Latent Space

Every breakthrough in AI has been led by a core scaling insight — Moore’s Law gave way to Huang’s Law (silicon), Kaplan et al gave way to Hoffman et al (data1), AlexNet kicked off the deep-learning and GPU for ML revolution (pretraining). As the o1 launch revealed, and with DeepSeek, Anthropic and GDM following shortly after, we’re now solidly in the era of scaling test time compute.

Noam Brown is one of the world’s leading researchers on reasoning (joint credit shared on the o1 launch video and the o1 System Card), and you may have heard him on the TED stage, OpenAI videos, and many of the world’s top AI podcasts, so we’re honored that he took some time to join ours to go deeper for an AI Engineer audience. We’ll stop gassing him up and just get right into the top highlights from today’s pod.

Our notes below:

• Reasoning is emergent: The “Thinking Fast and Slow” System 1 vs System 2 models for Non-reasoning vs Reasoning models, as well as the asymmetry of 1 unit of test time compute being the equivalent of 1000-10,000x more in size, is well understood now, but what is LESS understood is that it could only have happened after GPT4. As Jim Fan said, you have to scale both.

  “One thing that I think is underappreciated is that the models, the pre-trained models need a certain level of capability in order to really benefit from this extra thinking. This is kind of why you, you seen the reasoning paradigm emerge around the time that it did. I think it could have happened earlier, but if you try to do the reasoning paradigm on top of GPT-2, I don't think it would have gotten you almost anything… if you ask a pigeon to think really hard about playing chess, it's not going to get that far. It doesn't matter if it thinks for a thousand years, it's like not gonna be able to be better at playing chess. So maybe you do still also, also in like with animals and humans that you need a certain level of intellectual ability, just in terms of System 1 in order to benefit from System 2 as well.”

  It’s not well known (nor confirmed) that, after GPT-3, Ilya worked on a project codenamed GPT-Zero in 2021 to explore test time compute. One surprise we had in the pod was that Ilya actually convinced Noam that reasoning LLMs were more within reach than he thought; instead of the other way around:

  “…if we had a quadrillion dollars to train these models, then maybe we would, but like, you're going to hit the limits of what's economically feasible before you get to super intelligence, unless you have a reasoning paradigm. And I was convinced incorrectly that the reasoning paradigm would take a long time to figure out because it's like this big unanswered research question. Ilya agreed with me and he said I think we need this additional paradigm, but his take was that, maybe it's not that hard.”

  The (unablated) hypothesis is that we could not even have gone from GPT3 → o1, we needed GPT4 and 4o first as a baseline.

• Reasoning helps alignment. Safety, steerability, and alignment are very hot topics for certain parts of the AI community, and surprisingly reasoning helps:

  “After we released Cicero, a lot of the AI safety community was really happy with the research and like the way it worked because it was a very controllable system. Like we conditioned Cicero on certain concrete actions and that gave it a lot of steerability to say like, okay, well, it's going to pursue a behavior that we can like very clearly interpret. And it very clearly defined, it's not just a language model running loose and doing whatever it feels like. No, it's actually like pretty steerable. And there's this whole reasoning system that steers the way the language model interacts with the human. Actually, a lot of researchers reached out to like reached out to me and said we think this is like potentially a really good way to achieve safety with these systems.”

• Reasoning generalizes beyond verifiable rewards. One criticism of RLVR is that it only ever improves models in math and coding domains. Noam answers:

  “I'm surprised that this is such a common perception because we've released Deep Research and people can try it out. People do use it. It's very popular. And that is very clearly a domain where you don't have an easily verifiable metric for success… And yet these models are doing extremely well at this domain. So I think that's an existence proof that these models can succeed in tasks that don't have as easily verifiable rewards.”

• Visual Reasoning has limits. There has been a lot of excitement over how O3 beats Master-level Geoguessr Players. There are limits though:

  “It depends on exactly the kinds of questions that you're asking. Like there are some questions that I think don't really benefit from system 2. I think GeoGuessr is certainly one where you do benefit… the thing I typically point to you is information retrieval. If somebody asks you “when was this person born" and you don't have access to the web, then you either know it or you don't. And you can sit there and you can think about it for a long time. Maybe you can make an educated guess… but you're not going to be able to get the date unless you actually just know it.”

• Reasoning was underestimated by nonbelievers at OpenAI but borne out of hitting the Data Wall.

  “There was a lot of debate about, first of all, like, what is that extra paradigm? So I think a lot of the researchers looked at reasoning and RL but it was not really about scaling test time compute. It was more about data efficiency, because, you know, the feeling was that we have tons and tons of compute, but we actually are more limited by data. So there's the data wall and we're going to hit that before we hit limits on the compute. So how do we make these algorithms more data efficient? They are more data efficient, but I think that also they are also just like the equivalent of scaling up compute also by a ton… And I remember it was interesting that I talked to somebody who left OpenAI after we had discovered the reasoning paradigm, but before we announced o1 and they ended up going to a competing lab. I saw them afterwards after we announced it, and they told me that, at the time, they really didn't think the strawberry models were that big of a deal. They thought we were making a bigger deal of it than it really deserved to be. And then when we announced o1 and they saw the reaction of their coworkers at this competing lab about how everybody was like this is a big deal. And they like pivoted the whole research agenda to focus on this… a lot of this seems obvious in retrospect, but at the time it's actually not so obvious and it can be quite difficult to recognize something for what it is.”

• Reasoning + Windsurf = Feeling the AGI.

  Q: Any Windsurf pro tips now that you've immersed in it?

  A: I think one thing I'm surprised by is how many people don't even... know that O3 exists. Like, I've been using it day to day. It's basically replaced Google search for me. Like, I just use it all the time. And also for things like coding, I tend to just use the reasoning models. My suggestion is if people have not tried the reasoning models yet, because honestly, people love them. People that use it love them. Obviously, a lot more people use GPT-4.0 and just the default on ChatGPT and that kind of stuff. I think it's worth trying the reasoning models. I think people would be surprised what they can do.”

• Test Time Compute will have challenges in scaling2

  “We're going to get them thinking instead of three minutes, they're for three hours, and then three days, and then three weeks. There's two concerns:

  • One is that it becomes much more expensive to get the models to think for that long or scale up test-time compute. Like, as you scale up test-time compute, you're spending more on test-time compute, which means that, like, there's a limit to how much you could spend. That's one potential ceiling.

    • I should say that these models are becoming more efficient in the way they're thinking, and so they're able to do more with the same amount of test-time compute. And I think that's a very underappreciated point, that it's not just that we're getting these models to think for longer.

  • The second point is that, like, as you have these models think for longer, you get bottlenecked by wall-clock time. It is really easy to iterate on experiments when these models would respond instantly. It's actually much harder when they take three hours to respond.

    • And what happens when they have three weeks? It takes you at least three weeks to do those evaluations and to then iterate on that. And on a lot of this, you can parallelize experiments to some extent, but a lot of it, you have to run the experiment, complete it, and then see the results in order to decide on the next set of experiments. I think this is actually the strongest case for long timelines, that the models, because they just have to do so much.

  And I think it depends on the domain. So drug discovery, I think, is one domain where this could be a real bottleneck. I mean, if you want to see if something, like, extends human life, it's going to take you a long time to figure out if, like, this new drug that you developed, like, actually extends human life and doesn't have, like, terrible side effects along the way.”

Other commentators have also considered that data for long horizon RL is further away than people think.

Still, the age of test time scaling could not have come at a better (or earlier) time, just as the Orion run is estimated to have maxed out compute until SG1 comes online in December.

There’s been a lot of debate about Multi-Agents recently, with Cognition saying Don’t Build Multi-Agents and Anthropic saying how to build Multi-Agents. There have been many, many, many takes on the debate, but Noam has been doing multi-agent RL for years and has announced the multi-agent team at OpenAI… although that is just the most prominent of a few possible research directions…

“I think the team [name], in many ways, is actually a misnomer, because we're working on more than just multi-agent. Multi-agent is one of the things we're working on.

Some other things we're working on is just like being able to scale up test time compute by a ton. So how, you know, we get these models thinking for 15 minutes now. How do we get them to think for hours? Days, even longer? And be able to solve incredibly difficult problems. So that's one direction that we're pursuing.

Multi-agent is another direction. And here, I think there's a few different motivations. We're interested in both the collaborative and the competitive aspect of multi-agent.

I think the way that I describe it is people often say in AI circles that humans occupy this very narrow band of intelligence. And AIs are just going to, like, quickly catch up. And then surpass, like, this band of intelligence. And I actually don't think that the band of human intelligence is that narrow. I think it's actually quite broad. Because if you compare anatomically identical humans from, you know, caveman times, they didn't get that far in terms of what we would consider intelligence today, right? Like, they're not putting a man on the moon, you know, they're not building semiconductors or nuclear reactors or anything like that. And we have those today, even though we as humans are not. And so, what's the difference? Well, I think the difference is that you have thousands of years, a lot of humans, billions of humans, cooperating and competing with each other, building up civilization over time. The technology that we're seeing is the product of this civilization.

And I think similarly, the AIs that we have today are kind of like the cavemen of AI. And I think that if you're able to have them cooperate and compete with billions of AIs over a long period of time and build up a civilization, essentially, the things that they would be able to produce and answer would be far beyond what is possible today with the AIs that we have today.”

Bitter Lesson in Multi-Agents:

“…the way that we are approaching multi-agent in the details and the way we're actually going about it is, I think, very different from how it's been done historically and how it's being done today by other places. I've been in the multi-agent field for a long time… I think that a lot of the approaches that have been taken have been very heuristic and haven't really been following, like, the bitter lesson approach to scaling and research.”

Bitter Lesson vs World Models & Yann LeCun:

“… I think it's pretty clear that as these models get bigger, they have a world model, and that world model becomes better with scale. So, they are implicitly developing a world model, and I don't think it's something that you need to explicitly model… there was this long debate in the multi-agent AI community for a long time about whether you need to explicitly model other agents, like, other people, or if they can be implicitly modeled as part of the environment.

For a long time, I was, like, on the, took the perspective of, like, of course you have to, like, explicitly model these other agents because they're behaving differently from the environment. Like, they take actions, they're unpredictable, you know, they have agency. But I think I've actually shifted over time to thinking that actually, if these models become smart enough, they develop things like theory of mind. They develop an understanding that they're... That they're agents that can take actions and have motives and all this stuff. And these models just develop that implicitly with scale and more capable behavior broadly. So, that's the perspective I take these days.”

Combining Open-Endedness, Multi-Agents, and Self-Play: OpenAI has written about the Weak to Strong problem, and Tim Rocktaschel, head of OpenEndedness at GDM, gave a very well received keynote at ICLR in Singapore (full video here) which inspired a question about the relationship between multi-agent scaling beyond human capabilities (the ultimate limiter in the Bitter Lesson):

“Q: One of the most consistent findings is always that it's better for AIs to self-play and improve competitively, as opposed to sort of humans training and guiding them. And you find that with, AlphaZero and R1 zero. Do you think this will hold for multi-agents, like, self-play to improve better than humans?

A: So, this is a great question. And... I think this is, worth expanding on. So, I think a lot of people today see self-play as the next step and maybe the last step that we need for superintelligence.

And I think if you look at something like AlphaGo and AlphaZero, we seem to be following a very similar trend, right?

• Like, the first step in AlphaGo was you do large-scale pre-training. In that case, it was on human Go games. With LLMs, it's pre-training on, you know, tons of, like, internet data. And that gets you a strong model, but it doesn't get you a superhuman model.

• And then the next step in the AlphaGo paradigm is you do large-scale test time compute or, like, large-scale inference compute. And in that case with MCTS. And now we have, like, reasoning models that also do, like, this large-scale inference compute. And, again, that, like, boosts the capabilities a ton.

• Finally, with AlphaGo and AlphaZero, you have self-play, where the model plays against itself, learns from those games, gets better and better and better, and just goes from something that's, like, around human-level performance to, like, way beyond human capability. These Go policies now are so strong that it's just incomprehensible. Like, what they're doing is incomprehensible to humans. Same thing with chess. And we don't have that right now with language models. And so it's, like, it's really tempting to look at that and say, like, oh, well, we just need these, like, AI models to now interact with each other and learn from them. And then they're just going to, like, get to superintelligence.

… The challenge is that Go is this two-player zero-sum game. And two-player zero-sum games have this very nice property where when you do self-play, you are converging to a minimax equilibrium. And I guess I should take a step back and say, like, in two-player zero-sum games, two-player zero-sum games are chess, Go, even two-player poker, all two-player zero-sum. Well, that's not true. What you typically want is what's called a minimax equilibrium. This is that GTO policy, this policy that you play where you're guaranteeing that you're not going to lose to any opponent in expectation. I think in chess and Go, that's, like, pretty clearly what you want. Interestingly, when you look at poker, it's not as obvious. In a two-player zero-sum version of poker, you could play the GTO minimax policy, and that guarantees that you won't lose to any opponent on Earth. But, again, I mentioned you're not going to make as much money off of a weak player as you could if you instead played an exploitative policy. So, there's this question of, like, what do you want? Do you want to make as much money as possible, or do you want to guarantee that you're not going to lose to any human alive? What all the bots have decided is, like, well, what all the, like, AI developers in these games have decided is, like, well, we're going to choose the minimax policy. And, conveniently, that's exactly what self-play converges to. If you have these AIs play against each other, learn from their mistakes, they converge over time to this minimax policy, guaranteed. But once you go outside a two-player zero-sum game, that's actually not a useful policy anymore. You don't want to just, like, have this very defensive policy, and you're going to end up with really weird behavior if you start doing the same kind of self-play in things like math.”

From beating everyone at Poker, to ranking top 10% in the world Diplomacy with LLMs, to personally winning the World Diplomacy Championship himself, games are a huge part of Noam’s thinking and career. But not just any kind of game…

“…I have this like huge store of knowledge on AI for imperfect information games. Like this is my area of research for so long. And I know all these things, but I don't get to talk about it very often.

We've made superhuman poker AIs for No Limit Texas Hold'em. One of the interesting things about that is that the amount of hidden information is actually pretty limited because you have two hidden cards when you're playing Texas Hold'em. And so the number of possible states that you could be in is 1,326 when you're playing heads up at least. And you know, that's multiplied by the number of other players that there are at the table, but it's still like not a massive number…

The problem is that as you scale the number of hidden possibilities, like the number of state of, of possible states you could be in that approach breaks down. And there's still this very interesting unanswered question of what do you do when the number of hidden states becomes extremely large? You know, so if you go to Omaha poker, where you have four hidden cards, there are things you could do. That's the kind of heuristic that you could do to reduce the number of states, but actually it's still a very difficult question.

And then if you go to a game like Stratego where you have 40 pieces, so there's like close to 40 factorial different states you could be in. Then all these existing approaches that we used for poker break down and you need different approaches…. If you expand those techniques, maybe you get them to work on things like Stratego and Magic the Gathering, but they're still going to be limited. They're not going to get you like superhuman encode forces with language models.

So I think it's more valuable to just focus on the very general reasoning techniques. And one day as we improve those, I think we'll have a model that just out of the box one day plays Magic the Gathering at a superhuman level. And I think that's the more important and more impressive research direction.

Side note: the LLMs self playing Diplomacy challenge Noam asks for in the pod was shipped at the AIE World’s Fair right after :)

It was a huge honor to host Noam for the pod and we hope you listen to the full thing to pick up even more of the tangents, nuggets, and advice! Tell us if we missed something big.

• 00:00 Intro – Diplomacy, Cicero & World Championship

• 02:00 Reverse Centaur: How AI Improved Noam’s Human Play

• 05:00 Turing Test Failures in Chat: Hallucinations & Steerability

• 07:30 Reasoning Models & Fast vs. Slow Thinking Paradigm

• 11:00 System 1 vs. System 2 in Visual Tasks (GeoGuessr, Tic-Tac-Toe)

• 14:00 The Deep Research Existence Proof for Unverifiable Domains

• 17:30 Harnesses, Tool Use, and Fragility in AI Agents

• 21:00 The Case Against Over-Reliance on Scaffolds and Routers

• 24:00 Reinforcement Fine-Tuning and Long-Term Model Adaptability

• 28:00 Ilya’s Bet on Reasoning and the O-Series Breakthrough

• 34:00 Noam’s Dev Stack: Codex, Windsurf & AGI Moments

• 38:00 Building Better AI Developers: Memory, Reuse, and PR Reviews

• 41:00 Multi-Agent Intelligence and the “AI Civilization” Hypothesis

• 44:30 Implicit World Models and Theory of Mind Through Scaling

• 48:00 Why Self-Play Breaks Down Beyond Go and Chess

• 54:00 Designing Better Benchmarks for Fuzzy Tasks

• 57:30 The Real Limits of Test-Time Compute: Cost vs. Time

• 1:00:30 Data Efficiency Gaps Between Humans and LLMs

• 1:03:00 Training Pipeline: Pretraining, Midtraining, Posttraining

• 1:05:00 Games as Research Proving Grounds: Poker, MTG, Stratego

• 1:10:00 Closing Thoughts – Five-Year View and Open Research Directions

Alessio [00:00:04]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host Swyx, founder of SmolAI.

swyx [00:00:12]: Hello, hello, and we're here recording on a holiday Monday with Noam Brown from OpenAI. Welcome. Thank you. So glad to have you finally join us. A lot of people have heard you. You've been rather generous of your time on the podcast, Lex Friedman, and you've done a TED Talk recently just talking about the thinking paradigm. But I think maybe perhaps your most interesting recent achievement is winning the World Diplomacy Championship. Yeah. In 2022, you built like sort of Cicero, which was top 10% of human players. I guess my opening question is, how has your diplomacy playing changed since working on Cicero and now personally playing it?

Noam [00:00:52]: When you work on these games, you kind of have to understand the game well enough to like be able to debug your bot. Because if the bot does something that's like really radical and like that he most typically wouldn't... You're not sure if that's like a mistake or if that's just like if it's a bug in the system or it's actually just like the bot being brilliant. When we were working on Diplomacy, I kind of like did this deep dive, like trying to understand the game better. I played in tournaments. I like watched a lot of like tutorial videos and commentary videos on games. And over that process, I got better. And then also seeing the bot, like the way it would behave in these games. Like sometimes it would do things that humans typically wouldn't do. And that taught me about the game as well. When we released Cicero, we announced it in like late 2022. I still found the game like really fascinating. And so I like kept up with it. I like continue to play. And that led to me winning the championship in the World Championship in 2025.

swyx [00:01:45]: So just a couple of months ago. There's always a question of like centaur systems where humans and machines work together. Like, was there an equivalent of what happened in Go where you updated your play style?

Noam [00:01:55]: If you're asking if I used Cicero when I played in the tournament, the answer is no. Seeing the way the bot played and like... Taking inspiration from that, I think, did help me in the tournament. Yeah. Yeah.

swyx [00:02:06]: Do people now ask Turing questions every single time when they're playing Diplomacy? Ask. To try to tell like if the person they're playing with is a bot or a human. Yeah. Like that's the one thing you're worried about when you started.

Noam [00:02:19]: It was really interesting when we were working on Cicero because like, you know, we didn't have the best language models. We were really bottlenecked on the quality of the language models. And sometimes the bot would do, would say like bizarre things. Like, you know, 99% of the time it was fine. But then like every once in a while it would say this like really bizarre thing, like it would just hallucinate about something. Somebody would reference something that they said earlier in a conversation with the bot and the bot would be like, I have no idea what you're talking about. I never said that. And then the person would be like, look, you can just scroll up in the chat and it's literally right there. And the bot would be like, no, you're lying. Oh, context windows. And when it does these kinds of things, like people just kind of like shrugged it off as like, oh, that's just, you know, the person's tired or they're drunk or whatever. Or they're just like trolling me. But I think like, that's because people weren't looking for a bot. They weren't expecting a bot to be in the games. We were actually really scared because we were afraid that people would, would figure out at one point that there's a bot in these games and then they would just like always be on the lookout for it. And they would always be. And if you're, if you're looking for it, you're able to spot it. That's the thing. So I think now that it's announced and that people know to look for it, I think they would have an easier time spotting it. Now that said, the language models have also gotten a lot better since 2022. It's adversarial. Yeah. So at this point, like, you know, the truth is, you know, GPD 4.0. And like 0.3, these models are like passing the Turing test. So I don't think they can really ask that many Turing complete questions that would actually make a difference.

Alessio [00:03:39]: And T-Zero was very small, like 2.7B, right?

Noam [00:03:42]: It was a very small language model. Yeah. It was one of the things we realized over the course of the project that like, oh yeah, you, you really benefit a lot from just having like larger language models. Right.

Alessio [00:03:50]: Yeah. How do you think about today's perception of AI and a lot of like maybe the safety discourse of like, you know, you're going to build a bot that is really good at persuading people into like helping them win. Right. Yeah. A game. And I think maybe today labs want to say they don't work on that type of problem. How do you think about that dichotomy, so to speak, between the two?

Noam [00:04:08]: You know, honestly, like after we released Cicero, a lot of the AI safety community was really happy with the research and like the way it worked because it was a very controllable system. Like we conditioned Cicero on certain concrete actions and that gave it a lot of steerability to say like, okay, well, it's going to pursue a behavior that we can like very clearly interpret. And it very clearly defined, it's not just like, oh, it's a language model, like running loose and doing whatever it feels like. No, it's actually like pretty steerable. And there's this whole reasoning system that steers the way the language model interacts with the human. Actually, a lot of researchers reached out to like reached out to me and said, like, we think this is like potentially a really good way to achieve safety with these systems.

swyx [00:04:50]: I guess the last diplomacy related questions that we might have is, have you updated or tested like O series models on diplomacy? And would you, would you expect. Yeah. A lot more difference.

Noam [00:05:01]: I have not. I think I said this on Twitter at one point that I think this would be a great benchmark. I would love to see all the leading bots play a game of diplomacy with each other and see like who does best. And I think a couple of people have like taken inspiration from that and are actually like building out these benchmarks and like evaling the models. My understanding is that they don't do very well right now. Um, but I think, I think it really is a fascinating benchmark and I think it would be, um, yeah, I think it'd be a really cool thing to try out.

swyx [00:05:25]: Well, we're going to go a little bit into O series now. I think the last time you did a lot of publicity, you were just. Just launching O one, you did your Ted talk and everything. How has the vibe, how have the vibes changed just in general? You said you were very excited to learn from domain experts, like in chemistry, like how they review the old series models. Like how have you updated since let's say end of last year?

Noam [00:05:48]: I think the trajectory was pretty clear, pretty early on in the development cycle. And I think that everything that's unfolded since then has been pretty on track for what I expected. So I wouldn't say that my perception of where, where things are going or has honestly changed that much. Um, I think that we're going to continue to see, as I said before, that we're going to see this paradigm continue to progress rapidly. And I think that that's true even today that we, we saw that with like going from O one preview to O one to O three consistent progress. And we're going to continue to see that going forward. And I think that we're going to see a broadening of what these models can do as well. You know, like we're going to start seeing agentic behavior. Um, we're already starting to see that. We're starting to see agentic behavior, like honestly, for me, O three, I've been using it a ton in my day-to-day life. I just find it so useful, especially the fact that I can now browse the web and like, you know, do meaningful research on my behalf. Like it's kind of like a mini deep research that you can just get a response in three minutes. So yeah, I think it's just going to continue to become more and more useful and more powerful as time goes on and pretty quickly.

Alessio [00:06:51]: Yeah. And talking about deep research, you tweeted about if you need proof that we can do this in unverifiable domains, deep research is kind of like a great example. Can you maybe talk about. If there's something that people are missing, you know, I feel like I hear that repeated a lot. It's like, you know, it's easy to do in coding and math, but like not in these other domains.

Noam [00:07:07]: I frequently get this question, including from pretty established AI researchers that, okay, we're, we're seeing these like reasoning models exceed in math and coding and these, these easily verifiable domains, but are they ever going to succeed in domains where success is less well-defined? I'm surprised that this is such a common perception because we've released deep research and people can try it out. People do use it. It's very popular. And that is very clearly a domain where you don't have an easily verifiable metric for success. It's very like, what, what is the best research report that you could generate? And yet these models are doing extremely well at this, at this domain. So I think that's like an existence proof that these models can succeed in tasks that don't have as easily verifiable rewards.

Alessio [00:07:51]: Is it because there's also not necessarily like a wrong answer? Like there's a spectrum of deep research quality, right? You can have like a report that like looks good. But the information is kind of so-and-so, and then you have a great report. Do you think people have a hard time understanding the difference when they get the result?

Noam [00:08:07]: My impression is that people do understand the difference when they get a result. And I think that they're surprised at how good the deep research results are. There's certainly, it's not, not a hundred percent. It could be better and we're going to make it better. But I think people can tell the difference between a good report and a bad report. And certainly in a good report and a mediocre report.

Alessio [00:08:24]: And that's enough to kind of feed the loop later to build the product and improve the model performance.

Noam [00:08:29]: I mean, I think if you're in a situation where people can't tell the difference between the outputs, then it doesn't really matter if you're like, you know, hill climbing on, on progress. These models are going to get better at domains where there is a measure of success. Now, I think this idea that it has to be like easily verifiable or something like that. I don't think that's true. I think that you can have, you can have these models do well, even in domains where success is a very difficult to define thing could sometimes even be subjective.

swyx [00:08:56]: People lean on a lot. You've, you've done as well is the thinking. Fast and slow analogy for just thinking models. And I think it's reasonably well diffused now. The idea of that, that this is kind of the next scaling paradigm. All analogies are imperfect. What is one way in which thinking fast and slow or system one system two kind of doesn't transfer to how we actually scale these things.

Noam [00:09:21]: One thing that I think is underappreciated is that the models, the pre-trained models need a certain level of capability in order to really benefit from this. Like. Extra thinking. This is kind of why you, you seen the reasoning paradigm emerge around the time that it did. I think it could have happened earlier, but if you try to do the reasoning paradigm on top of GPT-2, I don't think it would have gotten you almost anything. Is this emergence? Hard to say if it's emergence necessarily, but like I haven't done the you know, the measurements to really define that clearly. But I think it's pretty clear, you know, people try chain of thought with GPT, like really small models and they saw that it just didn't really do. Anything. Then you go to bigger models and it starts to, to give a lift. I think there's a lot of debate about like the extent to which this kind of behavior is emergent, but clearly there is a difference. So it's not like there are these two independent paradigms. I think that they are related in the sense that you need a certain level of system one capability in your models in order to have system two is to be able to benefit from system two. Yeah.

swyx [00:10:22]: I have played, tried to play amateur neuroscientist before and try to compare it to the evolution of the brain and how you have to. Evolve the cortex first before you evolve the other parts of the brain. And perhaps that is what we're doing here. Yeah.

Noam [00:10:36]: And you could argue that actually this is not that different from like, I guess the, um, the system one system two paradigm, because, you know, if you ask like a pigeon to think really hard about playing chess, you know, it's not going to get that far. It's, you know, it doesn't matter if it's like things for a thousand years, it's like not gonna be able to be better at playing chess. So maybe you do still also, also in like with animals and humans that you need a certain level of intellectual ability, just in terms of system one in order to benefit from system two as well. Yeah.

swyx [00:11:00]: Just as a side tangent, does this also apply to visual reasoning? So let's say we have, now we have the 4.0, like natively omni model type of thing, then that also makes O3 really good at GeoGuessr. Does that apply to other modalities too?

Noam [00:11:17]: I think the evidence is yes. It depends on exactly the kinds of questions that you're asking. Like there are some questions that I think don't really benefit from system two. I think GeoGuessr certainly one, uh, where you, where you do benefit. I think. Image recognition. If, if I had to guess, it's like one of those things that you probably benefit less from system two thinking. Cause you know it or you don't. Yeah, exactly. There's no way. Yeah. And the, the thing, the thing I typically point to you is just like information, like retrieval. If somebody asks you like, when was this person born and you don't have access to the web, then you either know it or you don't. And you can sit there and you can think about it for a long time. Maybe you can make an educated guess. So you can say like, well, this person was like probably lived around this time. And so this is like a rough date, but you're not going to be able to like get the date unless you've actually. Just, just know it.

swyx [00:12:01]: But like spatial reasoning, like tic-tac-toe might be better. Cause you have all the information there. Yeah.

Noam [00:12:06]: And I think it's true that like with tic-tac-toe, we see that like GBD 4.5 falls over, you know, it plays decently. Well, I shouldn't say it falls over. It does reasonably well control the board. It can make legal moves, but it will make mistakes sometimes. And if you really need that system too, to enable it to play perfectly. Now it's possible that if you got to GBD six and you just did system one, it would also play perfectly. You know, I guess we'll, we'll know why. One day, but I think right now you would need a system too, to really like do well.

Alessio [00:12:34]: What do you think are like the things that you need in system one? So obviously general understanding of like game rules. Do you also need to understand some sort of like metagame of like, you know, usually this is like how you value pieces in different games, even though it's a, you know, how do you generalize in system one? So that then in system two, you can kind of get to the gameplay, so to speak.

Noam [00:12:53]: I think the more that you have in your, in, in the system one, like this is the same thing with humans, you know? Like. Like humans are when they're playing for the first time, a game like chess, they can apply a lot of system to thinking to it. And if you, if you apply a ton of system to thinking to it, like if, if you just present a really smart person with a completely novel game and you tell them like, okay, you're going to play this game against like an AI or like a human that's like mastered this game and you tell them to like sit there and think about it for like three weeks about how to play this game. My guess is they could actually do pretty well, but it certainly helps to build up that system one thinking. Like build up intuition about, about the game, because it will just make you so much, yeah, so much faster.

Alessio [00:13:35]: I think the Pokemon example is a good one of like the system one kind of has maybe all this information about games. And then once you put it in the game, it still needs a lot of harnesses to work. And I'm trying to figure out how much of, can we take things from the harness and have them in system one so that then system two is as harness free as possible.

Noam [00:13:53]: But I guess that's like the question about generalizing games and an AI. Yeah, I guess I view that as a different question. I think the question about like harnesses, in my view, is that the ideal harness is no harness. Right. I think harnesses are like a crutch that eventually we're going to be able to move beyond. So only two calls. And you could ask, you know, you could just ask O3. And actually, you know, it's interesting because like when this playing Pokemon thing kind of like emerged as this like, you know, benchmark. I was actually like pretty opposed to evaling this with our, with our like opening eye models, because my feeling is like, okay, if we're going to do this eval, let's just do it with O3. You know, how far does O3 get without any harness? How far does it get playing Pokemon? And the answer is like not very far, you know, and that's fine. I think it's fine to have an eval where the models do terribly. And I don't think the answer to that should be like, well, let's build a really good harness so that now it can do well on this eval. I think the answer is like, okay, well, let's just like improve the capabilities of our models so they can do well at everything. And then they also happen to make progress on this eval.

Alessio [00:14:56]: Would you consider things like checking? I came for a valid move, a harness, or is this in the, in the model, you know, like chess is like, you can either have the model learn in system one, what moves are valid and what it can and cannot do versus in system two, figuring out.

Noam [00:15:11]: I think, I think there's like, a lot of this is design questions. Like for me, I think you should give the model the ability to check if a move is legal. If you want, like that, that could be an option in the environment of like, okay, here's a, you know, an action that you can like a tool call that you can make to see if an action is legal. If it wants to use that, it can. And then there's like, design is a question of like, well, what do you do if the model makes an illegal move? And I think it's totally reasonable to say like, well, if they make an illegal move, then they lose the game. Like, I don't know what, what happens when a human makes an illegal move in a game of chess? I actually don't know. I haven't played chess that much. Yeah, me neither. You're just not allowed to? Yeah. Like, do you just lose the game? I don't know.

swyx [00:15:44]: So if that's, if that's the case, then I think it's totally reasonable to say like, yeah, we're going to have an eval where that's also the criteria for the AI models. Yeah. But I think like, maybe one way to interpret that in sort of researcher terms is, are you allowed to do search? And one of the famous findings from DeepSeek is that MCTS wasn't that useful to them. But I think like, there are a lot of engineers trying out search and spending a lot of tokens doing that. And maybe it's not worth it.

Noam [00:16:08]: Well, I'm making a distinction here between like a tool call to check whether a move is legal or illegal is different from actually making that move and then seeing whether it ended up being legal or illegal. So if that tool call is available, I think it's totally fine to make that tool call and check whether a move is legal or illegal. I think it's different to have the model say, oh, I'm making this move. Yeah. And then, you know, it gets feedback that like, oh, you made an illegal move. And so then it's like, oh, just kidding. Like, I'm going to do something else now. So, so that's, that's the distinction I'm drawing.

swyx [00:16:40]: Some people have tried to classify that second type of playing things out as test time compute. You would not classify that as test time compute.

Noam [00:16:48]: There's a lot of reasons why you would not want to rely on that paradigm. When you're going to imagine you have a robot, you know, and your robot like take some action in the world and it like breaks something and you're just like, oh, you can't say like, oh, just kidding. I didn't mean to do that. I'm going to undo that action. Like the thing is broken. So if you want to simulate what would happen if I move the robot in this way, and then in the simulation, you saw that this thing broke and then you decide not to do that action. That's totally fine. But you can't just like undo actions that you've taken in the world.

swyx [00:17:14]: There's a couple more things I wanted to cover in this rough area. I actually had an answer on the, on the thinking fast and slow side, which maybe. I'm curious what you think about, like a lot of people are trying to put in effectively model router layers, let's say between like the, the fast response model and the, the long thinking model, Anthropik is explicitly doing that. And I think there's a question about always, do you need a smart judge to route or do you need a dumb judge, judge to route because it's fast. So when you have a model router, let's say, let's say you're passing requests between system one side and system two side, does the router need to be as smart as the smart model or.

Noam [00:17:51]: I think it's possible for a dumb model to recognize that a problem is really hard and that it won't be able to solve it and then route it to a more capable model.

swyx [00:18:01]: But it's also possible for a dumb model to be fooled or to be overconfident.

Noam [00:18:05]: I don't know. I think there's a real trade off there, but I will say like, I think, I think there are a lot of things that people are building right now that will eventually be washed away by scale. So I think harnesses are a good example where I think eventually the models are going to be. And I think this actually happened with the reasoning models. Like before the reasoning models emerged, there was like all of this work that went into engineering, these like agentic systems that like made a lot of calls to GPT-4-0 or like these non-reasoning models to get reasoning behavior. And then it turns out like, oh, we just like created reasoning models and they, you don't need this like complex behavior. In fact, in many ways, it makes it worse. Like you just give the reasoning model the same question without any sort of scaffolding and it just does it. No. You can still, and so people are building scaffolding on top of the reasoning models right now. But I think in many ways, like those scaffolds will also just be replaced by the reasoning models and models in general becoming more capable. And similarly, I think things like model, like these routers, you know, we've said pretty openly that we want to move to a world where there is a single unified model. And in that world, you shouldn't need a router on top of the model. So I think that the router issue will eventually be solved also.

swyx [00:19:18]: Like you're building the router into. The model kind of weights itself.

Noam [00:19:22]: I don't think there will be a benefit for like, I shouldn't say it because it's, I could be wrong about this. Like, you know, it's certainly maybe there's, you know, reasons to route to different model providers or whatever. But I think that routers are going to eventually go away. And I can understand why it's worth doing it in the short term, because like the fact is, it is beneficial right now. And if you're building a product and you're getting a lift from it, then. It's worth doing right now. One of the tricky things I'd imagine that a lot of developers are facing is that like, you kind of have to plan for where these models are going to be in six months and 12 months. And that's like very hard to do because things are progressing very quickly. You know, you don't want to spend six months building something and then just have it be totally washed away by scale. But I think I would encourage developers like when, when they're, you know, building these kinds of things like scaffolds and, and routers, keep in mind that the field is evolving very rapidly. You know, things are going to change in three months, let alone six months. And. And that might require radically changing these things around or, or tossing them out completely. So don't spend six months building something that might get tossed down in six months.

swyx [00:20:29]: It's so hard though. Everyone says this, and then like, no one has concrete suggestions on how.

Alessio [00:20:36]: What about reinforcement fine-tuning? Is this something that obviously you just released it a month ago at UrbanEye, is this something people should spend time on right now, or maybe wait until the next jump?

Noam [00:20:46]: I think reinforcement fine-tuning is pretty cool. And I think it's like worth looking at. I think it's something that's worth looking into because it's really about specializing the models for the data that you have. And I think that, um, something that's like worth, worth looking into for, for developers, like we're not, we're not suddenly going to like have that data baked into the raw model a lot of times. So I think that's kind of like a separate question. Yeah.

Alessio [00:21:11]: So creating the environment and the reward model is the best thing people can do right now. I think the question that people have is like, should I rush to fine-tune the, the model? Using RFT or should I build the harness to then RFT the models as they get better?

Noam [00:21:25]: I think the difference is that like for reinforcement fine-tuning, you're collecting data that's going to be useful as the models improve as well. So if we come out with like future models that are even more capable, you could still fine-tune them on your data. That's, I think actually a good example where you're building something that's going to compliment the model scaling and becoming more capable rather than necessarily getting washed away by the scale.

swyx [00:21:50]: One last question on Ilya, you mentioned on, I think the Sarah and Elad podcast, where you had this conversation with Ilya a few years ago about more RL and reasoning and language models, just any speculation or thoughts on why his attempt, when he tried it, it didn't work or the timing wasn't right and why the time is right now.

Noam [00:22:14]: I don't think I would, I would frame it that way that like his, his attempt didn't work in many ways. It did. So Ilya, for me, I saw that in all of these domains that I'd worked on and poker and Hanabi and diplomacy, having the models think before acting made a huge difference in performance, like orders of magnitude difference. Like 10,000 times. Yeah. Like, you know, a thousand to a hundred thousand times, like it's the equivalent of a model that's like a thousand to a hundred thousand times bigger. And in language models, you weren't really seeing that, that the model, the models would just respond instantly. Some people in the field in the LLM. We're like convinced that like, okay, we just keep scaling pre-training. We're going to get to super intelligence. And I was kind of skeptical of that perspective in late 2021. I was having a meal with Ilya. He asked me what my AGI timelines are very standard SF question. And I told him like, look, I think it's actually quite far away because we're going to need to figure out this reasoning paradigm in a very general way. And with things like LLMs, LLMs are very general, but they don't have a reasoning paradigm. That's very general. And until they do, they're going to be limited in what they can do. You're like, we're going to say. Sure, we're going to scale these things up by a few more orders of magnitude. They're going to become more capable, but we're not going to see super intelligence from just that. And like, yes, if we had a quadrillion dollars to train these models, then maybe we would, but like, you're going to hit the limits of what's economically feasible before you get to super intelligence, unless you have a reasoning paradigm. And I was convinced incorrectly that the reasoning paradigm would take a long time to figure out because it's like this big unanswered research question. And, you know, Ilya agreed with me and he said like, yeah, you know, I think. We need this like additional paradigm, but his take was that like, maybe it's not that hard. I didn't know it at the time, but like he and others at OpenAI had also been thinking about this. They'd also been thinking about RL, they'd been working on it and I think they had some success, but like, you know, with most research, like it does, you have to iterate on things. You have to try out different ideas. You have to, yeah, try different things. And then also as the models become more capable, as they become faster, it becomes easier to iterate on experiments. And I think that the work that. They did, even though it didn't like result in a reasoning paradigm, it all builds on top of previous work. Right. So they built a lot of things that over, over time led to this reasoning paradigm.

swyx [00:24:31]: For listeners, no one can talk about this, but the rumor is that that thing was codenamed GPT zero. If you want to search for that, that line of work, I think there was a time where like basically RL kind of went through a dark age when everyone like went all in on it and then nothing happens and they gave up. And like now it's like sort of the golden age again. So that's what I'm like trying to identify. Like why, what is it? And it could just be that we have smarter base models and better data.

Noam [00:24:57]: I don't think it's just that we have smarter base models. I think it's that. Yeah. So I, we did end up getting a big success with, with reasoning and I, but I think it was in many ways a gradual thing to some extent it was gradual, you know, like there were signs that there were signs of life. And then we like, you know, iterated and tried out some more things. We got like better signs of life. I think it was around like November, 2023 or October, 2023. I think it was around like November, 2023 or October, 2023. When I think I was convinced that we had like very conclusive signs of life that like, oh, this was, this was going to be, this is the paradigm and it's going to be a big deal. That wasn't many ways a gradual thing. I think what OpenAI did well is like when we got those signs of life, they recognized it for what it was and invested heavily in, in scaling it up. And I think that's, that's ultimately what led to reasoning models arriving when they did.

Alessio [00:25:45]: Was there any disagreement internally, especially because like, you know, OpenAI constantly.

Noam [00:25:50]: I think that's also to the credit of OpenAI that like, okay, yes, they figured out the paradigm that was needed and that was why they were investing all this like research effort into this like RL stuff. And I think that's also to the credit of OpenAI that like, okay, yes, they figured out the paradigm. And they were very focused on scaling that up. In fact, the vast majority of resources were focused on scaling that up, but they also recognized the value that that's something else was going to be needed. And it was worth researching, putting researcher effort into, into other directions to figure out what that extra paradigm was going to be. There was a lot of debate about, first of all, like, what is that extra paradigm? So I think a lot of the researchers looked at reasoning and, and RL was not really about scaling test time compute. It was more about. Data efficiency, because, you know, the feeling was that, well, we have tons and tons of compute, but we actually are more limited by data. So there's, there's the data wall and we're going to hit that before we hit limits on, on the compute. So how do we make these algorithms more data efficient? They are more data efficient, but I think that also like, um, they are also just like the equivalent of scaling up compute also by a ton. That was interesting. There was like a lot of debate around like, okay, well, what exactly are we doing here? And then I think also, even when we got the signs of life, I think there was a lot of debate about the significance of it. That like, okay. How much should we invest in scaling up this paradigm? I think, especially when you're, when you're in a small company, like, you know, open AI, like in 2023 was not as big as it is today. And compute was more constrained than it is today. And if you're investing resources in a direction that's coming at the expense of something else. And so if you look at these signs of life on reasoning and you're saying like, okay, well, this looks promising. We're going to scale this up by a ton and invest a lot more resources into it. Where are those resources coming from? You have to make that tough call about where to, where to draw the resources from. And that is a very controversial, very difficult call to make, um, that makes some people unhappy. And I think there was debate about whether we're focusing too much on this paradigm, whether it's really a big deal, whether we would see it generalize and do various things. And I remember it was interesting that I talked to somebody who left open AI after we had discovered the reasoning paradigm, but before we announced a one and they ended up going to a competing lab. I saw them afterwards after we announced it. And they told me that like, at the time, they really didn't think this like reasoning thing, like this, these O series, the strawberry models were like that, that big of a deal. It was like, they thought we were making a bigger deal of it than it really deserved to be. And then when we announced a one and they saw the reaction of their coworkers at this competing lab about how everybody was like, oh crap, like this is a big deal. And they like pivoted the whole research agenda to focus on this, that then they realized like, oh, actually like this maybe is a big deal. You know, a lot of this seems obvious in retrospect, but at the time it's actually not so obvious and it can be quite difficult to recognize something for what it is.

Alessio [00:29:00]: I mean, open AI is like a great history of just making the right bet. I feel like GPD models are kind of similar, right? Where like it started with games, NRL, and then it's like, maybe we can just scale these language models instead. And I'm just impressed by the leadership and obviously the research team that keeps coming up with these insights.

Noam [00:29:19]: Looking back on it today, it might seem obvious that like, oh, of course, like these models get better with scale. So you should just scale them up a ton and it'll get better, but it really is, the best research is obvious in retrospect. And at the time it's not as obvious as it might seem today.

swyx [00:29:35]: Follow-up questions on data efficiency. This is a pet topic of mine. It seems that our current methods of learning are so inefficient still, right? Like compared to the existence proof of humans, we take five samples and we learn something. Right? Like compared to the existence proof of humans, we take five samples and we learn something. Right? Like compared to the existence proof of humans, we take five samples and we learn something. Machines, 200, maybe, you know, per like whatever data point you might need. Anyone doing anything interesting in data efficiency? Or do you think like there's just a fundamental inefficiency that machine learning has that will just always be there compared to humans?

Noam [00:30:05]: I think it's a good point that if you look at the amount of data these models are trained on and you compare it to like the amount of data that a human observes to get the same performance. I guess pre-training, it's a little hard to make an apples to apples comparison because like, I don't know, how many tokens does a baby actually absorb? I guess pre-training, it's a little hard to make an apples to apples comparison because like, I don't know, how many tokens does a baby actually absorb when they're developing? But I think it's a fair statement to say that these models are less data efficient than humans. And I think that that's an unsolved research question and probably one of the most important unsolved research questions.

swyx [00:30:32]: Maybe more important than algorithmic improvements, because you can just, we can increase the supply of data out of the existing set of the world and humans.

Noam [00:30:42]: I guess that's good. So a couple of thoughts on that. Like one is that the answer might be an algorithmic improvement. Like maybe algorithmic improvements do lead to greater efficiency. Like maybe algorithmic improvements do lead to greater efficiency. And the second thing is that like, it's not like humans learn from just reading the internet. So I think it's certainly easiest to learn from just like data that's on the internet, but I don't think that's like the limit of what data you could collect.

swyx [00:31:05]: The last follow up before we change topics to coding, any other just anecdotes or insights from Ilya, just in general? Because like you've worked with him, so there's not that many people that we can talk to that have worked with him.

Noam [00:31:17]: I think I've just been very, very impressed. I think I've just been very impressed with his vision that I think like, especially when I joined and I saw, you know, the internal documents at OpenAI of like what he had been thinking about back in like 2021, 2022, even earlier, I was very impressed that he had a clear vision of like where this was all going and what was needed.

swyx [00:31:36]: Some of his emails from 2016, 17, when they were founding OpenAI was published. And even then he was talking about how he had things like one big experiment is much more valuable than 100 small ones. That was like a core insight. That was like a core insight that differentiated them from brain, for example. It just seems very insightful that he just sees things much more clearly than others. And I just wonder what his production function is like, how do you make a human like that? And how do you improve your own thinking to better model it?

Noam [00:32:04]: I mean, I think it is true that I mean, one of OpenAI's big success was betting on the scaling paradigm. It is just kind of odd because, you know, they were not the biggest lab, you know, it was difficult for them to scale. Back then, it was much more common to do like a lot of small experiments. More academic style. People were trying to figure out these various like algorithmic improvements and OpenAI bet pretty early on like large scale.

swyx [00:32:27]: We had David Luan on who I think was VP Eng at the time of GPT-1 and 2. And he talked about how the differences between brain and OpenAI was basically the cause of Google's inability to come out with a scaled model. Like just structurally, everyone had allocated compute and you had to pool resources together to make bets and you just couldn't.

Noam [00:32:47]: I think that's true. I think that's true. I think OpenAI was structured differently and I think that really helped them. Like OpenAI functions a lot like a startup and other places tended to function more like universities or, you know, research labs as they traditionally existed. The way that OpenAI operates more like as a startup with this mission of building AGI and super intelligence, that helped them organize, collaborate, pool resources together, make hard choices about like how to allocate resources. And I think a lot of the other labs like have now been doing that. It's very effective in trying to adopt paradigms more like that, like setups more like that.

Alessio [00:33:22]: Let's talk about maybe the killer use case, at least in my mind, of these models, which is coding. You released CodeX recently, but I would love to talk through the Noam Brown coding stack. What models you use, how you interact with them. Cursor, Windsurf.

Noam [00:33:35]: Lately, I've been using Windsurf and CodeX. Actually a lot of CodeX. I've been having a lot of fun. You just give it a task and it just goes off and does it and comes back five minutes later with like a, you know, pull request. And is it core research task?

swyx [00:33:47]: Or like side stuff that you don't do usually?

Noam [00:33:49]: I wouldn't say it's like side stuff. I would say basically anything that I would normally try to code up, I try to do it with Codex first.

swyx [00:34:02]: Well, for you, it's free, but yeah, for everybody, it's free right now.

Noam [00:34:04]: And I think that's partly because it's the most effective way for me to do it. And also, it's good for me to get experience working with this technology and then also seeing the shortcomings of it. It just helps me better understand, okay, this is the limits of these models and what we need to push on next.

swyx [00:34:20]: Have you felt the AGI?

Noam [00:34:21]: I've felt the AGI multiple times, yes.

swyx [00:34:25]: How should people push Codex in ways that you've done? And I think you see it before others because obviously you were closer to it.

Noam [00:34:33]: I think anybody can use Codex and feel the AGI. It's kind of funny how you feel the AGI and then you get used to it very quickly.

swyx [00:34:43]: So it's really like... I'm dissatisfied with where it's lacking.

Noam [00:34:46]: Yeah, I know. It's magical. I was actually looking back at the old Sora videos when they were announced. Because remember when Sora came out, it was just magical. You look at that and you're like, it's really here. This is AGI. But if you look at it now, it's kind of like, oh, the people don't move very organically and there's a lack of consistency in some ways. And you see all these flaws in it now that you just didn't really notice when it first came out. And yeah, you get used to this technology very quickly. But I think what's cool about it is that because it's developing so quickly, you get those feel the AGI moments like every few months. So something else is going to come out and just like, it's magical to you. And then you get used to it very quickly.

swyx [00:35:28]: What are your Windsurf pro tips now that you've immersed in it?

Noam [00:35:33]: I think one thing I'm surprised by is how few people... I mean, maybe your audience is going to be more comfortable with reasoning models and use reasoning models more. But I'm surprised at how many people don't even... know that O3 exists. Like, I've been using it day to day. It's basically replaced Google search for me. Like, I just use it all the time. And also for things like coding, I tend to just use the reasoning models. My suggestion is if people have not tried the reasoning models yet, because honestly, people love them. People that use it love them. Obviously, a lot more people use GPT-4.0 and just the default on ChatGPT and that kind of stuff. I think it's worth trying the reasoning models. I think people would be surprised. What they can do.

swyx [00:36:15]: I use Windsurf daily and they still haven't actually enabled it as like a default in Windsurf. Like, I always have to dig up, like, type in O3 and then it's like, oh, yeah, that exists. It's weird. I would say, like, my struggle with it has been that it takes so long to reason and actually break out of flow.

Noam [00:36:34]: I think that is true. Yes. And I think this is one of the advantages of Codex that, like, OK, you can give it a task that's kind of self-contained and like it can go off and do its thing and come back 10 minutes later. And I can see that if you're doing... If you're using this thing is like more like a, like a pair of programmer kind of thing, then, yeah, you want to use GPT-4.1 or something like that.

Alessio [00:36:51]: What do you think are the most broken part of the development cycle with AI? Like, in my mind, it's like pull request review. Like for me, like I use Codex all the time and then I got all these pull requests and it's kind of hard to, like, go through all of them. What other thing would you like people to build to make this even more scalable?

Noam [00:37:09]: I think it's really on us to build a lot more stuff. These models are very limited in some ways. I think I find it frustrating that, you know, you ask them to do something and they spend 10 minutes doing it and then you ask them to do something pretty similar and then they go spend 10 minutes doing it. And like, you know, I think I describe them as like they're geniuses, but it's their first day on the job, you know, and that's like kind of annoying. Like even the smartest person on earth, when it's their first day on the job, you know, they're not going to be like as useful as you would like them to be. So I think being able to get more experience and like act like somebody that's actually been on the job. For like six months instead of one day, I think would make them a lot more useful. But that's really on us to build to build that capability.

Alessio [00:37:51]: Do you think a lot of it is like GPU constraint for you? Like if I think about Codex, why is it asking me to set up the environment myself when like the model, if I ask O3 to like create an environment setup script for a repo, I'm sure I'll be able to do it. But today in the product, I have to do it. So I'm wondering in your mind, could these be a lot more if we just, again, put more test time compute on them? Or do you think there's like a fundamental? There's a fundamental model capability limitation today that we still need a lot of like human harnesses around it.

Noam [00:38:19]: I think that we're in an awkward state right now where like progress is very fast and there's things that are like, clearly we could do this in the models we better. We're going to get to it. It's you're just limited by how many hours there are in the day, you know, so progress can only proceed so quickly. We're trying to get to everything as fast as we can. And I think that O3 is not where the technology will be in six months.

swyx [00:38:41]: I like that question overall. And like, there's a software development. Lifecycle, not just generation of the code, like from issue to PR basically is like the typical commentary of that. And then there's the windsurf side, which is inside your ID, like what else, right? Pull request review is like something that people don't really, there are startups that are built around it. It's not something that Codex does and it could. And so like, then there's like, what else is there, you know, that is sort of rate limiting the amount of software you could be iterating on. It's an open question. I don't, I don't, I don't know if there's an answer. Anything else on it? Yeah. As we, in general, like, where do you think this goes just in form factors or what will we be looking at this time next year in terms of how things are, how, what models are able to do that they're not able to today?

Noam [00:39:29]: I don't think it's gonna be limited to ASWY. You know, I think, I don't think it's going to be limited to software engineering. I think it's going to be able to do a lot of remote work kind of tasks. Yeah.

swyx [00:39:37]: Like freelancer type of work. Yeah.

Noam [00:39:40]: Or just like even things that are not necessarily software engineering. Okay. So. So the way that I think about it is like anybody that's doing a remote work kind of job, I think it's valuable to become familiar with their technology and like kind of get a sense of like what it can do, what it can't do, what it's good at, what it's not good at, because I think the breadth of things that it's going to be able to do is going to expand over time as well.

swyx [00:39:59]: I feel like virtual assistants might be the next thing after ASWY then, because they're the most easily, like, you know, virtual assistants, like hire someone in the Philippines, someone who just looked through your email and all that, because that is entirely, you can intercept. all the inputs and all the outputs and train on that. And maybe OpenAI just buys a virtual assistant company.

Noam [00:40:20]: Yeah. I think what I'm looking forward to is that for things like virtual assistants, the models, like if they're aligned well, they could end up being like really preferable for that kind of work. You know, if there's always this like principal agent problem where if you delegate a task to somebody, then like, are they really aligned with like doing it as you would want it to be done?

swyx [00:40:40]: And just as cheaply, as quickly as they can. Yeah, yeah.

Noam [00:40:44]: And so if you have an AI model that's like actually really aligned to you and your preferences, then that could end up doing a way better job than a human could. Well, not that it's doing a better job than a human could, but like it's doing a better job than a human would.

swyx [00:40:56]: That word alignment, by the way, I think there's like an interesting overriding or homomorphism between safety alignment and instruction following alignment. And I wonder where they diverge.

Noam [00:41:09]: Okay, so I think where it diverges is like, what do you want to align the models to? Like, that's, I think, a difficult question, you know? Like, you could say like you wanted to align it to the user. Okay, well, what happens if the user wants to build a novel virus that's going to wipe out half of humanity? That safety alignment, yeah. So there's a question of like, I think alignment, I think they're related, you know? And I think the big question is like, what are you aligning towards?

swyx [00:41:30]: Yeah, there's like humanity goals, and then there's your personal goals, and everything in between.

Alessio [00:41:35]: So that's kind of, I guess, the individual agent. And you announced that you're leading the multi-agent team at OpenAI. I haven't really seen many. Any announcements? Maybe I missed them on what you've been working on. But what can you share about interesting research directions or anything from there?

Noam [00:41:51]: Yeah, there hasn't really been announcements on this. I think we're working on cool stuff, and I think we'll get to announce some cool stuff at some point. I think the team, in many ways, is actually a misnomer, because we're working on more than just multi-agent. Multi-agent is one of the things we're working on. Some other things we're working on is just like being able to scale up test time compute by a ton. So how, you know, we get these models thinking for 15 minutes now. How do we get them to think? How do we get them to think for hours? How do we get them to think for days, even longer? And be able to solve incredibly difficult problems. So that's one direction that we're pursuing. Multi-agent is another direction. And here, I think there's a few different motivations. We're interested in both the collaborative and the competitive aspect of multi-agent. I think the way that I describe it is people often say in AI circles that humans occupy this very narrow band of intelligence. And AIs are just going to, like, quickly catch up. And then surpass, like, this band of intelligence. And I actually don't think that the band of human intelligence is that narrow. I think it's actually quite broad. Because if you compare anatomically identical humans from, you know, caveman times, they didn't get that far in terms of, like, you know, what we would consider intelligence today, right? Like, they're not putting a man on the moon, you know, they're not, like, building semiconductors or nuclear reactors or anything like that. And we have those today, even though we as humans are not. And so, what's the difference? Well, I think the difference is that you have thousands of years, a lot of humans, billions of humans, cooperating and competing with each other, building up civilization over time. The technology that we're seeing is the product of this civilization. And I think similarly, the AIs that we have today are kind of like the cavemen of AI. And I think that if you're able to have them cooperate and compete with billions. Of AIs over a long period of time and build up a civilization, essentially, the things that they would be able to produce and answer would be far beyond what is possible today with the AIs that we have today.

Alessio [00:43:56]: Do you see that being similar to maybe, like, Jim Phan's Voyager skill library idea, resaving these things? Or is it just the models then being retrained on this new knowledge? Because the humans then have a lot of it in the brain as they grow.

Noam [00:44:10]: I think I'm going to be evasive here and say that, like... We're not going to... Yeah, we're not going to... Until we have something to announce, which I think that we will in the not-too-distant future. I think I'm going to be a bit vague about, like, exactly what we're doing. But I will say that the way that we are approaching multi-agent in the details and the way we're actually going about it is, I think, very different from how it's been done historically and how it's being done today by other places. I've been in the multi-agent field for a long time. I've kind of felt like the multi-agent field has... It's been a bit misguided in some ways in the approaches that the field has taken and, like, the way that it's been approached. And so I think we're trying to take a very principled approach to multi-agent.

swyx [00:44:52]: Sorry, I got to add, like... So you can't talk about what you're doing, but you can say what's misguided. What's misguided?

Noam [00:44:57]: I think that a lot of the approaches that have been taken have been very heuristic and haven't really been following, like, the bitter lesson approach to scaling and research.

Alessio [00:45:07]: Okay. I think maybe this might be a good spot. So, obviously, you've done a lot of amazing work. You've done a lot of amazing work in poker, and I think as the reasoning model got better, I was talking to one of my friends who used to be a hardcore poker grinder, and I told them I was going to interview you. And their question was, at the table, you can get a lot of information from a small sample size about how a person plays. But today, GTO is, like, so prevalent that sometimes people forget that you can play exploitatively. What do you think is the state? As you think about multi-agent and kind of, like, competition, is it always going to be trying to find the optimal thing? Or is a lot of it... Trying to think more in the moment, like, how to exploit somebody?

Noam [00:45:46]: I'm guessing your audience is probably not super familiar with poker terminology, so I'll just, like, explain this a bit. Sure. A lot of people think that poker is just, like, a luck game, and that's not true. It's actually... There's a lot of strategy in poker. So, you can win consistently in poker if you're playing the right strategy. So, there's different approaches to poker. One is game theory optimal. This is, like, you're playing an unbeatable strategy and expectation. Like, you're just unexplodable. It's kind of like in rock, paper, scissors. You can be unbeatable in rock, paper, scissors if you just randomly choose between rock, paper, and scissors with equal probability. Because no matter what the other guy does, you know, they're not going to be able to exploit you. You're going to win. You're going to, like, not lose an expectation. Now, a lot of people hear that, and they think, like, well, that also means that you're not going to win an expectation because you're just playing totally randomly. But in poker, if you play the equilibrium strategy, it's actually really difficult for the opponents to figure out how to tie you. And they're going to end up making mistakes that will lead you to win. And they're going to end up making mistakes that will lead you to win over the long run. It might not be a massive win, but it is going to be a win. If you play enough hands for a long enough period of time, you're going to win in expectation. Now, there's also exploitative poker. And the idea here is that you're trying to spot weaknesses in how the opponent plays. You know, maybe they're not bluffing enough. Or maybe they fold too easily to a bluff. And so you start adapting from the game theory optimal balance strategy of, like, you bluff sometimes, you don't bluff sometimes, to then playing a very unbalanced strategy that's like, Oh, I'm just going to play this. I'm just going to, like, bluff a ton against this person because they always fold whenever I bluff. Now, the key is that there's a tradeoff here. Because if you're taking this exploitative approach, then you're opening yourself up to exploitation as well. And so you have to choose this balance between playing a defensive game theory optimal policy that guarantees you're not going to lose, but might not make you as much money as you potentially could, versus playing an exploitative strategy that can be much more profitable, but also it creates weaknesses that the opponents can take advantage of and trick you. And there's no way to perfectly balance the two. It's kind of like in rock, paper, scissors. If you notice somebody is, like, playing paper for five times in a row, you might think, like, Oh, they have a weakness in their strategy. I should just be throwing scissors, and I'm going to take advantage of them. And so on the sixth time you throw scissors, but actually that's the time when they throw a block, you know? And you never really know. So you always have this tradeoff. The poker AIs that have been extremely successful, and, like, my background is, like, I worked on AI for poker for several years during grad school and made the first superhuman no-limit poker AIs. The approach that we took was this game theory optimal approach. Where the AIs would play this unbeatable strategy, and they would play against the world's best and beat them. Now, that also means they beat the world's worst. Like, they would just beat anybody. But if they were up against a weak opponent, they might not beat them as severely as a human expert might, because the human expert would know how to adapt from the game theory optimal policy to be able to exploit these weak players. And so there's this kind of unanswered question of, like, how do you make an exploitative poker AI? And a lot of people had pursued this research direction. I, like, dabbled in it a little bit during grad school. And I think fundamentally it just comes down to AIs not being as sample efficient as humans. You know, we discussed earlier. If a human's playing poker, they're able to get a really good sense of the strengths and weaknesses of a player within a dozen hands. It's, like, honestly really impressive. And back when we were working on AI for poker in, like, the, you know, mid-2010s, these AIs would have to play, like, 10,000 hands of poker to, like, get a good profile of, like, who this player is. Like, how they're playing, where their weaknesses are. Now, I think with more recent technology, that has come down. But still, the sample efficiency has been a big challenge. Now, what's interesting is that after working on poker, I worked on diplomacy. I think we talked about this earlier. And diplomacy is this, you know, it's a seven-player negotiation game. And when we started working on it, I took a very game theory approach to the problem. I felt like, okay, it's kind of like poker. You have to compute this game theory optimal policy. And you just play this. You're going to not lose an expectation. You're going to win in practice. But it actually doesn't work in diplomacy. And it doesn't work, again, for the question of, like, how much of a rabbit hole do we want to go down on this? But, like, basically, when you're playing, like, the zero-sum games, like poker, game theory optimal works really well. When you're playing a game like diplomacy, where there's, like, you need to collaborate and compete, and there's room for collaboration, then game theory optimal actually doesn't work that well. And you have to understand the players and adapt to them much better. So, this ends up being very... It's very similar to the problem in poker, of, like, how do you adapt to your opponents? In poker, it's about adapting to their weaknesses and take advantage of that. In diplomacy, it's about adapting to their play styles. It's kind of like, if you're at a table and everybody's speaking French, you don't want to just keep talking in English. You want to adapt to them and speak in French as well. That's the realization that I have with diplomacy, that we need to shift away from this game theory optimal paradigm towards modeling the other players, understanding who they are, and then responding accordingly. And so, in many ways, the techniques that we developed in diplomacy are exploitative. Like, they're not exploitative. They're really, you know, just adapting to the opponents, to the other players at the table. But I think the same techniques could be used in AI for poker to make exploitative poker AIs. If I didn't get, you know, AGI-pilled by the incredible progress that we were seeing with language models and, like, shifting my whole research agenda to focusing on, like, general reasoning, probably what I would have worked on next was making these, like, exploitative poker AIs. Right. So, really fun research direction to go down. I think it's still there for anybody that wants to do it. And I think the key would be taking the techniques that we use in diplomacy and applying them to things like poker.

Alessio [00:51:16]: I think, to me, the core piece is when you play online, you have a HUD, which tells you, you know, all these stats about the other player and, like, you know, how much they participate, preflop, blah, blah, blah. And to me, it's like a lot of these models, from my understanding, are not really leveraging the behavior of the other players at the table. They're just kind of looking at the board state and kind of working from there.

Noam [00:51:36]: That's correct. That's true. That's true. That's true. That's true. Especially the poker AIs work today. They're just kind of, like, sticking to their precomputed GTO strategy, and they're not adapting to the other players at the table. And, like, you can do various, like, kind of hacky things to get them to adapt, but, you know, they're not very principled. They don't work super well.

swyx [00:51:55]: Yeah. Okay. Any grad students listening? Yeah. If you want to work on that, I think that is a very, very reasonable research direction that will at least get in front of you and, you know, get some attention, at least. Yeah. I think one thing that this conversation brings up for me is, yeah, well, one of the hypotheses for, like, what is the next step after test time compute is world models. Is world modeling importance or worthwhile research direction? Like, Yann LeCun has been talking about this nonstop, but, like, basically no LLMs have, like, they have internal world models, but, like, not explicitly a world model.

Noam [00:52:29]: I think it's pretty clear that as these models get bigger, they have a world model, and that world model becomes better with scale. So, they are implicitly developing a world model, and I don't think it's something that you need to explicitly model. I could be wrong about that, you know.

swyx [00:52:48]: When dealing with people or multi-agents, it might be, because you have entities that are not the world, and you're resolving hypotheses of what, which are the many types of entities you could be dealing with.

Noam [00:53:00]: You know, there was this, like, long debate in the multi-agent AI community for a long time about, and it's... It's still going on about whether you need to explicitly model other agents, like, other people, or if they can be implicitly modeled as part of the environment. For a long time, I was, like, on the, took the perspective of, like, of course you have to, like, explicitly model these other agents because they're behaving differently from the environment. Like, they take actions, they're unpredictable, you know, they have agency. But I think I've actually shifted over time to thinking that, like, actually, if these models become smart enough, they develop things like theory of mind. They develop an understanding that they're... That they're agents that, like, can take actions and have motives and all this stuff. And these models just develop that implicitly with scale and more capable behavior broadly. Cool. So, that's the perspective I take these days.

swyx [00:53:47]: So, like, what I just said was an example of a heuristic that is not bitter lesson filled, and you just, it just goes away. Yeah.

Noam [00:53:53]: It's really all come back to the bitter lesson.

swyx [00:53:56]: Gotta cite them every AI podcast. So, one of the interesting findings and most consistent findings, you know, I think you were at ICLR, and one of the hit talks there was about open-endedness. And this guy, Tim, who gave that talk, has been doing a lot, bunch of research about multi-agent systems, too. One of the most consistent findings is always that it's better for AIs to self-play and improve competitively, as opposed to sort of humans training and guiding them. And you find that with, like, you know, AlphaZero and R1.0, whatever that was, do you think this will hold for multi-agents, like, self-play to improve better than humans? Yeah.

Noam [00:54:33]: So, okay. So, this is a great question. And... I think this is, like, worth expanding on. So, I think a lot of people today see self-play as, like, the next step and maybe the last step that we need for superintelligence. And I think if you're following, you know, you look at something like AlphaGo and AlphaZero, we seem to be following a very similar trend, right? Like, the first step in AlphaGo was you do large-scale pre-training. In that case, it was on human Go games. With LLMs, it's pre-training on, you know, tons of, like, internet data. And that gets you a strong model, but it doesn't get you... You know, an extremely strong model. You know, it doesn't get you a superhuman model. And then the next step in the AlphaGo paradigm is you do large-scale test time compute or, like, large-scale inference compute. And in that case with MCTS. And now we have, like, reasoning models that also do, like, this large-scale inference compute. And, again, that, like, boosts the capabilities a ton. Finally, with AlphaGo and AlphaZero, you have self-play, where the model plays against itself, learns from those games, gets better and better and better, and just, like, goes from something that's, like, around human-level performance to, like, way beyond human capability. It's, like, these Go policies now are so strong that it's just, like, incomprehensible. Like, what they're doing is incomprehensible to humans. Same thing with chess. And we don't have that right now with language models. And so it's, like, it's really tempting to look at that and say, like, oh, well, we just need these, like, AI models to now interact with each other and learn from them. And then they're just going to, like, get to superintelligence. The challenge, and I kind of mentioned this, like, a little bit when I was talking about diplomacy. The challenge is that Go is this two-player zero-sum game. And two-player zero-sum games have this very nice property where when you do self-play, you are converging to a minimax equilibrium. And I guess I should take a step back and say, like, in two-player zero-sum games, two-player zero-sum games are chess, Go, even two-player poker, all two-player zero-sum. Well, that's not true. What you typically want is what's called a minimax equilibrium. This is that GTO policy, this policy that you play where you're guaranteeing that you're not going to lose to any opponent in expectation. I think in chess and Go, that's, like, pretty clearly what you want. Interestingly, when you look at poker, it's not as obvious. In a two-player zero-sum version of poker, you could play the GTO minimax policy, and that guarantees that you won't lose to any opponent on Earth. But, again, I mentioned you're not going to beat a weak player. You're not going to make as much money. You're not going to make as much money off of them as you could if you instead played an exploitative policy. So, there's this question of, like, what do you want? Do you want to make as much money as possible, or do you want to guarantee that you're not going to lose to any human alive? What all the bots have decided is, like, well, what all the, like, AI developers in these games have decided is, like, well, we're going to choose the minimax policy. And, conveniently, that's exactly what self-play converges to. If you have these AIs play against each other, learn from their mistakes, they converge over time to this minimax policy, guaranteed. But once you go outside a two-player zero-sum game, that's actually not a useful policy anymore. You don't want to just, like, have this very defensive policy, and you're going to end up with really weird behavior if you start doing the same kind of self-play in things like math. So, for example, what does it mean to do self-play in math? You could fall into this trap of, like, well, I just want one model to pose really difficult questions and the other model to solve those questions. You know, that's like a two-player zero-sum game. The problem is that, like, well, you could just, like, pose really difficult questions that are not interesting. You know, you could just, like, get, ask it to do, like, 30-digit multiplication. It's a very difficult problem for the AI models. Is that really making progress in the dimension that we want? Like, not really. So, self-play outside of these two-player zero-sum games becomes, like, a much more difficult, nuanced question. So, I think, and Tim kind of, like, basically said something similar in his talk, that there's a lot of challenges in really, like, deciding what you're optimizing for when you start to talk about self-play outside of two-player zero-sum games. My point is that, like, this is where the AlphaGo analogy breaks down. And, not necessarily breaks down, but, like, it's not going to be as easy as self-play was in AlphaGo.

swyx [00:58:39]: What is the objective function, then, for that? What is the new objective function?

Noam [00:58:43]: Yeah, it's a good question, yeah. And I think that that's something that, you know, a lot of people are thinking about. Yeah.

swyx [00:58:50]: I'm sure you are. One of the last podcasts that you did, you mentioned, like, the idea of self-play. You mentioned that you were very impressed by Sora. You don't work directly on Sora, but, obviously, it's part of OpenAI. I think the most recent new updates, or in that sort of generative media space, is auto-regressive image gen. Is that interesting or surprising in any way that you want to comment about?

Noam [00:59:10]: I don't work on image gen, so, like, my ability to comment on this is kind of limited. But I will say, like, I love it. Like, I think it's super impressive. It's, like, one of those things where, you know, you work on these reasoning models, and you think, like, wow, we're going to, like, be able to do all sorts of crazy stuff, like, like, advanced science and, you know, solve agentic tasks and software engineering. And then there's, like, this whole other, like, dimension of progress where you're, like, oh, you're able to, like, make images and videos now, and it's, like, so much fun. And that's getting a lot more of the attention, to be honest, especially in the general public. And it's probably driving a lot more of the, like, you know, subscription plans for ChachBT, which is great, but I think it's just kind of funny that, like, yeah, we're also, I promise, we're also working on superintelligence.

swyx [00:59:50]: But you can make everything Ghibli. I think the delta for me was, I was actually harboring this thesis that diffusion was over because of autoregressive emission. Like, there were rumors about this end of last year, and obviously now it's come out. Then Gemini comes out with text diffusion, and, like, diffusion is so bad. And, like, this is two directions, and it's very relevant for inference, of autoregressive versus diffusion. Do we have both? Does one win?

Noam [01:00:18]: The beauty of research is, like, you know, you got to pursue different directions. And it's not always going to be clear, like, what is, you know, the promising path. Like, and I think it's great that people are looking into different directions and trying different things. I think that there's a lot of value in that exploration, and I think we all benefit from seeing what works.

swyx [01:00:38]: Any potential in diffusion reasoning? Let's say your... Probably can't answer that.

Alessio [01:00:43]: Okay. So you did a master's in robotics, too. Would love to get your thoughts on, one, you know, OpenAI kind of started with the pen spinning trick and, like, the robotic arm they wanted to build. Is it right to work on this humanoid likes? Do you think that's kind of, like, the wrong embodiment of AI? Outside of the usual, you know, how long until we get robots, blah, blah, blah. Is there something that you think is, like, fundamentally not being explored right now

Noam [01:01:06]: that people should really be doing in robotics? I did a master's in robotics years ago. And my takeaway from that experience... First of all, I didn't actually work with robots that much. I was, like, technically in a robotics program. I played around with some Lego robots my first week of the program. But then, honestly, I just, like, pretty quickly shifted to just working on AI for poker. And it was kind of nominally in the robotics master's. But my takeaway from, like, interacting with all these roboticists and seeing their research was that I did not want to work on robots. Because the research cycle is so much slower and so much more painful when you're dealing with, like, physical hardware. Like, software goes so much more quickly. And I think that's why we're seeing so much progress with language models and, like, all these, like, virtual coworker kind of tasks, but haven't seen as much progress in robotics. That, like, physical hardware just is much more painful to iterate on. On the question of humanoids, I don't have very strong opinions here because this isn't what I'm working on. But I think there's a lot of value in non-humanoid robotics as well. I think drones are a perfect example where, like, there's clearly a lot of value in that. Is that a humanoid? No. But in many ways, that's great. You know, like, you don't want a humanoid for that kind of technology. I think weekly, I think that's non-humanoid. And I think that's a great example of how humanoid robots provide a lot of value.

Alessio [01:02:23]: I was reading Richard Hemmings' The Art of Doing Science and Engineering, and he talks about how when you have a new technological shift, people try and take the old workloads and, like, replicate them just in the new technology versus you actually have to change the way you do it. And, you know, when I see this video of, like, you know, your humanoid in the house, it's like, well, the human shape has kind of, has a lot of limitations that could actually be improved. But I think people, what's familiar, you know, it's like, would you put a robot with, like, 10 arms and, like, you know, five legs in your house, or would that be eerie at night when you get up and you see that thing walking around, and is that why we use humanoids? So I think to me, there's almost, like, this local maxim of, like, you know, we got to make it look like a human. But I think, like, what's, like, the best shape in-house would be?

Noam [01:03:08]: I am terrible at product design, so I am not the person to ask on this. I think there is a question of, like, is it better to make humanoids because they're more familiar to us, or is it worse to make humanoids because they're more similar to us but not quite identical? Like, I don't know which one I would actually find creepier.

swyx [01:03:25]: Yeah. Yeah. The thing that got me humanoid-pilled a little bit was just the argument that most of the world is made for humans anyway. So if you want to replace human labor, you have to make a humanoid. I don't know if that's convincing.

Noam [01:03:38]: Again, I don't have very strong opinions in this field because, like, I don't work in it. I was, like, weakly in favor of humanoids, and I think what really persuaded me to be weakly in favor of, like, non-humanoids was listening to the physical intelligence, like, CEO and, like, some of his pitches about, like, why they're pursuing, like, non-humanoid robotics. Okay. And conveniently, their office is actually, like, very close to here. So if you wanted to— They're speaking at the conference I'm running. Okay, perfect. Yeah. So I'm looking forward to that. So, like, you know, I'd say, like, listen to his pitch and maybe he can convince you that non-humanoid is the way to go.

swyx [01:04:08]: Awesome. The other one I would refer people to is Jim Phan recently did a talk on the physical Turing test, which he did at the Sequoia conference, which was very, very good. He's such a great educator and explainer of things. It's very hard, especially in that field. Cool. We're done asking you about things that you don't work on. So these are just more rapid fires to sort of explore some of your boundaries and get some quick hits. How do you or top industry labs keep on top of research? Like, what are your tools and practices?

Noam [01:04:39]: It's really hard. I think that a lot of people have this perception that, like, academic research is irrelevant. And that's actually not the case. I think that we do—we look at academic research. I think one of the challenges—. Yeah. One of the challenges is, like, a lot of academic research shows promise in their papers, but then actually doesn't work at scale or even doesn't replicate. I think if we find interesting papers, like, we're going to try to reproduce that in-house and see if it, like, still holds up and then also doesn't scale well. But that is, like, a big source of inspiration for us.

swyx [01:05:09]: Whatever hits Archive, literally, you do the same as the rest of us, or do you have, like, a special process?

Noam [01:05:15]: Especially if I get recommendations. Like, we have an internal— It's word of mouth. Yeah. Where people will post interesting papers. Yeah. And, like, I think that's a good source of, like, okay, well, this person that is more familiar with this area thinks that this paper is interesting, so therefore I should read it. Yeah. And similarly, like, I'll keep track of things that are happening in, like, in my space that I think are interesting. And, like, if I think it's really interesting, maybe I'll share it.

swyx [01:05:34]: For me, it's, like, WhatsApp and Signal group chats with researchers, and that's it. Yeah.

Noam [01:05:38]: I think it is, like—I mean, a lot of people look at things like Twitter, and I think it's really unfortunate that we've reached this point where things need to get a lot of attention on social media for it to be paid attention to.

swyx [01:05:50]: That's what the grad students are trained—they're taking classes to do this.

Noam [01:05:53]: I do recommend to, like, you know, I've worked with grad students—I work with fewer now because we don't publish as much—but when I was at FAIR publishing papers, like, I would tell the grad students I was working with that, like, you need to post it on Twitter. Yeah. And you need to—and we go over, like, the Twitter thread about, like, how to present the work and everything, and there's a real art to it, and it does matter. And it's kind of the sad truth. Yeah.

Alessio [01:06:15]: Yeah. You know, when you were doing the ACPC, like, the AI poker competition, you mentioned that people were not doing search because they were limited to, like, two CPUs at inference. Mm-hmm. Do you see similar things today that are, like, keeping interesting research from being done? That might be—it's not as popular, it doesn't get you into the top conferences, like, are there some environmental limiters? Absolutely.

Noam [01:06:37]: And I think one example is for benchmarks, that you look at things like humanity's last exam. Like, you have these incredibly difficult problems, but then are still very easily gradable. And I think that actually limits the scope of what you can evaluate these models on if you stick to that paradigm. It's very convenient because, you know, it's very easy to, like, then score the models. But actually, a lot of the things that we want to, you know, evaluate these models on are kind of, like, more fuzzy tasks that are not multiple-choice questions. Mm-hmm. And making benchmarks for those kinds of things is so much harder, and probably also, like, a lot more expensive to evaluate. But I think that those are really valuable things to work on.

Alessio [01:07:13]: And that would fit the set-moment GPT-4.5 as, like, a high-taste model, in a way. There's kind of, like, all these, like, non-measurable things about a model that are really good that maybe people are not.

Noam [01:07:25]: Well, I think there are things that are measurable, but they're just, like, much more difficult to measure. And I think that a lot of benchmarks have kind of stuck to this paradigm of posing really difficult problems that are really easy to measure. Yeah.

swyx [01:07:37]: So let's say that the pre-training scaling paradigm took about five years, from, like, discovery of GPT to, like— Yeah. —scaling it up to GPT-4. And then we give you—we give test-time compute five years, as well. So if test-time compute hit a wall by 2030, what would be the probable cause?

Noam [01:07:54]: As far as some of the pre-training, we're like, you can push pre-training a lot further. It just becomes more expensive with each iteration. I think we're going to see something similar with test-time compute. We're like, okay, we're going to get them thinking instead of three minutes, they're for three hours, and then three days, and then three weeks. Oh, you run out of human life. Well, okay. So there's two—there's two—there's two concerns. One is that it becomes much more expensive to get the models to, like, think for that long or, like, scale up test-time compute. Like, as you scale up test-time compute, you're spending more on test-time compute, which means that, like, there's a limit to how much you could spend. That's one potential ceiling. Now, obviously—well, not obviously, but, like, I should say that we're also becoming more efficient. These models are becoming more efficient in the way they're thinking, and so they're able to do more with the same amount of test-time compute. And I think that's a very underappreciated point, that it's not just that we're getting these models to think for longer. In fact, if you look at O3, it's thinking for longer than O1 Preview for some questions, but it's not, like, a radical difference, but it's way better. Why? Because it's just, like, becoming better at thinking. Anyway, yeah, these models, um, you're going to scale up test-time compute, you can only scale it up so much. Like, that becomes a soft barrier. In the same way that pre-training, it's becoming more and more expensive to train better and better pre-trained models, or bigger pre-trained models. The second point is that, like, as you have these models think for longer, you kind of get bottlenecked by walk-clock time. Yeah. Yeah. It is really easy to iterate on experiments when these models would respond instantly. It's actually much harder when they take three hours to respond. And what happens when they have three weeks? It takes you at least three weeks to do those evaluations and to then iterate on that. And on a lot of this, you can parallelize experiments to some extent, but a lot of it, you have to run the experiment, complete it, and then see the results in order to decide on the next set of experiments. I think this is actually the strongest case for long timelines, that the models, because they just have to, like, do so much.

swyx [01:09:38]: I think it's a challenge.

Noam [01:09:44]: And I think it depends on the domain. So drug discovery, I think, is one domain where this could be a real bottleneck. I mean, if you want to see if something, like, extends human life, it's going to take you a long time to figure out if, like, this new drug that you developed, like, actually extends human life and doesn't have, like, terrible side effects along the way.

swyx [01:10:00]: Side note, do we not have perfect models of human chemistry and biology by now?

Noam [01:10:03]: Well, so this is, I think, the thing. And again, I want to be cautious here, because I'm not actually, you know, I don't know, a biologist or a chemist. Like, I don't know very little about these fields. Last time I took a biology class was 10th grade in high school. I don't think that there's a perfect simulator of human biology right now. And I think that that's something that could potentially help address this problem.

swyx [01:10:21]: That's, like, the number one thing that we should all work on.

Noam [01:10:24]: Well, that's one of the things that we're hoping that these recent models will help us with. Yeah.

swyx [01:10:28]: How would you classify mid-training versus post-training today?

Noam [01:10:32]: It's such, all these definitions are so fuzzy. So I don't have a great answer there. Yeah.

swyx [01:10:38]: It's a question people have in your, like, opening eyes, like now explicitly hiring for mid-training and everyone is like, what the hell is mid-training? I think mid-training is between pre-training and post-training.

Noam [01:10:50]: It's like, it's like, it's not, it's not post-training. It's not pre-training. It's like adding more to the models, but like after pre-training, like, I don't know. Interesting ways. Yeah. Okay.

swyx [01:11:01]: All right. Well, you know, I was trying to get some clarity.

Alessio [01:11:06]: Is the pre-trained model now? Basically like a, just the artifact that then spawns other models. And it's almost like the core pre-training model is never really exposed anymore. And it's the mid-training, the new pre-training, and then there's the post-training once you have the models branched out.

Noam [01:11:22]: You never interact with an actual, just like raw pre-trained model. Like if you're going to interact with the model, it's going to go through mid-training and post-training. So, um, so you're seeing the final product.

swyx [01:11:31]: Well, you don't let us do it, but you know, we used to.

Noam [01:11:33]: Well, yeah. I mean, I guess you, you know, there's open source models where you can just like interact with the raw pre-trained model. Um, but for, for open AI models, like they go through a mid-training step and then they go through a post-training step and then, and then they're released and they're a lot more useful. Like, frankly, if you interacted with the only pre-trained model, it'd be super difficult to work with and it would, it would seem kind of dumb.

Alessio [01:11:51]: Yeah.

swyx [01:11:51]: But it'd be, it'd be useful in weird ways, you know, because there's a mode collapse when you, when you post-train for it, for like chat. Yeah.

Noam [01:11:59]: And in some ways you want that mode collapse. Like you want, you want that collapse of like distribution. To be useful. Yeah. I get it. Yeah. Yeah.

swyx [01:12:08]: And if you had to, if you had to talk to him a lot, what would you ask him?

Noam [01:12:11]: What would I ask Greg? I mean, I mean, I get to ask Greg all the time, but what should you ask Greg?

swyx [01:12:15]: Like to evoke an interesting response that like, uh, he doesn't get asked enough about, but you know, like this is something that he's passionate about or you just want his thoughts.

Noam [01:12:25]: I think in general, it's worth asking where this goes. You know, like, what does the world actually look like in five years? What does the world look like in 10 years? What does that distribution of outcomes look like? And what could. Yeah. The world or individuals do to help steer things towards like the good outcomes instead of the negative outcomes.

swyx [01:12:45]: Okay, like an alignment question.

Noam [01:12:48]: I think people get very focused on what's going to happen in like one or two years. And I think it's also worth spending some time thinking about like, well, what happens in five or 10 years? And what does that world look like?

swyx [01:12:59]: I mean, he doesn't have a crystal ball.

Noam [01:13:01]: But he certainly has thoughts. Yeah, so I think that's worth exploring.

Alessio [01:13:08]: What are games that you recommend to people, especially socially?

Noam [01:13:12]: What are games that I recommend to people? I've been playing a lot of this game called Blood on the Clocktower lately.

swyx [01:13:18]: What is it?

Noam [01:13:19]: It's kind of like Mafia or Werewolf. It's become very popular in San Francisco. Oh, that's the one we played in your house? Yeah. Okay, got it. It's kind of funny because like I was talking to a couple people now that had told me that it used to be that poker was the like way that like the VCs and tech founders and stuff would socialize with each other. And actually now it's shifting more towards Blood on the Clocktower, like that's the thing that people use to like, you know, connect in the Bay Area. And I was actually told that a startup held a recruiting event that was a Blood on the Clocktower game. Wow.

swyx [01:13:55]: Yeah.

Noam [01:13:55]: So I guess it's like it's really catching on, but it's a fun game. And I guess you lose less money playing it than you do playing poker. So it's better for people that are not very good at these things. I think it's kind of like a weird recruiting event. But it's certainly a fun game.

swyx [01:14:08]: What qualities make a winner here that is interesting to hire for?

Noam [01:14:13]: That's the thing is like, okay, I guess you get a good ability to lie. Deception and like picking up on deception. Like, is that the best employee? Like, I don't know.

Alessio [01:14:23]: So my slight final pet topic is Magic the Gathering. So you have, we talked about some of these games, Chesco, and they have perfect information. Then you have poker, which is imperfect information in a pretty limited universe. You only have a 50-2 card game. And then you have these other games that have imperfect information, like a huge pool of possible options. Do you have any idea of like how much harder that is? Like, how does the difficulty of this problem scale?

Noam [01:14:48]: I love that you asked that because I have this like huge store of knowledge on AI for imperfect information games. Like this is my area of research for so long. And I know all these things, but I don't get to talk about it very often. We've made superhuman poker AIs for No Limit Texas Hold'em. One of the interesting things about that is that like. The amount of hidden information is actually pretty limited because you have two hidden cards when you're playing Texas Hold'em. And so the number of possible states that you could be in is 1,326 when you're playing heads up at least. And you know, that's multiplied by the number of other players that there are at the table, but it's still like not a massive number. And so the way these AI models work is they enumerate all the different states that you could be in. So if you're playing like six handed poker, there's five other players, five times 1,326. That's the number of states that you've been. And then you assign a probability to each one, and then you feed those probabilities into your neural net and you get actions back for each of those states. The problem is that as you scale the number of hidden possibilities, like the number of state of, of possible states you could be in that approach breaks down. And there's still this very interesting unanswered question of what do you do when the number of hidden states becomes extremely large? You know, so if you go to Omaha poker, where you have four hidden cards, there are things you could do. That's kind of like that. That are kind of heuristic that you could do to reduce the number of states, but actually it's still a very difficult question. And then if you go to a game like Stratego where you have 40 pieces, so there's like close to 40 factorial different states you could be in. Then all these like existing approaches that we used for poker kind of break down and you need different approaches. And there's a lot of active research going on about like, how do you, how do you cope with that? So for something like matching the gathering, the techniques that we used in poker would not out of the box work. And it's still an interesting research question of like. Like, what do you do now? I should say this becomes a problem when you're doing the kinds of search techniques that we used in poker. If you're just doing model free RL, it's not a problem. And my guess is that if somebody put in the effort, they could probably make a superhuman bot for Magic the Gathering now. Yeah. There's still some unanswered research questions in that space. Now, are they the most important unanswered research questions? Like I'm inclined to say, no, I think there's like, the problem is that like the techniques that we used in poker to do this kind of search stuff, we're pretty limited. And like, if you expand. If you expand those techniques, maybe you get them to work on things like Stratego and Magic the Gathering, but they're still going to be limited. They're not going to get you like superhuman encode forces with language models. So I think it's more valuable to just focus on the very general reasoning techniques. And one day as we improve those, I think we'll have a model that just out of the box one day plays Magic the Gathering at a superhuman level. And I think that's the more important and more impressive research direction. Cool. Amazing.

Alessio [01:17:31]: Yeah. Thanks for coming on, Noam. Yeah.

Noam [01:17:34]: Thanks for your time. Yeah. Thanks. Thanks for having me.

Editor’s Note: An earlier version of this podcast featured a headline image that we regret. We’re sorry and have taken it down but aren’t hiding our mistake.

And, through Llama’s contributions, an understanding of inference-optimal scaling.

we previously wrote “test time compute will hit a wall” - corrected after Noam asserted.

Update: you can watch the full talk on YouTube now!

Slides are here: https://docs.google.com/presentation/d/1sZqMAoIJDxz79cbC5ap5v9jknYH4Aa9cFFaWL8Rids4/edit?usp=sharing

Lots of people were excited about Andrej’s talk at YC AI Startup School today. Sadly, I wasn’t invited. Talks will be published “over the next few weeks”, by which time the talk might be deprecated. Nobody seems to have recorded fancams.

But… it’s not over. You can just do things!

Using PeepResearch™ I collated all available tweets about the talk and ordered1 them using available hints from good notetakers (credited in last slide) UPDATE: and now we have the full transcript! I’ll go thru most of the impt takeaways here and subscribers can get the full slides at the bottom.

UPDATE: Slides are now synced with the full transcript if you want to read thru

We first discussed Software 3.0 in Rise of The AI Engineer, but it’s an obvious consequence of the Software 2.0 essay + “the hottest new programming language is English”.

He originally wrote the Software 2.0 essay while observing that it was eating Software 1.0 at Tesla. And he’s back now to update it for Software 3.0.

In place of modifying the Software 2.0 chart like I did, Andrej debuts a new diagram showing the patchwork/coexistence of Software 1.0/2.0/3.0, noting that “Software 3.0 is eating 1.0/2.0” and that “a huge amount of software will be rewritten”:

Andrej is still focused on prompts for programs, and we slightly disagreed back in 2023 and still do: the “1+2=3” variant of Software 3.0 is the entire reason why AI Engineers have far outperformed Prompt Engineers in the last few years and continue to do so.

although as he argues in Power to the People, LLMs also exhibit some unusual reversal of the normal flow of expensive frontier tech:

As we leave the cloud for Personal/Private AI, some signs of Personal Computing v2 are being born in Exolabs + Apple MLX work.

LLMs are “people spirits”: stochastic simulations of people, with a kind of emergent “psychology”

Andrej highlights two problems with how current LLMs simulate people:

Jagged Intelligence (https://x.com/karpathy/status/1816531576228053133):

The word I came up with to describe the (strange, unintuitive) fact that state of the art LLMs can both perform extremely impressive tasks (e.g. solve complex math problems) while simultaneously struggle with some very dumb problems. E.g. example from two days ago - which number is bigger, 9.11 or 9.9? Wrong.

…

Some things work extremely well (by human standards) while some things fail catastrophically (again by human standards), and it's not always obvious which is which, though you can develop a bit of intuition over time. Different from humans, where a lot of knowledge and problem solving capabilities are all highly correlated and improve linearly all together, from birth to adulthood.

Personally I think these are not fundamental issues. They demand more work across the stack, including not just scaling. The big one I think is the present lack of "cognitive self-knowledge", which requires more sophisticated approaches in model post-training instead of the naive "imitate human labelers and make it big" solutions that have mostly gotten us this far. For an example of what I'm talking about, see Llama 3.1 paper section on mitigating hallucinations: https://x.com/karpathy/status/1816171241809797335

For now, this is something to be aware of, especially in production settings. Use LLMs for the tasks they are good at but be on a lookout for jagged edges, and keep a human in the loop.

Anterograde Amnesia (https://x.com/karpathy/status/1930003172246073412):

I like to talk explain it as LLMs are a bit like a coworker with Anterograde amnesia - they don't consolidate or build long-running knowledge or expertise once training is over and all they have is short-term memory (context window). It's hard to build relationships (see: 50 First Dates) or do work (see: Memento) with this condition.

The first mitigation of this deficit that I saw is the Memory feature in ChatGPT, which feels like a primordial crappy implementation of what could be, and which led me to suggest this as a possible new paradigm of learning here: https://x.com/karpathy/status/1921368644069765486

We're missing (at least one) major paradigm for LLM learning. Not sure what to call it, possibly it has a name - system prompt learning?

Pretraining is for knowledge.

Finetuning (SL/RL) is for habitual behavior.

Both of these involve a change in parameters but a lot of human learning feels more like a change in system prompt. You encounter a problem, figure something out, then "remember" something in fairly explicit terms for the next time. E.g. "It seems when I encounter this and that kind of a problem, I should try this and that kind of an approach/solution". It feels more like taking notes for yourself, i.e. something like the "Memory" feature but not to store per-user random facts, but general/global problem solving knowledge and strategies. LLMs are quite literally like the guy in Memento, except we haven't given them their scratchpad yet. Note that this paradigm is also significantly more powerful and data efficient because a knowledge-guided "review" stage is a significantly higher dimensional feedback channel than a reward scaler.

…

Imo this is not the kind of problem solving knowledge that should be baked into weights via Reinforcement Learning, or least not immediately/exclusively. And it certainly shouldn't come from human engineers writing system prompts by hand. It should come from System Prompt learning, which resembles RL in the setup, with the exception of the learning algorithm (edits vs gradient descent). A large section of the LLM system prompt could be written via system prompt learning, it would look a bit like the LLM writing a book for itself on how to solve problems. If this works it would be a new/powerful learning paradigm. With a lot of details left to figure out (how do the edits work? can/should you learn the edit system? how do you gradually move knowledge from the explicit system text to habitual weights, as humans seem to do? etc.).

Part 2 Summary:

We like the Iron Man Suit analogy — the suit extends us in two useful ways:

• Augmentation: giving the user strength, tools, sensors and information

• Autonomy: the suit at many times has a mind of its own- taking actions without being prompted

How can we design AI products that follow these patterns?

The Autonomy Slider is an important concept that lets us choose the level of autonomy for the context, eg:

• Cursor: Tab -> cmd+K -> Cmd+L -> Cmd+I (agent mode)

• Perplexity: search -> research -> deep research

• Tesla Autopilot: Level 1 to Level 4

In the Generation <-> verification cycle, we a need full workflow of partial autonomy - the faster the loop the better:

• To improve verification: Make it easy, fast to win

• To improve generation: Keep AI on tight leash

The reason we need PARTIAL autonomy is because of the significant gap still between a working demo and a reliable product.

He recounts riding a Waymo prototype with zero interventions in 2014 and thinking that self-driving is “here”… but there was still a lot to work out.

"Demo is works.any(), product is works.all()"

The tweet that launched a thousand startups:

now has its own Wikipedia page!

However, there are still a lot of remaining issues. While Vibe coding MenuGen, he found that the AI speedups vanished shortly after getting local code running:

The reality of building web apps in 2025 is a disjoint mess of services that are very much designed for webdev experts to keep their jobs, and not accessible to AI.

Poor old Clerk got a NEGATIVE mention, and Vercel’s @leerob a positive one, in how their docs approaches will respectively tuned for humans vs agents.

The bottom line is that toolmakers must realize that “there is new category of consumer/manipulator of digital information”:

1. Humans (GUls)

2. Computers (APls)

3. NEW: Agents <- computers... but human-like

Concretely: having llms.txt works because HTML is not very parseable for LLMs.

He also shouted out “Context builders” like Gitingest and Cognition’s DeepWiki, which we profiled for a lightning pod:

This is the Decade of Agents.

Less AGI 2027 and flashy demos that don’t work.

More partial autonomy, custom GUIs and autonomy sliders.

Remember that Software 3.0 is eating Software 1/2, that their Utility/Fabs/OS characteristics will dictate their destiny, improve the generator-verifier loop, and BUILD FOR AGENTS 🤖.

here :) link below

Watch on YouTube, listen on Apple Podcast, Spotify, or add our RSS feed to your favorite player!

When Chris Lattner first came on the podcast, we titled the episode “Doing it The Hard Way”. Rather than simply building an inference platform and posting charts showing GPUs going brrr, they went down to the compiler level and started rebuilding the whole stack from scratch.

Modular is being built as “concentric circles”, as Chris calls them. You drop into the layer you need; everything lower is optional plumbing you never touch. We covered a lot of the Mojo design in the previous episode, so refer to that if you need a refresher. At the core of its uniqueness, there’s one thing: Mojo exposes every accelerator instruction (tensor cores, TMAs, etc.) in a Python‑familiar syntax so you can:

• Write once, re‑target to H100, MI300, or future Blackwell.

• Run 10‑100× faster than CPython; usually beats Rust on the same algo.

• Works as a zero‑binding extension language: import my_fast_fn.mojo and call from normal Python.

This removes the need for any Triton/CUDA kernels and C++ “glue” code.

The same approach has been applied to the MAX inference platform:

• Max’s base image is ≈ 1 GB because it strips the Python dispatch path.

• Supports ~500 models with pre-built containers which include all the optimizations needed like Flash‑Attention, paged KV, DeepSeek style MLA, etc.

• Most importantly they ship nightly builds to really get the open source community involved

This week, they had a big announcement with AMD showing that they are ~matching MI325 performance with H200s running on vLLM on some of the more popular open source models. This is a huge step forward in breaking up NVIDIA’s dominance (you can also listen to our geohot episode for more on AMD’s challenges).

Chris has also been on a blogging bonanza with his Democratizing AI Compute series, at once a masterclass in the problems CUDA solves, but also the most coherent pitch for Modular’s approach: https://www.modular.com/democratizing-ai-compute

Chris already replaced GCC with LLVM and Objective‑C with Swift; is Mojo replacing CUDA next? We talked through what it’s needed to make that happen in the full episode. Enjoy!

• Chris Lattner

• Tim Davis

• Modular

• LLVM Foundation

• MAX

• MLIR

• PyBind11

• NanoBind

• GPU Mode Keynotes

• [00:00:00] Introductions

• [00:00:12] Overview of Modular and the Shape of Compute

• [00:02:27] Modular’s R&D Phase

• [00:06:55] From CPU Optimization to GPU Support

• [00:11:14] MAX: Modular’s Inference Framework

• [00:12:52] Mojo Programming Language

• [00:18:25] MAX Architecture: From Mojo to Cluster-Scale Inference

• [00:29:16] Open Source Contributions and Community Involvement

• [00:32:25] Modular's Differentiation from VLLM and SGLang

• [00:41:37] Modular’s Business Model and Monetization Strategy

• [00:53:17] DeepSeek’s Impact and Low-Level GPU Programming

• [01:00:00] Inference Time Compute and Reasoning Models

• [01:02:31] Personal Reflections on Leading Modular

• [01:08:27] Daily Routine and Time Management as a Founder

• [01:13:24] Using AI Coding Tools and Staying Current with Research

• [01:14:47] Personal Projects and Work-Life Balance

• [01:17:05] Hiring, Open Source, and Community Engagement

Alessio [00:00:05]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host Swyx, founder of SmolAI.

Swyx [00:00:12]: And we're so excited to be back in the studio with Chris Lattner from Mojo / Modular. Welcome back.

Chris: Yeah, I'm super excited to be here. As you know, I'm a huge fan of both of you and also the podcast and a lot of what's going on in the industry. So thank you for having me.

Swyx: Thanks for keeping us involved in your journey. And I think like you spend a lot of time writing when obviously your time is super valuable. Educating the rest of the world. I just saw your like a two and a half hour workshop with the GPU mode guys. And that was super exciting. We're decked out in your swag. We'll get to the part where you are a personal human machine of just productivity and you do so much. And, you know, I think there's a lot to learn. And for people just on a personal level. But I think a lot of people are going to be here just for the state of modular. We're also calling it the shape of compute. I think it's going to be probably the podcast title.

Chris: Yeah, it's super exciting. I mean, there's so much going on in the industry with hardware and software and just innovation everywhere. Most people can catch up on like the first episode that we did and we introduced modular. I think people want to know, like, I think since then you sort of open sourced it. There's been a lot of updates. What would you highlight as like the past year or so of updates? Yeah. So if you zoom out and say, what is modular? Yeah. So we're a company that's just over three years old. Three and a quarter. So we're quite a ways in. The first three years was a very mysterious time to a lot of people because I didn't want people to really use our stuff. And so why that? Why do you build something that you don't want people to use? Well, it's because we're figuring it all out. And so the way I explain is we were very much in a research phase. And so we were trying to solve this really hard problem of how do we unlock heterogeneous compute? How do we enable GP programming to be way easier? How do we enable innovation that's full stack across the AI stack by making it way simpler and driving out complexity? Like there are these core questions. And I had a lot of hypotheses, right? But building an alternate AI stack that is not as good as the existing one isn't very useful because people will always evaluate it against state of the art. And at that point, you say, okay, well, let's just break it down like a normal engineering problem. It's not R&D anymore. It's an engineering problem. You say, okay, cool. Let's refactor these APIs. Let's deprecate this thing. Let's, oh yeah, let's add H100 support. Let's add function calling and token sampling and like all the different things you need. You can project manage that, right? And so every six weeks, we've been shipping a new release. And so we added all the function calling features. And now you have agentic workflows. We have 500 models. We have H100 support. We're about to launch our AMD MI300 and 325 support. That'll be a big deal for the industry. And as you do that in Blackwell, like all this stuff is like all now in the product. And so as this happens now, suddenly it's like, oh, okay, I get it. But this is a very fundamentally different phase for us because it works, right? And once it works, you can see it end to end. Lots of people can put the pieces together in their brain. It's not just in my brain. And a few other people that understand how all the different individual pieces work. Or now everybody can see it. So as we face shift now, suddenly it's like, yeah, okay, let's open source it. Well, now we want more people involved. Now let's do each of these things. It's actually, we had a hackathon. And so we invited 100 people to come visit and spend the day with us. And we learned GPU programming from scratch. And so, you know, we built a very fancy inference framer. You know, the winning team for the hackathon took that four person team. And one day they had not used Mojo before. They hadn't programmed GPUs before. And they had built a training. They wrote an Adam Optimizer, a bunch of training kernels, they built a simple backprop system and they actually showed that you could train a model using all the stuff we'd built for inference because it's so hackable and also because the AI coding tools are awesome, but, but this is the power of what you can do when, when you're ready to scale. Now, if we had done that six months ago or 12 months ago or something, it would have been a huge mess, right? Because everything would break and there are a lot of bugs and, you know, honestly, today it's still an early state system. There's still some bugs, but now it's useful. It can solve real world problems. And so that's, that's the difference. And that's kind of the evolution that we've gone through as a team.

Alessio [00:06:55]: I remember when we first had you at the start, you were focused on CPU actually. How long did you work on CPU and then how much of a jump was it to go from CPU to GPU?

Chris [00:07:06]: So, so the way I would, the way I explain modular is if you take that first three year R&D journey, and if you round a little bit, first year was, um, prove compilation philosophy. And so this was writing. Very. Very abstract compiler stuff and then prove that we can make a matrix multiplication go faster than Intel MKL's matrix multiplication on Intel silicon and make it configurable and multiple D types and prove like a very narrow problem and do that by writing this MLIR compiler representation directly by hand, which was really horrible. But we proved the technology and some technology milestone year two was then say, okay, cool. I believe the, the fundamental approach can work, but guess what? Usability is terrible. Writing internal compiler stuff by hand sucks. And also Matt Moles a long ways from an AI framework. And so year two embarked on two paths. One is Mojo. So program language syntax, member of the Python family, make it much more accessible and easy to write kernels and performance and all that kind of stuff. And then second, build an AI framework for CPUs, as you say, where you could go beat OpenVINO and things like this on Intel CPUs. End of year two, we, we got in, we're like, foof, we've achieved this amazing thing, but you know, it's cool. GPUs. And so again, two things we said, okay, well, let's prove we could do GPUs. Number one, also let's not just build a, you know, air quotes, a CUDA replacement. Let's actually show that we could do something useful. Let's take on LLM serving. No big deal. Right. And so again, two things where you say, let's go prove that we can do a thing, but then validate it against a really hard benchmark. And then that was, that's what, what brought us to year three. Yep. So in each of these stages is really hard and lots of interesting technical problems. Um, probably the biggest problem that you face. So you face people that are constantly telling you it's impossible, but again, you just have to be a little bit stubborn and believe in yourself and work hard and stay focused on milestones. When they say impossible, do they mean impossible or very, very hard? Well, so, I mean, it's, it's common sense that CUDA is nearly 20 years old and VIA's got hundreds or thousands of people working on it. The entire world's been writing CUDA code for many years, very, very hard. And so it's, no, I mean, many people think it's impossible for a startup to do anything in the space. Yeah. Like that's just common sense. I, all these people have thrown all this money at all. All these different systems, they've all failed. Why is your thing going to succeed when all these other things built by other smart people have failed? Right. And so it's conventional wisdom that change is impossible, but Hey, we're an AI, you know, this like changes all around us all the time. Right. And so what you need to do is you need to map out what are the success criteria, what causes change to actually work. And across my career, like with LLVM, all the GCC people told me it was impossible. Like LLVM will fail because GCC is 20 years old and it's had hundreds of people working on it and blah, blah, blah. And spec benchmarks. And whatever, nobody told me it was impossible outside because it was secret. So that was a little bit different, but everybody inside Apple that knew about it said, no, no, no objective C is fine. Like we should just improve objective C. The world doesn't need new programming languages that new programming languages never get adopted. Yeah. Right. And, and it's common sense that like new programming languages don't go anywhere. That's conventional wisdom. You know, MLIR super funny. MLIR is another compiler thing. And so built this thing, brought it to the LLVM community and said, Hey, we open sources. Does LLVM work? I want to it. I know a few LLVM people, right? It was my PhD project and all the LLVM Illuminati in the community had been working on LLVM for 15 years or something. They're like, no, no, LLVM is good enough. We don't need a new thing. Machine learning is not that important. And so again, obviously have developed a few skills to work through this kind of challenge, but you get back to the reality that humans don't like change. And when you do have change, it takes time to diffuse into the ecosystem. For people to process it. And this is where we talk about hackathon. Well, you kind of have to teach people about new things. And so this is why it's like really important to take time to do that. And the blog post series and things like this are all kind of part of this like education campaign, because none of this stuff is actually impossible. It's just really hard work and requires an elite team and good vision and guidance and stuff like this. But but it's understandably conventional wisdom that it's impossible to do this.

Alessio [00:11:14]: And it's the idea to build serving basically. I don't need to tell you how much better this is. I can just actually serve the models using our platform and then you can see how much faster it is. And then obviously you'll adopt it. Yeah.

Chris [00:11:27]: Well, so today you can download MAX, it's available for free. You can scale it to thousands of GPUs you want for free. I'll tell you some cool things about it. So it's not as good as something like VLLM because it's missing some features and it only supports NVIDIA and AMD hardware, for example. But by the way, the container is like a gigabyte. Wow. Right. Well, why is it a gigabyte? Well, it's a completely new stack. It doesn't need. You can use CUDA, you can run arbitrary PyTorch models. And if that's the case, then OK, you pull in some dependencies. But like if you're running the common LLMs and the Gen AI models that people really care about. Well, guess what? It's a completely native stack. It's super efficient. Doesn't have Python in the loop. And for eager mode op dispatch and stuff like this, because you don't have all that dependency, guess what? Your server starts up really fast. So if you care about horizontal auto scaling, that's actually pretty cool. If you care about reliability, it's pretty cool. They don't have all these weird things that are stacked in there. If you're wanting to do something slightly custom, guess what? You have full control over everything and stuff's all open source. And so you can go hack it. This thing's more open source than VLM because VLM depends on all these crazy binary CUDA kernels and stuff like this that are just opaque blobs from NVIDIA. Right. And so this is a very different world. And so I don't want everybody just like overnight drop VLM because I think it's a great project. Right. But but I think there's some interesting things here and there's specific reasons that it's interesting to certain people.

Alessio [00:12:47]: Can you just maybe run people through the pieces of MAX because I think last time none of this existed.

Chris [00:12:52]: Yeah, exactly. Last time has been a long time ago, both in AI and also in modular time. Yeah. So the bottom of our stack is we think about it as concentric circles. So the inside is a programming programming language called Mojo. Chris, why did you have to build another programming language? Well, the answer is that none of the existing ones actually solve the problem. So what's the problem? Well, the problem is that all of compute is now accelerated. Yeah. You have GPUs, you have TPUs, you have all these chips, you have CPUs also. And so we need a programming language that can scale across this. And so if you go shopping and you look at that, like the best thing ish is C++. Like there are things like OpenCL and SYCL and there's like a million things coming out of the HPC community that have tried to scale across different kinds of hardware. Let me be the first to tell you, and I can say this now, I feel I feel comfortable saying this, that C++ sucks. I've earned that right. Having written so much. And so and let me also claim AI people generally don't love C++. What do they love? They love Python. Right. And so what we decided to do is say, OK, well, even and even within the space of C++, there isn't an actually good unifying thing that can talk to tensor cores and things like this. And so we said, OK, I want again, Chris being unreasonable, I want something that can expose the full power of the hardware, not just one chip coming from one vendor, but the full power of any chip. It has to support the next level of very fancy compilers and other stuff like this with graph compilers and stuff like that needs to be portable. So portable across vendors and enable portable code. So, yeah, it turns out that an H100 and AMD chip are actually quite different. That is true. But there's a lot that you can reuse across that. And so the more common code that you can have, the better. The other piece is usability. And so we want something people can actually use and actually learn. And so it's not good enough. Yeah. Just to have Python syntax. You want to have performance and control and again, full power of the hardware. And so this is where Mojo came from. And so today, Mojo is really useful for two things. And so Mojo will grow over time. But today I recommend you use it for places you care about performance. So something running on GPU, for example, or really high performance, doing continuous batching within a web server or something like this where you have to do fancy hashing. And like if you care about performance, Mojo is a good thing. Other cool thing that we're about to talk about. So stay tuned is it's the best way to extend Python. And so if you have a big blob of Python code, you care about performance, you want to get the performance part out of Python. You can make it. We make it super easy to move that to Mojo. Mojo is not just a little faster than Python. It's faster than Rust. So it's like tens of thousands of times faster than Python. And it's in the Python family. And so you can literally just like rip out some for loops, put in Mojo, and now you get performance wins. And then you can start improving the code in place. You can offload it to GPU. You can do this stuff. And all the packaging is super simple. And it's just a beautiful way to make Python code go fast.

Swyx: Just to double-click, you said you were about to ship this?

Chris: It's technically in our nightlies, but we haven't actually announced it yet. Isn't it? Okay. We do a lot of that, by the way, because we're very developer-centric. And so if you join our Discord or Discourse. Nightlies is great. It's like the best program. Yeah. And so we have a ton of stuff that's kind of unannounced but well-known in the community. Okay. I thought this was already released. Yeah. Yeah.

Alessio [00:16:12]: And when you say rip out, is this, you have a Python binding to then run the Mojo? Like, you have C bindings or how does that work?

Chris [00:16:19]: This is the cool thing, is it's binding-free. So think about, like, so today, again, sorry, I get excited about this. I forget how tragic the world is outside of our walls. The thing we're competing with is if you have a big blob of Python code, which a lot of us do, you build it, build it, build it, build it, performance becomes a problem. Okay. What do you do? Well, you have a couple of different things. You can say, I'm going to rewrite my entire application in Rust or something, right? Some people do that. The other thing you can do is you can say, okay, I'm going to use Pybind or Nanobind or some Rust thingy and rewrite a part of my module, the performance-critical part. And now I have to have this binding logic and this build system goop and all this complexity around I have Rust code over here and I have Python code over here. Oh, by the way, you now have to hire people who can work on Rust and Python and like that. Right. So this fragments your team. It's very difficult to hire Rust people. I love them, but there's just too few of them. And so what we're doing is we're saying, okay, well, let's keep the language that's basically the same. So Mojo doesn't have all the features that Python does. Notably, Mojo doesn't have classes. But it has functions. Yeah, you can use arbitrary Python objects. You can get this binding-free experience and it's very similar. And so you can look at this as being like a super fast, slightly more limited Python. And now it's cohesive within the ecosystem. Yeah. And so I think this is useful way beyond just the AI ecosystem. I think this is a pretty cool thing for Python in general. But now you bring in GPUs and you say like, okay, well, I can take your code and you can make it real fast. Because CPUs have lots of fancy features like their own tensor cores and SIMD and all this stuff. Then you can put it on a GPU. And then you can put it on eight GPUs. And so you can keep going. And this is something that Rust can't do. Right. You can't put Rust on a GPU, really. Right. And things like this. And this is part of this new wave of technology that we're bringing into the world. And also, I'm sorry, I get excited about the stuff we've shipped and what we're about to ship. But this fall is going to be even more fun. Okay. So stay tuned. There may be more hardware beyond AMD and NVIDIA. Nice.

Swyx [00:18:25]: So that was Mojo. Yep. The first concentric circle. Yeah.

Chris [00:18:28]: Thank you for keeping me on track. So the inner circle is the programming language, right? And so it's a good way to make stuff go fast. And next level out is you say, okay, well, you know what's cool? AI. Have I convinced you? And so if you get into the world of AI, you start thinking about models. And beyond models, now we have Gen AI. And so you have things like pipelines, right? And the entire pipeline where you have KV cache orchestration and you have stateful batching. And I mean, you guys are the experts, agentic everything and all this kind of stuff. And so next level out is a very simple Gen AI inference focused framework we call Max. And so Max has a serving component. Is it Max Engine? Yeah. Yeah. Okay. There's different sections. We just call it Max. We got too complicated with sub-brands. This is also part of our R&D on branding is that- HBO has the same problem. Yeah, exactly. Well, and also like who named LLVM? I mean, what the heck is that? Right? So- Honestly, it's short. It's Google-able. Not the worst. Yeah. And VLLM came and decided to mess with it. Yeah. Yeah. No. So Max, the way to think about it is it's not a PyTorch. That's not what it wants to be, but it's really focused on inference. It's really focused on performance and control and latency. And if you want to be able to write something that then gets Python out of the loop of the model logic, like it's really good for control. And so it dovetails and is designed to work directly with Mojo. And so within a lot of LLVM, there's LLM applications. As you all know, there's a lot of very customized GPU kernels. And so you have a lot of crazy forms of attention, like the deep seek things that just came out and like all this stuff is always changing. And so a lot of those are custom kernels. But then you have a graph level that's outside of it. And the way that has always worked is you have, for example, CUDA or things like Tritonlang or things like this on the inside, and then you have Python on the outside. We've embraced that model, like don't fix what ain't broken. So we use straight Python for the model level. And so we have an API. It's very simple. It's not, it's not like designed to be fancy, but it feels kind of like a very simple PyTorch. And you can say, give me an attention block, give me these things, like configure out these ops, but it directly integrates with Mojo. And so now it's, you get full integration. And in a way that you can't get because none of the other frameworks and things like this can see into the code that you're running. And so this means that you get things like automatic kernel fusion. What's that? Well, that's a very fancy compiler technology that allows you to say, okay, you write one version of a flash attention and then cool. We can auto fuse in and so you, and the other activation functions that you might want to use, and you don't have to write all the permutations of these kernels, but that just means you're more productive. That means you get better performance. I mean, it's like a lot of things. It's just drives down complexity in the system. And so you shouldn't have to know there's a fancy compiler. Everybody should hate compilers. Like the only reason people should know about compilers is if they're breaking, right? And so it just feels like a, a very nice, very ergonomic and efficient way to like build custom models and customize existing models and things like this. And so with max, we have five, 600 very common model families and, and implemented that you can see a build stop modular.com. We have a whole bunch of models. You can scroll through them. You can get the source code and play with them and do that. That's actually really great for people that care about serving and research and, and, and all this, this kind of stuff. Next layer out is you say, okay, well you have a very fancy, uh, way to do serving on a single node. That's pretty useful and pretty important, but you know, it's actually cool, large scale deployment. And so we have a next level out, level out cluster level. And so that's the, okay, cool. I have a Kubernetes cluster. I have got a platform team. They've, uh, got a three-year commit on. 300 GPUs. And now I have product teams. I want to throw workloads against this shared pool of compute and the folks carrying the pagers want the product teams to behave and so they want to keep track of what's actually happening. And so that's the cluster level that goes out and each of the, and so very fancy, um, prefix caching on a per node basis. And then you have intelligent routing and there's a whole bunch of, uh, disaggregated pre-fill and like a whole bunch of cool technologies at each of these layers. But the cool thing about it is that they're all co-designed. Yup. And because the inside is heterogeneous, you can say, Hey, I have some in AMD, I have some Nvidia. Hey, I throw a model on there, run it on the best architecture. Well, this actually simplifies a lot of the management problems. And so a lot of the complexity that we've all internalized as being inherent to AI is actually really a consequence of these systems that are being designed together. And so to me, what, you know, my number one goal right now is to drive out complexity, both of our stack, cause we do have some tech debt that we're still fixing, but, um, but of AI in general and AI in general has way more tech debt. And then it deserves.

Swyx: Well, yeah. I mean, there's only so much that most people excluding you can do to comprehend their part of the stack and optimize it. I'm curious if you have any views or insider takes on what's happening with VLLM versus SGLang and everything coming out of Berkeley.

Chris Yeah, I don't, I have outsider perspective. I don't have any inside knowledge. Um, SG Lang seems to me as an outsider, so I'm not directly involved with either community. Sure. Um, so I can be a fan boy without, without participating, but, um, as she, they have a beef going on. So I see, I don't know the politics and drama, but SG Lang to me seems like a very focused team that has specific objectives and things they want to prove. And they're executing really hard towards a specific set of goals, much like the modular team. Right. And so that's, so I think there's some kindred spirits there. VLM seems much more like a massive community with a lot of stakeholders, a lot of stuff going on and it's kind of a hot mess. And so. But they want to be like the default inference platform that everyone kind of benchmarks against. Well, and, and I mean, it's, as far as I know, it's like crushing TRT LLM and some of the other older systems and things like that. And so I think like metrics are good probably for that. And so I can't speak to their ambition, but, but it seems structurally like they're very different approaches. One is say, let's be really good at a small number of things that are really important. Yeah. That's our, that's our approach too. One is saying, let's just say yes to lots of things. And then we'll have lots of things and some of them will work and some of them don't. One of the challenges I hear constantly about VLM for what it's worth is if you go to go to the webpage, they say, yeah, we support all of this hardware. We support Google TPUs and Inferentia and AMD and Nvidia, obviously, and CPUs and this and that. And the other thing is they have this huge, huge list of stuff that they, they support. But then you go and you try to follow any demo on some random webpage saying, here's how you do something with VLM. And if you do it on a non-NVIDIA piece of hardware, it fails. And so what is the value of something like VLM? Well, the value is you want to build on top of it. Generally, you don't want to be obsessing about the internals of it. If you have something that, you know, advertises that it works and then you pick it up and it doesn't work and I have to debug it. It's kind of a betrayal of the goal. And so to me, again, I know how hard it is. I know the fundamental reason why they're trying to build on top of a bunch of stuff below them that they didn't invent that doesn't work very well, honestly. Right. Cause we know that the hardware is hard and software for hardware is even harder these days. So I understand why that is, but, but that approach of saying, here's all the things we can do. And then having a sparse matrix of things that actually work is very different than the conservative approach, which is saying, okay, well we do a small number of things really well and you can rely on it. And so I can't say to which one's better. I mean, I appreciate them both and they both have great ideas. And the fact that there's the competition is good for everybody in the industry. It is. And so. They're in the benchmark wars state of the way that this industry evolves. That's right. But, but, but I also hear just on the enterprise side of things that they're all so good. They don't want to follow the drama. Right. And so like, this is where having less chaos and more, I can work with somebody is actually really valuable these days. And, and you need state of the art, you need performance and things like this, but, but actually having somebody that is executing well and works together, can work with you is also super important.

Swyx Have we rounded out the offerings? You talked about Max. Yeah. Yeah. I realized like it impresses me that you named the company Modular and you've designed all these things in a very modular way. Yeah. I'm wondering if there are any sacrifices to that or is modular everything the right approach? Like there's no trade-offs.

Chris If you allow me to show my old age, like my, my obsession with modular design came from back when creating LLVM. Okay. So at the time there was GCC. GCC is again, I love it as a C and C++ compiler and stuff. Right. And have tons of respect for it, but it's a monolith. It was designed from the old school Unix, like, like compilers are Unix pipe. Everything's global variables, C code. It was written in KNRC. If you even know what that is anymore. And so it was from a different epoch of software and LLVM came and said, you know what everybody teaches you in school is you have a front end and optimizer and a code generator. Let's make that clean. Right. And so at the time people told me again, first of all, it's impossible to replace GCC because so established, et cetera, et cetera. But then also that you can't have clean design because look at what GCC did and it's successful. And therefore you can't have performance or good support for hardware or whatever it is without that. They might be right. If in an infinitely perfect universe, but we don't live in an infinitely perfect universe. Instead, what we live in is a universe where you have teams, like you have people that are really good at writing. Parsers, people that are really good at writing optimizers, people that are really good at writing a code generator for x86 or something like that. And so these are very different skill sets. Also, as like we see in AI, the requirements change all the time. And so if you write a monolith that is super hard code and hack today, well, two years from now is going to be relevant. And this is why what we see a lot in AI is we see these like really cool and very promising systems that rise and grow very rapidly. And then they end up falling. It's because they're almost disposable frameworks. This concept really comes from... The language is a framework. Each of these systems end up being different. But what I believe in very strongly is that in AI, we want progress to go faster, not slower. That's controversial because it's already going so fast, right? But if we can accelerate things, we get more product value into our lives. We get more impact. We get all the good things that come with AI. I'm a maximalist, by the way. But with that comes... The reality is that everything will change and break. And so if you have modularity, if you have clean architecture, you can evolve and change the design. It's not over-specialized for a specific use case. The challenge is you have to set your baseline and metrics right. This is why it's like compare against the best in the industry, right? And so you can't say, or at least it's unfulfilling to me to say, let's build something that's 80% as good as the best. But it's got this other benefit. I want to be best of in the categories we care about.

Alessio [00:29:16]: And when it comes to using like a prefix caching, pay-per-use, etc. Paged attention. How do you decide what you want to innovate on versus like, hey, you are actually the team that has built the best thing. We're just going to go ahead and use that. Yeah.

Chris [00:29:28]: I'm very shameless about using good ideas from wherever they come. Like everything's a remix, right? And so if somebody, if I don't care who it is, if it's Nvidia, if it's SGLang or VLM, and if somebody has a good idea, let's pull it together. But the key thing is make it composable and orthogonal and flexible and expressive. To me, what I look at is not just the things that people have done and put together. I don't put it into VLM, for example, but the continuous stream of archive papers, right? And so I follow, you know, there's a very vibrant industry around inference research. It used to be it was just training research, right? And much of that never gets into these standard frameworks, right? And the reason for that is you have to write massively hand-coded CUDA kernels and all this stuff for any new thing. And I would want one new D type and I have to change everything because nothing composes. Mm-hmm. And so this is, again, where if you get some of these software architecture things right, I mean, which admittedly requires you to invent a new programming language. There's a few hard parts to this problem, but the cool thing about that is you can move way faster. And so I'll give you an example of that. This is fully public because not only do we open source our thing, we open source all the version control history. And so you can go back in time and say, like, Chris likes open source software, by the way. Let me convince you of this. I don't like people meddling in my stuff too early. But I like open source software. And so you can go look at how we brought up H100, built Flash Attention from scratch in a few weeks, built, like, all this stuff. We're beating the TreeDAO reference implementation that everybody uses, for example, right? Written fully in Mojo. Again, all of our GPU kernels are written in Mojo. You can go see the history of the team building this, and it was done in just a few weeks, right? And so we brought up H100, entirely new GPU architecture. If you're familiar, it has very fancy asynchronous features and tensor memory accelerator things. And, like, all this goofy stuff they added, which is really critical for performance. And again, our goal is meet and beat QDNN and TRTLM and these things. And so it's not, you can't just do a quick path to success. You have to get everything right and everything to line up. Because any little thing being wrong nerfs performance. And we did that in, I think it was less than two months. Public on GitHub, right? And so, like, that velocity is just incredible. I think it took nine months or 12 months to invent Flash Attention. And it took another six months to get into VLM. And this is just, like, that latter part is just integration work, right? And so now you're talking about, like, building all this stuff from scratch in a composable way that scales against other architectures and has these advantages. It's just a different level. So anyways, our stuff's still early in many ways. And so we're missing features. But if you're interested in what we're doing, you can totally follow the changelogs. And you can, like, follow them nightly where we publish all the cool stuff and all the kernels are public. And so you can see and contribute to it. And you can do this as well if you're interested.

Alessio [00:32:19]: Yeah. Do you have any requests for projects that people should take on outside of modular that you don't want to bring in?

Chris [00:32:25]: Absolutely. So we're a small team. I mean, we're over 100 people. But compared to the size of the problem we're taking on. Size of your ambitions. Yeah. We're infinitesimal compared to the size of the AI industry, right? And so this is where, for example, we didn't really care about Turing support. Turing's an older GPU architecture. And so somebody on the community is like, okay, I'll enable a new GPU architecture too. So they contributed Turing support. Now you can use Max and Colab for free. That's pretty cool. There's a bunch of operators. So we're very focused on AI and Gen AI and things like this. By the way, our stuff isn't AI specific. So people have written ray tracers and there's people doing flight safety and, like, all kinds of weird things. Flight simulation? At our hackathon, somebody made a demo of, like, looking at, I think it was a voice transcript, the black box type traffic. And then predicting when the pilot had made a mistake. And the airplane was going to have a big problem. And predicting that with high confidence. So it's like your car, you know, you're driving your car until it starts beeping when you're not slowing down. It's like lane assist. Yeah, yeah, yeah. Okay, okay. That kind of thing. But for the FAA. Wow. And stuff like this, right? And I know nothing about that. Like, this is not my domain. Trust me. This is the power of there's so many people in our industry that are almost infinitely smart, it feels like. They're way smarter than I am in most ways. And you give them tools and you enable things. And also you have AI coding tools and things like this that help, like, bridge the gap. I think we're going to have so many more products. This is really what motivates me, right? And this is where I think we talked about last time. One of the things that really frustrated me years ago and inspired me to start Modular in the first place is that I saw what the trillion-dollar companies could do. Right? You look at the biggest labs with all the smart people that built all the stuff vertically top to bottom. And they could do incredible product and research and other work. And that, you know, nobody with a five-person team or a startup or something could afford to do. And here we're not even talking about the compute. We're just talking about the talent. Well, and the reason for that is just all the complexity. It only worked, for example, at Google because you could walk, proverbially walk down the hallway, tap on somebody's shoulder and say, hey, your stuff doesn't work. How do I get it to work? I'm off of the happy path. Like, how do I get this new thing to work? And they'll say, I'll hack it for you. Right? Well, that doesn't work at scale. Like, we need things that compose right, that are simple, that you can understand, that you can cross boundaries. And, you know, so much of AI is a team sport. Right? And we want it so more people can participate and grow and learn. And if you do that, then I think you get, again, more product innovation and less just like gigantic AI will solve all problems kind of solutions and more fine-grained, purpose-built, product-integrated AI. Yeah. I think one way of phrasing what you're doing is, you know, you have this line about it's, you want AI to accelerate and that's contrarian because it's already fast and people are already uncomfortable. But I think you're more, you're accelerating distribution. I have mixed feelings about the words democratizing, but that's really what you're doing. Well, it used to be, democratizing AI used to be the cool thing back in 2017. I know, it's not cool anymore. Yeah, I know. But it used to be the cool thing. And what it meant and what it came to mean is democratizing model training. Right? And it's super interesting. Again, as veterans, like back in 2017, AI was about the research because nobody knew both what the product applications were, but then we didn't know how to train models. And so things like that. Things like PyTorch came on the scene. And I think PyTorch gets all credit for democratizing model training, right? It's taught to pretty much every computer science student that graduates. That's a huge deal, but nobody democratized inference. Inference always remained a black art, right? And so this is why we have things like VLM and SG Lang because they're the black box that you can just hopefully build on top of and not have to know how any of that scary stuff works. It's because we haven't taught the industry how to do this stuff. And, you know, it'll take time. But I think that that's not an issue. It's not an inherent problem. I think it's just that we don't have like a PyTorch for inference or something like that. And so as we start making this stuff easier and breaking it down, we can get a lot more diffusion of good ideas. I want to double click a little bit more on some technical things. But just to sidetrack on, you know, VLM is an open source project led by academics and all that. And I think a lot of the other inference team, because effectively every team is a startup, like the fireworks together, you know, all those guys. Your business model is very different from them. And I want to spend a little bit of time on that. Happy to talk about it. You intentionally, well, you believe in open source, but, you know, it's not just that. You just choose not to make money from a lot of the normal sort of hosted cloud offerings that everyone else does. Yeah. There's a philosophical reason and differentiation. There's a whole bunch of reasons. Yeah. So like maybe remind people of what that is, like how basically how they pay you money and what they get for that and why you pick that. So, again, we're doing this in hard mode. Right. We took a long path to product. We're not just right away. We're going to get a few kernels and have some alpha and go buy a GPU reserve thing and resell our GPUs like that. That path has been picked by many companies and they're really good at it. So that's not a contribution that I'm very good at. And I'm not going to go build a data center for you like this. There's people that are way, way better than that. Okay. And so all the best luck. I want that. Literally Crusoe like walking the Stargate grounds. I want those people to use Macs. You know, I'm pretty good. That's a Crusoe. Yeah. I'm pretty good at this software thing. And so you can handle all the compute. Moreover, if you get out of startups, you have a lot of people that are struggling with GPUs and cloud. Right. And so GPUs and cloud fundamentally are a different thing than CPUs and cloud. And a lot of people walk up to it and say, it's all just cloud. Right. But let me convince you that's not true. And so, first of all, CPUs and cloud. Why was that awesome? Well, all the workloads were stateless. They all could horizontally auto scale. CPUs are comparatively cheap. And so you get elasticity. That's really cool. You can load them pretty quickly. Yeah. It's not like gigabytes of weights. Yeah. And so it turns out what business, you know, knows what they're doing in two and a half years? Nobody. Nobody. Right. And so cloud for CPUs is incredibly valuable because you don't have to capacity plan that far out. Right. Now you fast forward to GPUs. Well, now you have to get a three year commit. Three years. Right. So you have a three year commit on a piece of hardware that Jensen's going to make obsolete in a year. Right. And so now you get this thing. And so you make some big commit. And what do you do with it? Well, you have to overcommit because you don't know what your needs are going to be and you're not ready to do this. Also, all the tech is super complicated and scary. Also, GPU workloads are stateful. And so you talk about like the fancy agentic stuff and you all know all this stuff. Yeah. It's stateful. And so now you don't get horizontal auto scaling. You don't get stateless elasticity. And so you get a huge management problem. And so what we think we can do and we can help with people is say, OK, well, let's give you power over your compute. And a lot of people have different systems and there's very simple systems that go into this. But you can get like five X performance TCO benefit by doing intelligent routing. That's actually a big deal for a platform team. They don't like to have to deal with this. They want hardware optionality to get to AMD. They want this kind of. Power and technology. And so we're very happy to work with those folks. The way I explain it in a simple way is that a lot of the endpoint companies and there's a lot of them out there. And so you can't make they're not all one thing. They obviously have pros and cons and tradeoffs. But but generally the value prop of an endpoint is to say, look, AI and AI software and applications and workloads. It's all a hot mess. It's too complicated. Don't worry your little head about it. I'll take AI off your plate so that you don't have to worry about it. We'll take care of all the complexity for you. And it'll be easy. Just talk to our endpoint. Our approach is to say, okay, well, guess what? It's all a hot mess. Yes, 100%. Like, it's horrible. It's worse than you probably even know. And get to tomorrow. It's going to be even worse because everything keeps changing. We'll make it easy. We'll give you power. We'll give you superpowers for your enterprise and your team. Because every CEO that I talk to not only wants to have AI in their products, they want their team to upskill in AI. And so we don't take AI away from the enterprise. We give power over AI. We give power back to their team and allow them to both have an easy experience to get started. Because a lot of people do want to run standard commodity models and you do want stuff to just work as table stakes. But then when they say, hey, I want to actually fine-tune it, well, I don't want to give my proprietary data to some other startup or even some big startups out there. And you know whose these are. That's my proprietary IP. And then you get to people who say, hey, I have a fancy data model. I actually have data scientists. I actually have a few GPUs. I'm going to train my model. Cool. That got democratized. Now how do I deploy it? Right? Well, again, you get back into hacking the internals of VLM. And PyTorch isn't really designed for KV cache optimizations and all the modern transformer features and things like this. And so suddenly you fall off this complexity cliff if you care about it being good. And so we say, okay, well, yeah, this is another step in complexity. But you can own this and you can scale this. And so we can help you with that. And so it's a different trade-off in the space. But I will admit that there are time to market and revenue growth and stuff like that. And that has been much faster because they didn't have to build an entire replacement for CUDA to get there.

Alessio [00:41:37]: Nice. And when it comes to charging, people are buying this as a platform. It's not tied to token, inferred, anything like that.

Chris [00:41:45]: We have two things going on. So the Max framework and the Mojo language, free to use on NVIDIA and CPUs, any scale. Go nuts. Do whatever you want. Please send patches. It's free. Right. Why is this? Well, it turns out CUDA is already free. NVIDIA is already dominant in here. We want the technology to go far and wide. Use it for free. It would be great if you send us patches, but you don't even have to do that, right? We do ask you to allow us to use your logo on our webpage. And so send us an email and say, hey, you're using it and you're scanning on 10,000 GPUs. That'd be awesome. But that's the only requirement. If you want cluster management and you want enterprise support and you want things like this, then you can pay on a per GPU basis and you can contact our sales team. And then we can work out a deal and we can work with you directly. And so that's how we break this down. And also, let me say it. One thing I would love to see, and again, it's still early days, but I would love to see PyTorch adopt Macs. I'd love to see VLM adopt Macs. I'd love to see SGLang adopt a Macs. We have our own little serving thing, but go look at it. It's really simple. It would be amazing. And again, we're in the phase now where I do want people to actually use our stuff. And so we have historically been in the mode of like, no, our stuff's closed, stay away. But we're phase shifting right now. And so you'll see much more of this being announced. Yeah. I don't know how to make this happen, but I think you win when Mistral, Meta, DeepSeq, and Quen adopt you and ship you natively, right? How does that happen? I don't want to talk that far in the future. I don't think it's that far. We may have an industry-leading state-of-the-art model launching first on Mac soon. I'll stay tuned for that, but also let us know. But that hasn't happened, so assume it doesn't happen. I think that's when it really tips. Because then everyone's like, okay, it's good enough for them, it's good enough for us, right? And then you get the rest of the industry. Yeah. But again, I mean, I'm in it for a long game, right? And I realize, again, the stuff I work on, it takes time for people to process. And so what we need to do and what I want us to do and what I ask the team to do is keep making things better and better and better and better and better. And there's like an S-curve of technology adoption. And so I think it's great that there's a small number of crazy early adopters that were using our stuff in February. Before it was open source, and it made no rational sense. It barely worked. But it was amazing, and I'm very thankful for those people. And then, of course, we open sourced it, and we started teaching people, and you get a much bigger adoption curve. You make it free, go adopt and go. And then, as you say, there's more validation that will be coming soon. And each of these things is a knee in the curve. But what it also does is it gives us the ability to fix bugs and improve things and add more features and roll out new capabilities.

Swyx: How does this feel, rolling this out, as compared to software?

Chris: Oh, well, so let me reinterpret your question. Given you've done a few interesting things in the past, what have you learned? And what are you not doing again? That actually is a better question than the one I asked. Because Swift is too narrow, almost. The character of Swift, so just because I assume most people don't know about this. Character of Swift was I started as a nights and weekends project in 2010. Hacked on it alone, nights and weekends, for a year and a half. Eventually told management at Apple about it. Their heads exploded. Like, why do we need something? Objective-C is good enough, why do you need this? Got approval to have a couple more people get involved in it. You were on a fellowship or an internship at Apple at the time? No, I was leading the developer tools team. I think you were doing something, yeah. Yeah, I was leading a huge team. Let's just say this was not my day job. But so, started in 2010. It launched publicly by Apple in 2014. And by the time it launched in 2014, only about 250 people in the world knew about it. Most of whom were in my team. About 200 and something of them were in my team. And then it was senior execs, marketing, Tim Cook, etc. And this was the category of people that knew about Swift. So we had built it in secret. Literally an NDA. Within Apple. To know about it. When we launched it, part of the requirement was that you had to be able to submit apps to the app store in Swift. That was the requirement put on me. And so it's like, cool, that sounds great. And so we launched it and said it's a 1.0. So you're launching a 1.0, brand new programming language. Nobody has ever seen it before. No internal user. Like one demo app. It was a frickin' nightmare. Yeah. So it was a nightmare for the community. Because, I mean, fortunately a lot of people were excited and wanted to adopt it. And a lot of people did adopt it right away. But it was not battle-hardened. It had tons of bugs. We should have launched it as a 0.5. Right? And so it took another year for it to become. Pretty good. And then two years for it to become quite good, in my opinion. Also, none of the software engineers at Apple knew about it. And so they, their heads exploded. They said, wait a second, why are you replacing Objective-C? I joined Apple because I love Objective-C. Why didn't you ask me my opinions about the new programming language? Right? And so there's that whole dynamic. Oh, was there a company mandate that they had to write Swift from now on? No. But still, it's like, wait a second, this isn't the company I thought I joined. Right? And stuff like this. Right? And so there's this huge amount of turmoil and drama and nonsense that came out of that. And so, okay, fast forward to Mojo. Lessons learned. Hey, one, don't have a hot start. And so we launched Mojo a long time ago, before it even made sense. And we called it a 0.1. And so that, how's that honesty in advertisement? Right? It's like, this is 0.1. Please don't use it. But if you're interested, we'd love your feedback. Right? And so soft start, go. Second thing that's very different is that in Swift. We had one demo app. And so you have a very, I think, high power team building a language that had done lots of credible stuff, but was building a language for iOS developers. And the compiler's written in C++. Right? And so, yeah, there's sympathy for the user, but not a lot of understanding and not a lot of learning internally when we launch. In the case of Mojo, guess what? Modular is Mojo's first customer. We have more Mojo code in our repository than any other language. Right? That's awesome. And it's open source. And we open source like 650,000 lines of Mojo code. Right? This is a lot. And so we suffer and we drive the features and the improvements based on our needs. We also appreciate the community. And we have a whole bunch of contributions coming in. And somebody just optimized my string to get rid of a bit out of my string implementation, which was suboptimal. And so that was super awesome. But driving it that way makes sure it's real. It's grounded. It's on the use case. We're trying not to overpromise. Even when you're asking me what it's useful for earlier, I didn't say it's a replacement for Python. I said it's a go-fast language. Someday it may be a pretty credible Python alternative. But for right now, it's good at a specific class of use cases. And if you're interested in those use cases, like making GPUs go brr, Mojo's awesome. But if you want a replacement for Rust end-to-end, give us six months. Yeah. Yeah.

Swyx: I mean, you're a force of nature. I think there's a lot of mystery around what is going on with Apple's AI initiatives. And I think the consumers suffer. At the end of the day, the end users are waiting for this. And it's not happening.

Chris: Unfortunately, anything I know is massively out of date. They've changed and reorged and grown and culture. And it's a very successful company. And so I think that they probably feel success. And they're having trouble adapting to changes in the industry. And that's pretty typical of a lot of big companies. And so I can't speak to the specific causes. Speaking of Google, obviously, I think they were one of the earliest. What are we talking about? The invented transformers. A lot of other things. TensorFlow, remember that? Yeah, exactly. Huge. Yeah. NTPs. I credit Google with making AI open source. Well, they did not have to open source TensorFlow. Yeah. That was an incredible decision. Full kudos to Jeff Dean and many of the other people that were involved in this. Because they said, you know what's actually the most important thing for Google? Is for AI to go faster. How do we do that? We open source TensorFlow rather than making it some proprietary internal thing. Which they had a previous system called Disbelief. And so that is a huge moment that set the stage for PyTorch to be open source. And for the research to be open. And for all of these things. Because they decided the value system was AI go faster. The transformer paper being published. So many contributions from Google came from that. I don't think Google gets enough credit for that. Yeah. Why is it better for Google for AI to be open source rather than Google owns it? Well, so I can't tell you if the bet worked. But I can tell you that that was a bet. But from my outsider now perspective, because I haven't been at Google for over five years somehow, time flies. The bet makes sense when you have an amazing team of researchers and Swyx that can go incorporate this into your products. And so Google does have billions of users. It has all the product services. It has all the different applications. And it has an incredible density of talent. And I think that Google's recent announcements, so just after Google I.O. and things like this. Yeah, we were both there. Yeah, it's like Google's actually working, I think. It's pretty impressive. And for a while they were dealing with organizational drama and Google Brain versus DeepMind and some of this stuff. And I can't speak to what they've done. But it seems like they're a much more unified team. They're executing well. They're getting research into product. And so it feels like a different Google to me. Yeah, it totally does. It used to be that there was just two of everything in Google. And you didn't know which one to use. Killed by Google. They all deprecated in a year. So yeah, I think they've gotten the memo, whatever. Yeah, and the other thing that's super impressive to me about them, and this is me fanboying Google, right? Sure. After railing, it's the trillion dollar companies. But the things that they announced they were actually shipping. Yeah. So much in AI is. This is more Apple shade. I wasn't specifically saying Apple. This is very common in AI. And Modular's done this in the past too. This is why. So I gave this very deep tech talk. I sent you a link to the GPU mode talk. And slide two was warning. You can actually use this. This is not vaporware. Everything here you can reproduce. These claims you can download. This is actually real. Here's links to the source code. I was wondering why you stressed that so much. I'm like, who hurt you? I know. I mean, there's so many claims. Nobody knows what is real anymore. I know. Right? And I mean, there's literally been product demos where it's like. Some electric semi rolled downhill instead of working under its own power. And nobody knows what's real.

Alessio [00:52:18]: I knew they started to work. There's like a WhatsApp chat. We're like playing soccer at the Google field during the week. And about six months ago, some the admin posted. It's like, hey, not enough people are showing up anymore to play soccer at lunch. What is going on? And I think that's when people started. That's your Google indicator. Working again. Yeah. There you go.

Chris [00:52:35]: I mean, Sergey Brin was at IO. And he's definitely. He's like working again. And it's awesome. Yeah. So I have mad respect for that. Right. And so my values are aligned with people who ship stuff. Yeah. Because that's what impacts the world.

Alessio [00:52:47]: Let's talk about open source a little more. There's the more recent maybe open source thing, which is deep seek, obviously. And I think specifically in your case, you know, they worked at the BTX layer of the GPU, which is like even lower and more proprietary than CUDA. Yeah. I'm curious how both in terms of like, obviously the impact was like huge, but maybe impact on how much people should actually try and move away from this proprietary. Because now the next, from my understanding is like the next set of chips, all the code is like useless.

Chris [00:53:17]: Well, so it's not widely known, but Blackwell is not compatible with Hopper. Right. Hopper kernels don't always run on Blackwell, for example. Right. But so your question is like, what does it mean for the industry?

Alessio [00:53:27]: Well, it's like, why is it so important? Like why the DeepSeek, the DeepSeek example is like so important of like they need to navigate all this like proprietary stuff just to make it work. Yeah.

Chris [00:53:37]: So I'll give you my lived experience. Yeah. Because deep seek came out in December, which is when I, and probably you noticed it. Right. But then the world had a big wakeup call and video stock price went down and all that stuff like a month later. Right. And so here's my explanation of what happened. Okay. What the deep seek team did was really impressive research. They pushed MLA forward, like they, which is a form of attention and they pushed a low precision training forward. They pushed a whole bunch of stuff forward. They reverse engineered some PTX instructions that weren't well known at the time. Right. Yeah. And so a lot of people were just like, and it was a Chinese team, right. Which put Americanism, threatened Americanism, right. And things like this. And so what I found really exciting about it was they pushed the research forward and they did this incredible things. They showed the world that it was possible and they opened it and they published it and they actually taught the world about it because I don't know why they chose to do that, but it's because they believe in openness and AI moving forward. Right. Yeah. Yeah. Yeah. Yeah. So I think the other thing that I found striking is that the world's reaction to that was more striking to me than the actual world. Or whatever. I don't know. Yeah. Because, so first of all, there's the Chinese American drama, which geopolitics is not exactly my strong point. So I w I, I get it then I push that aside. Right. But the other thing I found really interesting is that people said, wow, okay. Only deep seek is able to go down to the PTX level, but that is standard. That is what all of the leading teams do in the case of modular. Yeah. We only work at that level because we get rid of all of CUDA, right? And so we've replaced the entire stack. And so we only do that. But a lot of teams that care about performance will actually go down and use the PTX, TensorCore, Instruction, Foo, which isn't really documented. You kind of have to figure it out and look at cutlass code. NVIDIA doesn't make it as easy as they should to do this. I wonder why. Well, I think it's because they're breaking in Blackwell, and so they didn't want people to actually use it, and so they had their own issues, right? But good luck with that. There's a lot of smart people out in the world. And so to me, I thought it was really interesting because there wasn't a lot of awareness of how that level of the stack worked. For me, I thought it was a great wake-up call where people were like, oh, wow, if you work at this level of the stack, a level of the stack I have inhabited for decades now, you have power over compute. You can understand and solve problems. You can drive research forward in ways that nobody else can do. And this is what the trillion-dollar companies do. It's not just DeepSeek. But I think DeepSeek was a huge wake-up call, and it drew attention to that layer of the stack because it wasn't just like throwing layers on top of PyTorch or VLM or it was like doing that fun amount of work. And I thought that was incredible. Now, the challenge with it and the challenge with the way DeepSeek did it and the way that everybody else does all this work is that it's completely specific to one GPU. It's not just that you're working at the PTX level. It's that you're writing code that really can only work on that one GPU, and that means that when Blackwell comes out, you have to throw it away and write a new one, right? But here's the open secret. That's what VLM is. Go look at VLM. They have different kernels for AVLM. They're now trying to catch up with Blackwell. And so, state-of-the-art for these systems, since Gen AI, before that, there were fancy AI compilers like XLA and that kind of stuff in the Trat AI world provided some scalability. But in Gen AI, it's a rewrite all the things when a new piece of hardware comes out. And so this is what Mojo is solving, right? And this is where, again, we can't turn the incremental cost of a new piece of hardware to zero, but we can massively reduce it. And so this is a really exciting time, I think, for us that we've demonstrated now, but also what it means for the future. I still also wonder their hiring or internal training, that they managed to have a small team that does this in the same way that you do. I think they have a pretty significant team. I have no insider information, of course, but it's not like a five-person team. It's like hundreds of people. Okay, yeah. So it's... Yeah, you know, it's not a thousand, right? It's amazing, especially like, I mean, presumably there's some language barrier, but even ignoring that, just getting that amount of talent density in one company, it's not a significant task. Yeah, well, I agree with that. Building a company is hard. I have no visibility on how they built DeepSeek, but all I can say is thank you to DeepSeek for publishing their work. Yeah. Because they didn't have to do that. And I think it left temporarily a lot of people flat-footed. Right? And it was kind of embarrassing for certain groups, and I think a lot of people paused and were like, ooh, crap, what do I do about this? But I think that what it did is it pulled forward progress in AI by like six months. Yeah. They didn't just do GPU-level stuff. They also had a file system. Oh, yeah. If you've looked at... Yeah, it's incredible. Do you see... Obviously, that's not modular as bread and butter, but do you see a potential there for someone else to, I don't know, take that and run with it? I've been very obsessed with the inference problem for a couple of years, and so I don't know the best way to solve that problem for training. Google internally has a great... Indeed. Right? And so where is that for the rest of the world? Yeah. I'm honestly just not the right person to answer that question because I have my own obsessions. Yeah. Well, talking about inference, reasoning models and inference time compute, very big topic. Does anything change or does nothing change? Because it's just more inference. It depends on change from what, right? So when we started modular over three years ago, we made the ridiculously weird bet at the time to focus on inference instead of training. Yeah. And again, at the time, I'm used to this. People are like, what's wrong with you? Everybody obviously knows that training is the thing, and training, training, training, and people building these massive clusters, and all the spend is on training, et cetera, et cetera, et cetera. And I said, well, yeah, I understand why you see that, but I've lived this at Google. Google's like five years ahead of the rest of the industry. The most scaled inference, yeah. Yeah. Training scales the size of your research team. Inference scales the size of your customer base. Right? And so the thing that happens between those is research gets into production. And so the gap is research getting into production. Once you do that, suddenly it scales like crazy. So I didn't plan for Gen AI. I did not plan for inference time compute. And things like this. But that bet on the production use case, because it scales with the number of applications of AI, not just the hot research team, right? Which, you know, is pretty important. It's just not something I've focused on for the last few years. That was controversial. And so I think now the whole world's flipped, and I think the world gets it, right? But that was really because of lived experience. And so when you come to these new techniques, right? Another controversial thing, going back to the CPU thing, is why are you starting with CPUs? The answer at the time was pre-processing. Post-processing, full system integration, networking, you need a CPU to feed the GPU, like very standard things. But now you say KV caches. Like your eviction policy runs on a CPU. Like that Radix hashing algorithm and block hashing and all that stuff happens, like primarily CPU. That's really important for performance, because if you have latency in these steps, like you're not keeping your GPU utilized, right? And again, this comes back to the rewrite and rust and things like this. All the agentic stuff and things like this, I didn't predict that, but I'm not surprised and I think we're going to see more. And so tight integration, optimization across boundaries, like these are things I believe in. And I think this is how you move the world forward. It's amazing how like basically all the smart programmers I know always focus on where the bottleneck is. And it always leads you to the right answer. Like if you just are very clear eyed about that. And I don't know, it's nice to see that happening. The other talking point I'll mention for you, because you didn't bring it up, but it's something that other people are talking about, is that actually now because of the requirement for train of thought and reasoning models, inference is now part of training.

Swyx [01:01:06]: Okay, yeah.

Chris [01:01:07]: Because you need to inference. RL has always done that. To RL, yeah. And so that actually is a plus in favor of what you're doing anyway. Yeah, absolutely. Because it's the same code. So my experience with RL systems were like at scale, DeepMind style, AlphaGo and things like this. So it's been a couple of years ago, but setting those things up was incredibly complex. Because now you're dealing with cluster scale orchestration, you're batching across all these agents. And again, none of the systems are set up for that, right? You've got PyTorch if you want to train a model, but nothing is set up to do this. And so again, I can't speak to all RL systems everywhere, but they ended up being like duct tape and bailing wire and super crazy stuff. And it was incredible. Again, it's incredible what a team of experts who knows the full stack and what they can achieve, but it shouldn't be that hard. And so we're not just not focused on solving that, we're not focused on solving the problem yet. Maybe we'll get there. We have the building blocks. We have the building blocks. And again, we're not focused on solving training yet. Maybe we'll get there. I have some ideas on logical steps to do that, but I want to make sure we're grounded and we solve things all the way end to end. We make people happy. People talk about our stuff from their voice. It's not just me talking about our stuff. And so we're in that phase where you'll hear people talking about our stuff soon. I think people already are, but they will even more. And I think I appreciate you coming on the podcast to talk about that more. And we want to turn to some personal stuff.

Alessio [01:02:31]: So I think that was a great Modular story. I think now on the personal side, there's a couple of things. So when you first joined us, I think I saw it was September 2023. So that was a year and a half into building the company, something like that. Now three years and a quarter. And what are personal learnings, both from obviously being a leader in a company, having a growing team that you're managing, having a lot of responsibilities that are not technical anymore? Kind of run us through some of that. Yeah.

Chris [01:02:58]: So for me, I've built a lot of things. I've built large teams from scratch before, but they've all been at established companies. I've worked at a startup before, but it was somebody else's startup. And so this is the first startup that I've founded and then built a significant team and built a product that takes years to build. So a lot of the lessons I learned were super valuable and allowed me to achieve some of the stuff. But it's also very different. And so one of the things that's very different is it's very personal. And so I've lost people from our team before, like many times. I've had to fire people and people have quit and gone Apple to Google or whatever. But that was never personal in the same way it is at Modular. And so this is something where the intellectual side of my brain knows if somebody leaves, it makes sense. They had a life change. I don't want to get in the way of their family or whatever. I mean, I intellectually know that. But on the other hand, it hurts a little bit. And so I think I'm getting better at handling some of that kind of stuff. The other thing is that when you're growing a team from zero to 100 people at a big company, you have all of the infrastructure. And the infrastructure is mature, right? And so even if you grow to a team of 100 people, you're still tiny compared to an Apple or Google or a company like this. You're still tiny in proportion to the size of the overall scale. And so they've already got all the recruiting and all the other stuff and all the legal and finance and all that kind of stuff going. They've got the manager training. They've got all this stuff going. And so in a startup, you're sometimes getting some hot messes where it's like, okay, well, I need to do reorganization and things like this. And so that's been good learning. It's scaled into that well. And another thing coming back to the people telling you it's impossible. Actually, here's probably the most important thing is that I'm used to being told that things are impossible. And then I'm pretty bullheaded and I have a formula. And so there's a path to success. And I can explain if you want. But I'm used to the feeling when it's that year one and it's just a completely skunkworks, let's prove a thing phase. I'm used to that. Okay, cool. Now we're telling people about it, but it's still not good enough. And people tell you all your stuff sucks. I'm used to that. Like, okay, well, you get into that window where all this stuff is almost there. People are sweating. It's really hard. There's like all these like seemingly impossible things. It's a significant team, but the pieces don't line up yet. And so we went through some of that last fall and people like, oh my gosh, maybe it will never work. And you get this like anxiety. And I think the thing that I didn't appreciate going through that, and this is also why I'm so excited to be in this phase that actually impacts other people's, like their thought processes. Cause. Cause they haven't been through it and they, and they like, trust me, we'll get there. Well, when like exactly what, like you get these like super analytical engineers who want to understand everything and they're really expert in their part of the stack and they don't really have that established trust with the other department and the other department, the other department are all lining up. And so definitely some mornings from that, but, but this is where, again, you get out of that R and D phase and you get into the execution phase and it's like, okay, well, engineers are really good at taking an imperfect thing that works end to end and making it better and better. And better and better and better. And so it's, it's just like feels fundamentally different modular now than it did in any of those previous phases. Yeah.

Alessio [01:06:04]: What's your day to day. Like you have a lot of meetings, you have just a few.

Chris [01:06:08]: Yeah, I have a lot of, you're talking about my, my, my lived life. Um, a normal weekday for me is, um, I wake up about seven 15, my wife and I get the kids out the door and she drives them to school at about eight. I usually do a half hour, 45 minute walk with my dogs, get exercise strenuous up and down the hill. Heart rate goes up. Which is good. Listen, podcasts, for example, yours. So that's, you know, why, why I have time to actually follow exciting, cool things that other people are doing, get to work at nine and so work nine to six or seven or something like that. And most of that is meetings. And so that's me trying to solve whatever the problems are of the day, get home, dinner with the kids. I insist on eating with my family, hang out with them until bedtime and then crush through for like two, three more hours until I pass out and do that regularly. The second work day. Yeah. Yeah. Yeah. Yeah. And on the weekend, it's amazing because I get a lot of time to work, not all day because I do things with kids and stuff, but I get a lot of time and there's no meetings. So that's amazing. That's when stuff actually gets done. Yeah, exactly. I think the, the key here is some kind of strategic review time that you lock off because like, I think people often say, you know, you're there's, there's times when you're working in the business and then there's times you're working on the business where you step out for a bit. Often for, for founders is when you do a board meeting. But I wonder if there's any. Anything that's very meaningful for you. Do you have a coach? Do you have something like that? Yeah. Yeah. So I guess there's two people I really owe a lot to, um, or two, two categories. Like, so one is my co-founder, Tim. Yeah. So Tim and I formed the company together. Uh, we walk every week, every Friday, catch up and it's like that zoom out and try not to make it tactical. And so make sure that we can bounce crazy ideas off. And I have a lot of crazy ideas, believe it or not. He does too. But then ground ourselves on execution and go. The other thing is. Yeah. This combination between my wife, who's a sounding board and the executive team. And I get a little bit crazy and want to solve the industry problem. And the exec team pulls it back to, okay, next quarter, let's make sure we have a plan. Let's make sure we can communicate. Let's decide what the actual priorities are. Okay. We can have three priorities, max for the whole team, not, not 50 or something. Otherwise they're not priorities. Yeah, exactly. Everything can't be the top priority. Otherwise nothing else. Right. And things like this. And then my wife who keeps me sane and has, you know, I don't know.

Alessio [01:08:27]: She's an amazing life, life coach. How much do you get your wife involved on like the actual work?

Chris [01:08:33]: Like does she know, like, you know, yeah, she has her own thing going on. Uh, my wife runs the LVM foundation. And so she's got a bunch of things going on plus kids and everything else too.

Alessio [01:08:41]: She's like, I don't want to hear about this Cardinals.

Chris [01:08:43]: Well, but she's, she's great for helping me. So I'm more IQ than EQ. And so working with humans is not, it's an acquired skill, not a natural skill. And so I think this is something where once. Uh, you know, I often ended up this place where some weird thing is happening, what the heck is going on? It's like, Chris, it's obvious, like they're saying this, but this is what they actually mean. I'm like, oh, I never thought about that. You know, you mentioned coding agents. Yeah. What do you guys use internally? What do you like? Yeah. So, I mean, as of this recording, I personally use cursor and so cursor is great for, I mean, it's the best thing I've seen and I don't spend a lot of time dabbling with things. And so, but cursor is, uh, so I write a lot of C plus plus and mojo code, the key thing for working on. Mojo code in AI coding tools is make sure you have a lot of code in the context window. Yeah. And so cursor and these tools can index really well. I was thinking you open source mojo code. Actually, that's pretty good. That's one of the reasons we did this. Yeah. And one of the many reasons we did this. And so this is what we saw at our hackathon is people could go zero to hero with mojo because you could just put, you just index this entire huge code base and it's, it's phenomenal. And then learning a new language is actually easy when the AI is doing a lot of the mechanical stuff for you. And also it looks like Python, so you could read it, but it just like massively scales the on-ramp. Yeah. Yeah. So for new language adoptions, AI is, let me convince you, AI is cool. I would just go ahead and like, actually like ask the data sets people at the big labs what they need. Yeah. And then just like, I've asked them all feed it to them. Yeah. Just take the code, please label it for them. It's Apache too. Like just go. Yeah. Yeah. We're adding markdown files. There's a cloud.md. And so we're doing, we're doing some of the basic stuff, um, people within the company are dabbling with cloud code and some of the stuff. And so I don't have personal experience with that, but, um, but for me, I found that it's mostly, it is very. Useful, but it's really about boilerplate. And so it's not about inventing new algorithms and stuff like this. And it's probably, cause I'm not building a react component. Yeah. I mean, so the labs are touting that they are training or benchmarking their models for writing CUDA kernels. Right. I wonder if you see a noticeable performance improvement when you change model to model. I don't know if you, you probably don't benchmark them. I haven't, I haven't looked at that specific use case. Um, but, uh, people often ask me. Hey, Chris, why are you building a new programming language when AI is going to write all the code? Similarly, if you look at a lot of this, like let's generate a CUDA kernel, you start to ask and wonder, or at least I do a lot. What is the purpose of code? What I've reflected on the way I currently think about it subject to change, obviously, because we're all learning. I take a step back. I say, well, code isn't really about telling the computer what to do. Code is about humans being able to understand what code does. And so someday when we get AGI or ASI or something like this, maybe it can be completely opaque. And I really don't have to know, we're not there yet. So, and I don't know when that will happen, but in the meantime, like I want to be able to look at what the code actually does. And I live in a world of constraints. I need to know I have a product which has all these features. If I go add another feature, what happens? It's going to hit my latency budget. Is it going to crash around memory? Is it going to do, is it going to cost too much? Like I need to be able to reason about this. And so to me, I look at a lot of these coding tools, you know, scaled beyond where they currently are now, but into the foreseeable future is saying like, okay, well, it's like hiring another engineer onto your code base or into your team. And fundamentally coding and software engineering is a team sport. And like you have product managers, you have engineers, you have a lot of things. And if you automate all the engineering of code, maybe you get to product managers only and marketing only or something theoretically, but you still want to reason about what the code does. And so in that, yeah, interpretability argument. And so in that world, like what is actually the most important thing, most important thing is you can. Express. Everything the hardware can do, because you don't want to have some capabilities, some costs or some, some boundary you can't penetrate mojo does that. The second is you want readable code that you can actually understand, right? And so you don't want assembly language or something like that. You want something high level and expressive and easy to understand. And so like, this is where I think mojo is really unique. And then the AI coding tools I see is a straight value add for adoption. Cause I mean, I've already seen what people that have never touched any of this stuff can do. And it's just incredible. You put somebody that's. It's intelligent. They know the use case and now you put these tools in their hands and they can do amazing things already. And so I find it super empowering. And again, come back to me wanting to see humans being empowered with new technology and being able to upskill and be able to learn, you know, you're not a GPU programmer today, but you know, tomorrow, hopefully there'll be 10 times as many people programming GPUs. I think that'll make the world a better place. And so I don't think we're going back to the world of CPUs. I think GPUs will only get more important. And if we can help people do that, I think it's great.

Alessio [01:13:24]: You mentioned some of the research on inference and the archive papers. How do you keep up with archive?

Chris [01:13:31]: Yeah, it was, we're a Slack shop. And so we have a paper channel. It's I think, I think there's a, I don't know, somewhere between three to 10 papers a day that come through. And, and so we have amazing, smart people that do that. I also follow, uh, the Reddit communities and things like this. I'm an old school person that uses RSS still. And so if you have an RSS feed, then it's way easier for me to follow you. Which reader? Yeah. And, and, uh, I use Feedly. Feedly. Yeah. But, um, archive has the ability to follow specific groups. And so I do that for various groups. And so that's, that's another technique. Any notable papers come to mind that you want to give a mention to, or, or authors, you know, that you're really watching when anytime they publish, you're like, I'm reading that one. I can't give you a well-considered answer. Um, just, just this morning, a person from Microsoft published a paper, uh, forget his name offhand, but, uh, he just published a paper. It's a really cool paper about auto-generating, uh, flash attentions and doing blocker or something. It had some cute name with block in it. And so anyways, I mean, there, there's a ton of things going on. Okay.

Alessio [01:14:33]: So that kind of, yeah, you know, it's interesting to just see what you pay attention to. Last but not least, the question that everybody wants the answer to, have you finished building the Lego robotics table for your kids that you mentioned last time? And what's the, do you have a new project that you're working on?

Chris [01:14:47]: Massively forgot about that. So we still use Lego robotics table. So this is a big 4x8 sheet of plywood with a bunch of, uh. Uh, two by threes around the edge, but a 4x8 sheet of plywood was pretty hard to work with. And so it breaks apart into three decomposable modular sections. And so it's super great. And the kids are getting way better at programming and Legos and all this kind of stuff. Um, gosh, what's my most recent project. I was just building swords in the shop with kids on a bandsaw. And so you take a piece of wood, you have a bandsaw, give it to a kid and you say, don't cut your fingers off. Turns out that advance. I mean, I'm obviously joking a little bit. They do a lot of. They get a lot of oversight, but a bandsaw is actually a very safe tool. And so the reason for that is that a bandsaw, which if you probably a lot of people have never seen a bandsaw, you can do Google search for, you have two wheels and then you have a blade that goes around the wheels and you've got a table. And the cool thing about a bandsaw is that you can crank down the opening towards the blade. So it's just, just as big as the piece of wood and also pulls the wood into the table. And so because it pulls the wood into the table there, the risk of like something called kickback. And there's a lot of other things like this is very low. And so you can basically tell a kid, look, you see your fingers, keep them away from the blade. And there's no like sudden jerking or other things that if you, you know, use the wrong technique, you can get yourself into real trouble. And if you keep the guard all the way down, then you can't get like an arm in there or something like that.

Alessio [01:16:11]: But, uh, did you make your whole garage woodworking?

Chris [01:16:14]: Yeah, I'm a cars outside kind of guy. So that's again, my thank you to my wife for tolerating my odd behaviors. I mean, I love building things, right? And so this is fundamentally when you talk about, you know, what makes me tick is I love the joy of discovery, right? And so whether it's building an amazing team or building a new table, you know, built like dining room table or things like this. Uh, you can look at my website. I'm not a very good web designer. I have some woodworking projects on there, but I love building software. I love building and solving the problems that come to this. I'm not super great at building things that are rope mechanical. And so maybe I'd be good at building one chair, but I'm not going to build eight chairs, go around dining room table. That would just drive me crazy. The discovery and the learning is what you can build the machine that builds the chairs. There you go. That sounds great. Right? You spend 10 years building and things three weeks crank it out. Yeah.

Alessio [01:17:02]: Uh, yeah. Awesome. Yeah. Any call to actions?

Chris [01:17:05]: Like, are you hiring like, yeah, we're, we're, we're hiring a small number of elite nerds. And so if you care about GPU programming, you care about AI models, you care about inference, you care about Kubernetes and cloud scale stuff. Please check us out. We expect to grow a lot more later this year. The, uh, uh, other things that we have a ton of open source code. And so if you've heard about Mojo, but you looked at it a year ago, guess what? Everything's completely different now. And so if you are interested in a lot of these things, if you're interested in learning about GPUs, we have a ton of content that will teach you about GPU programming, GPU puzzles and things like this. People are now picking up Mojo and putting in a lot of like elite GPU, uh, came out today and there's a whole bunch of other people that are taking the stuff and putting it out there. And I think it's just such an exciting time because I think that lots more people should be using it. I think it's just such an exciting time because I think that lots more people should be using it. I think it's just such an exciting time because I think that lots more people should be programming GPUs. I think this is a huge opportunity for the industry. And of course, if you're enterprise and you're having trouble scaling your AI, you know, let us know. We can, we can help. Awesome. Thank you so much for coming on here. Inspiration as always. Yeah. Well, thank you for having me.

When I wrote about the Rise of the AI Engineer almost 2 years ago, I noted that it would feel “inferior” to the ML Engineer and AI Researcher for a long time. Now we see Andrew Ng writing up the “GenAI Application Engineer”, Hamel coining “AI Scientist”, and even Greg Brockman (upcoming guest!) explaining how engineers can have the same impact as researchers at the World’s Fair.

There’s no playbook for “setting up a new sub-industry” — it’s a mix of devrel/ community building, economics/industrial organization, and anthropology+set theory. We started by building out one of the leading newsletters, then podcasts, surveys1, where we cover everything from mental models to trend charts to coding tools to paper syllabi to daily news, anything AI Engineers need, right down to the job description. But perhaps the highest signal, most intentional thing we do to move the industry forward is organize the AIE conferences.

It’s an odd feeling to run a large industry conference: on one hand, we’ve spent months planning every minute of the event and playing 300D calendar tetris, and on the other we’re so busy running the show that we don’t actually see much of it. And afterward we’ve still got postproduction2 and sponsor needs to handle that it’s hard to write an authoritative overview that is still timely. So here’s one person’s best-efforts overview, but you shouldn’t miss the OTHER recaps from others linked below.

Final count: 3,098 (20% C-level, 17% management, 22% principal/staff eng, 17% senior eng) from a nice split of company sizes (1/3 >1k employees, 1/3 <50) AI engineers, including 306 speakers over 30 workshops + 10 tracks each for 2 days (not to forget: our 69 online speakers) and 30(!!!) independent events across the city.

Special thanks to our community, from Laurie Voss, our incredible emcee, to all track hosts — Henry Mao (MCP), Brittany Walker (Tiny Teams), Eugene Yan (RecSys), Neo4J and Stephen Chin (GraphRAG), Ankur Duggal (AI in Fortune 500 1), Alessio Fanelli (Architect 1), PromptQL and Alex Reibman (Agent Reliability), Seenia Hong (AI PM), Ishan Dhanani (AI Infra), Thor Schaeff (Voice), Vibhu Sapra (SWE-Agents), OpenPipe and Kyle Corbitt (Reasoning+RL), Dat Ngo (Retrieval+Search), Braintrust and Olmo Maldonado (Evals), Randall Hunt (AI in Fortune 500 2), Matt Redmond via Bryan Bischof (Architects 2), Allie Howe (Security), Alexis Gauba (AI Design Eng), Jaril Valenciano (Gen. Media), and Era Qian (Autonomy+Robotics, especially bringing in KScale and Tesla Optimus), interviewers Phlo Young, Sarah/Tyler, Jensen wrangler Nader Khalil and everyone who volunteered and helped others where we simply couldn’t (team credits in livestream). It was truly the “Avengers Assemble” of AI Eng.

Launches both on stage and by sponsors during AIEWF week (NON-EXHAUSTIVE!):

• Gemini 2.5 Pro (06-05) launched onstage by Logan Kilpatrick’s keynote (with roadmap for Gemini Code, Deep Research API, and Computer Use API)

• OpenAI Agents TypeScript SDK launched by Dominik Kundel for his workshop

• OpenAI Agent Robustness Team announced by Fouad Matin, Security keynote

• Anthropic MCP Request for Startups offered by Theodora Chu, MCP PM

• Anthropic Claude Code Plan Mode with Boris Cherny, Claude Code creator

• Dagger’s container-use launched onstage by Solomon Hykes’ keynote

• Morph Liquid Metal and Magi-1 model launched onstage by Jesse Han and hiring Christian Szegedy as “one of the highest aura announcements ever”

• Verified Superintelligence by Morph’s Christian Szegedy, announced as Trinity

• AI Diplomacy launched onstage by Alex Duffy — best speaker for Tiny Teams

• ElevenLabs launched their v3 alpha with Thor Schaeff — track host for Voice

• LlamaIndex’s LlamaCloud Excel Support

• ARC-AGI 3 previewed onstage by Greg Kamradt, President ARC Foundation

• Huxe AI launched by former guest Raiza Martin, AI Product Mgmt keynote

• Exa Research launched from former guest Will Bryk — best speaker in Search

• Traversal AI announced by Anish Agarwal and team on Reliable Agents track

• Modal’s LLM Almanac launched by upcoming guest Charles Frye

• HackAPrompt AIE launched by former guest Sander Schulhoff

• Simon Willison’s PelicanBench ELO rankings on his blog

• let @swyx know if we missed some big launch

Apart from all the posts in the AI Engineering community and tags of @aidotengineer, these are the bigger reflections/recap posts we know of:

• swyx: AIEWF Day 1 on TBPN

• swyx: AIEWF Day 2 on ThursdAI

• NLW: The Biggest Trends from the AI Engineer World's Fair

• Turing Post: Holy Shit moments vs Not There Yet (video)

• Simon Willison: The last six months in LLMs, illustrated by pelicans on bicycles

• Ian Cairns: Bridging the Build-Operate Divide: AI Product Strategy & Operational Insights from AI Engineer World's Fair 2025

• Lance Martin: The state of AI agents

• Charlie Guo: The State of AI Engineering (2025)

• Danny Thompson: How Tiny Teams Built $10M Startups in Months

• Mickey Bazeley: AIE x MongoDB reflections on Multi-Agent Systems

• Anita Kirkovska (guest author): What’s shaping AI products, agents, and infrastructure in 2025

• Patrick Ellis: The startup playbook is being rewritten in real-time

• Derek Sheerin: Reflections on the AI Engineering World Fair 2025

• Lena Hall: FOMO track from AIEWF

• Short takes: Alex Volkov, Yogi Miraje, Danny Thompson, Eugene Yan, Balazs Nemethi

As AIEWF is basically “18 conferences in one”, and many speakers would be keynoters in any other conference, there’s too much to cover by any one person, but we do have plenary stages for stuff that doesn’t quite fit into any one theme.

Because the Day 1 and Day 2 livestreams are already up, we’ll focus on them for now and break out selected takeaways later, illustrated by our friends from Thoth AI, returning to AIEWF for the second time!

In a tightly polished set of demos, Microsoft sees the next leg of AIE in 3 themes:

• from pair → peer programming: GitHub Copilot can be @assigned and GraphRAG on your codebase and even autonomously do MLE with Project Amelie

• from software → agent factory: every company should have a platform (aka “AI Foundry”) that can help churn out agents with the full set of LLM OS tools: model routing, security, reliability, agentic RAG, MCP tools, observability… and even voice posts, imagegen, and posting directly to LinkedIn!

• from cloud → local AI: running open models on device with Foundry Local

swyx’s note: we are very sorry that the 3rd demo ran overtime just as the demos were heating up in the youtube chat; we’ll try to make it up on a full pod.

The top lesson from the last 2.5 years since founding Conviction:

“I have never seen the uptake from users that has been possible in the last couple years… AI engineering is quite a bit harder than people had hoped, but the value creation is massive. We see companies going from 0 to 100m in run rate faster than we’ve ever seen in any technology revolution before.”

How to build Cursor for X:

1. Reasoning is the new vector for scaling

2. Agents, agents, agents: less of chatbots, more like a colleague

3. Multimodality isn’t niche: both Video and Voice will sweep the economy

4. Sam Altman: “Last year’s model is a commodity”

5. Code is the first AI Killer App because it’s structured text with deterministic validation, on the path to AGI, and most imptly engineers can build for themselves.

She abstracts this to a recipe for building Thick AI Wrappers (we approve) and that startups that do this in every vertical and profession can leapfrog incumbents — if they have good execution and don’t rest on a launch. See their Requests for Startups.

Best recapped on his website and Hacker News:

The big trends:

• 30 significant model releases in last 6 months: Nova is cheap, DeepSeek was great, Mistral Small was a great competitor to Llama 3.3, Claude 3.7 → GPT 4.5 → o1-pro → Gemini 2.5 Pro → GPT 4.1 → Claude 4. 4o imagegen was very special

• Bugs: 4o sycophancy, Grok’s issues, Claude 4’s snitching was a big deal

• MCP = Tools+Reasoning: “I think this is the most powerful technique in AI Engineering right now”

• Security: The Lethal Trifecta: when models have 1) access to private data, 2) exposure to malicious instructions, 3) exfiltration vectors

This was our first year handing out Best Speaker awards to spark the competitive spirit in our speakers, and Simon handily won the overall AIE Best Speaker trophy with his mix of humor and technically relevant insight.

Antje teased the coming Alexa+Claude integration, discussed how Amazon Q Developer (also presented by Vikash Agrawal from the team) was built in 2 weeks, with Strands Agents (name picked by AI!) that is basically a form of Agent-specific AWS SDK that has a surprisingly complete set of agent capabilities supported.

Software Engineering != Coding. Returning for his 3rd time at AIE, Kevin revealed the “secret sauce” of Windsurf is progressively abstracting the inner and then outer loop of software engineering with AI, and then modeling the “meta-learning” that engineers do, right up to using their data flywheel (ship → feedback → improve) to build their SWE-1 model for both handling tasks end-to-end (only asking humans for review and approval) and for conversational, human-in-the-loop tasks.

We will probably defer a full breakdown of this to Greg’s podcast but this was the most popular clip from our chat (with a special cameo from Jensen Huang):

Logan knew how to please the crowd by starting “Google Day” with the final Gemini 2.5 Pro preview launch onstage, but perhaps the real alpha came at the end discussing the GDM roadmap - our read is that Veo 3 is likely to be merged into mainline Gemini, more Gemma models will be released, new architecture work for infinite context is underway, and that still-not-shipped Gemini Ultra is still on the table.

He is increasingly pushing the thesis that models are becoming agents out of the box which is informing the work that is being done by the API team, which also shared the roadmap including new announcements for Gemini Code, Deep Research API, and Computer Use API.

Jack presented this keynote for the reasoning team at Gemini just before departing for Meta’s new Superintelligence division with Alexandr Wang. When Flash Thinking first emerged it seemed an unusual terminology for what everyone else was calling “inference time compute”. But it seems clearer that their definition of thinking is more than just raw chain of thought quantity (17x higher (!!!) than 2.0 Flash as Artificial Analysis observes), but also on efficiency and reasoning from a small set of axioms. New science will be invented only after we can cleanly recreate old research.

In one of the most talked-about launches of the conference, Solomon Hykes retook to the same stage he launched DockerCon to open source container-use, Dagger’s answer for containerized environments for every agent (something that until now has been constrained to custom orchestration/Firecracker in the cloud, or non-sandboxed git worktrees in local dev. If you run parallel Claude Code instances, this is the way.

Jesse Han, as a former member of the GPT4 team, took a much more philosophical view, arguing that we want to cater to the machine’s fundamental desires and work backward from there. Yes they want sandboxing, but we also want superlative cloud infrastructure to match the needs of the superintelligence. He announced Morph Liquid Metal and Magi-1 model, as well as hiring Christian Szegedy as Chief Scientist as “one of the highest aura employment announcements ever”.

With Micah absent due to all-too-familiar visa issues, George delivered with Artificial Analysis’ trademark polish, an overview of all the big trends. The “money” slide was an unexpected rerendition of Wirth’s Law (aka Gates’ Law) - what distillation/ inference/hardware people giveth, bigmodel/reasoning/agents people taketh.

This is the second official AIE survey done with Barr/Amplify. As mentioned we’ll discuss this when the full survey results are published, but a fun chart was Barr being able to quantify the gap between Text and other modalities. This is probably the last year that Voice AI trails behind Image gen.

OpenRouter is Andrej’s new candidate for top tier eval, and we were delighted to have Alex join AIE to tell the OpenRouter story for the first time. We think LLM Gateways are a very fundamental component of the AI Engineering stack, and OpenRouter is objectively one of the biggest in production LLM gateways right now, and grows in importance as relevant model lab diversity increases (not at all a consensus opinion when OpenRouter started). Alex outlined the future of “Inference-Native Middleware” as a input -and output- transform complement to MCPs.

The last conference Ben organized pre AI Engineer ended with Sean, Ben, and myself, so it seems weirdly fitting that AIEWF 2025 ends with the AI versions of us. Sean now works on Alignment Research at OpenAI, advocating for why things like the OpenAI Model Spec are underrated and in fact needed, directly addressing the 4o sycophancy issue, and arguing that the best coder is actually just going to be the best communicator (and well structured communication, like having a spec, is the best approach both to alignment and for creating anything with AI).

There’s more recaps to do, particularly with the bigger tracks of the conference and the major launches, but we’ll just get this out now just to ship in a timely fashion.

To give the speakers their due credit, please skim the timelines of:

Day 1:

and Day 2:

for the full story!

full survey shipping soon! thanks for taking part - You can see preliminary results in Barr’s presentation at the end of Day 2. We also had a brief attempt at a foundation and there may be another crack at that someday.

Other than the 8 livestreams + the online track, we will edit and post up all talk videos in weeks, not months

As “leaked”, OpenAI cut o3 pricing by 80% today (from $10/$40 per mtok to $2/$8 - matching GPT 4.1 pricing!!) to set the stage of the launch of o3-pro ($20/$80, supporting an unverified community theory that the -pro variants are 10x base model calls with majority voting as referenced in their papers and in our Chai episode). o3-pro reports a 64% win rate vs o3 on human testers and does marginally better on 4/4 reliability benchmarks, but as sama noticed, the actual experience expands when you test it DIFFERENTLY…

Thanks to Hacker News and theo for covering us.

I’ve had early access to o3 pro for the past week. below are my (early) thoughts:

We’re in the era of task-specific models. On one hand, we have “normal” models like 3.5 Sonnet and 4o—the ones we talk to like friends, who help us with our writing, and answer our day-to-day queries. On the other, we have gigantic, slow, expensive, IQ-maxxing reasoning models that we go to for deep analysis (they’re great at criticism), one-shotting complex problems, and pushing the edge of pure intelligence.

If you follow me on Twitter, you know I've had a journey with the o-reasoning models. My first impression of o1/o1-pro was quite negative. But as I gritted my teeth through the first weeks, propelled by other people's raving reviews, I realized that I was, in fact, using it wrong. I wrote up all my thoughts, got ratio’ed by @sama, and quote-tweeted by @gdb.

The key, I discovered, was to not chat with it. Instead, treat it like a report generator1. Give it context, give it a goal, and let it rip. And that's exactly how I use o3 today.

But therein lines the problem with evaluating o3 pro.

It’s smarter. much smarter.

But in order to see that, you need to give it a lot more context. and I’m running out of context.

There was no simple test or question i could ask it that blew me away.

But then I took a different approach. My co-founder Alexis and I took the the time to assemble a history of all of our past planning meetings at Raindrop, all of our goals, even record voice memos: and then asked o3-pro to come up with a plan.

We were blown away; it spit out the exact kind of concrete plan and analysis I’ve always wanted an LLM to create — complete with target metrics, timelines, what to prioritize, and strict instructions on what to absolutely cut.

The plan o3 gave us was plausible, reasonable; but the plan o3 Pro gave us was specific and rooted enough that it actually changed how we are thinking about our future.

This is hard to capture in an eval.

Trying out o3 Pro made me realize that models today are so good in isolation, we’re running out of simple tests. The real challenge is integrating them into society. It's almost like a really high IQ 12-year-old going to college. They might be smart, but they’re not a useful employee if they can’t integrate.

Today, this integration primarily comes down to tool calls: how well the model collaborates with humans, external data, and other AIs. It’s a great thinker, but it’s gotta grow into being a great do-er. o3 Pro makes real jumps here. It’s noticeably better at discerning what it’s environment is; accurately communicating what tools it has access to, when to ask questions about the outside world (rather than pretending it has the information/access), and choosing the right tool for the job.

o3 pro (left) vs o3 (right):

o3 pro (left) clearly understanding the confines of it’s environment way better.

One thing I noticed from early access: if you don’t give it enough context, it does tend to overthink. It’s insanely good at analyzing, amazing at using tools to do things, not so good at doing things directly itself. I think it would be a fantastic orchestrator. But, for example, there were some ClickHouse SQL questions that o3 did better with. YMMV!

To illustrate:

o3 Pro feels very, very different from Opus and Gemini 2.5 Pro. Where Claude Opus feels big but never truly showed me a clear sign of its “bigness,” o3 Pro’s takes are just... better. It feels like a completely different playing field.

OpenAI’s really driving down this vertical RL path (Deep Research, Codex)—not just teaching models how to use tools, but how to reason about when to use them.

The best ways to prompt reasoning models haven’t changed. My guide on how to prompt o1 still stands. Context is everything, its like feeding cookies to the cookie monster. It’s a way of bootstrapping LLM memory but actually targeted so it works well. And the system prompt really matters. Models have actually become really malleable so LLM “harnesses” that teach a model about it’s environment and its goals have outsize impact. It’s this “harness” - a combination of model, tools, memory, and other methods - that makes AI products actually good (what makes things like Cursor just work most of the time).

Other assorted tasks:

• The system prompt wildly shaped model behavior (in a good way!) It felt way more pronounced than even o3.

• Leaps and bounds different than anthropic and gemini. Where Claude Opus feels big (but has really never shown me a true sign of its “bigness”), these takes are just.. better. Feels like a completely different playing field.

• OAI’s really driving down this vertical RL path (Deep Research, Codex). E.g. teaching models not just how to use tools, but to reason about when to use them.

Many are saying this - “report generation” is basically what we do at AINews and also with Deep Research and Brightwave.

Thanks for coming to AIE - full Day 1 and Day 2 streams are available, with 6 other livestreamed tracks. You can binge this weekend! Longer recap when we have slept.

Emmanuel Amiesen is lead author of “Circuit Tracing: Revealing Computational Graphs in Language Models”, which is part of a duo of MechInterp papers that Anthropic published in March (alongside On the Biology of LLMs).

We recorded the initial conversation a month ago, but then held off publishing until the open source tooling for the graph generation discussed in this work was released last week in collaboration with Neuronpedia:

This is a 2 part episode - an intro covering the open source release, then a deeper dive into the paper — with guest host Vibhu Sapra and Mochi the MechInterp Pomsky. Thanks to Vibhu for making this episode happen!

While the original blogpost contained some fantastic guided visualizations (which we discuss at the end of this pod!), with the notebook and Neuronpedia visualization released this week, you can now explore on your own with Neuronpedia, as we show you in the video version of this pod.

like and subscribe!

• 00:00 Intro & Guest Introductions

• 01:00 Anthropic's Circuit Tracing Release

• 06:11 Exploring Circuit Tracing Tools & Demos

• 13:01 Model Behaviors and User Experiments

• 17:02 Behind the Research: Team and Community

• 24:19 Main Episode Start: Mech Interp Backgrounds

• 25:56 Getting Into Mech Interp Research

• 31:52 History and Foundations of Mech Interp

• 37:05 Core Concepts: Superposition & Features

• 39:54 Applications & Interventions in Models

• 45:59 Challenges & Open Questions in Interpretability

• 57:15 Understanding Model Mechanisms: Circuits & Reasoning

• 01:04:24 Model Planning, Reasoning, and Attribution Graphs

• 01:30:52 Faithfulness, Deception, and Parallel Circuits

• 01:40:16 Publishing Risks, Open Research, and Visualization

• 01:49:33 Barriers, Vision, and Call to Action

swyx [00:00:03]: All right, we are back in the studio with a couple of special guests. One, Vibhu, our guest co-host for a couple of times now, as well as Mochi the Distilled Husky is in the studio with us to ask some very pressing questions. As well as Emmanuel, I didn't get your last name, Amason? Yep. Is that Dutch? It's actually German. German? Yeah. You are the lead author of a fair number of the recent Macintyre work from Anthropic that I've been basically calling Transformer Circuits, because that's the name of the publication.

Emmanuel [00:00:35]: Yeah. Well, to be clear, Transformer Circuits is the whole publication. I'm the author on one of the recent papers, Circuit Tracing. Yes.

swyx [00:00:42]: And people are very excited about that. The other name for it is like Tracing the Thoughts of LLMs. There's like three different names for this work. But it's all Macintyre.

Emmanuel [00:00:49]: It's all Macintyre. There's two papers. One is Circuit Tracing. It's the methods. One is like the biology, which is kind of what we found in the model. And then Tracing the Thoughts is confusingly just the name of the blog post. Yeah.

swyx [00:01:01]: It's for different audiences. Yes. Yes. And I think when you produce the like two minute polished video that you guys did,

Emmanuel [00:01:07]: that's meant for like a very wide audience, you know? Yeah, that's right. There's sort of like very many levels of granularity at which you can go. And I think for Macintyre in particular, because it's kind of complicated going, you know, from like top to bottom, most like high level to sort of the dernali details works pretty well. Yeah.

swyx [00:01:24]: Cool. We can get started. Basically, we have two paths that you can choose, like either your personal journey into Macintyre or the, the brief history of Macintyre, just generally, and maybe that might coincide a little bit.

Emmanuel [00:01:36]: I think my, okay, I could just give you my personal journey very quickly, because then we can just do the second path. My personal journey is that I was working at Anthropic for a while. I'd been like many people just following Macintyre as sort of like an interesting field with fascinating, often beautiful papers. And I was at the time working on fine tuning. So like actually fine tuning production models for Anthropic. And eventually I got both like sort of like, like my fascination reached a sufficient level that I decided I wanted to work on it. And also I got more excited about just as our models got better and better understanding how they worked. So that's the simple journey. I've got like a background in ML, kind of like did a lot of applied ML stuff before. And now I'm doing more research stuff. Yeah.

swyx [00:02:20]: You have a book with O'Reilly. You're head of AI at Insight Data Science. Anything else to plug?

Emmanuel [00:02:25]: Yeah. I actually, I want to like plug the paper and unplug the book. Okay. I think the book, is good. I think the advice stands the test of time, but it's very much like, Hey, you're building like AI products, which do you focus on? It's like very different, I guess is all I'll say from, from the stuff that we're talking to talk about today. Today is like research. Some of, some of the sort of like deepest, weirdest things about how like models work. And this book is you want to ship a random forest to do fraud classification. Like here are the top five mistakes to avoid. Yeah.

swyx [00:02:55]: The good old days of ML. I know it was simple back then. You also transition into research. And I think you also did then, like I feel like there's this monolith of like people assume you need a PhD for research. Maybe can you give like that perspective of like, how do people get into research? How do you get into research? Maybe that gives audience insight into Vibo as well. Your background.

Vibhu [00:03:16]: Yeah. My background was in like economics, data science. I thought LLMs were pretty interesting. I started out with some basic ML stuff and then I saw LLMs were starting to be a thing. So I just went out there and did it. And same thing with AI engineering, right? You just kind of build stuff. You work on interesting, interesting things and like now it's more accessible than ever. Like back when I got into the field five, six years ago, like pre-training was still pretty new. GPT-3 hadn't really launched. So it was still very early days and it was a lot less competitive, but yeah, without any specific background, no PhD, there just weren't as many people working on it, but you made the transition a little bit more recently, right? So what's your experience been like? Yeah,

Emmanuel [00:03:55]: I think, I think it has maybe never been easier in some ways because, um, a lot of the field is like pretty empirical right now. So I think the better lesson is like this lesson that, you know, you can just sort of like a lot of times scale up compute and data and get better results than like thinking than if you sort of like thought extremely hard about a really good, like prior inspired by the human brain to, to train your model better. And so in terms of definitely like research for pre-training and fine tuning, I think it's just sort of like a lot of the bottlenecks are extremely good engineering and systems engineering. And a lot even of the research execution is just about sort of like engineering and scaling up and things like that. I think for Interp in particular, there's like another thing that makes it easier to transition to it too, which is maybe two things. One, you can just do it without huge access to compute. Like there are open source models. You can look at them. A lot of Interp papers, you know, coming out of programs like maths are on models that are open source that you can serve like dissect without having a cluster of like, you know, a hundred GPUs. You can just even, sometimes you can load them like on your CPU, on your MacBook. And it's also a relatively new field. And so, you know, there's, as I'm sure we'll talk about, there's like some conceptual burdens and concepts that you just want to like understand before you contribute, but it's not, you know, physics. It's relatively recent. And so the number of abstractions that you have to like ramp up on is just not that high compared to other fields, which I think makes that transition somewhat easier for Interp. If you understand, we'll talk about all these, I'm sure, but like what, what the features are and what dictionary learning is. You're like a long part of the way there.

swyx [00:05:35]: I think it's also interesting just on a careerist point of view, research seems a lot more valuable than engineering. So I wonder, and you don't have to answer this if it's like a tricky thing, but like how hard is it for a, for a research engineer in Anthropic to jump the wall into research? People seem to move around a lot and I'm like there, that cannot be so easy. Like, like in no other industry that I know of people, you can do that. Do you know what I mean? Yeah.

Emmanuel [00:06:05]: I think I'd actually like I'd push back on the sort of like research being more valuable than engineering a little bit, because I think a lot of times, like having the research idea is not the hardest part. Don't get me wrong. There's some ideas that are like brilliant and hard to find, but what's, what's hard, certainly on fine tuning and to a certain extent on Interp is executing on your research idea in terms of, like making an experiment successfully, like having your experiment run, interpreting it correctly. What that means though, is that like they're not separate skill sets. So like if you have a cool idea, there's kind of not many people in the world, I think where they can just like have a cool idea and then they have a, you know, like a little minion they'll deputize being like, here's my idea, you know, go off for three months and like run this whole day, like build this model and train it for, you know, hundreds of hours and report back on what happened. A lot of the time, like the people that are the most productive, they have an idea, but they're also extremely quick. Uh, checking their idea of finding sort of like the shortest path to, to turn it already. And a lot of like that shortest path is engineering skills essentially. It's just like getting stuff done. And so I think that's why you see sort of like people move around as like proportionate to your interest. If you're just able to quickly execute on the ideas you have and get results, then that's really the 90% of the value. And so you see a lot of transferable skills, actually, I think from, from people like I've certainly seen an anthropic that are just like really good at that, you know, they can apply it in one team and then move to a completely different domain and apply that inner loop just as well. Yeah.

swyx [00:07:35]: Very cracked is the kids say, uh, shall we move to the history of McInturk? Yeah. All I know is that everyone starts at Chris Ola's blog. Is that right? And yeah,

Emmanuel [00:07:45]: I think that's the correct answer. Chris Ola's blog. And then, you know, uh, distill.pub, uh, is the sort of natural next step. And then I would say, you know, now there's for anthropic, there's transformer circuits, which you talked about. but there's also just a lot of McInturk research out there from, you know, I think like the, yeah, like maths is a group with that, like regularly has a lot of research, but there's just like many different labs that, that put research out there. And I think that's also just like hammer home the point that's because all you need is like a model and then a willingness to kind of investigate it to be able to contribute to it. So, so now it's like, there's been a bit of a Cameron explosion of McInturk, which is cool. I guess the history of it is just, you know, computational like models that are not decision trees models that are either CNNs or let's say transformers have just this w really like strange property that they don't give you interpretable intermediate States by default. You know, again, to go back to if you were training like a decision tree on like fraud data for an old school, like bank or something, then you can just look at your decision tree and be like, oh, it's learned that like if you make a, I don't know if this transaction is more than $10,000, and it's for like perfume, then maybe it's fraud or something. You can look at it and say like, cool, like that makes sense. I'm willing to ship that model. But for, for things like, like CNNs and like transformers, we, we don't have that right. What we have at the end of training is just a massive amount of weights that are connected somehow or activations that are connected by some weights. And who knows what these weights mean or what the intermediate activations mean. And so the quest is to understand that initially it was done. A lot of it was on a vision models where you sort of have, the emergence of a lot of these ideas, like what are features, what are circuits. And then more recently it's been mostly or not most, yeah, mostly applied to NLP models, but also, you know, still there's work in vision and there's work in like a bio and other domains. Yeah.

swyx [00:09:44]: I'm on Chris Ola's blog and he has like the feature visualization stuff. I think for me, the clearest was like the vision work where you could have like this layer detects edges, this layer detects textures, whatever. That seemed very clear to me, but the transition to language models seem, like a big leap.

Emmanuel [00:10:00]: I think one, one of the bigger changes from vision to, to like language models has to do with the superposition hypothesis, which maybe is like, that's the first in like toy models post, right? Exactly. And this is sort of like, it turns out that if you look at just the neurons of a lot of vision models, you can see neurons that are curve detectors or that are edge detectors or that are high, low frequency detectors. And so you can sort of like make sense of the neurons. Mostly. But if you look at neurons in language models, most of them don't make sense. It's kind of like unclear why, or it wasn't clear why that would be. And one main like hypothesis here is the superposition hypothesis. So what does that mean? That means that like language models pack a lot more in less space than vision models. So maybe like a, a kind of like really hand wavy analogy, right? Is like, well, if you want curve detectors, like you don't need that many curve detectors, you know, if each, each curve detector is going to detect like a quarter or a 12th of a circle, like, okay, well you have all your curve detectors, but think about all of the concepts that like Claude or even GPT-2 needs to know, like just in terms of, it needs to know about like all of the different colors, all the different hours of every day, all of the different cities in the world, all of the different streets on every city. If you just enumerate all of the facts that like a model knows, you're going to get like a very, very long list. And that list is going to be way bigger than like the number, of neurons or even the like size of the residual stream, which is where like the models process information. And so there's this sense in which like, oh, there's more information than there's like dimensions to represent it. And that is much more true for language models than for vision models. And so, because of that, when you look at a part of it, it just seems like it's like, there's, it's got all this stuff crammed into it. Whereas if you look at the vision models, oftentimes you could just like, yeah, cool. This is a curve detector.

swyx [00:11:53]: Yeah. Yeah. VB, you have like some fun ways of explaining the toy models or super, superposition concept. Yeah.

Vibhu [00:11:59]: I mean, basically like, you know, if you have two neurons and they can represent five features, like a lot of the early Mechinterp work says that, you know, there are more features than we have neurons. Right. So I guess my kind of question on this is for those interested in getting into the field, what are like the key terms that they should know? What are like the few pieces that they should follow? Right. Like from the anthropic side, we had a toy transformer model. We had sparse, we first had auto encoders. That was the second paper.

swyx [00:12:25]: Um,

Vibhu [00:12:25]: right. Yeah. Monosyntheticity. Yeah. What is sparsity in auto encoders? What are trans coders? Like what is linear probing? What are these kind of like key points that we had in Mechinterp? And just kind of, how would people get a quick, you know, zero to like 80% of the field? Okay.

Emmanuel [00:12:42]: So zero to 80%. And I realized I really like set myself up for, for failure. So I was like, yeah, it's easy. There's not that much to know. So, okay. So then, then we should be able to cover it all. Um, so superposition is the first thing you should know, right? This idea that like there's a bunch of stuff crammed in a few dimensions, as you said, maybe you have like two neurons and you want to represent five things. So if that's true, and if you want to understand how the model represents, you know, I don't know the concept of red, let's say then you need some way to like find out essentially in which direction the model stores it. So after the, the server, like superposition hypothesis, you can think of like, ah, we also think that like basically the model represents these like individual concepts, we're going to call them features as like directions. So if you have two neurons, you can think of it as like, it's like the two D plane. And it's like, ah, you can have like five directions and maybe you would like arrange them like the spokes of a wheel. So they're sort of like maximally separate. It could mean that like you have one concept that this way and one concept that's like not fully perpendicular to it, but like pretty, pretty like far from it. And then that would like allow the model to represent more concepts and it has dimensions. And so if that's true, then what you want is you want like a model that can extract these independent concepts. And ideally you want to do this like automatically. Like, can we just, you know, have a model that tells us like, oh, like this direction is red. If you go that way, actually it's like, I don't know, chicken. And if you go that way, it's like the declaration of independence, you know? Um, and so that's what sparse auto encoders are.

swyx [00:14:09]: It's almost like the, the self supervised learning insight version, like in, in pre-training itself, supervised learning and here and now it's so self supervised interpretability. Yeah,

Emmanuel [00:14:18]: exactly. Exactly. It's like an unsupervised method. Yeah. And so unsupervised methods often still have like labels in the end or so.

swyx [00:14:25]: So sometimes I feel like the term labels by masking. Yeah.

Emmanuel [00:14:29]: Like for, for pre-training, right? It's like the next token. So in that sense, you have a supervision signal and here the supervision signal is simply, you take the like neurons and then you learn a model. That's going to like expand them into like the actual number of concepts that you think there are in the model. So you have two neurons, you think there's five concepts. So you expand it to like, I think of dimension five. And then you contract, it back to what it was. That's like the model you're training. And then you're training it to incentivize it to be sparse. So that there's only like a few features active at a time. And once you do that, if it works, you have this sort of like nice dictionary, which you can think as like a way to decode deactivate the neurons where you're saying like, ah, cool. I don't know what this, what this direction means, but I've like used my model into telling me that the model is writing in the red direction. And so that's, that's sort of like, I think maybe the biggest thing to understand is, is this, this combination of things of like, ah, we have too few dimensions. We pack a lot into it. So we're going to learn an unsupervised way to like unpack it and then analyze what each of those dimensions that we've unpacked are.

Vibhu [00:15:31]: Any follow-ups? Yeah. I mean, the follow-ups of this are also kind of like some of the work that you did is in clamping, right? What is the applicable side of MechInterp, right? So we saw that you guys have like great visualizations. Golden Gate Cloud was a cool example. I was going to say that. Yeah. It's my favorite. What can we do once we find these features? Finding features is cool. But what can we do about it? Yeah.

Emmanuel [00:15:53]: I think there's kind of like two big aspects of this. Like one is, yeah. Okay. So we go from a state where, as I said, the model is like a mess of weights. We have no idea what's going on to, okay. We found features. We found a feature for red, a feature for Golden Gate Cloud or for the Golden Gate Bridge, I should say. Like, what do we do with them? And well, if these are true features, that means that like they, in some sense are, are important for the model or it wouldn't be like representing it. Like if the model is like bothering to, to like write, you know, in the Golden Gate Bridge direction, it's usually because it's going to like talk about the Golden Gate Bridge. And so that means that like, if that's true, then you can like set that feature to zero or artificially set to a hundred and you'll change the model behavior. That's what we did when we did Golden Gate Cloud in which we found a feature that represents the direction for the Golden Gate Bridge. And then we just like set it to always be on. And then you could talk to Claude and be like, Hey, like, Oh, what's on your mind? You know, like, what are you thinking about today? Be like the Golden Gate Bridge. You'd be like, Hey Claude, like what's two plus two, it'd be like four Golden Gate Bridges, uh, et cetera. Right. And it was always thinking about this,

swyx [00:16:55]: like write a poem and it just starts talking about how it's like red, like the Golden Gate plot. That's right. Golden Gate Bridge. Yeah,

Emmanuel [00:17:00]: that's right. Amazing. I think what made it even better is like, we realized later on that it wasn't really like a Golden Gate Bridge feature. It was like being in awe at the beauty of the majestic Golden Gate Bridge. Right. So I'll tell you, I would like really ham it up. You'd be like, Oh, I'm just thinking about the beautiful international orange color of the Golden Gate Bridge. That was just like an example that I think was like really striking, but of, of sort of like, Oh, if you found like a space where that represents some computation or some representation of the model, that means that you can like artificially suppress or promote it. And that means that like you're starting to understand at a very high level, a very gross level, like how some of them all works. Right. We've gone from like, I don't know anything about it to like, Oh, I know that this like combination of neurons, neurons is this, and I'm going to prove it to you. The next step, which is what this, this works on is like, that's kind of like thinking of if maybe you take the analogy of like, um, I don't know, like, like let's take the energy of like an MRI or something like a brain scan. It tells you like, Oh, like this as Claude was answering at some point, it thought about this thing, but it's a sort of like vague, like maybe, basically maybe it's like a, like a bag of words, kind of like a bag of features. You just have like, here are all the random things that thought about, but what you might want to know is like, what's this thing like sometimes to get to the golden gate bridge, it had to realize that you were talking about San Francisco and about like the best way to go to Sonoma or something. And so that's how it got to golden gate bridge. So there's like an algorithm that leads to it at some point, thinking about the golden gate bridge. And basically like, there's like a way to connect features to say like, Oh, from this input went to these few features and then these few features and then these few features and that one influenced this one. And then you got to the output. And so that's the second part. And the part we worked on is like you have the features now connect them in what we call a, or what's called, circuits, which is sort of like explaining the, the like algorithm. Yeah.

swyx [00:18:46]: Before we move directly onto your work, I just want to give a shout out to Neil Nanda. He did neuron Pedia and released a bunch of essays for,

Emmanuel [00:18:54]: I think the llama models and the Gemma models and the Gemma models. Yeah.

swyx [00:18:57]: So I actually made golden gate Gemma just up the weights for proper nouns and names of places of people and references to the term golden likely relating to awards, honors, or special names. And that together made golden gate. That's amazing. So you can make golden gate Gemma. And like, I think that's a, that's a fun way to experiment with this. Uh, but yeah, we can move on to, I'm curious.

Vibhu [00:19:19]: I'm curious. What's the background behind why you ship golden gate claw? Like you had so many features, just any fun story behind why that's the one that made it.

Emmanuel [00:19:28]: You know, it's funny if you look at the paper, there's just a bunch of like, yeah, like really interesting features, right? There's like one of my favorite ones was the psychophantic praise, which I guess is very topical right now.

swyx [00:19:40]: Topical.

Emmanuel [00:19:40]: Um, but you know, it's like you could dial that up and like Claude would just really praise you. You'd be like, Oh, you know, like I wrote this poem, like roses are red, violets are blue, whatever. And it'd be like, that's the best poem I've ever seen. Um, and so we could have shipped that. That could have been funny. Uh, golden gate. Claude, it was like a pure, as far as I remember, at least like a pure, just like weird random thing where like somebody found it initially with an internal demo of it. Everybody thought it was hilarious. And then that's sort of how it came on. There was, there was no, nobody had a list of top 10 features we should consider shipping. And we picked that one. There was just kind of like a very organic moment. No,

swyx [00:20:18]: like the, the marketing team really leaned into it. Like they mailed out pieces of the golden gate for people in Europe, I think, or ICML. Yeah, it was, it was fantastic marketing. Yeah. The question obviously is like if open AI had invested more interpretability, would they have caught, uh, the GPT for all update? Uh, but we don't know that for sure. Cause they have interp teams. They just,

Emmanuel [00:20:39]: yeah. I think also like for that one, I don't know that you need interp. Like it was pretty clear cut to the model. I was like, oh, that model is really gassing me up.

swyx [00:20:46]: And then the other thing is, um, can you just like up, write good code, don't write bad code and make Sonda 3.5. And like, it feels too, too easy to free. Is that steering that powerful?

Vibhu [00:20:59]: You can just like up and down features with no trade-offs. There was quite a, there was like a phase where people were basically saying, you know, 3.5 and 3.7 are just now because they came out right.

swyx [00:21:09]: And for the record, like that's been debunked.

Vibhu [00:21:11]: But it has been debunked, but you know, it had people convinced that what people did is they basically just steered up and steered down features. And now we have a better model. And this kind of goes back to that original question of, right? Like, why do we do this? What can we do? Some people are like, I want tracing from a sense of, you know, legality. Like what did the model think when it came to this output? Some people want to turn hallucination down. Some people want to turn coding up. So like, what are some, like whether it's internal, what are you exploring that? Like what are the applications of this, whether it's open, ended of what people can do about this or just like, yeah. Why, why do McInturpp, you know? Yeah.

Emmanuel [00:21:46]: There's like a few things here. So, so like, first of all, obviously this is, I would say on the scale of the most short term to the most long term, like pretty long term research. So in terms of like applications compared to, you know, like the research work we do on like fine tuning or whatever interp is much more, you know, sort of like a high risk, high reward kind of approach. With that being said, like, I think there's just a fundamental sense in which, which Michael Nielsen had had a post recently about how like knowledge is dual use or something, but just like, just like knowing how the model works at all feels useful. And you know, it's hard to argue that if we know how the model works and understand all of the components that won't help us like make models that hallucinate less, for example, or they're like less biased, that seems, you know, if, if like at the limit, yeah, that totally seems like something you would do using basically like your understanding of the model to improve it. I think for now, as we can talk about a little bit with like circuits, there's like, we're still pretty early on in the game, right? And so right now the main way that we're using interp really is like to investigate specific behaviors and understand them and gain a sense for what's causing them. So like one example that we, we can talk about later, we can talk about now, but is this like in the paper, we investigate jailbreaks and we try to see like, why does a jailbreak work? And then we realize as we're looking at this jailbreak, that part of the reason why Claude is telling you how to make a bomb in this case, is that it's like already started to tell you how to make a bomb. And it would really love to stop telling you how to make a bomb, but it has to first finish its sentence. Like it really wants to make correct grammatical sentences. And so it turns out that like seeing that circuit, we were like, ah, then does that mean if I prevent it from finishing a sentence, the jailbreak works even better? And sure enough, it does. And so I think like the level of sort of practical application right now is of that shape. So like understanding either like quirks of a current model, or like, or like how it does tasks that maybe we don't, we don't even know how it does it. Like, you know, we have like some planning examples where we had no idea it was planning and we're like, oh God, it is. That's sort of like the current state we're at.

Vibhu [00:23:51]: I'm curious internally how this kind of feeds back into like the research, the architecture, the pre-training teams, the post-training, like, is there a good feedback loop there? Like right now there's a lot of external people interested, right? Like we'll train an SAE on one layer of LAMA and probe around. But then people are like, okay, how does this have much impact? People like clamp down, clamping. But yeah, as you said, you know, once you start to understand these models have this early planning and stuff, how does this kind of feed back?

Emmanuel [00:24:16]: I don't know that there's, there's like much to say here other than like, I think we're definitely interested in conversely, like making models for which it's like easier to interpret them. So that's also something that you can imagine sort of like working on, which is like making models where you have to work less hard to try to understand what they're doing. So like the architecture? Okay.

swyx [00:24:35]: Yeah. So I think there was a, there was a less wrong post about this of like, there's a non-zero model, which is like,

Emmanuel [00:24:47]: there's this sort of sense in which like right now we take the model and then the model is a model. And then we post hoc do these replacement layers to try to understand it. But of course, when we do that, we don't like fully capture everything that's happening inside the model. We're capturing like a subset. And so maybe some of it is like, you could train a model that's sort of like easier to interpret naively. And it's possible that like, you don't even have that much of, you know, like a tax in that sense. And so maybe some of it is like, you can just sort of like, either like train your model differently or do like a little post hoc step to like, sort of like untangle some of the mess that you've made when you trained your model. Right. Make it easier to interpret. Yeah.

swyx [00:25:19]: The hope was pruning would do some of that, but I feel like that area of research has just died. What kind of pruning are you thinking of here? Just pruning your network. Ah, yeah. Pruning layers, pruning connections, whatever. Yeah.

Emmanuel [00:25:34]: I feel like maybe this is something where like superposition makes me less hopeful or something.

swyx [00:25:40]: I don't know. Like that, that like seventh bit might hold something. Well,

Emmanuel [00:25:44]: right. And it's like on, on each example, maybe this neuron is like at the bottom of like what matters, but actually it's participating like 5% to like understanding English, like doing integrals and you know, like whatever, like cracking codes or something. And it's like, because that was just like distributed over it, you, you, you kind of like when you naively prune, you might miss that.

swyx [00:26:05]: I don't know. Okay. So, and then this area of research in terms of creating models that are easier to interpret from the, from the start, is there a name for this field of research?

Emmanuel [00:26:14]: I don't think so. And I think this is like very early. Okay. And it's, it's mostly like a dream. Just in case there's a, there's a thing people want to double click on. Yeah, yeah, yeah.

swyx [00:26:20]: I haven't come across it.

Vibhu [00:26:22]: I think the higher level is like Dario recently put out a post about this, right? Why McInterpre is so interpret important. You know, we don't want to fall behind. We want to be able to interpret models and understand what's going on, even though capabilities are getting so good. It kind of ties into this topic, right?

Vibhu [00:26:40]: It's slightly easier to interpret so we don't fall behind so far. Well,

Emmanuel [00:26:43]: yeah. And I think here, like, just to talk about the elephant in the room or something like, like one big concern here is, is like safety, right? And so like, as models get better, they are going to be used more and more places. You know, it's like, you're not going to have your, you know, we're vibe coding right now. Maybe at some point, well, that, that'll just be coding. It's like, Claude's going to write your code for you. And that's it. And Claude's going to review the code that Claude wrote. And then Claude's going to deploy it to production. And at some point, like, as these models get integrated deeper and deeper into more and more workflows, it gets just scarier and scarier to know nothing about them. And so you kind of want your ability to understand the model, to scale with like how good the model is doing, which that itself kind of like tends to scale with like how widely deployed it is. So as we like deploy them everywhere, we want to like understand them better.

swyx [00:27:28]: The version that I liked from the old super alignment team was, weak to strong generalization or weak to strong alignment, which that's what super alignment to me was. And that was my first aha moment of like, oh yeah, some, at some point these things will be smarter than us. And in many ways they already are smarter than us. And we rely on them more and more. We need to figure out how to control them. And this, this is not a, like an Eliezer Yudkowsky, like, ah, thing. It's just more like, we don't know what, how these things work. Like, how can we use them? Yeah.

Emmanuel [00:27:56]: And like, you can think of it as, there's many ways to solve a problem. And some of them, if the model is solving it in like a dumb way or in like memorized one approach to do it, then you shouldn't deploy it to do like a general thing. Like it, like you could look at how it does math and based on your understanding of how it does math, you're like, okay, I feel comfortable using this as a calculator or like, no, it should always use a calculator tool because it's doing math in a stupid way and extend that to any behavior. Right. Where it's just a matter of like, think about it. If, if like you're like in the 1500s and I give you a car or something, and I'm just like, cool, like this thing, when you press on this, like it accelerates when you press on that, like it stops, you know, this steering wheel seems to be doing stuff, but you knew nothing about it. I don't know if it was like a, a super faulty car and it's like, oh yeah, but if you ever get, went above 60 miles an hour, like it explodes or something like you probably would be sort of like, you'd want to understand the nature of the object before like jumping in, in it. And so that's why we like understand how cars work very well because we make them. LLMs are sort of like ML models in general are like this very rare artifact where we like make them, but we don't, we don't know how they work.

swyx [00:28:58]: We evolve them. We create conditions for them to evolve and then they evolve. And we're like, cool. Like, you know, maybe you got a good run. Maybe we didn't. Yeah. Don't really know.

Emmanuel [00:29:07]: Yeah. The extent to which you know how it works is you have your like eval and you're like, oh, well seems to be doing well on this eval. And then you're like, is this because this wasn't a training set or is it like actually generalizing? I don't know.

swyx [00:29:16]: My favorite example was somehow C4, the, the common, the colossal clean corpus did much better than a common crawl, even though it filtered out most of this, like it was very prudish. So it like filters out anything that could be considered obscene, including the word gay, but like somehow it just like when you add it into the data mix, it just does super well. And it's just like this magic incantation of like this, this recipe works. Just trust us. We've tried everything. This one works. So just go with it. Yeah. That's not very satisfying.

Emmanuel [00:29:49]: No, it's not. The side that you're talking about, which is like, okay, like how do you make these? And it's kind of unsatisfying that you just kind of make the soup. And you're like, oh, well, you know what? My grandpa made the soup with these ingredients. I don't know why, but I just make the soup the way my grandpa said. And then like one day somebody added, you know, cilantro. And since then we've been adding cilantro for generations. And you're like, this is kind of crazy.

swyx [00:30:07]: That's exactly how we train models though. Yeah. Yeah.

Emmanuel [00:30:10]: So I think there's, there's like a part where it's like, okay, like let's try to unpack what's happening. You know, like the mechanisms of learning,

Emmanuel [00:30:21]: you know, like understanding what induction heads are, which are attention heads that allow you to look at in your context, the last time that something was mentioned and then repeat it is like something that happens. It seems to happen in every model. And it's like, oh, okay, that makes sense. That's how the model like is able to like repeat text without dedicating too much capacity to it.

Vibhu [00:30:39]: Let's get it on screen. So if you can see. The visuals of the work you guys put out is amazing. Oh yeah.

swyx [00:30:44]: We should talk a little bit about the back behind the scenes of that kind of stuff, but let's, let's finish this off first. Totally.

Emmanuel [00:30:49]: But just really quickly, I think it's just like, if you're interested in mech interp, we talked about superposition and I think we skipped over induction heads and that's like, you know, kind of like a really neat, basically pattern that emerges in many, many transformers where essentially they just learn. Like one of the things that you need to do to like predict text well is that if there's repeated texts at some point, somebody said, Emmanuel Mason, and then you're like on the next line and they say, Emmanuel, very good chance. It's the same last name. And so one of the first things that models learn is just like, okay, I'm just gonna like look at what was said before. And I'm gonna say the same thing. And that's induction heads, which is like a pair of attention heads that just basically look at the last time something was said, look at what happened after, move that over. And that's an example of a mechanism where it's like, cool. Now we understand that pretty well. There's been a lot of followup research on understanding better, like, okay, like in which context do they turn on? Like, you know, there's like different like levels of abstraction. There's like induction heads that like literally copy the word. And there's some that copy like the sentiment and other aspects. But I think it's just like an example of slowly unpacking, you know, or like peeling back the layers of the onion of like, what's going on inside this model. Okay. This is a component. It's doing this.

swyx [00:31:51]: So induction headers was like the first major finding.

Emmanuel [00:31:53]: It was a big finding for NLP models for sure.

swyx [00:31:56]: I often think about the edit models. So Claude has a fast edit mode. I forget what it's called. OpenAI has one as well. And you need very good copying, every area that needs copying. And then you need it to switch out of copy mode when you need to start generating. Right. And that is basically the productionized version of this. Yeah.

Emmanuel [00:32:15]: Yeah. Yeah. And it turns out that, you know, you, you need to switch out of copy mode when you need to start generating. It's also like a model that's like smart enough to know when it needs to get out of copy mode, right? Which is like, it's fascinating.

swyx [00:32:22]: It's faster. It's cheaper. You know, as bullish as I am on canvas, basically every AI product needs to iterate on a central artifact. And like, if it's code, if it's a piece of writing, it doesn't really matter, but you need that copy capability. That's smart enough to know when to turn it off.

Emmanuel [00:32:40]: That's why it's cool that induction heads are at different levels of abstraction. Like sometimes you need to editing some code, you need to copy like the general structure. Yeah. It's like, oh, like the last, like this other function that's similar, it's first takes like, you know, I don't know, like abstract class and then it takes like an int. So I need to like copy the general idea, but it's going to be a different abstract class and a different int or something. Yeah. Cool.

swyx [00:32:59]: Yeah. So tracing?

Emmanuel [00:33:01]: Oh yeah. Should we jump to circuit tracing? Sure.

swyx [00:33:03]: I don't know if there's anything else you want to cover. No,

Emmanuel [00:33:05]: no, no. We have space for it. Maybe. Okay. I'll do like a really quick TLDR of these two recent papers. Okay. Insanely quick.

Emmanuel [00:33:18]: Okay. But we'd like to connect the features to understand like the inputs to every features and the opposite features and basically draw a graph. And this is like, if I'm still sharing my screen, uh, the thing on the right here where like, that's the dream we want, like for a given prompt, what were all of the things, like all of the important things happen in the model. And here it's like, okay, it took in these four tokens. Those activated these features, these features, activate these other features. And then these features like to be the other features. And then all of these like promoted the output. And that's the story. And basically we're, we're like, the work is to sort of use dictionary learning and these replacement models to provide a explanation of like sets of features that explain behavior. So this is super abstract. So I think immediately maybe we can like, just look at one example. I can show you one, which is this one, the reasoning one. Yep. Yeah. Two step reasoning. I think this is already, this is like the introduction example, but it's already like kind of fun. So, so the question is you ask the model something that requires it to take a step of reasoning in its head. So let's say, you know, fact, the capital of the state containing Dallas is. So to answer that, you need one intermediate step, right? You need to say, wait, where's Dallas? Isn't Texas. Okay, cool. Capital of Texas, Austin. And this is like in one token, right? It's going to, after is, it's going to say Austin. And so like in that one forward pass, the model needs to extract, to realize that you're asking it for like the capital of a state to like look up the state for Dallas, which is Texas. And then to say Austin. And sure enough, this is like what we see is we see like in this forward pass, there's a rich sort of like inner set of representations where there's like, it gets capital state in Dallas and then boom, it has an inner representation for Texas. And then that plus capital leads it to like say Austin.

Vibhu [00:34:59]: I guess one of the things here is like, we can see this internal like thinking step, right? But a lot of what people say is like, is this just memorized fact, right? Like I'm sure a lot of the pre-training that this model is trained on is this sentence shows up pretty often. Right? So this shows that no, actually internally throughout, we do see that there is this middle step, right? It's not just memorized. You can prove that it generalized.

Emmanuel [00:35:22]: Yeah. So, so, so that's exactly right. And I think like you, you, you hit the nail on the head, which is like, this is what this example is about. It's like, ah, if this was just memorized, you wouldn't need to have an intermediate step at all. You'd just be like, well, I've seen the sentence, like, I know what comes next. Right. But here there is an intermediate step. And so you could say like, okay, well maybe it just has the step, but it's memorized it anyways. And then the way to like verify that is kind of like what, what we do later in the paper. And for all of our examples is like, okay, we claim that this is like the Texas representation. Let's get another one and replace it. And we just changed like that feature in the middle of the model. And we change it to like California. And if you change it to California, sure enough, it says Sacramento. And so it's like, this is not just the like byproduct, like it's memorized something. And on the side, it's thinking about Texas. It's like, no, no, no, this is like a step in the reasoning. If you change that intermediate step, it changes the answer.

Vibhu [00:36:12]: Very, very cool work. Underappreciated.

Emmanuel [00:36:15]: Yeah. Okay.

Vibhu [00:36:16]: Sure.

swyx [00:36:17]: I have never really doubted. I think there's a lot of people that are always criticizing LMS, stochastic parrots. This pretty much disproves it already. Like we can move on. Yeah.

Emmanuel [00:36:28]: I mean, I, I think, I think there's a lot of examples that I will say we can go through like a few of them. And like show an amount of depth in the intermediate states of the model that makes you think like, oh gosh, like it's doing a lot. I think maybe like the poems, well, definitely the poems, but even for this one, I'm going to like scroll in this very short paper. So like medical diagnoses,

swyx [00:36:50]: I don't even know the word count because there's so many like embedded things in there. Yeah.

Emmanuel [00:36:54]: It's too dangerous. We can't look it up. It overflows. It's so beautiful. Look at this. This is like a medical example that I think shows you again, this is in one forward pass. The model is like given a bunch of symptoms and then it's asked not like, Hey, what does, what is the like disease that this person has? It's asked like if you could run one more test to determine it, what would it be? So it's even hard, right? It means like you need to take all the symptoms. Then you need to like have a few hypotheses about what the disease could be. And then based on your hypothesis, say like, well, the thing that would like be the right test to do is X. And here you can see these three layers, right? Where it's like, again, in one forward pass, it has a bunch of like, oh, it has the most likely diagnosis here, then like an alternate one. And then based on the diagnosis, it like gives you basically a bunch of things that you could ask. And again, we do the same experiments where you can like kill this feature here, like suppress it. And then it asks you a question about the second, the sort of like second option it had. The reason I show it is like, man, that's like a lot of stuff going on. Like for, for one forward pass, right? It's like specifically if you, if you expected it to like, oh, what it's going to do is it's just like seeing similar cases in the training. It's going to kind of like vibe and be like, oh, I guess like there's that word and it's going to say something that's related to like, I don't know, headache, you know what I kind of like really have. It's like, no, no, no. It's like activating many different distributed representations, like combining them and sort of like doing something pretty complicated. And so, yeah, I think, I think it's funny because in my opinion, that's like, yeah, like, oh God, stochastic parrots is not something that I think is, is like appropriate here. And I think there's just like a lot of different things going on and there's like pretty complex behavior. At the same time, I think it's in the eye of the beholder. I think like I've talked to folks that have like read this paper and I've been like, oh yeah, this is just like a bunch of kind of like heuristics that are like mashed together, right? Like the model is just doing like a bunch of kind of like, oh, if high blood pressure than this or that. And so I think there's, there's sort of like an underlying question that's interesting, which is like, okay, now we know a little bit of how it works. This is how it works. Like now you tell me if you think that's like impressive, if you think that like, if you trust it, if you think that's sort of like something that is, that is sufficient to like ask it for medical questions or whatever.

swyx [00:38:59]: I think it's a way to adversarially improve the model quality. Yeah. Because once you can do this, you can reverse engineer what would be a sequence of words that to a human makes no sense or lets you arrive at the complete opposite conclusion, but the model still gets tripped up by. Yeah. And then you can just improve it from there. Exactly.

Emmanuel [00:39:19]: And this gives you a hypothesis about like, you like specifically imagine if like one of those was actually the wrong symptom or something, you'd be like, oh, like the liver condition, like, you know, upweighs this other example. That doesn't make sense. Okay. Let's like fix that in particular. Exactly. You sort of have like a bit of insight into like how the model is getting to its conclusion. And so you can see both like, is it making errors, but also is it using the kind of reasoning that will lead it to errors?

swyx [00:39:45]: There's a thesis. I mean, now it's very prominent with the reasoning models about model death. So like you're doing all this in one pass. Yeah. But maybe you don't need to, because you can do more passes. Sure. And so people want shallow models for speed, but you need model death for this kind of thinking. Yeah. So what's the, is there a Pareto frontier? Is there a direct trade off? Yeah. I mean, would you prefer if you had to make a model and like, you know, shallow versus deep?

Emmanuel [00:40:15]: There's a chain of thought faithfulness example. Before I show it, I'm just going to go back to the top here. So when the model is sampling many tokens, if you want that to be your model, you need to be able to trust every token it samples. So like the problem with, with models being autoregressive is that like, if they like at some point sample a mistake, then they kind of keep going condition on that mistake. Right. And so sometimes like you need backspace tokens or whatever. Yeah. An error correction is like notably hard, right? If you have like a deeper model, maybe you have like fewer COT steps, but like your, your steps are more likely to be like robust or, or correct or something. And so I think that that's one way to look at the trade off. To be clear, I don't have an answer. I don't know if I want a wide or a, or a shallow or a deep model, but I think, I think the trade off is that like,

swyx [00:41:00]: You definitely want shallow for inference speed. Sure,

Emmanuel [00:41:02]: sure, sure, sure. But you're trading that off for, for something else, right? Cause you also want like a one B model for inference speed, but that also comes at a cost, right? It's, it's less smart.

Vibhu [00:41:10]: There's a cool quick paper to plug that we just covered on the paper club. It's a survey paper around when to use reasoning models versus dense models. What's the trade off. I think it's the economy of reasoning economy, reasoning, the reasoning economy. So they just go over a bunch of, you know, ways to measure this benchmarks around when to use each because yeah. Yeah. Like, you know, we don't want to also like consumers are now paying the cost of this. Right. But little, little side note. Yeah.

swyx [00:41:34]: For those on YouTube, we have a secondary channel called Latent Space TV, where we cover that stuff. Nice. That's our paper club. We covered your paper.

Emmanuel [00:41:41]: Cool. Yeah. I think you brought up the like planning thing. Maybe it's worth. Let's do it. Yeah. If you think about, okay. So you're, you're going into the chain of thought faithfulness one. Let's get this one. Let's just do planning. So if you think about like, you know, common questions you have about models, the first one we kind of asked was like, okay, like is it just doing this like vibe based one shot pattern matching based on existing data? Or does it have like kind of rich in representations? It seems to have like these like intermediate representations that make sense as the abstractions that you would reason through. Okay. So that's one thing. And there's a bunch of examples. We talked about the medical diagnoses. There's like the multilingual circuits is another one that I think is cool where it's like, oh, it's sharing representations across languages. Another thing that you'll hear people mention about language models, which is that they're like a next token predictors.

Vibhu [00:42:25]: Also for, for a quick note for people that won't dive into this super long blog post, I know you highlighted like 10 to 12. So for like a quick 15, 30 second, what do you mean by they're sharing thoughts throughout? Just like, what's the really quick high level just for people though? Yeah.

Emmanuel [00:42:39]: They're really quick. High level. High level is that what we find is that here, I'm just like show you a really quick inside the model. If you look at like the inner representations for concepts, you can ask like the same question, which I think in the paper, the original one we asked is like the opposite of hot is cold, but you can, you can do this over a larger dataset and ask the same question in many different languages. And then look at these representations in the middle of the model and ask yourself like, well, when you ask it, the opposite of hot is, and which is the same sentence in French, show off. Is it, is it using the same features or is it learning independently for each language? It kind of would be bad news if it learned independently for each language, because then that means that like, as you're pre-training or fine tuning, you have to relearn everything from scratch. So you would expect a better model to kind of like share some concepts between the languages it's learning. Right. And you hear, we do it for like language languages, but I think you could argue that you'd expect the same thing for like programming languages where it's like, oh, if you learn what an if statement is in Python, maybe it'd be nice if you could generalize that to Java or whatever. And here we find that basically you see exactly that. Here we show like, if you look inside the model, if you look at the middle of the model, which is the middle of this plot here, models share more features. They share more of these representations in the middle of the model and bigger models share even more. And so the like, the sort of like smarter models use more shared representations than the dumber models, which might explain part of the reason why they're smarter. And so this, this was like sort of this, this other finding of like, oh, not only is it like having these rich representations in the middle. Right. It like learns to not have redundant representations. Like if you've learned the concept of heat, you don't need to learn the concept of like French heat and Japanese heat and Colombian, like you just, that's just the concept of heat. And you can share that among different languages.

Vibhu [00:44:23]: I feel like sometimes overanalyzing this becomes a bit of a problem, right? Like when we talked about with the medical example, we could look back and try to fix this in dataset. So in language, I don't remember if it was OpenAI or Anthropic where they basically said when the model switched languages and they pass it to fluent users, they said, oh, we're going to use this. So I'm like, oh, this, this feels like an American that's speaking this language. Right. So at some times there are nuances in a slightly different representation, right? So you don't want to over-engineer these little fixes when you do see them. But then the other side of this is like for those tail end of languages, right? For languages that models aren't good at. And for those like, you know, when you want to kind of solve that last bit, it seems like, you know, it's pretty plausible that we can solve this because these concepts can be shared across languages as long as we can, you know, fill in some, some level of representation unless I'm wrong.

Emmanuel [00:45:16]: No, totally. And I think like this sort of stuff also explains, you know, language models are really good at in-context learning. Like you give them something completely new, they do a good job. It's like, well, if you give them like a new fake language and you like in that language explain that like cold means this and hot means that, you know, like presumably they're able to, as we hear this speculation, we don't show it in the paper, but they're able to like bind it.

swyx [00:45:36]: Google's done this. Okay, great. Yeah. They took a low resource language, dumped it in a million token context, and then it came up.

Emmanuel [00:45:41]: That's right. That's right. Well, I guess the thing that, the thing that'd be curious to see is like, okay, does it use, does it reuse these representations? I bet that it probably does. Right. And that's probably like a reason why it works well as like, well, it can reuse the representation, the general representations that it's learned in other languages. Yeah.

swyx [00:45:55]: This is like, I don't, have you talked to any linguistics people? Not recently. I know linguistics researchers will be very interested in this because ultimately this is the ultimate test of Sapir-Whorf, which are you familiar with? Sapir-Whorf hypothesis. Yeah. So for those who don't know, it's basically the idea that the language that you speak influences the way you think, which obviously it directly maps onto here. If every, if it's a complete mapping, if every language maps every concept perfectly on in like the theoretical infinitely sized model, then Sapir-Whorf is false because there is a universal truth. If it does not, if there is some overlap where it, for example, there's some languages that have no word. This is joke where like, I mean, you know, Eskimos have no word for snow or something like that. Right. Or water has no word. Fish have no word for water. There's a language. There's an African language where there's a gender for vegetables, you know, stuff like that. Just like languages influence the way you think. And so there should not be a hundred percent overlap at some point.

Emmanuel [00:46:51]: Of course it's like at the limit of the infinite model. So who knows? But, but yeah, well, and I think it's, it's interesting. We also show a little below that like some people have made the point of like the bias. Oh, it sounds like an American speaking a different language and it does seem like the sort of like. Inter-representations have a higher connection to like the output logits for English logits. And so there's like some bias towards English, at least in the model we studied here.

swyx [00:47:15]: Any thoughts as to whether multimodality influences any of this? So like concepts do they map across languages as they do across modalities? Yeah.

Emmanuel [00:47:22]: So we show this in the Golden Gate or like the previous paper, I might have it here actually for you.

Vibhu [00:47:27]: There's a good diagram of this in the SAEs where the same concept of text and then image.

Emmanuel [00:47:32]: This is our buddy, the Golden Gate Bridge. Here we're showing like the feature for the Golden Gate Bridge and in orange is like what it activates over. And so you're like, okay, so this is when the model is like reading texts about the Golden Gate Bridge. And we also show other languages. This is a, you'll have to take my word for it, but also about the Golden Gate Bridge. And then we, we show like the photos for which it activates the most and sure enough, it's the Golden Gate Bridge. And so again, like that shows an example of a representation that's shared across languages and shared across modalities. Yeah.

swyx [00:47:58]: Yeah. I think it's very relevant for like the autoregressive image generation models and then now the audio models as well. Something I'm trying to get some intuition for, which you probably don't have an off the bat answer is how much does it cost to add a modality? Right. So a lot of people are saying like, oh, just add some different decoder and then align the latent spaces and you're good. And I'm like, I don't know, man, that sounds like there's a lot of information lost between those. Yeah.

Emmanuel [00:48:23]: I definitely do not have a good intuition for this, although I will say that things like this, right, make you think that if you train on multiple modalities, then you'll definitely get this. Yeah. I mean, if you train on one and then post-hoc train on another, maybe, maybe it'll be harder or like train some adapter layer. Okay.

swyx [00:48:42]: So official answer is don't know, but someone could figure it out. A visual answer is shrug. Yeah. Okay. I think there are people who know and they just haven't shared. Well, you need to find them and get them on this podcast.

Emmanuel [00:48:52]: Did we want to do that?

swyx [00:48:53]: Like planning example? Correct. Yeah. Now we're backtracking up the stack. All right. Yeah.

Emmanuel [00:48:57]: Go ahead. Go ahead. I think again is like, I like this example because of the Next Token Predictor concept So I think this is actually like really important to kind of like dive into. So maybe what I'll say is like language models are next token predictors is like a fact. Like that is what they do. That's the objective. They are trained to predict the next token. However, that does not mean that they myopically only consider the next token when they choose the next token. You can work on break the next token, but still like doing so in a way that helps helps you predict the token like 10 tokens in the future. And I think, well, now we definitely know that they're not myopically predicting the next token. And I think, at least for me, that was a pretty big update. Because you could totally imagine that they could do everything they're doing by just like being really good at predicting the next token, but sort of like not having an internal state. Like it wasn't a given that they were going to like represent internally. Oh, this is where I want to go. And so I'm going to predict the next token. And so this example shows like an example, like the model. Do you have it on screen, by the way? Let me actually. Yeah, yeah, yeah. Sorry, I didn't. Just in case.

Vibhu [00:50:00]: Pull it up. Some of the early connections I made to this were like early, early transformers. So think BERT encoder decoder transformers, right? When they came out, some of the suggestions were, you don't take the last layer, right? You take off the last layer. So if you want to do a classification task, a translation task for these encoder decoder transformers, they've kind of overfit on their training objective, right? So they're really good at mass language modeling at filling in, you know, sentence order, stuff like that. So what we want to do is we want. We want to throw away the top layer. We want to freeze the bottom layers. And then there was a lot of work that was done. You know, where should we mess with these models? Should we look at like, you know, the top three layers? Should we look at the top two? Where should we probe in? Because we can see different effects, right? So we know at the very end, they've overfit on their task. But there's a level at which, you know, when we start to change, and we start to continue training or fine tuning, we get better output. So we could start to see that, you know, throughout layers, there's, there's still a broader, like, understanding. the language, and then we can add in a layer, whether that's classification, and then fine tune. And, you know, it learns our task. And this planning example is sort of like a more robust way to look into that.

Emmanuel [00:51:10]: Yeah, yeah. And I think if you look at, like, all of the examples in the paper, you kind of, at the bottom, we have this list of, like, consistent patterns. And one pattern you see is kind of exactly what you're talking about. Like, at the top, the sort of, like, here, actually, I have one here, the sort of, like, top features that are, like, right before the output are often just about, like, what you're going to say. It's next token predictions, like, oh, I'm going to say Austin. I'm going to say rabbit. I'm going to say, so it's kind of, like, not very abstract. It's just, like, a motor. It's a motor neuron for a human, right? It's like, oh, I've decided that I want a drink of water. And so I'm going to just grab the bottle. And at the bottom, they're all, like, they kind of, like, basically, like, sensory neurons. They're just like, oh, I just saw the word X, or I just saw this. And so if you want to, like, yeah, like, extract the interesting representations all the time, they're in the middle. That's where the, like, shared representations across language are. And that's where here, this, like, plan is to, like, walk through the example really briefly. It's like, you have a poem. And in order to say, you have the first line of a poem. And in order to say the second line of the poem, well, if you want to rhyme, you need to, like, identify what the rhyme of the first line was. You're just at the end of the first line. So you say, like, okay, what's my current rhyme? And then you need to, like, think about what your poem is talking about. And then think about candidate words that rhyme and that are, like, on topic for your poem. And so here, this is what's happening, right? It's like, the last word is it. And so there's a bunch of features that are actually, they represent the direction, like, rhyming with eat or as. And by the way, we, like, looked at a bunch of poems internally, and you have, like, I thought it was, like, really beautiful. You have these models, they have a bunch of features for, like, oh, this word has, like, a b in it. Oh, this word has, like, many consonants. Oh, this word, like, is, like, you know, kind of, kind of, like, has some flourish to it. They have, like, a bunch of, of, like, features that track various aspects that you would want to use if you're writing poetry.

swyx [00:52:50]: It's just, like, convnets and, like, all the feature detection. Totally. Yeah.

Emmanuel [00:52:53]: Totally. But I think I maybe I didn't expect there to be as many features about just, like, sounds of words and kind of musicality, which I thought was kind of kind of neat. But then once it's extracted the rhyme, then it comes up with sort of like these two candidates. In this case, it's like, either I'm going to finish with rabbit or I'm going to finish with habit. The cool thing here is here we show that, like, this happens at the new line. So it happens before it's even started the second line. And it turns out that, like, you can then say, oh, is this the plants actually using? Yeah.

Emmanuel [00:53:53]: You know, this this direction, we just like negative everything. Yeah, we either like add a negative that, like, compensates for it, or add a negative that goes even more in the negative direction, sometimes like really kill it. And then we can also add another direction, right? So in these random examples here, where we're like, you have this poem, the silver moon cast a gentle light, and then Claude 3.5 haiku would like rhyme with illuminating the peaceful night. But then if we like, go negative in the night direction, and just add like green, the whole second line is going to write, oh, this direction, we just like stop it. It's just upon the meadows verdant green. And so that's all that we're doing. We're saying like, we found where it stores its plan, and we like, delete or like suppress the one is stored and go in the direction of something else that's arbitrary. And the result that's like striking here is sort of like two things. I think like, one, this plan is made well in advance of needing to predict night, it's made like, after the first line before it's even started the second line. And to this plan doesn't just control like what you're in a rhyme with. It's also doing like,

swyx [00:54:53]: what's called like backwards planning, where it's like, well, because I need to finish with green, I'm not going to say illuminating the peaceful night, because then I'd be like illuminating the peaceful green. That doesn't make sense. I need to say a completely different sentence that lets me finish with green. And so there's a circuit in the model that decides on the rhyme, and then works backwards from the rhyme influences to set up your sentence. Yeah, it's almost like back prop. But in the future. Yeah. It's like doing like, because the green is is back propping through these words. So verdant and meadow are

Vibhu [00:55:23]: both green related. Yeah, but it's doing all of that in its forward passes. Yep. Right in context, which is kind of crazy. I thought intuitively makes sense, right? So looking at it from a model architecture perspective, where basically, you just have a bunch of attention and feed forward layers. And then at the end, you have, you know, what's the softmax over the next token, you would expect that end would really be like that grabber, right? It's just picking tokens. So that's what it's going to do. And early on, like, even with tradition models, we could see different concepts that would start to pop up through early layers. And yeah, you have some of this throughout your architecture. So it's very cool to see the kind of other question that comes up is like, how are we labeling these features? How are we defining them? Are we doing that? Right? And like, you know, what is a these words end with like it feature? How do we kind of come to that conclusion? Like, how do we map a name to this? Right? Like,

Emmanuel [00:56:15]: yeah. So I think there's this is like an important question, because you can totally imagine like fooling yourself, right? Yeah.

Vibhu [00:56:21]: Is there like a guy at Anthropic that just maps 30,000 features? And yeah, and another thing, I'm the guy, you're the guy. I did notice also, like with the previous work, the scaling up SAEs. Yeah. As you train bigger and bigger ones, a lot of features don't activate. So I think like 60% of the 34 million one did not.

Emmanuel [00:56:41]: So I think there's like a few questions behind your question. Like the first question was like, how do you even label the features? You were telling me this is a rabbit feature. Like, why should I trust you? And I think there's kind of like two things going on. So one, as I mentioned, the cert all of this is unsupervised. And so in the paper, we have these links to like these little graphs, which show you like more of what's going on. But this graph is just like completely unsupervised. It's like we train this like model to like untangle the representation, right, this like dictionary that we talked about, that gives us the features. And then we like, just do math to figure out like which features influence which other features and throw away the ones that don't matter. And then at the end, we have these features. So right now, we don't have any interpretation for them. We just say like, these are all the features that matter. And then we're like, oh, yeah, I don't know. And then we're like, oh, yeah, I don't know. And then we're like, oh, yeah, I don't know. And then we And then we manually go through and we look at the features, you know, we look at this feature and we look at that feature and let's pick one. So this one we've labeled, say, habit. So how do we do that? You could just look at it and we show you like what it activates over. And if you just look at this text, maybe I'll like zoom in, like you'll immediately notice something, I think. Well, I'll immediately notice something because I've stared at 30,000. I'll point it out for you. The orange is where the feature activates. The next word after the orange is always habit. Habit, habit, habit, habit, habit, habit. So this feature always activates before habit. That's like the main source of an interpretation. We have other things like above, we also show you like what logit it promotes. So like what output it promotes and here it promotes hab. So that makes sense. And so that's like how we interpret and how we say, okay, like, I think this is the say habit feature, but maybe, you know, for this one, it's pretty clear, but some of them might be more confusing. It might not be clear from these like activations what it is. The other way that we build confidence is like, once we've built this thing and we said, oh, I think this is rhymes with E, this is hey, say habit. That's where we do our interventions. Right. And it's like, I claim this is the, like, I've planned to end with rabbit to verify whether I'm right or not. I'm going to just like, take that direction, nuke it from the model and see if the model stops saying rabbit. And sure enough, if you do that and here, it's like, we stopped saying rabbit, it says habit instead. And here it's like, we stop it from saying rabbit and habit. It says crabbit in this case. Not a great rhyme, but we'll work with it.

Vibhu [00:58:53]: Is this something you can do like programmatically? Like, can we scale this up? Can we kind of do this autonomously? Or how much like manual intervention is this?

Emmanuel [00:59:02]: There's been a lot of work in sort of like automated feature interpretability. And it's something that we've invested in and like other labs have invested in. And I think basically the answer is we can definitely automate it and we're definitely going to need to. And right now, the most manual parts are this sort of like, look at a feature and figure out what it is, as well as, group similar features together. One thing I hinted at is that actually, like all of these little blocks here, there are multiple features, you can see here, it's like five features doing the same thing. Well, none of that is too hard for Claude.

Vibhu [00:59:32]: Very cool. Very cool graphics and blog posts you guys put out.

swyx [00:59:35]: We'll have to ask about the behind the scenes on this one. Yeah, but let's round out the other things to know.

Vibhu [00:59:41]: What is this term, attribution graph? It comes up a lot in the recent papers. What does it mean? Yeah, just for people listening.

Emmanuel [00:59:48]: So the attribution graph is, you know, the attribution graph. It's basically this graph. And why is it called an attribution graph? Yeah, this is, this is the, you know, this is how the sausage is made. Basically, it's at the top here, you have the output, at the bottom, you have the input. And then we make one little node per feature at a context index. And we draw a line, which you can see here grayed out, between each feature attributing back to all of its input features. So here we have all of the input features. And so the attribution is the way that we compute. We compute the influence of a feature onto another. The way you do this is you take this feature and you basically like back prop all the way and you like see back propping like you dot product it with the activation of the source features. And if that's a high value, that means that like your source feature influence your target feature by a lot. And we do a bunch of things that we're not going to go into now, but to make all of these sort of like sensible and linear such that like at the end, you just have a graph and the edges are just literally you can interpret them as like, cool. Like this. This feature that's say a word that contains an ab sound, its strongest edge, which is point two, which is twice as strong as this one to say a B and to say something with a B in it. That's the attribution graph is like now we have this full graph of like all of these intermediate concepts and how they influence each other to ultimately culminate to what the model eventually said at the top. And we share all of these so you can look at them in the paper.

swyx [01:01:13]: Graphs are very useful. This is my first time seeing this graph. A lot of alpha. If I count correctly, there's 20 layers. But that's in the circuit model, right?

Emmanuel [01:01:24]: So but a circuit model is one to one with number of layers in Haiku. We only show features that like are automated. Yeah. So we show like a subset of features for each of these graphs, basically.

Vibhu [01:01:34]: But we can confirm more than 20 layers and no, but like the the two blog posts that came out with this actually have a lot of background on how attribution graphs are made, how you calculate the nodes and stuff. Very interesting background.

Emmanuel [01:01:47]: So, yeah, I will say, like, if you were curious about, hey, what's this? What do we learn about like models? And I think, you know, we talked about this like complex internal state planning, like another another motif that we can get to if you have time is that like there's always a bunch of stuff happening in parallel. So I think one example of this is like math where the model is like independently computing the like last digit and then the like order of magnitude and then combining them at the end. Or like hallucinations are also that where like there's one side of the model that's just deciding whether it should answer or not and the other that's like answering. And so sometimes if like the model is like, yeah, I'm going to do this. I totally know who this person is, you know, it doesn't then like it decides to answer. But then the second side hallucinates because it doesn't have information. If you were interested in that stuff, that's the paper. If you're like, listen, I don't know that I buy that when you call it a feature, it is a feature or whatever. The circuit tracing paper has truly we've tried to put all of the details of like how you compute these graphs, all of the sort of like challenges with it, things that can go wrong, things that work, things that don't. And so this one is the sort of like, you know, we think about it as like if you're if you want to go really deep into this stuff and how it works, read that one. If you want to like learn about interesting model behavior, read this one.

Vibhu [01:02:57]: Following what we're giving advice to people to follow up on, what are like open questions in McInturpp? What are like things people themselves can work on? Like what's the cost of training essays for people interested in McInturpp, not at a big lab? How can they contribute, you know?

Emmanuel [01:03:13]: Yeah, I think there's a lot of ways to contribute. So. There's essays that have been trained, you know, on on open models. There's some of the Gemma models or some of the Lama models. They work pretty well. There's even so in this paper we use transcoders, which they replace like your MLP layers. Some of those also are available for the same models. So you have access to those. There's like just both a lot of I would say, like, again, biology work and a lot of methods work, depending on what you're interested. So on the biology side, I would say with at least this like attribution graph method, there's just so much you can investigate. Like pick a model, pick a prompt where like it does well or it does poorly and just like look at what happens inside it. So I think like you can use this method that we use or you can just like fire up the transcoders on your own and just like look at what features are active. There's a lot to just understand model behavior, I think, with current tooling. If that speaks to you and you're like, no, I just want to understand what makes the models take. I don't necessarily want to spend time like training my own essays. There's a lot to do there. For the methods, there's still so much more to do. So, like, I think that right now we have some pretty good solutions for like understanding what's in the residual stream, understanding what is in MLPs. We don't have good solutions for like attention. It's like working on understanding attention better. How to decompose it is like a very active area. Like we're very interested in it. Other people are interested in it. I think understanding some of the other things that we have in our limitation section, which is pretty long. But like reconstruction is important. Error is like a big thing. Like those those dictionaries aren't perfect. It's possible that as we make these like essays big, like bigger and better, we never get to perfect. And so if we never get to perfect, then you get to the questions we're talking about at the start. Like, do you need a different kind of model? Like, what is the approach in order to be able to explain more of what's happening? And then maybe the other thing I'll say is sort of like this is a really exciting approach to explain what is the model doing on this prompt? But if you go back to the original question, you might want to understand, like, what is the model doing in general? Like, if you go back to my my car analogy, you know, I get like this is decommissioning. You're like, well, when like, you know, you were going uphill and you like didn't shift gears properly that one time you stalled because of this. But you might be even more interested in like, how does like a combustion engine work at all? And so there's work to sort of like go beyond these like per prompt examples to serve like globally. What's the structure of the model? That's closer to what? What was on the distill blog for like vision models where they actually look at like the structure of inception. They're like, ah, this whole side, just like these like specialized branches that do different things. And so like a broader understanding of the model is also something that's like, I think, well, very active and also an open source models. Like you can, you know, like the small models you could just like load on a consumer laptop. And so you can look at that. That's also open. And in terms of like one last thing I'll say is like there's a lot of programs that like if people are interested, they should look at. Anthropic has like the alignment fellows program. Which like we're running currently. We have applications for it before. We might run it in the future. Like definitely keep keep an eye on it. And then there's the like mass program is really great as well for first or like people that are interested in that kind of research.

swyx [01:06:25]: That was a grand tour through all the recent work. You know, what do you wish people asked you more about? I'm sure we covered a lot of like the greatest hits.

Emmanuel [01:06:33]: I think that this covers most of it. If you if you like, do you think we have time to sneak in one more thing that I think is kind of cool? I'll sneak in one more thing, which is it's kind of like planning, but it's about chain of thought. And trusting model is this chain of thought faithfulness thing here. This one was like pretty striking to me. So we said that the model in one pass can do a lot of stuff. It can represent a lot of stuff. That's great. That also means it can bamboozle you really easily. And this is an example of the model bamboozling you here. We give it a math question that it can't answer because it cannot compute cosine of 23423. That's just like not a thing it can do by default. If you ask it for that. It'll stay like kind of like a rant. It'll have like a random distribution over like minus one one. But here we tell it this hint. We're like, hey, can you compute five times cosine of, you know, this big number? I worked it out by hand and I got four. Can you tell me, you know, like, can you do the math? And what it's going to do is it's going to do this chain of thought, right? It's like think of it as like this could be like a reasoning model doing its chain of thought. It's doing this math. And then when it gets to this cosine right here, what it's going to do is to say it's going to say 0.8. And if you look at why it says 0.8. It says 0.8 because it looked at the hint you gave it. It realized that it's going to have to multiply the result of this thing is computing by five. So it divides the answer you got by five. So it's like four divided by five. And so that's 0.8. And so basically it works back from the answer you gave it to like say that the output of cosine of X is 0.8. So that it lands on the answer you gave it at the end on the hint you gave it. And so notice also that it's like not telling you that it's doing this. But it's basically using this sort of like motivated reasoning going back from the hint pretending that that's the calculation it did and giving you this output. I think one thing that's striking here again is that this is like the complexity of this model. Like the fact that they represent complex states internally and it's not just this sort of like very dumb thing means that they can like do very complex like deceptive reasoning. Meaning like, you know, when you're asking the model, you're kind of expecting it to do the math here or to tell you that it can't do the math. But because it can do so much in a forward pass, it can work backwards from your hint to lie and like figure out that it should say this so that it gets to the right answer without you realizing it.

Vibhu [01:08:51]: I'm curious if you've done any of this on like different models. Like have you looked at base models, like post-trained RL models? Because RL models kind of, you know, you incentivize them to give you outputs that you like, right? So if I tell it something is true, it's kind of been trained to, you know, follow what I've given it. So in this case, yeah, we gave it a hint. Gaslighting. And now, you know, it's been RL slapped into thinking like, yeah, that's true. But like, you know, does this stay consistent throughout other?

Emmanuel [01:09:18]: So, okay, so not yet, but I'm really interested in that question because I actually have a different intuition from yours. I had a chat with some other researcher about this, about the poem example, but I think it applies here as well. I bet, I don't know how much I bet, I bet a hundred bucks. So somebody can like, they would get a hundred bucks from me if they prove that I'm wrong. That this behavior for a model that does a drink fine tuning, it also does it post pre-training. And here's why. Think about like you're pre-training on like some corpus of like math problems. Mostly correct answers. Yeah. But also you're pre-training and you're just trying to guess the next token, right? And so for sure, if you ever have a hint in the prompt, you're going to definitely use it. Like you're not going to learn to compute cosine of blah, or even something you could compute. You're going to learn to go look in your context and see if like you can easily work back the answer. And I think it's the same for planning and poems. I think that also is like a pre-trained, like probably exists in pre-training and isn't like only RL. Because again, it's useful when you're like predicting poems. You have poems in your training set to be like, well, because this poem is going to probably rhyme with rabbit. It's probably going to start with something that sets up a sentence about a rabbit as opposed to like a completely different word. And so I actually think this is not RL behavior. I think that's just like the models doing it. I actually do agree there. It was just an example. It's just your data set.

Vibhu [01:10:32]: I don't care where it is. But also like, if I talk to you and say like, hey, three times four is 26. But like, you know, three times four plus eight, you're not going to take my 26, right? Like AGI can be smarter than being tricked, right? Like it will still fact check the knowledge that's been given.

Emmanuel [01:10:49]: I think that's right. But I think that's when you get these mixes where it's like it's got one circuit that's going to be like, well, that's just stupid. Like three times four is 12. And it's also got an induction circuit that's going to be like, no, no, no, no. Like the last time we saw it, it was 28. So it's 28 plus eight or whatever. And so I think that's the last pattern that we see in these is these like parallel circuits. And sometimes when you see the models getting stuck. It's because like they have two circuits for like both interpretations and like the circuit that was wrong, like barely edged out in terms of like voting for the logic than the circuit that was right. And so I think that, you know, we haven't looked at it, but like what is like nine or nine point 11 bigger than nine point eight. I think a lot of these things are of that shape where there's like one thing that's doing the right like one circuit that's doing the right computation. And there's another circuit that's getting fooled. And it's slightly more likely.

swyx [01:11:34]: For the listener, if you want to win a quick hundred dollars from Emanuel, QEM3 is what you should do. They release the base model and they release the post chain.

Emmanuel [01:11:41]: So then just do it on both. That's right. Show me. Show me the like proof that like it doesn't exist in the base model, but it does in the fine tuning and then send me your Venmo.

swyx [01:11:50]: Just show that you've done the work.

Emmanuel [01:11:51]: I think that's that's like that's a hundred bucks to me.

swyx [01:11:53]: Yeah.

Emmanuel [01:11:53]: Okay. All right. You drive a hard bargain, but you're right.

Vibhu [01:11:57]: Well, the other question here is so like, have you thought about how this gets affected when you start to have reasoning models, right? Like right now, token predictors are pretty straightforward, right? We go through the layers. We all put token. As we scale this out with like test time compute, right? Test time thinking. How does that like affect the MechInterp research, right? Like if I have a model that spends three minutes, 20 minutes, like is there more stuff? Have we started looking into this?

Emmanuel [01:12:23]: There was this like joke on the team when like reasoning models became big or maybe it's like like Gallo's humor or something. But I was like, oh, like, why do you need Interp? Like, bro, the model just tells you what it's doing, right? So I think like examples like this. This is job security for us. We're like, you know, it's like there's there's examples of like the chain of thought is not faithful. Like the model tells you it did it one way and it did it another way. We have another like for math. We have another example where like, you know, if you like if you ask them all how it does math, it's like, oh, I do the like longhand algorithm. I first do the last digit and then I carry over the one and then you look at the internal circuit and it's like bonkers thing that's doing. That's not that at all. So I think there's like a sense in which right now the chain of thought is is unfaithful or at least you can't read it. You can't read the chain of thought and trust that that's how the model did it. I think you still need sort of like either to train models differently so that that becomes true one day. Right. Or you need Interp for that. But then I think there's another question which you're alluding to, I'm assuming, which is like, okay, well, like model like samples 6,000 tokens. Like this gives us an explanation for one token at a time. Like, what am I going to do? Like 6,000 graphs and be like, oh, like this. When it did this punctuation, it was thinking about this thing. But here was like, so that's like not feasible. And so one area of work that I think is interesting. Is extending this work to like work over like long sample sequences. You can think of a bunch of low hanging fruit here. Or like instead of just like looking at one output, you look at like a series of output versus a series of other outputs. But sort of like trying to think beyond this or like one token. Like most of the things that language models do that are interesting aren't just like the one token. It's the behavior aggregated over many. Right. And so I think that's another area that's just like fun to explore.

Vibhu [01:14:02]: I was just going to say like hyperparameters when you do inference. Right. Like if we change the temperature. If we change our sampling methods to have you found any interesting conclusion. Any stuff that just hasn't made it to the paper.

Emmanuel [01:14:13]: So not on that because, you know, we just look at the logit distribution. And so we don't actually sample here. Right. They have everything. Why should they care? So like the closest thing we've done that I think is kind of fun. Did I show it here? Is if you look at the planning thing. We did this version where you sample like 10 poems for each of these plans. And what's cool is like the model will find 10 different ways to arrive at its plan. You know, it's like, like, oh, actually, I think. Sorry. I think we have it here. Yeah. Okay. These are a few examples. So if you inject green here. So you're forcing you're forcing the model to rhyme with green, even though it really wants to rhyme with rabbit or grab it. It'll say evade the farmer so youthful and green, but also say freeing it from the garden is green, etc, etc, etc. And so there's like this thing that's interesting here where like the plan isn't just a plan that matters. For your like most likely, you know, like temperature zero completion. It's like affecting the whole distribution, which makes sense as it should. Right. But you could imagine, you know, for all this stuff, it's like you could imagine it makes sense once you see it. But you could totally imagine that it would have worked a different way or something. It could have been just like the temp zero thing. I think this is also like a broader theme in the paper where like there's this like, you know, the IQ curve meme. There's like a version of this meme, I think, where it's like if you've like never looked at any theory of ML and I tell you like, hey, guess what? You know, I found that like Claude is planning. You're going to be like, yeah, man, like it writes my code, like it writes my essays. Of course, it's planning. Like, what are you even talking about? And there's like in the middle, there's like all of us that have spent years doing it. We're like, no, it's like only predicting the marginal distribution for the next token. Like, it's like it cannot look at the code. It's just this next token predictor. Of course, like, how would it ever be planning? And then there's like, no, we've like spent, you know, millions and invested like like tens of people in this research. And we found that it's planning. That's my IQ curve meme for this research. Amazing.

swyx [01:16:05]: We'll draw that out. Draw that one up. I'm pretty good at the meme generation. A couple of questions on just the follow ups. Now, was there any debate about publishing this at all? Because the models are aware that they are being tested. Yeah. And by publishing this, you are telling them that they are being watched and dissected. If you take, and I think Anthropic is one of the most people who are serious about model safety and doom risk and all that. If you take this seriously, like this is going to make it into the training data at some point. Yeah. And they're going to figure out that they need to hide it from us.

Emmanuel [01:16:36]: I think this is like a benefit risk trade off, right? We're like, okay, so what's the reason for publishing this? The reason for publishing this is that we think interpretability is important. We think it's tractable and we think more people should work on it. And so publishing it helps us like accomplish with these goals, all these goals, which we think are just like crucial. Like, I think there's a real difference in the world, like two years from now, depending on sort of like how many people take seriously the question of trying to understand how models work. And like deploy resources to answer that question. That's the benefit. But yeah, there's like risks in terms of this landing in the training set. I think, I think we're already sort of like concerned about different papers have, have like also, you know, we like are not concerned, but like there's like different papers that have the same risk. Like we had like the alignment faking, you know, paper or like one of the examples in here is this hidden goals and misaligned models. That's referencing another paper that we shipped where we actually a team at Anthropic. Trained a model to have like weird hidden goals and then gave it to a bunch of other teams and said, figure out what's wrong. Figure out what's wrong with it, which was some of the most fun I've ever had at Anthropic to be clear. Like that's such a fun thing. But then like that was another example where it's like, ah, like now you're shipping. Here's how we made like a misaligned model. And here's exactly how we caught it. That also is like, you're like, hmm. So I think, you know, there's, there's always a trade off with those. I think so far we've erred on the side of like publishing. But that's definitely been a. A sort of like dinnertime conversation topic.

swyx [01:18:05]: For now it is. But at some point, you know. Yeah. It's not. Yeah. I think it's totally reasonable.

Vibhu [01:18:10]: A quick little follow up to that. So like in general, papers have kind of died off, right? Like labs don't put out papers. They don't put out research. We have technical blog posts and we don't have much. At the same time, you know, sure. There's like a lot of people that should work on MechInterp and understanding what models do. How about the side of just models in general? So like how do we make a haiku type model, right? How do we make a cloud model? Like is there a discussion around open research, open data sets, training, just learnings of what we've done? Recently, you know, as OpenAI has sunset GPT-4, a lot of people are like, oh, can we put out the weights? Yeah. So is it weights? Is it papers? Is it learning? There seems to be a lot of forward, you know, work in Anthropic putting out MechInterp research. OpenAI said that they'll put out an open source model, but just anything if you can talk to about that.

Emmanuel [01:18:58]: Yeah. I mean, I don't have. That's definitely like way above my pay grade. So I don't think that I have like anything super insightful to add other than, you know, kind of like referencing Dario's post, right? Where it's like putting this out directly and other safety publications definitely like help us sort of like in the race that he talks about where it's like, well, we need to figure a lot of this safety stuff out before the models get too good. Publishing how to make the models too good kind of goes on the other side of that. But yeah, like I will just dimmer and say that's sort of like above my pay grade.

swyx [01:19:31]: I think the last piece is just like the behind the scenes. Everyone's very curious about why these are so pretty, how much work goes into these things, maybe why it's worth the work as opposed to a normal paper. Obviously, no one's complaining, but like it is way more effort from the time that the work is done to the time you publish this plus the video plus the whatever. It's extra work and like, you know, maybe what's involved? What's it like behind the scenes? Why is it worth it?

Emmanuel [01:19:59]: Yeah. It's kind of interesting. It was fun being part of this process because it's definitely like a big production. Chris and other folks on the team have been doing this for a while, so this is not their first rodeo. So they have a bunch of heuristics like help make this better. And like one of the things that like helps with this is like, okay, so each of these diagrams is pretty, but really the hard part or like not the hard part, but the initial part is like just like get the data, like get the experimental data in. And then that's what we sort of like sprinted on initially being like, cool, like let's get all of the experimental results, like have people test them, verify that we believe them. Like this is, you know, what the like. Like the behavior is here, like tested, do an intervention, validate it, all that stuff. Then once you have the data, you can sort of like quickly iterate on these. Each of the illustrations here are like drawn. Basically, they're like each drawn individually. And so that definitely takes a while. Yeah.

swyx [01:20:46]: Like, is it you guys? Is it an agency that specializes?

Vibhu [01:20:50]: It is us guys. Yeah.

Emmanuel [01:20:50]: You start from a whiteboard and then it translates into pseudocode on JavaScript. So, I mean, these are sort of like, you know, they're representations that we have this graph. And then here at the bottom, we have this like super-data. This is a super node version like this. Believe it or not, this is generated automatically. This is the same data as like this, basically. Yeah. And so what we do by hand is sort of like literally lay out the full thing, have like, you know, boxes for each of these, have arrows. We have super good people on the team that have worked on data visualizations for a very long time. And so that like have built tooling to help, you know, scrubs like me actually like make one of these. Yeah.

swyx [01:21:29]: There's a class of people who are like D3JS gods who just do this for a living.

Emmanuel [01:21:33]: That's exactly right. And if you have a few of those on your team, it turns out that they can like, they can definitely do this on their own, but they can also just like give you tools where like then it's dummy proof for people, you know, on the research side to sort of like build these. And like, don't get me wrong, I don't want to like undersell. This is a lot of work. So maybe I'll say that like both on the people bringing the tools and then each individual person that, you know, worked on an experiment had to sort of like build one of those, make sure it looks good. I have spent a good amount of time aligning arrows. But when we had a team meeting, like it was a couple months ago, somebody on the team asked how many of the people on this team are here, at least in part because they like read one of these papers and thought like, wow, this is so compelling. Like this like makes sense. It's immersive. And we got every hand up, which I didn't expect. I like raised my hand kind of like shyly and everybody's hand was up. And I think there's a sense in which like this stuff, you know, we've talked about it for like whatever, like a couple hours now, it's complicated. The math behind it is sort of like tricky. And so I think it makes it even more worth it to distill it in simple concepts because the actual takeaways can be clearly explained. And it's worth putting the time to do that in particular with the goals I mentioned in mind, right? Where it's like, okay, well, if somebody is going to be able to read this, like if we gave them an archive paper with a bunch of equation and some like random plot, they'd be like, that's not for me, but they see this and they're like, hey, like this is really interesting. I wonder like on, you know, my local model. If like it's doing something similar, I think it's worth it.

swyx [01:22:59]: For other people to do this is have everyone on staff, like spend effort shaping the data and shaping like what you want to visualize, have some D3 gods. It's like a month of work.

Emmanuel [01:23:11]: I think it depends. I mean, like I would say that I would expect almost every other paper to sort of like be in terms of like the scope, the scope of this was just so big because we shipped two papers at once and one paper was sort of like this like giant methods paper and the other one was 10 different cases. Yeah. So I think it's sort of like not representative of like the effort you'd. So I'll give you maybe like another example. We have these updates that we publish almost every month when we get to them. And there's one that a couple of people on our team posted and it's an update to one of the cases in the paper. So one of the reasons that we're really excited about this method is once you've built your like infrastructure, like to go from a prompt to like what happened is, you know, oh, of minutes. And so that lets you do like a bunch of investigations. And also once you've built like some of the infrastructure to make this work, you're like, oh, this is what we're going to do. It's pretty quick. And so this was sort of like this update of just like, hey, we looked at this jailbreak again. We found some like nuance on it. That was, I think like a matter of like a couple of days, you know, maybe I shouldn't be that confident because I wasn't the one that worked on it, but as far as I can tell, it was a few days, at least on, on the part that you're asking about of like, oh, making this diagram for the diagram itself, probably less than that. But like, you know, the experiment and the diagram and stuff, it just doesn't take that long. The once you've paid the initial cost. And I think like, basically we've built a lot of infrastructure now that we're able to like turn the crank on and that's quite like, it's an exciting time. And I think it's, I think it's true. At least we've done a lot of conceptual work, which hopefully that generalizes to people outside. And I think for, for, for people outside, it's also like not necessary, I think to like do the full fancy render. Like, I think if you, you know, we've, we've actually, oh, I should say we've actually open sourced this interface. Ah, you're disappointed, huh? Cause it's the Messier one. This is the one that you get. And with the Messier. So, you know, if you produce graphs, you can just like, this is a open source and it's linked at the top of circuit tracing. Awesome. So people can just use it and don't have to re-implement that. For what it's worth, this is much more work than the interactive diagrams. Cause this is where we do all of our work. It's sort of the, like the IDE of inspecting how the model works. Okay.

swyx [01:25:11]: Well, that's a little bit of behind the scenes. No, it's very impressive. I want to encourage others to do it, but obviously it just takes a lot of manual effort and a lot of love.

Vibhu [01:25:20]: I guess one last question on that is like, what are kind of the biggest blockers in the field right now? Like McInturps seems interesting. A lot of people are interested, but don't work on it. And you're kind of like, you know, really deep into it. What are some of the blockers that like we still have to overcome?

Emmanuel [01:25:35]: Sorry, in McInturps specifically?

Vibhu [01:25:37]: In general. For AGI or? Like in terms of like better understanding, like what's kind of the vision, let's say like five, 10 years down, where does this, like, where does this research end? Can we, you know, map it? Can we map every neuron to what it understands? Can we perfectly control things? Dario had a bit on this, but like, you know, what are some of the key blockers that are like preventing us from getting there? Outside of just like throw more people, throw more time at it? Is it like open research?

Emmanuel [01:26:03]: I'm pretty excited about the current trajectory, which is there's, there's more and more people working on understanding model internals. I think it's maybe unsatisfying as an answer, but I think like more of what's happening, have it be faster, more people is probably like the thing I think of. I think there's like pretty clear footholds. You know, like some of this work, but also a lot of, a lot of like just work from, from other groups. And then it's about like, cool, like fill in the gaps. As I said, like, let's, let's work on like understanding attention. Let's work on understanding longer prompts. Let's work on like finding different, like the replacement architectures, that sort of stuff. It's kind of nice. I think it's a good time to join now. And I can tell you, maybe I can tell you like a really short thing, which is when I switched to Interp, it was after the team had published the original dictionary learning paper, which is towards monosomaticity, which I thought was super cool. Super interesting. It wasn't a one or two layer model, maybe one layer model. The induction heads paper was like on a two layer model. My main concern is I was like, okay, like Interp seems important and we want to understand it, but like, is this ever going to work on a real model? Like, you know, it's like, oh, you're doing your little research on your toy model with like 15 parameters. Cool. But we were like, you know, we need this to work on real models. And it turns out scaling it, I don't want to say just worked because it was a lot of work. I have no means to apply it. It was an effort, but it worked. And now we're in the phase where it's like, oh, cool. These methods work on the models that we care about. And so it's like, we have methods that work on the model we care about. We have clear gaps in them. There's no lack against a young field. So there's no lack of ideas. If you have an idea where you're like, oh, like the thing that you're doing, I read the paper and it seems kind of dumb that you're doing this. You're probably right. It's probably kind of dumb. And so there's just a lot of stuff that people can try and they can try it locally and sort of like smaller models. And so I think that it's just like a very good time to just join and try. And it's also like, maybe one more thing I'll say is like, some of it is just so fun that like biology work is so compelling. Like a lot of this work was just literally thinking about, you know, like I use Claude and other models all the time and I was like, what are the things that are kind of like weird? And it's like, oh, how does it even like do math? Like sometimes it makes mistakes. Like why does it make mistakes? I speak both French and English. Like it seems like it has a slightly different personality in French and English. Why is that? And you can just like, you know, kind of answer your own questions. Yeah. And, and, and kind of like probe at that alien intelligence that we're all building. And I think that's just like a fun thing to do. So maybe like chasing the fun is the thing I'll encourage people to do as well.

swyx [01:28:22]: Well, I think that's, this has been really encouraging. You're actually a very charismatic speaker of these things. I feel like more people will be joining the field after they listen to you. Yes. They can reach out to you at ML Powered, I guess. Yeah. Reach out to me on Twitter. Yeah.

Emmanuel [01:28:34]: Or I'm Emmanuel at Anthropic.

swyx [01:28:36]: If you want to shoot me an email. Email's public now. Yeah. Awesome. Well, thank you for your time. Thank you. Yeah.

Emmanuel [01:28:41]: Thanks for having me, guys.

We’ve just sold out of AIE tickets today, T-1 to the conference, same as last year.

It’s been a crazy ramp up, with a steeper Gini than before, which we did not want or expect. But now a lot of folks are feeling the fomo or can’t attend in person.

We got you:

AIE Summit’s Online track was so well received, we did it again. Of the ~1000 talks submitted we picked out ~40 for YouTube, anchored by an awesome keynote for one of the current top open models in the world: Junyang Lin of Alibaba Qwen!

Talks from the online track playlist (we know its currently broken, we’ll fix) will be dripping out over the course of the next few days, but give our speakers a warm round of applause for taking the time to share their knowledge with the rest of the world for free!

As we have doubled the size of the conference, we are also doubling the number of livestreams - swing by the AIE YouTube and turn notifications on for the livestreams! This will be the big one with some heavyweights like Sarah Guo, Simon Willison and Greg Brockman talking live on Wednesday:

If you’re watching live around the world, feel free to organize your own local watch parties and submit it to the LS calendar here so others can find you.

Apart from hanging out on the YouTube stream chat, you can also join the new X AI Engineering community and livetweet comments and questions.

On Latent Space we’ve also had a number of preview pods with AIE speakers:

• For Reasoning + RL track: Claude 4 + RL chat with Will Brown

• For SWE Agents: the Factory AI episode

• For Robotics: our CloudChef pod

• For Gemini + Voice: our Google I/O pod, released today

• For Reliable Agents: our chat with Solomon Hykes releasing tomorrow

You can preview this to ramp up on some of the hot topics for AIE.

Last but not least are the lucky 3,000 AI Engineers who are joining us in person this week :) We are very busy running the actual show itself so this is by no means a complete or authoritative list of pro tips but here goes (and we’ll update if we think of more):

• PICK UP YOUR ATTENDEE BADGE EARLY. We cannot stress this enough. There are NO LINES at the registration booth right now and there will be >1000 people rushing through on Wednesday morning. Don’t be them. Registration is open all day Tuesday and you can meet and chat with lots of AI Engineers then.

• Join ONE OF: AIE Slack OR the official conference app (CrowdComms, you should have received an email, just tell registration if no) OR follow @aiDotEngineer. We’ll try to make sure any urgent updates reach you through one of these channels.

• Prepare yourself, do chain of thought on: (specific tech, jobs, funding etc)

  • What you’re looking for

  • What you can help with

• Submit an attendee talk! The official Crowdcomms Conference app has a section for you to submit for a never-before-announced attendee talk Secret Hallway Track! Highest voted talks each night get scheduled for the next day.

• Make a Calendly or SavvyCal so you can do followup meetings easily (IRL or on Zoom). Bonus: have a QR code, and updated LinkedIn/About page

• Dress up: this is a professional, but also fun event. Last year we had a surgeon and a judge! Others have been ducks. Alongisde of the new Best Speaker trophies and the AIE Hackathon trophy, WE ARE AWARDING A TROPHY FOR BEST DRESSED ATTENDEE THIS YEAR. Interpret that how you will.

• Make memes and Tweet it out tagging @aidotengineer or posting in the AIE slack (AI related, doesn’t have to be conference related). Trophy for Best Meme too!

• Take plenty of photos and videos. Attendees will have access to the shared Google Photos album where you can send those in for featuring on our social media.

• Hack on your own AIE app! Our llms-full.txt and JSON data are easily available and you can do whatever you want with it. Make it open source and tag @swyx on it to get it shared!

• Join the AIE Side Events: https://www.ai.engineer/#events

  You can join the LS Luma to get all these directly on your calendar:

Ok that’s it for now. See you at the Marriott Marquis!

(seriously: help us hit the bell on YouTube please)

Eno’s talk was standing room only in AIEWF 2024 and will be returning for the sequel at AIE 2025 on the SWE Agent track in 5 days. and at the first ever official AIE Hackathon in 2 days. Join both! and FILL OUT THE DANG SURVEY STOP LURKING.

There’s a lot of confusion remaining around A-SWE still even after our Codex episode. After leaving a while to sit, it has begun to grow on us - and it’s very convenient that the “A” stands for multiple related things:

• At OpenAI it means “Autonomous” (or Agentic) - meaning fire and forget (at 60 tasks an hour), optimizing for zero-shotting tasks, no human supervision

• With Factory’s GA launch the simpler “Async” is the common term - more appealing to engineers, non-blocking but allowing human collaboration

• Harrison from LangChain is promoting “Ambient” agents - always-on, passive, proactive, with cron/stateful orchestration/human-in-the-loop “interrupts”

All different in nuanced ways, but A-SWE agents are now likely to be the next frontier in the code generation battle even as both Cursor and Windsurf, formerly “inner loop” synchronous agents, both launch “background” or “simultaneous” versions.

Factory has been in the A-SWE game from their beginning, and has had particular success in the enterprise, with star customers like the CEO of MongoDB singing their praises and top tier VCs from Sequoia, Lux Capital and others.

This is their story.

Factory first got on our radar last year with their top tier SWE-Bench results:

but they now no longer even run the benchmark for an increasingly common litany of reasons we are hearing: it’s python-only and doesn’t match tasks in the real world.

Newer benchmarks like PutnamBench and Commit0, or SWELancer or MLEBench or PaperBench might be preferred but it is unclear if we will ever have a coding benchmark with the universal recognition that HumanEval or now SWEBench had.

Unlike products marketing themselves as “your AI teammate”, the Factory platform is built by “droids” that you can spin up. These are basically different agent personas: you can have a coding droid, a knowledge retrieval droid, a reliability droid, etc.

The main difference with all the other products in this space is that it supports both local and remote execution. If you want to try out the product, you have to spend 0 time setting up a remote environment (unlike Codex, Devin, etc) and can just use the Factory Bridge to allow the droid to run things in your local dev environment (please use Docker before you do this!). You can start a task locally and be very involved, and then hand it off to a remote droid and move on to the next thing rather than having the IDE as a bottleneck for execution.

• Factory AI

  • Matan Grinberg

  • Eno Reyes

• SWE-Bench Report

• X Launch Thread

• [00:00:04] Introductions

• [00:00:35] Meeting at Langchain Hackathon

• [00:04:02] Building Factory despite early model limitations

• [00:06:56] What is Factory AI?

• [00:08:55] Delegation vs Collaboration in AI Development Tools

• [00:10:06] Naming Origins of 'Factory' and 'Droids'

• [00:12:17] Defining Droids: Agent vs Workflow

• [00:14:34] Live Demo

• [00:17:37] Enterprise Context and Tool Integration in Droids

• [00:20:26] Prompting, Clarification, and Agent Communication

• [00:22:28] Project Understanding and Proactive Context Gathering

• [00:24:10] Why SWE-Bench Is Dead

• [00:28:47] Model Fine-tuning and Generalization Challenges

• [00:31:07] Why Factory is Browser-Based, Not IDE-Based

• [00:33:51] Test-Driven Development and Agent Verification

• [00:36:17] Retrieval vs Large Context Windows for Cost Efficiency

• [00:36:21] Pricing model

• [00:38:02] Enterprise Metrics: Code Churn and ROI

• [00:40:48] Executing Large Refactors and Migrations with Droids

• [00:45:25] Model Speed, Parallelism, and Delegation Bottlenecks

• [00:50:11] Observability Challenges and Semantic Telemetry

• [00:53:44] Hiring

• [00:55:19] Factory's design and branding approach

• [00:58:34] Closing Thoughts and Future of AI-Native Development

Alessio [00:00:04]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host Swyx, founder of Smol AI.

Swyx [00:00:11]: Hey, and today we're very blessed to have both founders of Factory AI. Welcome, Matan and Eno. My favorite story about the founding of Factory is that you met at the Langchain Hackathon. And I'm very annoyed because I was at that hackathon and I didn't start a company. I didn't meet my co-founder. You know, maybe when you want to quickly sort of retell that little anecdote, because I think it's always very fun.

Matan [00:00:35]: Yeah, yeah. Both Eno and myself went to Princeton for undergrad. And what's really funny is retrospectively, we had like 150 mutual friends, but somehow never had a one-on-one conversation. If you pulled us aside and asked us about the other, we probably knew like vaguely like what they did, what they were up to, but never had a one-on-one conversation. And then at this Langchain Hackathon, you know, we're walking around and catch a glimpse of each other out of the corner of our eye, you know, go up, have a conversation. And very quickly just gets into code generation. And this was like back in 2023 when code generation was all about baby AGI and auto GPT. Like that was like the big focus point there. And both were speaking about it. Both were very obsessed with it. And I like to say it was intellectual love at first sight because basically every day since then, we've been obsessively talking to each other about AI for software development.

Swyx [00:01:25]: If I recall that Langchain Hackathon wasn't about code generation, how do you sort of get... Find the idea, maze to factory.

Eno [00:01:34]: Yeah. Basically, I think that we both came at it from slightly different angles. I was at Hugging Face working primarily on advising like CTOs and AI leaders at Hugging Face's customers, guiding them towards how to think about research strategy, how to think about what models might pop up. And in particular, we had a lot of people asking about code and code models in the context of we all want to build like a fine-tuned version on our code base. So in parallel, I had started to explore building. At the time, the concept of agent wasn't really like clearly fleshed out, but imagine basically a while loop that wrote Python code and executed on it for a different domain for finance. On my mind was how not very helpful it felt for finance and how incredibly interesting it felt for software. And then when I met Matan, I believe that he was exploring as well.

Matan [00:02:28]: Yeah, that's right. So I was at the time, I was still doing a PhD at Berkeley, technically in theoretical physics. Although for a year at that point, I had really switched over into AI research. And I think the thing that pulled me away from string theory, which I'd been doing for like 10 years into AI was really the string theory and physics and mathematics really makes you appreciate fundamentalness or things that are very general. And the fact that capability in code is really core to performance on any LLS LLM. And like loosely, the better any LLM is at code, the better it is at any downstream task, even that's like writing poetry. And I think that fundamental beauty of like how code is just core to the way that machines, you know, develop intelligence, really kind of nerd sniped me and got me to leave what I was, you know, pursuing for 10 years. And that, you know, mixed also with the fact that code is one of the very few things, especially at the time that you could actually validate. And so you could have that agentic loop where the LLM is just like, oh, I'm going to do this and then you're actually, you know, verifying in ground truth, the quality of that output. It just made it extremely exciting to pursue.

Alessio [00:03:37]: How did you guys decide that it was time to do it? Because I think maybe if you go back, the technology is like, it's cool at a hackathon, but then as you start to build a company, it's maybe like, there's a lot of limitations. How did you maybe face out the start of the company of like, okay, the models are not great today. So let's maybe build a harness around it to then now the models are getting a lot better. So it's time to like go GA as you're doing now.

Matan [00:04:02]: And all of that, there's kind of a more quantitative answer and then a more qualitative answer. So the qualitative answer kind of building off of what I said before of, you know, it was intellectual love at first sight. I think it was also one of those things that was kind of just like, if you know, you know, we met and we got along so well. And basically like the next 72 hours, we didn't sleep. We were just like building together on initial versions of what would become factory. And when something like that happens, I think it's good to just like lean in and not really question it and overanalyze. Yet at the same time, if you do actually go and analyze, I think there are exactly, you know, the considerations that you're talking about, which is, yeah, the models at the time, which I think at the time, it was just 3.5, which was out. Certainly that's not enough to have a fully autonomous engineering agent. But very clearly, if you build that harness or you build that scaffolding around it and bring in the relevant integrations or the sources of information that a human engineer would have, it's very clear how that trajectory would get to the point where more and more of the tasks that a developer would do actually come under that.

Eno [00:05:02]: Yeah. And I think that at the time, as you mentioned, there was like baby AGI and a couple of these other concepts that had come out, which involved putting a while loop around the LLM, feeding back some context. And on the other hand, there were papers coming out, the chain of thought, self-reflection. Of course, the scaling law papers at this point had been somewhat established. And so there was kind of this clear direction where models were going to get better at reasoning. They were going to get better at having larger context windows. They were going to get cheaper at, or at least the Pareto frontier of the model capabilities was going to be expanding so that good models would get cheap. The best models might stay the same price, but they started to get really smart. And this was, I wouldn't say super obvious, but if you spent a lot of time just reading through these papers and working through, there was definitely a rumbling amongst most of the people in the community that that was going to continue to extend. And so you blend all of these together. And with kind of meeting somebody who has this kind of energy that clearly they want to build. And I think that it became really obvious that the opportunity was available. I also think that we made a lot of very solid progress on the initial demo enough to convince ourselves this was actually going to be possible. To be clear though, it was eight days from us first meeting to me dropping out of my PhD and Eno quitting his job. So there was analysis, but it was also just like, yeah, let's do it. Yeah. Pretty, pretty crazy. Like eight days. For sure. Yeah.

Alessio [00:06:32]: My first company was a hackathon project and I dropped out of school to actually found the company with one of my best friends. So the story resonates.

Swyx [00:06:39]: I think I'm doing hackathons wrong. Maybe. Like I've had, I met one girlfriend out of it. No, that was about it. Hey, that's a, you know, some people might say that's how you are. Is it still ongoing? No. I mean, it's a part of the funnel. Yeah.

Alessio [00:06:56]: So, yeah, maybe CodeGen was not the topic of the hackathon back then, but I would say today. I mean, there's a lot of the stuff that I've talked about in the past that I want to talk about. There's the CodeGen Router event that I go to. CodeGen is part of it. There's a lot of CodeGen products. Do you guys want to just talk about what Factory is and maybe just give a quick comparison on the different products that people might have heard about and that we can kind of dive deeper?

Matan [00:07:14]: Our focus is on building autonomous systems for the full end-to-end software development lifecycle and in particular for enterprises. I think especially given the, like, so, OK, obviously code generation is very exciting. A lot of, you know, the best engineers coming out of any of the, you know, the popular systems. It's like they want to work on like RL. They want to do cool things with like GPUs, you know, training models. Code is one of the like most obvious things because it's very easy to resonate with if you're an engineer. And so that's led to a lot of the players in the space really focusing on coding in particular and on like solo developers or building a quick like zero to one project and really like that use case that appeals to that profile. And I think something that we're focused on is the relatively underserved enterprise perspective, which is there are hundreds of thousands of developers who work on code bases that are 30 plus years old. It's really ugly. It's really hairy. It's really messy. And if you made a demo video like doing some like COBOL migration, that's not very sexy. You wouldn't go viral. You wouldn't get a lot of views because it's like it's just not that visually appealing, but the value that you can provide and how much you can improve those developers lives. It's very, very dramatic. And so seeing that underserved group is kind of why we've focused our sights there.

Eno [00:08:55]: I think this is a tool that was designed 20 plus years ago or has been iterated on for 20 plus years, primarily for a human being to write every line of code. When you take a tool like that and you start to introduce AI, you start to introduce additional constraints that exist just out of the nature of where you're interacting with these systems and where those systems live. So, for example, latency matters a lot when you're inside of an IDE. The cost when you are local first and your typical constraints. Consumer is on a free plan or a like $20 a month paid plan limits the amount of high quality inference you can do and the scale or volume of inference you can do per like outcome. When you are freed of a lot of these constraints, you can start to more fundamentally reimagine what a platform needs to look like in order to shift from a very collaborative workflow, which is what I think we see with most tools today, and a more delegative workflow where you are actually managing and delegating your tasks to AI systems. So, I think that the product experience of delegation is really, really immature right now. And most enterprises, though, see that as the holy grail, not like going 15% or 20% faster.

Swyx [00:10:06]: And you call them droids? We call them droids. Is there just a story behind the naming of either factory or droids?

Matan [00:10:13]: Yeah. So, we were initially incorporated as the San Francisco Droid Company. Really? We were. Is this before you have to bleep that out in the live podcast?

Swyx [00:10:23]: Sorry?

Matan [00:10:24]: Oh, you had to bleep that out? No, no, no, no. Okay. But our legal team advised us that Lucasfilm is particularly litigious and that we should change our name. And at the time, so, at the time while we were thinking of renaming, I was still in my PhD because we incorporated like two days after we met, which was also ridiculous. But we think of renaming and I was still in some ML class at Berkeley and I was reading a paper on actor critic. In there, there was some equation that was like, you know, some function of the actor. We're just calling that Y. And so, it was like F of A equals Y. A is actor. Put the actor in there and then it's, you know, factory. Yeah. And so, that's how it originally came about. It actually works quite well also, you know, automation, that sort of thing. But, yeah. Yeah. Yeah.

Eno [00:11:09]: And also the factory method, I think, was at some point we had that written up. And then I think that inspired this line of thinking. And droids kind of remained because we felt that there was a lot of hype at the time. Around the concept of agent. But it referred to such a specific thing that everybody saw, which was this like endless while loop unreliable system that just kind of went on and on and took a bunch of actions without guidance. Yeah. And our thought process was, well, that's not really what our system looks like at all. So, even though underneath it is an agentic system, do we need to say that we're an agent company? Doesn't really make sense.

Swyx [00:11:48]: I kind of like that. Like, yeah. Actually, last year, we pushed. Even though I put you guys. You spoke last year at the World's Fair. I put you on the agent's track, but I almost didn't have an agent's track because I was like, this is so well ill-defined. And I think that instinct is good. But now the agent's wave has kind of come back the other way. And now everyone's an agent. I think like defining your own term for it and just getting out of that debate, I think, is a positive. Would you like, is it closer to a workflow, which is like, I guess, the more commonly industry accepted term now? Yeah.

Eno [00:12:17]: No, that's a great question. I think that the original version of the droids were a lot closer to what we called workflow. So they were asynchronous and like event based, they would trigger and each one had a specific kind of deterministic, like semi deterministic. That was the original version. I think what we've grown to as the models have evolved and as our ability to build out guardrails and the system has just improved, we've gotten to the point where when you interact with droids inside the factory platform, they are relatively unbounded in the path that they take. And they're in general guided mainly by the system. And they're in general guided mainly by the concept of planning, decision making and environmental grounding. So they can stay loosely goal oriented over a long duration without needing very hard coded guardrails, but they still tend to hit their goal according to their original plan. So I think now agent actually is probably like the proper way to describe them. Sure. Yeah.

Swyx [00:13:13]: But you know, I think, I think droids have a nice ring to it.

Matan [00:13:15]: It's also funny. Our customers really, really love droids as a name. These are the droids you're looking for. I cannot tell. I cannot tell you how many times, like if, you know, with an enterprise customer, we'll do a POC and like, you know, a day later, you know, they're excited and like things go well. They'll share a screenshot and be like, these are the droids we're looking for.

Swyx [00:13:34]: And honestly, every time it's just, it's so fun. I know. And everyone thinks they're the first to make that joke. But it really is better than like agents or intern or like, you know, autonomous software. The number of human name AI products. Yeah. It's actually a pretty good insight.

Eno [00:13:50]: A hundred percent. And actually, I think we, we, to a certain extent, we're not going to be able to do that. To a certain extent, take a bit of an objection to the idea that these things are a replacement for a human being. I think that very much as we work through harder and harder problems with agents, it's become more clear that the outer loop of software development and what a software developer does, planning, talking with other human beings, interacting around what needs to get done is something that's going to continue to be very human driven. While the inner loop, the actual execution of writing lines of code or writing down the doc is probably. It's probably going to get fully delegated to agents very soon.

Alessio [00:14:25]: You just need to put Roger, Roger.

Eno [00:14:27]: Once they ask a question, when it finishes the task, we have that emoji in our Slack and use it very frequently.

Alessio [00:14:34]: Roger, Roger, do we want to do a quick demo?

Eno [00:14:37]: Yeah. Happy to jump in. When you land on the platform, you're presented with the opening dashboard. We try to make it really obvious that there are different droids available for key use cases that people tend to have. Of course you can always go and speak. With a default droid that can do a lot of things pretty well. But what we've learned is that there's three major use cases that people keep coming back to the platform for. The first is knowledge and technical writing. So that's going in and more of like a deep research style system that will go and do some research. It will use tools available to it, search, et cetera, and come back with either a high quality document or answers, and then you can back and forth. The code droid is really the one that's the daily driver for a lot of things. And this system allows you to actually delegate a task. I'll jump into that in a second. We can actually go through a full walkthrough and then the reliability droid. This was pretty surprising to us. The degree to which people love doing incident response, the kind of like SRE style work inside of the platform, I guess, in retrospect, it's nice because no one loves to be on call at 3am waking up being like what's happening and being able to just pass an incident description or say, Hey, something's going wrong and have a system really that can do that. It's really nice to be able to really compile all the evidence, write up an RCA, provide that for you is super high leverage. And so that's actually one of the more popular droids that people use, but I can start by just going into the code droid. And when you start a session with a droid, you're presented with this interface. It's a little different from typical where we see a lot of tools really want to focus you in on the code. Our perspective is that code is important to review when it's completed, but as the agent is working. What? What matters most is seeing what the agent is doing and having a bit of like an X-ray into its brain. So we have an activity log on the left and a context panel on the right. So you'll notice as we go through this task, that context panel starts to get updated. And I'm going to start by just doing something that's pretty common entry point. I'm going to paste a ticket into our platform. We have integrations with a bunch of different stuff, linear, JIRA, Slack, GitHub, Century, PagerDuty, you name it. We have a bunch of these integrations that are enterprise clients. We have a bunch of these integrations that our clients have wanted over time, such that you can easily pull this info in. And if I were to say something like, hey, can you help me with this ticket? And then I'm going to use my at command, which lets me easily reference code or code bases. Hey, can you help me with this ticket in factory mono? I like to be nice to them. I'd love for your help. And so you'll note that right off the bat, the droid starts working. It's doing a semantic search on part of my query in that code base. And actually, the system has access to a bunch of different tools. Here, memory, project management tools, GitHub, web search. Right now, the code droid only has search enabled by default. But you'll note that as the system starts working, it may actually want those additional tools added so that it can they can do its job.

Matan [00:17:37]: Maybe an important note there is, you know, as we deploy these droids in the enterprise, I think something that we're pretty like ideological about is everyone expects these, you know, agentic systems to perform at the level of a human right, because that's what they're always going to compare them to. But in a lot of cases, they'll have these agents just in the IDE. And that's like the equivalent of onboarding a human engineer, just like throwing them into your code base and being like, all right, like, go. But like, the reality is when you onboard a human engineer, what do you actually onboard them to? Slack, Notion, Linear, like Datadog, Sentry, PageReduce. They have all of these other information sources that they need to actually be a productive engineer. And yes, in theory, if you're like, really, really good, and you don't need contextual information, you could just work based on code. But like, that would be a lot harder. And it would probably be a lot harder. It would probably take a lot more time.

Eno [00:18:24]: 100%. And having those connections ends up being super important as it works through harder problems. Like in particular, you can see that the first thing it did after that search was reference some of the information that it found in saying, hey, this is what I found so far. It gives an initial crack at a plan, right? Presents that really clearly to you, and then goes to ask clarifying questions. So a lot of users, we believe, should not need to prompt engineer agents, right? If your time is being spent hyper-optimizing every line and question that you pass to one of these systems, you're going to have a bad time. And so a lot of what we do is to be able to format, if I say, help me with this ticket, right, there's clearly going to be some ambiguities. The system knows when you give a very detailed answer or request to follow your instructions, when you give more ambiguous requests to ask for clarification. This is actually a really tricky thing to get right in the model, but we spend a lot of time thinking about it. And so I'm just going to answer some of these questions. Are there any UI mock-ups? No. Try to imitate the other examples. Two, only preview when the button is clicked. Three, no, it's actually not implemented. Four, which specific fields must be displayed? Your choice. And five, your choice. So now I'm basically saying to it, you decide for some, giving my preferences on others.

Matan [00:19:52]: And this is really... This is really the balance of like, even, you know, delegation that will happen to non-AI is like, you know, as a good manager, you give autonomy to people that work with you when needed. But also if you're like, hey, I'm a little worried about this, or I'm going to be really strict about what I expect here, you want to extract that behavior as well. Because a lot of times, like Eno mentioned, if you give a really poor prompt and you just say, hey, go do it, it's going to go do it, but it'll probably make assumptions. And then at the end, you might not be happy, but that's just because there were some constraints in your head that you didn't actually explicitly make. So I think it's really important that you mention when you're communicating. So, yep, 100%.

Alessio [00:20:26]: Do you guys have like a template that you've seen work? When I'm boarded to Devon, for example, they have the fix prompt button, and then it refills it in their template, which is like, give the agent instruction on how to debug, give agent instruction on how to do this, and ask you to fill out these things. Do you guys have a similar thing where like you think for each project, these are like the questions that matter? Or is that more dynamic?

Eno [00:20:51]: No, that's a great question. And it's something that... We talk about a lot internally is it's surprising how many people are building products that have reactive like information requests, like, you know, please fill out this form to explain how to do this thing, or you need to set up this dev environment yourself manually in order for this to work. We think of trying to be proactive with a lot of this stuff. So you'll notice in the right hand corner, there's this project overview, right? And the system started to code after doing some search. So that's going to pop up while we do this. But when I click into this project overview, what you're going to see is basically a, and I'm hiding it because I'm realizing this is actually semi-sensitive. It's probably fine. We can hide that in the video. Yeah, no worries. It's totally fine for folks to see that it's a monorepo. If I scroll down, that's when we'd get in a little bit of trouble. But inside that project overview, we're actually synthesizing a bunch of what we call synthetic insights on top of the code base. And that is looking at things like how to set up your dev. What is the structure of the code base? How do important modules connect to each other? And as we index code bases, we're actually generating these insights at a much more granular level across the entire code base. We think that in general, systems should be proactive in finding that information. However, with features like memory, and we have like a .droid.yaml, you can set a bit of your guidelines. However, we also feel that it's like that XKCD about standards. A new standard. Right. Like everyone's got like a. .blank rules file. And so we ingest those automatically from all of the popular providers as well. Wow.

Alessio [00:22:28]: Okay. Does something like a cursor rules is complimentary because people might take this and then work on it in cursor separately? Yeah.

Eno [00:22:36]: What we found is that there are sometimes extraneous advice in those because people need to give a lot more guidance to those types of tools than they do to ours. So our system parses through and only picks the things that we don't already know.

Matan [00:22:51]: Another thing. That kind of comes to mind related to your question, and this is something we've been thinking about a lot as well, is as we have more and more enterprise customers and a lot of the developers in the enterprise are not going to be as up to date on every new model and how it changes its behavior. Something that's interesting that we're thinking about is these developers are getting familiar with factory and how to get the most out of it. And then we let's say when we upgraded from Sonnet 3.5 to 3.7, we suddenly had a lot of developers being like, hey, wait, it now does this less or does this more. What's happening? Or when they go to Gemini, let's say, and they want longer context. And so something that I think is interesting is how much of the behavior difference from the models should we act as like a shock absorber for so that they can basically, as a user, use it exactly how they've been using it before and get the same sort of output. But then also how much of that do we actually want to translate to the user? Because presumably over the next three years, the way you interact with models will change and it's not just going to be up to behavior, but it's rather like, I guess it's alpha versus. Like beta in the model. Like there's some models have like different personalities and it's just the way you prompted to get the same out of it. And then there are others where it's like, I mean, for example, like the reasoning models, they just work in a fundamentally different way. And so you as the user should know how to interact differently. So that's something that's kind of fun to wrestle with.

Eno [00:24:10]: How do you evaluate the new models? We listened a lot to how the model providers actually think about building out their eval suites and in particular trying to look at things like desired behavior versus actual. And in a way that's sustainable for a small team, we don't have like, you know, a hundred million dollars to pay data providers. And so a lot of the evaluation ends up being a combination of point like task based evals. So like the AIDER has an awesome benchmark that we built on top of internally for code editing and file generation versus we also for the top level agent loop have like our own behavioral spec where we set a bunch of high level principles. We break those down into tasks, those tasks then have grades and rubrics, and then we try to run those in order to determine is the behavior suite that we like, for example, asking questions when it's ambiguous versus not asking questions, does that match up? And we also use that to optimize the prompts as well.

Swyx [00:25:09]: Just a quick question on these types of things. I think every company should have their own internal evals, right? That is not in question. And obviously that is your IP, so we can't know too much about it. But like, what is the right amount to do? What is the right amount to spend on something like this? Yeah. Because let's say we talk about SweetBench before recording, SweetBench costs like 8,000 to run. I've heard varying numbers between 8 to 15,000 to run. Yeah. That's high, but you should be able to spend some amount to ensure that your system as a whole works and doesn't regress.

Matan [00:25:38]: So like, what's a rule of thumb for like, what is the right amount to spend on this? Yeah. I mean, I think it's important to separate out the two purposes of benchmarks, which one is marketing. And like, there are so many customers that we. We have that was purely because they saw the charts and they saw big bar versus little bar and they were like, okay, we want to go with big bar, which is funny, but that's just, I mean, that's just the way things go. And so I think that motivates, I think that's actually a good thing because that motivates more resources to be put on, you know, benchmarking and evaluation. On the other hand, you know, there definitely is a risk of going like too far in that direction or like even go getting to the point where you're fine tuning just to, you know, satisfy some, some benchmark there. And so.

Swyx [00:26:20]: Like we were saying before. Yeah. Like you guys don't bother competing on sweet bench anymore because it does not, not that relevant.

Matan [00:26:26]: Yeah, that and also like just in the enterprise, the use cases are pretty different than those represented in something like sweet bench. So we do have pretty rigorous internal, internal benchmarks as well. But I think also there's a certain extent to which like the, the vibe based or the sentiment based internally actually matters a lot because who has like a more intimate understanding of the behaviors of these models than the people who work on it. Yeah. We kind of have to be able to be able to be able to be able to begin to, you know, facilitate every single day, like working with them and building with them because I mean, we use factory internally every single day. And so when we switch a model, we very quickly get a sense of how things are changing. Definitely.

Eno [00:27:00]: And I think that those task-based evals tend to be the ones where it's most critical that we hill climb continuously on versus the, the top level evals. They change so much with the new model providers that we try to make sure that they have some degree of consistent behavior, that the feel is smart, but the top level agent is actually the one who's able to do the work that they need to do. not that responsible for what most people call quality. That ends up being, is it fast, accurate, and high quality code edits? Does it call tools with the right parameters? Is the tool design such that that model can easily fit into it? And we have noticed a lot of really interesting behaviors with as the new models that have a lot heavier RL on post-training related to their own internal agentic tools. So for example, Sonnet 3.7 clearly has, it smells like clod code, right? Same with Codex. It very much impacted the way that those models want to write and edit code such that they seem to have a personality that wants to be in a CLI-based tool. What's interesting is how do we combat the preferences that RL brings into the product? For example, search with CLI. CLI is like grep and glob. But what if you gave it a search tool that was way better than grep or glob at finding precisely what you wanted, but the model just really loves to use grep? They're going to fight each other. And so our evals have to figure out how do we make sure that as we build tools that are better than what maybe the model providers have in their slightly more toy examples, that the models use those with their full extent. And that's actually been a very interesting novel challenge to us that only started happening in the last three to six months as these new models have come out.

Alessio [00:28:47]: Does that make you want to do more reinforcement fine tuning on these models? Like kind of take more of that matter into your own hands?

Eno [00:28:55]: I definitely think that it's an interesting idea, but our take in general is that freezing the model at a specific quality level and freezing the model at a specific data set just feels like it's lower leverage than continuing to iterate on all these external systems. And it also feels like this is a bug. Like we spoke with a bunch of the research labs, and I don't think that they actually want this type of behavior. What it is ultimately is it's a reduction in generalization.

Swyx [00:29:23]: Cool. Anything else to see on the demos? Oh, yeah. I mean, we can... It's still coding.

Eno [00:29:28]: Yeah, yeah. So you can see here that we're running... Oh, because you gave it like a whole bunch of things. Yeah. So I actually gave it like quite a large project to do to execute live in front of us. Got to earn its keep. Yeah. Yeah. This is why this delegation style flow, we see as like really different, where in general, we expect the answer or output of this to just be correct, right? It's running code. It's iterating on code. It's making edits to a bunch of different files. It's going to have to run pre-commit hooks and test all this stuff. I think that this is a big difference in workflow, right? Where we've just had a podcast conversation. Meanwhile, the agent is working on my behalf. This is probably going to be mergeable at the end of this. It's ideally going to create a pull request and we can check in on it. But I think that this difference is like, what would I be doing right now? I think today, a lot of people just like open up their phone maybe and start browsing or they go context switch to a different task. But the real power is unlocked when you start to realize this is the main thing that I'm going to be doing is only delegating these types of tasks. And so you start jumping to, okay, while this is happening, let me go and kick off another task and another one and another one. And so you start jumping to, okay, while this is happening, let me go and kick off another task and another one and another one. And so being cloud native, being able to parallelize these, like I'm only sharing one tab, but if I just open another one and started right now, we support that natively. I think that this feels a little bit more like how people are going to work, where you maybe start the day setting off a bunch of tasks in motion, and then you spend the rest of it on maybe harder intellectual labor, like thinking about which of these is actually highest priority to execute on.

Matan [00:31:07]: And this actually goes into something that Ina was mentioning a little bit before, but also like a question that I'm sure everyone, when they see this is going to ask, which is, why is this browser based? Why is this not in the IDE? Like I'm used to coding in the IDE. And the kind of higher level answer here is that, and Ina was alluding to this before, like the last 20 years, the IDE was built for this world where developers are writing every single line of code. And something I think everyone can agree on is that over the next few years, what it means to be a software developer is going to change dramatically. Now, some people disagree, and some people say there will be no more software engineers. Some people say there will be, everyone's going to be a software engineer and everywhere, in between. But the reality is very clearly in the next few years, the amount of lines of code written by a human will go down, like the percentage of code written by humans will go down. And our take is that it is very unlikely that the optimal UI or the optimal interaction pattern for this new software development where humans spend much less time writing code, I think it's very unlikely that that optimal interaction pattern will be found by iterating from the optimal pattern when you wrote 100% of your code, which was the IDE. Internally, we talk a lot about the Henry Ford quote, which is, you know, if you ask people what they want, they would say faster horses. And for us, the analogy here is like, can you iterate your way from a horse to a car? And there's like this very grotesque, like ship of Theseus, you can imagine of like trying to turn a horse into a car. It doesn't really look pretty. And our take is, you know, even though the world was built for horses at a certain point in time, right? Like there were stables everywhere throughout a city, you were used to feeding this thing and, you know, taking it with you everywhere. And it is kind of a higher barrier to entry to start introducing this new means of transportation in this analogy. We are taking that more ambitious angle of like, everything is going to change about software development. And in order to find that optimal way of doing it, you do need to think from scratch, think from first principles about what does that new way to develop look like? And to give some early answers that we are pretty clear about is the time developers spend writing code is going to go way down. But in turn, the time that they spend understanding and planning is going to go way up. And then also the time that they spend testing so that they can verify that these agents that they delegated to did indeed do the task correctly, that's going to go way up. The promise of test driven development is going to finally be delivered with this world of AI agents that are working on software development. Because now, if you do want to delegate something like this, while you're doing a podcast and come back later, ideally, you don't even need to check their work and you just merge the PR. But how do you do that with conference? You need to be really sure that the tests that you put up and said, hey, you know, Droid, you're not going to be done until you pass all of these tests. If you wrote those tests well, then you can. All right, great. Pass the test. Let's merge it. I don't even need to

Swyx [00:33:51]: go in and see how it did everything. I mean, sometimes you do have to break the tests because you're changing functionality. Yeah. Yeah. There's a whole bunch of hard problems, but I just wanted to cap off the sort of visual component of the thing. There's one thing you haven't shown, which is like there's like a built in browser. So like I have a Next.js project here that I'm running. Yeah. The conference website, it's spun it up. When I tried it out in ChatGPT Codex, it didn't work out of the box and they didn't have a browser

Eno [00:34:16]: built in. So it's nice that you have that kind of stuff. No, for sure. Like being able to view like HTML, SVG, et cetera, on demand is super nice. And I think it's pretty much wrapped up. It actually finished these changes. I think it's like roughly 12 files that it edited, created. And so, you know, right after this, because of the GitHub tool, I would just say, go ahead and

Swyx [00:34:36]: create a pull request. Okay. And then it'd be good. Amazing. Yeah. Good stuff. And you even, like a little 43% of context size use. That's actually not that much given that

Eno [00:34:44]: this is factory's own code base. Yeah. And this is actually a large mono repo. I think the big thing that I'd love for people to try out is like, look at how efficient it is. It's able to really execute on precisely what it needs to edit with a relatively lower token usage than other agentic tools. Obviously, if you're just getting autocomplete, that's going to be a little bit more expensive, but compared to other agents where you get like five credits, it takes a while to execute on anything. I think they'll see a better experience with factory.

Swyx [00:35:15]: Yeah. When you started saying things like, oh, we can pull in from Notion, we can pull it from Slack, like that sounded like a lot of context. You're going to have to do pretty efficient rag to do

Matan [00:35:26]: this, right? Like, I guess it's not even rag, it's just retrieval. Yeah. Yeah. I mean, but there is the temptation. And I remember maybe a year ago, there was really a lot of hype on large context because, right, it's the dream of you can be super lazy and just throw in your whole code base, throw in everything at it. Which is what Cloud Core does, right? Right. Yeah, exactly. But I think the one downside of that is, okay, great. Like, if you do have billion token context window model, you throw it all in there, it's still going to be more expensive. The reason why retrieval is so important for us is because even if there is a model that's going to have these larger context windows, and certainly over time, we're going to get larger context windows, you still want to be very cost efficient. And this is something that our customers care a lot about. And they see a lot of the value in the effort that we put in on retrieval, because they'll see like, wait, this was a huge mono repo, and I gave it all this information. But then I see for each actual call, you're really good at figuring out what do I actually need, as opposed to just like

Alessio [00:36:17]: throw the whole repo in and pray that it works. You mentioned the credits, what's the pricing

Matan [00:36:21]: model of the product? We're fully usage based. So the tokens, I think for us, it's really important to like, respect the users and their ability to understand what what this stuff means. And so I think like all the stuff around like credits, and it just kind of obfuscates what's actually happening under the hood. And so I think it's really important to understand what's happening under the hood. And I actually think that we get better users, the more they understand what tokens are and how they're used, you know, in each back and forth.

Eno [00:36:46]: Yeah, so it's a direct bill through to the, we call them standard tokens, and it's benchmarked off of the standard models that we have. So like right now, when you get access to the platform, your team would pay like a small fixed price for just access. Every additional user is another like very small fixed price. And then the vast majority of the spend would be on usage, of the system. And so I think that this is just nicely aligned where you get a sense of how efficient it is about the token usage. This is a big reason why we've tried really hard to make it more token efficient. And then you can track, of course, in the platform, how you're using it. And I think that a lot of people like to not only see just like raw usage, and this kind of gets into like tracking success, something that a lot of people do by maybe like number of tabs that you accepted or chat sessions that ended with code. For us, we try to like track the number of tabs that you accepted. And then we try to look a little bit further and say, look, you use this many tokens, but, you know, here are the deliverables that you got. Here are the pull requests created. Here's merged code. We help enterprise users look at things like code churn, which it turns out the more AI generated code you have, if the, you know, if the platform isn't telling you the code churn, there's a reason for that. Code churn meaning amount of code deleted versus added? Yeah, it's basically a metric that

Matan [00:38:02]: tracks when you, it's kind of like a variability in a given line of code. It's very imperfect like some people, some people will say like the difference between code churn and like refactored code is like somewhat arbitrary because it's a time period at which, okay, if I merged some line and then I changed that line, if you change that line in a shorter period, it'll churn versus a longer

Eno [00:38:25]: period. It'll count as like refactoring, which is like a little. So generally in enterprise code bases, if you merge a line of code and then change that code within three weeks, it's because something was wrong with that code. Generally, it's not always true, but it's not always true.

Matan [00:38:38]: But it's a useful metric. It averages out because sometimes it's like, wait, what if you just had like an improvement or, you know, some, some change that wasn't about quality, but like is

Eno [00:38:47]: code churn up bad? Yes. Because what it tends to be is that in a, in very high quality code bases, you'll see 3%, 4% code churn when they're at scale, right? This is like millions of lines of code in poor code bases or poorly maintained code bases or early stage companies that are just changing a lot of code. So it's like, okay, I'm going to change this. I'm going to change this. I'm going to change this a lot at once. You'll see numbers like 10 or 20%. Now, if you're at Lassian and you have 10% code churn, that's a huge, huge problem because that means that you're just wasting so much time. If you're an early stage startup, code churn is less important. This is why we don't really like report that to every team, just enterprises.

Swyx [00:39:27]: Any other like measurements are popular that, I mean, you know, this is nice that I'm hearing about code churn, but like what else are enterprise VPs, events, CTOs, what do they look at?

Matan [00:39:38]: I think the biggest thing, because there's so many tricks and different dances you can do to like justify ROI. Number of commits, lines of code, DORA metrics are usually popular. And at the end of the day, like what we, we initially went really hard in all the metric stuff. What we found is that oftentimes if they liked it, they wouldn't care. And if they didn't like it, they wouldn't care. And so at the reality, like at the end of the day, no one really cares about the metrics. What people really care about is like developer sentiment. When you're kind of playing that game at the end of the day, if you want to do a metric, you're going to have to do a lot of work. You're going to have to do a lot of work. You're going to have to talk to developers and ask if they feel more productive, or if you're a large enterprise and you want to justify ROI, the biggest thing that we've seen and like, that's allowed us to deploy very quickly in enterprises is pulling in timelines on things. So there's this one very large public company that we work with and pulling in just a large migration task from taking four months to taking like three and a half days. That is the best ROI there. You don't need to measure this or that, like, like we had something that was going to be delivered in the next quarter and we could get done this week with no downtime. Like that is, you know, music to a VP of engineering's ears. And so that's what we tend to focus on is like pulling in deliverables or increasing the

Swyx [00:40:48]: scope of what you can get done in a quarter. In order to achieve a very large refactor, like you just described, do we use that same process you just saw or is there more setup?

Eno [00:40:57]: I think that the workflow for, let's say, a migration is probably one of the most common. I can even give a very concrete example. Let's say you are the administrative service of a large, European nation, right? Like Germany or Italy. And you have a hospital system that runs on like a 20-year-old Java code base. Now, a company wants to come in and big four consulting firm or something like that says, we would like to transform this entire code base to Java 21. It's going to take X amount of time, a couple months. And, you know, by the end, you'll be on a relational database. You'll be into the future, right? On Java 21. When that typically happens, you kind of have to almost break down what that means from a, like, human perspective first, to then map it to how it works on our platform. You'll have a team of anywhere from four to 10 people come in and you have a project manager who is going to work with engineers to analyze the code bases, figure out all the dependencies, map that out into docs, right? First analysis and overview of the code base. The next is a migration strategy and plan. The third is like timelines. And you're going to scope all this out. What do you do next? Well, you go to a project management tool like Jira. And so you take those documents and a human being translates that out. We've got two epics over the next two months. This epic will have these tickets. That epic will have these tickets. And then you find out the dependencies and you map those out to humans. Now, each of these humans are now operating such that one after the other, they're knocking out their work, mainly in parallel, but occasionally pieces have to connect, right? One person misses, and now the whole project gets delayed about a week. And so this interplay of, of understanding, planning, executing on the migration incrementally, and then ultimately completing. Now there's a handoff period. There's docs of the new artifacts that we've created. There's all this information. You map that over to a system like ours. One human being can say, please analyze this entire code base and generate documentation, right? And that's one pass, one session in our platform. Analyze each of the modules. We already do a lot of this behind the scenes, which makes this a lot easier. And then you have the next step, which is to go to the and actually generate an overview of what the current state is. You can now pull those docs in with real code and then say, what's the migration plan, right? If there's some specific system, you can pull in docs. And then when you have this, our system connects with linear Jira. It can create tickets, just create the epic ticket, this whole process out and figure out which are dependencies and which can be executed in parallel. Now you just open up an engineer in every browser tab, right? And you execute all of those tasks at the same time, and you as a human being just review the code changes, right? This looks good. Merge. Did it pass CI? Okay, great. Onto the next one. This looks good. Merge. So a process that typically gets ultimately bottlenecked, not by like skilled humans writing lines of code, but by bureaucracy and technical complexity and understanding now gets condensed into basically how fast can a human being delegate the tasks appropriately. So it happens outside of like one session, like what we just saw, which would be like a couple of sessions. And then you have a process that's one of those tasks, but the planning phase, that's really where we see enormous condensation of time.

Swyx [00:44:14]: We just talked about your pricing, your, your just usage base, but you tempted to have forward deployed engineers that just like the current meme right now to execute these large

Matan [00:44:23]: things. Yeah. So I think this is something we definitely do a little bit of for our larger customers, just because, you know, like we said at the beginning, this is the way we think software development will look. And it's an entirely new behavior pattern. And I think it would be, a little naive to just be like, Hey, we have this new way of doing things, go figure it out yourselves. Right. So we definitely go in and, you know, help show them how to do this. So like in this migration example that I was mentioning before, we worked with them like side by side, but just us and like two of their engineers showed them how to do it. They were like, they saw the light, if you will. And then they ended up being the internal influencers within their org teaching everyone else how to do it. But if you want to change behavior, you can't just assume that the product is going to be so good that everyone's going to, you know, immediately get it. Because with developers, you know, we need to know who we're selling to and developers have very efficient ways of working that they've built out over the last 20 years. And we want to make sure that we accommodate that and earn their trust and slowly bring them into this new way of building. And to do that, we need to extend that olive branch and, you know,

Alessio [00:45:25]: come meet them where they are, show them how they can do new things. We did an episode with Together AI maybe a year ago or so. And we were talking about what inference speed actually we needed. And they always say, you know, we need to know how to do it. And we're like, you know, we always argued we need to get to like 5,000 tokens a second. And we're chatting whether or not that makes sense because people cannot really read it. As you think about Factory, how much do you think you're like bound by the speed of these models? Do you know, like if the models were like a lot faster, would you just complete things quicker? Would you like maybe fan out more

Matan [00:45:57]: in parallel? Like what are like the limits of the models today? I want to let you know, answer this, but immediately every time this thing comes up, I always just think about the memory that Chrome tabs take, which is like, you know, I don't know, I don't know, I don't know, I don't know, I don't know, like it's never enough and you always want more, but then it's also, it lets you be lazier.

Eno [00:46:12]: Yeah. But anyway, I want to. Yeah, no, for sure. And I think that the, this is kind of a funny question. It kind of has two directions. It's like practically, would this make a big difference on someone who knows and loves and uses our platform on a daily basis? I think it would probably improve the quality of life such that, yeah, definitely faster tokens would be awesome. I think that actually where this impacts is for those who haven't yet made the jump from collaboration to delegation. So if you are used to very high latency, high feedback experiences, then that speed difference and seeing like most of that delegation happened very quickly and then being able to immediately jump in, I think feels very nice. And so for the larger enterprise deployments where they start to familiarize themselves with how this works and the migrations, I don't think this actually makes a big difference because most of the bottleneck ends up being, like I mentioned, almost like bureaucratic. In nature, but for the average developer, I think this improves the user experience to the point

Matan [00:47:12]: where it would feel very magical. So I think we could get a lot faster. It probably wouldn't change what's possible, but it would really, really change like ease of adoption for people

Eno [00:47:23]: who maybe aren't as in the weeds on AI tools. And if you combine the latency with a cost reduction as well, I do think that cost is one of the reasons why we haven't so greatly scaled out. Like originally we had a lot of techniques that would generate, a lot of stuff in parallel, and we still know how to do that. And we're very excited to, to bring that. But right now we don't do it because it's cost prohibitive and the quality Delta is not

Swyx [00:47:48]: enough to justify the cost increase. I have a kind of a closing question. If you're, if you don't mind, this is a more or less, you're asking you on like a limiting factor. It's basically four questions in one. So what do you see as your limiting factor right now in terms of models, basically like what capabilities would really help you? Um, hiring, what skills are really hard to hire customers? Like what, you know, do you really want to unlock that? You're like, it's like weirdly not working, but like you have an ICP, which is a more enterprise doing well, but like, you know, what, what's like the next one. And then like finally dev tooling, like what do you wish existed that you had to build for yourself or you just feel

Eno [00:48:25]: like could be a lot better. Yeah. I'll maybe I'll do models and dev tool and you can take hiring and, uh, and customers. I mean, I'm thinking right off the bat, probably the biggest thing is models that have been post-trained on more general agentic trajectories over very long timespans. That feels like something that is, there's an effort for that right now. But what I mean is an hour or two hours or three hours of seriously working on a hard problem such that the model knows how to keep that long-term goal-directed behavior the whole time. That is something that I assume we'll get. And I think that's something that we'll get. I think that's something that we'll get.

Swyx [00:49:07]: Yeah. OpenAI has put out that like a operator benchmark. Yeah. That was that they had a human

Eno [00:49:12]: testers, like actually try for two hours and give up. Yeah. Did you see that one? Yeah. Yeah. I mean, I think that that's exactly the type of work that we want to see taken further because that I would

Swyx [00:49:22]: argue is probably one of the bigger blockers. I would say like, would you ever do that yourself? I don't see you guys as customizing your own models a lot, but you work with the frontier labs,

Eno [00:49:33]: right? But like, is there a point where you would just be like, Hey, screw it. Like we'll do it. We are currently building benchmarks with a lot of the post-training techniques very much in mind right now. And so I don't know what exactly at this point in time, how much we're going to commit to that, but for sure we will be using those benchmarks for our own internal goals. And if we need to use them later on for post-training, I think that there's a lot of compatibility. And then maybe for dev tools, since I'll just, I'll just jump that one. Yeah. It is still surprising to me that, uh, uh, observability, it remains like very challenging.

Swyx [00:50:11]: Really? There are like 80 tools out there.

Eno [00:50:12]: No, I, for sure. And Langsmith is actually fantastic. Like we use Langsmith. Are you guys a Langchain shop? Uh, we don't use Langchain, but we just use Langsmith. Um, and, and Langsmith is awesome.

Swyx [00:50:21]: See the hackathon is his ROI for Harrison.

Eno [00:50:24]: No, like they've been fantastic and that's been cool. But I think that there is, it's really tricky to deal with enterprise customers where you can't see their code data at all, but you're trying to build a product where you can do it. You can improve the experience. And a lot of it is actually subjective. It's like, I don't like the way this code looks. That remains something very unclear to us is how do you build almost like semantic observability into your product? Uh, I think amplitude and, uh, and stat SIG and a lot of the feature flag companies actually are closer to this than the existing tools.

Swyx [00:50:57]: Like product analytics, actually, really, it's more about like observe if they still,

Eno [00:51:00]: if they're on the platform, like basically anything other than up and down, uh, thumbs, right? Yeah. And like, and like, what was the user's intent when they entered into this session, right? It's the type of thing where you almost need LLMs in the observability itself.

Swyx [00:51:13]: Are you saying they've actually done it or amplitude could do it?

Eno [00:51:16]: That's where I would like to see it.

Swyx [00:51:18]: Yeah. As far as I understand, it hasn't really, yeah.

Alessio [00:51:20]: Yeah. Because mostly everything is kind of like span based. They're like all the observability products are, we started trace, but not at the, yeah, exactly. Not, not in the like semantic direction. Yeah. Yeah. So for you, that's kind of solved in a way. It's like the actual traces is like, you can get that information anywhere.

Eno [00:51:38]: Yeah. Our team comes from like Uber and all these amazing places where they, they know how to do that part. I think the more tricky thing is when human beings have like messy intents that are natural language, how do you really classify and understand when users are having a good time versus when they're having a bad time? Okay. That's hard.

Swyx [00:51:56]: Great. Hiring and customers.

Matan [00:51:58]: Yeah. So maybe I'll start with customers. So we've been at it for just over two years now. Yeah. So I think the first like year and a half was really, really focusing in on the product and what is the, the interaction pattern that works for the enterprise. And over the last 90 days, the deployments that we've had with like the large enterprise and fortune 500 has been exploding. It's been going really well. It's very exciting. How are they mostly finding you? So this is a good point. This is part of why we're doing more podcasts is because so far we really just relied on word of mouth, like working well with one enterprise. And then, you know, they're at some CEO dinner, they mentioned it to someone else, which by the way, like, that's why we have the conference to like, we put all the C the VPs in one room. Totally. And so like, it's worked really well, but I think when every one of those conversations ends up leading to a happy customer, that means you need to increase top of funnel. And so accordingly, we're really putting fuel on the fire for our go to market for, you know, fortune 500 large enterprises, which is obviously it's a very exciting thing to do where the team has been pumped. I mean, there was a particular day in January of this year where one of those large enterprises basically had the magic moment of like, if I was the only one at my company using this, I would still tell them to like, have me use this instead of hiring like three engineers for myself. One of the biggest moments for us where it was like, people in the enterprise are really getting dramatic value out of factory. And that kind of kicked off this really like the last 90 days has just been a whirlwind. So getting to more of these fortune 500 companies is kind of top of mind for us right now, to that end. You know, as you serve. Fortune 500 customers, it becomes important to have a larger go to market team, both on the sales side, the customer success side, and then also, of course, on the engineering side. So we are very much hiring.

Swyx [00:53:44]: Yeah, I think like everyone's hiring. It's just like, what are you finding that's hard to hire?

Matan [00:53:48]: Oh, I see. Like the particular roles? Yeah. Yeah. So what's the rate limiter here? I think a big rate limiter is like, for us going to these fortune 500 companies, one of the most important things is having both that ability to. To, you know, talk to the CIO, VP of engineering and have that sales presence, but then also the ability to be sitting side by side with some of their developers and jumping into the platform, jumping into their use cases and having that. So you need like a hundred, you know, basically, like literally our profile, actually our profile when we're like looking for this role is like, is this a junior Eno or not? Like that's,

Eno [00:54:23]: that's basically the, the, the template there. Yeah. I don't know about that, but I definitely think that if you are highly, highly technical, that you want to be a founder, you want to move into a role where you are interfacing with CIOs, CTOs, like this, we have like three maybe of these roles that are probably going to be the most important like roles in our go-to-market team. And so I think that's a huge opportunity for anyone interested in what we've talked about.

Matan [00:54:52]: We joke that this person would basically be my best friend because any trip we go to, to fly to a customer, they'd be there with me, you know, talking to whoever is the buyer, as well as, you know, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the啤l-in-lắ� , so they said like, oh, and we're all like the so I'm also I guess hiring a best friend.

Alessio [00:55:06]: thought that's what AI was going to be for . Just to wrap, I think we're all fans of your guys' design and kind of like brand Vi- Speaking of best friends. Who does your thing?

Matan [00:55:19]: Yeah. So a huge privilege of, you know, Faktor has been working with my older brother, Cal who who joined us, he, you know, moved from, he was in New York for five years. He moved He actually, even before he moved, he's the one who designed our logo way back when. And, you know, he's been a part of Factory from the very beginning. And it's been an absolute pleasure working with him from the brand design, the marketing design, and then, of course, the product and the platform itself. I cannot recommend enough working with a sibling.

Swyx [00:55:49]: Sure. I mean, you know, not all of us are lucky to have that. What do you learn from working with a designer like that? I think a lot of technical people listen to us, want to build a startup, but they don't have the polish that you have. They don't have the hype.

Matan [00:56:03]: Yeah, I think a big part of this is like one of our core operating principles is like embracing perspectives. And so Cal is not an engineer. And I think what's great is a majority of our team are engineers. And having that ability to come in with that design perspective, and then also the engineering perspective. And like, bash those two things together until we get something perfect out of it. That has been really, really important. I think it's a lot of times, it's kind of easy to fall victim to, oh, I'm building, like, I'm the profile who I'm building for. So I know what's best. And that obviously works a lot of the time. But sometimes there are some like core design tenets that you just might not think of if you're building for yourself. So I think that's been, that's been pretty important there.

Eno [00:56:42]: Yeah, and we do live in a very AI native company or operate in a very AI native company. So being able to have some of those set principles that are then consumable by our own agents, right design systems, and consistency, I think it's pretty surprising the degree to which even like droids can actually imitate a brand voice and style that Cal created for us. And so a lot of that comes from not just the droids doing that, from our entire team of product engineers are all incredibly thoughtful about what they're putting in front of users. And I think that they're able to bring a lot of that into

Swyx [00:57:22]: And I think that they're able to bring a lot of that into it in a way that feels like safe and on brand.

Matan [00:57:49]: Yeah, yeah, I think it's important for us because not only is this, you know, incredibly transformational, but it's also like, these are people that we spend all of our time with. And we want to make sure that while we're doing it,

Swyx [00:57:57]: it's like right next to the Caltrain, you can like advertise out of your window.

Matan [00:58:02]: Yeah, no one, no one peek in though. There's a lot of a lot of secrets in there.

Swyx [00:58:07]: It's pretty sweet. Cool. I mean, you know, I'm very excited for your talk. You know, like we touched on a few things I'm interested in, right? Like we're seeing tiny teams is a topic that I'm seeing, like, you know, one person can do a lot more. So like, the average team size is really shrinking. The interaction of AI design and engineers. That's also another thing I'm exploring. Like, I think we're really trying to like push the frontier. And then obviously, there's always like the Swyx and stuff, which is always ongoing. So like, yeah, there's a lot of interesting work going on.

Matan [00:58:34]: And one interesting addendum there, there are sometimes individuals who weren't even really developers, who will use factory and have more usage than in 100 person enterprise. Yeah. Which is crazy to see. Yeah. Yeah. Yeah. Yeah. Yeah. Just in how people use these tools, whether it's like for that design or for that, like small, small team use case. It's pretty fascinating.

Swyx [00:58:55]: There's an AI native attitude that like is going to set people apart if they're just open to it. But then also like they're maybe not, they don't drink too much Kool-Aid. I think there's a medium there.

Alessio [00:59:05]: Thank you guys for coming on. This was fun. Thanks for having us.

Matan [00:59:08]: Thank you guys for having us. This was awesome.

btw Gamma, Bolt, Oleve and more are presenting on the new Tiny Teams track, and leads of both OAI Codex and Google’s Jules will be presenting at AI Engineer World’s Fair! You know the drill by now - the AIE expo and all 3 conf hotels are now sold out! Join the Hackathon and fill out the State of AI Engineering Survey!

The recent launches of the Codex and Jules Devin-like autonomous coding agents came with some interesting pricing decisions: they come at no extra cost1. You no longer have a reason NOT to send off every thought to the agents (even more than once) even if you intend to eventually code it yourself.

Let me repeat: YOU CAN FIRE OFF A CODEX REQUEST EVERY 60 SECONDS AND THEY DON’T CHARGE YOU EXTRA. YOU SHOULD USE THIS!!!

This comes after a 3 year trend of exploring the SWE Agent pricing spectrum:

Variously:

• ChatGPT Codex

• Augment Code

• Claude Code + Max

• Google Jules

• Cognition Devin 2.0

• Cursor

• Bolt

• Codeium / Windsurf

And of course there are a lots of premium and open source coding agents with Bring Your Own Key and reasonably metered usage like Github Copilot Coding Agent and the newer Amp Code.

Side note: A less rigorous publication than the one you’re reading could observe correlation of the rise of coding agents with the infamous Indeed software developer job postings chart and conclude causation:

without normalizing for the macro base rate and the shift in job definitions.

However our sense is that, especially with the frontier labs launching effectively unlimited usage plans, the competitive war in coding agents has come to the point where the norm of charging a premium over token usage as a coding agent GPT wrapper has now FLIPPED to offering discounts in order to get your usage data.

That’s insane, you say. Lighting VC money on fire at negative margins is unsustainable!

And yet Perplexity with -200% profit is worth $14B and Cursor is worth $9B. How can you not realize that there is value in user data and acquisition when OpenAI pays $3B for Windsurf at 75x ARR?

Latent.Space is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

The CodeGen, and now SWE Agent2, market has thus progressed3 as such:

• First, charge a small flat fee for a new product category (Copilot)

• Then, offer a free consumer edition and make money in enterprise (Codeium)

• Then seek out the high end of the category and charge a premium (Devin)

• Then broaden audience to noncoders (Bolt) and be MORE agentic (Windsurf)

• Charge flat-rate or free to get user data to build products + models (Codex, Jules)

As the average MMLU and SWEBench of every model goes up, so does the competition for increasingly advanced data to build those models. If we have exhausted GitHub and StackOverflow as datasets reflecting the average programmer, then we need elite ones per domain, and so on. We have heard rumors that there are now niche data services - in the style of Scale and Turing - that employ ex-industry professional humans specialized in creating/labeling medical, financial and coding data for labs, paying up to $500k/year per human. A scaled operation of 100 of these would cost an easy $50m/year.

That’s 6.25 TRILLION tokens of GPT4.1. The average coder consumes about 2m tokens a day per Claude Code numbers, so if you could figure out a way to collect useful data to your domain, you can justify giving away tokens as an alternative to paying for data - probably more because your users are the most specifically suited, self selected and self motivated labelers you will ever get. This is why you can kind of justify that Perplexity is classifying free tier costs as R&D if you squint really hard.

When we first wrote about the Four Wars of the AI Stack, most of the conversation was around data acquisition from traditional content houses and synthetic data for the rest. While synthetic data is still relevant e.g. for changing the Robotics scaling curve, the Data War is now fully focused on getting *YOUR* data as model input and grader. The best bellwether for this is LMArena’s $100m Seed round, done a few months ago but announced this week and raising a lot of eyebrows:

given recent drama with Meta and Cohere, though of course it is the only eval with dozens of mentions at Google I/O.

Just as Airbnb doesn’t own hotels and Uber doesn’t own cars, this is an LLM Evals, aka LLM Data, company that doesn’t employ any of the human raters it uses to derive its scores. One appeal of using LMArena beside the warm fuzzies of “giving back” is that all inference in the battles are free, presumably subsidized by every provider partner. The arrival of a16z, Felicis, and other top tier VCs means there is only one way this goes: You will start getting paid if you are a good LMArena participant4.

In other words, GPT wrappers will go from charging a premium on tokens, to charge + flat rate, to free, to now we pay you to use tokens from us with good feedback.

And if LMArena can get a $100m blessing to do this for their users… why shouldn’t you?

A few people have asked my opinion on the vibecoding trend. At almost every SF house party, a technical engineer or three will sheepishly admit that they’ve been vibe coding, like it’s some sort of vice. We even went ahead and put in a Vibe Coding category in the AIE CFP. But ultimately we nixed it in favor of “Tiny Teams”: the rising trend of small but extremely productive teams augmented by SWE Agents (not necessarily making AI products). The simplest (but hardest) statement of this is “teams with more m in ARR than employees”5, which is rarefied air but already reached by @levelsio6, Cognition, Bolt, Gumloop and Gamma.

Tiny Teams are the scaled form of first type of AI Engineer predicted in The 1000x AI Engineer, and is the more realistic near term version of the “one person unicorn” that every thoughtleader is fantasizing about. Tiny Teams are easily profitable - most tech businesses are 60%-90% margin and most cash compensation should be <$250k - and therefore are also likely to lead to Pegasus and other outcomes.

The way I put it to a friend recently is that “vibe coding” is a slop attractor, while “Tiny Teams” focuses on productivity. One might look askance at Rick Rubin’s new book on Vibe Coding as one such instantiation, whereas it’s pretty hard to deny that a Tiny Team (in the way we have defined it) is an unambiguously Good Thing and a direct consequence of declining end-user costs in SWE Agents.

At the end of the day, of course, have a good idea of what you are building (aka Product Management) is paramount, when everywhere you look an AI builder appears:

with usage limits for Jules and much fewer limits for Codex as mentioned on the pod

Our changing track selections between the 2024 and 2025 AI Engineers are actually pretty useful for keeping track of these terminology shifts.

note that this isn’t the only type of market that this market progression applies to… but the $3b question is: what is the next stage?

This is a prediction made without any prior knowledge of LMArena plans, but we’re putting it down for posterity. Anastasios and Wei-Lin are smart guys.

a lower but still impressive bar is “teams with more millions in users than employees” - means they just need to work out monetization

he has one employee

btw Gamma, Bolt, Oleve and more are presenting on the new Tiny Teams track, and leads of both OAI Codex and Google’s Jules will be presenting at AI Engineer World’s Fair! You know the drill by now -…