Latent Space

The World’s Fair is now TEN days away! Today’s guest Will was one of the most popular speakers of AIE NYC, and is coming back for the new RL+Reasoning track with Misha Laskin, Nathan Lambert, Christian Szegedy, Greg Kamradt, Kyle Corbitt and more. Join the top AI Engineer gathering of the year! If you want JUST the hallway track, Expo tickets are now on sale - with ~30 of the Explorer discounts left.

In an otherwise heavy week packed with Microsoft Build, Google I/O, and OpenAI io, the worst kept secret in biglab land was the launch of Claude 4, particularly the triumphant return of Opus, which many had been clamoring for. We will leave the specific Claude 4 recap to AINews, however we think that both Gemini’s progress on Deep Think this week and Claude 4 represent the next frontier of progress on inference time compute/reasoning (at last until GPT5 ships this summer).

Will Brown’s talk at AIE NYC and open source work on verifiers have made him one of the most prominent voices able to publicly discuss (aka without the vaguepoasting LoRA they put on you when you join a biglab) the current state of the art in reasoning models and where current SOTA research directions lead. We discussed his latest paper on Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment and he has previewed his AIEWF talk on Agentic RL for those with the temerity to power thru bad meetup audio.

is on YouTube!

• 00:00 Introduction to the Podcast and Guests

• 01:00 Discussion on Claude 4 and AI Models

• 03:07 Extended Thinking and Tool Use in AI

• 06:47 Technical Highlights and Model Trustworthiness

• 10:31 Thinking Budgets and Their Implications

• 13:38 Controversy Surrounding Opus and AI Ethics

• 18:49 Reflections on AI Tools and Their Limitations

• 21:58 The Chaos of Predictive Systems

• 22:56 Marketing and Safety in AI Models

• 24:30 Evaluating AI Companies and Their Strategies

• 25:53 The Role of Academia in AI Evaluations

• 27:43 Teaching Taste in Research

• 28:41 Making Educated Bets in AI Research

• 30:12 Recent Developments in Multi-Turn Tool Use

• 32:50 Incentivizing Tool Use in AI Models

• 34:45 The Future of Reward Models in AI

• 39:10 Exploring Flexible Reward Systems

Alessio (00:01.136)

Hey everyone, welcome to a Lightning plus Emergency News, Late in Space podcast episode. I'm Alessio, partner and CTO at Decibel and joined by my co-host Wix, founder of Small AI.

Swyx (00:05.95)

Ha ha.

Swyx (00:12.776)

Hey, hey. And yeah, honestly, we knew that Cloud 4 was coming. And we just didn't. We're just too busy to have a dedicated episode. So this is our makeup dedicated episode with a special guest, Brown from now I can say it, Prime Intellect.

Will Brown (00:29.518)

Hey, how's it going? Great to be on and Circus and I have known each other for a little bit and this is my first time on the podcast, I believe. Great to chat with you guys. Big news day, I guess. So lots of stuff out in the world. There's always a news day.

Swyx (00:47.988)

I think this week is particularly heavy for some weird reason. Like, Monday was Microsoft Build, Tuesday, Wednesday, Google, and today is Claude. I wonder what tomorrow will bring.

Will Brown (00:56.322)

Yeah, we had IO and then had IO and then the two.

Swyx (01:00.16)

Yeah, different IOs, exactly. Yeah, so we actually were supposed to record this morning, and we all wanted to watch the Cloud Keynote, so we went and watched the Cloud Keynote. Obviously, a good model, big model. They're really emphasizing coding. They didn't really talk much about reasoning, to be super honest. They were just like, it runs for longer now. What are you guys' takes?

Will Brown (01:29.326)

Yeah, so mean, like one thing I've kind of been seeing coming for a little bit that I think people are kind of also all aware of now is that like, the thing that's going to make the next wave of stuff be powerful is just like everyone wants better agents. Everyone wants models that can like go off and do stuff. And like reasoning was kind of like a precursor to that a little bit. Like, I mean, I always think of like open eyes, like five levels framework, where like chat bots was like the RLHF era.

And then Reasoners was like the O1 and R1. But like really what people were thinking of was Reasoners are a step on the path towards agents. And so I can kind of see why Cloud, why Anthopiq is not like, we have the best Reasoner. They're really like showing off their suite agent and like tool use and like function calling benchmarks, multi-turn stuff. Because I think that's really like what people care about more for actual applications as opposed to like...

did really good on this math competition. Like the math competition was like, that stuff was all like a signal that was supposed to think we were getting somewhere. But the thing we were getting towards for a lot of people at least is practical agents.

Swyx (02:40.871)

Alas you

Alessio (02:41.178)

Yeah, I think the extended thinking mode, I think they removed the uppercase. I think in the Cloud 3 release, was like extended thinking, kind of like capitalize and not just like extended thinking with tool use. So I think they're also, yeah, downplaying whether or not it's reasoning or not. I think they're trying to merge everything together. And it's not, I mean, I didn't realize that, but extended thinking could not.

used tools before, the way they worded it, and now they can in Oppos 4, so that's great. But yeah, they haven't put it as far center as last time.

Swyx (03:18.483)

Do we have any, this is already veering off from Claude directly into speculation, but do we have any idea if there are any material differences between how Claude extended thinking works versus the old series models? Do we know?

Will Brown (03:35.758)

The biggest difference seems to be, and this is kind of a thing that's been, I mean, this is all speculation, of course, but from the start, Anthrop had always kind of had this little thinking thing where you could, sometimes even like Cloud 3.5 would do like a tiny bit of thinking. And it was really just like deciding which tool to use for the most part. Like if it was doing an artifact in the Cloud UI, it would have this little thing.

where it would think for like two sentences about which tool to use. And it seemed like Anthropix kind of attitude has been that extended thinking is an instance of tool use. And that it's the kind of thing you want to equip the model with the ability to do, but it's not like, it's a thinking model. It's just a sink for the model to like brain vomit because that brain vomiting will help it like find a nice thing to do next.

In the same way that doing search or doing code execution are like ways to kind of get more information on the path towards like finishing a problem.

Swyx (04:42.929)

Yeah, inference time compute, as they say. I did meet somebody who claimed to have coined to define the scratch pad paper. And this was obviously before the Jason Wei chain of thought paper. But it's all the same sort of method, general family of techniques.

Will Brown (04:51.105)

No.

Swyx (04:56.979)

I think the question for me is also like, is there some model routing going on? Like are they different models, the thinking, non-thinking? Or are they the same models with like, just like you turn off the end of turn token generation. that's it.

Will Brown (05:13.165)

I think these models should be the same model and AnthropicNet is what they're doing well. Like it's not that hard to like, Quinn did it in a very kind of like simple way and they kind of talked about how they did it a little bit. But it's not like too difficult to like have whether or not a model thinks like be the sort of thing. I mean, like obviously all of this stuff is like hard at like serious scale, but like conceptually at least it's not like a big

problem to solve about how would you ever do it. It's like, no, we have reinforcement learning, can kind of like, or we just SFT on like different things, can kind of teach models skills like that pretty directly.

Swyx (05:54.674)

Yeah. You have some work that you've published recently on GRPO and relationship for doing a lot of work on multi-turn RL. I think I wanted to just kind of round out any other Claude highlights that you guys saw. There is controversy that I'm leaving towards the end. But any other technical highlights that you guys want to focus on?

Will Brown (06:09.975)

Sure, sure, yeah.

Will Brown (06:14.219)

Right, right. Okay, yeah.

I mean, I think it seems like a really cool model. But I think like a Calamity's like tweeted this earlier today. It seems like it's linear progress, which is like great. But it doesn't feel like there's not anything that I've seen from it. That feels like a paradigm shift in terms of like, the sorts of stuff Daria talks about, which like I think maybe we're still on the path to get there. And it feels like this is just like, keep going up.

in terms of complexity of agents. think the one thing that to me was really nice to see, I haven't like done too much testing myself yet. But in their reported benchmarks, the reward hacking issues to like, sonnet three seven loves to like do stuff that to me feels reward hacky in the sense of like, it'll try to, you ask it a coding question, and it would like, do your question, and then seven other things also. Presumably because

there was some RL environment where there wasn't really a penalty for doing that or there was an enough penalty and covering its bases like was more likely to pass test cases on some coding thing. Like you could imagine like a sweet bench kind of thing where there's a minimal diff that is really what you want, but there's like you could do a ton of other stuff and put all these other things in place that as long as you don't, as long as it's not enough that you trip over your feet, it's just like extra stuff that's there if it helps pass the test cases.

And what I really think you want to do with these models is like kind of min-max. Like you want the models to like do the thing and no more. And they had some internal benchmark for this that went from like 45 % down to 15 % for both for Sonnet and for Opus as opposed to 3.7. And so I'm hopeful that these models are much more like friendly to code with and maybe more like trustworthy. Like that's the thing that I kind of...

Will Brown (08:14.007)

have buckets for models of like how much can I trust them in a code base? Especially something beyond like a single file like old Gemini to me was very trustworthy. GPT 4.1 is very trustworthy. New Gemini is not, 3-Sonic is not, 03 is not. I haven't decided which bucket new, new sign-up, new outputs are gonna fall into.

Swyx (08:35.453)

trustworthy in terms of reward hacking.

Will Brown (08:38.381)

Just like not gonna make them like the they're gonna do the right thing in the code base and like worst case they'll do it like dumb but they're not gonna like go break a bunch of stuff. They're not gonna leave a bunch of like extraneous comments and helper functions all over the place that aren't really needed or like make seven new files just to have them there. Like this is the sort of thing that 3.7 does a lot. Yeah.

Swyx (09:00.401)

Yeah, way too eager. Yeah, it was like I already have the function of my code base. I would just make a new one just because it felt like it. Yeah, one thing I often wonder about those things is like, just for RL environments in general, why is a token cost more of a thing in the penalties? That's the one rule above all.

Like you can actually skip a lot of reward hacking by just, hey, the more tokens you use, the worse it is.

Will Brown (09:31.502)

I mean, that's not what the model, they're selling you tokens. They want you to use more tokens. But like, so like there's that element of it. But I think also it's that there was this initial kind of reaction of everybody of like, more tokens is better. If you look at the line, it goes up as you spend more tokens, your accuracy goes up. And so I think the pressure to like really tamp down on token usage was not that serious for a lot of people, especially because the companies are like,

Swyx (09:34.139)

This okay.

Will Brown (10:01.617)

to sell you more tokens. it is the sort of thing that you can have some more controls over. like, when did this in kind of like a very kind of abrupt way, where they can like you can in the UI, can like set a token budget and just like truncates the thought. it seems like artificially truncating the thought is actually like fine. Like the model can even if like it got cut off mid sentence with an injected like think token.

these are smart enough models that they can kind of finish with the best that they got from that point. And so that's like one way to do it. The other and like that's becoming a kind of standard API feature now is like your think budget. Claude has that. Yeah, we did a little bit of experimentation with that in our last Interlock2 run at Prime Intellect, which it was before I joined. But thinking budgets are the kinds of things that you can insert into a reinforcement learning objective. And

Swyx (10:34.449)

Compensate, yep.

Will Brown (11:00.575)

You can see the model get better at targeting the right amount of thinking based on, let's say something goes in your system prompt. You can have the prompt just say, use X amount of tokens. doesn't need to be like, but if you've trained the model to respect this, would hope that if you execute this correctly, the model learns to roughly think the right amount.

Swyx (11:24.325)

Okay, this actually changed my opinion of thinking budgets because previously I was thinking that reasoning effort was better than thinking budgets. Thinking budgets is of like a max cutoff. It's a target, right? It's not, it's not a, like it's a, right, right, right. Yeah, because I actually want to set effort. I don't super care about cutoff apart from the cost and like giving me, you know, 64 bits of cutoff or whatever doesn't matter.

Will Brown (11:34.037)

the same thing. think. Right. So the effort is a target probably. Yeah.

Will Brown (11:53.708)

I'm not sure that they're like that different. think like we don't know how they do it under the hood. But my guess is that the whole reasoning effort thing is essentially a token budget that the model has been like RL to like, you would hope that you get different behavior. So the model when it's told it has a short thinking budget, you would hope that uses slightly different strategies that are better versus it has a high budget. It's more willing to like do lots of math calculations, for example. But I think conceptually,

It's really just about the model has some amount of room that can think for in tokens and it's trying to do that well, hopefully.

Swyx (12:29.479)

Yeah.

Alessio (12:30.224)

Do you think we're gonna have these as hyperparameters for like much longer or do you think this is kind of like, know, as we're early in this like recently models, more of the stuff is exposed and then it gets moved away from the user?

Will Brown (12:44.705)

I think in chat interfaces, it probably won't stick around. Like I don't think we're always going to have the drop down of like oath for many and for many high that feels silly. I do think it's a thing that developers want, especially because once you've kind of built around a certain model, like a lot of these writers are hoping you stick with the one model and are not switching all the time. You do need a knob to control costs and also latency. And so that is one kind of useful knob to expose developers.

for controlling this quality versus cost and latency.

Swyx (13:24.443)

Awesome. Cool on all of that. I think the elephant in the room, let's talk about it, is this controversy around opus or opus, right? Snitching on you.

Will Brown (13:38.303)

Mm-hmm. Yeah, I mean, so I have lot of opinions, but I think...

Swyx (13:42.961)

So for those out of the loop, let's recap because I feel like you're closer to this than I am. Like I learned about it from you.

Will Brown (13:49.102)

Sure, sure, yeah. this was someone from Anthropoc, I'm not gonna name him, because I know he doesn't wanna have all this attention on him, he deleted the tweet. But it was essentially like going through different things that people found during safety stress testing of Clod. And so this is not like, what's Clod gonna do for you? I think people took this out of context pretty badly. And so there's a fair point there that it's like, people are really reading into the one sentence much more than they should.

Swyx (13:57.522)

Of course you did.

Will Brown (14:19.201)

But this is thing Anthropic does a lot is they really stress test their models. They try to put their models in situations where they can really see like what could an adversary get them all to do or what does the model do if it's in a situation where there's no right answer. like, I think a lot of the kind of headline Anthropic like safety results, especially related to reward hacking and kind of deviation and alignment faking are all things to me that seem like a rock and a hard place situation where

the model has two objectives it's given that are conflicting with each other and it has to pick one. And no matter which one it picks, it's going to sound terrible. Like it's either following the user's instructions or it's following like common norms. And once you kind of accept either of those, it's going to do the thing that is aligned with that set of like guidelines. So in the case of like, if your model's goal is to be like maxing helpful to the user,

then it would help a user like build a bomb. If a model's goal is to be maximally helpful to society and a user's asking you to build a bomb, it's gonna be like, no, that's bad. I have to do something to stop this. Like you kind of have to pick a goal. And like maybe the right answer is the model just differs and it's like, no, I'm gonna stop talking. But people also get mad when you tell them like that the model will stop talking to you or like refuse to do anything. Like there's just no, it's, there's no, it's.

no way to kind of win and make everybody happy. But I do think like, like they report this because they think it's important to have people understand the safety implications of these models and to understand like, okay, how bad would it be if someone was trying to use this? Could this like meaningfully help someone commit crime or violence or whatever? And so like that's what they have like their state safety framework for. And the things that happen in these like,

blog posts and threads and papers about like the model trying these things, they're kind of putting these models in a scenario that elicits these things. Like it's the sort of thing that you would imagine a very smart human might also do in those situations. Like let's say you are told like accomplish some vague under specified goal at any cost. And you really like want to solve that goal. Think like game shows like survivor, I think is a good example of like

Will Brown (16:43.029)

or Lord of the Flies, any of these kind of canonical situations of people who are put in a weird spot and have to go do stuff and figure it out how to do it. They're kind of crafting these environments for the models and just looking at it and seeing what happens. And so I think it is a little silly to overanalyze behaviors in either direction of like, the model is reporting you to the police or the model is going to go help you find uranium on the dark web. Well, these models can kind of do, there's no, like they're

The base model in general of LLM is not artificially constrained in any way. Like with the right prompt it will do whatever up to its intelligence limit. And so like the question is just how do you constrain the space from all possibilities down to like a more reasonable set and like that's hard. So.

Swyx (17:32.347)

Okay, you actually gave a serious answer, which I totally respect. I was more looking for shitposts. you're treating this as though, like, yep, this is what the problem actually is, which is totally fine. And yeah, I mean, that's what you are as a researcher, right?

Will Brown (17:35.693)

I mean...

Will Brown (17:50.86)

Yeah, I mean, I think tweeting is fun. Like it's it's cathartic to like just kind of like get a post out. So like, when I saw the one about the uranium thing, I was like, let me tweet it. So the tweet was like, we found that quad can go search the dark web to look for like uranium. And I was like, here are the top 10 things that builders are using in their agentic rag applications with the new groundbreaking quad for

And it was just like, silly, both making fun of like, LinkedIn, like, thread posters as well as just like the funniness of the scenario that they were talking about.

Swyx (18:24.721)

Hehehehe

Swyx (18:29.587)

Yeah, this is it.

Alessio (18:30.128)

Does any of this make you think differently about what tools to give an LLM? I know they deleted the tweet, but it's basically like, before if you're putting all these MCPs, like, yeah, you have email access and all of this. now it's like, well, maybe I don't want to give email access all the time if you're going to snitch on me with the email access.

Will Brown (18:31.932)

Hahaha

Will Brown (18:37.761)

Hmm.

Will Brown (18:49.353)

Yeah, I mean, I think coding with these models, especially like quad, I did a fair amount, like for a few weeks, I was doing a lot of quad code with three seven, mostly for kind of random side projects. I never really got to the point where I found it was helpful for a thing that was like a large existing code base. but if it's like, Hey, I want to like cook something up in a few hours for fun. Pretty good at that. But these become messy and they become hard to maintain. And you get to a point where it's like,

nothing is working, I just gotta like dig in and fix it all myself. And so I think part of that is that the models have access to like a terminal and you can do a lot of stuff in a terminal. MCP is kind of a way of constraining the action space. like in like canonical RL, people talk about like states, actions rewards, policies as like the things that are like the moving parts. And

models generally are trained in like old school rl but like a very fixed action space of like what are the keys on the video game i can hit but with that alone it's like text text is like kind of unbounded and what you can do with it in a terminal there's not much you can't do in a terminal and so if you're training models i got a lot of flack for this one people were it was both people who were like

Swyx (20:09.191)

I'm just showing this. Wait, Flack? Why?

Will Brown (20:14.795)

the notation is stupid and bad, but RL is really simple or like RL is like complicated and it's like everyone has a different opinion on what RL means. And I was trying to just like, be like, hey, it's actually kind of complicated. And I wasn't picking this up like, the definition of the NMDP is complicated. I was like, no, there's just like a lot of moving parts. And to think about it, to like do anything, especially if you to change any like part of the system, like here's a question, hypothetical, like

What happens if you have two LLMs learning together? How do you reason about that? How do you like think about that? Is this going to be a stable system or not stable system? What if they're like kind of cooperative, but kind of not cooperative? And then they're training to work together, but also want to backstab each other. Like this is kind of the environment people are finding themselves all in all the time out in the real world. But if you want to make AI's do this, have to like translate this into code and math. And the more complex your

goals are with this thing, the more complex the math gets. like, RL is like one math language that kind of exposes these primitives. But like, I think a lot of people are like, oh, I can follow the equations. That means I understand it. And it's like, well, sure. But like, also, there's, it's like this, I don't know, end body problem thing, where you can freeze it and look at it. It's like, oh, how does one thing moving affect everything else? And what are the cascading ripple effects? And like,

Swyx (21:41.415)

Wow, you just brought three body problem into this. Amazing.

Will Brown (21:45.421)

Like, in like the physics version, not the show. Yeah, yeah, yeah.

Swyx (21:48.786)

Yeah, yeah, No, no, mean, yeah, that actually, actually very impossible to model. Like, I guess you can like simulate it, but like even then, yeah.

Will Brown (21:54.378)

Mm-hmm.

Will Brown (21:58.85)

But it's chaotic. It's like it's sensitive to initial conditions. So like you can't really say like this is one of those things that like like why does no one predict the weather a year out? There's no I don't think anyone has anything that's like good at long term weather forecasting. I'm beyond like I don't know climate trends, but no one can predict whether it's going to rain in Seattle on a given day in a year. Even if you think like like the system's predetermined, like we did. It's all clouds bumping off each other and whatnot. Mountain ranges. We kind of know how these things work.

Swyx (22:09.095)

Mm-hmm.

Swyx (22:14.195)

climate.

Swyx (22:25.363)

So the butterflies are flapping their wings. mean, you got to let it play out. If we had no butterflies, we could predict it. Interesting. OK, so I guess we can sort of round it out, unless there's any more of the controversy. I think there is. I think that the system card is actually very good. They probably went too hard on it.

Will Brown (22:29.431)

But exactly, yeah.

And so, RL is very sensitive to butterflies.

Swyx (22:54.451)

compared to like normal system cards. And it's a little bit confusing whether this is marketing or are they just like, no, we really super care about safety. And part of this is like Apollo just being Apollo, pushing the frontier of red teaming, right? So they're going to report the things because it's extremely good at Apollo marketing. Yeah.

Will Brown (23:13.005)

Right. Yeah.

I think they're really there seem to still be like trying to be creative with their kind of consumer marketing. Like it feels like people in the world like love quad or have ground type of quad, but still had a phase where they were using it a ton. But it hasn't really broken out to general people in the way. And it feels like a lot of their marketing that I've seen is like a little confusing. Like it feels like they've done a really good job at crafting a brand image that appeals to a segment of the population.

who has certain considerations that they really like that a model has a deep personality or whatever. The sorts of people who I think also really like GBT 4.5, many of them really loved Claude III Opus, the big model smell. But a lot of people just don't care. And I think Anthropec is trying to figure out how to...

Swyx (24:12.763)

Yeah, just wanted to use it as a tool.

Will Brown (24:18.177)

like appeal to that audience. The LMSys, Sycophanty, Roporo, the people who love those models, different crowd. And it's a larger crowd. And that's a tough problem to solve as a company.

Swyx (24:30.171)

It is. What's your quick take on Ella Marina getting $100 million?

Will Brown (24:37.313)

We'll see. Like I imagine that they partner with company labs in different capacities to.

Swyx (24:45.405)

Probably making a lot of money, yeah.

Will Brown (24:47.725)

Like I'm not, I'm not in the business of trying to point the finger at like saying they definitely did this, but if I was a company that was able to raise it, that kind of valuation, and I had just had a long public partnership with Metta or eventually public partnership for a thing, which we've kind of seen was Metta had the ability to do a lot more back and forth than a lot of other labs did. I would imagine that there's some compensation going on there or access to data. And so like,

I think being an eval company puts you in a really hard spot. Some people are talking about this on Twitter, like just that to be an eval company, you kind of have to sell to the labs, but selling to the labs doesn't really like kind of wrecks to reviles. And so it's like this dog chasing its tail.

Swyx (25:26.973)

Mm-hmm.

Swyx (25:32.529)

Because your incentives are like your customer. Yeah, yeah, So in finance, would, mean, you you are from Moenstany. So this is the credit rating agencies. Like literally your customer is the one that you're supposed to govern, but they're also your customers. So then you have to be like nice to them or they'll just go to the next one.

Will Brown (25:39.469)

Yeah. Yeah. Exactly. Yeah.

Will Brown (25:53.26)

Yeah, I mean, I do think that like the best source of evals going forward is probably going to be academia. And so this is the thing that I tell people who are like starting a PhD, which is like find things that are like cheap to work on as a PhD student. Because you cannot go pre train a foundation model really on your own, but you can build a really good, really clever eval. And like we're

churning through evals at the time, we saturate them, we always need more. It's not the kind of thing that is ever going to end. And so that's the task of translating vibes of what is good or bad about a model into very precise scientific questions, I think is an important problem. It's a problem that you can get by a lot more with brain power rather than dumping capital into it.

You need to like pay for the API costs, like that is generally the kind of thing that either you can get covered within academic grants or like industry sponsors or the kind of thing that just like there's versions of these things that are like small sample size that get you on get on the radar or you kind of pick and choose which models you can afford to email. But it's like an accessible field of research. And it's one that like the incentives of academia I think are quite good for, which is like write a splashy paper.

that says something interesting about the broader field rather than, we want to make this one look like the winner.

Swyx (27:25.179)

Yeah, I think a lot of grad students still don't have taste. I don't know how better to put it. just, you go to enough academic conferences and I'm like, why did you work on this, man? Like, you're so smart. You're capable of better. So how do you teach taste?

Will Brown (27:31.735)

That's fair, yeah.

Yeah.

Yeah.

I think, I mean, I can tell how I did it originally, which is like, I think you always want to be thinking pretty far ahead and you want to be like making kind of educated bets about what the world looks like in the state in the few years. like, you have to say like, what are the questions that no one's even talking about? And this is like not an easy thing to do. You have to like really convince yourself that you're kind of right about the way things at least might go.

Like when I was like in, I finished up undergrad, late, like they finished in 2019 then went right into grad school. Um, but like towards the end of the 2010s, like we had like, I'll go to deep mind, uh, doing all this multi-agent RL stuff that was like really cool. Um, then it was like, okay, this stuff kind of works like AI is like going somewhere. Multi-agent systems are kind of going somewhere still very early stages, but what's going to happen once this gets there.

Will Brown (28:41.533)

And it seemed like, okay, these things are all going to be like continually learning in parallel as this big multiplayer game, basically. And if you look at the math, the math was kind of like undercooked and there's like some really hard open questions that are still open questions in multi-agent learning theory. And so like that was my focus. was, which was like, how do I like learn about this? How do I learn to think about this stuff better? And at some point I kind of got tired of proving theorems and was like, okay, let's just go build the thing. But I think like,

You want to think about like, whether you're doing theory or experiments, like you have to lay out a few different conditional statements to get to the point where you really doing interesting research that's beyond like, just loathing through that people are like, obviously going to be working on in parallel. You want to be jumping ahead of the curve a little bit. Um, I think my last, I don't know, this isn't like, I wasn't the first person to do this, but like, it was pretty clear to me.

like after O1 and before R1 that like RL was going to work and that that was going to intersect with agents where the solution was going to be like RL Batilius. That seemed like the direction things were going to go. And so that was like, I don't think that was a very risky research bet, but it was like a research bet that seemed to work out.

Swyx (30:12.071)

Yeah, speaking of which, you just published the paper. Now I have the full context as you were an advisor on this, and one of your grad students was doing the work, something like that.

Will Brown (30:15.169)

Yes.

Will Brown (30:22.495)

Yeah, so was me with Sillyong was my intern. This was kind of the last major thing I was working on at Morgan Stanley. And this kind of was in parallel with the verifiers as the repo that I've been building out major updates that coming very soon, by the way, I'm very excited about some stuff. But it kind of was something I really started in earnest, like January, kind of in the follow up to I'd had the GRPO demo thing go viral.

And was like, wait, there's something to this format reward thing. like, this is like a proper repo. the other one. Yeah, yeah, the other one was like just a gist. This one is like a repo for like multi-turn tool use. RL, with GRPO. And so like, in some ways the paper is like, it's the first paper that's really like, actually there's been a couple of papers that people have used the repo for.

Swyx (30:54.675)

It was literally like a GitHub gist, right? Or something.

No, no, the GRPO. Yeah.

Will Brown (31:21.559)

but it's one where a lot of the stuff from the original GRP demo gist gets kind of extended to the multi-turn RL tool use setting. And so there's a lot of experiments here about like, okay, how do you actually get models to use tools? How do you incentivize tool use? Because something we'd see is that if you set these models up to use tools, they just won't. Like if you say, hey, here's a question, you have access to these tools, do as many rounds of tool calling as you want, and then submit your answer, they'll just submit their answer.

because they like are, especially for like small models, like they aren't already trained to use tools. They don't really want to because they don't necessarily have that instinct and they're pretty bad at like function calling and format instruction following. And so what you would see is like when they use a tool, they would like mess up the JSON and then they'd be like, that didn't work. And it threw me, it got me out of focus and it would be more likely following that that the model would just like go off the rails.

because they would get like an error message from the parser. And so the safe option for the models is just to like stay in this basin of like, just do think then respond. Same with like normal formatting awards too. Like if you want models to use thinking tokens, you kind of have to like incentivize that. You have to either do a little bit of like SFT warmup or you have to like reward them for doing it. Otherwise they will not follow it 100 % of the time on format alone versus like

a model like R1, like 100 % of the time, it is going to use its think tokens. You are not going to ever see R1 just like talk normally without the thinking section. And so like, you kind of do have to decide what you want the model to do. Like, this is a little bit like a user facing question of like, what, what behavior of the model should the default be? And if you have, if you want it to do a certain thing, if you want it to be a tool use agent model,

Like it does help considerably to like actually have this incorporated into the reward. The kind of key trick in the paper to get around this problem. So, okay, one kind of reward hack these models would do is like, they would do like a dummy tool call where they would like learn to ask this, use the same Google search every time and ignore it. like the, questions would be like, okay, here's some like MMLU style question, go figure out the answer.

Will Brown (33:47.734)

use web search and if you start rewarding them for like tool use, they will use the tool, but they don't really want to like have to, they want to like be very safe with it. And all of these questions, like they do kind of know a lot of the answers already. And I think calibrating the right difficulty of your questions for RL is like an important problem that we're still kind of figuring out. but they would like do silly versions of tool use where they aren't actually using the tool to assist in their reasoning.

they're using it to get the reward. And so we kind of have to do a credit assignment thing of like, okay, did the tool result in information? And so for these experiments we were doing, the trick was like, okay, does the like some string matching thing involving the ground truth answer and the return search results from Wikipedia. So did the model actually start to thing that retrieved useful information for the question? And so this is like, but the framework is more general than just that it's that

I'm once you have a new a way to do intermediate evaluation if you can evaluate like the quality of an inter-other mid-year estate now you can kind of rewrite the g r p o advantage calculation to take this into account. Because I think this is less a problem like ppo if you know ppo is like the old school rl and it also is what people use for our lhf. But in the context of g r p o g r p o is like great for.

like leaning heavy on highly parallel inference compute, it's more memory efficient for the actual training process. It's much easier to do in a distributed fashion because you have less gradient syncing and less model weight copies. It's kind of like DPO on steroids, I think is one way to think about it, but it's also gets around a lot of the pitfalls of DPO both in that it's like online by default, as well as that you have this large set rather than just a pair of completions. So you do get like some intermediate credit assignment a little bit.

via this group comparison. But for tool use, seems to be far enough out of distribution of small models, doing, incorporating this turn level. So the way that I've been thinking about it is like, in like canonical RL, the state action are like things that you do many rounds of like, take an action, go to a new state, take an action, go to a new state. And for a while people thought about LLM RL as like, each token's action and the new sequence is a new state. And you can kind of do that.

Will Brown (36:10.733)

But you can also think of each turn as an action where, yeah, where the state is the response you get back from the tool call. And now you like have a different way of designing your algorithms to take into account credit assignment, which is that like, and it also is like a little more flexible from a reward perspective. it like feels like people are moving in the direction of model based rewards where you either LLM is a judge where

Swyx (36:14.053)

Yeah, that's more likely.

Will Brown (36:39.681)

the judge sees the correct answer or it has questions it's supposed to verify as properties of the response. Just because that's much more flexible than like trying to write these little parsers, like writing a math parser to check if a math question is right is like not that easy actually because there's so many edge cases and you want to handle like latex support and markdown and like equivalent fractions. And it's like, just like let a model do that. Don't don't have a 2000 line Python script that does that.

Swyx (36:45.341)

Mm-hmm.

Will Brown (37:09.599)

And so...

Swyx (37:09.629)

Sorry, let me clarify. Math parser to verify that the math is right. And you have a latex parser inside it.

Will Brown (37:17.193)

sorry, yeah, so like a lot of models naturally will like think in latex because they've been trained on lot of archive like tech. And so if you're doing like an R1 and people are like, math is easy to verify, the easy to verify still is usually like this very long piece of code that has to handle lots of annoying edge cases. And even then it's like 98%.

Swyx (37:22.695)

I didn't know that. yeah, that makes sense. I I guess.

Swyx (37:38.799)

Yeah. Hmm. Hmm. Okay. Interesting.

Will Brown (37:42.882)

Because like, it's a freeform response that is like, there's not only one way to write an equation. Like if you have two valid mathematical expressions that are equivalent, but they're also like symbolic. Like you need to verify that two symbolic expressions are correct. One of which might be written as code. One of which might be written as latech. One of which might be like written as word. Like you can't do it if it's words with these like literal parsers, but they try to cover a lot of these cases. That's also why you'd see models put boxed around their final answer a lot.

Swyx (38:04.787)

There's pseudocode out there, yeah.

Will Brown (38:12.767)

is that it's one hack is that it's much easier to kind of verify the right piece of the information if you know exactly where it's going to live rather than like the model saying, the answer to the question is four. Then you have to like parse away the answer to the question is, just throw that out. so it's like deterministic rewards are like nice if you can get them to work, but they're also really painful. And they're pretty hard to generalize across domains. Like for math, the easiest is when the final answer is an integer.

and lives in the same spot. Like there's a box that's going to be an integer. And so this is one of the reasons like everyone used GSMAK for so long is because it's like mostly integers. think it, A E is all integers. It's super easy to verify these things and to parse them. But as you go to, and multiple choice too, multiple choice is super easy to verify. But anything that's a little bit more flexible, deterministic, like rule-based rewards start to break down.

Swyx (38:44.645)

you

Swyx (39:08.754)

Alright.

Will Brown (39:10.197)

And, the model based direction seems to be pretty promising. And I think underexplored for like, what if you use LM as a judge in your RL loop? think kind of going back to like, Anthropics has been talking about this for a long time via constitutional AI. In that case, it was less about the LM judging and giving a direct reward to the model and more about training a reward model that was doing like token level advantage estimates. the P which is the PPO way of doing it.

But it seems like you can kind of do that for GRPO2 and other flavors of RL where you can incorporate like, you know, the reward model can basically be an LLM where like it's fine tuned to like be more calibrated maybe and to have the right kind of range of responses. But you could also have it be a reasoner. You could have it be something that is able to do tool calling. There's no reason why the full power of LLMs

Swyx (39:46.684)

A full reward model.

Swyx (39:51.794)

Yeah.

Swyx (39:58.418)

Yeah.

Will Brown (40:09.517)

can't be offloaded or can't be also given to the process of evaluating whether or not an answer is correct or satisfies a certain set of criteria. And so I think like, that's the direction I'm most excited about is like really pushing on kind of beyond deterministic rule-based rewards into like these more flexible things. And I think you want to do this both at like a, so okay, that paradigm is not going to work super well with

uh... turn a lot of token level rewards but i think it does work with turn level rewards like can the alum verify like whether a certain search query was useful? Sure. Um, like there's a lot of these questions that are pretty granular that alums can like basically nail all the time if it's a good enough alum and you can incorporate that into RL with that sort of approach.

Swyx (40:53.745)

Yeah, you decompose it.

Swyx (40:59.795)

Awesome. I think that was all the topics that we had prepped. Alessio, I think you're also pretty good on that. Obviously, it will take some time to figure out Cloud 4. Anything you want to plug? We already talked about your talk, I guess, coming up.

Will Brown (41:16.557)

Sure, yeah. I'll be at A.Engineer in a weeks, yeah, coming up. I'm also doing a course thing. Yeah, mine's gonna be, that's gonna be a lot of fun. I'm also collaborating with Kyle Corbett from OpenPype to do a course, which is both of us have our open source projects that we like, are agentic RL focused and kind of we've been friends for a while and are trying to do something that's a little more...

Swyx (41:20.947)

Two and fourth. Yeah. Your track is particularly hype.

Will Brown (41:44.777)

structured as like a way of kind of getting information out into the world for people who I think we're especially thinking about like kind of practical use cases for agents and helping people giving people a kind of outlet to learn more about like how the stuff works and yeah more coming soon about that.

Swyx (42:04.06)

Awesome. Well, I think that's it. Yeah, thanks for coming on at very short notice. I'm glad we can make this happen. We'll do part two with Callow and do full prime intellect thing whenever you guys are ready. Awesome.

Alessio (42:05.4)

Awesome. Thanks for coming on, Will.

Will Brown (42:14.286)

Sweet. Awesome. That'll be fun. Great.

The World’s Fair is 2 weeks away, and early bird tix have sold out! We’re happy to share that Fouad Matin of the OpenAI Codex team, Michael Truell of Cursor, Kevin Hou of Windsurf, Boris Cherny of Claude Code, Scott Wu of Cognition, Josh Albrecht of Imbue, Itamar Friedman of Qodo/AlphaCodium and Eric Simons of Bolt, and many more SWE Agent luminaries will be sharing updates and launches.

Join the top AI Engineer gathering of the year!

Since the first leaks about OpenAI’s coding agents in January, the speculation has been rampant, but with the Codex CLI release last month and today’s ChatGPT Codex research preview, the A-SWE (automated software engineering) agent roadmap is slowly coming into focus, with more agentic coding tools on the way.

On the livestream, Greg Brockman mentioned a number of pro user habits for using ChatGPT Codex: write modular code, and use linters, formatters, and commit hooks extensively to give the machine a faster feedback loop.

In this chat with the Codex team though, we found a good deal MORE suggestions for best practices to upgrade your vibe coding to one-shot WHAM coding:

• Have an Abundance mindset: Codex has unlimited usage during the research preview, so you should use a ton of it, for every little idea.

  • We confirmed that you can run up to 60 concurrent Codex instances per hour, meaning you can fire up a Codex task every minute

• Groom your Agents.md: the new hierarchical Agents.md is designed to capture all your instructions to the model and grow as model intelligence grows. You can even ask Codex to write its own Agents.md…

• Make your codebase discoverable: a well named and organized codebase helps Codex navigate the filesystem as well as a brand new engineer might.

• Ask it what it should do: You don’t have to prompt anything if you are out of ideas - having Codex create tasks for itself is an underrated skill

• Use it in mobile ChatGPT: as demoed, the mobile site (not yet the app) unlocks a different form of fire-and-forget usage than when you use it on the desktop

• Use in-distribution languages with types: Python > Ruby, TS > JS

There’s lots more little tips in the podcast and yet to be discovered!

We were honored to have the chance to test the new product, but our impressions changed a great deal once we talked to the Codex team for this podcast and found out how power users use Codex:

We’ve already talked about the four types of SWE Agents, and on its surface ChatGPT Codex looks very similar to Devin (Scott Wu’s critique here), but the end to end RL in the new `codex-1` finetune of o3 leads to remarkably improved benchmarks:

• 00:00 Introduction to the Latent Space Podcast

• 00:59 The Launch of ChatGPT Codex

• 03:08 Personal Journeys into AI Development

• 05:50 The Evolution of Codex and AI Agents

• 08:55 Understanding the Form Factor of Codex

• 11:48 Building a Software Engineering Agent

• 14:53 Best Practices for Using AI Agents

• 17:55 The Importance of Code Structure for AI

• 21:10 Navigating Human and AI Collaboration

• 23:58 Future of AI in Software Development

• 28:18 Planning and Decision-Making in AI Development

• 31:37 User, Developer, and Model Dynamics

• 35:28 Building for the Future: Long-Term Vision

• 39:31 Best Practices for Using AI Tools

• 42:32 Understanding the Compute Platform

• 48:01 Iterative Deployment and Future Improvements

Alessio [00:00:06]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host Swyx, founder of Small AI.

swyx [00:00:15]: Hello, hello, calling in from Singapore here, but we are in the remote studio because the opening IT team keeps shipping, and today they just live streamed and released ChatGPT Codex. Welcome to Josh, who I think we've talked about, we've met while you were at Airplane, right?

Josh [00:00:34]: Yeah, yeah, you know, I've been building DevTools for a bit now, and I have to talk to you when I'm building DevTools.

swyx [00:00:42]: I mean, you know, you have now seen me complain a lot when things happen, so I don't know if it's a good or bad thing.

Alexander [00:00:51]: It's a gift, man. Feedback is a gift. Thank you.

swyx [00:00:55]: Alexander, we're new to each other, but you've been leading a lot of the Codex testing and demos and stuff.

Alexander [00:01:01]: Yeah, hey, I'm Alexander, I'm on the product team here.

swyx [00:01:04]: Awesome. So, yeah, we're going to just assume that everyone's watched the live stream. You also released a blog post with a bunch of, like, test demo videos, basically a bunch of, like, it's very interesting. I noticed in the demo videos, it was, like, individual engineers sitting by themselves, very lonely, and then they're just talking to their AI friends, coding with them. I don't know if that's the vibe you want to give off, but, like, that's how I came across.

Alexander [00:01:30]: Yeah, man, those videos, we were going for. Like, maximum authentic, just, like, engineers talking about how it helps them.

swyx [00:01:38]: Yeah, I'll take the feedback. But, no, I mean, it's true. I mean, sometimes, you know, on-call is a lonely job. Like, mobile engineer is a lonely job. Like, there's not that many of those. So, yeah, totally. But, anyway, so what did you guys individually do? Maybe we can kind of start there. How did you get pulled into the project? And we'll start from there.

Alexander [00:01:59]: Yeah, maybe I can go first, because then we have to have a fun story about how we started working together.

Alexander [00:02:05]: Okay, so, actually, before working at OpenAI, I was working on a native Mac OS software called Multi, which is, like, about, like, it was kind of like a pair programming tool, but we thought of ourselves as working on, like, human-to-human collaboration. And then, basically, as Chachapy and stuff came around, we started thinking about, like, oh, what if instead of a human pair programming with a human, it was, like, a human pair programming with an AI? So, I'll skip this whole journey, but that was this whole journey, and then we all ended up joining OpenAI. And I was working. We were mostly working on desktop software, and then we shipped reasoning models. And, you know, I'm sure you guys were, like, ahead of the curve in terms of understanding the value of reasoning models, but for me, like, it's kind of, like, starts off as better chat, but then when you can give it tools, you can actually, like, make it an agent, right? Like, an agent is, like, a reasoning model with, like, tools and environment, guardrails, and then maybe, like, training on, like, specific tasks. So, anyways, we got super interested in that, and we were just, like, starting to think about, like, okay, how do we bring reasoning models? Into desktop. And at the same time, here at OpenAI, there was a lot of experiments going on with, like, giving these reasoning models access to terminals. I wasn't working on those first experiments, to be clear, but those were, like, that was, like, the first true, like, wow, I really feel the AGI moment that I had. It was actually while I was talking to David Kaye, a designer who was working on this thing called Scientist, and he showed me this demo of it updating itself. And, like, nowadays, like, I don't know if any one of us, like, I don't know if any one of us, like, I don't know if any one of us would be, like, the most impressed to, like, change the background color.

swyx [00:03:37]: Modifying its own code?

Alexander [00:03:38]: Yeah. And then, you know, it was, like, they had hot reloading set up. So, I was just, like, mind blown at the time. And I don't know, it's still a super cool demo. And so, we kind of were experimenting with a bunch of these, and I sort of joined one of the teams that was, like, tinkering with this. And, you know, we kind of realized, like, hey, it's just, like, super valuable to figure out how to give a reasoning model access to the terminal.

Alexander [00:04:06]: And then, now, we have to figure out, like, how to make that, like, a useful product and how to make it safe, right? Like, you can't just let it go loose on your local file system, but that's, like, where people were initially trying to use it. So, a lot of those learnings ended up becoming the Codex CLI, which shipped recently. You know, a lot of the work there, like, you know, the thinking that I'm, like, most proud of is, like, enabling things, like, full auto mode. And, like, when you do that, we actually, like, increase the amount of sandboxing, so that's still safe for you. And then, so, you know, we were working on... You know, these types of things, and then we started realizing we want to let the model think for longer. We want to have a bigger model. We want to let the model, like, do more things safely, like, without having to do any approvals. And so, we thought, you know, maybe we should give the model its own computer, the agent its own computer. And then, at the same time, we were also experimenting with, like, putting the CLI in, like, our CI, so it could automatically fix tests. We did this, like, crazy hack to, like, get it to automatically fix, like, linear tests. We did this, like, really good center issue tracker. And so, then we ended up, sort of, like, creating this project that is Codex, which is basically, like, really the concept of giving the agent access to a computer. Actually, I realized that. I don't know if you were asking. Well, I personally did. But, anyways, I told the story. I hope that's okay. Sure.

swyx [00:05:21]: No, it's... I mean, you weave your personal story into the larger narrative anyway. But, yeah. And I'm sure Josh has a part, too.

Josh [00:05:28]: Yeah. Yeah. So, my story is somewhat different. I've had OpenAI for two months here. And, you know, I've been using it for a long time. And it's been one of the most fun, chaotic two months of my life. But maybe I'll start back at the company I had founded a few years back called Airplane. We were building an internal tool platform. The idea is to let you build internal tools, but really lean into developers and make that really easy. And it sounds unrelated, but in many ways, like, the similar themes started coming up. What's the right form factor for doing local development? How do you deploy tooling to the cloud? How do you run code in the cloud? How do you compose all these primitives of storage and compute and UI to let developers build software really quickly? I like to joke that we were just, I don't know, two years too early. Towards the end, we were playing around with, like, GPT 3.5 and, you know, trying to really make... It was really cool. It could actually build, like, a React view really quickly, right? And I think, you know, if we had kept going on it, maybe it would have turned into some of the AI builders that you see today. But that company ended up getting acquired by Airtable, where it ran some of the AI engineering teams there. And for me personally, towards the beginning of this year, I saw the progress we were making in software, agentic software development. And for me, it was a bit of, like, my own moon landing kind of moment that I suspected was about to happen, right? Whether or not I was involved in the next two years, I think we are going to build an agentic software engineer. And so I talked to my friend over there, and I was like, hey, are you guys working on something like this? And, you know, he gives me a wide-eyed look, he's like, I'm not allowed to tell you anything, but maybe you could talk to the team. And so, very fortunately, this was right when, you know, Alex and the folks were spinning off things. And I remember actually, you know, in our interview, we riffed on the form factor, right? Should it be CLI? The issues with that, waiting for it to finish and not being able to interrupt all the time. Wanting to run it four times, ten times in parallel. Right? And, you know, at that point, I said, maybe it should be both. And we sort of are, you know, going for that right now. But, yeah, I don't know, I'll just say, like, I was very excited and still very excited to just be pushing this forward. And I think Codex is still really early, excited to share it with the world, but there's a lot more to build. Yeah.

Alexander [00:07:52]: I'll say it was a very fun conversation when we first met, because he came in, I've never had this happen before. It was like, here's exactly, like, kind of the change that I see in the world. And therefore, the type of product that I want to build. I know you can't confirm if you're working on it, but just so you know, this is the only thing I want to work on. And then I was like, I asked just a few open-ended questions, and we immediately got into, like, some of the core debates around the form factor of the tool.

swyx [00:08:16]: And I was like, okay, this is awesome to work on. I think a DevTools person can spot another DevTools person like that.

Alessio [00:08:23]: Yeah. Blink twice if you're working on this.

swyx [00:08:26]: But for what it's worth, early iPhone team at Apple was the same, because iPhone team members did not know if they were on the same team, they're not allowed to tell each other. So they had to, like, triangulate. Wow.

Alessio [00:08:41]: And talking about form factor, so you mentioned the CLI, which you already released, and I think there's other, you know, Cloud Code, AIDR, a bunch of other tools out there. Should people think of Codex and ChargeGPD as like a hosted Codex CLI? Like, are there big differences between the two? Let's talk about that.

Josh [00:09:00]: Yeah.

Alessio [00:09:00]: Go for it.

Josh [00:09:01]: Yeah. I think of it as, I think that's a short of it, right? Allowing you to run Codex agents in OpenAI's cloud. But I think that the form factor, like, it's a lot more than just where the computer runs, right? It's how does this bind to the UI? How does this scale out over time? How do you manage caching and permissioning? And how do you do the collaboration story? And so let me know if you disagree. But I think the... Yeah. I think that really is like form factor is the core of it.

Alexander [00:09:35]: Yeah. It's like, it was pretty, it's been honestly a really fun journey. Like the other day, or maybe last night, like in the AM, Josh was sleeping because he had to do the live stream. I didn't have to. But anyway, a bunch of us were like looking back at the dock where we planned what we were going to ship. And we were like, man, we had a lot of scope creep.

Alexander [00:09:57]: And effectively, all that scope creep was kind of like incrementally made sense because we kept leaning further and further into it. And so I think the most important thing for us was like, you know, not just like a model that's good at coding, but rather this is an agent that is good at like independent software engineering work. And like, the more we lend into that, the more like things started to feel really special. So, you know, we could, I'm going to just label and then set aside like the entire conversation around like the compute platform that Josh has like, you know, been leading. But let's just take the model, for example. You know, we don't just want it to be good at code. And we don't just want it to solve, say, Swybench tasks. Swybench is an eval, for those who don't know, that has a certain way of functionally grading outputs. Because if you look at a lot of Swybench passing outputs from an agent, they're not really PRs that you would merge because the code style might be different. It works, but the code style is different. So we spent a lot of time making sure that our model is graded adhering to instructions, graded inferring code style so that you don't have to tell it. But let's say that you got then a PR, that it was like the code style was good, it followed your instructions well. It still might be really hard to merge if you have this enormous description, just like model of how it thought about building it. And you probably need to pull it onto your computer to test the change and validate that it works. And maybe that's okay if you're just running one change. But in a future world that we imagine where actually maybe the majority of code is actually being written by agents that we're delegating to, doing tasks in parallel. It becomes critically important that you can actually integrate those changes easily as the human developer. So, for instance, some of the other stuff we started to train was PR descriptions, like, let's really nail this idea of a good, concise PR description that highlights the relevant things. So our model will actually write a nice, short PR description with a PR title that adheres to your repo format. We have a way to prompt that more if you want with agents.md. And then in the PR description, it'll actually cite relevant code that it found along the way. Or write a code that it found along the way. Or write a relevant code in its PR so that you can, like, mouse over and just see it. And perhaps my favorite thing is actually the way we handle testing. So the model will attempt to test its change, and then it will tell you in this, like, really nice way with just, like, a checkbox kind of thing whether or not those tests passed. And again, it will cite if the test passed, like, a deterministic reference to the log. So you can, like, read it and be like, okay, I know that this test passed, right? Or if the test failed, it'll be like, hey, this didn't work. I feel like you need to install it. You need to install, like, PMPM or whatever. And you can, like, read the log and, like, see what it is. So those are some of the things that I think – I've lost track of the original question. But anyways, those are some of the things that we, like, have been really leaning into as we build this, like, basically, like, software engineering agent in the cloud.

Josh [00:12:47]: I think also just, like, it feels very different, right? You can, like, look at the features. But I think for me the feeling is, like, there's – it takes a leap of faith the first few times. You're like, I'm not really sure if this is going to work. And it goes off for, like, 30 minutes, right? But then it comes back, and it's like, wow, this agent went out, wrote a bunch of code, wrote scripts to help code mod its own changes, right, tested this, and it really went through the full end-to-end of thinking about the change it wants to make. And I, you know, had no faith at the start that it was going to be able to successfully do it. And then after using it a bit, you're like, wow, like, it actually, you know, pulled through. Yeah. Yeah. Yeah. Yeah. Yeah. It's something that's, like, hard to really, like, summarize. You have to really try it. But it finally feels very different. And, yeah, that feels special.

Alessio [00:13:38]: Yeah. I used it. I opened up VR for it a few minutes ago. I was in the lucky first 25% of people to get the rollout. Yeah, it's very nice. You know, it kind of shortcut it because it couldn't figure out how to run RSpec in Rails. And so it just checked the syntax of the Ruby file, and it was like, looks good to me. But I think it doesn't have the agents.md yet. So I think once I set that up, it'll be good.

swyx [00:14:01]: No, just don't use Ruby, man. Python is one. Oh, yeah. That's still issues.

Alessio [00:14:06]: Once it's good enough to migrate the whole thing, then I'll do that.

Alexander [00:14:09]: I mean, it is funny that there is, just briefly on the note of, like, don't use Ruby or not, there's, like, a bunch of things that I think teams can do to, like, make better use of AI agents. Oh, please. Stop using Ruby.

Alessio [00:14:20]: Number two. No, no, no.

swyx [00:14:21]: But, yeah, if you could list some things out that's, like, best practices, you know, I noted from the live stream that they mentioned ProUBI. So, like, users install linters and formatters so that, basically, these are in-the-loop verifiers that the agent can kind of use, right? Yeah. So, which, like, you know, it turns out to be dev best practices as well, but now the agents can auto-use it. Commit hooks have always been a tricky thing for humans because I've been on teams that were, like, no, everything has to have a commit hook. And then I've also been on teams that were, like, no, like, this thing gets in the way of committing, so let's rip everything out. But actually, for agents, it's actually really good to have commit hooks. Yeah.

Josh [00:14:58]: I mean, you took the words out of my mouth. I think the three I was going to say would be, one, agents.md. And, like, we put a lot of effort into making sure the agent, like, understand this hierarchy of instructions, right? You can put them in subdirectories, and it'll understand which ones take precedence over which others. So, over time, right, like, I mean, we also have, like, O3 and Fora writing our agents.md files for us. I love these.

swyx [00:15:24]: I love the tips. You actually open sourced the prompt descriptions here. Yeah.

Josh [00:15:29]: Yeah. Yeah. Anything you highlight? Yeah. I mean, I think I would start simple and not try to overdo it. And, like, a simple agents.md will get you a long way rather than no agents.md. And then it's more of, like, you learn over time, right? What we would really like to do is auto-generate this at some point for you based on the PRs you create and the feedback you give. But, you know, we figured we'd ship faster rather than later. But...

swyx [00:15:56]: As you mentioned, you have O3, O4 writing agents.md for you as well.

Josh [00:16:00]: Yeah. Like, I'll, like, give it my entire directory, right, and let it...and just say, like, hey, produce an agents.md. Well, actually, these days, I'm using code one to do it because it can...codex one, sorry, to traverse, like, your directory tree and generate those things for you. So, yeah, I would recommend slowly, gradually investing in agents.md. And then, you know, you took the words out of my mouth, like, getting very basic linting formatting up nets you really big wins. Because it's similar to how, like, if you open a new project in VS Code, right, like, you get some out-of-the-box checking. The agent's starting, like, as a human, you're sort of starting without that advantage. And so, this is trying to give that back to the...and, yeah, I don't know, do you have anything else? Yeah.

Alexander [00:16:47]: So, one analogy there. And then, actually, I have just some thoughts we've observed of even, like, using other coding agents, like, just any coding agent, you know, how to prepare for that. But, like, you know, the analogy that comes up. I kind of like is, like, you...so if you start with, like, a base reasoning model, actually, you basically have this, like, really precocious, like, incredibly intelligent, incredibly knowledgeable and, like, weirdly, spikily intelligent, you know, college grad, you know. And but we all know if you hire, like, that person and put them, like, ask them to do software engineering work, like, independently, like, there's just a lot of practices that they're not going to know about. And so, kind of a lot of what we've done with CodexOne is basically give them, like, some of it, its first few years of job experience and, like, that's effectively what the training is, right? So that it just, it kind of knows more of these things. And, like, if you think about it, like, a PR description is a classic example of that, like, writing a good PR description, right? And possibly knowing what not to put in it, actually, right? And then, so that's what you get there. So now you have this, like, this, like, weirdly, weirdly knowledgeable, spikily intelligent college grad with a few years of job experience and then every time you kick off a task, it's kind of, like, their first day at your college. And so, agents.md is basically a method for you to, kind of, compress that, like, test time exploration that it has to do so it can, like, know more. And as Josh said, obviously, we want to, like, right now, it's a research preview, so you have to, like, update it yourself, but there's, like, a lot of ideas we have for how to make that automatic. Yeah. So that's just a fun analogy.

Josh [00:18:15]: Yeah. Maybe the last one I'll say is, like, make your code base discoverable, all right? It's, like, the equivalent of maintaining good engineering practices for new hires that you make. Yeah. So, like, if you want to make your code base faster, right? A lot of my prompts start with, like, I'm working in this subdirectory. Here's what I'd like to accomplish, right? Can you please do it for me? And so, giving that guidance, that scoping, helps.

Alexander [00:18:40]: Yeah. Okay. I'll give you three, sorry, three things for, like, generally. So first, like, language choice. I was hanging out with a friend the other day who's a bit of a latecomer to AI, and he was, like, oh, yeah, I want to try building, like, an agent's product, like, should I build it in JavaScript? Yeah. Yeah. And I was, like, you're still using JavaScript? Like, no wonder. Like, you just, like, you know, use at least TypeScript. Like, give it some types. So, I mean, I think that's a basic one. I don't think anyone listening to us now needs to, like, be told this. Another one is, like, you just make your code modular, right? The more modular and testable it is, the better. But you don't even have to write the tests. Like, an agent can write the tests, but you kind of need to design the architecture to be modular, right? I saw this presentation recently by someone here who was, like, they weren't vibe coding. It was, like, professional software engineer. But using, like, tools like Codex to, like, build a new system. And they got to build a system from scratch. And there was kind of this graph of, like, their commit velocity. And then their system had, like, some traction. So then it was, like, okay, now we're going to port it into, you know, the monolith that is the overall ChatGPT codebase that, you know, has seen ridiculous hypergrowth and so maybe is not, like, the most architecturally preplanned. And, like, their commit rate, the same engineer, same tooling, actually the AI tooling continues to improve, their commit rate just, like, plummets, right? And so I think the other thing is just, like, yeah, architecture, like, good architecture is, like, even more important than ever. And, like, I guess the fun thing is, like, for now that's the thing that humans are really good at. So, like, you know, kind of good, you know, important for the software engineers to do their job.

swyx [00:20:08]: I don't know. Just don't look at my codebase. Yeah, well, definitely don't look at mine.

Alexander [00:20:12]: But the last thing is just kind of a fun story, which is the code name, the internal code name for our project is WAM, like W-H-A-M.

Alexander [00:20:21]: And we chose it, actually, I was working with a research lead, and he was, like, hey, make sure you grep the codebase before you choose the code name. So we, like, searched the codebase, and the string WAM was, like, only present, like, in a few larger strings and never present as its own string. And that means that whenever we prompt, we can be very efficient. We can just say in WAM, right? And then WAM code that is, like, you know, for our web codebase or our server codebase or in, like, our shared types or anywhere. It's, like, really efficient for the agent to find, right? Whereas let's pretend that, like, alternatively, we would have called our product, like, ChatGPT code, you know? Not saying we didn't consider that. Then it would be super hard for the agent to figure out where we wanted to direct it to. And so we'd probably have to provide, like, more relative, like, folder paths. So there's a lot of this stuff, like, as you start to think ahead, like, oh, I'm going to have an agent that's going to be using terminal to grep, then, you know, you can start, like, naming things intentionally.

Alessio [00:21:19]: Would you name? Would you start naming things? Would you start naming things less for humans' readability and more for agent readability? Like, what's kind of the trade-off in your mind?

Josh [00:21:28]: Yeah. It's interesting because I definitely had different priors come in at OpenAI, and, like, I currently believe that the systems are actually very convergent. Like, there's a lot of... Maybe it's because as long as you see humans and AI writing it, like, maybe there's a world where it's only AI that's maintaining a codebase and the substance change. But the moment you sort of have to break that fourth wall and a human is coming in, doing code review, deploying the code, like, it has human fingerprints all over it, right? And so how humans communicate to AI, where to make the changes, how humans communicate a bug that needs to be done, or communicate business requirements, right? All those things aren't going to go away immediately. And so I think the whole system still feels actually very human. I think there's a cooler answer I could say that's like, oh, no, it's like this alien thing. It's completely different. But I don't know. I think it's like these are started off as large language models. There's a lot rooted in AI. There's a lot rooted in human communication. Yeah.

Alexander [00:22:25]: By the way, if there's somewhere you want to take this, you should actually cut us off because I realize we're just kind of monologuing between each other. But it's...

Alessio [00:22:30]: No, no. I think this also ties to the agents.md, right? It's like, why is it called agents.md and not readme.md? There's kind of like, I guess, in your mind, some fundamental difference with how the agents and the human consumes the information. So I'm curious if you think that's at the class naming level, it's just at the instruction level. Like, where does it break down?

Alexander [00:22:49]: Mm-hmm. Yeah. So this is like a few options for this naming, right? Which we considered. So you could go for readme.md. You could go for contributors.md, right? You could go for like codexagent.md and then like maybe like codexcli.md is like these two separate files, right? But like that are sort of like branded.

swyx [00:23:08]: There's also cursor rules, widthsurf rules, everyone has rules. Yeah. Yeah.

Alexander [00:23:12]: And then you could go for agents.md, right? And so like there are a few trade-offs here. I guess one is like openness and one is specificity, I suppose. And so when we thought about it, you know, we thought about, well, probably there's things that you want to tell an agent that you don't need to tell a contributor. And similarly, there's things you want to tell contributors to like really like help them set up in your repo or whatever that you don't need to tell the agent. The agent can just figure that out. So we were like, okay, maybe this is going to be different. And like, you know, the agent's going to read your readme anyways. So like maybe agents.md ends up being like the stuff that you need to tell the agent, that it's not like automatically figuring out from the readme. So we kind of made that decision. Then we considered, okay, there are like different form factors of agents, right? Like the most special thing about what we are building and shipping is like, it's just like an out of the box way to use a cloud-based agent that can do many tasks in parallel and can like think for a long time and can use a lot of tools safely, right? And so we thought like, well, you know, how fundamentally different is the set of instructions that you want to give that from an agent that you're working with more collaboratively? On your computer? We had a good amount of debates about that, to be completely honest. And then we ended up concluding like, actually, those sets of instructions aren't different enough that we need to namespace this file. If there is something you need to namespace, you could probably just like say it in plain language within the file. Then the last thing we consider is like, well, okay, how different do we think the instructions you have to give like our agent are to the instructions you might give to an agent running on a different model or built by a different company? And we just think it kind of sucks if you have to like create like all these different agents or whatever. You know, it's like part of why we made the codec CLI open source is like a lot of problems like safety issues that you need to figure out for how to deploy these things safely. And no one should have to figure these out like more than once. So that's why we went for like a non-branded name.

Josh [00:25:01]: And I have one specific example of why, you know, readme and agent.md are different. For agents, I don't think you really have to tell it code style. It looks at your code base and writes code that's consistent to that, whereas like a human's not going to take its time, sorry, their time to go through the code base and, you know, follow all the conventions, right? So that's just one example. Like, you know, at the end of the day, like there are differences between how these two kinds of developers approach it.

swyx [00:25:38]: Cool. I think that's a really good set of advice. I think you just gave us our episode title, like we're just going to call it best practices for using ChatGPT codecs. Isn't it? You know what I mean? We are going to want best practices. So I noticed something that's very interesting, right? I think there's always two versions in terms of building agents, one which is you try to be more controlling, you try to make it more deterministic, and then the other, you try to just prompt it and trust the model. And I think your approach is very much prompt it, trust the model. I see inside of the agent.md system prompt that, you know, you just prompt it to like behave the way that you want and you just expect the model to behave it. Obviously, you have control of the model so you can train it if it doesn't do well. But like one thing that makes me question it is how do you fit everything in context? Like what if I just have a super long HSMD, you know, in your live stream you had it demoing on like the OpenAI model repo, which is just giant, right? So how do you manage caching and context windows and all that?

Josh [00:26:48]: Yeah. I mean, would you believe it? Would you believe me if I told you right now that it all fits in the context window?

swyx [00:26:55]: Not the OpenAI repo. No, but sorry, everything that the agent needs to... Right. So you reify the HSMD, you put it at the top, right? It's just like another system prompt.

Josh [00:27:05]: No, actually, it's a file that the agent knows how to like grep and set for, right? Okay. Because there might be multiple ones. And so you can actually see it in the work log, right? It's like going to look for it. It very aggressively looks for an agent and it's been trained to do that. I'll say it's been really interesting joining OpenAI and like seeing how, you know, when you're thinking about where models are going and what AI products will look like years from now, you know, you design products in a different way, right? Like before OpenAI, especially when you don't have access to a team of researchers and many main GPUs, you're building these deterministic programs, right? A lot of scaffolding around how to do that. But you don't really let the model operate at its fullest capacity, right? It was interesting when I just joined, actually, I got a lot of pushback saying like, hey, why don't we just like hard code, like, listen, you keep using this tool wrong. Let's just say in our prompt, don't do that. And then the researchers will be like, no, no, no, we don't do that. We're like, we're going to do it the right way. We're going to teach the model why this is the right way to do it. And I think that's like related to this overall thought. Like, where do you put the deterrent? Where do you put the deterministic guardrails in? And where do you really let the model think, right? Similar conversation around planning. Should we just have an explicit planning stage where it's like, think out loud first, write down what you're going to do, and then go do it? Sure. But what if the task is really easy? All right. Do you really want to think of this whole time? What if it needs to like replant as it goes? Like, do you have all these like if else conditions, heuristics to do that? Or do you train a really good model that knows how to switch between those modes of thinking? And so it's tough. Like, I definitely have advocated for a lot of these things. Like, little guardrails here and there until like the next training runs done. But I think that's really like, we're really building for this future where the model is able to make all these decisions. What's really important is that you give it the right tool, right? You give it ways to manage context, manage memory, manage, you know, ways to explore the code base. Those still are really important.

Alexander [00:29:11]: Yeah. That's like, yeah, super well said, like, I think building here is like, super fun and different. And like, you know, the model isn't all the product, but the model is the product, right? And you kind of like need to have this kind of like humility in terms of like thinking about like, okay, well, what are the things that like, we, there's like three parties, right? There's the user, the developer and like the model maybe, right? What are the things that the user like just needs to decide up front? And then what are the things that like we, the developer are going to be able to decide better than the model? And then what are the things that the model can just decide best, right? And like that. You kind of, every decision is like, just has to be one of those three and you know, it's not like everything's the model. Like for instance, we have two buttons in the UI right now, like ask and code and like, you know, those, those probably could get inlined into, into like the decisions the model makes. But you know, right now it was just really like, it made sense to kind of just give the user choice upfront because we spawn a different container for the model first, based on what button you press. So like if you, if you ask for code, we put all the dependencies in, I'm going to oversimplify here. But if you don't ask for code, if you're just asking a question, we like do a much quicker container setup before the model like gets any choice. And so, you know, that's maybe a user decision. There's some places where, you know, user and developer decisions kind of come together around the environment. But like ultimately a lot of like, a lot of like agents that I see are really impressive, but it's basically like part of what's impressive is it's like a bunch of developers building, building this like really bespoke state machine around a bunch of like short model calls. And so then the upper bound of like complexity of problem that the model can tackle is kind of actually just what can fit in the developer's brain. Right. And over time we want these models to capture like, or to, to solve for much more complex problems, you know, like just by themselves on like more and more complex individual tasks. And then eventually you could really imagine that you get like a team of agents working together maybe with like one agent that's kind of managing those agents and you know, the complexity just explodes. And so we really want to like get as much of that complexity, as much of that state machine as possible, like pushed into the model. And so you end up with these kind of two modes of building, like in one place you're like building product, UI and rules. And in the other case, you're, you still have to do work to get the model to learn something, but rather what you have to do is you have to figure out like, what are the right things that this model needs to see during its training to like learn something. And so it's still a lot of human work to like figure out how to get that change, but it's like a very different way of thinking of like, we're going to get the model to see this.

Alessio [00:31:43]: But how do you build the product to get the signal? So if you think about the code in ask, it's almost, you're basically getting the user to label the prompt in a way, right? Because they say, ask, this is an ask prompt code. This is a code prompt. Are there any other kind of like fun product designs, like as you built this of like, okay, we think the model can learn this, but we don't have the data. This is how we architect codecs to kind of help us collect the data.

Josh [00:32:09]: I think file context and scoping is, we don't have great built in things. Like that right now. But it's like one of the obvious things that we need to add is another example of this, right? Like you could have, we're often usually pleasantly surprised as, oh, it was able to find the exact file that I was thinking about, but it takes some time, right? And so a lot of times you'll shortcut a bunch of chain of thought by just saying, hey, I'm looking at this directory, can you go through? So I think that'll probably be there for a bit until, you know, you have some better architectural indexing and search capabilities. Okay.

Alexander [00:32:44]: Yeah. I'll add to this. Like, I'm actually going to double down on my thing about like, where do we, how do we think about it? Like, so, you know, one thing we might consider is like context window management, right? And like, should we intervene here? And so we could do a product intervention, right? Like write some code to intervene and then kind of the next level of thinking maybe like a little bit more AGI pilled is like, okay, let's, let's get the model to like see context window management stuff in this training. I can't even come up with an example now at this point, cause I'm too AGI filled, but like, I don't know. We could come up with something that it has to see to like learn how to manage its context. Like, but it's like a specifically tasks related to context window. But then the most AGI pilled thing to do is like, to be like, we don't actually need to think about this problem. The model will just figure it out. All we have to do is give it harder and harder problems. And then like, it will just have an emergent property of managing its own context. Because that's the only way it can like solve these problems. Right. So I'm kind of slightly like oversimplifying here, but like, you know, basically the model learns to manage its context. And so you, when you were talking about like it working in the monorepo. It learns how to be like efficient with the way that it spends its tokens as it's like browsing and like setting and like, you know, in your example of like, there's a giant agents.md, like, I guess we, you know, it would just need, we would just to show it some versions where there was that. And so it learns it can't, it shouldn't read the whole thing every time and should like first figure out how many lines it has, et cetera. So anyways, summarizing, I'm like, we just need to keep giving it harder and harder problems. And a lot of these things that we might be very tempted to like build a sub intervention for, like, it will just have to figure out and if it doesn't figure it out, maybe it didn't matter.

swyx [00:34:15]: Sure. Yeah. Uh, I totally get, uh, that I think like we don't really have online models yet. Right. And that's kind of what you need for, for your vision to be real. Um, and for what it's worth, I wasn't thinking about like a giant agents.md, I was just thinking about like hierarchical nested agents.md with, with like a lot of code, um, and like, I think one issue where you have this version where the model is the product is your, your dev cycle as the codex team. Right. Like you have to kind of, it's not as tight because you have to be like, okay, every time there's a bug. All right. Now I need to go get data. Um, and where do you get the data? I don't, I don't know. Like maybe employees use it. Maybe you have like, you buy it from vendors and like you hire some human raters or whatever, and then you have to train it in and then you have to go test it again. It's very slow, isn't it? And like expensive. Yeah.

Alexander [00:35:09]: I think it's definitely, yeah, I think it's definitely like from a, from a building perspective, you, you have to do this. When you're really willing to play the, like the long-term vision of like, we're going to build a better model, maybe even a better model bespoke for a certain like functional purpose like codex one. And then we're going to generalize the learnings from that model into like an even bigger model. That's like getting all these other like learnings from other functional purposes and like these together will like become a really powerful thing. And that's, that's like kind of like the philosophy we've have with training models so far. Um, and it has been working, but it's, it's definitely like a long-term play, you know, like another example, we, we do, we do do this on occasion. Like for example, uh, recently we released GPT 4.1, like really good coding model. And again, that was like based on like working, we were like, Hey, we want to invest better in this area. Let's hang out with a bunch of developers, understand their feedback, how things work, um, you know, create some evals. And you know, like you said, this is like, it's a lot of work to do that. Um, but then we end up with a great model and even more exciting, we can then like take those learnings and like put them into our mainline model. And then everything benefits and you kind of the, the, the, the sort of philosophical view. I don't know if I can like factually prove it or not. Maybe someone here can, is that like, if you can like build, do something very specific for like a specific purpose, actually, when you bring that and you bring it to the, into the generalized model, like you might even get outsized returns on that because there's like transfer from all these different domains.

swyx [00:36:37]: Um, okay, cool. Uh, I think we had a couple factual things to wrap up on, on just like Codex itself. And then we wanted to double click on the compute platform stuff, which, uh, I think, uh, Josh, you wanted to, to cover more on. Um, so I, I noticed in the details, it was, um, between one to 30 minutes in length. Um, is it, is that a hard cutoff? Um, have you had it go, go for longer? Um, any, any comment on the, the task time?

Josh [00:37:02]: Yeah. I mean, I just checked the code base before this, uh, cause someone else has a similar question. Uh, are hard cutoffs an hour right now? Although don't hold us to that. It may change over time. Um, the longest is I've seen two hours when, uh, in development mode and the model one, off the rails. Um, so, you know, I, I, but I think 30 minutes is a great ballpark for the kind of tasks that we're trying to solve, right? These are hard tasks that require a lot of iteration, uh, and testing and, uh, the model needs that time.

Alexander [00:37:31]: Yeah. I mean, yeah, I, I think actually like our average is like pretty, it's significantly lower than 30. Yeah. But if you give it a hard, if you give it a hard task, you'll, you'll end up at 30.

swyx [00:37:40]: Yeah. I mean, um, you know, I think there's, there's a couple analogies here. One. Uh, I think the operator team released a benchmark where they, they had to cut off for two hours. And then the other one is the meter paper, which I don't know if has been circulating where they estimated that the, uh, the current average autonomous time is like an hour and it's maybe doubling every seven months. Um, so like an hour sounds right, but also, I mean, that's the median. So there's going to be some that are, that go longer than that. Yeah, totally. Is this part of the, uh, you had cutoffs for a few, like 23, um, sweet bench, very few, very rare examples that were not runnable. Was that part of it, uh, in terms of length or was there, it was just something else.

swyx [00:38:20]: Yeah.

Alexander [00:38:21]: Yeah. To be honest, I'm not exactly sure, but I feel like there's a bunch of sweet bench cases that actually are like an invalid might be too strong of a word and a little bit out, you know, not sure, but like, I feel like there's like issues with running them. So they just don't work. Okay. Yeah.

swyx [00:38:35]: Um, and then max concurrency, is there a concurrency limit? If I have like five, 10, a hundred simultaneous.

Alexander [00:38:41]: Five and 10 is totally fine. Do we actually have a, I feel like we did like introduce like a. A limit for fraud reasons. Like, I don't know what it is. Yeah.

Josh [00:38:47]: I think right now it's 60 an hour. Um, but. Wow.

Alexander [00:38:53]: So one per minute, I'm just going to, yeah, but look, this, this is literally the point, right? Like it is. So, so like longterm, we actually don't want you to have to think about if you're like delegating or like pairing with AI, like if you imagine an AGI, like super assistant, you just like ask, you just talk to it and it just does stuff. It answers quickly if it needs to, you know, it takes a long time and you also don't have to like only talk to it. It's also just like present your tools. Right. So that's, that's like the longterm thing, but in the near term, like, yeah, this is a tool you delegate to. And um, the way to use it that we see, like, you know, going back to the, I guess the, maybe the title of this podcast of like best practices, it's like, you must have an abundance mindset and you must think of it as like, like not using your time to explore things. And so like, you know, often when something, a model is going to work on your computer and it's going to work on your computer, you're like really craft the, the, the prompt because, you know, then it's, it's going to use your computer for a while and maybe you can't. Um, but the way we see people who like love Codex the most using it is they don't, they think for like maybe 30 seconds max about their prompt. It's just like, oh, I have this idea, like, boom, oh, there's this thing I want to do like, boom. Oh, like I just saw this bug or like this customer feedback thing, like, and you just send it off. And so, yeah, like the more you're running in parallel, actually, I think that, I mean, the happier we are and like the happier we think like users are when they see it, like, that's just the vibe of the product really.

swyx [00:40:13]: I would, I would pass my own anecdotes. So I was on the trusted testers team for this thing, uh, as you both of you well know, and I was using, I found out I was using it wrong. I was using it like cursor, like I had my chat window open and I watched it code and then I realized I wasn't supposed to, and I was like, oh, like you guys are just firing the things off and like, you know, going on about your day. And, um, yeah, that, that, that was a change in mindset. Yeah. One. Yeah.

Alexander [00:40:39]: Real quick. I'll keep it brief. Like one thing that's quite fun is like, use it on your phone because somehow just like being on your phone, just like flips the way people think about things. So like we made the website responsive and you know, we'll make, we'll pull it into the app eventually. Um, so like try it, like it's actually like super fun and satisfying. Okay.

swyx [00:40:56]: So yeah, it's not, yeah, cause there was a, there was a voice, uh, there was one of the videos that was showing the mobile engineer coding with it on his phone, but it's not available in ChatGPT's app. Okay. Yeah. Yeah.

Alessio [00:41:08]: Yeah. Not yet. Just one question I got from the mobile. I got the notifications that I get, uh, when it starts the task, it says starting research the same way, the deep research notification is, uh, is it using deep research as a tool or did you just reuse the same, uh, notification that we just used the same notification? Yeah. So you mentioned the compute platform. You mentioned how you share some of the infrastructure with RL. Can you maybe just give people a high level of like what the codex has access to? What it doesn't have access to? Like it doesn't look like people can run commands, uh, themselves. They can only instruct the model to do it. Um, any other things people should keep in mind? Yeah.

Josh [00:41:48]: So, and I'll say, uh, it's a, it's an evolving discussion as we figure out what parts, uh, we can give folks access to and the agent and vice, like, and what we need to like hold back for, for now. Right. And so we're learning and it's really, we would like to give, uh, humans and agents alike as much access as possible within safety and security, uh, constraints. Um, what you can do today, right, is as a human, um, set up an environment, set up scripts that get them run. Uh, these scripts typically will be installing dependencies. I expect that to be maybe 95% of the use case there, uh, and just really get all the right binaries in place for your agent to use. Um, we actually do have like a bit of an environment editing experience, whereas a human, you can drop into a REPL, try things out. Um, so, you know, um, uh, please don't abuse it, but, uh, there's definitely ways, uh, for you to interact with the environment there.

Alexander [00:42:43]: Um, we, we laugh about that because like earlier I mentioned scope creep, like we weren't planning on having a REPL, like to like interactively update your environment, but like, you know, anyway, we try, Josh was like, oh man, we need this. And so that was like an example. Scope creep. Thanks for doing it.

Josh [00:42:57]: We do have rate limits in place and we do monitor that very carefully, but you know, there's interactive bits of there to like, to, to get that going. But, um, once the agent starts running, right, um, what we actually do today, and we're hoping to like evolve on this is we'll, um, cut off internet access, uh, because we still don't fully understand what letting loose an agent in its own environment is going to do. Right. Um, you know, for now the safety tests all have come back very sturdily, like, you know, it's not susceptible to sorts of certain exfiltration attempts on prop injection, but there's still a lot of risks to this category. So we don't know. And, um, that's why to start. Yeah. I mean, it's, it's not conservative there. And when the agent's running, it doesn't have a full network access, but you know, I love to be able to change that, right. Allow it to give limited access, certain domains or certain, uh, repositories. And so all this to say, it's like something we're evolving as we build out the right systems to support that. Um, not sure that quite touches on your original question. Um, the last thing though, that I do want to mention is like, there's an interactivity, um, element with like, you know, as the agent's running. Yeah. Sometimes you're just like, oh, I want to like correct it, tell it to like go somewhere else or, um, you know, let me maybe fill this part out and, uh, then you can take back over. Right. We haven't quite solved those problems either. What we really wanted to start was to like shoot for the fully independent, just like deliver massive value in one shot kind of approach. Um, but yeah, we're definitely thinking about how we can, uh, weave human and agents together, um, better.

swyx [00:44:34]: Uh, I mean, uh, for what it's worth. Yeah. I think the one shot thing is, um, a good angle that, that the other, the other people, you know, like, uh, and this is me comparing you to, to alternatives like Devin and factory and all the others. Um, there are more focused on multi-shot human feedback, all these, but like, um, you know, I, so I, I have a web website I'm working on and I, and I gave it a request and I compared it all the others. That was my test for, uh, Codex, uh, and it did one shot it. I posted it. Uh, uh, the screenshot is a tweet, uh, just earlier today. Okay. Um, and it's, uh, I think it's really good, especially if you're running 60 at a time. so I think that, that really makes sense, but it, it's, it is a very ambitious goal because like, um, you know, human feedback is a crutch that we like to use. Um, it also, I think makes us write more tests, which is annoying cause I don't like to write tests, but now I have to write tests. Fortunately, I can, I'm now getting Codex to write my own tests and I really liked the, on the, on the live stream as well. You can just kind of ask it to just look at your code base and just suggest stuff to do. Cause I don't even have the energy to figure out what I should be doing.

Alexander [00:45:40]: Yeah. I delegated delegation. I thought that was a great line. Yeah.

Josh [00:45:43]: Um, and we're not saying one form factor is better than the others, right? Like, you know, I love using Codex CLI, um, and, uh, it's really, we just want, like, as we talked about in our interview, when I was interviewing over an AI, like you really want both modes. Um, but I think what we see as like the role of Codex here is to really push the frontier. Yeah. On that sort of single shot autonomous, um, software engineering.

Alexander [00:46:10]: Yeah. Like, I kind of think of the research preview as like our, like our thought experiment. It's like, you know, what, what is coding agent like in its purest, like most like AGI pilled or scale pilled, scale pilled form look like. And then maybe, I mean, for me personally, like, I dunno, like part of what excites me about working at OpenAI, it's not just like solving for developers, but it's just really thinking about like, how does AGI like benefit all of humanity and like, what does that feel like to non-developers as well? And so like, for me, like what's really interesting is like thinking of Codex as an experiment for like what it'll feel like to be in other functions, um, you know, doing work. And like the goal for me to, to build towards is this is a vision where it's like, you know, we do the work that's like ambiguous or creative or hard to automate in whatever way. And, but otherwise we just have like agents, like that we're delegating most of the work to. Um, but these agents, they're, they're not like this, like long horizon thing versus short horizon. They're just like kind of ubiquitously available. Um, why? Because they're not with you. Um, so, uh, yeah, we, we decided to like take the purest form to start, which we thought would be the smallest scope thing to ship and probably isn't. Uh, but, uh, yeah, and then we're going to bring these things together.

swyx [00:47:18]: Uh, okay. I think we have time for a couple of questions. Um, we, you, I'm just going to double click on the research preview a bit. Um, it is a research preview. Why? What, what is left? What do you think would qualify it to be a full release? Um, you know, on, on the live stream, Greg mentioned this seamless, uh, journey. Um, you know, the transition between cloud and CLI, uh, is it that, or is there, is there, are there other things on your mind?

Alexander [00:47:42]: I mean, to be completely honest, you know, we, the part of why we believe so much in iterative deployment, like I can give you some of my thoughts now, but like also we're really curious to see. Um, so because this is like such a new form factor. Um, but you know, some of the items that are top of mind for me are like multimodal inputs. you know, we've, we've talked. Yeah. I know you like that. Right. Yeah. Like another, another example would be like, you know, just giving it a little bit more access to the world. Um, you know, a lot of folks have, have requested for like forms of network access. Um, you know, I also think that, um, right now kind of the, the UI that we shipped is actually one that we iterated around. It's like a fun story there, but like, um, and it's one that people find useful, but it's, it's definitely not the final form of what it is. And like, we would love for it to be like much closer with the tools that developers spend time in. So like, those are some of the themes we're thinking about. Um, but you know, to be clear, we'll iterate and figure that out.

Alessio [00:48:35]: Um, I wanted to ask, why did you put finding a typo as one of the onboarding things? Because I used it and then I saw it and it's literally just grepping for potential type. It's like searching, grepping for like Selenium with an N or like something with like some T G N instead of N G, but it went through like 50 of these and then finally found will misspell this W I L without the two thing. But it was really cool to see what it thought, like default, uh, spelled as like D E F U A L T. It's like, it just grabbed all these different things. And then eventually got there. Like, why, why, why did you make that task?

Josh [00:49:20]: Honestly, T when I were talking about it and he was like, listen, it would be funny if I had a typo as I typed this prompt out and, uh, just make it a little bit meta, um, you know, nervous fingers on a, uh, on a live stream. Uh, so maybe optimized for the, for that. Okay. I have noticed it. Um, you know, it likes to do T E H to look for the, um, and so it's a work in progress.

Alessio [00:49:45]: That was great. Um, any parting thoughts called to action? Are you growing the team? Do you want a specific feedback from the community?

Josh [00:49:57]: Yeah. I think for me, like the one thing that, um, you know, um, is really on my mind for getting better at, um, over the next few months is really helping you customize your environment, uh, more, um, in a more high fidelity manner. Um, it turns out the good news is the agent can do a lot of really good work with only the basics, right? Um, it's much like if your dev machine is borked and you're sort of looking at your editor, but none of the type checks are working. You, a lot of folks can still actually do a lot of good work, but, um, how do you get close that last, you know, 30, 40%. And it's really hard because, uh, there's such a wide variety of environments out there. Um, but especially would love feedback from folks on how they would like to see their environment change. How do they want to see their environment customized? Do they want to just ship us a darker image where they rather have a support dev containers, right? Um, so the form factor of how you do the DX of how you do, um, environment customization still very much an open question that we need to improve on.

Alexander [00:50:53]: Yeah. Big plus one to that. And I think for me, maybe the thing that I'm most interested in is, Hey, this is, this is like a new shape of, of tool to collaborate with. Uh, and I'm just really interested for people to try working with it in as many different ways as possible. And like, kind of figure out like, where does it work well in your workflow? Like you said, you know, you mentioned you earlier, you were trying to use it kind of like your IDE and then you realized it was different. Right. So I would just love for people to take advantage, especially now, like we're very intentionally just providing like very generous rate limits. So people can try it. Like, we just want you to try it and figure out what sticks, what works, what doesn't, how do you prompt it? And then we want to learn from that and use that to lean in. So yeah, my, my, I guess my parting call to action is like, please go try it out, um, in ChatGPT. Use it as much as you can, especially now, and then let us know, uh, like how you like to hold it. Basically.

swyx [00:51:44]: Yeah. I'm, I'm, uh, worried about the pricing when it happens, but yeah, I'm going to abuse this. Why not? Why not? Yeah. Send us feedback on pricing too. Yeah. Um, okay. Uh, you're pro it's too early to talk about pricing, right? Uh, yeah, it's too early now. Yeah. Okay. All right. Um, but, uh, yeah, based on cloud code, that's the thing that people are worried about. Right. Um, and cloud has started to introduce some kind of fixed pricing. Yeah. And I think it's a, it's a huge mess. Like it's, there's no right answer. Everyone just wants the cheapest form of code attention they can get. Um, so yeah. Good luck. Thanks.

Josh [00:52:19]: I mean, my take is, I don't know, it's going to make it into, but like we aim to deliver out a lot of value. Right. And it's on us to show that and really make people realize like, wow, this is like doing very economically valuable work for me. And I think a lot of the pricing can fall from that. Um, but yeah, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, I think that, I think the, that's where the conversation should start. Um, are we actually delivering that value?

swyx [00:52:44]: Yeah. Awesome. All right. Well, thank you so much. Yeah. Thanks for working on this and thanks for sharing your time. Um, it is, uh, it's been, it's been a long time coming, but I think people can start seeing. OpenAI in general, is getting very serious by agents. It's not just coding, but coding obviously is the, the one group that is self-accelerating that I think, um, I've, obviously you guys are super passionate about. It's really inspiring to see.

Alexander [00:53:08]: Yeah. Yeah, super excited to just, like, ship everyone this coding agent and then, yeah, like, bring it together into just, like, the general AGI super assistant. Yeah, so thanks for having us on.

Alessio [00:53:19]: Thank you, guys.

Alexander [00:53:20]: Cool. Thank you.

Last few days for AI Engineer early bird tickets - most of the speakers have been announced! Limited tix for volunteers are available, and you can also get discounted tickets via the official AIE Hackathon and the State of AIE survey! The AIE Expo has sold out but we have opened a few slots for the Hackathon (winners present at AIE)!

The AI coding wars have now split across four battlegrounds:

• AI IDEs: with two leading startups in Windsurf ($3B acq. by OpenAI) and Cursor ($9B valuation) and a sea of competition behind them (like Cline, Github Copilot, etc).

• Vibe coding platforms: Bolt.new, Lovable, v0, etc. all experiencing fast growth and getting to the tens of millions of revenue in months.

• The teammate agents: Devin, Cosine, etc. Simply give them a task, and they will get back to you with a full PR (with mixed results)

• The cli-based agents: after Aider’s initial success, we are now seeing many other alternatives including two from the main labs: OpenAI Codex and Claude Code. The main draw is that 1) they are composable 2) they are pay as you go based on tokens used.

Since we covered all three of the first categories, today’s guests are Boris and Cat, the lead engineer and PM for Claude Code. If you only take one thing away from this episode, it’s this piece from Boris: Claude Code is not a product as much as it’s a Unix utility.

This fits very well with Anthropic’s product principle: “do the simple thing first.” Whether it’s the memory implementation (a markdown file that gets auto-loaded) or the approach to prompt summarization (just ask Claude to summarize), they always pick the smallest building blocks that are useful, understandable, and extensible. Even major features like planning (“/think”) and memory (#tags in markdown) fit the same idea of having text I/O as the core interface. This is very similar to the original UNIX design philosophy:

Claude Code is also the most direct way to consume Sonnet for coding, rather than going through all the hidden prompting and optimization than the other products do. You will feel that right away, as the average spend per user is $6/day on Claude Code compared to $20/mo for Cursor, for example. Apparently, there are some engineers inside of Anthropic that have spent >$1,000 in one day!

If you’re building AI developer tools, there’s also a lot of alpha on how to design a cli tool, interactive vs non-interactive modes, and how to balance feature creation. Enjoy on your podcast app or YouTube!

• Cat Wu

• Boris Cherny

• Claude Code

• Aider

• MCP creators episode

• Husky

• TMUX

• Bitter Lesson

• Cyclomatic Complexity

• Interpretability Research by Chris Olah

• [00:01:59] Origins of Claude Code

• [00:04:32] Anthropic’s Product Philosophy

• [00:07:38] What should go into Claude Code?

• [00:09:26] Claude.md and Memory Simplification

• [00:10:07] Claude Code vs Aider

• [00:11:23] Parallel Workflows and Unix Utility Philosophy

• [00:12:51] Cost considerations and pricing model

• [00:14:51] Key Features Shipped Since Launch

• [00:16:28] Claude Code writes 80% of Claude Code

• [00:18:01] Custom Slash Commands and MCP Integration

• [00:21:08] Terminal UX and Technical Stack

• [00:27:11] Code Review and Semantic Linting

• [00:28:33] Non-Interactive Mode and Automation

• [00:36:09] Engineering Productivity Metrics

• [00:37:47] Balancing Feature Creation and Maintenance

• [00:41:59] Memory and the Future of Context

• [00:50:10] Sandboxing, Branching, and Agent Planning

• [01:01:43] Future roadmap

• [01:11:00] Why Anthropic Excels at Developer Tools

Alessio [00:00:04]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host Swyx, founder of SmolAI.

Swyx [00:00:12]: Hey, and today we're in the studio with Cat Wu and Boris Cherny. Welcome. Thanks for having us.

Cat [00:00:17]: Thank you.

Swyx [00:00:18]: Cat, you and I know each other from before. I just realized Dagster as well. Yeah. And then Index Ventures and now Anthropic. It's so cool to see a friend that you know from before now working at Anthropic and shipping really cool stuff. And Boris, you're a celebrity because we were just having you outside, just getting coffee, and people recognize you from your video. Wasn't that neat?

Boris [00:00:43]: Yeah, I definitely had that experience like once or twice in the last few weeks. It was surprising.

Swyx [00:00:48]: Yeah. Well, thank you for making the time. We're here to talk about Claude Code. Most people probably have heard of it. We think quite a few people have tried it.

Boris [00:00:56]: But let's get a crisp, upfront definition. What is Claude Code? Yeah. So Claude Code is... What is Claude in the terminal? So, you know, Claude has a bunch of different interfaces. There is desktop, there's web, and yeah, Claude Code, it runs in your terminal. Because it runs in the terminal, it has access to a bunch of stuff that you just don't get if you're running on the web or on desktop or whatever. So it can run bash commands, it can see all of the files in the current directory, and it does all that agentically. And, yeah, I guess maybe it comes back to, like, maybe the question under the question is, like, you know, where did this idea come from? And, yeah, part of it was, we just want to learn how Claude, we want to learn how people use agents. We are doing this with the CLI form factor, because coding is kind of a natural place where people use agents today. And, you know, there's kind of product market fit for this thing. But, yeah, it's just sort of this crazy research project. And obviously, it's kind of bare bones and simple. But, yeah, it's like an agent in your terminal. That's how the best stuff starts.

Alessio [00:01:59]: Yeah. How did it start? Did you have a master plan? Did you have a master plan to build Claude Code?

Boris [00:02:04]: There was no master plan. When I joined Anthropic, I was experimenting with different ways to use the model kind of in different places. And the way I was doing that was through the public API, the same API that everyone else has access to. And one of the really weird experiments was this quad that runs in a terminal. And I was using it for kind of weird stuff. I was using it to, like, look at what music I was listening to and react to that. And then, you know, like, screenshot my, you know, video player and explain. Like, what's happening there and things like this. And this was, like, kind of a pretty quick thing to build. And it was pretty fun to play around with. And then at some point, I gave it access to the terminal and the ability to code. And suddenly, it just felt very useful. Like, I was using this thing every day. It kind of expanded from there. We gave the core team access, and they all started using it every day, which was pretty surprising. And then we gave all the engineers and researchers at Anthropic access. And pretty soon, everyone was... I was using it every day. And I remember we had this DAU chart for internal users. And I was just watching it, and it was vertical, like, for days. And we're like, all right, there's something here. We got to give this to external people so everyone else can try this, too.

Alessio [00:03:17]: Yeah.

Boris [00:03:18]: Yeah, that's where it came from.

Alessio [00:03:19]: And were you also working with Boris already? Or did this come out, and then it started growing? And then you're like, okay, we need to maybe make this a team, so to speak.

Cat [00:03:28]: Yeah, the original team was Boris, Sid. And Ben. And over time, as more people were adopting the tool, we felt like, okay, we really have to invest in supporting it, because all our researchers are using it. And this is, like, our one lever to make them really productive. And so at that point, I was using QuadCode to build some visualizations. I was analyzing a bunch of data, and sometimes it's super useful to, like, spin up a streamlet and, like, see all the aggregate stats at once. And QuadCode made it really, really easy to do. So I think I sent Boris, like, a bunch. at some point, Boris was like, do you want to just work on this? And so that's how it happened.

Boris [00:04:07]: It was actually a little, like, it was more than that on my side. You were sending all this feedback, and at the same time, we were looking for a PM. And we were, like, looking at a few people. And then I remember telling the manager, like, hey, I want Cat. Aww.

Alessio [00:04:19]: I'm sure people are curious. What's the process within Anthropic to, like, graduate one of these projects? Like, so you have kind of, like, a lot of growth. Then you get a PM. When did you decide, okay, we should, like, it's ready to go? It'll be opened up.

Boris [00:04:32]: Generally, at Anthropic, we have this product principle of do the simple thing first. And I think that the way we build product is really based on that principle. So you kind of staff things as little as you can and keep things as scrappy as you can because the constraints are actually pretty helpful. And for this case, we wanted to see some signs of product market fit before we scaled it.

Swyx [00:04:51]: Yeah, I imagine. So, like, we're putting out the MCP episode this week. And I imagine MCP also now has a team around it in much the same way. Hmm. It is now very much officially, like, sort of, like, an Anthropic product. So I'm kind of curious for Cat, like, how do you view PMing something like this? Like, what is, I guess, you're, like, sort of grooming the roadmap. You're listening to users. And the velocity is something I've never seen coming out of Anthropic.

Cat [00:05:16]: I think I PM with a pretty light touch. I think Boris and the team are, like, extremely strong product thinkers. And for the vast majority of the features on our roadmap, it's actually just, like, people who have a lot of experience. People building the thing that they wish that the product had. So very little actually is tops down. I feel like I'm mainly there to, like, clear the path if anything gets in the way and just make sure that we're all good to go from, like, a legal marketing, et cetera, perspective. Yeah. And then I think, like, in terms of very broad roadmap or, like, long-term roadmap, I think the whole team comes together and just thinks about, okay, what do we think models will be really good at in three months? And, like, let's just make sure that what we're building is really consistent. Yeah. Like, what do we think is really compatible with, like, the future of what models are capable of?

Swyx [00:06:01]: I'd be interested to double-click on this. What will models be good at in three months? Because I think that's something that people always say to think about when building AI products. But nobody knows how to think about it because it's, everyone's just, like, it's generically getting better all the time. We're getting AGI soon, so don't bother. You know, like, how do you calibrate three months of progress?

Cat [00:06:20]: I think if you look back historically, we tend to ship models every couple of months or so. So three months is just, like, an arbitrary number. I think the direction that we want our models to go in is being able to accomplish more and more complex tasks with as much autonomy as possible. And so this includes things like making sure the models are able to explore and find the right information that they need to accomplish a task, making sure that models are thorough in accomplishing every aspect of a task, making sure the models can, like, compose different tools together effectively. Yeah. These are the directions we care about. Yeah.

Boris [00:06:57]: I guess coming back to code, this kind of approach affected the way that we built code also because we know that if we want some product that has, like, very broad a product market fit today, we would build, you know, a cursor or a wind serve or something like this. Like, these are awesome products that so many people use every day. I use them. That's not the product that we want to build. We want to build something that's kind of much earlier on that curve and something that will maybe be a big product, you know, a year from now or, you know, however much time from now. As the model improves. And that's why code runs in a terminal. It's a lot more bare bones. You have raw access to the model because we didn't spend time building all this kind of nice UI and scaffolding on top of it.

Alessio [00:07:38]: When it comes to, like, the harness, so to speak, and things you want to put around it, there's one that may be prompt optimization. So obviously I use Cursor every day. There's a lot going on in Cursor that is beyond my prompt for, like, optimization and whatnot. But I know you recently released, like, you know, compacting context features and all that. How do you decide how thick it needs to be on top of the CLI? So that's kind of the share interface. And at what point are you deciding between, okay, this should be a part of clock code versus this is just something for the IDE people to figure out, for example?

Boris [00:08:10]: Yeah, there's kind of three layers at which we can build something. So the, you know, being an AI company, the most natural way to build anything is to just build it into the model and have the model do the behavior. The next layer is probably scaffolding on top, so it's like quad code itself. And then the layer after that is using quad code as a tool in a broader workflow, so to compose stuff in. You know, so for example, a lot of people use code with, you know, Tmux, for example, to manage a bunch of windows and a bunch of sessions happening in parallel. We don't need to build all of that in.

Boris [00:08:42]: Compact, it's sort of this thing that kind of has to live in the middle because it's something that we want to work when you use code. You shouldn't have to pull in extra tools on top of it. And rewriting memory in this way isn't something the model can do today. So you have to use a tool for it. And so it kind of has to live in that middle area. We tried a bunch of different options for compacting, you know, like rewriting old tool calls and truncating old messages and not new messages. And then in the end, we actually just did the simplest thing, which is ask Claude to summarize the, you know, the previous messages and just return that and that's it. And it's funny when the model is so good, the simple thing usually works. You don't have to over-engineer it. We do that for Claude who plays Pokemon too.

Swyx [00:09:23]: Just kind of interesting to see that. Yeah. It's like pattern re-emerging.

Alessio [00:09:26]: And then you have the Claude.md file for the more user-driven memories, so to speak. It's like kind of like the equivalent of maybe cursor rules, I would say.

Boris [00:09:36]: Yeah. And ClaudeMD, it's another example of this idea of, you know, do the simple thing first. We had all these crazy ideas about like memory architectures and, you know, there's so much literature about this. There's so many different external products about this and we wanted to be inspired by all this stuff. But in the end, the thing we did is ship the simplest thing, which is, you know, it's a file that has some stuff. And it's auto-read into context. And there's now a few versions of this file. You can put it in the root, or you can put it in child directories, or you can put it in your home directory. And we'll read all of these in kind of different ways. But yeah, it's the simplest thing that could work.

Swyx [00:10:07]: I think like people are interested in comparing and contrasting, obviously, because to you, obviously, this is the house tool. You work on it. People are interested in like figuring out how to choose between tools. There's the cursors of the world. There's like the devins of the world. There's aiders and there's ClaudeCode. And, you know, there's a lot of stuff. We can't try everything all at once. My question would be, where do you place it in the universe of options? Well, you can ask Claude to just try all these tools.

Swyx [00:10:36]: No self-favoring at all.

Boris [00:10:39]: Claude plays engineer. I don't know. We use all these tools in-house, too. We're big fans of all this stuff. Like ClaudeCode is, obviously, it's a little different than some of these other tools in that it's a lot more raw. Mm-hmm. Like I said, there isn't this kind of big, beautiful UI on top of it. It's raw access to the model. It's as raw as I get. So if you want to use a power tool that lets you access the model directly and use Claude for automating, you know, big workloads, you know, for example, if you have like a thousand Lint violations and you want to start a thousand instances of Claude and have it fix each one and then make a PR, then ClaudeCode is a pretty good tool. Got it. It's a tool for power workloads for power users. And I think that's just kind of where it fits. Yeah.

Alessio [00:11:23]: It's the idea of like parallel versus kind of like single path, one way to think about it, where like the IDE is really focused on like what you want to do versus like ClaudeCode. You kind of more see it as like less supervision required. You can kind of spin up a lot of them. Is that the right mental model?

Boris [00:11:40]: Yeah. And there's some people at Anthropic that have been racking up like thousands of dollars a day with this kind of automation. Most people don't do anything like that, but you totally could do something like that. Yeah. We think of it as like a Unix utility. Mm-hmm. Right? So it's like the same way that you would compose, you know, grep or cat or, oh, cat. Or something like this. Nice. The same way you can compose code into workflows.

Alessio [00:12:03]: The cost thing is interesting. Do people pay internally or do you get free? If you work at Anthropic, you can just run this thing as much as you want every day? It's for free internally. Nice. Yeah. I think if everybody had it for free, it would be huge.

Alessio [00:12:20]: Because like, I mean, if I think about, I pay Coursera. I use millions and millions of tokens in Coursera. That would cost me a lot more in Claude Code. And so I think like a lot of people that I've talked to, they don't actually understand how much it costs to do these things. And they'll do a task and they're like, oh, that costs 20 cents. I can't believe I paid that much. How do you think, going back to like the product side too, is like, how much do you think of that being your responsibility to try and make it more efficient versus that's not really what we're trying to do with the tool?

Cat [00:12:51]: We really see Claude Code as like the tool that gives you the smartest abilities out of the model. We do care about cost insofar as it's very correlated with latency. And we want to make sure that this tool is extremely snappy to use and extremely thorough in its work. We want to be very intentional about all the tokens that it produces. I think we can do more to like communicate the cost with users. Currently, we're seeing costs around like $6 per day. per active user. And so it's like, it does come out to a bit higher over the course of a month in Cursor. But I don't think it's like out of band. And that's like roughly how we're thinking about it.

Boris [00:13:36]: I would add that I think the way I think about it is it's an ROI question. It's not a cost question. And so if you think about, you know, an average engineer salary and like what, you know, we were talking about this before the podcast. Like, engineers are very expensive. And if you can make an engineer 50, 70% more productive, that's worth a lot. And I think that's the way to think about it.

Swyx [00:13:58]: So if you're saying, if you're targeting Claude to be the most powerful end of the spectrum, as opposed to the less powerful, but faster, cheaper side of the spectrum, then there's typically people who recommend a waterfall, right? You try this faster, simple one. That doesn't work. You upgrade, you upgrade, you upgrade. And finally you hit Claude Code, at least for people who are token constrained that don't work at Ntopic. And part of me wants to just fast track all of that. I just want to fan out to everything all at once. And once I, once I'm not satisfied with the next one solution, I'll just sort of switch to the next. I don't know if that's real.

Boris [00:14:32]: Yeah, we're, we're definitely trying to make it a little easier to make Claude Code kind of the tool that you use for all the different workloads. So for example, we launched thinking recently. So for any kind of planning workload where you might've used other tools before you can just ask what, and that'll use, you know, chain of thought. To, to think stuff out. I think we'll get there.

Swyx [00:14:51]: Maybe we'll do it this way. How about we recap like sort of the brief history of Claude Code, like between when you launched and now there, there've been quite a few ships. How would you highlight the major ones? And then we'll get to the thinking tool.

Boris [00:15:04]: And I think I'd have to like check your Twitter to remember.

Cat [00:15:09]: I think a big one that we've gotten a lot of requests for is web fetch. Yep. So we worked really closely with our legal team to make sure that, you know, the code that we shipped as secure of an implementation as possible. So we'll web fetch if a user directly provides an URL, whether that's in their call.md or in their message directly, or if a URL is mentioned in one of the previously fetched URLs. And so this way enterprises can feel pretty secure about letting their developers continue to use it. We shipped a bunch of like auto features, like autocomplete, where you can, you know, press tab to complete a file name or file path. Autocompact so that users feel like they have like infinite context since we'll compact behind the scenes. And we also shipped auto accept because we noticed that a lot of users were like, hey, like Claude Code can figure it out. I've like developed a lot of trust for Claude Code. I wanted to just like autonomously edit my files, run tests, and then come back to me later. So those are some of the big ones.

Swyx [00:16:15]: Vim mode, custom slash commands.

Cat [00:16:18]: People love Vim mode. Yeah. So that was a, that was a top request too. That one went pretty viral. Yeah.

Boris [00:16:24]: Yeah. Memory. Those are recent ones. Like the hashtag to remember.

Swyx [00:16:28]: So yeah. I mean, I'd love to dive into, you know, on the technical side, any of them that was particularly challenging. Paul from AIDR always says how much of it was coded by AIDR, you know? So then the question is how much of it was coded by Claude Code? Obviously there's some percentage, but I wonder if you have a number. Like 50? 80? It's pretty high. Probably near 80 I'd say. Yeah. Yeah.

Boris [00:16:48]: It's very high.

Cat [00:16:49]: Yeah. That makes sense. It's a lot of human code review though.

Boris [00:16:52]: Yeah. What a, what a human code review. I think some of the stuff has to be handwritten and some of the code can be written by quad. And there's sort of a wisdom in knowing which one to pick and what percent for each kind of task. So usually where we start is quad writes the code. And then if it's not good, then maybe a human will dive in. There's also some stuff where like I actually prefer to do it by hand. So it's like, you know, intricate data model refactoring or something. I won't leave it to quad because I have really strong opinions and it's easier to just do it and experiment than it is to explain it to quad. So yeah, I think that nets out to maybe like 80, 90% quad written code overall. Yeah.

Alessio [00:17:28]: We're hearing a lot of that in our portfolio companies, like more like series A companies is like 80, 85% of the code they write is ad generated. Yeah. Yeah. Well, that's a whole different discussion. The custom slash command. I had a question. How do you think about custom slash command MCPs? Like how does this all tie together? You know, is the slash command and clock code kind of like an extension of the MCP? Are people building things that should not be MCP, but are just kind of like self-contained things in there? How should people think about it?

Boris [00:18:01]: Yeah. I mean, obviously we're big fans of MCP. You can use MCP to do a lot of different things. You can use it for custom tools and custom commands and all this stuff. But at the same time, you shouldn't have to use it. So if you just want something really simple and local, you just want, you know, some essentially like prompt that's been saved, just use local commands for that. Over time, something that we've been thinking a lot about is how to re-expose things in convenient ways. So for example, let's say you had this local command. Could you re-expose that as an MCP prompt? Yeah. Because clock code is an MCP client and an MCP server. Or similarly, let's say you pass in a custom, you know, like a custom bash tool. Is there a way to re-expose that? Re-expose that as an MCP tool? Yeah. We think generally you shouldn't have to be tied to a particular technology. You should use whatever works for you. Yeah.

Alessio [00:18:48]: Because there's some like Puppeteer. I think that's like a great way, great thing to use a clock code, right? For testing. There's like a Puppeteer MCP protocol, but then people can also write their own slash commands. And I'm curious, like where MCP are going to end up being, where it's like maybe each slash command leverages MCPs, but no command itself is an MCP because it ends up being customized. I think that's what people are still trying to figure out. It's like, should this be in the runtime or in the MCP server? I think people haven't quite figured out where the line is.

Boris [00:19:20]: Yeah. For something like Puppeteer, I think that probably belongs in MCP because there's a few like tool calls that go in that too. And so it's probably nice to encapsulate that in the MCP server.

Cat [00:19:30]: Whereas slash commands are actually just like prompts. So they're not actually tools. We're thinking about how to expose more customizability options so that people can use them. People can bring their own tools or turn off some of the tools that quad code comes with. But there's also some trickiness there because we want to just make sure that the tools people bring are things that quad is able to understand and that people don't accidentally inhibit their experience by maybe bringing a tool that is like confusing to quad. So we're just trying to work through the UX of it.

Boris [00:20:05]: Yeah. I'll give an example also of how this stuff connects. For quad code internally in the GitHub repo, we have this GitHub action that runs. And the GitHub action invokes quad code with a local slash command. And the slash command is lint. So it just runs a linter using quad. And it's a bunch of things that are pretty tricky to do with a traditional linter that's based on static analysis. So for example, it'll check for spelling mistakes, but also it checks that code matches comments. It also checks that, you know, we use a particular library for, you know, a specific command. So it checks for network fetches instead of the built-in library. There's a bunch of these specific things that we check that are pretty difficult to express just with lint. And in theory, you can go in and, you know, write a bunch of lint rules for this. Some of it you could cover, some of it you probably couldn't. But honestly, it's much easier to just write a one bullet in Markdown in a local command and just commit that. And so what we do is quad runs through the GitHub action. We invoke it with slash project colon lint. So which just invokes that local command. It'll run the linter. It'll identify any mistakes. It'll make the code changes. And then it'll use the GitHub MCP server in order to commit the changes back to the PR. And so you can kind of compose these tools together. And I think that's a lot of the way we think about code is just one tool in an ecosystem that composes nicely without being opinionated about any particular piece.

Swyx [00:21:27]: It's interesting. I have a weird chapter in my CV that makes me, I was the CLI maintainer for Nellify. And so I have a little bit of a dive. There's a decompilation of Claude Code out there that has since been taken down. But it seems like you use CommanderJS and React Inc is like the public info about this. And I'm just kind of curious, like at some point you're just, you're not even building Claude Code. You're kind of just building a general purpose CLI framework that anyone, any developer can hack to their purposes. You ever think about this? Like this level of configurability is more of like a CLI framework. Or like some new form factor that doesn't exist before.

Boris [00:22:09]: Yeah. It's definitely been fun to hack on a really awesome CLI because there's not that many of them. Yeah. But yeah, we're big fans of Ink. Yeah.

Swyx [00:22:17]: Vadim Demede. So we actually used him, used React Inc for a lot of our projects.

Boris [00:22:22]: Oh, cool. Yeah.

Swyx [00:22:23]: Yeah.

Boris [00:22:24]: Ink is amazing. It's like, it's sort of hacky and janky in a lot of ways. It's like you have React and then the renderer is just translating the React code to like ANSI language. Yeah. That's the way to render. And there's all sorts of stuff that just doesn't work at all because ANSI escape codes are like, you know, it's like this thing that started to be written like the 1970s. And there's no really great spec about it. Every terminal is a little different. So building in this way, it feels to me a little bit like building for the browser back in the day where you had to think about like Internet Explorer 6 versus Opera versus like Firefox and whatever. Like you have to think about these cross-terminal differences a lot. Yeah. So yeah, big fans of Ink because it helps abstract over that. We're also, we use Bun. So big fans of Bun. That's been, it makes writing our tests and running tests much faster. We don't use it in the runtime yet.

Swyx [00:23:13]: It's not just for speed, but you tell me, I don't want to put words in your mouth, but my impression is they help you ship the compilation, the executable. Yeah, exactly.

Boris [00:23:21]: So we use Bun to compile the code together. Yeah.

Swyx [00:23:24]: Any other pluses of Bun? I just want to track Bun versus Deno conversations. Yeah. Deno's in there. Yeah.

Boris [00:23:33]: I actually haven't used Deno back. It's been a while. Yeah. I remember Ryan.

Swyx [00:23:37]: That's what a lot of people say. Yeah. Yeah.

Boris [00:23:39]: Ryan made it back in the day and it was like, there were some ideas that I think were very cool in it, but yeah, it just never took off to that same degree. Yeah. Still a lot of cool ideas. Like being able to NPM, just import from any URL, I think is pretty amazing.

Swyx [00:23:51]: That's the dream of ESM. I was going to ask you one other feature, then we can get to the thinking tool of AutoAccept. I have this little thing I'm trying to do. I'm trying to develop thinking around for trust in agents, right? When do you say, all right, go autonomous? When do you pull the developer in? And sometimes you let the model decide. Sometimes you're like, this is a destructive action, always ask me. And I'm just curious if you have any internal heuristics around when to AutoAccept and where all this is going.

Cat [00:24:25]: We're spending a lot of time building out the permission system. So Robert on our team is leading out this work. We think it's really important to give developers the control to say, hey, these are like the allowed permissions. Generally, this includes stuff like the model's always allowed to read files or read anything. And then it's up to the user to say, hey, it's about to edit files, it's about to run tests. These are like probably the safest three actions. And then there's like a long list of other actions that users can either allow list or deny list based on regex matches with the action.

Alessio [00:24:59]: How good is that? Is writing a file ever be unsafe if you have version control? I think that's.

Boris [00:25:04]: Yeah, I think it's I think there's like a few different probably like aspects of safety to think about. So it could be useful just to break that out a little bit. So for file editing, it's actually less, I think, about safety, although there is still a safety risk. Because what might happen is, let's say the model fetches a URL and then there's a prompt injection attack in the URL. And then the model writes malicious code to disk and you don't realize it. Although, you know, there is code review as like a separate thing. Kind of where there is as protection. But I think generally for file rights, the model might just do the wrong thing. That's the biggest thing. And what we find is that if the model is doing something wrong, it's better to identify that earlier and correct it earlier. And then you're going to have a better time. If you wait for the model to just go down this like totally wrong path and then correct it 10 minutes later, you're going to have a bad time. So it's better to usually identify failures early. But at the same time, there's some cases where you just want to let the model go. So, for example, if Claude Code is. You know, it's writing tests for me. I'll just hit shift tab. Enter auto accept mode and just let it run the tests and iterate on the tests until they pass. Because I know that's a pretty safe thing to do. And then for some other tools like bash tool, it's pretty different. Because quad could run, you know, rm, rf, slash. And that would suck. That's not a good thing. So we definitely want people to be in the loop to catch stuff like that. The model is, you know, trained and aligned to not do that. But, you know, these are non-deterministic systems. So, like, you still want a human in the loop. I think that generally the way that things are trending is kind of less time between human input. Did you see the meter paper?

Swyx [00:26:43]: No. They established a Moore's law for time between human input, basically. And it's basically doubling every three to seven months is the idea. And Anthropic is currently doing super well on that benchmark. It's roughly about autonomous for 15 minutes. And it's at the 50th percentile of human effort, which is kind of cool. Highly recommend that.

Alessio [00:27:04]: I put cursor in YOLO mode all the time and just run it.

Swyx [00:27:08]: But it's fine. Which is vibe coding, right? Yeah.

Alessio [00:27:11]: And there's a couple of things that are interesting when you talked about alignment and the model being trained. So I always put it in a Docker container. And I have a prefix every command with, like, the Docker compose. And yesterday, my Docker server was not started. And I was like, oh, Docker is not running. Let me just run it outside of Docker. And I'm like, whoa, whoa, whoa, whoa, whoa. You should start Docker and run it in Docker. You can now go outside. That is, like, a very good example of, like, you know, sometimes you think it's doing something and then it's doing something else. And for the review side, I would love to just chat about that more. I think the linter part that you mentioned, I think maybe people skipped it over. It doesn't register the first time. But, like, going from, like, rule-based linting to, like, semantic linting, I think it's, like, great and super important. And I think a lot of companies are trying to do how do you do autonomous PR review, which I've not seen one that I use so far. They're all kind of, like, mid. So I'm curious how you think about closing the loop or making that better and figuring out especially, like, what are you supposed to review? Because these PRs get pretty big when you buy code. You know, sometimes I'm like, oh, wow. Oh, GTM. You know, it's like, am I really supposed to read all of this? It kind of seems, most of it seems pretty standard. But, like, I'm sure there are parts in there that the model wants. I would understand that are, like, kind of out of distribution, so to speak, to really look at. So, yeah, that's a very open-ended question. But any thoughts you have would be great.

Boris [00:28:33]: The way we're thinking about it is quad code is, like I said before, it's a primitive. So if you want to use it to build a code review tool, you can do this. If you want to, you know, build, like, a security scanning, vulnerability scanning tool, you can do that. If you want to build a semantic linter, you can do that. And hopefully with code, it makes it so if you want to do this, it's just a few lines of code. And you can just have quad write that code also. Because quad is really great at writing GitHub actions.

Cat [00:28:58]: Yeah, one thing to mention is we do have a non-interactive mode, which is, like, what quad uses in these, or how we use quad in these situations to automate quad code. And also a lot of our, the companies using quad code actually use this non-interactive mode. So they'll, for example, say, hey, I have, like, hundreds of thousands of tests in my repo. Some of them are out of date. Some of them are flaky. And they'll send quad code. So they'll send quad code to look at each of these tests and decide, okay, how can I update any of them? Like, should I deprecate some of them? How do I, like, increase our code coverage? So that's been a really cool way that people are non-interactively using quad code.

Swyx [00:29:38]: What are the best practices here? Because when it's non-interactive, it could run forever. And you're not necessarily reviewing the output of everything, right? So I'm just kind of curious, how is it different in non-interactive mode? What are, like, the most important hyperparameters or arguments to solve?

Boris [00:29:55]: Yeah, and for folks that haven't used it, so non-interactive mode is just quad dash p. And then you pass in the prompting quotes, and that's all it is. It's just the dash p flag. Generally, it's best for tests that are read-only. That's the place where it works really well. And you don't, you know, super have to think about permissions and running forever and things like that. So, for example, a linter that runs and doesn't fix any issues. Or, for example, we're working on a thing where we use quad with dash p to generate the change log for quad. So every PR is just looking over the commit history. And being like, okay, this makes it into the change log, this doesn't. Because we know people have been requesting change logs, so we're just getting quad to build it. So generally, non-interactive mode, really good for read-only tests. For tests where you want to write, the thing we usually recommend is passing a very specific set of permissions on the command line. So what you can do is pass in dash dash allow tools. And then you can allow a specific tool. So, for example, not just bash, but, for example, git status. Or git diff. So you just give it a set of tools that it can use.

Swyx [00:30:57]: Or, you know, edit tool, for example. It still has default tools like file read, grep, system tools like bash and ls, and memory tools.

Boris [00:31:04]: So it still has all these tools, but allow tools just lets you, instead of the permission prompt, because you don't have that in the non-interactive mode. It's just kind of pre-accepting.

Cat [00:31:15]: And we'd also definitely recommend that you start small. So, like, test it on one test. Make sure that it has reasonable behavior. Iterate on your prompt. Then scale it up to 10. Make sure that it succeeds. Or if it fails, just, like, analyze what the patterns of failures are. And gradually scale up from there. So definitely don't kick off a run to fix, like, 100,000 tests. Yeah.

Swyx [00:31:36]: I think the... So at this point, I just, you know, I want to... This tagline is in my head that basically at Anthropic, there's Claude Code generating code. And then Claude Code also reviewing its own code. Like, at some point, right? Like, different people are setting all this up. You don't really govern that. But it's happening.

Boris [00:31:53]: Yeah, we have to be, you know, at Anthropic, there's still a human in the loop for reviewing. And I think for, you know, for ASL, this is important. So, like, for general, like, model alignment and safety.

Swyx [00:32:02]: What's ASO?

Boris [00:32:03]: Oh, so ASL, this is, like, the kind of the safety levels.

Swyx [00:32:07]: Yeah, right, right.

Boris [00:32:08]: What does it stand for?

Cat [00:32:09]: Autonomous safety level. Autonomous.

Boris [00:32:12]: Essentially, like, it's like a...

Swyx [00:32:13]: Sorry, I'm not used to the acronyms. We have a lot of these. But you've published stuff. I know. I just don't know what they're called internally. Yeah. But the point of the thing I was thinking about was we have, you know, VPs of N, CTOs listening. Like, this is all well and good for the individual developer. But the people who are responsible for the tech, the entire code base, the engineering decisions, all this is going on. My developers, like, I manage, like, 100 developers. Any of them could be doing any of this at this point. What do I do to manage this? How does my code review process change? How does my change management change? I don't know.

Cat [00:32:48]: We've talked to a lot of VPs and CTOs. Yeah. They're really excited about it. They actually tend to be quite excited because they experiment with the tool. They download it. They ask it a few questions. And, like, Claude Code, when it gives them sensible answers, they're really excited because they're like, oh, I can understand this nuance in the code base. And sometimes they even ship small features with Claude Code. And I think through that process of, like, interacting with the tool, they build a lot of trust in it. And a lot of folks actually come to us and they ask us, like, how can I roll it out more broadly? And then we'll often, like, have sessions with, like, VPs of Dev Prod and talk about these concerns around how do we make sure people are writing high-quality code. I think in general, it's still very much up to the individual developer to hold themselves up to a very high standard for the quality of code that they merge. Even if we use Claude Code to write a lot of our code, it's still up to the individual who merges it to be responsible for, like, this being well-maintained, well-documented code that has, like, reasonable abstractions. And so I think that's something that will continue to happen where Claude Code isn't its own engineer that's, like, committing code by itself. It's still very much up to the ICs to be responsible for the code that's produced. Yeah.

Boris [00:34:06]: I think Claude Code also makes a lot of this stuff, a lot of quality work becomes a lot easier. So, for example, like, I have not manually written a unit test in many months.

Cat [00:34:15]: And we have a lot of unit tests. We have a lot of unit tests.

Boris [00:34:17]: And it's because Claude writes all the tests. And, you know, before, I felt like a jerk if on someone's PR, I'm like, hey, can you write a test? Because, you know, they kind of know they want to… For code coverage? Is that still relevant? For code coverage, yeah. Okay. And, you know, they kind of know they should probably write a test and that's probably the right thing to do. And somewhere in their head, they make that trade-off where they just want to ship faster. And so you always kind of feel like a jerk for asking. But now I always ask because Claude can just write the test. Right. And, you know, there's no human work. You just ask Claude to do it and it writes it. And I think with writing tests becoming easier and with the writing lint rules becoming easier, it's actually much easier to have high-quality code than it was before.

Swyx [00:34:56]: What are the metrics that you believe in? Like, is it… A lot of people actually don't believe in 100% code coverage. Because sometimes that is kind of optimizing for the wrong thing. Arguably, I don't know. But like, obviously, you have a lot of experience in different code quality metrics. But what still makes sense?

Boris [00:35:11]: I think it's very engineering team dependent, honestly. I wish there was a one-size-fits-all answer. Yeah. Just give me the one solution. For some teams, test coverage is extremely important. For other teams, type coverage is very important. Especially if you're working in, you know, very strictly typed language. And, you know, for example, avoiding, like, NEs and JavaScript and Python. Yep. I think cyclomatic complexity kind of gets a lot of flack. But it's still, honestly, a pretty good metric just because there isn't anything better in terms of ways to measure. Yeah. For code quality. Okay.

Swyx [00:35:43]: And then productivity is obviously not lines of code. But do you care about measuring productivity? I'm sure you do.

Boris [00:35:50]: Yeah. You know, lines of code, honestly, isn't terrible. Oh, God. It has downsides. Yeah. It's terrible. Well, lines of code is terrible for a lot of reasons. Yes. But it's really hard to make anything better. It's the least terrible. It's the least terrible. There's, like, lines of code, maybe, like, number of PRs. How green your GitHub is. Yeah.

Cat [00:36:09]: Yeah. The two that we're really trying to nail down are, one, decrease in cycle time. So how much faster are your features shipping because you're using these tools? So that might be something like the time between first commit and when your PR is merged. It's very tricky to get right, but one of the ones that we're targeting. The other one that we want to measure more rigorously is, like, the number of features that you wouldn't have otherwise built. Hmm. We have a lot of channels where we get customer feedback. And one of the patterns that we've seen with Claude Code is that sometimes customer support or customer success will, like, post, hey, like, this app has, like, this bug. And then sometimes 10 minutes later, one of the engineers on that team will be, like, Claude Code made a fix for it. And a lot of those situations when you, like, ping them and you're, like, hey, that was really cool, they were, like, yeah, without Claude Code, I probably wouldn't have done that because it would have been too long. It would have been too much of a divergence from what I was otherwise going to do. It would have just ended up in this long backlog. So this is the kind of stuff that we really want to measure more rigorously.

Boris [00:37:14]: That was the other AGI-appealed moment for me. There was a really early version of Claude Code many, many months ago. And this one engineer at Anthropic, Jeremy, built a bot that looked through a particular feedback channel on Slack. And he hooked it up to code to have code automatically put up PRs with just fixes to all this stuff. And some of this stuff, you know, it didn't fix every issue. But it fixed a lot of the issues. You say, like, 10%, 50%? You know, this was, like, early on, so I don't remember the number. But it was surprisingly high to the point where I became a believer in this kind of workflow. And I wasn't before.

Alessio [00:37:47]: SAPM isn't that scary, too, in a way? Where you can build too many things, it's almost like maybe you shouldn't build that many things. I think that's what I'm struggling with the most. It's, like, it gives you the ability to create, create, create. But then at some point, you've got to support, support, support.

Swyx [00:38:02]: This is the Jurassic Park, like, your scientist is so preoccupied with what you're doing. Yeah, exactly.

Alessio [00:38:06]: I don't know if we should. Yeah. How do you make decisions? Like, now that the cost of actually implementing the thing is going down as a PM, how do you decide what is actually worth doing?

Cat [00:38:15]: Yeah, we definitely still hold a very high bar for net new features. Most of the fixes were, like, hey, this functionality is broken or this, like, there's a weird edge case that we hadn't addressed yet. So it was very much, like, smoothing out the rough edges as opposed to building something completely net new. For net new features, I think we hold a pretty high bar. High bar that it's very intuitive to use. The new user experience is, like, minimal. It's just, like, obvious that it works. We sometimes actually use Claude Code to prototype instead of using docs. Yeah, so you'll have, like, prototypes that you can play around with. And that often gives us a faster feel for, hey, is this feature ready yet? Or, like, is this the right abstraction? Is this the right interaction pattern? So it gets us faster to feeling really confident about a feature. But it doesn't circumvent the process of us making sure that the feature definitely fits in, like, the product vision.

Boris [00:39:12]: It's interesting how as it gets easier to build stuff, it changes the way that I write software. Where, like Cat's saying, like, before I would write a big design doc. And I would think about a problem for a long time before I would build it sometimes for some set of problems. And now I'll just ask Claude Code to prototype, like, three versions of it. And I'll try the feature and see which one I like better. And then that informs me much better and much faster than a doc would have. And I think we haven't totally internalized that transition yet in the industry.

Alessio [00:39:41]: Yeah, I feel the same way for some tools I build internally. People ask me, could we do this? And I'm like, I'll just, yeah, just build it. It's like, well, it feels pretty good. We should, like, polish it, you know? Or sometimes it's like, no, that's not.

Swyx [00:39:57]: It's comforting that, you know, like, your max cost is, I mean, even at Anthropic where it's theoretically unlimited. Yeah. The cost is roughly $6 a day. That gives people peace of mind. Because I'm like, $6 a day? Fine. $600 a day, we have to talk.

Alessio [00:40:12]: Like, you know. Yeah. I paid $200 a month to make Studio Ghibli photos. So it's all good. That is totally worth it.

Cat [00:40:19]: You mentioned internal tools. And that's actually a really big use case that we're seeing emerge. Because a lot of times, if you're working on something operationally intensive, if you can spin up an internal dashboard for it. Or, like, an operational tool where you can, for example, grant access to a thousand emails at once. A lot of these things, you don't really need to have, like, a super polished design. You kind of just need something that works. And Claude Code's really good at those kinds of zero to one tasks. Like, we use Streamlit internally. And there's been, like, a proliferation of how much we're able to visualize. And because we're able to visualize it, we're able to see patterns that we wouldn't have otherwise. If we were just looking at, like, raw data.

Boris [00:41:04]: Yeah. Like, I was working on also this, like, side website last week. And I just showed Claude Code the mock. So I just took the, you know, the screenshot I had, dragged and dropped it into the terminal. And I was like, hey, Claude, here's the mock. Can you implement it? And it implemented. And it looked like, you know, it sort of worked. It was a little bit crummy. And I was like, all right, now look at it in Puppeteer and, like, iterate on it until it looks like the mock. And then it did that three or four times. And then the thing looked like the mock. Yeah. This was just all manual work before.

Swyx [00:41:32]: I think we're going to ask about, like, two other features of, I guess, the overall agent pieces that we mentioned. So I'm interested in memory as well. So we talked about autocompact and memory using hashtags and stuff. My impression is that your, like you say, simplest approach works. But I'm curious if you've seen any other requests that are interesting to you or internal hacks of memory that people have explored that, like, you know, you might want to surface to others.

Boris [00:41:59]: There's a bunch of different approaches to memory. Most of them use external stores of various sorts. Like Chroma. Exactly. Yeah. There's a lot of projects like that. And, yeah, it's either a K value or kind of, like, graph stores. That's, like, the two big shapes for these. Are you a believer in knowledge graphs for this stuff? You know, I'm a big, if you talked to me before I joined Anthropic and this team, I would have said, yeah, definitely. But now, actually, I feel everything is the model. Like, that's the thing that wins in the end. And it just, as the model gets better, it subsumes everything else. So, you know, at some point, the model will encode its own knowledge graph. It'll encode its own, like, KV store if you just give it the right tools. Yeah. But, yeah, I think the specific tools, there's still a lot of room for experimentation. We just, we don't, we don't know yet.

Swyx [00:42:45]: In some ways, are we just coping for lack of context length? Like, are we doing things for memory now that if we had, like, 100 million token context window, we don't care about?

Cat [00:42:55]: I would love to have 100 million.

Boris [00:42:57]: I mean, you know, some people have claimed to have done it. We don't know if that's true or not. I guess here's a question for you, Sean. If you took all the world's knowledge and you put it in your brain. Yeah. And let's say, you know, there is, like, some treatment that you could get to make it so your brain can have any amount of context. You have, like, infinite neurons. Is that something that you would want to do or would you still want to record knowledge externally?

Swyx [00:43:19]: Putting it in my head is, like, different for me trying to use an agent tool to do it because I'm trying to control the agent. And I'm trying to make myself unlimited, but I want to make the tools that I use limited because then I know how to control them. And it's not even, like, a safety argument. It's just more like I want to know what you know. And if you don't know a thing, then sometimes that's good. Like the ability to audit what's in it. Yeah. And I don't know if this is the small brain thinking because this is not very bitter lesson, which is, like, actually sometimes you just want to control every part of what goes in there in the context. And the more you just, you know, Jesus takes care of it. Like, if you really trust the model, then you have no idea what it's paying attention to.

Boris [00:43:58]: Yeah. I don't know. Did you see the Mac interpretability stuff from Chrisola and the team that was published? Like last week? Yeah, last week. Yes. What about it? I wonder if something like this is the future. So there's an easier way to audit the model itself. And so if you want to see, like, what is stored, you can just audit the model.

Swyx [00:44:15]: Yeah. The main salient thing is that they know what features activate it per token and they can tune it up, suppress it, whatever. But I don't know if it goes down to the individual, like, item of knowledge from context, you know. Not yet.

Boris [00:44:30]: Yeah. But I wonder, you know, maybe that's the bitter lesson version of it.

Swyx [00:44:34]: Right, right. Any other comments from memory? Otherwise, we can move on to planning and thinking.

Cat [00:44:38]: We've been seeing people play around with memory in quite interesting ways, like having Claude write a logbook of all the actions that it's done. So that over time, Claude develops this understanding of what your team does, what you do within your team, what your goals are. How you like to approach work. We would love to figure out what the most generalized version of this is so that we can share broadly. I think with things like hard code, like, I think when we're developing things like hard code, it's actually less work to implement the feature and a lot of work to tune these features to make sure that they work well for general audiences, like across a broad range of use cases. So there's a lot of interesting stuff with the memory, and we just want to make sure that it works well out of the box before we share it broadly.

Swyx [00:45:29]: Agree with that.

Boris [00:45:30]: I think there's a lot more to be developed here. I guess a related problem to memory is how do you get stuff into context? Knowledge base. Knowledge base, yeah. And originally, we tried very, very early versions of Claude actually used RAG. So we, like, indexed the code base, and I think we were just using Voyage. So, you know, just off-the-shelf RAG, and that worked pretty well. And we tried a few different versions. There was RAG, and then we tried a few different kinds of search tools. And eventually, we landed on just agentic search as the way to do stuff. And there were two big reasons, maybe three big reasons. So one is it outperformed everything. By a lot. By a lot. And this was surprising. In what benchmark? This was just vibes, so internal vibes. There's some internal benchmarks also, but mostly vibes. It just felt better.

Swyx [00:46:14]: And agentic RAG, meaning you just let it look up in however many search cycles it needs. Yeah.

Boris [00:46:20]: Just using regular code searching, you know, glob, grep, just regular code search. Regular code search, yeah. Yeah. So there was, like, one. And then the second one was there was this whole, like, indexing step that you have to do for RAG. And there's a lot of complexity that comes with that because the code drifts out of sync. And then there's security issues because this index has to live somewhere. And then, you know, what if that provider gets hacked? And so it's just a lot of liability for a company to do that. You know, even for our code base, it's very sensitive. So we're kind of, we don't want to upload it to a third-party thing. It could be a first-party thing. But then we still have this out-of-sync issue. And agentic search just sidesteps all of that. So essentially, at the cost of latency and tokens, you now have really awesome search without security downsides.

Alessio [00:47:01]: Well, memory is like planning, right? There's kind of, like, memory is, like, what I like to do. And then planning is, like, now use those memories to come up with a plan to do these things.

Swyx [00:47:10]: There was one. Or maybe put it as, like, memory is sort of the past. Like, what we already did. And then planning is kind of what we will do. And it just crosses over at some point. Yeah.

Alessio [00:47:19]: I think the maybe slightly confusing thing from the outside is what you define as thinking. So there's, like, extensive thinking. There's the think tool. And it's kind of, like, thinking as in planning, which is, like, thinking before execution. And then there's, like, thinking while you're doing, which is, like, the think tool. Can you maybe just run people through the difference?

Swyx [00:47:40]: I'm already confused listening to you.

Boris [00:47:42]: Well, it's one tool. So Quad can think if you ask it to think. Generally, the usage pattern that works best is you ask Quad to do a little bit of research. Like, use some tools, pull some code into context, and then ask it to think about it. And then it can make a plan and, you know, do a planning step before you execute. There's some tools that have explicit planning modes. Like, RooCode has this and Quine has this. Other tools have it. Like, you can shift between, you know, plan and act mode or maybe a few different modes. We've sort of thought about this approach. But I think our approach to product is similar to our approach to the model, which is Bitter Lesson. So just freeform. Keep it really simple. Keep it close to the metal. And so if you want Quad to think, just tell it to think. Be like, you know, make a plan. Think hard. Don't write any code yet. And it should generally follow that. And you can do that also as you go. So maybe there's a planning stage and then Quad writes some code or whatever. And then you can ask it to think and plan a little bit more. You can do that anytime.

Alessio [00:48:45]: Yeah, I was reading through the think tool blog post. And it said, while it sounds similar to extended thinking, it's a different concept. Extended thinking is what Quad does before it starts generating. And then think it, once it starts generating, how do you stop and think? Is this all done by the Quad code harness? So people don't really have to think about the difference between the two, basically, is the idea? Yeah, you don't have to think about it. Okay. That is helpful. That is helpful. Because sometimes I'm like, man, am I not thinking right?

Boris [00:49:16]: Yeah. Yeah. And it's all chain of thought, actually, in Quad code. So we don't use the think tool. Anytime that Quad code does thinking, it's all chain of thought.

Swyx [00:49:24]: I had an insight. This is, again, something we had, a discussion we had before recording, which is in the Claude Place Pokemon HacCathon. We had access to more sort of branching environments feature, which meant that we could take any VM state, branch it, play it forward a little bit, and use that in the planning. And then I realized. The TLDR of yesterday was basically that it's too expensive to just always do that at every point in time. But if you give it as a tool to Claude and prompt it, in certain cases, to use that tool, seems to make sense. I'm just kind of curious, like your takes on overall, like, sandboxing, environment, branching, rewindability, maybe, which is something that you immediately brought up, which I didn't think about. Is that useful for Claude? Or Claude has no opinions about it?

Boris [00:50:10]: Yeah, I could talk for hours about this.

Swyx [00:50:12]: Claude probably can too. Yeah? Yeah. Let's get original tokens from you, and then we can train Claude on that. By the way, that's like explicitly what this podcast is. We're just generating tokens for people. Is this the pre-training or the post-training? It's a pre-trained data set. We got to get in there. Oh, man.

Boris [00:50:28]: Yeah. How do I buy? How do I get some tokens? Starting with sandboxing, ideally, the thing that we want is to always run code in a Docker container. And then it has freedom. And you can kind of snapshot, you know, with other kind of tools later on top. You can snapshot, rewind. Do all this stuff. Unfortunately, working with a Docker container for everything is just like a lot of work, and most people aren't going to do it. And so we want some way to simulate some of these things without having to go full container. There's some stuff you can do today. So, for example, something I'll do sometimes is if I have a planning question or a research type question, I'll ask Claude to investigate a few paths in parallel. And you can do this today if you just ask it. So say, you know, I want to refactor X to do Y. Can you research three separate ideas for how to do it? Do it in parallel. Use three agents to do it. And so in the UI, when you see a task that's actually like a sub-Claude, it's a sub-agent that does this. And usually when I do something hairy, I'll ask it to just investigate, you know, three times or five times or however many times in parallel. And then Claude will kind of pick the best option and then summarize that for you.

Alessio [00:51:33]: But how does Claude pick the best option? Don't you want to choose? What's your handoff between you should pick versus I should be the final decider?

Boris [00:51:43]: I think it depends on the problem. You can also ask Claude to present the options to you.

Swyx [00:51:48]: Probably, you know, it exists at a different part of the stack than Claude Code specifically. Claude Code as a CLI, like you could use it in any environment. So it's up to you to compose it together. Should we talk about how and when models fail? Because I think that was another hot topic for you. I'll just leave it open. Like, how do you observe Claude Code failing?

Cat [00:52:06]: There's definitely a lot of room for improvement in the models, which I think is very exciting. Most of our research team actually uses Claude Code day to day. And so it's been a great way for them to be very hands on and like experience the model failures, which makes it a lot easier for us to target these in model training and to actually provide better models, not just for Claude Code, but for like all of our coding customers. I think one of the things about the latest Sonnet 3.7 is it's a very persistent model. It's like very, very motivated to accomplish the user's goals. But it sometimes takes the user's goal very literally. And so it doesn't always fulfill what like the implied parts of the request are, because it's just so narrowed in on like, I must get X done. And so we're trying to figure out, okay, how do we give it a bit more common sense so that it knows the line between trying very hard and like, no, the user definitely doesn't want that. Yeah.

Boris [00:53:06]: Like the classic example is like, hey, go on, get this test to pass. And then, you know, like five minutes later. It's like, all right, well, I hard-coded everything. The test passes. I'm like, no, that's not what I wanted. Hard-coded the answer. Yeah. But that's the thing. Like it only gets better from here. Like these use cases work sometimes today, not, you know, not every time. And, you know, the model sometimes tries too hard, but it only gets better.

Swyx [00:53:27]: Yeah.

Cat [00:53:28]: Yeah. Like context, for example, is a big one where like a lot of times if you have a very long conversation and you compact a few times, maybe some of your original intent isn't as strongly present as it was when you first started. And so maybe the model like forgets some of what you originally told it to do. And so we're really excited about things like larger effective context windows so that you can have these like gnarly, like really long hundreds of thousands of tokens, long tasks, and make sure that quad code is on track the whole way through. Like that would be a huge lift, I think, not just for quad code, but for every coding company.

Swyx [00:54:04]: Fun story from David Hershey's keynote yesterday. He actually misses the common sense of 3.5 because 3.7 being so persistent. 3.5 actually had some entertaining stories where apparently it like gave up on tasks and just 3.7 doesn't. And when quad 3.5 gives up, it started like writing a formal request to the developers of the game to fix the game. And he has some screenshots of it, which is excellent. So if you're listening to this, you can find it on the YouTube because we'll post it. Very, very cool. One form of failing, which I kind of like. I kind of wanted to capture was something that you mentioned while we're getting coffee, which is that quad code doesn't have that much between session memory or caching or whatever you call that. Right. So it reforms the whole state for whole coffee every single time. So as to make the minimum assumptions on the changes that can happen in between. So like how consistent can it stay? Right. Like I said, I think that one of the failures is that it forgets what it was doing in the past. Unless you explicitly opt in via Claude.md. Or whatever. Is that something you worry about?

Cat [00:55:11]: It's definitely something we're working on. I think like our best advice now for people who want to resume across sessions is to tell Claude to, hey, like, write down the state of this session into this text doc. Probably not the Claude.md, but like in a different doc. And in your new session, tell Claude to read from that doc. Yeah. But we plan to build in more native ways to handle this specific workflow.

Boris [00:55:36]: There's a lot of different cases of this, right? Like sometimes you don't want Claude to have the context. And it's sort of like Git. Sometimes I just want, you know, a fresh branch that doesn't have any history. But sometimes I've been working on a PR for a while and like I need all that historical context. Right. So we kind of want to support all these cases. And it's tricky to do a one size fits all. But generally our approach to code is to make sure it works out of the box for people without extra configuration. So once we get there, we'll have something.

Alessio [00:56:02]: Do you see a future in which the commits play a bigger part? Of like in a pull request, like how do we get here? You know, there's kind of like a lot of history in how the code has changed within the PR that informs the model. But today the models are mostly looking at the current state of the branch. Yeah.

Boris [00:56:20]: So Claude for, you know, for some things that will actually look at the whole history. So for example, if it's writing, if you tell Claude, hey, make a PR for me, it'll look at all the changes since your branch diverged from main. And then, you know, take all of those into account when generating the pull request message.

Cat [00:56:35]: You might notice it running git diff as you're using it. I think it's pretty good about just tracking, hey, what changes have happened on this branch or so far and just make sure that it's like understands that before continuing on with the task.

Boris [00:56:51]: One thing other people have done is ask Claude to commit after every change. You can just put that in the ClaudeMD. There's some of these like power user workflows that I think are super interesting. Like some people are asking Claude to commit after every change. So that they can rewind really easily. Other people are asking Claude to create a work tree every time so that they could have, you know, a few Claudes running in parallel in the same repo. I think from our point of view, we want to support all of this. So again, Claude code is like a primitive and it doesn't matter what your workflow is. It should just fit in.

Alessio [00:57:23]: I know that 3.5 Haiku was the number four model on AIDR when it came out. Do you see Claude code have a world in which you have like a commit hook that uses maybe Haiku to do something like the lintel? Or stuff and things like that continuously. And then you have 3.7 as the more.

Boris [00:57:39]: Yeah, you could actually do this if you want. So you're saying like through like a pre-commit hook or like a GitHub action or?

Alessio [00:57:46]: Yeah, yeah, yeah. Say, well, kind of like run Claude code, like the lintel example that you had. I want to run it at each commit locally, like before it goes to the PR.

Boris [00:57:55]: Yeah, so you could do this today if you want. So in the, you know, if you're using like Husky or like whatever pre-commit hook system you're using or just like Git pre-commit hooks. Just add a line quad dash p.

Alessio [00:58:05]: And then, you know, whatever instruction you have and that'll run every time. Nice. And you just specify Haiku. It's really no difference, right? It's like maybe it'll work a little worse, but like I still support it.

Boris [00:58:15]: Yeah, you can override the model if you want. Generally, we use Sonnet. We default to Sonnet for most everything just because we find that it outperforms. Yeah. But yeah, you can override the model if you want. Yeah.

Swyx [00:58:25]: I don't have that much money to run commit hook on 3.7. Just as a side note on pre-commit hooks. I have worked in places where they insisted on. Having pre-commit hooks. I've worked at places where they insisted they'll never do pre-commit hooks because they get in the way of committing and moving quickly. I'm just kind of curious. Like, do you have a stance or recommendation? Oh, God. That's like asking about tabs versus spaces. A little bit. But like, you know, I think it is easier in some ways to like if you have a breaking test, go fix the test with clock code. In other ways, it's more expensive to run this at every point. So like there's trade-offs.

Boris [00:59:02]: I think for me, the biggest trade-off is. You want the pre-commit hook to run pretty quickly so that if you're either if you're a human or if you're a quad, you don't have to wait like a minute. Yeah. So you want the fast version. Yeah. Yeah. So generally, you know, pre-commit, you know, for our code base should run. Just types. Yeah. It's like less than, you know, five seconds or so. Like just types and lint maybe. And then more expensive stuff you can put in the GitHub action or GitLab or whatever you're using. Agreed.

Swyx [00:59:26]: I don't know. I like putting prescriptive recommendations out there so that people can take this and go like, this guy said it. We should do it in our team. And then that's that's a basis for decisions. Yeah. Cool. Any other technical stories to tell? You know, I wanted to zoom out into more product-y stuff, but, you know, you can get as technical as you want.

Boris [00:59:47]: I don't know. Like one anecdote that might be interesting is the night before the code launch, we were going through to burn down the last few issues and the team was up like pretty late trying to trying to do this. And one thing that was bugging me for a while is we had this like markdown rendering that we were using. And it was just, you know, it's like the markdown rendering in Quad today is beautiful. And it's just like really nice rendering in the terminal. And it is bold and, you know, headings and spacing and stuff very nicely. But we tried a bunch of these off the shelf libraries to do it. And I think we tried like two or three or four different libraries. And just nothing was quite perfect. Like sometimes the spacing was a little bit off between a paragraph and like a list. Or sometimes the text wrapping wasn't quite correct. Or sometimes the colors weren't perfect. So each one had all these issues. And all these markdown renderers are very popular. And they have, you know, thousands of stars on GitHub and have been maintained for many years. But, you know, they're not really built for a terminal. And so the night before the release at like 10 p.m., I'm like, all right, I'm going to do this. So I just asked Quad to write a markdown parser for me. And they wrote it. Zero shot. Yeah. It wasn't quite zero shot. But after, you know, like maybe like one or two prompts, they got it. And, you know, that's the markdown parser that's in code today and the reason that markdown looks so beautiful. That's a fun one.

Swyx [01:00:59]: It's interesting what the new bar is, I guess, for implementing features like this exact example where there's libraries out there that you normally reach for that you find, you know, some dissatisfaction with for literally whatever reason. You could just spin up an alternative and go off of that. Yeah.

Boris [01:01:17]: I feel like AI has changed so much and, you know, literally in the last year. But a lot of these problems are, you know, like the example we had before, a feature you might not have built before or you might have used a library. Now you can just do it yourself. Yeah. I think the cost of writing code is going down and productivity is going up. And we just have not internalized what that really means yet. Yeah. But yeah, I expect that a lot more people are going to start doing things like this, like writing your own libraries or just shipping every feature.

Alessio [01:01:43]: Just to zoom out, you obviously do not have a separate Claude Code subscription. I'm curious what the roadmap is. Like, is this just going to be a research review for much longer? Are you going to turn it into an actual product? I know you were talking to a lot of CTOs. Is there going to be a Claude Code enterprise? What's the vision?

Cat [01:02:04]: Yeah. So we have a permanent team on Claude Code. We're growing the team. We're really excited to support Claude Code in the long run. And so, yeah, well, we plan to be around for a while. In terms of subscription itself, it's something that we've talked about. It depends a lot on whether or not most users would prefer that over pay-as-you-go. So far, pay-as-you-go is the most popular. I think pay-as-you-go has made it really easy for people to start experiencing the product because there's no upfront commitment. And it also makes a lot more sense with a more autonomous world in which people are scripting Claude Code a lot more. But we also hear the concern around, hey, I want more price predictability if this is going to be my go-to tool. So we're very much still in the stages of figuring that out. I think for enterprises, given that Claude Code is very much like a productivity multiplier for ICs. And most ICs can adopt it directly. We've been just like supporting enterprises as they have questions around security and productivity monitoring. And so, yeah, we've found that a lot of folks see the announcement and they want to learn more. And so we've been just engaging in those.

Swyx [01:03:16]: Do you have a credible number for the productivity improvement? Like for people who are not in Anthopic that you've talked to, like, you know, are we talking 30%? Some number would help justify things.

Boris [01:03:29]: We're working on getting this. Yeah. We should. Yeah. It's something we're actively working on. But anecdotally for me, it's probably 2x my productivity. My God. So I'm just like, I'm an engineer that codes all day, every day. Yeah. For me, it's probably 2x. Yeah. I think there's some engineers at Anthropic where it's probably 10x their productivity. And then there's some people that haven't really figured out how to use it yet. And, you know, they just use it to generate like commit messages or something. That's maybe like 10%. So I think there's probably a big range and I think we need to study more.

Cat [01:03:58]: For reference, sometimes we're in meetings together and sales or compliance or someone is like, hey, like, we really need like X feature. And then Forrest will ask a few questions to like understand the specs. And then like 10 minutes later, he's like, all right, well, it's built. I'm going to merge it later. Anything else?

Cat [01:04:18]: So it definitely feels definitely far different than any other PM role I've had.

Alessio [01:04:23]: Do you see yourself opening that channel of the non-technical people? Talking to Claude Code and then the instance coming to you, which like they're ready to find and talk to it and explain what they want. And then you're doing kind of the review side and implementation.

Boris [01:04:37]: Yeah, we've actually done a fair bit of that. Like Megan, the designer on our team, she is not a coder, but she's landing pull requests. She uses code to do it. She designs the UI? Yeah.

Cat [01:04:48]: And she's landing PRs to our console product. So it's not even just like building on Claude Code. It's building like across our product suite in our monorepo. Right.

Boris [01:04:57]: Yeah, yeah. And similarly, you know, our data scientist uses Claude Code, right? Like, you know, like BigQuery queries. And there was like some finance person that went up to me the other day and was like, hey, I've been using Claude Code. And I'm like, what? Like, how do you even get it installed? You don't even use Git. And they're like, yeah, yeah, I figured it out. And yeah, they're using it. They're like, so Claude Code, you can pipe in because it's a Unix utility. And so what they do is they take their data, put it in a CSV, and then they take the, they cat the CSV, pipe it into code. And then they ask it code questions about the CSV. And they've been, they've been using it for that.

Alessio [01:05:32]: Yeah. That would be really useful to me because really what I do a lot of the times, like somebody gives me a feature request. I kind of like rewrite the prompt. I put it in agent mode and then I review the code. It would be great to have the PR wait for me. I'm kind of useless in the first step. Like, you know, taking the feature request and prompting the agent to write it. I'm not really doing anything. Like my work really sucks. It starts after the first run is done.

Swyx [01:05:59]: So I was going to say, like, I can see it both ways. So like, okay, so maybe I would simplify this to in the, in the workflow of non-technical people in the loop. Should the technical person come in at the start or come in at the end? Right. Or come in at the end and the start. Obviously that's the highest leverage thing because like sometimes you just need the technical person to ask the right question that the non-technical person wouldn't know to ask. And that really affects the implementation.

Alessio [01:06:23]: But isn't that the bitter lesson of the model? That the model will also be good at asking the follow-up question. Like, you know, if you're like telling the model, hey.

Swyx [01:06:31]: That's what I trust the model to do the least. Right. Sorry, go ahead. Yeah.

Alessio [01:06:35]: If you're like the model, hey, you are the person that needs to translate this non-technical person request into the best prompt for Claude Code to do a first implementation. Yeah. Like, I don't know how good the model would be today. I don't have an eval for that. But that seems like a promising direction for me. Like it's easier for me to review 10 PIs. Than it is for me to take 10 requests, then run the agent 10 times and then wait for all of those runs to be done and review.

Boris [01:07:04]: I think the reality is somewhere in between. We spend a lot of time shadowing users and watching people at kind of different levels of seniority and kind of technical depth use code. And one thing we find is that people that are really good at prompting models from whatever context, maybe they're not even technical, but they're just really good at prompting. They're really effective at using code. And if you're not very good at prompting, then code tends to go off the rails more and do the wrong thing. So I think in this stage of where models are at today, it's definitely worth taking the time to learn how to prompt model as well. But I also agree that, you know, maybe in a month or two months or three months, you won't need this anymore because, you know, the bitter lesson always wins. Please. Please do it. Please do it, Anthropic.

Swyx [01:07:43]: I think there's a broad interest in people forking or customizing Claude Code. So we have to ask, why is it not open source? We are investigating. Ah.

Boris [01:07:54]: Okay. So it's a not yet. There's a lot of tradeoffs that go into it. On one side, our team is really small and we're really excited for open source contributions if it was open source. But it's a lot of work to kind of maintain everything and like look at it. Like I maintain a lot of open source stuff and a lot of other people on the team do too. And it's just a lot of work. Like it's a full time job managing contributions and all this stuff.

Swyx [01:08:21]: Yeah. I'll just point out that you can do source available and that's, you know, solves a lot of individual use cases. Without going through the legal hurdles of a full open source.

Boris [01:08:30]: Yeah, exactly. I mean, I would say like there's nothing that secret in the source. And obviously it's all JavaScript, so you can just decompile it. Compilation's out there. It's very interesting. Yeah. And generally our approach is, you know, all the secret sauce, it's all in the model. And this is the thinnest possible wrapper over the model. We literally could not build anything more minimal. This is the most minimal thing. Yeah. So there's just not that much in it.

Swyx [01:08:51]: If there was another architecture that you would be interested in that is not the simplest, what would you use? What would you have picked as an alternative? You know, like, and we're just talking about agentic architectures here, right? Like there's a loop here and it goes through and you sort of pull in the models and tools in a relatively intuitive way. If you were to rewrite it from scratch and like choose the generationally harder path, like what would that look like?

Cat [01:09:14]: Well Boris has rewritten this, Boris and the team have rewritten this like five times.

Swyx [01:09:19]: Oh, that's a story.

Boris [01:09:21]: Yeah.

Cat [01:09:21]: It is very much the simplest thing I think by design. Okay.

Boris [01:09:25]: So it's got simpler. It got simpler. It doesn't go more complex. We've rewritten it from scratch. Yeah. Probably every three weeks, four weeks or something. And it just like all the, it's like a ship of Theseus, right? Like every piece keeps getting swapped out and just cause quad is so good at writing its own code. Yeah.

Swyx [01:09:41]: I mean, at the end of the thing, the thing that's breaking changes is the interface, the quad MCP, blah, blah, blah. Like all that has to kind of stay the same unless you really have a strong reason to change it. Yeah.

Cat [01:09:51]: I think most of the changes are to make things more simple. Right. Like to share interfaces across different components.

Cat [01:10:00]: Because ultimately we just want to make sure that the context that's given to the model is in like the purest form and that the harness doesn't intervene with the user's intent. And so very much a lot of that is just like removing things that could get in the way or that could confuse the model. Yeah.

Boris [01:10:16]: On the UX side, something that's been pretty tricky and the reason that, you know, we have a designer working on a terminal app is it's actually really hard to design for a terminal. There's just like, there's not a lot of literature on this. Like I've been doing product for a while. So like I kind of know how to build for apps and for web and, you know, for engineers in terms of like tools that have DevEx, but like terminal is sort of new. There's a lot of these really old terminal UIs that use like curses and things like this and for very sophisticated UI systems. But these are all, they all feel really antiquated by the UI standards of today. And so it's taken a lot of work to figure out how exactly do you make the app feel like fresh and modern and intuitive in a terminal. Yeah. And we've had to come up with a lot of that design language ourselves.

Swyx [01:11:00]: Yeah. I mean, I'm sure you'll be developing over time. Cool. Closing question. This is just more general. Like, I think a lot of people are wondering, Anthopic has, I think it's easy to say the best brand for AI engineering, like, you know, developers and coding models. And now with like the coding tool attached to it, it just has the whole product suite of model and tool and protocol. Right. So, and I don't think this was obvious one year ago today. Like when Claude 3 launched, it was just, it was just more like, this is a general purpose models and all that. But like Claude Sonnet really took the scene as like the sort of coding tool of choice and I think built Anthopic's brand and you guys are now extending. So why is Anthropic doing so well with developers? Like, it seems like there's just no centralized, every time I talk to Anthropic people, they're like, oh yeah, we just had this idea and we pushed it and it did well. And I'm just like, there's no centralized strategy here. Or like, you know, is there an overall overarching strategy? Sounds like a PM question to me. I don't know. I would say like Dario is not like breathing down your neck going like build the best dev tools. Like he's just, you know, letting you do your thing. Everyone just wants to build awesome stuff.

Cat [01:12:11]: It's like, I feel like the model just wants to write code. Yeah, I think a lot of this trickles down from like the model itself being very good at code generation. Like we're very much building off the backs of an incredible team. I think that's the only reason why Claude Code is possible. I think there's a lot of answers to why the model itself is good at code, but I think like one high level thing would be so much of the world is run via software and there's like immense demand for great software engineers. And it's also something that like you can do almost entirely with just a laptop or like just a dev box or like some hardware. And so it just like is an environment that's very suitable for LLMs. It's an area where we feel like you can unlock a lot of economic value by being very good at it. There's like a very direct ROI there. We do care a lot about other areas too, but I think this is just one in which the models tend to be quite good and the team's really excited to build products on top of it.

Alessio [01:13:17]: And you're growing the team you mentioned. Who do you want to hire? Yeah. We are. Who's like a good fit for your team?

Boris [01:13:24]: We don't have a particular profile. So if you feel really passionate about coding and about the space, if you're interested in learning how models work and how terminals work and how like, you know, all these technologies that are involved. Yeah. Hit us up. Always happy to chat.

Alessio [01:13:42]: Awesome. Well, thank you for coming on. This was fun.

Cat [01:13:45]: Thank you.

Boris [01:13:45]: Thanks for having us.

Cat [01:13:46]: This was fun.

Update: Steven Sinofsky agrees: 1) Schticks get tired, 2) Helpfulness gets less helpful as skill level improves. Society advances by the number of operations you can do without thinking, not by the number of times you are complimented on your brilliance.

The original title of this post was “Please don’t put memory everywhere”. The proximate cause was the rollout of improved memory in ChatGPT1 this month. Many many many people at OpenAI hyped it up, the priesthood anointed it “possibly life-changing”, and Rich Sutton called it the Era of Experience2. However, the number of serious people reporting issues (bad assumptions, hesitation to use, social calculation) have caused even some of the OpenAI folks to concede3 that we may need an “innie” and “outie” ChatGPT with severed memories between them.

But this goes deeper than memory. Of course there’s a relevant Silicon Valley clip:

To quote:

Gilfoyle: It's bad enough it has to talk, does it need fake vocal tics like “uh”?

Dinesh: Well that just makes it sound more human.

Gilfoyle: Humans are shit. This thing is addressing problems that don't exist. It's solutionism at its worst. We are dumbing down machines that are inherently superior.

Dinesh: Whatever, you gave your server a human name, you named it Anton.

Gilfoyle: yeah but Anton doesn't call me anything. He grimly does his work, and he sits motionless till it's time to work again. We could all take a page from his book.

The logical end state of making our agents stateful and personable are actually not too far off from this meme:

At the heart of the conflict are the two schools of thought in building AI products:

• The Clippy School: the chatty, supportive, personal “friend”/”companion”, where we prioritize “personality” and “personalization”, disfluencies and human affects.

• The Anton School: the concise, efficient, auditable tool/assistant, where the only things we care about are either SOTA quality or speed/reliability of “good enough”.

This ongoing gap is as wide as the wordcel/shape rotator divide of 2022:

• the Antonites cannot understand why there are so many people excited about AI Consciousness. They say F*** you, show me the prompt. They sit on Hacker News and complain about reliability. They often don’t believe that LLMs demonstrate emergence or reasoning, and are aligned with Team Big Workflows.

• the Clippies cannot understand why people are creeped out by permanent, pervasive memory by default in ChatGPT. They want to remove knobs and selectors from AI interfaces. They talk about Claude as “he” and have the One Thread. They burn AGI effigies and go to exclusive parties with Team Big Model.

The Clippy vs Anton divergence is the current most serious obstacle to general intelligence in a practical sense as well — separate post-trains that optimize for chat use-cases are now being produced by both Llama 4 and chatgpt-latest, variously boosting performance by 144 Elo and 123 Elo4 respectively, whereas 20th-ranked Claude 3.7 Sonnet does a lot better on code use-cases like Aider5.

This past 72 hours in ChatGPT-land have been a rollercoaster:

With some early positive reviews now completely drowned out by the acknowledged extreme glazing of the model, initially funny, but very quickly tiresome and patronizing:

It is unclear the extent to which Mikhail Parakhin, former Bing AI lead who worked closely with OpenAI, speaks for OpenAI, but he blames this on an RLHF oopsie:

Genuine mistakes and legitimate concerns about the tone-deaf rollout aside, there is of course a Pareto frontier of Helpful, Harmless and Honest per a given Model Spec that ChatGPT-4o of April 2025 failed6, but even if the ChatGPT update had stayed on the frontier, there is STILL a choice to be made between “brutal honesty” (Anton) and “diplomatic/supportive” (Clippy) that humans cannot unanimously decide on (often, not even the same human would agree on vs their own past preference).

Good RLHF can move models to the HHH frontier, and can even move out the frontier. Good Memory can remember preferences and personalize model behavior on the fly. But until ChatGPT can read our minds and moods, it will never really know which of the many selves we contain are currently in the driver’s seat.

In this case, for the here and now, the simple “out” is to offer toggles for personality. (Custom GPTs should have unlocked this, but the fact that they aren’t is telling)

Of course, the toggle is itself an implicit admission that we have yet failed to reach AGI.

This isn’t really a post about AI so much as about HCI - Human-Computer Interaction, and an age-old debate about what role we want computers to play in our lives. Everybody says they want computers to augment us, not replace us, but then their actions often lead to the exact opposite result. This is a battle as old as technology.

In Tech’s Two Philosophies, Ben Thompson contrasts Steve Jobs’ famous marketing pitch for Apple - that computers are the bicycle of the mind - against Mark Zuckerberg’s view of technology - “not only does Facebook want to do things for you, it wants to do things… [that] would not be done otherwise”. Less diplomatically put, Facebook wants to fill your mind and influence it. If Apple makes bicycles, Facebook makes opium (and connections, but that’s merely its origin story).

Here is the very crux of the Anton vs Clippy battle - a revisitation of Tech’s core dichotomy of its place in human society.

There are some ideas as to what the Improved Memory system prompt is, but of course one never knows for sure and these do change over time.

Imagine what using ChatGPT for throwaway questions and tasks would -ACTUALLY- be like if what Rich envisions here actually came to pass…

Of course, sama says you can opt out of Improved Memory, and this is technically true in that “Temporary conversation” is one click away. But the new chat experience always toggles that off again (as of this writing). Every competent technologist knows the power of defaults.

As of time of writing, ChatGPT-4o-latest (2025-03-26) is 1408 score vs GPT-4o-2025-05-13 is 1285 - the more recent 2024-08-06 variant scores *lower* at 1265.

The gradual evolution of Claude from its early (poorly received) consumer centric billboards and general reputation for personality and “big model smell”, to today’s image as a favorite developer AI lab from models to tools to protocols, has a bit of a Star is Born contrast dynamic as OpenAI, the “API for AGI” platform pours ever more effort in releasing new features in premium ChatGPT tiers, and looks more and more like a consumer AI company comparable to Google, with plans for its own social network.

Specifically, it tries to be so “helpful” that it becomes disingenuously dishonest and therefore loops back around to extremely unhelpful

Expo slots are sold out, and speaker CFPs are coming together! We’re really excited to bring you a great show. Super Early Bird tix for AI Engineer World’s Fair sell out this week, so get your tickets now and have your voice heard in the 2025 survey!

In our Winds of AI Winter in July ‘24 we first introduced the “LLM OS” as one of the new battlegrounds of the Four Wars of AI. Everything that is a tool in ChatGPT, can (and will) turn into a startup. After having Exa (Search) and Browserbase (Browser) on the pod, it’s time for E2B (Code Interpreter).

If you’ve followed us for a while, you might be familiar with them as they first released in 2023 as a cloud for AI agents, and were one of the first places you could use swyx’s smol-developer!

In the last 2 years, they have grown from a handful of users to being used by ~50% of the Fortune 500 and generating millions of sandboxes each week for their customers. As the “death of chat completions” approaches, LLMs workflows and agents are relying more and more on tool usage and multi-modality.

The most common use cases for their sandboxes:

• Run data analysis and charting (like Perplexity)

• Execute arbitrary code generated by the model (like Manus does)

• Running evals on code generation (see LMArena Web)

• Doing reinforcement learning for code capabilities (like HuggingFace)

The big trend that E2B is building on is that infrastructure management is moving away from devs to AI:

• 2023: AI Engineers use E2B to build code interpreter into their product

• 2024: E2B powers generative UIs and vibe coded apps

• 2025: AI agents use E2B to manage a virtual computer, run code and serve results back to the users

The amount of time each sandbox ran for went up >10x from 2024 to 2025, mostly driven by new long-running agents experiences like Manus that their customers are building.

If you missed, you can check out Vasek’s full workshop from AIE World’s Fair (and signup for our next one!)

• E2B

• Building Manus AI (with E2B)

• LMArena + E2B

• Open-R1 + E2B

• Composio

• Mastra

• Exa

• Browserbase

• Why Isn't Usage Based Billing A Bigger Category?

• [00:00:00] Introductions

• [00:00:37] Origin of DevBook -> E2B

• [00:02:35] Early Experiments with GPT-3.5 and Building AI Agents

• [00:05:19] Building an Agent Cloud

• [00:07:27] Challenges of Building with Early LLMs

• [00:10:35] E2B Use Cases

• [00:13:52] E2B Growth vs Models Capabilities

• [00:15:03] The LLM Operating System (LLMOS) Landscape

• [00:20:12] Breakdown of JavaScript vs Python Usage on E2B

• [00:21:50] AI VMs vs Traditional Cloud

• [00:26:28] Technical Specifications of E2B Sandboxes

• [00:29:43] Usage-based billing infrastructure

• [00:34:08] Pricing AI on Value Delivered vs Token Usage

• [00:36:24] Forking, Checkpoints, and Parallel Execution in Sandboxes

• [00:39:18] Future Plans for Toolkit and Higher-Level Agent Frameworks

• [00:42:35] Limitations of Chat-Based Interfaces and the Future of Agents

• [00:44:00] MCPs and Remote Agent Capabilities

• [00:49:22] LLMs.txt, scrapers, and bad AI bots

• [00:53:00] Manus and Computer Use on E2B

• [00:55:03] E2B for RL with Hugging Face

• [00:56:58] E2B for Agent Evaluation on LMArena

• [00:58:12] Long-Term Vision: E2B as Full Lifecycle Infrastructure for LLMs

• [01:00:45] Future Plans for Hosting and Deployment of LLM-Generated Apps

• [01:01:15] Why E2B Moved to San Francisco

• [01:05:49] Open Roles and Hiring Plans at E2B

Alessio [00:00:03]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host Swyx, founder of Small AI.

swyx [00:00:11]: Hey, and this is a little bit of a Latent Space Discord reunion, because we have Vasek in the house from B2B, welcome. Hey, good to see you guys, thank you for having me. Help me with your last name, because I realize I've never had to pronounce it. Mlejski.

Vasek [00:00:25]: No, no, well, I guess close in, but Mlejski. Mlejski. Yeah, and my official legal first name is Václav, but everyone pronounces it as Václav, which I hate, so it's just Vasek.

swyx [00:00:37]: Okay, awesome. We're both invested in E2B in different ways, but you and I go back the furthest. I just realized three years ago when you were working on DevBook, and you were interested in that developer experience angle, and somehow you pivoted to E2B, maybe you want to tell that story.

Vasek [00:00:55]: Yeah, so Tomas, my co-founder, who's our CTO. We've been interested in DevTools for quite a long time, like six, eight years. And before DevBook, there was a bunch of iterations. There were different iterations and pivots of DevBook. We just stopped renaming things at some point and just went with DevBook. The one you are talking about was interactive documentation for developers. So basically the idea was, we wanted to, instead, you as a developer, when you come to a tools doc, a website, instead of reading about everything and then Googling it and trying it in your coding editor, the idea was to give you interactive experience in the browser. So you would have pre-made interactive guides, playgrounds, you could try things right away. And the company, the owner of the docs, would prepare the experience for you, because you would also be trying everything in the browser. They would now see if you get stuck anywhere, what you are doing. So you get it very valuable. So we started onboarding analytics. We actually built an interactive playground for Prisma. I think it's still up. They are still using it. The last time I checked, a couple of months ago. So it was like interactive guides and playground that you could try out Prisma without, you know, having to manually set up all the databases and everything. So you could just try Prisma right away. And that was like the very, very first version of actually our infrastructure that we are offering now. Basically sandboxes. It was sandboxes. It literally was sandboxes, the same technology, but just completely unskillable.

swyx [00:02:35]: So then did that somehow in 2024-ish turn to E2B? 2023, I think March 2023.

Vasek [00:02:43]: And we were pretty burned out. Tomas and I, we were working from Prague, from the Czech Republic, from my apartment. Like nothing was really moving, no growth. And GPT 3.5 came out, like really first model, kind of good-ish. So we took a like, let's take 10 days break, two weeks break from DevBook and because like everyone was trying things with AI, it was very clear, like this is something where the future might go. So we wanted to just like from out of curiosity, try things out. We wanted to build like a DevIn kind of like thing. The first idea we had was like, let's automate our work because with every project we were starting. There was a set of tools you always wanted, want to integrate in your back end, like Stripe, like for SaaS business, like Stripe, analytics, Slack notifications, emails, sending out emails. And so we gave the agent the tools to run code and we needed some kind of sandbox. We were like, yeah, that's a good coincidence. We have sandbox from DevBook and we posted about it on Twitter. Basically, the agent actually pulled GitHub repository. Wrote code, started the server, tested everything. And at that point, I think we deployed it to Railway. Railway had like the best DX and it was easiest to just plug it, plug it in into the agent. I tweeted about it and I think Greg Brockman like retweeted it like, hey, like I don't remember exactly what he said. I need to find a tweet. But it was around the time where OpenAI was retweeting or OpenAI co-founders were retweeting all the things that people are doing. Like to show like what you can do with GPT-3.5 and people like it had like half a million views after a few days. And so we were with Tomas, we were like, we got to do something like people are interested in. So we just open sourced it, the repository, and it was named, the organization was like the AI company. Like we had no name. I kind of wish like we could have like a stick, like a legal stick with it. We just came up with it. With a name, it was like E2B because you take English, you convert it to bits. That's how it started. We started building community around it. And like two days later, we were like, let's focus on the sandbox part, not on the agent part. And we actually had a bunch of hypothesis behind it, like why it might be more interesting than building the agent.

Alessio [00:05:19]: Yeah. And then you had the small developer, run small developer on E2B.

swyx [00:05:23]: That was a little bit later. Yeah. What was the, what, what, so this is in 2020.

Alessio [00:05:28]: When you first launched. E2B was kind of like a AI agents cloud. What did you call it? 23.

Vasek [00:05:33]: The hypothesis was like agents, code gen agents especially will need some kind of environment to run the code. And the same way developer needs a laptop or something, you know. So, but we, we struggled a lot with how the product should actually look like and what's like a go to market, the first version. And so the high level idea was like, we will host your agent. Everything from actually deploying it to then monitoring it and, and it will also have this environment running the code. One of the first like test project was taking Sean's small agent project and deploying it inside our sandbox and just giving the agent tools to pull the GitHub repositories, work on that, do a PR and then post it on GitHub, the PR on GitHub, which was very, very popular. Like, like there was. It was clear, it was clear that like, um, there was something very interesting.

swyx [00:06:31]: It's amazing how, how much people went with that. It was literally not meant to do that. It was meant to do Chrome extensions. Yeah.

Vasek [00:06:37]: And I think like you really, you had good insight that you let the agent plan the work in, in Markdown file basically. And like right on the spec, which I think even now, when you look at deep research agents, like that's sort of oftentimes what they are doing, like they plan everything. Yeah.

swyx [00:06:55]: These days I would say they should do more structured output than Markdown. Um, but, uh, you know, that's, that's an implementation detail. Yeah.

Vasek [00:07:01]: So that was taking off, but it attracted a little bit different audience than we wanted. Um, so, uh, because it's, it's attracted a lot of people who nowadays would be using tools like lovable, for example, you know, they just wanted to build projects for them. And of course, like it wasn't really working at the time, like it worked for like one simple website, but the moment you wanted something more complex, like it was.

swyx [00:07:27]: I've been also reflecting on why I stopped working on it and there was actually a, uh, so when I built it, it was with, uh, cloud three, the new cloud three launches was trying to utilize the a hundred K contexts of cloud three and I use the same project to do this, to try to repeat the demo that I made for myself a month afterwards. And it wasn't anywhere as smart. So cloud three got dumber, but it looks like I like made up the demo or something, but no, like literally I just reran the same code. Yeah. And it just, it just was not as capable. And I, I mean, I think to some extent, this is like, RN Jesus, you know, hurting me or like, it's the title, it's like a different month. So like the, the model is like different or something, but I think also like basically people had this vision of what they wanted and then they tried to do it in reality and they couldn't do it because the models are not ready.

Vasek [00:08:16]: So a lot of feedback we got, um, at some point people thought like small agents is, is from us because like we had a website, like deploy small agent, but we were no, no, like that's, that's like Sean's where he should, he should go to his repository, uh, submit an issue.

swyx [00:08:32]: No, they could, they could clone it.

Vasek [00:08:34]: It was like, you know, I did it however you want for, for, for us, it was more like a test. If the environment, the sandbox is useful for the agent, if it can sort of scale and, and I can work, which worked well, but then it took us another, I would say six months to actually find the right, you know, go to market strategy, which was. Code interpreting for us. So typically AI data analysis, uh, data visualization inside like a headless Jupiter type of, uh, notebook and environment. Specifically Jupiter? It doesn't need to be Jupiter. The important part is that, um, you don't need to explain the model and the model doesn't need to care about how to keep the state of the program running. So it was, it was, especially with our earlier models, I think the models are now smarter, but they kept producing like code snippets and thought like. Yeah. They can reference to previous, uh, you know, variables and, and, and functions, definitions, uh, which if you had only normal code execution, like you run the code snippet and then you finish, it wouldn't work. So you need some kind of rebel environment. And, um, what was, I think, especially early on, like Python was the language that the models were probably the best or one of the first languages where the models were working very well. Looking back at it, there was like a really strong pool from people just wanting to visualize data and talk with their data. Python was really good at it and it makes this well with Jupiter type of environment because you get charts out of the box and you can also support interactive charts. Jupiter itself isn't the right environment to actually do it because like, uh, because all the technical problems that you will run into once you are start doing, once you start doing it on scale. So it's, it, it gets slower and slower and actually like what we are coming into is like, we are building our own thing, uh, internally, just like to support all of them specifically, it's got like your own runtime.

Alessio [00:10:35]: At what point did you start going from just code interpreter, run code to like expand? Because now you have, you know, people doing RFT, you have computer use and all of those things. Like when were the models ready for people to start using it? Just, you know, you demoed at one of our first events as well. And I think there's always this lag between the infrastructure. Structure that you build and like the capabilities of the model. When did you go from just code interpreter to start saying, okay, now it's time to do computer use. Now it's time to do RFT.

Vasek [00:11:04]: That was probably end of 24, uh, start of 25. So when you look even at our data, like use 24, we are growing, we are growing good, but 25 is like up to the right. And so it feels like at 24 people are like figuring out these agents, um, and, and building them and trying them. And 25 is everyone. Moving them into production and finding more and more use cases around end of 24, start of 25. We started seeing things like using sandbox for, uh, reinforcement learning type of use cases or using sandbox as a, for computer use, uh, which was very interesting when Anthropic launched their computer use. We had like a desktop version of a sandbox that was sitting like in our GitHub repository for six months. We were like, okay, this is probably interesting, but no model can actually use it. So, uh, when Anthropic announced it, it was like, oh, like we have something here, uh, we can show you using, uh, with, with like lovables and, and, and Blitz's type of products also like started using the sandbox for more than just like run code snippet, like data analysis and then deep research agents that, that has been something really big in the last few months.

swyx [00:12:19]: How does deep research agents, are you referring to Manus or?

Vasek [00:12:23]: Yeah, Manus.

swyx [00:12:24]: For example, um, one of the companies. Deep research, I typically think of as like a, uh, uh, search, like a web search heavy task. Doesn't really use a code interpreter in any way. I have some idea that Manus uses E2B in an interesting code, code interpretive way to do deep research. What's the difference?

Vasek [00:12:42]: Yeah, I think it's a good idea to stop thinking about the sandbox just for code interpreting and more about like a runtime, code runtime for the LLM or the agent. But the use case for the. Sandbox, it's a very horizontal in a sense that it can cover everything from the agent needs to create a file, make a to-do list. It uses a browser, not inside the center where that's like separate, you have browser use, uh, run, uh, use being used for research on the website, but then you download the data somewhere you need to transform the data. You want to do data analysis, you want to actually write a small app, you want to create a, uh, Excel sheet. So it's, uh, the same way you are using, you know, your laptop as a human, that is the same useful for the agent. So you can think about it as more like a dev box. And at the same time, the agent that's using it is also like a very, very, uh, good developer, good accountant, good, uh, like slides, greater researcher. And so you are just basically giving it tools to let it do the job even better and faster.

Alessio [00:13:52]: Yeah. Um, yeah. And when you say up into the right, I just want to share some numbers from the investor updates. So March 24, which we're talking about, I don't know, you shared this, you shared these, you shared these publicly, but yeah, March 24, you were doing 40,000 sandboxes. You want to say how many you've done last month, March 20, 25?

Vasek [00:14:15]: Uh, I 15 million around that.

Alessio [00:14:20]: So in one year you've gone from 40,000 to 50,000. So in one year you've gone from 40,000 to 50,000. And yeah, I think you can kind of see the slope, especially from like the summit, three point seven release. And I think this is like an interesting model versus infrastructure. I think there's this usual, like VCO, we should invest in like the tools, you know, the picks and shovels instead of the application layer. But I think it's, this is like the first time where like the infrastructure is lagging the applications.

Vasek [00:14:45]: That's a good point. Like, I think 24 was all about like the agent couldn't use the whole sandbox. And now, uh, yeah, sometimes we are actually like catching up on, with some features for the, for the LMS, uh, that they need more than what we have, uh, at the moment. Yeah.

swyx [00:15:03]: I guess like what we're doing here, you know, you are another one of the LLMOS companies that we are talking to. We also did one with browser base and, um, I'm not sure who else would qualify under that, that term. Exa. Exa. Yeah. Yeah. Uh, so like, basically like you don't specifically yourself use LLMs. Internally in your products, but you enable others to work, to augment their LLMs with your infrastructure. Yeah. Um, I don't know any reflections on just the general LLMOS landscape. You have other competitors, like, you know, how, how is this evolving? How do you, how do you position in it?

Vasek [00:15:38]: A lot of people are saying like, if you are a GPT wrapper in 23, you were really bad positioned, uh, because like all the value will be captured by the AI labs. It's a good to be GPT wrapper because you get all the advantages from a new model. You just switch it. I mean, the just is like not so simple, probably have evals, um, need to change your prompt a little bit, but I would say it's increasingly easier to switch models. So, uh, we need to think about it the same way. Like you, our users are switching models a lot. We need to be agnostic, uh, to the, to the LLMs. Oftentimes, like people want to deploy us in their cloud or on-prem, on-premise. So that's also something very important. I think a good analogy here. It's sort of like technologically, it's kind of, you want to be the Kubernetes of the world for the agent, but with much better DX, uh, and, and easier, easier to use.

swyx [00:16:32]: One thing I'm thinking about also is like, what is valuable real estate to occupy in the LMOS? Um, and I have this spectrum, I think on the browser based episode I was talking about, like either you can focus on browser emulation or you can focus on the VM, or you can do like a custom Python sandbox, like a model does. You know, like, uh, would you say that you are the most general of all of them?

Vasek [00:16:56]: Yes, it's very general, but that's not really historically how you want to market it because people don't know what to do with it. Okay. Um, so that, like we, we started as, when we had our website, uh, in 24, it said something like computer, like cloud computer for AI. People didn't just understand like what to do with it. So we had to like literally show them, like first you change it to code. Interpreting, because that's something people knew from open AI. And then you just show them very, very specific use case. And you use that to get an early traction and early set of users. And we spent a lot of time, I mean, Teresa from our team spent a lot of time, especially her, uh, on educating the market, you know, and the developers. So, uh, and I think this is the space, the AI space is you sometimes have to like show developers what they might need. And you kinda have to like trust your gut. Like this might go in this direction, maybe like this could make sense and show them like what they can build with it, because it's very hard to imagine what you can build with things that you don't even have. Right. So like, why would you need forking sandboxes or checkpointing sandboxes? Like, how is that useful? Well, it turns out if you are building some Monte Carlo type of a thing, uh, search, uh, for our agent, like it's very useful. Uh, but like to actually go for a developer who might not be like in the AI deep research type of thing. Like, it's not. It's not obvious. So, uh, you want to be agnostic, you want to be general, but you want to also show people like very clear use cases, how they can use you. And over the time they didn't get educated and realize, okay, there's more use cases. I understand the platform. Uh, they start coming up with their own ideas, but onboard people with general use cases is very hard for that's at least that's what we learned hard way. Yeah.

Alessio [00:18:47]: And you also don't really tailor to like the more DevOps infrastructure. Yeah. Person, which I think a lot of the other sandboxes are like, oh, we have a G visor runtime and we have all these like different terms that you don't really know if you're like the AI engineer type.

Vasek [00:19:01]: Yeah, that's exactly true that, um, our user, uh, isn't like an infra engineer. Um, even ML engineer usually isn't our type of a user. It's like AI engineer, you know, like the front definition you have, uh, I still remain very bullish web developers and JavaScript world, TypeScript world. Even though we have like a ton of usage from Python, um, but there's so many web developers and it's easier and easier to use LMS. I really think that you need to cater to these type of developers and make it like things simple for them. And if they want to dive in, they can, chances are like, they don't even want to, they want to focus on building. Yeah. Yeah.

swyx [00:19:41]: Product developers, not infrastructure developers. It's very interesting. Yeah. I mean, like this, this whole GPT wrapper versus model lab thing is not like, you know, it's better to work on wrapper. Over, over model labs, because you know, the model as people are making a lot of money, it's just that there is room for wrappers and the wrappers also do make money. And I think people were not seeing that in 2023. Yeah. And this is what we saw.

Vasek [00:20:02]: It's not, you know, binary. It's not either or.

swyx [00:20:05]: Yeah. Yeah. I think, uh, and then, uh, just a quick check, uh, do you know, like the rough percentage between JavaScript and Python?

Vasek [00:20:12]: A slightly less JavaScript, uh, so like from number of downloads of our SDK. per month, it's like 250,000 JavaScript, close to around half a million Python. Yeah. Something around that. Two to one. Yeah. Yeah.

swyx [00:20:27]: Interesting. I mean, if, if the use case is really for code interpreting, generating charts and all these, then Python wins. Yeah.

Vasek [00:20:34]: Yeah. Exactly. Python wins for that. But once you go into more like, for example, building apps, I generated apps, then it's JavaScript winning, right? Because you probably have, you have frameworks, all the Svelte, Next.js. It's Vue.js, it's frameworks, which is like, um, JavaScript thing. It's interesting because I was thinking that it doesn't really matter what code you want the LM to produce. Like it wouldn't dictate what kind of user is using us. Right. But if you think about it, when developers are building like AI data analysis, it's typically Python developer. When developers building like a type of, um, V zero, like a use case level, like a use case is web developer. It's a product developer. So I don't know. I think this is like super obvious, um, that I wouldn't think that that would be the case.

swyx [00:21:23]: There's all sorts. I mean, both, um, argument is that you should, you should want to use something like a web container where it's like run on your own browser and it's all, it's all, you know, free and very fast. I guess what the one more critical question, like, let's say I do know my infra, well, what is the point of an, uh, cloud for AI, like computer for AI? Like, like, so basically why can I use existing tools like railway? A railway wants to go after your customers, right?

Vasek [00:21:50]: So like, why does there need to be an AI focused, you know, virtual machine, you know, sandbox or execution environment, whatever you call it. What we offer is sort of orthogonal to cloud. So beforehand, you don't know what kind of code you will be running in our cloud. So you can't really optimize it. Right. So there's no like a built deploy step, uh, per se, everything happens ad hoc during runtime. So you need to solve problems. Like you want to install dependencies. Right. You want to be able to pull GitHub repositories very, very fast. So then, uh, how do you, the work workloads we are running can go from five seconds to five hours. So that also like changes even the pricing model a lot. Uh, how do you make sure that everything makes sense for you and for the user? Like, uh, uh, from the pricing point of view, from the like unit economics and also from like just the infrastructure, uh, point of view where the, uh, you know, the sandboxes are getting placed inside your cluster, the security model is, uh, is also different usually because also comes down to, you don't know beforehand what code you will run. So by default, it's untrusted code and you need to have complete isolation between these sandboxes to make sure that you're running the right code. So you need to make sure first that like, if something happens in one sandbox, it doesn't affect other sandboxes, but also you want to know about security inside the sandbox. So as the LLMs are getting better, you want to know what's happening inside the sandbox. There's like a fun story from HuggingFace, uh, when they are using, when they were using us and are using us for their, uh, open R1 model. And one of the developers, uh, he shared it online. So I think it's okay to say that. So he lost access to their cluster because the LLMs are not working. So the LLM decided to change permissions. If that happens with us, we just like kill the sandbox and get, get a new one. Uh, he, uh, it takes like, I don't know, 150 milliseconds for them. Like they had to take down the whole cluster, set up everything. Didn't even know that this happened. So it's like, like the model, the compute model, uh, TLDR is the compute model security model is different from the current cloud providers. And you need to think from the first days about it differently. And then people use.

Alessio [00:24:09]: Depending on the, because the difference that maybe people don't think about is like, you can generate the code with the Python SDK, but it can still run Lua code or like R code. Like it's not always matched to the runtime of the SDK.

Vasek [00:24:23]: Yeah, you can, it's a general machine. So whatever you can run on a Linux, you can run inside a sandbox. And, uh, we had users like running, of course, like Python code, but a C++, we had, um, Fortran, uh, someone running Fortran. Which is like, why they were doing some, some like using very old API for like banking or something like that, but you can also start a server, uh, inside a sandbox. And then you want it to be accessible from the internet. It's a very general machine. And the challenge is like, how do you make everything fast? For example, how do you make everything secure at the same time? You need to make it accessible enough and controllable by the LM and observable for the human. So you're kind of building for two personas for a human developer, the AI engineer. Yeah. And for the LM who's like using the sandbox. Yeah.

Alessio [00:25:12]: The composability thing is something I had not thought about before, but later, like you mentioned, it's like, you know, you might need to run Fortran to access one API. And then in the next step, you'll need to take that data and run it in a Python, uh, script. And then you're going to expose that through JavaScript to something else. And like you guys can switch the runtime. Yeah. Halfway. Yeah.

Vasek [00:25:32]: And, and in idle world, you don't want to kill the sandbox, start a new one, uh, or maybe. Like create a new sandbox template. You just want to keep using the computer. And because we are in cloud, we can do it in a way that you can get more Ram. You can get more CPU, you can get less CPU. So you are really paying only for what you need. And as the LLM is doing more and more, uh, you can have very elastic sandbox and keep adding features. And the goal where we think this is getting going is the LLM, like decides what they want to do and how it wants to have. The sandbox configured. So it's basically starts controlling the infrastructure itself and creating sandboxes themselves.

swyx [00:26:15]: While we're talking about the technical details, I just wanted to let you tell people about any other technical details. Like what is the box where we get, what kind of Linux, uh, what size of whatever, what details matter here?

Vasek [00:26:28]: So you get Ubuntu box, you can customize it and anything Debian based is going to work. You can even add, you know, graphical interface. So if you want. To, you can run like legit, not headless, but like legit Ubuntu computer on it.

swyx [00:26:45]: So that you can do this sort of take out like an old operator type experience. Like you take over control.

Vasek [00:26:50]: We have SDK called desktop SDK that does this for you out of the box and it supports, uh, VNC. Uh, so you can also have like a human in the loop type of a thing and control everything and stream and see what's happening. So by default, you get two CPUs and half a gig of. RAM on the free tier and you can customize it on our landing page. We say up to eight gigs, but if you tell us what's your use case, you can go to 64 gigs of RAM. We have users, uh, using such a buffet sandboxes. You can go to 16 CPUs. I think, uh, if I'm not mistaken, and storage is free. Storage is free. Um, I think there's like a lot of things that are free that we probably need to think about a you know, with the dev tools, especially infra. I always see this pattern. And I had this like naive idea idea as well that founder says like, oh, this AWS pricing, it's terrible. Like you are GCP. Like you have no idea what you are paying for. There's so many like small items you need to pay for. We will start a new infrastructure company. You just were, you're just going to pay $200 per month. And then you just pay for like pure compute that works until you start scaling and you figure out, okay. Like actually like people are doing like weird stuff that I didn't expect. Like, okay. Having lots of traffic producing. We have a customer that proposed petabyte of data. Uh, I mean, it's not free to host petabyte of data and that's growing. So then you start introducing, okay, probably they should pay some amount for ingress, egress, for storage and the pricing gets increasingly complex. So I just think it's very interesting like phenomenon that, uh, you start with this brave idea that everything will be super simple, but.

swyx [00:28:38]: And we only do code interpreting, so we only need to price for compute, right?

Vasek [00:28:42]: Yeah, exactly.

swyx [00:28:44]: And then you wake up in the real world and it's messy. Yeah, so I know about this from my career in cloud. And so the common refrain is that, I call this the first principle of technology, everything can be broken down into some combo of compute, storage, and networking. If you fail to price one of them, you will get abused.

Vasek [00:29:03]: Because you're essentially offering free storage or compute or something, right?

swyx [00:29:08]: Yeah. For those interested in this idea, there's a fourth one, which is basically the control plane, or like the off layer, the IAM policies and all that. And maybe you can call that security as well. It's like the fourth layer that people kind of pay for as its own independent thing. For those also interested, HashiCorp has more breakdowns from this, like David McJanet, which I think is very interesting. If you're just in the business of running a cloud infrastructure company, you should know these things. Many hundreds of businesses have run into the exact same problems. You should just not repeat them and just learn whatever the best practice is.

Vasek [00:29:43]: Yeah, I think the billing model has been figured out many times. So I don't think it's like the challenge is not figuring it out. The challenge is, I think, is introducing it sometimes quick enough. You actually need to do changes on your infrastructure to make sure you know about all this data that's happening and moving one way or another. But yeah, I completely agree with you. Like this problem, we are not the first one that are having it. Yeah. Do you use one of the usage-based billing providers? Orb. We are talking with Orb right now. So we are... Orbin? Meter? Open meter is another one, I think. Or meter. Yeah, metronome.

Alessio [00:30:23]: Metronome, yeah, metronome is the other one.

Vasek [00:30:25]: We have been using Stripe, Stripe's usage, which has been a little bit sometimes rougher around the edges. Yeah, this is the thing, like, we shouldn't spend so much engineering on it. Like, I want to outsource it because that's not our product. Someone else should be, like, focusing on this full time. And it's actually pretty non-trivial, like, to make sure you have everything right. And you really don't want to make mistakes here.

swyx [00:30:48]: Is there anything that you're really looking for that would say, like, OK, that's really what we want? That maybe Orb or Metronome haven't really adjusted for AI yet?

Vasek [00:30:56]: I don't think this is an AI-specific problem. This is infrastructure as you know it. For us, some things that didn't work, when we look at some of the providers where, like, the cut they took from the revenue, for example, they take from you. So some of the pricings, I don't know what's the latest, but, like, I don't remember which one was it, honestly, but I knew that pricing was basically they take a small cut from the revenue each month, which is basically, like, what Stripe does when you are processing payments. It's just like the value was really high. And then it's a lot about how hard it would be to... integrate it, like, how much time we are spending on it, because we know we don't want to build this in-house. It's more like, is the switch worth it, basically?

Alessio [00:31:41]: I'm curious your updated takes. I know you had the why isn't usage-based building a bigger category. I'm curious now with AI, with, like, token-based pricing, if you have updated thoughts.

swyx [00:31:51]: So for people who don't know context, at Netlify, we went from relatively flat tier-based pricing to usage-based billing. Because that's basically how all infrastructure companies should eventually go, because you have some whales who use a lot of infrastructure and some who don't use that much, and you shouldn't charge the same for both of them. The fun insight was that you would think that at an IS, PaaS company, that revenue is, like, the most important problem to work on, and directly impacts the company's revenue and valuation and all that. No engineers wanted to do it. And I was like, why? Like, it's... It's... We actually looked around for a long time, we tried to hire, and then we couldn't hire, so we ended up putting one of our most senior engineers on it, and she took a year to ship the whole billing project that was presumably a board-level objective, which was like, hey, let's change from this pricing plan to this pricing plan. How hard can that be? Turns out very hard, because you have to instrument everything, even the things that you're, like, I don't know if we'll ever use this.

Vasek [00:32:56]: Yeah, that's exactly what I meant. That's why storage is free. Yeah, but the first thing is, like, you need to know about everything that's happening inside of Stanford, inside of the cluster. Yeah, even your logs. How big are your logs? Yeah. It's crazy hard. The pricing is, like, actually the, not the figuring out the business model, but the integration of it and implementation is actually a lot of engineering.

swyx [00:33:18]: Yeah, soft limits, like, hard limits, like, do you cut off people when, once they bust the limits? Probably not, because they get pissed at you. Yeah. And then they also get pissed at you if you don't cut them off, because then you send them a big bill. Yeah. So you just, there's no winning.

Vasek [00:33:29]: Yeah, exactly. Um... Also, additional problem that people are asking us, like, I want to run the agent for five hours, and you can do that, and you might be a very early startup, so, like, our goal isn't to, like, cash you out, our goal is, like, you can use as much of E2B as you can and just grow, but the more, the longer you run the sandboxes, the larger is going to be your bill. So I think that's, like, it doesn't really need to correlate with how much product market fit you have, for example, how much users you have. Because with these agents, they can work for a long time, even if you are, like, a pretty early-stage company, if you have a few users, so...

swyx [00:34:08]: Sure. I mean, yeah, I think, but there's still a question of soft limit, hard limit, that kind of stuff. Yeah, yeah, yeah. So just to answer your question on, like, billing, and for those who are interested, we actually had the CTO of Orb send in a talk for a remote track for the New York summit on, like, what he thinks pricing for agents looks like, and I think, basically, you are reselling tokens, right? The base layer is coming from either your open model cloud provider or your closed model lab API, and you resell them, and a lot of them, some people have a lot of markup on them, like, certain unnamed AI builders, and some people have negative markup on them. Like, they are basically selling you at a discount, you should buy as much as you want, because they are using VC monies to subsidize your thing. And I think that seems fine. People often, Simon Wilson often asks for, like, bring your own key solution, where, like, you know, I will have my control over, like, my relationship with them. I will have my relationships and my pricing and my credits with my LLM providers, and I will, but I want to use your app, so I'll give you my keys, and then you use the keys on behalf of me. That doesn't seem to be as popular as I think, as I understand it. I don't know if you've had that request.

Vasek [00:35:14]: It's sort of like in crypto, use your own hardware wallet. Always going to be, like, less popular than just going with the thing that's...

swyx [00:35:21]: Yeah, yeah. So literally, Alex from OpenRouter is the only person who has implemented this, and it's fine for, you know, individual use cases, because there's a lot of free tiers for it. For individuals. But yeah, I mean, I think pricing-wise, people are trying to move that discussion on, like, you know, are you positive margin on your tokens or are you negative margin on your tokens? And there's some economic reality there, but they're trying to move that to the agent work, which is, what is the value of the human labor you're replacing? Which is a whole different thing, right? So, like, instead of comparing on cost of goods sold, you're comparing on value delivered, and that is much higher.

Alessio [00:36:00]: Yeah, I feel like if we could get to a good market on, like, bid-ask of, like, work being done, and then you can kind of arbitrage how many tokens you need. But I think today the agents are so unreliable and unpredictable that, like, it's hard to price ahead of time. Because you could price any software engineering task per task, right? It's like, do a new button, it's like $500, and then I can arbitrage that. But today there's no certainty of that, and I'm curious.

swyx [00:36:24]: Yeah, you would think, like... Also, this is why I was very excited about Replit when they first launched their marketplace. The marketplace credit thing. I'm not...

Alessio [00:36:32]: If Replit can't do it, then I don't know if anyone can. Right. The other technical thing we talked about before is forking. You also mentioned it before, and sandbox checkpoints and things like that. Is it something you have today, and then what are people using that for? Like, our example was the Cloud Place Pokemon hackathon. Are there more enterprise use cases where you see people request forking and checkpoints?

Vasek [00:36:56]: Yeah, we don't have this publicly now yet. But that's something we are working on to release somewhat soon. We have persistence, which is like the prerequisite to the core. You mount a volume. Yeah, and well, mount a volume, but also memory persistence, which is very interesting. So you basically can pause the whole sandbox while even when all the code is, you know, with all the context of the code execution and resume to it, resume it later and come back to it, I don't know, like two weeks later. And it's still going to be there. But the continuous session time is limited to 24 hours, right? No, that's limited in the beta in 30 days. Okay. Do you mean, are you asking about...

swyx [00:37:35]: I don't know, I was looking at your pricing pages.

Vasek [00:37:36]: Are you asking about like when the sandbox is running? The sandbox can run up to 24 hours, but when it's paused, it can be paused for like a month. I personally think this is like one of the cases where you kind of need to show the developers why it's useful. And this is something I think is going to be very useful as the agents are getting better and LMs are getting better. Because you will be able to parallelize problem solving, essentially. So you will, instead of having a single agent doing one thing, you might have multiple agents trying different paths. And if you imagine like a tree or a graph, every node is like a snapshot sandbox, like a checkpoint sandbox. And then from that node, you fork the sandbox and go to the next state. Eventually you find the right path, right? It's kind of like a, it's a tree search. So the forking and checkpointing solves the problem. It solves the local state problem. It doesn't solve the remote state because that's something you can't really control. But it solves a local state for the agent and can then come back to a state and you don't need to replay the whole session or trying to force the LM to do the same thing again. You just have, you have the chat history and you have even all the steps inside the sandbox that went to it.

Alessio [00:38:55]: Do you feel like you want to help? With like the forking and then the re-merging, because I think people understand the forking, but then it's like, okay, how do I monitor which of the leaves is successful and then how do I merge them back in the thing? Do you think that's something that you want to help people do? Like kind of spread out, parallelize and then find the winner? Or is that something people should do on their own?

Vasek [00:39:18]: I think this is a sort of like a framework discussion on top of E2B. So this is like, we are looking at it. I think eventually like. Okay. We should go more higher level and I don't know if like framework is the right type of a thing. We like to, in the team, think about it as a toolkit, like instead of like building opinionated framework, how to build agents, we give you sort of like a wrapper around E2B that makes it really easy for the LMS to, for example, merge these states or navigate these three. So I think it will eventually move there. I think it's a good question to ask, like how it's going to look like. I still think like building a framework is very hard in the AI as things are moving very, very fast.

swyx [00:40:04]: Curious about frameworks. Do you see any rise in popular frameworks that we should be keeping tabs on?

Vasek [00:40:10]: Well, I have a, I have, I don't know if this is an unpopular opinion, but like people keep telling me, uh, LinkedIn isn't popular, but if you look at the stats, like it has 20 million downloads per month. How can you have not popular framework when it does, it has like 20 million downloads and it's growing, I think. Um, I think there's, there's a slight bubble, uh, in, I don't know if it's like people in SF, but like developers thinking Langchain isn't popular or used. There's one framework that's interesting. I think it's called, uh, Mastra, um, from the, one of the founders of Sam Bagwad. Yeah. He's in the discord. So that I, I like, which looks very promising. I also like it's like TypeScript first, uh, which I thought like is the right decision because as, as we talked about, like, being more bullish on the product developers instead of like a Python machine learning type of a developer, I've seen a lot of more like a toolkits or I don't know how to call them. Like for example, Composio has been one that like gives you all the tools at once. There's been more tools, more tools like that, which is interesting approach. I think also browser base is stagehand is super exciting because in my head, it's not a framework. It's like, it doesn't necessarily. Like dictate how the agent should behave. It just gives it a good tools to navigate a website. And it's very elegant, uh, like it has like, I think they added like three methods by act, see, and, and something.

swyx [00:41:40]: Yeah.

Vasek [00:41:40]: There's three APIs. Yeah.

swyx [00:41:41]: Three is observe. Yeah. Observe. Yeah. Yeah. We talked about it in that episode. No, it's cool. Uh, actually I wasn't expecting that many names to come out, but these are good names for people to know. I would agree with most of them as, as, uh, they're in the conversation of, of tooling. If, and a lot of people listen to us for like. Oh, what's going on in SF, right? Like, yeah.

Vasek [00:41:58]: Actually, it's a good question because like when you ask it, I would expect to be like bigger framework boom out is because like it would make more sense to build more sense to build a framework now instead of 23, because things are a little bit more stable. Big problem with frameworks is that the idea of framework should probably be that things aren't changing underneath your hands, which if you launch your framework in 23, uh, it was, uh, so that's not a fun thing to be at as a developer. Who's using the framework. I think some things are still changing. I mean, things are a hundred percent still changing, but I'll slightly, I would say like some things are clearer than in 23.

swyx [00:42:35]: Like, I don't think people realize, but like, I'll just say it out here. Like chat competitions is dying. So like any framework that was built in that era with like no conception of real time, no conception of omni-modal or multimodal native things, they will probably not age very well. Yeah.

Vasek [00:42:52]: Yeah. Probably as we like, what's the next after chat based. Yeah. Yeah. Probably as we like, what's the next after chat based. Yeah. Probably as we like, what's the next after chat based interaction with LM.

swyx [00:42:57]: Yeah. You need chat competitions and you also need reasoning with streaming and the streaming interactions of agents is also not mapped out very well, but like all agent frameworks will have to adjust to that basically.

Vasek [00:43:09]: Good question to ask when thinking all about dev tools with LM, is my dev tool more relevant as the LMs are getting smarter and as people need less prompting? I'm, I'm, for example, like really bad at prompting, but like I can get more work done over the years because the LMs are getting better. So that means like there's less need for prompt management type of a thing. Yeah.

swyx [00:43:30]: The talk from Ramp at our New York conference was basically about this, like how do you set up your, your architecture so that you benefit from 10,000 X improvements in models rather than every time you, you know, model improves, you have to kind of throw out your existing workflows. Yeah.

Vasek [00:43:47]: The question we ask very often when thinking about the features and like how to position ourselves in the ecosystem.

Alessio [00:43:55]: We have gone 51 minutes without talking about MCPs, which is tragic. Yeah. How do you do that?

swyx [00:44:02]: How do you talk about AI infrastructure and no MCP?

Alessio [00:44:04]: Right? It's kind of crazy. But since you mentioned the prompts that we just had the episode with the MCP creators and they were, I wouldn't say not frustrated, but maybe they hope more people will use the prompts and resources and MCP servers instead of just the, the tool costs. I'm curious if you've seen any fun use cases with like remote MCPs on, on E2B or. Yeah. Any other things like that?

Vasek [00:44:26]: Yeah.

swyx [00:44:27]: MCPs watching it very closely, but like still undecided, but actually not what it is, but kind of like what to do with it. There's no MCP on your docs, bro.

Vasek [00:44:37]: Uh, go to, if you go to a GitHub, we have like MCP server. Um, but yeah, we, we seen people using E2B to, uh, host MCPs. Like, but then I would say like, I don't think you need E2B for that. You can just. Probably you can run it. It's not like even like optimized for it probably. It's not like the, there are probably better ways to do that. I think people are a little bit too much focused on the protocol side of it. I even like calling it a protocol. I don't know. It's just like a server. Uh, and there's like a. A server end client. Yeah. Server end client. And there's some agreed. So I guess like in a sense it is a protocol, but like, uh, I see a lot of people like comparing it to email protocols and such, which is like, I don't know. It seems a little bit far stretch to me, uh, at this current moment, but maybe I'm missing something. Like I'm, I think it's super interesting idea. I just haven't had direct, I guess the right insight about it yet. One last thing I wanted to add, like, because we have a bunch of users that added E2B our MCP server to their like registries. Uh, so I think at least at the moment, like what's more useful is like higher order MCPs. I was talking with Henry from Smith area around it. Uh, about it, uh, that, that, you know, and he was saying exactly this, uh, telling me exactly this concept, like it's unclear who's using the MCP at the moment. Is it a developer? Is it a like end user or is it another agent? If it's a developer, then it might be like, makes sense to have like a sandbox creation of sandbox in the MCP. But if it's an end user, probably like it's too low level primitive. So he wants some kind of higher order MCP. Like instead of us offering like a sandbox. We would be offering like a way to build a cogen agent with MCP or something like that. Or run a cogen agent that's using E2B MCP like in the background. So I have more like a, like these type of unanswered questions in my head about MCPs.

Alessio [00:46:39]: I mean, they are all running locally right now. Mostly. I don't think there's many remote MCP servers yet. I know that people are pushing for it.

Vasek [00:46:47]: It looks like there's more remotely actually. I don't know. That's what I heard from people like managing these registries of MCPs.

swyx [00:46:55]: Well, they're incentivized to tell you that. Yeah.

swyx [00:46:59]: I fully agree with that confusion about what to do. I do think that every dev tools company needs some kind of MCP strategy for better or worse. It's annoying as it is. I think it does start with having an API though, rather than like a SDK first experience. Because then you, people can just wrap in whatever. Language that they want. And then you also, for you particularly, you might want to have a distinction in your strategy for MCP clients versus MCP servers. Because those might be different things. And particularly, I liked what you said about the higher order MCP for the end user who doesn't really care about implementation detail. I think that is fantastic for E2B where like the agents can just spin up an E2B instance in the background and they don't even know about it. Yeah.

Vasek [00:47:48]: In the ideal world, we have like mcp.e2b.dev. First, it needs to have figured out authentication. So the agent should just ask to have, I guess, an account created for it or whatever it is it's going to be. And then it can just like launch a sandbox, do the execution there, and you can come back to it a month later and the state is still there. So that I think makes a ton of sense. Then you can start building higher order things on top of that, which could be very interesting. Yeah. I like what you said about having API first versus SDK first approach. I think this is very, very important for the LLMs. And that's how we are building our whole new dashboard, the infrastructure. So everything needs to be API controllable from like public APIs for users because eventually the LLMs will want to control and gather this data. Okay. So that's a big shift for E2B because you're SDK first. It has been like underneath API base all the time. But now we will go more into it. And it was more like in terms of prioritization. So we needed to start with humans to get to LLMs and first build for human developer and then now building for like a LLM developer first. Yeah.

swyx [00:49:03]: I'll just call out that since we did the MCP episode, they announced their update to the spec that they added an off component to the spec itself. It seems to be just based on OAuth 2.1. And I assume like, you know, that's the first easy step. That's the easiest thing to do. But there's no effective distinction between an agent and an user. We never had that.

Vasek [00:49:22]: I think this is more like a broader question of you have like all the websites that has optimized everything for humans. But now you will have like agents visiting those websites. Like what are the incentives there? You know, like probably you as a website owner, you want to know it's an agent because you spend so much time and money optimizing everything for humans. So I think like the dynamics on the internet might get really weird if you don't have a distinction. This is a human, this is an agent. Yeah.

Alessio [00:49:52]: We tried to record an episode with Matthew Prince, CEO of Cloudflare yesterday. And then we had technical difficulties, but they're building a lot of that.

swyx [00:49:58]: You can see the stats. Yeah.

Alessio [00:50:00]: They mentioned, for example, used to be, you know, Google will be a two to one crawl to referral ratio. So for every two pages they will read, they will send you one visitor. Yeah. He said OpenAI is 250 to one. So they'll read 250 of your pages and send you one visitor. And then Throbic was like 6,000 to one. Wow. So they'll read 6,000 pages before they send one person back to your website. So obviously we have Jeremy Howard who's been working on LLMs.txt to kind of have a separate interface for that. I think today people don't really curate the LLM experience. I think a lot of the LLM TXT that people are making is like automated, take a website and turn it into LLM.txt. But like, that's not really what you want to do. It's like, how do you separate completely the two things? You know, like if you're an e-commerce store, the LLM TXT should have your own inventory and like one thing, you know, shouldn't have a search button.

swyx [00:50:51]: I feel like obviously LLMs.txt is a good movement that improves the legibility of these doc sites for LLMs. But I feel like it's kind of like a halfway measure. Like, I think we've failed with agents if we are reshaping the human environment for agents. Like there must be two of everything. What's your human side and what's your agent side? Like, no, like agents should just be the human side. Yeah. Why are we making any special dispensation for these things?

Alessio [00:51:17]: Yeah, I think the monetization is the only thing. Like too many websites are like ads driven, kind of like... You need to look at it, right? Yeah, I think you need to change. Once we figure that out, you know, I think you can use the same interface. But I think today just like all these pop-ups, it's like, fuck man, stop popping. Okay, so Bitcoin solves this. Yeah.

swyx [00:51:34]: I'm just kidding. But anyway, yeah. I don't know if you have a take on all this.

Vasek [00:51:39]: I have like a general rule that there's a... Usually when something new comes out, people have this tendency to recreate the thing that already exists for the new thing. Like that's like the internet for agents versus you already have all the infrastructure for the old thing and probably might be easier to teach the new thing to use the old thing. I think it's very common coming from developers that it will be a perfect world when you have nice, clear distinction between these two. But actually the world is super messy. So nothing comes to my mind that they would like end up that... You have clear distinction between like two type of sort of entities or we created like a new internet for just mobile phones. You actually have both desktop website version and mobile phone version, right? It's much more always complicated and messy in the real world than having a nice, clear distinction, which is something I think developers strive for just from working with code because you want this nice, clear distinction. But humans are more complicated than that. So I think like... Everything ends up being sort of mixed and you need to adapt to it. For sure. Yeah.

swyx [00:52:48]: I think we'll do this for 20 years and then we'll figure out the... Yeah.

Vasek [00:52:51]: And probably it will be somewhere in between that you have like a world where agents are using human internet, but the internet changed because you have agents or something like that.

swyx [00:53:00]: This reminds me of some conversation that I think... I don't know who was making this analogy. Like in the mobile era, we had m.yourdomain.com and then we had www.yourdomain.com. And now we might have like just llm.yourdomain.com. And that's just the LLM experience. mcp. Oh, yeah. Way better. Yeah. Consumes everything. Cool. We're just going to go through all the rest of the use cases. We can refer people to your website. But I always want people to have a good mental map of when they should go to E2B and what other people are using E2B so that they don't miss out, right? So it's AI data analysis, data visualization, coding agents, generative UI, code gen evals, and computer use. Do you think that's like the sequence of most popular data analysis, least popular is computer use?

Vasek [00:53:47]: Most experimental is computer use, I would say. It still gets a lot of traction when you share the demos of it. It's just Manus.

swyx [00:53:56]: Who else is doing this? I basically don't see anyone else.

Vasek [00:53:59]: When you say computer use, I imagine like a graphical interface. Manus, as far as I know, isn't running that. So what do you also call a kind of computer use but without graphical interface? Because you are using the full computer, but more from like code point of view. But that's a different discussion. I would say like computer use is very exciting, but experimental from what I've seen. Yeah, okay. I think for real computer use, you really want to support more platforms than just Linux.

swyx [00:54:29]: Yeah, Windows, right?

Vasek [00:54:31]: And then it gets, it might be more licensing battle than technical battle.

swyx [00:54:37]: Yeah, lawyers always win. Yeah. We'll probably have Erik for the next question. We'll probably have Erik from PIG at some point talk about his movements on Windows. So that was going to go into evals, right? And also how that links to RFT. Yeah, just like, can you tell more stories about the Open R1 projects, how you work with them and any other like academics that are working with E2B that you think could be possible for like the researcher model training use case or fine-tuning use case?

Vasek [00:55:03]: Yeah, the way Hugging Face will build the Open R1 project is using us is during like the reinforcement learn, CodeGen reinforcement learning step where the R1 model, the Open R1 model has a training step where they give it a code problem and the model needs to generate and run code somewhere. Then you have reward function basically giving you zero or one telling you if that was a good solution or bad solution, then you improve the model. So kind of like the feedback loop. They are using the E2B sandboxes to run many hundreds of these sandboxes, thousands of these sandboxes per training step. So they can achieve like big parallelization. We started very fast. You don't need to use your GPU cluster for that, which is like very expensive to use for this type of workloads. And also, and that goes with the story that I mentioned in the beginning is that you don't need to worry about like the LLM actually like changing permissions in your cluster and then you can access the cluster because everything is isolated and secure from each other. So I think that's very, very interesting use case because we've had a few other companies like reaching out and started using us this way, building models, foundation models. It's like when we started E2B, that wasn't the use case that we had in mind. So, but it's like makes total sense if you also, if you think about like life cycle of an AI agent. Like if you, it makes a lot of sense for us to be like from the earliest stage possible and the earliest stage is probably model training. So this fits very, very, very nicely in that, in the use case. We actually released a case study with Hugging Face that's on our website that people can read.

Alessio [00:56:53]: Have you seen people also use that to evaluate agents they want to use or is it mostly people doing training?

swyx [00:56:58]: We've seen people using E2B for evals. Yeah, that's what I'm thinking. Like it should be very easy to run like suites. We've seen people using E2B bench and all these on E2B. Yeah.

Vasek [00:57:07]: This is more foreshadowing, but we will be launching in the next couple of months, like a startup and research program for people and universities and researchers like doing exactly these type of things. Different use case, but we, for example, work with LM Arena folks from Berkeley that are using us to compare models in AI app generation and we run the AI generated app. He wrote that in the equation, right? Yes. I did. I did. I think that's only possible thanks to Alessio. And I'm not joking because first he connected it and then he actually wrote it.

swyx [00:57:41]: Man, the things you have to do to win deals these days. Full stack value add.

Vasek [00:57:46]: He was already our investor. So I think that also shows it even in a better light because you clearly see the person is not interested just to win the deal, but actually, you know. To increase the value of the investment. Exactly. Like how many VCs are actually fixing your bug in your code base?

Alessio [00:58:03]: Let's make a YouTube short of this part so I can share it. You should put it on our landing page. Yeah, exactly.

Alessio [00:58:12]: Awesome, man. So let's talk about, since you mentioned VCs, a lot of VCs that passed on you before because you only do code execution, it's kind of like a small market. I would love for you to maybe also paint the picture, you know. So you just mentioned it's cheaper than the GPU cluster than Hugging Face has. Do you want to do GPUs in the future? You mentioned Railway. That was easy to do. Do you want to compete with Railway down the line? Like where do you see E2P going?

Vasek [00:58:37]: So GPU question is an interesting one. The GPU market is hard. Like you are competing on the compute internal, I think it's hard because eventually you will have competitors and everyone will be like pricing it a little lower and no one is making any margin. So that's like, but it just opens new use cases for you. And for example, even with the data analysis, if you just run, I think it's Pandas code from, there's a recent update, like recent update. If you just run it on GPU, it's like twice as much faster. If we want to like do really big AI data analysis, you need that. So I think like GPUs for us, and also if you want to like have LLM train small machine learning models, maybe you want to like build full games, you really want to offer the full cloud computer, but in cloud, that's very elastic. So I think GPUs are there on the roadmap. I wouldn't say it's like the thing that we are immediately working on, but eventually it makes like a lot of sense for us to offer this. And so what was the other question? On the, do you want to host the apps that you are building too? We are very well positioned that the LLM is doing all the development work with us. Then you need to deploy it somewhere. So it's like very next natural next step. Eventually, like we want the LLMs to work. We want the LLMs to deploy these services, apps that are building and have them manage it. And developer is more like in the backseat, like looking at things, if everything is working correctly. If your swarm of agents is working correctly. It also requires a slightly different infrastructure for deploying. But I think there's a big advantage in knowing what developers are building on our platform. And then, because then you can see like what's the ideal, even from technical point of view, you have all the insights that you need to actually affect your business. So you can effectively deploy it and kind of then cover the full lifecycle of building the app. But now it's not built by the human developer. It's built by the AI. So TLDR, yes, probably down the road somewhere. We want to build essentially like the new AWS, but for LLMs.

Swyx [01:00:45]: So we're just going to move out and just to wrap up. One of the interesting things that I saw you do was you were originally check-based. And I'll be very honest. When I first invested in the early round that you did, I was like, these guys know how to recruit in Czech Republic. It's like a competitive advantage. And then the next thing I know, you're moving to SF with your You show up to our office.

Vasek [01:01:15]: That's hilarious. Actually, I think you offered even your place to live for.

swyx [01:01:23]: So why move to SF? Do you think that everybody in your kind of similar situation should? Any pros and cons that you have?

Vasek [01:01:31]: I think it's you can definitely build a DevTool company from Europe. I think it's a lot about question of how easy you want it to be in the earlier days. Especially if you are building like a, you know, kind of like red ocean versus blue ocean waters. If you are building in a field that already exists and you have like large competitors, you're building something that's 10 times, 100 times better. It's probably, you probably don't need to be NSF. Eventually, you probably will need some kind of US base because for sales and customers. But you can very well build this from Europe. I know great companies doing that because all it's the knowledge is already among all the developers. But I would maybe argue that it might be harder to find people that are comfortable with fast iteration loop and changing things early on. You know, almost we are pivoting every week, every month. But the main motivation for us was we just wanted to be very close to our users. And it was clear after a few weeks that SF is becoming this AI hub. And what we used to do and we still do it sometimes, but slightly less because of like not having that much time and resources. But we just met with the customers that had problems, our users. And we just like implemented it to be for them, like next to them. Like we made a PR.

swyx [01:03:02]: I call this the Collison installation.

Vasek [01:03:05]: By the way, I don't know if this is like a known thing, but do you know how many times they did that? 200? Twice, three times. Oh, lol. I asked about like three years ago when I had a chance to ask a question from Patrick Collison. Like how many times you did the Collison installation? And he was like three times. But then after that, you probably don't want to do that because you want to automate things and focus on other stuff. Nice. But it's exactly like do the things that don't scale. Three times. Are you kidding? Exactly three.

Vasek [01:03:40]: But my whole point was that how many such users I can meet in Prague in a week versus in San Francisco in a week. In Prague, it's going to be probably one. And then I can't meet anyone else for the next half of the year because I just don't have users in Prague. But all of my users, our users were here. So we could just keep repeating doing that again and again. I would even argue we did it too much. We could have automated slightly faster. But it's very useful feedback that you can get. And you can then start moving much, much, much faster. So that was important. And I think also eventually you are in like you are in the B2B business even from the early days. And it's good to have good relationship with people. And just like meeting in person is just better than meeting over Zoom. Well, I mean, that's why we do this in person.

swyx [01:04:32]: But yeah. I mean. You know, obviously I run a conference. I'm very sympathetic to people meeting in person. Right. But I also want there to be some hope for people who are who are never going to come to SF that like they can still get involved. That's partially why we do this podcast is to get them involved in the community.

Vasek [01:04:48]: And I mean, we started a new office in Prague. Yes. You're hiring again in Prague. Yeah. And I think that there's really good talent in Europe, in Czech Republic, for example. I don't know. I can't speak for other countries, but I can imagine it's very similar. Once you have like a clear idea of what your product looks like, then you can find really good expert on certain part of your infrastructure on database and things like that. And just like have a top talent, get top talent for that. The reason we didn't want to do it early on was because we kind of didn't know ourselves what we were building. And we had to figure it out in person with users here. But now we feel much more strong about like knowing the roadmap for the company and for a product. And so it's much easier to hire people that have our difference from us and explain them what they are building even remotely. Sometimes if you are not there and you're just communicating through Slack.

Alessio [01:05:49]: Just to wrap, what are the roles that you're hiring for?

Vasek [01:05:52]: So we are hiring distributed systems engineers. We are hiring platform engineers, AI engineers. We are also hiring account manager and customer success engineer. So kind of all over the place, we see a lot of market pool and momentum being built up. So we want to double down and move even faster because we see all the potential of what we can do. And just want to kind of like you want to pour the gas on the fire on the spark. And that's how I feel where we are now at E2B.

Alessio [01:06:25]: Awesome, man. Thank you so much for coming on. Yeah.

Vasek [01:06:27]: Thank you for having me. It's great.

swyx here! We’re delighted to bring you another guest poster, this time talking about something we’ve been struggling to find someone good to articulate: TDD for AI. This is part of a broader discussion we’re seeing as teams of AI Engineers want templates and models for how the SDLC adapts in the age of AI (e.g. Sierra’s ADLC)

Anita’s talk came out of the online track for the NYC Summit, and it’s warm reception gives us enough confidence to re-share it with the broader LS community.

AIEWF talk selection is very selective - we have over 500 talks submitted with the deadline closing soon - but we have just gotten resources for the online track again for AIEWF - send in your talk and we’ll fit good ideas in where-ever we have room!

from AI Engineer Summit’s Online Track:

Based on the success of the online talk, we asked Anita to convert it into a blogpost for easier scanning/discussion. What follows is Anita’s updated, short blog version of the main takeaways, but it’s definitely handy to have the full talk to doubleclick into each stage and get more context!

Historically with Test Driven Development (TDD), the thing that you're testing is predictable. You expect the same outputs given a known set of inputs.

With AI agents, it's not that simple. Outcomes vary, so tests need flexibility.

Instead of exact answers, you're evaluating behaviors, reasoning, and decision-making (e.g., tool selection). This requires nuanced success criteria like scores, ratings, and user satisfaction, not just pass/fail tests.

And internal evals aren’t enough. You need to make continuous adjustments based on real-world feedback.

So how can you build a process around this?

At Vellum, we’ve worked with hundreds of companies refining this process. If you're after a "self-improving" magic agent, this isn’t the article for you.1 We believe in observability and debugging, because that’s what actually works.

What follows is a breakdown of the five stages you can follow to enable TDD-based AI development for your organization and ship reliable agentic applications.

First you must start by clearly defining the problem you're solving with AI.

Not every issue is an AI problem. Sometimes a simpler solution using traditional software works better. Map out the user journey clearly from start to finish. Identify exactly where AI adds value, how users will interact with the AI-generated outputs, and isolate any areas where friction might occur.

For example, if you’re developing a customer support chatbot, specify clearly which types of customer issues (intents) your AI will handle, and which you'll escalate to humans.

Second, your AI will only be as good as the context (data) you provide. Understand exactly what data you have, its quality, and how you’ll access it (MCP or otherwise). Think about how fresh it needs to be or how often you'll need updates. And don't forget privacy!

Finally, plan for future readiness by answering questions like:

• Workflow-Agent tradeoffs: If I want to add agentic capabilities later, will it be easy?

• Reusability: Is this system modular enough to easily build upon or reuse components?

• Areas of Technical Risk: What components are likely to change and how will I test for regressions when I change them?

Once you have your process outlined, it’s time to validate whether your idea can be built with AI.

Unlike traditional software engineering, LLMs are non-deterministic, unpredictable, and getting your AI working right requires a lot of trial and error.

The second stage of the TDD process requires you to first validate that AI works in the context of your use-case.

You start by building an MVP. The MVP can be as simple as a prompt that you want to test with different models. You follow some prompting structure, test different techniques and evaluate at a small scale whether these models can work with your use-case.

For example, if your system should extract data from a PDF, you might test Gemini 2.0 Flash Thinking mode or Mistral OCR, and do a quick comparison on how the results compare. At this stage you should optimize for rapid iteration cycles to narrow in on the best implementation quickly.

An MVP might also look like an orchestrated workflow, doing several things in sequence or looping through multiple steps. Think of using RAG, pulling details from longer documents, calling tools (functions), or keeping track of things with short- or long-term memory.

If your MVP relies on all these pieces working together, you'll want to test it out with a few examples early on, just to confirm that the whole setup actually makes sense for your use case.

But for production-ready solutions, chances are that you’ll need to evaluate hundreds or thousands of these tests.

That brings me to the next stage.

At this stage, you should work with product and domain experts who will know how to tell whether your AI system is "doing a good job."

Say you're building a chatbot that suggests real estate properties based on user preferences:

• You'd need to work with the product team to figure out things like what questions the chatbot should ask, how it filters results, and how users interact with it.

• At the same time, you'd check with the legal team about what regulations you may need to stay in compliance with, like fair housing laws, privacy rules, and what you can or can't say when suggesting properties.

Using this knowledge, you’ll be creating a database with all these examples and the correct responses. Then you’ll run automated tests to see which of these test will pass/fail.

At this point, you probably know that with AI solutions you’ll end up evaluating more things than just “correctness”. Some examples include checking for correctly retrieved context from a vector database, JSON validity, semantic similarities and more.

Then you keep repeating this loop: create tests → run evals → make small change to prompts/logic to fix for that test → check for regressions.

Notice the last part: Check for regressions.

Fixing a prompt for one test case can easily introduce regressions to other test cases. You need to have a mechanism that will be able to check for these regressions and flag them for review.

Now, let’s say that through evals, you’ve built the confidence you need to put your AI system in front of customers in production. What's next?

As you've probably noticed, AI development needs a lot of trial and error, and small tweaks can break the system.

This means that when it comes time to integrate your AI system with your application layer, you'll get massive benefit from two concepts:

• Decoupled deployments

• Release management

If you decouple the deployment lifecycle of your AI systems from that of your application layer, you'll be able to ship changes to your AI features quickly and roll back easily if things go wrong.

You won't be bound to the release cycle of the rest of your product.

Second, and just as important, is having solid release management. Good release management means version-controlling the system as a whole and allowing for different environments point to different versions. For example, your application layer's staging environment might point to a newer, less stable release of your AI system, whereas your production environment might point to an older, more tried-and-true version.

This separation also allows you to run the same evaluations every time you push an update, so you can clearly see what's improving, and what's getting worse. You should be able to easily answer questions like:

• Regressions: Did the new update cause more mistakes or unexpected behavior?

• A/B testing: Can we easily compare performance with previous versions?

• Rollbacks: If something goes wrong, can you quickly switch back to a safe, earlier version?

Now, lets say your AI solution is in production and is being used by your customers. Amazing!

However, even now, it's likely your AI feature will come across some user input you could never have expected and produce an output that leaves users dissatisfied. The best AI teams capture this data, flag edge-cases, and create a feedback loop such that they can run new experiments to account for them.

This brings us to observability.

This is the final stage in this list, but in many ways it feels like stage 0.

This is where you need to capture all edge cases that show up in production, and successfully update your AI solution to account for them without regressing in other ways.

For example, you can capture implicit or explicit feedback from your customers to learn if a given AI response or workflow completion was useful for your end user.

Two things are very useful here:

• Graph or Tracing view (looking at all executions, API invocations, received data)

• Feedback loops

The more complex your system gets, the more you'll need observability.

The more agentic your system becomes, the more you'll rely on observability.

Use a graph view to track important details—things like unique executions, API calls, parameters you're sending, and data you're getting back—at every step of your workflow. Drill down into each and every execution, and easily spot issues.

It’s like a continuous loop honestly — your work is never done.

It will get more stable over time, but in order to get full control of your system you need to tame it first.

Once deployed, the TDD looks something like this: capture feedback loops → evaluate → improve → regression check → deploy.

Vellum’s solutions are designed around this TDD vector.

We’ve worked closely with hundreds of companies who have successfully put agentic TDD into practice — and we’ve designed our platform around those best practices.

Our platform gives you the flexibility to define and evaluate agent behaviors, capture real-world feedback, and continuously improve your AI—all in one place.

If you want to see a demo, happy to do one — book a call with us here.

Ironically, Augment Code’s talk is exactly about this!

Quick reminder: AI Engineer CFPs close soon! Take a look at “undervalued tracks” like Computer Use, Voice, and Reasoning, and apply via our CFP MCP (talks OR workshops, we’ll figure it out).

Relevant to today’s quick post we do have an Agent Reliability track. Also: take the 2025 State of AI Engineering Survey!

The AI attention economy has enabled a hypeboi priesthood who exist in a state of perpetual performative orgasmic nirvana, minds continually blown as every launch Changes Everything, vibing at gigahertz oscillations of “it’s so over” vs “so back”. OpenAI’s “Practical Guide to Building Agents” is the latest such earth shatterer:

This guide, however, has been less well received than Anthropic’s equivalent.

If you watch his multiple appearances with us, Harrison Chase is not someone who is quick to “anger”, so calling this guide “misguided” and doing a word by word teardown can seem like fighting words for him1.

At the heart of the battle is a core tension we’ve discussed several times on the pod - team “Big Model take the wheel” vs team “nooooo we need to write code” (what used to be called chains, now it seems the term “workflows” has won).

You should read Harrison’s full rebuttal for the argument, but minus the LangGraph specific parts, the argument that stood out best to me was that you can replace every LLM call in a workflow with an agent and still have an agentic system:

And the ideal agent framework lets you start from one side of the spectrum and move to the other, optimizing for making code easy to change:

You’ll find this necessary because sometimes you DO want to reverse decisions from having too many agents - as fellow speaker Augment Code found in their #1 SWE-Bench entry:

To be clear it’s easy to understand where the Big Model folks are coming from: if you work with Big Lab enough, you’ve seen hundreds of engineer-hours of hand tuned workflows obliterated overnight with the next big model update — the AI Engineer equivalent of learning the Bitter Lesson again and again. This is why “AI engineering with the Bitter Lesson in mind” was such a resonant topic at the Summit (now at 124k views across platforms):

Specifically, I think the success of both OpenAI and Gemini’s Deep Research this year primarily leveraging O3 to reason through research planning and execution, and later Bolt and Manus AI doing the same with Claude, with very little workflow engineering, has demonstrated that there’s a lot to be said for building general purpose agents that simply augment models without the “inductive bias” constraints of workflows. O-team researcher Hyung Won Chung noted that adding more structure gets you wins in the short term, but that structure tends to lose in performance as the model (pretrain or inference compute) keeps scaling up.

If your goal is to build AGI, to build a horizontal platform, particularly one targeting non-technical consumers who are confused by even having a model selector, then it’s an understandable position to take, and (even encourage, for the purposes of dataset/human feedback collection).

Ultimately the reason I argue Harrison isn’t -actually- taking a fighting stance is he leaves room for the spectrum to exist and makes a remarkably (for someone with obvious skin in the game) balanced argument that you’re going to just want options for doing both:

I find this hard to debate - if meaningful conversation is to be had, it really is more about where the current state of this Pareto frontier really is today (I’m not sure it is convex yet) and how to move it out.

What -is- true is that there is such a thing as bad ideas to avoid in creating workflows that will DEFINITELY get steamrolled, and also the converse - workflow systems that maintain value as their underlying models get upgrades - as we saw last year with AlphaCodium’s initial release in Jan and then its value persisting “out of distribution” in Nov after the release of o1 - as we discuss on our pod covering Flow Engineering.

The other pretty cool thing that Harrison did in his piece was publish a full comparison table of all relevant Agent Frameworks today, although of course even he couldn’t escape the McCormick trap. It’s useful to test our descriptive metaframework of everybody’s Agent definitions against a new out-of-distribution Agents definition:

I think this is a remarkably fair shopping list of abstractions and features for the discerning Agent Engineer — it also articulates why you feel certain gaps when an Agent Framework promises you the world and yet you can’t do some things easily.

To callback to our intro, if your mind is continually blown, it can never be made up. I think that helping people make up their minds is a valuable service to the community.

If you like learning from this kind of debate, we’re doubling down on the success of the Dylan v Frankle showdown from last year’s NeurIPS, and also accepting submissions for what we’re calling “The Great Debates” - good faith debaters from two sides of a relevant industry debate. Everybody wins, but the people who are best able to change minds win the most. Apply in pairs!

As I’ll argue: they’re actually not! Harrison is ever the diplomat.

Thanks to your support, the 2024 World’s Fair was a big success, and AIE Summit in NYC has surprisingly seen ~3x more in online impact than that despite the selectively small in-person numbers, with almost 80k people tuning in to our *livestream* alone.

Spots on the AIE Expo are selling out really quickly and we’re now looking at about 3,000 AI Engineers joining us in person this June (with up to 100x more online).

We’ve already had hundreds of people apply to speak, but as the curation team, we want to seek out the best AI Engineers in the world (no prior speaking exp required, no famous corp required, just good ideas, demos, and real life experience that might be useful to your fellow AIEs) and ensure a sufficient diversity of thought and content.

PLEASE ALSO SEND THIS TO YOUR FRIENDS TO ENCOURAGE THEM TO APPLY!

Lots of people have seen our Call for Speakers (aka CFP) via the AIE newsletter, but many have either been putting it off and/or missed the announcement, so we’re making a dedicated call for speakers here. You can read the CFP for our prompts on the things we are seeking, but submit the best talk ideas you have anyway and we’ll find a stage for it if the team likes it enough - e.g. our online track on YouTube. Speakers get free tix + flights + accom (for out-of-towners - with a sponsorship in lieu if company covers!) All nonaccepted, nontrivial submissions get discounted tickets as a show of appreciation.

Some tracks have been unexpectedly competitive, and others uncompetitive. You should take note accordingly when shaping a talk/SHARING IT WITH A FRIEND :)

We of course have an llms.txt, but, as the first conference with a dedicated MCP track, we’ll not only have Anthropic’s MCP team presenting a State of the Union, but also had to create our own open source MCP server! Try it now!

Add this server: https://ai-engineer-wf-2025.swyx-5de.workers.dev/sse

And then chat with our conference and submit your talk proposal right from within Cursor, Windsurf, Claude Code, or your MCP client of choice :)

Important note: submissions here will all be considered for acceptance, but speakers with accepted talks from the MCP process will need to fill out Sessionize anyway to complete their acceptance when we notify you via email. Sorry, we know this is suboptimal, but Sessionize doesn’t have a write API and we need a single source of truth.

ok enough yapping. Please submit your talks by this weekend, and tell your best/shy AI engineer friends!

Barr from Amplify and the massively popular Frontier Feud is running the 2025 State of AI Engineering Survey!

Take the survey for the Amazon gift card raffle and to have your voice heard in AIE!

We’ll keep this brief because we’re on a tight turnaround: GPT 4.1, previously known as the Quasar and Optimus models, is now live as the natural update for 4o/4o-mini (and the research preview of GPT 4.5). Though it is a general purpose model family, the headline features are:

• Coding abilities (o1-level SWEBench and SWELancer, but ok Aider)

• Instruction Following (with a very notable prompting guide)

• Long Context up to 1m tokens (with new MRCR and Graphwalk benchmarks)

• Vision (simply o1 level)

• Cheaper Pricing (cheaper than 4o, greatly improved prompt caching savings)

We caught up with returning guest Michelle Pokrass and Josh McGrath to get more detail on each!

We are calling for the world’s best AI Engineer talks for AI Architects, /r/localLlama, Model Context Protocol (MCP), GraphRAG, AI in Action, Evals, Agent Reliability, Reasoning and RL, Retrieval/Search/RecSys , Security, Infrastructure, Generative Media, AI Design & Novel AI UX, AI Product Management, Autonomy, Robotics, and Embodied Agents, Computer-Using Agents (CUA), SWE Agents, Vibe Coding, Voice, Sales/Support Agents at AIEWF 2025! Fill out the 2025 State of AI Eng survey for $250 in Amazon cards and see you from Jun 3-5 in SF!

Coreweave’s now-successful IPO has led to a lot of questions about the GPU Neocloud market, which Dylan Patel has written extensively about on SemiAnalysis. Understanding markets requires an interesting mix of technical and financial expertise, so this will be a different kind of episode than our usual LS domain.

When we first published $2 H100s: How the GPU Rental Bubble Burst, we got 2 kinds of reactions on Hacker News:

1. “Ah, now the AI bubble is imploding!”

2. “Duh, this is how it works in every GPU cycle, are you new here?”

We don’t think either reaction is quite right. Specifically, it is not normal for the prices of one of the world’s most important resources right now to swing from $1 to $8 per hour based on drastically inelastic demand AND supply curves - from 3 year lock-in contracts to stupendously competitive over-ordering dynamics for NVIDIA allocations — especially with increasing baseline compute needed for even the simplest academic ML research and for new AI startups getting off the ground.

We’re fortunate today to have Evan Conrad, CEO of SFCompute, one of the most exciting GPU marketplace startups, talk us through his theory of the economics of GPU markets, and why he thinks CoreWeave and Modal are well positioned, but Digital Ocean and Together are not.

However, more broadly, the entire point of SFC is creating liquidity between GPU owners and consumers and making it broadly tradable, even programmable:

As we explore, these are the primitives that you can then use to create your own, high quality, custom GPU availability for your time and money budget, similar to how Amazon Spot Instances automated the selective buying of unused compute.

The ultimate end state of where all this is going is GPU that trade like other perishable, staple commodities of the world - oil, soybeans, milk. Because the contracts and markets are so well established, the price swings also are not nearly as drastic, and people can also start hedging and managing the risk of one of the biggest costs of their business, just like we have risk-managed commodities risks of all other sorts for centuries. As a former derivatives trader, you can bet that swyx doubleclicked on that…

• SF Compute

  • Evan Conrad

  • Ethan Anderson

  • John Phamous

  • The Curve talk

• CoreWeave

• Andromeda Cluster

Like and subscribe!

• [00:00:05] Introductions

• [00:00:12] Introduction of guest Evan Conrad from SF Compute

• [00:00:12] CoreWeave Business Model Discussion

• [00:05:37] CoreWeave as a Real Estate Business

• [00:08:59] Interest Rate Risk and GPU Market Strategy Framework

• [00:16:33] Why Together and DigitalOcean will lose money on their clusters

• [00:20:37] SF Compute's AI Lab Origins

• [00:25:49] Utilization Rates and Benefits of SF Compute Market Model

• [00:30:00] H100 GPU Glut, Supply Chain Issues, and Future Demand Forecast

• [00:34:00] P2P GPU networks

• [00:36:50] Customer stories

• [00:38:23] VC-Provided GPU Clusters and Credit Risk Arbitrage

• [00:41:58] Market Pricing Dynamics and Preemptible GPU Pricing Model

• [00:48:00] Future Plans for Financialization?

• [00:52:59] Cluster auditing and quality control

• [00:58:00] Futures Contracts for GPUs

• [01:01:20] Branding and Aesthetic Choices Behind SF Compute

• [01:06:30] Lessons from Previous Startups

• [01:09:07] Hiring at SF Compute

Alessio [00:00:05]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host Swyx, founder of Smol AI.

Swyx [00:00:12]: Hey, and today we're so excited to be finally in the studio with Evan Conrad from SF Compute. Welcome. I've been fortunate enough to be your friend before you were famous, and also we've hung out at various social things. So it's really cool to see that SF Compute is coming into its own thing, and it's a significant presence, at least in the San Francisco community, which of course, it's in the name, so you couldn't help but be.

Evan: Indeed, indeed. I think we have a long way to go, but yeah, thanks.

Swyx: Of course, yeah. One way I was thinking about kicking on this conversation is we will likely release this right after CoreWeave IPO. And I was watching, I was looking, doing some research on you. You did a talk at The Curve. I think I may have been viewer number 70. It was a great talk. More people should go see it, Evan Conrad at The Curve. But we have like three orders of magnitude more people. And I just wanted to, to highlight, like, what is your analysis of what CoreWeave did that went so right for them?

Evan: Sell locked-in long-term contracts and don't really do much short-term at all. I think like a lot of people had this assumption that GPUs would work a lot like CPUs and the like standard business model of any sort of CPU cloud is you buy commodity hardware, then you lay on services that are mostly software, and that gives you high margins and pretty much all your value comes from those services. Not really the underlying. Compute in any capacity and because it's commodity hardware and it's not actually that expensive, most of that can be sort of on-demand compute. And while you do want locked-in contracts for folks, it's mostly just a sort of de-risk situation. It helps you plan revenue because you don't know if people are going to scale up or down. But fundamentally, people are like buying hourly and that's how your business is structured and you make 50 percent margins or higher. This like doesn't really work in GPUs. And the reason why it doesn't work is because you end up with like super price sensitive customers. And that isn't because necessarily it's just way more expensive, though that's totally the case. So in a CPU cloud, you might have like, you know, let's say if you had a million dollars of hardware in GPUs, you have a billion dollars of hardware. And so your customers are buying at much higher volumes than you otherwise expect. And it's also smaller customers who are buying at higher amounts of volume. So relative to what they're spending in general. But in GPUs in particular, your customer cares about the scaling law. So if you take like Gusto, for example, or Rippling or an HR service like this, when they're buying from an AWS or a GCP, they're buying CPUs and they're running web servers, those web servers, they kind of buy up to the capacity that they need, they buy enough, like CPUs, and then they don't buy any more, like, they don't buy any more at all. Yeah, you have a chart that goes like this and then flat. Correct. And it's like a complete flat. It's not even like an incremental tiny amount. It's not like you could just like turn on some more nodes. Yeah. And then suddenly, you know, they would make an incremental amount of money more, like Gusto isn't going to make like, you know, 5% more money, they're gonna make zero, like literally zero money from every incremental GPU or CPU after a certain point. This is not the case for anyone who is training models. And it's not the case for anyone who's doing test time inference or like inference that has scales at test time. Because like you, your scaling laws mean that you may have some diminishing returns, but there's always returns. Adding GPUs always means your model does actually get. And that actually does translate into revenue for you. And then for test time inference, you actually can just like run the inference longer and get a better performance. Or maybe you can run more customers faster and then charge for that. It actually does translate into revenue. Every incremental GPU translates to revenue. And what that means from the customer's perspective is you've got like a flat budget and you're trying to max the amount of GPUs you have for that budget. And it's very distinctly different than like where Augusto or Rippling might think, where they think, oh, we need this amount of CPUs. How do we, you know, reduce that? How do we reduce our amount of money that we're spending on this to get the same amount of CPUs? What that translates to is customers who are spending in really high volume, but also customers who are super price sensitive, who don't give a shit. Can I swear on this? Can I swear? Yeah. Who don't give a shit at all about your software. Because a 10% difference in a billion dollars of hardware is like $100 million of value for you. So if you have a 10% margin increase because you have great software, on your billion, the customers are that price sensitive. They will immediately switch off if they can. Because why wouldn't you? You would just take that $100 million. You'd spend $50 million on hiring a software engineering team to replicate anything that you possibly did. So that means that the best way to make money in GPUs was to do basically exactly what CoreWeave did, which is go out and sign only long-term contracts, pretty much ignore the bottom end of the market completely, and then maximize your long-term contracts. With customers who don't have credit risk, who won't sue you, or are unlikely to sue you for frivolous reasons. And then because they don't have credit risk and they won't sue you for frivolous reasons, you can go back to your lender and you can say, look, this is a really low risk situation for us to do. You should give me prime, prime interest rate. You should give me the lowest cost of capital you possibly can. And when you do that, you just make tons of money. The problem that I think lots of people are going to talk about with CoreWeave is it doesn't really look like a cloud platform. It doesn't really look like a cloud provider financially. It also doesn't really look like a software company financially.

Swyx [00:05:37]: It's a bank.

Evan [00:05:38]: It's a bank. It's a real estate company. And it's very hard to not be that. The problem of that that people have tricked themselves into is thinking that CoreWeave is a bad business. I don't think CoreWeave is explicitly a bad business. There's a bunch of people, there's kind of like two versions of the CoreWeave take at the moment. There's, oh my God, CoreWeave, amazing. CoreWeave is this great new cloud provider competitive with the hyperscalers. And to some extent, this is true from a structural perspective. Like, they are indeed a real sort of thing against the cloud providers in this particular category. And the other take is, oh my gosh, CoreWeave is this horrible business and so on and blah, blah, blah. And I think it's just like a set of perception or perspective. If you think CoreWeave's business is supposed to look like the traditional cloud providers, you're going to be really upset to learn that GPUs don't look like that at all. And in fact, for the hyperscalers, it doesn't look like this either. My intuition is that the hyperscalers are probably going to lose a lot of money, and they know they're going to lose a lot of money on reselling NVIDIA GPUs, at least. Hyperscalers, but I want to, Microsoft, AWS, Google. Correct, yeah. The Microsoft, AWS, and Google. Does Google resell? I mean, Google has TPUs. Google has TPUs, but I think you can also get H100s and so on. But there are like two ways they can make money. One is by selling to small customers who aren't actually buying in any serious volume. They're testing around, they're playing around. And if they get big, they're immediately going to do one of two things. They're going to ask you for a discount. Because they're not going to pay your crazy sort of margin that you have locked into your business. Because for CPUs, you need that. They're going to pay your massive per hour price. And so they want you to sign a long-term contract. And so that's your other way that you can make money, is you can basically do exactly what CoreWeave does, which is have them pay as much as possible upfront and lock in the contract for a long time. Or you can have small customers. But the problem is that for a hyperscaler, the GPUs to... To sell on the low margins relative to what your other business, your CPUs are, is a worse business than what you are currently doing. Because you could have spent the same money on those GPUs. And you could have trained model and you could have made a model on top of it and then turn that into a product and had high margins from your product. Or you could have taken that same money and you could have competed with NVIDIA. And you could have cut into their margin instead. But just simply reselling NVIDIA GPUs doesn't work like your CPU business. Where you're able to capture high margins from big customers and so on. And then they never leave you because your customers aren't actually price sensitive. And so they won't switch off if your prices are a little higher. You actually had a really nice chart, again, on that talk of this two by two. Sure. Of like where you want to be. And you also had some hot takes on who's making money and who isn't.

Swyx: So CoreUv locked up long-term contracts. Get that. Yes. Maybe share your mental framework. Just verbally describe it because we're trying to help the audio listeners as well. Sure. People can look up the chart if they want to.

Evan: Sure. Okay. So this is a graph of interest rates. And on the y-axis, it's a probability you're able to sell your GPUs from zero to one. And on the x-axis, it's how much they'll depreciate in cost from zero to one. And then you had ISO cost curves or ISO interest rate curves. Yeah. So they kind of shape in a sort of concave fashion. Yeah. The lowest interest rates enable the most aggressive. form of this cost curve. And the higher interest rates go, the more you have to push out to the top right. Yeah. And then you had some analysis of where every player sits in this, including CoreUv, but also Together and Modal and all these other guys. I thought that was super insightful. So I just wanted to elaborate. Basically, it's like a graph of risk and the genres of places where you can be and what the risk is associated with that. The optimal thing for you to do, if you can, is to lock in long-term contracts that are paid all up front or in with a situation in which you trust the other party to pay you over time. So if you're, you know, selling to Microsoft or something or OpenAI. Which are together 77% of the revenue of CoreUv. Yeah. So if you're doing that, that's a great business to be in because your interest rate that you can pitch for is really low because no one thinks Microsoft is going to default. And like maybe OpenAI will default, but the backing by Microsoft kind of doesn't. And I think there's enough, like, generally, it looks like OpenAI is winning that you can make it's just a much better case than if you're selling to the pre-seed startup that just raised $30 million or something pre-revenue. It's like way easier to make the case that the OpenAI is not going to default than the pre-seed startup. And so the optimal place to be is selling to the maximally low risk customer for as long as possible. And then you never have to worry about depreciation and you make lots of money. The less. Good. Good place to be is you could sell long-term contracts to people who might default on you. And then if you're not bringing it to the present, so you're not like saying, hey, you have to pay us all up front, then you're in this like more risky territory. So is it top left of the chart? If I have the chart right, maybe. Large contracts paid over time. Yeah. Large contracts paid over time is like top left. So it's more risky, but you could still probably get away with it. And then the other opportunity is that you could sell short-term contracts for really high prices. And so lots of people tried that too, because this is actually closer to the original business model that people thought would work in cloud providers for CPUs. It works for CPUs, but it doesn't really work for GPUs. And I don't think people were trying this because they were thinking about the risk associated with it. I think a lot of people are just come from a software background, have not really thought about like cogs or margins or inventory risk or things that you have to worry about in the physical world. And I think they were just like copy pasting the same business model onto CPUs. And also, I remember fundraising like a few years ago. And I know based on. Like what we knew other people were saying who were in a very similar business to us versus what we were saying. And we know that our pitch was way worse at the time, because in the beginning of SF Compute, we looked very similar to pretty much every other GPU cloud, not on purpose, but sort of accidentally. And I know that the correct pitch to give to an investor was we will look like a traditional CPU cloud with high margins and we'll sell to everyone. And that is a bad business model because your customers are price sensitive. And so what happens is if you. Sell at high prices, which is the price that you would need to sell it in order to de-risk your loss on the depreciation curve, and specifically what I mean by that is like, let's say you're selling it like $5 an hour and you're paying $1.50 an hour for the GPU under the hood. It's a little bit different than that, but you know, nice numbers, $5 an hour, $1.50 an hour. Great. Excellent. Well, you're charging a really high price per GPU hour because over time the price will go down and you'll get competed out. And what you need is to make sure that you never go under, or if you do go under your underlying cost. You've made so much money in the first part of it that the later end of it, like doesn't matter because from the whole structure of the deal, you've made money. The problem is that just, you think that you're going to be able to retain your customers with software. And actually what happens is your customers are super price sensitive and push you down and push you down and push you down and push you down, um, that they don't care about your software at all. And then the other problem that you have is you have, um, really big players like the hyperscalers who are looking to win the market and they have way more money than you, and they can push down on margin. Much better than you can. And so if they have to, and they don't, they don't necessarily all the time, um, I think they actually keep pride of higher margin, but if they needed to, they could totally just like wreck your margin at any point, um, and push you down, which meant that that quadrant over there where you're charging a high price, um, and just to make up for the risk completely got destroyed, like did not work at all for many places because of the price sensitivity, because people could just shove you down instead that pushed everybody up to the top right-hand corner of that, which is selling short-term. Contracts for low prices paid over time, which is the worst place to be in, um, the worst financial place to be in because it has the highest interest rate, um, which means that your, um, your costs go up at the same time, your, uh, your incoming cash goes down and squeezes your margins and squeezes your margins. The nice thing for like a core weave is that most of their business is over on the, on the other sides of those quadrants that the ones that survive. The only remaining question I have with core weave, and I promise I get to ask if I can compute, and I promise this is relevant to SOF Compute in general, because the framework is important, right? Sure. To understand the company. So why didn't NVIDIA or Microsoft, both of which have more money than core weave, do core weave, right? Why didn't they do core weave? Why have this middleman when either NVIDIA or Microsoft have more money than God, and they could have done an internal core weave, which is effectively like a self-funding vehicle, like a financial instrument. Why does there have to be a third party? Your question is like... Why didn't Microsoft, or why didn't NVIDIA just do core weave? Why didn't they just set up their own cloud provider? I think, and I don't know, and so correct me if I'm wrong, and lots of people will have different opinions here, or I mean, not opinions, they'll have actual facts that differ from my facts. Those aren't opinions. Those are actually indeed differences of reality, is that NVIDIA doesn't want to compete with their customers. They make a large amount of money by selling to existing clouds. If they launched their own core weave, then it would be a lot more money. It'd make it much harder for them to sell to the hyperscalers, and so they have a complex relationship with there. So not great for them. Second is that, at least for a while, I think they were dealing with antitrust concerns or fears that if they're going through, if they own too much layers of the stack, I could imagine that could be a problem for them. I don't know if that's actually true, but that's where my mind would go, I guess. Mostly, I think it's the first one. It's that they would be competing directly with their primary customers. Then Microsoft could have done it, right? That's the other question. Yeah, so Microsoft didn't do it. And my guess is that... NVIDIA doesn't want Microsoft to do it, and so they would limit the capacity because from NVIDIA's perspective, both they don't want to necessarily launch their own cloud provider because it's competing with their customers, but also they don't want only one customer or only a few customers. It's really bad for NVIDIA if you have customer concentration, and Microsoft and Google and Amazon, like Oracle, to buy up your entire supply, and then you have four or five customers or so who pretty much get to set prices. Monopsony. Yeah, monopsony. And so the optimal thing for you is a diverse set of customers who all are willing to pay at whatever price, because if you don't, somebody else will. And so it's really optimal for NVIDIA to have lots of other customers who are all competing against each other. Great. Just wanted to establish that. It's unintuitive for people who have never thought about it, and you think about it all day long. Yeah.

Swyx: The last thing I'll call out from the talk, which is kind of cool, and then I promise we'll get to SF Compute, is why will DigitalOcean and Together lose money on their clusters? Why will DigitalOcean and Together lose money on their clusters?

Evan [00:16:33]: I'm going to start by clarifying that all of these businesses are excellent and fantastic. That Together and DigitalOcean and Lambda, I think, are wonderful businesses who build excellent products. But my general intuition is that if you try to couple the software and the hardware together, you're going to lose money. That if you go out and you buy a long-term contract from someone and then you layer on services, or you buy the hardware yourself and you spin it up and you get a bunch of debt, you're going to run into the same problem that everybody else did, the same problem we did, same problem the hyperscalers did. And that's exactly what the hyperscalers are doing, which is you cannot add software and make high margins like a cloud provider can. You can pitch that into investors and it will totally make sense, and it's like the correct play in CPUs, but there isn't software you could make to make this occur. If you're spending a billion dollars on hardware, you need to make a billion dollars of software. There isn't a billion dollars of software that you can realistically make, and if you do, you're going to look like SAP. And that's not a knock on SAP. SAP makes a fuck ton of money, right? Right. Right. Right. Right. There aren't that many pieces of software that you could make, that you can realistically sell, like a billion dollars of software, and you're probably not going to do it to price-sensitive customers who are spending their entire budget already on compute. They don't have any more money to give you. It's a very hard proposition to do. And so many parties have been trying to do this, like, buy their own compute, because that's what a traditional cloud does. It doesn't really work for them. You know that meme where there's, like, the Grim Reaper? And he's, like, knocking on the door, and then he keeps knocking on the next door? We have just seen door after door after door of the Grim Reeker comes by, and the economic realities of the compute market come knocking. And so the thing we encourage folks to do is if you are thinking about buying a big GPU cluster and you are going to layer on software on top, don't. There are so many dead bodies in the wake there. We would recommend not doing that. And we, as SF Compute, our entire business is structured to help you not do that. It's helped disintegrate these. The GPU clouds are fantastic real estate businesses. If you treat them like real estate businesses, you will make a lot of money. The cloud services you can make on that, all the software you want to make on that, you can do that fantastically. If you don't own the underlying hardware, if you mix these businesses together, you get shot in the head. But if you combine, if you split them, and that's what the market does, it helps you split them, it allows you to buy, like, layer on services, but just buy from the market, you can make lots of money. So companies like Modal, who don't own the underlying compute, like they don't own it, lots of money, fantastic product. And then companies like Corbeave, who are functionally like really, really good real estate businesses, lots of money, fantastic product. But if you combine them, you die. That's the economic reality of compute. I think it also splits into trading versus inference, which are different kinds of workloads. Yeah. And then, yeah, one comment about the price sensitivity thing before we leave this. This topic, I want to credit Martin Casado for coining or naming this thing, which is like, you know, you said, you said this thing about like, you don't have room for a 10% margin on GPUs for software. Yep. And Martin actually played it out further. It's his first one I ever saw doing this at large enough runs. So let's say GPT-4 and O1 both had a total trading cost of like a $500 billion is the rough estimate. When you get the $5 billion runs, when you get the $50 billion runs, it is actually makes sense to build your own. You're going to have to get into chips, like for OpenEI to get into chip design, which is so funny. I would make an ASIC for this run. Yeah, maybe. I think a caveat of that that is not super well thought about is that only works if you're really confident. It only works if you really know which chip you're going to do. If you don't, then it's a little harder. So it makes in my head, it makes more sense for inference where you've already established it. But for training there's so much like experimentation. Any generality, yeah. Yeah. The generality is much more useful. Yeah. In some sense, you know, Google's like six generations into the CPUs. Yeah. Yeah. Okay, cool. Maybe we should go into SF Compute now. Sure. Yeah.

Alessio [00:20:37]: Yeah. So you kind of talked about the different providers. Why did you decide to go with this approach and maybe talk a bit about how the market dynamics have evolved since you started a company?

Evan [00:20:47]: So originally we were not doing this at all. We were definitely like forced into this to some extent. And SF Compute started because we wanted to go train models for music and audio in general. We were going to do a sort of generic audio model at some points, and then we were going to do a music model at some points. It was an early company. We didn't really spec down on a particular thing. But yeah, we were going to do a music model and audio model. First thing that you do when you start any AI lab is you go out and you buy a big cluster. The thing we had seen everybody else do was they went out and they raised a really big round and then they would get stuck. Because if you raise the amount of money that you need to train a model initially, like, you know, the $50 million pre-seed, pre-revenue, your valuation is so high or you get diluted so much that you can't raise the next round. And that's a very big ask to make. And also, I don't know, I felt like we just felt like we couldn't do it. We probably could have in retrospect, but I think one, we didn't really feel like we could do it. Two, it felt like if we did, we would have been stuck later on. We didn't want to raise the big round. And so instead, we thought, surely by now, we would be able to just go out. To any provider and buy like a traditional CPU cloud would sell offer you and just buy like on demand or buy like a month or so on. And this worked for like small incremental things. And I think this is where we were basing it off. We just like assumed we could go to like Lambda or something and like buy thousands of at the time A100s. And this just like was not at all the case. So we started doing all the sales calls with people and we said, OK, well, can we just get like month to month? Can we get like one month of compute or so on? Everyone told us at the time, no. You need to have a year long contract or longer or you're out of luck. Sorry. And at the time, we were just like pissed off. Like, why won't nobody sell us a month at a time? Nowadays, we totally understand why, because it's the same economic reason. Because if you if they had sold us the month to month or so on and we canceled or so on, they would have massive risk on that. And so the optimal thing to do was to only to just completely abandon the section of the market. We didn't like that. So our plan was we were going to buy a year long contract anyway. We would use a month. And then we would. At least the other 11 months. And we were locked in for a year, but we only had to pay on every individual month. And so we did this. But then immediately we said, oh, shit, now we have a cloud provider, not a like training models company, not an AI lab, because every 30 days we owed about five hundred thousand dollars or so and we had about five hundred thousand dollars in the bank. So that meant that every single month, if we did not sell out our cluster, we would just go bankrupt. So that's what we did for the first year of the company. And when you're in that position. You try to think how in the world you get out of that position, what that transition to is, OK, well, we tend to be pretty good at like selling this cluster every month because we haven't died yet. And so what we should do is we should go basically be like this broker for other people and we will be more like a GPU real estate or like a GPU realtor. And so we started doing that for a while where we would go to other people who had who was trying to sell like a year long contract with somebody and we'd go to another person who like maybe this person wanted six months and somebody else on six months or something and we'd like combine all these people. Together to make the deal happen and we'd organize these like one off bespoke deals that looked like basically it ended up with us taking a bunch of customers, us signing with a vendor, taking some cut and then us operating the cluster for people typically with bare metal. And so we were doing this, but this was definitely like a oh, shit, oh, shit, oh, shit. How do we get out of our current situation and less of a like a strategic plan of any sort? But while we were doing this, since like the beginning of the company, we had been thinking about how to buy GPU clusters, how to sell them effectively, because we'd seen every part of it. And what we ended up with was like a book of everybody who's trying to buy and everyone is trying to sell because we were these like GPU brokers. And so that turned into what is today SF Compute, which is a compute market, which we think we are the functionally the most liquid GPU market of any capacity. Honestly, I think we're the only thing that actually is like a real market that there's like bids and asks and there's like a like a trading engine that combines everything. And so. I think we're the only place where you can do things that a market should be able to do. Like you can go on SF Compute today and you get thousands of H100s for an hour if you want. And that's because there is a price for thousands of GPUs for an hour. That is like not a thing you can reasonably do on kind of any other cloud provider because nobody should realistically sell you thousands of GPUs for an hour. They should sell it to you for a year or so on. But one of the nice things about a market is that you can buy the year on SF Compute. But then if you need to sell. Back, you can sell back as well. And that opens up all these little pockets of liquidity where somebody who's just trying to buy for a little bit of time, some burst capacity. So people don't normally buy for an hour. That's not like actually a realistic thing, but it's like the range somebody who wants, who is like us, who needed to buy for a month can actually buy for a month. They can like place the order and there is actually a price for that. And it typically comes from somebody else who's selling back. Somebody who bought a longer term contract and is like they bought for some period of time, their code doesn't work, and now they need to like sell off a little bit.

Alessio [00:25:49]: What are the utilization rates at which a market? What are the utilization rates at which a market? Like this works, what do you see the usual GPU utilization rate and like at what point does the market get saturated?

Evan [00:26:00]: Assuming there are not like hardware problems or software problems, the utilization rate is like near 100 percent because the price dips until the utilization is 100 percent. So the price actually has to dip quite a lot in order for the utilization not to be. That's not always the case because you just have logistical problems like you get a cluster and parts of the InfiniBand fabric are broken. And there's like some issue with some switch somewhere and so you have to take some portion of the cluster offline or, you know, stuff like this, like there's just underlying physical realities of the clusters, but nominally we have better utilization than basically anybody because, but that's on utilization of the cluster, like that doesn't necessarily translate into, I mean, I actually do think we have much better overall money made for our underlying vendors than kind of anybody else. We work with the other GPU clouds and the basic pitch to the other GPU clouds is one. So we can sell your broker so we can we can find you the long term contracts that are at the prices that you want, but meanwhile, your cluster is idle and for that we can increase your utilization and get you more money because we can sell that idle cluster for you and then the moment we find the longer, the bigger customer and they come on, you can kick off those people and then go to the other ones. You get kind of the mix of like sell your cluster at whatever price you can get on the market and then sell your cluster at the big price that you want to do for long term contract, which is your ideal business model. And then the benefit of the whole thing being on the market. Is you can pitch your customer that they can cancel their long term contract, which is not a thing that you can reasonably do if you are just the GPU cloud, if you're just the GPU cloud, you can never cancel your contract, because that introduces so much risk that you would otherwise, like not get your cheap cost of capital or whatever. But if you're selling it through the market, or you're selling it with us, then you can say, hey, look, you can cancel for a fee. And that fee is the difference between the price of the market and then the price that they paid at, which means that they canceled and you have the ability to offer that flexibility. But you don't. You don't have to take the risk of it. The money's already there and like you got paid, but it's just being sold to somebody else. One of our top pieces from last year was talking about the H100 glut from all the long term contracts that were not being fully utilized and being put under the market. You have on here dollar a dollar per hour contracts as well as it goes up to two. Actually, I think you were involved. You were obliquely quoted in that article. I think you remember. I remember because this was hidden. Well, we hid your name, but then you were like, yeah, it's us. Yeah. Could you talk about the supply and demand of H100s? Was that just a normal cycle? Was that like a super cycle because of all the VC funding that went in in 2003? What was that like? GPU prices have come down. Yeah, GPU prices have come down. And there's some part that has normal depreciation cycle. Some part of that is just there were a lot of startups that bought GPUs and never used them. And now they're lending it out and therefore you exist. There's a lot of like various theories as to why. This happened. I dislike all of them because they're all kind of like they're often said with really high confidence. And I think just the market's much more complicated than that. Of course. And so everything I'm going to say is like very hedged. But there was a series of like places where a bunch of the orders were placed and people were pitching to their customers and their investors and just the broader market that they would arrive on time. And that is not how the world works. And because there was such a really quick build out of things, you would end up with bottlenecks in the supply chain somewhere that has nothing to do with necessarily the chip. It's like the InfiniBand cables or the NICs or like whatever. Or you need a bunch of like generators or you don't have data center space or like there's always some bottleneck somewhere else. And so a lot of the clusters didn't come online within the period of time. But then all the bottlenecks got sorted out and then they all came online all at the same time. So I think you saw a short. There was a shortage because supply chain hard. And then you saw a increase or like a glut because supply chain eventually figure itself out. And specifically people overordered in order to get the allocation that they wanted. Then they got the allocations and then they went under. Yeah, whatever. Right. There was just a lot of shenanigans. A caveat of this is every time you see somebody like overordered, there is this assumption that the problem was like the demand went down. I don't think that's the case at all. And so I want to clarify that. It definitely seems like a shortage. Like there's more demand for GPUs than there ever was. It's just that there was also more supply. So at the moment, I think there is still functionally a glut. But the difference that I think is happening is mostly the test time inference stuff that you just need way more chips for that than you did before. And so whenever you make a statement about the current market, people sort of take your words and then they assume that you're making a statement about the future market. And so if you say there's a glut now, people will continue to think there's a glut. But I think what is happening at the moment. My general prediction is that like by the winter, we will be back towards shortage. But then also, this very much depends on the rollout of future chips. And that comes with its own. I think I'm trying to give you like a good here's Evan's forecast. Okay. But I don't know if my forecast is right. You don't have to. Nobody is going to hold you to it. But like I think people want to know what's true and what's not. And there's a lot of vague speculations from people who are not that close to the market actually. And you are. I think I'm a closer. Close to the market, but also a vague speculator. Like I think there are a lot of really highly confident speculators and I am indeed a vague speculator. I think I have more information than a lot of other people. And this makes me more vague of a spectator because I feel less certain or less confident than I think a lot of other people do. The thing I do feel reasonably confident about saying is that the test time inference is probably going to quite significantly expand the amount of compute that was used for inference. So a caveat. This is like pretty much all the inference demand is in a few companies. A good example is like lots of bio and pharma was using H100s training sort of the bio models of sorts. And they would come along and they would buy, you know, thousands of H100s for training and then just like not a lot of stuff for inference. Not in any, not relative to like an opening iron anthropic or something because they like don't have a consumer product. Their inference event, if they can do it right. There's really like only one inference event that matters. And obviously I think they're going to run into it. And Batch and they're not going to literally just run one inference event. But like the one that produces the drug is the important one. Right. And I'm dumb and I don't know anything about biology, so I could be completely wrong here. But my understanding is that's kind of the gist. I can check that for you. You can check that for me. Check that for me. But my understanding is like the one that produces the sequence that is the drug that, you know, cures cancer or whatever. That's the important deal. But like a lot of models look like this where they're sort of more enterprising use cases or they're so prior to something that looks like test time inference. You got lots and lots of demand for training and then pretty much entirely fell off for inference. And I think like we looked at like Open Router, for example, the entirety of Open Router that was not anthropic or like Gemini or OpenAI or something. It was like 10 H100 nodes or something like that. It's just like not that much. It's like not that many GPUs actually to service that entire demand. But that's like a really sizable portion of the sort of open source market. But the actual amount of compute needed for it was not that much. But if you imagine like what an OpenAI needs for like GPT-4, it's like tremendously big. But that's because it's a consumer product that has almost all the inference demand. Yeah, that's a message we've had. Roughly open source AI compared to closed AI is like 5%. Yeah, it's like super small. Super small. It's super small. Super small. But test time inference changes that quite significantly. So I will... I will expect that to increase our overall demand. But my question on whether or not that actually affects your compute price is entirely based on how quickly do we roll out the next chips. The way that you burst is different for test time.

Alessio [00:34:01]: Any thoughts on the third part of the market, which is the more peer-to-peer distributed, some are like crypto-enabled, like Hyperbolic, Prime Intellect, and all of that. Where do those fit? Like, do you see a lot of people will want to participate in a peer-to-peer market? Or just because of the capital requirements at the end of the day, it doesn't really matter?

Evan [00:34:20]: I'm like wildly skeptical of these, to be frankly. The dream is like steady at home, right? I got this $15.90. Nobody has $15.90. $14.90 sitting at home. I can rent it out. Yeah. Like, I just don't really think this is going to ever be more efficient than a fully interconnected cluster with InfiniBand or, you know, whatever the sort of next spec might be. Like, I could be completely wrong. But speaking of... I mean, like, SpeedoLite is really hard to beat. And regardless of whatever you're using, you just like can't get around that physical limitation. And so you could like imagine a decentralized market that still has a lot of places where there's like co-location. But then you would get something that looks like SF Compute. And so that's what we do. That's why we take our general take is like on SF Compute, you're not buying from like random people. You're buying from the other GPU clouds, functionally. You're buying from data centers that are the same genre of people that you would work with already. And you can specify, oh, I want all these nodes to be co-located. And I don't think you're really going to get around that. And I think I buy crypto for the purposes of like transferring money. Like the financial system is like quite painful and so on. I can understand the uses of it to sort of incentivize an initial market or try to get around the cold start problem. We've been able to get around the cold start problem just fine. So it didn't actually need that at all. What I do think is totally possible is you could launch a token and then you could like subsidize the crypto. You could compute prices for a bit, but like maybe that will help you. I think that's what Nuus is doing. Yeah, I think there's lots of people who are trying to do things like this, but at some point that runs out. So I would, I think generally agree. I think the only thread in that model is very fine grained mixture of experts that can be like algorithms can shift to adapt to hardware realities. And the hardware reality is like, okay, it's annoying to do large co-located clusters. Then we'll just redesign attention or whatever in our architecture to distribute it more. There was a little bit buzz of block attention last year that Strong Compute made a big push on. But I think like, you know, in a world where we have 200 experts in MOE model, it starts to be a little bit better. Like, I don't disagree with this. I can imagine the world in which you have like, in which you've redesigned it to be more parallelizable, like across space.

Evan [00:36:43]: But assuming without that, your hardware limitation is your speed of light limitation. And that's a very hard one to get around.

Alessio [00:36:50]: Any customers or like stories that you want to shout out of like maybe things that wouldn't have been economically viable like others? I know there's some sensitivity on that.

Evan [00:37:00]: My favorites are grad students, are folks who are trying to do things that would normally otherwise require the scale of a big lab. And the grad students are like the worst pilots. They're like the worst possible customer for the traditional GPU clouds because they will immediately turn if you sell them a thing because they're going to graduate and they're not going to go anywhere. They're not going to like, that project isn't continuing to spend lots of money. Like sometimes it does, but not if you're like working with the university or you're working with the lab of some sort. But a lot of times it's just like the ability for us to offer like big burst capacity, I think is lovely and wonderful. And it's like one of my favorite things to do because all those folks look like we did. And I have a special place in my heart for that. I have a special place in my heart for young hackers and young grad students and researchers who are trying to do the same genre of thing that we are doing. For the same reason, I have a special place in my heart for like the startups, the people who are just actively trying to compete on the same scale, but can't afford it time-wise, but can afford it spike-wise. Yeah, I liked your example of like, I have a grant of 100K and it's expiring. I got to spend it on that. That's really beautiful. Yeah. Interesting. Has there been interesting work coming out of that? Anything you want to mention? Yeah. So from like a startup perspective, like Standard Intelligence and Find, P-H-I-N-D. We've had them on the pod.

Swyx [00:38:23]: Yeah. Yeah.

Evan [00:38:23]: That was great. And then from grad students' perspective, we worked a lot with like the Schmidt Futures grantees of various sorts. My fear is if I talk about their research, I will be completely wrong to a sort of almost insulting degree because I am very dumb. But yeah. I think one thing that's maybe also relevant startups and GPUs-wise. Yeah. Is there was a brief moment where it kind of made sense that VCs provided GPU clusters. And obviously you worked at AI Grants, which set up Andromeda, which is supposedly a $100 million cluster. Yeah. I can explain why that's the case or why anybody would think that would be smart. Because I remember before any of that happened, we were asking for it to happen. Yeah. And the general reason is credit risk. Again, it's a bank. Yeah. I have lower risk than you due to credit transformation. I take your risk onto my balance sheet. Correct. Exactly. If you wanted to go for a while, if you wanted to go set up a GPU cluster, you had to be the one that actually bought the hardware and racked it and stacked it, like co-located it somewhere with someone. Functionally, it was like on your balance sheet, which means you had to get a loan. And you cannot get a loan for like $50 million as a startup. Like not really. You can get like venture debt and stuff, but like it's like very, very difficult to get a loan of any serious price for that. But it's like not that difficult to get a loan for $50 million. If you already have a fund or you already have like a million dollars under your assets somewhere or like you personally can like do a personal guarantee for it or something like this. If you have a lot of money, it is way easier for you to get a loan than if you don't have a lot of money. And so the hack of a VC or some capital partner offering equity for compute is always some arbitrage on the credit risk. That's amazing. Yeah. That's a hack. You should do that. I don't think people should do it right now. I think the market has like, I think it made sense at the time and it was helpful and useful for the people who did it at the time. But I think it was a one-time arbitrage because now there are lots of other sources that can do it. And also I think like it made sense when no one else was doing it and you were the only person who was doing it. But now it's like it's an arbitrage that gets competed down. Sure. So it's like super effective. I wouldn't totally recommend it. Like it's great that Andromeda did it. But the marginal increase of somebody else doing it is like not super helpful. I don't think that many people have followed in their footsteps. I think maybe Andreessen did it. Yeah. That's it. I think just because pretty much all the value like flows through Andromeda. What? That cannot be true. How many companies are in the air, Grant? Like 50? My understanding of Andromeda is it works with all the NFTG companies or like several of the NFTG companies. But I might be wrong about that. Again, you know, something something. Nat, don't kill me. I could be completely wrong. But the but you know, I think Andromeda was like an excellent idea to do at the right time in which it occurred. Perfect. His timing is impeccable. Timing. Yeah. Nat and Daniel are like, I mean, there's lots of people who are like... Sears? Yeah. Sears. Like S-E-E-R. Oh, Sears. Like Sears of the Valley. Yeah. They for years and years before any of the like ChatGPT moment or anything, they had fully understood what was going to happen. Like way, way before. Like. AI Grant is like, like five years old, six years old or something like that. Seven years old. When I, when it like first launched or something. Depends where you start. The nonprofit version. Yeah. The nonprofit version was like, like happening for a while, I think. It's going on for quite a bit of time. And then like Nat and Daniel are like the early investors in a lot of the sort of early AI labs of various sorts. They've been doing this for a bit.

Alessio [00:41:58]: I was looking at your pricing yesterday. We're kind of talking about it before. And there's this weird thing where one week is more expensive of both one day and one month. Yeah. What are like some of the market pricing dynamics? What are things that like this to somebody that is not in the business? This looks really weird. But I'm curious, like if you have an explanation for it, if that looks normal to you. Yeah.

Evan [00:42:18]: So the simple answer is preemptible pricing is cheaper than non-preemptible pricing. And the same economic principle is the reason why that's the case right now. That's not entirely true on SF Compute. SF Compute doesn't really have the concept of preemptible. Instead, what it has is very short reservations. So, you know, you go to a traditional cloud provider and you can say, hey, I want to reserve contract for a year. We will let you do a reserve contract for one hour, which is the part of SFC. But what you can do is you can just buy every single hour continuously. And you're reserving just for that hour. And then the next hour you reserve just for that next hour. And this is obviously like a built in. This is like an automation that you can do. But what you're seeing when you see the cheap price is you're seeing somebody who's buying the next hour, but maybe not necessarily buying an hour after that. So if the price goes up. Up too much. They might not get that next hour. And the underlying part of this of where that's coming from the market is you can imagine like day old milk or like milk that's about to be old. It might drop its price until it's expired because nobody wants to buy the milk that's in the past. Or maybe you can't legally sell it. Compute is the same way. No, you can't sell a block of compute that is not that is in the past. And so what you should do in the market and what people do do is they take. They take a block. A block of compute. And then they drop it and drop it and drop it and drop into a floor price right before it's about to expire. And they keep dropping it until it clears. And so anything that is idle drops until some point. So if you go and use on the website and you set that that chart to like a week from now, what you'll see is much more normal looking sort of curves. But if you say, oh, I want to start right now, that immediate instant, here's the compute that I want right now is the is functionally the preemptible price. It's where most people are getting the best compute or like the best compute prices from. The caveat of that is you can do really fun stuff on SFC if you want. So because it's not actually preemptible, it's it's reserved, but only reserved for an hour, which means that the optimal way to use as of compute is to just buy on the market price, but set a limit price that is much higher. So you can set a limit price for like four dollars and say, oh, if the market ever happens to spike up to four dollars, then don't buy. I don't want to buy that at that price for that price. I don't want to buy that at that price for that price for an hour. But otherwise, just buy at the cheapest price. And if you're comfortable with that of the volatility of it, you're actually going to get like really good prices, like close to a dollar an hour or so on, sometimes down to like 80 cents or whatever. You said four, though. Yeah. So that's the thing. You want to lower the limit. So four is your max price. Four is like where you basically want to like pull the plug and say don't do it because the actual average price is not or like the, you know, the preemptible price doesn't actually look like that. So what you're doing when you're saying four is always, always, always give me this compute. Like continue to buy every hour. Don't preempt me. Don't kick me off. And I want this compute and just buy at the preemptible price, but never kick me off. The only times in which you get kicked off is if there is a big price spike. And, you know, let's say one day out of the year, there's like a four dollar an hour price because of some weird fluke or something. If there are other periods of time, you're actually getting a much lower price than you. It makes sense. Your your average cost that you're actually paying is way better. And your trade off here is you don't literally know what price you're going to get. So it's volatile. But your actual average historically has been like everyone who's done this has gotten wildly better prices. And this is like one of the clever things you can do with the market. If you're willing to make those trade offs, you can get a lot of really good prices. You can also do other things like you can only buy at night, for example. So the price goes down at night. And so you can say, oh, I want to only buy, you know, if the price is lower than 90 cents. And so if you have some long running job, you can make it only run on 90 cents and then you recover back and so on. Yeah. So what you can kind of create as like a spot inst is what other the CPU world has. Yes. But you've created a system where you can kind of manufacture the exact profile that you want. Exactly. That is not just whatever the hyperscalers offer you, which is usually just one thing. Correct. SF Compute is like the power tool. The underlying primitives of like hourly compute is there. Correct. Yeah, it's pretty interesting. I've often asked OpenAI. So like, you know, all these guys. Cloud as well. They do batch APIs. So it's half off of whatever your thing is. Yeah. And the only contract is we'll return in 24 hours. Sure. Right. And I was like, 24 hours is good. But sometimes I want one hour. I want four hours. I want something. And so based off of SF Compute's system, you can actually kind of create that kind of guarantee. Totally. That would be like, you know, not 24, but within eight hours, within four hours, like the work half of a workday. Yes. I can return your results to you. And then I can return it to you. And if your latency requirements are like that low, actually it's fine. Yes. Correct. Yeah. You can carve out that. You can financially engineer that on SFC. Yeah. Yeah. I mean, I think to me that unlocks a lot of agent use cases that I want, which is like, yeah, I worked in a background, but I don't want you to take a day. Yeah. Correct. Take a couple hours or something. Yeah. This touches a lot of my like background because I used to be a derivatives trader. Yeah. And this is a forward market. Yeah. A futures forward market, whatever you call it. Not a future. Very explicitly not a future. Not yet a futures. Yes. But I don't know if you have any other points to talk about. So you recognize that you are a, you know, a marketplace and you've hired, I met Alex Epstein at your launch event and you're like, you're, you're building out the financialization of GPUs. Yeah. So part of that's legal. Mm-hmm. Totally. Part of that is like listing on an exchange. Yep. Maybe you're the exchange. I don't know how that works, but just like, talk to me about that. Like from the legal, the standardization, the like, where is this all headed? You know, is this like a full listed on the Chicago Mercantile Exchange or whatever? What we're trying to do is create an underlying spot market that gives you an index price that you can use. And then with that index price, you can create a cash settled future. And with a cash settled future, you can go back to the data centers and you can say, lock in your price now and de-risk your entire position, which lets you get cheaper cost of capital and so on. And that we think will improve the entire industry because the marginal cost of compute is the risk. It's risk as shown by that graph and basically every part of this conversation. It's risk that causes the price to be all sorts of funky. And we think a future is the correct solution to this. So that's the eventual goal. Right now you have to make the underlying spot market in order to make this occur. And then to make the spot market work, you actually have to solve a lot of technology problems. You really cannot make a spot market work if you don't run the clusters, if you don't have control over them, if you don't know how to audit them, because these are super computers, not soybeans. They have to work. In a way that like, it's just a lot simpler to deliver a soybean than it is to deliver it. I don't know. Talk to the soybean guys. Sure. You know? Yeah. But you have to have a delivery mechanism. Your delivery mechanism, like somebody somewhere has to actually get the compute at some point and it actually has to work. And it is really complicated. And so that is the other part of our business that we go and we build a bare metal infrastructure stack that goes. And then also we do auditing of all the clusters. You sort of de-risk the technical perspective and that allows you to eventually de-risk the financial perspective. And that is kind of the pitch of SF Compute. Yeah. I'll double click on the auditing on the clusters. This is something I've had conversations with Vitae on. He started Rika and I think he had a blog post which kind of shone the light a little bit on how unreliable some clusters are versus others. Correct. Yeah. And sometimes you kind of have to season them and age them a little bit to find the bad cards. You have to burn them in. Yeah. So what do you do to audit them? There's like a burn-in process, a suite of tests, and then active checking and passive checking. Burn-in process is where you typically run LINPACK. LINPACK is this thing that like a bunch of linear algebra equations that you're stress testing the GPUs. This is a proprietary thing that you wrote? No, no, no. LINPACK is like the most common form of burn-in. If you just type in burn-in, typically when people say burn-in, they literally just mean LINPACK. It's like an NVIDIA reference version of this. Again, NVIDIA could run this before they ship, but now the customers have to do it. It's annoying. You're not just checking for the GPU itself. You're checking like the whole component, all the hardware. And it's a lot of work. It's an integration test. It's an integration test. Yeah. So what you're doing when you're running LINPACK or burn-in in general is you're stress testing the GPUs for some period of time, 48 hours, for example, maybe seven days or so on. And you're just trying to kill all the dead GPUs or any components in the system that are broken. And we've had experiences where we ran LINPACK on a cluster and it rounds out, sort of comes offline when you run LINPACK. This is a pretty good sign that maybe there is a problem with this cluster. Yeah. So LINPACK is like the most common sort of standard test. But then beyond that, what you do is we have like a series of performance tests that replicate a much more realistic environment as well that we run just assuming if LINPACK works at all, then you run the next set of tests. And then while the GPUs are in operation, you're also going through and you're doing active tests and passive tests. Passive tests are things that are running in the background while somebody else is running, while like some other workload is running. And active tests are during like idle periods. You're running some sort of check that would otherwise sort of interrupt something. And then the active tests will take something offline, basically. Or a passive check might mark it to get taken offline later and so on. And then the thing that we are working on that we have working partially but not entirely is automated refunds, which is basically like, is the case that the hardware breaks so much. And there's only so much that we can do and it is the effect of pretty much the entire industry. So a pretty common thing that I think happens to kind of everybody in the space is a customer comes online, they experience your cluster, and your cluster has the same problem that like any cluster has, or it's I mean, a different problem every time, but they experience one of the problems of HPC. And then their experience is bad. And you have to like negotiate a refund or some other thing like this. It's always case by case. And like, yeah, a lot of people just eat the cost. Correct. So one of the nice things about a market that we can do as we get bigger and have been doing as we can bigger is we can immediately give you something else. And then also we can automatically refund you. And you're still gonna experience it like the hardware problems aren't going away until the underlying vendors fix things. But honestly, I don't think that's likely because you're always pushing the limits of HPC. This is the case of trying to build a supercomputer. that's one of the nice things that we can do is we can switch you out for somebody else somewhere, and then automatically refund you or prorate or whatever the correct move is. One of the things that you say in this conversation with me was like, you know, you know, a provider is good when they guarantee automatic refunds. Which doesn't happen. But yeah, that's, that's in our contact with all the underlying cloud providers. You built it in already. Yeah. So we have a quite strict SLA that we pass on to you. The reason why I'm like, hedging on this is because we have some amount of active checks, we have some amount of passive checks. There are always new genres of bullshit, and the new genres of bullshit might cause a customer to have bad experience. And the active or passive checks didn't catch it. And so then it's a manual process after that. Then we have like a literal thing in our website that you can just say, Hey, some hardware problem, please tell us. And then we will go and resolve it for you. How, I mean, cards don't change generation to generation. What is a new genre of bullshit? If every component piece in the cluster has maybe like a one in a hundred chance of failing, or maybe a one in a thousand chance of failing, or maybe one in 10,000 chance of failing, You discover them. You discover them. So there's ones that like maybe nobody saw, maybe you didn't see, or maybe only matters for this one cluster with this motherboard in this particular data center or something. There's new interactions that otherwise don't happen. Most problems are really common and you can adapt to them. Like, like a GPU falls off a bus is like one of the most common things that can happen. So it's not SF Compute's job to go fix those things. No, it totally is to some extent. Totally is to some extent. So we, we operate the cluster. So unlike a reseller, which is what we were doing before. Yeah. In almost all cases, we have BMC access. So if on your laptop, there's like the button in the top right hand corner that you can hold down to like re-image the machine, there's a similar thing in like server X that you is like this other box that kind of plugs in and it basically lets you reset the machine from outside. And it's like remote, it's a remote hand sort of thing. So we ask for this and we get this from a lot of our vendors, which means we have quite a lot of ability to solve problems for customers in a way that you might not actually get from a reseller. Oftentimes we are the person who's debugging your cluster. For most customers that we work with, we have Slack channel. Our entire engineering team gets put in the Slack channel. If there was a problem at 2am, we are the ones who are debugging your problem at 2am. Not always the case because we don't physically run the hardware cluster or like the data center itself, but most problems are solvable through this. So that's the auditing side. The other side is I think of a standardization or whatever you call it. Beyond auditing. The other part of the work is kind of standardizing the commodity contracts. Yeah. So there's two ways that we do that. One is that you set like a this or better list. So you set like a spec list and you say, oh, you're going to get like a common variability is the amount of storage on the cluster. And so you'll say like, oh, you're going to get X or better. And there's some guarantee minimum and sometimes you might get more. And then we're working on a persistent storage layer that might sort of abstract a lot of this way, but mostly it's that. And then there's like a white list of motherboards and various things. Genres of things. But the other part is we run the clusters from bare metal up. And so we make a thing that's this like it's a UEFI shim. And if you're not familiar with what UEFI is, a UEFI is like the sort of firmware modern version of BIOS. Modern meaning it's been around for like forever. But you know, BIOS is like really old. It's like this whole IBM thing. And you can write code that exists at the UEFI layer. And again, when you hear UEFI, you should think BIOS. And it does the same sort of thing. It does the same thing as a Pixie boot, but in environments in which Pixie boot doesn't necessarily always work for us. So it basically sits at your BIOS, downloads an image, boots into an image that's like custom for the user. And then on top of that image, we can throw Kubernetes on it. We can throw VMs on it or whatever you want. And at some point, we'll probably like do more stuff with that. But that's functionally what we can do. The nice thing, though, is that because you control from that layer, you can easily image an entire cluster. You make it all the same. You can run your performance tests all automated. So much nicer. Right. Than what we used to do. Yeah. I mean, that is a very important work. I think like for me, as a trader, I need standard contracts. And so there basically needs to be the safe of a GPU. Yes. What we functionally do is we have a market under the hood that is focused on the buyer and the seller, and it's optimized for them. And then beyond that, for a trader, you can standardize around a certain segment of it. And you can trade on that contract. That's the goal that we're trying to get to. But you start by making something that works really well for buyers and really well for sellers. For those who are not familiar with derivatives markets, I can go ahead and say this because the point of being cash settled, which is something that you mentioned, which I think people might miss, is that you don't have to take physical delivery of the GPUs. Right. And so it's a pure financial instrument, which actually does mean that almost for certain, there will be more volume on SFC's marketplace than actually change hands in GPU terms. To be super clear. We are not a derivatives market. This doesn't happen yet. Yeah. We are not a derivatives market. We may in the future work to create a cash settled future. We are not currently a derivatives market. We are an online spot market. Yeah. I just think like people, normies get really upset when they're like, then they learn things like, oh, like derivatives on mortgages are like 12 times larger than the mortgages themselves. Yes. Yeah. No, I, um, a common thing that people have talked to us about, or like a fear or concern, I think people have is like, oh, you're financializing. Compute. And this will like cause various problems of sorts. Subprime crisis. Yeah. Um, and I think, so first I think part of this is just because crypto caused a lot of people to think about finance in the like very de-gen way for the right word. Um, and then before that, um, the sort of 2008, 2009 crisis, um, caused people to think about it also in sort of like a de-genny way. And this is very much not our mindset. The reason to create a derivative at all, or the reason to create a future at all is a risk reduction thing. Um, that's what futures do. The reason why a farmer wants a future is because they have no idea what the weather is going to do. And they don't want to be on the hook, um, for like they have small margins and if things go wrong, they really, really want to have a locked in price. Um, so that way they can like continue to exist for the next year. Data centers are the same way. The way that they solve it today is you go out and you sign long-term contracts with your customers. What that does for you is it means your business is de-risked. Um, you don't have to worry about the revenue for the next year. But that means that the customer now has to worry about what they're going to do with all this compute. And if they don't optimally use it and so on and so on, and that just pushes everything onto the startups who then in turn, push it on to VCs. And so what the VCs are forced to do in order to invest in AI is they have to go and write big, giant valuations, like pre-revenue at ridiculous multiples. So what you've done by not having a future is you've inflated the venture capital market, and that is a bubble. That's totally going to pop. At some point, like a lot of the companies are not going to work and the valuations are not going to work. And what's going to happen is a lot of these funds aren't going to return back to their LPs. And that affects the broader market. The way that you solve that, the way that you add security to the entire economic system in this chain is you add a future. That's how we did it in lots of other markets. It doesn't have to be this like, oh my gosh, we're going to like speculate on GB prices and like whatever. No. The whole point of SF Compute is to reduce the risk. Reduce the technical risk. Reduce the financial risk. Let's just chill out a little bit. There's so much other random shit. It's supercomputers. There's AGI, whatever. No. Let's just like chill the fuck out. I mean, also like Dan is going, raising like at a $30 billion valuation for Ilya, you know, like. Yeah. If everybody else in all of AI is like pushing the hype and the extreme, everything we've been trying to do is go the other way. Like whole website is just like a fucking single page. Um, like the entire brand is just like, what if we were? We're like calm in nature. And then everything that we do as the product is just calm. What if we, what if we were the opposite force of the big hypey extreme thing? What if we just like chilled things out? And part of that was because we, in the beginning were at the whim of the hypey nature. Like our entire origin is every 30 days and we don't sell out, we're going to go crazy and just completely bankrupt the company. And so everybody in the company is just like, what if we just chilled out? What if, what if we stopped? Yeah. This is the first time I've ever heard derivatives are the way to chill out. Yes. No. Futures are the way to chill out. Futures are the way to chill out the entire industry. And um, we wouldn't be doing this if it wasn't that case. I like that.

Alessio [01:01:20]: You have a very nice brand with a, you know, clear sky. We have to ask about the website. Yeah. What was the inspiration behind it? Why did you not go the black neon, more cool thing and go the more nature?

Evan [01:01:33]: I don't think I really am a black neon sort of person. I say. I'm wearing black pants and I thought I was wearing a black shirt, but apparently I'm not. So, um, the actual, the actual thing was a lot of companies do this thing where they, their website, you go to there and it's like a magical experience and like everything is extreme and amazing and credible and then you go to the product and it's like some SaaS app or something. Um, and it's like not actually that exciting. And that expectation of being like really, really good. And then the fall off the drop of not being really, really good. Yeah. It's something that from a product perspective, I never want it to happen, especially because in the beginning, like our product was really bad. And so I don't want to set the expectation that it's going to be like an amazing experience. I want to set the expectation that it's going to be like a good price for short term bursts. And so what we did instead is we set the thing to be really low. You set your expectations really low and then you get a supercomputer for like millions of dollars cheaper than you would have otherwise gotten your supercomputer. And so you have the opposite expectation. You have like really low expectations that are like mild or met higher. Yeah. And I think that's like the correct way to do things. But also I think we were just like so sick of hype and excitement and, um, I just like really want to like not do that. It's weird. Like by, by being anti-hype, you have created hype. Like I would say like the, the, the, the, the vibes are immaculate, you know, like you just, you go to like at the, the bay, the Cal trade, you just put up like a banner. This just, just says SF compute. True. That banner was created about five minutes before we had to actually put something up like before the deadline was there. Yeah. So it opens up Microsoft word and you did some serif. What is the font? Exactly. I don't know. Yeah. That was, um, indeed. Um, the, yeah, I think every time we tried to do the, the only caveat to this, the only caveat that we ever violate this rule with, uh, is when we're pitching San Francisco, I think San Francisco is amazing. So sometimes you will see these like advertisements from the city. Yeah. The city. Um, so if there's a part of San Francisco computes brand, which are these beautiful like images of SF. Yeah. And I am the complete opposite about this. I am such a San Francisco promoter that any time we talk about the city, I want to show the city from the like eyes that we have, which is mostly just gorgeous, beautiful area with nature. Like a lot of people think about San Francisco and they think about like tech industry or they think, yeah, or the tenderloin or something like grind culture or something. And no, like I think about like the fog, um, and just like the gorgeous view over the bridge and just the fact that there is this like massive amount of optimism in the city. And it's the backdrop of that optimism is the most beautiful countryside in all of the world. And so anytime we talk about SF, you will see like, or like we have a billboard somewhere that's just like local friendly supercomputer or whatever. And then the backdrop is like beautiful and amazing. And that's because to some extent we're pitching the city and the people here. And I think that people in the city here are actually really amazing. And so you get to earn the brand. Um, cause the expectations are met. Whereas I think on our own product, I'm typically want it to be better. And so I set the brand a lot lower. Um, and then the expectations are higher. Um, and you still meet the expectations, but you, you set them a little lower. I know. Are you the designer? I know you have an artistic side. Um, so, uh, I was in the beginning, so, uh, I'm like a figurative artist, so I draw people. Um, but we've worked with a design firm. Airfoil was really excellent with us. And then, um, nowadays though, John Pham, um, had a design from Vercel. Yeah. John is unbelievably amazing. Yeah. Um, I think the amount of care and craft and attention to detail that he puts into just everything is so cool. Yeah. Um, like if you go on our buy page right now, you go to sfcompute.com slash buy. There is an Easter egg there that will, you should find. I almost don't want to spoil it, but you should go find that Easter egg. If you just like hover the mouse around the thing in the top right hand corner, um, you will, you'll find it. Yeah, tweet at Evan if you find it. And then the, um, other person is, um, Ethan Anderson, our COO, who, um, has this really, really cool design. He has a RISD design background. And so, uh, his, like, uh, he used to be sort of industrial designery. I'm probably going to say that wrong. He's probably not an actual industrial designer, but design background, same. So I think between me and John and, uh, Ethan, um, I think we. The source of the vibes. The source of the vibes. I had to ask. Yeah. Okay. So we're going to zoom out a little bit. One of the last things I wanted to ask you was actually like, I remember, I think the first time that you was in like kind of cello and you were working on your email. Oh yeah. Yeah. Yeah. And I have a favorite pet topic of mine. We were here with Dharmesh yesterday talking about someone, build an agent that reads my emails. Yeah. And you did. And I think I actually paid for the first one. You were, you were so excited in the early GPT three days. I was like, you were like, uh, I'm building the most expensive startup ever. Yeah. It's so expensive. Anyway. So the point being what I'm trying to get to is you are a very smart guy. You built email. You, you didn't like it. You pivoted away. I've seen other, like every year there's someone who is like, I will crack email. Yeah. And I'll, and then, and then they give up. Yeah. What is so hard about email? I didn't pivot away because the product or the idea was bad. I pivoted away because I was super burnt out. I did a startup for like four years. And the first thing didn't work out. Is this room service? Yeah, this is room service. So my startup before this originally started as Quirk, which was like a mental health app, but then Quirk had the same problems that basically every mental health app has, which is like your retention goes to zero if you work at in any capacity. And so switched and then said, okay, well I will do something that's closer to my actual background was like a distributed systems company called Room Service. Room Service went for about nine months and then sort of had the same problem that I think every other competitor Room Service has, which is mostly people building a house. And so then I went back to our investors at the time, which was Nat and Daniel and specifically Daniel told me that I should go stare at the ocean. And you know, I will find something else to do and just throw shit at the wall. And then I think, I think it was Gustav at YC. Maybe it was probably actually Dalton Caldwell. Dalton Caldwell, like just said, don't die. Like you can just keep doing things and don't die. And so I think I just got it in my head that you should just like keep trying things and not die. And I really, really, really did not want to die and didn't really know what to do. And so I just threw out like 40 products with the assumption that if you just keep trying things, you won't die. This is actually not the most ideal thing to do. You actually should totally just pick a thing and go with it. But my brain wasn't set on like, oh, I should do this particular thing. It was set on not die. And so I just kept going for a very long time for like four years. And by the end of it, I think I was just super burnt out. And I was going to do the email thing with one co-founder and then they quit. And then I was going to do an email thing with another co-founder. And then they fell in love and decided to go get married and you know all that. Okay. So it wasn't that email is intractable. I'm just trying to figure out like, look, is there something bad? Like, is this a graveyard of ideas, right? Everyone wants to do email and then nobody does because something. And I think it's just hard to make an email client. I think it's hard to make an email client. That is, it's a competitive space in which there are lots of things. I do think that the better version of that is something that looks closer to what Intercom is doing. And Intercom obviously existed beforehand. So you can think about like any product. Like, should you be doing it or should somebody else in the industry who already has the existing customer set do it? And I think Intercom has pretty much very successfully done like they already had the position to do it. Like, what do you actually need the AI to write your emails for? Like, most people don't need this. But what who does need this is like support use cases is pretty much there. And the people who are best able to execute on this is totally Intercom. So like props to Owen. I think that was like completely the correct move. Yeah. Closing thoughts. Call to action.

Alessio [01:09:07]: Yes. Oh, yeah, we are.

Evan [01:09:09]: We are hiring for two roles as of this recording. I don't know. Maybe this will change and we'll be hiring for different roles. So go to the website or whatever. But the first role is for traditional systems engineering. This is like low level systems or low level Linux systems. Yeah. So I'll rest most all of our code bases and rest. But we're not necessarily just looking for like rest engineers. We're specifically looking for like Linux people sort of pitches. You get to work on supercomputers. You get to work on one of the few places in supercomputers that I think has a pretty good business model and is like a like a working thing. And people generally seem to think that our vibe as of compute is very nice. The we have just an unbelievable. Excellent team. I think nowadays our CTO is Eric Park. He's the co-founder of Voltage Park, which is one of the other GPU clouds. And he is quite possibly the sweetest man I've ever met. He is extremely chill and also just extremely earnest and kind. And the rest of the team kind of feels that energy very strongly. And then the other role that we're hiring for is financial systems engineering, which I really should learn what it's not systems engineering, but we should really find a better name for this role. It's basically a fintech engineer. That it's we have the same problems as traditional fintech does. And that's like we have a ledger. We have recording requirements and all that stuff. This role is responsible for the not lose all the money. Cool. Like, we've got a whole bunch of money flowing through us. There is a bunch of stuff that you need to do in order to not lose all that money. And then the actual outcome of that work, besides not just losing all the money, which is very important, is that you end up with better prices for the vendors and better prices for the buyers. And this means that your grad student who is an engineer. Who is making the cancer cure or whatever and needs to be able to buy like 100K of compute to like scale up really big actually can do so. And that's I think the like this is part of the reason to work at SFC is that you're the things you do actually matter in a way that doesn't necessarily always at all the companies functionally. We run supercomputers like not soybeans or I don't know. It's a very cool place to work because your outcomes of what you do have real deal impact in a way that you don't always get when you're doing SaaS. Excellent pitch. I bet you've done that a lot, but it's nice to hear for the first time. I was going to say, like, you know, have you looked into Tiger Beetle, the dual entry accounting database? We have. That seems to be the thing if you want to make systems that don't lose money. Yes. Systems that don't lose money. There are lots of other things you have to do. Like you have to make things in a format that your accountants can read and then get audited and so on. It's not purely just the yeah, it's not purely just the tech. Cool. Awesome. Thank you so much. Of course.