Latent Space

We’ve been on a bit of a mini World Models series over the last quarter: from introducing the topic with Yi Tay, to exploring Marble with World Labs’ Fei-Fei Li and Justin Johnson, to previewing World Models learned from massive1 gaming datasets with General Intuition’s Pim de Witte (who has now written down their approach to World Models with Not Boring), to discussing the Cosmos World Model with with Andrew White of Edison Scientific on our new Science pod, to writing up our own theses on Adversarial World Models. Meanwhile Nvidia, Waymo and Tesla have published their own approaches, Google has released Genie 3, and Yann LeCun has raised $1B for AMI and published LeWorldModel.

Today’s guests have a radically different approach to World Modeling to every player we just mentioned — while Genie 3 is impressive, its many flaws demonstrate the issues with their approach - terrain clipping, noninteractivity (single player, no physics/no objects other than the player move), and maximum of 60 second immersion.

Moonlake AI (inspired by the Dreamworks logo) is the diametric opposite - immediately multiplayer, incredibly interactive, indefinite lifetime, capable of MANY different kinds of world models by simulating environments, predicting outcomes, and planning over long horizons. This is enabled by bootstrapping from game engines and training custom agents:

In Towards Efficient World Models, Chris Manning and Ian Goodfellow join Fan-Yun in explaining why their approach to efficiency with structure and casuality instead of just blind scaling is sorely needed:

SOTA models still show physical or spatial understanding glitches, such as solid objects floating in mid-air or moving “inside” other solid objects.

If the goal is to plan for the next action, how often is a high-resolution pixel view necessary for modeling the world? Our bet is that there is a disproportionately large share of economically valuable tasks where such detail is not required. After all, humans with a wide variety of sensory limitations have little difficulty doing almost everything in the world. Furthermore, for a large number of purposes, describing a scene or a situation in a few words of language (“the car’s tires squealed as it cornered sharply”) is sufficient for understanding and planning.

Experiments also show that humans only partially process visual input in a top-down, task-directed way, often making use of abstracted object-level modeling. In almost all cases, partial representations combined with semantic understanding are sufficient.

…

If the goal is to facilitate the understanding of causality in multimodal environments, then the world model—whether it is used in the virtual world or the physical world—must prioritize properties such as spatial and physical state consistency maintained over long time periods, and an ability to evolve the world that accurately reflects the consequences of actions. That’s what Moonlake is building.

Game engines are the right starting point abstraction to efficiently extract causal relationships, and building the interfaces and community (including their new $30,000 Creator Cup) to kickstart the flywheel of actions-to-observations.

We were fortunate enough to attend their sessions at GDC 2026 (the Mecca of Game Devs), and were impressed by the huge variety and flexibility of the worlds people were building with Moonlake’s tools already! Live videos on the pod.

00:00 Benchmarking Gets Hard
00:47 Meet Moonlake Founders
01:26 Why Build World Models
03:12 Structure Not Just Scale
05:37 Defining Action Conditioned Worlds
07:32 Abstraction Versus Bitter Lesson
14:39 Language Versus JEPA Debate
20:27 Reasoning Traces And Rendering Layer
37:00 Gameplay Over Graphics
38:02 Fiction Rules And World Tweaks
39:15 Code Engines Beat Learned Priors
41:10 Diffusion Scaling Limits
43:23 Symbolic Versus Diffusion Boundary
46:14 Platform Vision Beyond Games
50:24 Spatial Audio And Multimodal Latents
54:23 NLP Roots Hiring And Moon Lake Name

[00:00:00] Chris Manning: Think this whole space is extremely difficult as things are emerging now. And I mean, it’s not only for world models, I think it’s for everything including text-based models, right? ‘cause in the early days it seemed very easy to have good benchmarks ‘cause we could do things like question answering benchmarks.

[00:00:20] But these days so much of what people are wanting to do is nothing like that, right? You’re wanting to get some recommendations about which backpack would be best for you for your trip in Europe next month. It’s not so easy to come up with a benchmark, and it’s the same problem with these world models.

[00:00:41] swyx: Okay. We’re back in the studio with Moon Lake’s, two leads. I, I guess there’s other founders as well, but, sun and Chris Manning. Welcome to the studio.

[00:00:54] Fan-yun Sun: Thanks. Thanks, Chris. Thanks for having us.

[00:00:56] swyx: You’ve got, you guys have, come burst onto the scene with a really refreshing [00:01:00] new take of mold models.

[00:01:01] I would just want to, I guess ask how you, the two of you came together. Chris, you’re a legend in NLP and just AI in, in, in general. You’re, you’re his grad student, I guess

[00:01:10] Fan-yun Sun: Actually my co-founder.

[00:01:11] swyx: Oh, yeah.

[00:01:12] Fan-yun Sun: I should give a lot of credit to my co-founder, Sharon. Yeah. She was, she was actually working with Professor Fe Androgyn and then she ended up working with, Ron and Chris Manning here.

[00:01:22] And then, so I got connected through to Chris initially, actually through my co-founder,

[00:01:26] swyx: what is Moon Lake? What, what is, actually, I’m also very curious about the name, but like why going into world models?

[00:01:33] Fan-yun Sun: So I was working a lot. With actually Nvidia research during my PhD years on essentially generating interactive worlds to train reinforcement learning agents or embody EA agents.

[00:01:44] And then there’s two observations. One in academia and one in industry. An industry like folks at Nvidia are actually paying a lot of dollars to purchase these types of interactive worlds, whether it’s for the sake of evaluation or training the robots, or policies or models. And [00:02:00] then, in academia, same thing is happening.

[00:02:02] And more specifically, when I was actually working with Nvidia on the synthetic data foundation model training project, we were actually generating a lot of these synthetic data and showing that, hey, you can actually, these synthetic data are actually as useful as real world data when it comes to multimodal pre-training.

[00:02:16] But then, like I said, there’s a lot of dollars being paid out to like external vendors or, or like. Other folks to manually curate these types of data. It was very clear to us that, okay, on our way to, let’s call it embody general intelligence models need to learn the consequences behind their actions, which means that they need interactive data and the demand for those types of data are growing exponentially.

[00:02:38] But everybody’s sort of thinking about it from a pure, say, video generation perspective or something else. But we feel like the true actually opportunity is actually building reasoning models that can do these things, like how humans do these things today. So that’s a little bit on the genesis of Moon Lake, and I think the reason I got into world models was partly.

[00:02:59] A philosophical [00:03:00] take of the on the world where I like, believe the simulation theory and stuff like that. But on the other, on the other hand, it’s really just like, oh, like there’s an opportunity there that I feel like nobody’s doing it the way I think should be done.

[00:03:10] Chris Manning: I can say a little bit about that.

[00:03:12] Yeah. So of the overall goal is the pursuit of artificial intelligence and most of my career has been doing that in the language space and that’s been just extremely productive. As we all know, the story of the last few years, I don’t have to tell about how much we’ve achieved with large language models, but, uh.

[00:03:31] Although they have been extremely effective for ramping language and general intelligence, it’s clearly not the whole world. There’s this multimodal world of vision, sound, taste that you’d like to be dealing with more than just, language. And then the question is how to do it. And despite, a huge investment in the computer vision space, right, as the research field computer [00:04:00] vision has been for decades, far, far larger than the language space, actually.

[00:04:05] I think it’s fair. Say that, vision, understanding sort of stalled out, right? You got to object recognition and then progress just wasn’t being made right? If you look at any of these, vision language models, it’s the language that’s doing 90% of the work and the vision barely works. And so there’s really an interesting research question as to why that is and at heart, the ideas behind Moon Lake are an attempt to answer that, believing that there can be a really rich connection between a more symbolic layer of abstracted understanding of visual domains, which aren’t in the mainstream vision models, which are still trying to operate on the surface level of pixels.

[00:04:50] swyx: I think one of your blog posts, you put it as structure, not scale. Is that, a general thesis?

[00:04:57] Chris Manning: Yeah. Well, scale is good too.

[00:04:58] swyx: Yeah. Scale is good. Too

[00:04:59] lot,

[00:04:59] Chris Manning: [00:05:00] lots of data is good as well and scale, but nevertheless, you want the structure Yeah. To be able to much more efficiently learn.

[00:05:07] swyx: Yeah. The other thing I really liked also is you put out an example of what your kind of reasoning traces look like.

[00:05:12] Right. Which you would distill is the word that comes to mind. I don’t even think that’s a good, good description, but it would involve, for example, geometry, physics, affordances, symbolic logic, perceptual mappings, and what, what have you. But like that, that is the kind of example that involves, let’s call it spatial reasoning, role model reasoning as as compared to normal LM reasoning.

[00:05:35] Yeah.

[00:05:36] Vibhu: But also like taking it a step back. So how do you guys define world models? A lot of people see okay, you can do diffusion, you can do video generation. But, you guys put out quite a few blog posts. You put out a essay recently, we can even pull it up about efficient world models. You have a pretty like structural definition here, but for the general audience that don’t super follow the space, right.

[00:05:55] What’s, what’s the difference in what we see from like a video generation model to [00:06:00] a world gen A simulator? How do you kind of paint that last

[00:06:02] Chris Manning: year? Yeah, so I think this is actually a little bit subtle because, people look at these amazing generative AI video models, SAWA VO three, one of these things, and they think Genie, they think, oh, this is amazing.

[00:06:17] This is we’ve solved understanding the world because you can produce these generative AI videos, but. The reality is that although the visuals do look fantastic, those visuals actually are accompanied by an understanding of the 3D world, understanding how objects can move, what the consequences of different actions are, and that’s what’s really needed for spatial intelligence.

[00:06:49] So I mean, a term we sometimes use is that you need action condition, world models. That you only actually have a world model if you can predict, [00:07:00] given some action is taken, what is going to change in the world because of it. And in particular, that becomes hard over longer time scales. So if you’re simply, trying to.

[00:07:12] Predict the next video frame. That’s not so difficult. But what you actually want to do is understand the consequences, likely consequences of actions minutes into the future. And to do that, you actually much more of an abstracted semantic model of the world.

[00:07:32] swyx: Yeah, the question comes where you want to have more structure than is available in just predicting the next token.

[00:07:41] And typically, well, let’s, let’s call it the experience of the last five years has been that is just washed away by scale, right? So what is the right middle ground here that, you don’t ignore the bitter lesson, but also you. Can be more efficient than what we’re doing today.

[00:07:57] Chris Manning: One possibility [00:08:00] is, look, if we just collect masses and masses and masses and masses of video data, this problem will be solved.

[00:08:11] Under certain assumptions that could be true, but there are sort of multiple avenues in which it could not be true. The first is what’s really essential is understanding the, the consequences of actions producing an action conditioned world model. And if you are simply, collecting observational video data, which is the easy stuff to collect, when you’re sort of mining online videos, you don’t actually.

[00:08:41] Know the actions that are being taken to see how the video is changing. And so if you are never collecting directly actions and you are having to try and infer them from what happened in the observed video, that’s not impossible. But it’s very [00:09:00] hard and it’s not really established that you can get that to work at any scale yet.

[00:09:05] And so there’s a lot of premium on collecting action condition video data, which is part of why there’s been a lot of interest in using simulation so that you can be collecting data where you do know the actions, which isn’t quite limited supply, but there’s also in the limit of as much data as you could possibly have.

[00:09:28] Maybe the problem is eventually solvable, but. Even though we collect huge amounts of text data is always at a great level of abstraction, right? Language is a human designed, abstracted representation where there’s meaning in each token and it’s representing and abstraction of the world, right?

[00:09:51] As soon as you are describing someone as a professor, and as soon as you are saying that they’re condescending, right? These are very [00:10:00] abstracted descriptions of the world. It’s not at what you’re observing as pixel level, and to get to that kind of degree of abstraction, starting from pixels is orders and magnitude of extra data and processing.

[00:10:14] And so, although, we absolutely want to exploit, get as much data as possible, use the bitter lesson. Nevertheless, if there are ways in which you can work with five orders of magnitude less data than people working purely from pixels, you’re gonna be able to make a lot more progress, a lot more quickly.

[00:10:34] And that’s the bet here. And so you could just say that’s only wanting to be able to, do it more efficiently, do it more quickly, do it more cheaply. But I think it’s actually more than that, I think. One should be making the analogy to how human beings work at one level. You know? Yes, we have these high [00:11:00] resolution eyes and we can look and see a scene like a video, but all of the evidence from neuroscience and psychology is that most of what comes into people’s eyes is never processed.

[00:11:13] Right. That you are doing fairly fine ated processing of exactly what you’re focusing on. But as soon as it’s away from that of yeah, there’s another guy over there that you’ve sort of only processing top down this very abstracted semantic description of the world around you. And so, that’s what human beings are doing.

[00:11:33] They’re working with semantic abstractions and so. I think it is just the right representation. ‘cause we also have other goals we want to be able to do, real time worlds. So that means there’s a limit to how much processing you can do and we want to do long-term planning and consistency. And again, that favors abstraction.

[00:11:55] I mean, I guess there was actually a recent. Blog posts that [00:12:00] came out from our Friends of physical intelligence and, they were sort of heading in the same direction they were saying Oh, to the pay

[00:12:06] swyx: pay model.

[00:12:07] Chris Manning: Yeah. Yeah. To maintain a long term memory of what’s happening in the world. So we can, do longer term we actually storing text of what is, been happening in the world.

[00:12:19] Right. It is not such a successful strategy of trying to keep it all at a pixel level.

[00:12:24] Vibhu: And yeah, I mean, you can see it in video models like that Temporal consistency. We’re at a scale of train on, all the video data we have. We have it for maybe 30 seconds, a few minutes. That’s not the same as a game state played for half an hour.

[00:12:37] Right. I thought you guys break it down pretty well. You have a, you have a blog post about. Building multimodal worlds with an agent. I dunno if you guys wanna talk about this. This is one of the things I read, I

[00:12:48] swyx: thought, yeah, it’s the thing I talked about with the reasoning chain. Yeah.

[00:12:51] Vibhu: So there’s like different phases to this.

[00:12:53] It seems like it’s more of an agent, a scaffold, very different approach than just, type in a prompt and you, you don’t have the same consistency. [00:13:00] It also, like, for people that are listening, I, I would highly recommend reading it. It breaks down the problem in a different light, right?

[00:13:06] So like, what do you need to consider when you’re talking about video, like world game models, right? How would, what do you need to consider? What are the factors? What are the elements? What’s the state? So I don’t know if you guys have stuff to talk about for this one.

[00:13:19] Fan-yun Sun: Yeah. Actually, I wanted to add on a little bit Yeah.

[00:13:22] On our previous point, which is just like, change topics so quickly. I, I do feel like sometimes people confuse like, oh, like we’re taking an an, an method with abstraction. That means they don’t believe in bitter lesson. Like that’s just false, right? Like we are believed is a bitter lesson. But then I feel like the question that we always discuss is like, what is the right abstraction level today?

[00:13:42] The analogy I like to make is like, let’s just say we can encode and decode. Represent all of images, videos, audio and bytes. Then the most bitter lesson approached is to train a next byte prediction model as opposed to the next token prediction model where it’s just like, okay, it’s natively multimodal, can just, but it’s like, yeah, like [00:14:00] to, to Chris’s point, it’s like the scale and computing you need to achieve that.

[00:14:03] So that’s why we always come back to like, okay, what is the most efficient way to do it? And reasoning models to the point of this blog post is a showcase of like, Hey, we’re actually just like reasoning about the world and reasoning about. The aspects of the world that CAGR that matter for me to learn what I want to learn from this role model.

[00:14:21] swyx: Yeah, it’s like you’re improving the en encoder of whatever you’re, trying to model. And like a better representation would just represent the important things in less space. Yeah. Which would just be more efficient.

[00:14:33] Fan-yun Sun: Yeah.

[00:14:34] swyx: So yeah, I, I, I fully agree that it is not, antagonistic to, bitter lesson.

[00:14:38] I do wanna wanna mention one more thing. Is there any philosophical differences with the JPA stuff that, Yun is working on? I gotta go there. You, you, you, you’re, you’re imagining like some latent abstraction. I’m like, okay, fine. Let’s, let’s talk about it, right? Like it’s an elephant in the room.

[00:14:52] Chris Manning: Yeah.

[00:14:53] Chris Manning: There are philosophical differences. Jan Lacoon is a dear friend of mine, but. [00:15:00] He has never appreciated the power of language in particular, or symbolic representations in general. Yarn is a very visual thinker. He always wants to claim that he thinks visually and there are no words, symbols, or math in his head.

[00:15:21] Maybe that’s true of yarn. It’s certainly not the way I think. Um. But at any rate, the world according to yarn is the basic stuff of the, the world and of intelligence is visual and language is just. This low bit rate communication mechanism between humans and it doesn’t have much other utility and it’s far inferior to the high bit rate video, that comes into your eyes.

[00:15:53] And I think he’s fundamentally missing a number of important things [00:16:00] there. Think of this evolutionary argument looking at animals, right? That the closest analogies, the things with chimps, right? So chimpanzees, have fairly similar brains to human beings. They have great vision systems, they have great memory systems.

[00:16:18] They’ve got, better memory than we do of short term memories. They can plan, they can build primitive tools that, humans. Massively ahead in what we understand about the world, what we can plan, what we can build. And essentially what took off for us was that humans managed to develop language and that gave a symbolic knowledge, representation, and reasoning level, which just, okay if this sort of vaulting of what could be done with the intelligence in brains.

[00:16:59] So the [00:17:00] philosopher Dan de refers to language as a cognitive tool and argues that, humans unique among the creatures in the world have managed to build their own cognitive tools and language is the famous first example. But other things like, mathematics and programming languages are also cognitive tools.

[00:17:21] They give you an ability to. Think in abstractions, in extended causal reasoning chains. And that allows you to do much more. And we use that for spatial representation and intelligence and planning and gameplay as well. So we believe, and this is, underlying the specific technologies that Moon Lake is making, that symbolic representations are powerful.

[00:17:50] And you want to use that in your understanding of the visual world when you want a causal understanding, when you want to maintain long-term [00:18:00] consistency and prediction. And as I understand it, that’s just not in ya Koon’s worldview. So I think that’s the fundamental philosophical difference. Then there’s the specific model.

[00:18:11] He’s been advancing jpa, that’s a reasonable. Research bed is a direction as to, to head for building out a model of the visual world. To my mind, it’s sort of one reasonable research bed. It’s not really established. It’s the best one that everyone should be following,

[00:18:32] swyx: at least developed at scale, at Meta.

[00:18:34] But it’s not just vision, right? Like, I mean, JPA is a, just joint admitting prediction can be applied to anything really. And people have done it. The argument is that there is a latent representation or that is probably more. Suited to the task, then why not let machines do it for us instead of predefining it at all?

[00:18:50] And isn’t something like a JPA shaped thing the right answer? And if not, why not?

[00:18:55] Chris Manning: So I think there’s a part of jpa that’s right, which is [00:19:00] you do want to have a joint. Embedding that gives you a consistent model of the world. And Jan’s argument is you can never get that from auto aggressive language models ‘cause they’re sort of left to right churning out one token at a time.

[00:19:22] I guess this is where we’re the research arguments of the field, I’m not actually convinced that’s right. ‘cause although the token production is this auto aggressive, process that’s heading, left to right, I guess don’t have to be left to right. But anyway, in sequence of tokens we could have right to left Arabic.

[00:19:40] But although that’s true, all of the weights of the model that are internal to the transformer, they are a joint model of the model’s understanding of the world. And so I think you can think of the weights of the model as a form of. Joint representation, [00:20:00] and therefore it is plausible to think that could be the basis of a world model, which avoids, ya’s objections.

[00:20:10] swyx: I think I follow, and obviously that would touch on what Moon Lake eventually ends up doing as well. Right. Like, which it’s hard to tell because you put out the end results, but we don’t know the inputs that go into it. So it’s, it’s, that’s something that we have to figure out over time.

[00:20:25] Vibhu: Yeah. I mean, I guess this kind of breaks down some of the outputs. Do you wanna walk us through it?

[00:20:31] Fan-yun Sun: Yeah. So this, this really just walks us through the reasoning traces of like, okay. So that just say, if we wanna build a world in this context, it’s really just a game demo that, that shows the, the variety of interactions that this world model can build.

[00:20:45] And yeah, it’s really just a reasoning traces of like, okay it prompted to create a bowling game. Like how did it achieve what you saw? That level of causality, interaction and consistency, right? So yeah, this is almost just like a, an example of [00:21:00] like a reasoning traces. Very

[00:21:01] swyx: detailed.

[00:21:01] Fan-yun Sun: Yeah.

[00:21:01] Vibhu: Very, very detailed.

[00:21:02] You gotta you don’t even realize it, right? Like when a video is generated, what happens when a ball strikes a pin, right? So first, like you, there’s audio in that, like audio triggers happens, score increments, the world changes. Like pins have to start dropping. There’s a timer that goes on. It’s just like very similar to how now we’re used to reasoning for language models.

[00:21:20] There’s a whole state of what happens. So geometry, physics, all this stuff. And then yeah, there’s kind of that single prompt. So asset, ation all this stuff. It’s like a, it’s a nice view to see what’s going on.

[00:21:32] swyx: I think Sun is also too polite to point out that, both like Google’s genie, demos as well as world Labs is marble, do not have interactive worlds.

[00:21:41] Fan-yun Sun: That’s the benefit of having a reasoning model, right? Like, because you can, you can say, oh, like maybe in this particular context, I want to learn how to bowl. And then you can say, okay, then what is it important when it comes to learning how to bowl? Okay, maybe it’s like I need to understand the, the basic of like, physics and I want to throw it over [00:22:00] them.

[00:22:00] I wanna know that when I, when it resets it’s a new game. So I know that yeah, basically, you know to pick up the ball, you know that ball’s gonna cause the pins to fall down. You know that what’s important to this particular bowling game is to score and you know that the score corresponds to the number of pins that fell down.

[00:22:19] So it’s just like, if it’s a model that sort of knows what it. Looks like, knows what a bowling game looks like, but doesn’t actually allows you to practice over and over again and to understand that, oh, like what it takes to actually get a high score. Then it sort of doesn’t actually allow you to learn what you set out to learn within the world model.

[00:22:38] And I think this is really just one example of showing like the advantages of the approach that we’re taking over most the, let’s call it the zeitgeist, is today, when people talk about clinical role models,

[00:22:51] Chris Manning: right? So it sort of seems like the question to ask when there’s a world model is.

[00:22:58] Can I not [00:23:00] only just wander around the world and look at the beautiful graphics, can I interact with the objects in the world and see the right consequences of actions?

[00:23:11] Vibhu: And you also understand what the consequences would be if you do something right. So it’s not just like, okay, there’s one thing if I pick it up, something will happen.

[00:23:19] But, there’s 50 options and I know I can expect, I can infer what would happen if I do any of them. Right. So very different when you can actually see it play around with it.

[00:23:28] swyx: There,

[00:23:31] swyx: there’s two cheeky elements of that. I mean, the, the, the I guess, less ambitious one is, let’s really establish for listeners, why is this fundamentally different than writing Unity code, right?

[00:23:40] Like just creating a model to translate a prompt into Unity code

[00:23:44] Fan-yun Sun: so there is an underlying physics engine. Yeah. In that sense, there’s some overlapping things to Unity, but the way we think about it is like physics engine. Tools or code are cognitive tools like borrowing Chris’s term, right? Like tools [00:24:00] that the model can employ as means to an end.

[00:24:04] So today maybe you say, okay, in this particular context we care about physics, we care about the long-term causality consequences. Then yes, we deploy it, employ physics engine, and then maybe tomorrow we say, okay, we’re we’re training that. Just say drones where we only care about really fluid dynamics and the visual aspect of the world.

[00:24:25] Then, then yeah, maybe we don’t actually, the model actually doesn’t have to use a physics engine. Or maybe it employs other types of representation or physics engine to achieve the task. So yes, writing code for Unity is sort of similar to a tool that our A model can employ, but our goal is for a model to take a representation conditioned reasoning.

[00:24:46] Approach or process.

[00:24:47] swyx: Yeah,

[00:24:47] Fan-yun Sun: internally.

[00:24:48] swyx: Yeah. Using these things as just like general two calls. Right. Which I think is very interesting. The other more ambitious one is, some kind of recursive element where it becomes multiplayer, right? Like here, there’s a single player element, you’re not [00:25:00] modeling any other people involved.

[00:25:01] And that is a whole other thing.

[00:25:04] Fan-yun Sun: But in fact, we can really do multiplayers. Oh yeah, okay. I haven’t seen any double situations. So just actually just like prompt our, our model to say, Hey, like configure to multiplayer. Then it’ll do like this. You’ll be able to configure multiplayer

[00:25:16] swyx: great

[00:25:17] Fan-yun Sun: persistency database for you.

[00:25:18] Easy. Yeah.

[00:25:19] Vibhu: So what, what are like some of the current limitations in where we’re at? So there’s one approach of like, okay, scale up video predictors. Obviously there’s data issues. With approaches like this, is it data constraints? What are like the next steps? Is it real time? Like, so there’s one side of, write an agent to write Unity code, but okay, I want to be streaming a game real time.

[00:25:38] I want to have characters being also like agent, but where, where do we kinda see this scaling up? Right?

[00:25:44] Fan-yun Sun: Yeah, there’s definitely a data constraint. Like the more data, the, the better. This reasoning model can almost basically act as humans to like operate a variety of tools and softwares to build whatever’s necessary.

[00:25:57] And then there’s a sort [00:26:00] of fidelity constraint, which we’re actually solving with another model, which we can talk about later. But it’s like, it’s not as easy to get to photorealism with the approach that we’re taking. But we think there are better solutions to that, which is we can dive into later.

[00:26:14] Later.

[00:26:15] Vibhu: The one one thing you note here is it’s a diffusion model, right? So there’s, there’s a few approaches, diffusion caution, splatting, yeah, so Ry diffusion model, you guys wanna

[00:26:25] Fan-yun Sun: Yeah.

[00:26:25] Vibhu: Introduce,

[00:26:26] Fan-yun Sun: yeah, totally.

[00:26:26] Fan-yun Sun: So within our world modeling framework, we think there are two models that we train, right?

[00:26:31] Like, there’s the multimodal reasoning model that we just talked about that essentially handles. Mainly the, the causality, the persistency and logic determinism of the world. And then RY is our bet on saying, okay, like while all those model, can take care of all these things that we just talked about, it’s limitations compared to existing, say, video models, is that it doesn’t have as high of a pixel [00:27:00] ality right off the gate, right?

[00:27:02] And EE is to say, Hey, we can actually take whatever persistent representation that we generate with our multimodal reasoning model and learn to restyle it into photo photorealistic styles or arbitrary styles you want. So this model is almost to say, Hey, I’m going to respect the persistency and interactivity of the world that you created, but my only job is to make sure that its pixel distribution is close to what we want.

[00:27:29] Vibhu: Yeah.

[00:27:30] swyx: Great example right there. You kept the KL divergence.

[00:27:33] Fan-yun Sun: Oh. Where,

[00:27:34] swyx: no, no. I mean this, this is a, a classic like, how you don’t stray too far from the source material as you, you kept the kl, which is Oh yeah. Kind of cool. Yeah.

[00:27:43] Fan-yun Sun: Yeah.

[00:27:44] swyx: I mean, and the

[00:27:44] Chris Manning: difference is, and I mean sun was pointing at this, where sort of saying it’s in one way a more difficult path, but a better path that, typically the diffusion models are producing the whole scene and it looks lovely, [00:28:00] but there isn’t spatial understanding behind it, which is allowing for the real time graphics gameplay, the spatial intelligence, understanding the consequences of worlds where this is, taking a path where it is assuming an abstracted semantic model of the world’s state.

[00:28:20] And then the diffusion model is then being used on top of that to produce the high quality graphics.

[00:28:27] swyx: Is there an intended practical, or business use for this, or is it like a, like a demonstration of capabilities?

[00:28:34] Fan-yun Sun: We actually believe that this is gonna be the next paradigm of rendering. So it’s gonna replace how ra raizer, it’s gonna replace DLSS today because it not only has these pixel prior that’s learned from the world such that you can literally play any game in photo realistic styles, which is a lot of people’s desire when they do GTA, right?

[00:28:51] Like,

[00:28:51] Vibhu: all the mods, all the people adding perfect lighting and all this.

[00:28:54] swyx: So

[00:28:54] Fan-yun Sun: skins

[00:28:55] swyx: for worlds, let’s call it

[00:28:56] Fan-yun Sun: skins, let’s call it skin for worlds. I,

[00:28:58] Vibhu: it’s also like, you can call it skin, you can call it [00:29:00] customization. You can play it how you want, right?

[00:29:01] Fan-yun Sun: Yeah, exactly. And I think another thing that we really pointed out specific specifically in this blog is the programmability of it, right?

[00:29:09] So what this means is that this render historically render is always a derivative of the game state, right? You’re saying, oh, here’s the game state, I’m rendering out a frame. But here I’m saying actually this render can be part of the gameplay loop. I can say something along the lines of, if upon getting 10.

[00:29:26] Apples, I’m gonna, my weapon of choice, my bullet’s gonna turn into apples. And that’s, that’s possible because we can say, we can basically dynamically have certain game state trigger the, the preconditions to the render such that the rendering is now part of the game loop too. One thing is to just say, okay, it’s, it’s, it’s the appearance.

[00:29:47] But the second thing is also to say there’s these novel interactions that are possible because this render now has actually priors of the world.

[00:29:57] swyx: It is up to the artist to figure out what to do with it.

[00:29:59] Fan-yun Sun: It [00:30:00] is up to the creators. Yes.

[00:30:01] swyx: Yeah.

[00:30:01] Fan-yun Sun: And I also think that’s actually another big argument that we’re making and the reason that we’re picking, taking the bet we’re baking is that a lot of the times, whether it’s for embody AI gaming, like you want a layer where human can inject their intentions.

[00:30:15] So, for example, let’s just say in the context of gaming, it’s obviously like my creative intent, but maybe in the context of embodied ai, it’s like, oh, like I take this foundational policy and I want to actually fine tune it to deploy in my house. So you want to almost say, inject, have a layer where human can say, oh, here’s the distribution of things I want to create to achieve my goal.

[00:30:35] And I think 3D graphics as it as it is today, is basic, the layer for people to say, Hey, what do I care about in this world? And it allows, basically human intent to be expressed in these worlds much more explicitly and distributionally as opposed to just saying, Hey, I’m gonna generate like, arbitrary.

[00:30:54] And it’s like just prompts,

[00:30:55] swyx: it’s one of those things where like, I think you, you’re going to build up a series of models, right? [00:31:00] This is just one of, this is probably like the highest utility or heaviest, frequency one, I don’t dunno what to call this. Where like you Yeah. You can immediately drop this in on any game and you don’t need anything else that.

[00:31:10] That you guys do. But, I, I could see, I could see that I think the, the human intent is something that people are not even used to because we’re so used to static worlds or, worlds that just don’t react, or, I don’t know. It’s, it, you’re kind of blowing my mind right now with like, I’m, I wonder if you’ve talked to people at GDC Hmm.

[00:31:27] And what are they gonna do with it?

[00:31:30] Fan-yun Sun: Yeah. Now the stance that we take on this front is like, we’re not gonna be more creative than our users to ship

[00:31:35] swyx: it out.

[00:31:35] Fan-yun Sun: Yeah. But we wanna make sure that we’re building things in a way that really allows them to express their intent.

[00:31:41] swyx: The thing that you said about, here’s the distribution that I want.

[00:31:45] I think text may be too low of a bandwidth to. To really demonstrate, because I, I, there, I’m, I’m probably just gonna want to drop in a bunch of, reference assets and then you can figure it out from

[00:31:58] Vibhu: there. But you probably wanna do a, a mixture of [00:32:00] both, right? Like you throw in a few images. I wanted this style.

[00:32:02] Yeah. I want it to look like this. So it, it’s, it’s a mixture, right?

[00:32:05] Chris Manning: I, I think it’s a mixture. I mean, yeah, I mean there’s clearly a visual component of this, and it’s not that, everything can be text. ‘cause of course you want to give a visual look, but there’s also a massive amount of giving the overall picture of the look of the world and the behavior of things that you can express in a few words of text.

[00:32:32] And it be very time consuming and difficult to do via visual means. So I think, yeah, you want a combination of both.

[00:32:40] Vibhu: So one question I kind of have is, how do we go about evaluating world models? So like, there’s many axes, right? One is like, okay. I have preferences. How well do we adhere to prompts? One is the simulation.

[00:32:50] One is like do things, is there core logic that’s broken? So coming from we know how to evaluate diffusion, there’s fidelity, there’s [00:33:00] stuff like that. But what are some of the challenges that most people probably aren’t thinking about?

[00:33:04] Fan-yun Sun: Yeah, I think this is like a great question and probably one of the hardest questions in role models because like, I think it always comes back to what are you building this role model for?

[00:33:13] And depending on your end goal and purpose, the evaluation should defer. So in the context of games, then the most direct way of measuring is how much behind are people actually spending in this world that you create? And if your goal is to say, for example, in the context that we just talked about, like, hey, deploying, deploying action in body, a agent, then your, your end.

[00:33:33] Metric is then, okay, after training in these worlds that you generate how robust it is to when you actually deploy to the target environment. But then, it’s, it’s hard to measure these end metrics. So today people have like these proxy metrics that I call that basically try to measure what we really care about, which is the end metrics, but then frankly it’s different for every use case.

[00:33:57] Yeah,

[00:33:57] Vibhu: which seems like quite a challenge, right? Like in [00:34:00] in language models or video models. Image models, your benchmarks are proxies, right? People aren’t actually asking instruction, following tool use questions. They’re proxies of how well it will do downstream. But for this, so like, should teams, should companies have their own individual benchmarks outside of games?

[00:34:16] If you think of stuff like, okay, video production, movies, stuff like that, that also want to use world models. Should, should they sort of internalize like. Their own proxy. Is this something you guys do? Where, where does that connect

[00:34:28] Chris Manning: go? Yeah, I think this whole space is extremely difficult as things are emerging now.

[00:34:35] And I mean, it’s not only for world models, I think it’s for everything including text-based models, right? ‘cause in the early days it seemed very easy to have good benchmarks ‘cause we could do things like question answering benchmarks and could you answer the question based on these documents and the various other kinds of, do pieces of logical reasoning or math.

[00:34:58] But again, these are sort of. [00:35:00] And there were sort of visual equivalents of things like object recognition, right? For these small component tasks. These days so much of what people are wanting to do also with language models is nothing like that, right? You’re wanting to, have an interaction with the language model and get some recommendations about which backpack would be best for you for your trip in Europe next month.

[00:35:25] And it’s not the same kind of thing, right? And it’s not so easy to come up with a benchmark as to does this large language model give you an effective interaction for guiding you in a good way for shopping, right? So, and it’s the same problem with these world models. So if we take the game design case, well success is that a game designer can.

[00:35:57] Produce what they are [00:36:00] imagining in a reasonable amount of time. And that’s really the kind of macro task. That’s a very hard thing to turn into a benchmark and I think a lot of this is actually going to turn into people walking, walking with their feet. Right? I mean, I guess that’s what’s happening, at the large language model level, right?

[00:36:23] When people are choosing to use, GPT five or Gemini or clawed, individuals are trying out these different models and deciding, oh, I like the kind of answers that GT five gives me, or no, I feel like I get more accurate detail from Claude, right?

[00:36:43] Vibhu: It’s a lot of

[00:36:43] Chris Manning: vitech, a lot of people just using it.

[00:36:45] It’s vibe checking. I realize that, but it’s actually whether. People feel it’s giving them utility in what they want. Right.

[00:36:52] Vibhu: And the the interesting thing there is like a lot of people prefer the visual, right? This looks pretty, which is not the objective of what this is [00:37:00] for, right? It’s if a, if a game designer is working on something, they care about the game engine, right?

[00:37:04] The state, it’s, it can look whatever. You can fix that up later. Or you can have a really good game state and you can quickly edit it to 20. 20 different versions, like Keep State,

[00:37:14] Chris Manning: right?

[00:37:14] Vibhu: So

[00:37:14] Chris Manning: that’s a really important distinction, for and for speaking to Moon Lake strength, right? So, yeah, great visuals are lovely to look at for a few seconds, but gains are really all about the concept, the game play.

[00:37:33] And a lot of the time that doesn’t actually even require great visuals. I mean, there are just lots of very successful games which have relatively primitive visuals, and there are other games where people have spent millions producing photo realistic, visuals, and the game sucks, right? So, keeping those two axes apart is really important in thinking about what’s important in a [00:38:00] world model for different uses.

[00:38:02] swyx: This conversation is reminding me of some game review and fiction discussions I’ve, had in my sort of non-AI related life. Some, for some people might know Brandon Sanderson, who’s a very famous, fiction author, had, is is a big game reviewer. And he, he’s a big fan of video games where you change one thing about a normal what you might assume about, about the world.

[00:38:22] For example, Baba is you, I don’t know if you might have come across that, where like the rules change as you play the game. And also like where, you can do things like reverse time selectively or like change gravity selectively. And I think this is also reminds, reminds me of other kinds of world models that are created by authors.

[00:38:38] Where Ted Chang is, is my typical example where he’ll take the world that, you know today, but change one thing about it and, but then create a consistent world based on that. Which is long-winded answer of me to, of. For me to say is it’s it easy to create alternative roles that don’t exist, but you change one thing and then let’s, let’s run a whole bunch of people through it to see if it works.

[00:38:58] Chris Manning: My first dance will [00:39:00] be, that seems a lot easier and more conceivable to do using Techn technology like Moon Lakes than with some of the other world models out there, where the sun can actually make it happen. I’ll let him give a second answer.

[00:39:15] swyx: If I guess for you, you’re constrained by the game engine tool, right?

[00:39:18] Like at the end of the day, that’s the, that’s the thought, partner that you have. If I ask for something where like, if it never is allowed to reverse time or if gravity only ever works one way, then well that’s it. But sometimes gravity might change,

[00:39:33] Fan-yun Sun: but it’s a lot easier to change with code as opposed to a model that is learned primarily on data of.

[00:39:42] Real world and virtual worlds that are, I guess, like for example, junior, like there’s actually trained on a lot of real world data and a lot of virtual gaming data, and it’s hard to say maybe it’s easier to say, okay, I wanna change the visuals in like the time period of, of the world. Like, you can’t change gravity, for [00:40:00] example.

[00:40:00] Vibhu: I feel like you can to light bounds, right? Everything comes down to like, code is a better way to execute it, but the models aren’t that diverse and creative, right? You can say, okay, make gravity slower. It can do that, but it’s limited to your representation of how you text it out, right? Like they’re, they’re only gonna do a few iterations, whereas programmatically, if there’s a game engine under the hood, you can kind of go wild, right?

[00:40:22] So one of the, I dunno, one of the limitations of most models is that they’re very overtrained to one style. Right. And extracting diversity is pretty difficult. At least that’s something we’ve seen.

[00:40:35] Fan-yun Sun: I mean, are there examples you have in mind where you Existing models? Yeah. Like it would be easier to do that’s not using code.

[00:40:43] Certain types of creative intent or like transition state transitions,

[00:40:47] swyx: Clipping, other models, other wo models are very good at clipping through things. Clipping my, my, my legs clipping through a rock because it’s, it’s just, it’s just bad. [00:41:00] Like, you would have to struggle very hard with your stuff to actually make that happen.

[00:41:04] Which I think is maybe a topic that you actually prepared on, Gian Splatting versus, the other stuff.

[00:41:09] Vibhu: Yeah. Yeah. It’s just for those not super familiar, right? There’s a, there’s gian splatting, there is diffusion. Like what works, what scales up. I feel like in February when Soro one came out the blog post was literally titled like,

[00:41:21] swyx: you bring it up.

[00:41:22] You never know.

[00:41:23] Vibhu: World, world, video generation models are world simulators. It’s super bitter lesson pilled. Yeah, emer, a lot of it is emergence, right? So, not to go through their blog post, basically their whole thing was as you scale up all this consistency, all this stuff just kind of solves, it’s a very simple premise, right?

[00:41:41] They just scaled up, diffusion, and from there, this is, this is Feb 2024, how much can we, it’s already been two years, which is basically five years. How much more in AI time do we need to just scale up or, or do we hit a data cap? But I think we already talked about this a lot, right? Like this is back to the beginning discussion of what’s [00:42:00] appropriate for the time.

[00:42:01] And that seems like your approach, right?

[00:42:03] Fan-yun Sun: Yeah. The point I’m trying to make is that they’re very many, many different types of world simulators and like having a world simulator that can produce pixel coherency is very, very useful for games and, marketing and all these things, but it’s not as useful as people think when it comes to causal reasoning.

[00:42:25] When it comes to embodied ai. Yeah, like it this title is true. We’re not saying that it’s, it’s like, not a great world simulator, but actually in the blog that we, we, we, we wrote, the bet is more so that there are gonna be disproportionately large share of value of real world tasks or, and virtual tasks where high resolution pixel fidelity is not needed.

[00:42:47] Yes. Video models have their values.

[00:42:50] swyx: Yeah. This is at the absolute limit of my physics understanding, but one example that comes to mind is basically having to solve like ba the equivalent of a three [00:43:00] body problem in a deterministic Well, where the video models, which is approximated good enough. Yeah.

[00:43:08] Right. Like there’s, there’s some point at which your approach kind of runs into like the you now have to simulate the world. Please, thank you very much. And like you’re trying to do that, but only to the extent that the game engine lets you and like game engines cannot do some things.

[00:43:23] Fan-yun Sun: Yeah, no, I mean, I think the interesting or more technical question here actually is where do you draw the boundary between.

[00:43:32] What’s handled with, let’s say, diffusion prior and what, when? What’s handled with symbolic priors?

[00:43:38] swyx: Yes.

[00:43:38] Fan-yun Sun: Okay.

[00:43:38] swyx: Okay.

[00:43:39] Fan-yun Sun: Right. Let’s go there. Because this, this boundary can actually be fluid. Like I think like maybe what you’re trying to get at is like, okay, people are saying pixel prior, everything. But what we’re saying is, okay, there’s a boundary that we draw where this is where we think provides the most economical value for the domains and things that we care about today.

[00:43:59] [00:44:00] And I actually do think, and it’s something that we do internally all the time, which is like, okay, given new equations that we learn or new elements of the world and that we, we learn, or maybe some other knowledge that we acquire in the process of developing the models. Should we still be maintaining this line exactly as it is today?

[00:44:22] Or should we move it a little bit left or a little bit right? Right. Like sometimes that we realize that, oh, like maybe customers or, or folks like want certain things that are better handled with preop pryor as opposed to, symbolic prior than,

[00:44:34] swyx: yeah. Your, your skin thing is a, is a example moving it, right.

[00:44:37] Yeah.

[00:44:37] Or left. Yeah,

[00:44:37] Fan-yun Sun: exactly.

[00:44:38] swyx: I dunno what the, the left right is.

[00:44:39] Fan-yun Sun: Yeah, yeah, yeah. No the, the model.

[00:44:42] swyx: Yes.

[00:44:42] Fan-yun Sun: Actually we have a few iterations of them. They’re actually at slightly different

[00:44:45] swyx: I know boundaries. You should, you should do that. That’s a cool dimension to show.

[00:44:49] Fan-yun Sun: Yeah.

[00:44:50] swyx: Is quantum mechanics the diffusion prior of our world?

[00:44:55] Right. It’s like that’s the boundary of classical mechanics versus quantum. Right? Like, that’s it. At one [00:45:00] point God plays dice and the other point doesn’t.

[00:45:02] Fan-yun Sun: I dunno if Chris, you wanna say it, but I think, I think generally I feel like physics is better with symbol P priors.

[00:45:08] Chris Manning: Even quantum physics.

[00:45:09] Fan-yun Sun: Even quantum physics.

[00:45:11] swyx: Yeah. This is starts against to, MLST territory is, is what I call it, where, he, he likes to get philosophical. We, we we’re quite friendly.

[00:45:18] Vibhu: I mean, we need to get, we need to get singularity. I heard some of that.

[00:45:23] swyx: No, no, I think that is actually really helpful and man, I just want you to productize this like, as a product guy, I’m just like, oh, also

[00:45:32] Vibhu: a gamer, I

[00:45:33] swyx: wanna, it’s like a researcher, like, it’s cool.

[00:45:35] Like this is a, the theoretical, like you have a very good, I don’t know, like the way of thinking about these things, but I just wanna see you like, express it. I do think like your fundamentally things when, when you leave open new tools, like, okay, use, use human intent to incorporate it into how you render.

[00:45:52] Artists are gonna have to take like two to three years to figure out what to do with this. And you just don’t know.

[00:45:57] Chris Manning: Right. But I think, this is, [00:46:00] gives a much more approachable and controllable world for the society, which is the beauty, the beauty of, NLP, that that will enable it to be adopted and used.

[00:46:10] And we are very hopeful about that. Yeah,

[00:46:13] Fan-yun Sun: yeah. Yeah. I mean, we are, we are very focused actually on commercialization in the sense that like we do, we do really believe in the data flywheel app approach. Yeah. Where, we put this in the hands of the creators and the users and then they will teach us when, what capability our model should improve.

[00:46:27] And that’s why we are, we are actually, like products and beta

[00:46:31] swyx: Yeah. Focusing on gaming. What, what’s like the adjacent thing to gaming

[00:46:34] Fan-yun Sun: embody adjacent, basically. So maybe we can, we can I’ll maybe start with where we see the platform in three years. Yeah. Which is like, okay. The users would tell us what they want to achieve.

[00:46:45] The end goal could be, Hey, I just, I wanna make something to teach my kids the value of humility. Or it could be, Hey, I wanna fine tune my, drones to be really good at rescue situations. I could be vacuum robots. I want to like train [00:47:00] my manipulation or like vacuum robot to be very robust to my office, right?

[00:47:04] But it’s like, whatever it is, scenario robust to

[00:47:06] swyx: my office

[00:47:07] Fan-yun Sun: or like navigate very robustly in my office. But then it’s like, whatever end goal that you want, our role model will say, okay, given what you want to achieve, let me generate a distribution of environments such that I can train and evaluate whatever it is you want.

[00:47:24] Yeah. Right. Maybe for the purpose of games, it’s just the end simulation and that’s the end product for certain policies. It’s like I can train it within these environments and then help you see where your policy is failing or not. Yeah. And then, so I think,

[00:47:37] swyx: so in that case, much more of a training tool.

[00:47:40] Than in other training

[00:47:41] Vibhu: evaluation? Both. Right?

[00:47:43] swyx: Sure. Same. Same thing.

[00:47:43] Fan-yun Sun: Yeah, same thing. I think it’s just this role model that allows people to train any policy that can act in any multimodal environments.

[00:47:51] swyx: Would it be harder to reward hack? Is there an angle here where it is harder to reward hack? Like it’s just, I’ll just put it generally because I think that’s a, that’s obviously a key [00:48:00] problem that a lot of people face when in training agents in these environments, and I don’t know, can you solve it?

[00:48:07] Chris Manning: I think not necessarily. To the extent that there’s a mis specified reward that. It seems like it could be hacked in a more symbolic world or in a more pixel based world. I dunno if Sun’s got any thoughts, but I don’t think that’s really being solved.

[00:48:26] swyx: The other thing that comes to mind is just you could just build a better sawa as a video generator model, right?

[00:48:31] Because then you, you would move the diffusion, side a bit more further to the right. I think if I got the directionality correct. And that’s it.

[00:48:40] Vibhu: It’s better on domains, right? Like on consistency over now, or for sure it exists versus something doesn’t, right.

[00:48:46] Chris Manning: So

[00:48:46] swyx: yeah. Yeah. Is

[00:48:49] Vibhu: is a question more like, like

[00:48:51] swyx: I’m just riffing on like, how do you, what can you build, you know?

[00:48:54] Oh, with the stuff that you have. I do think that the minor, the academic does go immediately to training [00:49:00] and in eval evaluation, but like art tends to take unusual directions. Like you might end up,

[00:49:06] Chris Manning: okay. Yeah. But the question is, can you use this piece of software to develop compelling gameplay and. I don’t think you can take SOAR and produce compelling gameplay, right?

[00:49:19] If you want to have a world that you can wander around in a bit, you are good. But what are your abilities to have gameplay mechanics implemented the way you’d like them to be and to have things stay, with the long-term history of your gameplay that influences future actions. I think there’s just nothing there for that.

[00:49:39] swyx: Yeah, I do tend to agree. I, I’m just trying to sort of test the boundaries. I would also make the observation that as AAA games industry has developed the line between what is a movie and what is a game has blurred. And you, you, you do end up basically producing a two hour movie as part of your game.

[00:49:57] Fan-yun Sun: No, honestly, there, there’s so many actually [00:50:00] applications in adjacent markets that our world model can go into. Yeah. But yeah, it, it’s sort of fun to riff, riff on. Although on the execution side, we we, we need to stay focused with like, okay, what are the capabilities we want to unlock over time?

[00:50:11] And there’s a roadmap for that. But yeah, if we’re just riffing on sort of like the possibilities, I feel like, whether it’s endless Yeah, it’s like classic

[00:50:18] swyx: and the embedding for a possibility and endless in my mind, it’s very close. Yeah. I do wanna, focus on one, like weird choice. I, I don’t know if it’s weird.

[00:50:28] Maybe I’m, I got something here. Audio, right? You could have just said no audio And audio in my mind has a lot of recursion, whereas in video you can just do recasting and that’s much computationally much simpler. Audio just seems way harder. I don’t know if you wanna just comment on just the special 3D audio.

[00:50:46] Problem. Did you really have to do it? I guess you do to be immersive, but like a lot of people do treat it as like, well, you just stick a, a tt S model on top of

[00:50:57] Vibhu: Well, there’s a lot more to game audio than [00:51:00] just speech. Right. It’s not just

[00:51:01] swyx: tts. Yeah. Tts. S Fxt, GM Spatial in my mind Echoes

[00:51:06] Chris Manning: Yeah.

[00:51:06] swyx: And reflections.

[00:51:07] And I, I don’t even know what’s, what else? I don’t know what, what other problems in this space.

[00:51:13] Fan-yun Sun: Yeah, I think this point like the, it’s sort of a more, more pointing to the benefits of using an game engine as a tool that’s available to the model, right? Because like part of the spatial audio is from the code that is underlying the simulation.

[00:51:32] And while we do give our model access to other types of audio models as. Tools.

[00:51:39] swyx: None of them would be spatial, I think.

[00:51:41] Fan-yun Sun: But that’s exactly sort of more 0.2. We’re giving our model an abstraction or a suite of tools such that it’s able to achieve that. And you can argue that sort of spatial is like a, like a emergence out of the, the tools that we and abstraction that we provide to the agents.

[00:51:59] And I think that’s the beauty of [00:52:00] this, this, this approach is like there’s a lot of things kind of like how human’s built technology and they’re like Lego blocks that build on top of each other. And it’s the same thing here. There’s gonna be things that sort of just sort of emerges from being able to put these things together in like combinatorially interesting ways,

[00:52:14] Chris Manning: right?

[00:52:15] So this integrated audio model exploits the understanding and semantics of the Moon Lake world, right? And whereas in general for the Gen AI video models. There’s no actual integration across to audio at all, right? That someone might stick some music or stick a soundscape or whatever else on top of their video.

[00:52:44] So it’s not a silent video, but they’re in no way connected into a consistent world model. And there’s nothing that’s okay. An action is happening in the video. Therefore there should be a sound that’s [00:53:00] coming from this part of the visual field.

[00:53:03] swyx: Yeah.

[00:53:03] Vibhu: Is that different than Sora too? Does it not have audio?

[00:53:06] Not to say it’s not like

[00:53:08] swyx: amazing

[00:53:08] Vibhu: isn’t a spatial

[00:53:09] swyx: audio.

[00:53:09] Vibhu: It doesn’t,

[00:53:10] swyx: no. I’ve played around it with it enough. It just sounds like someone put an 11 laps voice on top of it and just tried to do the lip sync.

[00:53:18] Vibhu: Oh, yeah. I’ve seen, okay. Generate a dog at the beach and reactions to big wave and move

[00:53:23] swyx: around.

[00:53:23] It’s definitely like, so have the dog, have the dog move away from camera and see if the, the song goes down. It doesn’t. ‘Cause they don’t have facial audio.

[00:53:32] Fan-yun Sun: We do want to basically like we, our moral model, like the one we’re training is basically towards the goal of having a combined latent representation across all these different modalities.

[00:53:42] Right? Such that it can like reason across these different modalities. So for example, if I close my eyes and like you play a video, you play a sound of like a car skidding away from me. I almost can like, visually extrapolate that trajectory in my mind. And I think that type of capability, we want our model to be able to reason, right?

[00:53:59] And that’s the reason that [00:54:00] we’re sort of taking this multimodal reasoning approach. It’s like we want this combine late in space that can

[00:54:05] swyx: Yeah. Oh, you said late in space. We like that. Here we have to play the, the bell Every time that someone says late in space, no, you gotta train daredevil one. Where you, you, you, it’s only audio, but you have to work out.

[00:54:15] Where everything is.

[00:54:19] Cool. I I think that that was, that was about it for our Moon Lake coverage. I do think that we have like a couple of, Chris Madden questions on, on IR and, just any, any other sort of attention topics or n NLP topics.

[00:54:31] Vibhu: Okay.

[00:54:31] swyx: Go ahead.

[00:54:32] Vibhu: Well, no, I mean, yeah, it’s just fun. We talked a bit about how you guys met, but you basically, you, you were like the godfather of NLP per se, right?

[00:54:39] You spent the whole career from early embeddings, early early attention. You did 2015 attention for machine translation, everything. You, you had information retrieval, so RAG before rag, we just wanna shout that out and admire a lot of that. Right? So what prompted the switch over to world models?

[00:54:56] How, how’d all that come about?

[00:54:58] Chris Manning: To some answer it [00:55:00] is, the enthusiasms and creativity of students, but there’s a bit of a history there, right? So, yeah. So clearly most of my career has been doing stuff with language and how I got into research was thinking, ah, this is just so amazing how humans can produce speech and understand each other in real time.

[00:55:21] And somehow they managed to learn languages from their kids. How could this possibly happen? And so, yeah, starting off I was very focused on language, but as it sort of got into the 2000 and tens, I started, going, I’d been working on question answering, and then I started to get, interest in visual question answering.

[00:55:42] And that was an area where it was very noticeable. That the visual understanding was bad. Right. These were the days when like, it sort of seemed like there’s almost no visual [00:56:00] understanding. You were just getting answers that came from priors. So, if you asked how many people are sitting at the table, it’d always answer two regardless of how many, how many people you could see in the picture.

[00:56:11] And so it seemed like, oh, these models actually aren’t able to get semantic information outta IMA images. And so I was interested in that problem and tried to work more on that. And so then that required. Knowing more about what’s happening in vision and how you can represent visual information.

[00:56:34] And then things start, there started to be this revolution of, doing generative AI images. And then I had students that started looking at that before the era of Moon Lake. I was also working with Demi Gore, who founded pika. And so, and

[00:56:50] swyx: Ian obviously

[00:56:52] Chris Manning: with gans. Yeah. Though Ian was never my student, but yeah, Ian I was very aware for the, the whole decade there of Ian with Gans.

[00:56:59] [00:57:00] Yeah. And I mean, Ian was a Stanford undergrad, but yeah,

[00:57:03] Vibhu: richard des u.com, I believe he was your student.

[00:57:06] Chris Manning: Yeah. Yeah. And there were, there were links across at that stage as well. So there were several papers in that era of doing, I mean, so Andre Cap was a, PhD student at the same time as Richard.

[00:57:20] And so there was some joint language vision work in that era as well. It seems kind of ancient by modern standards, but yeah, we’re trying to go from sort of textural dependency graphs to visual scenes

[00:57:32] Vibhu: at a time. The glove embeddings really took over a lot of. T-F-I-D-F, like one hot encoding, all that.

[00:57:38] The early vision language models we saw were like lava style adapters, right? It’s, it’s technically still just embedding latent space. Let’s add image, let’s like mixed modality. So, and that, that’s one of the things you super put out there too, right?

[00:57:51] swyx: Yeah.

[00:57:51] Vibhu: Yeah.

[00:57:52] swyx: Yeah.

[00:57:55] swyx: Well, thank you for all of that. Thank you for all advancing the worlds on, world modeling.

[00:57:56] I honestly, do think that if people deeply understand everything we just [00:58:00] covered, they will see what’s coming. I think you guys have, made some, a really significant contribution here. What are you hiring for? What is the, what do people find? We, we agreed that the CTA was a hiring call.

[00:58:10] Yeah. Don’t we have a GI You don’t need, you don’t need engineers anymore, right?

[00:58:14] Fan-yun Sun: Yeah. On the model side we are actually striving towards basically a self-improving system. But what that means is that we need people to set up the self-improving system. So more, more specifically people who have the intersection of knowledge within co-generation and computer vision and graphics, right?

[00:58:30] Yeah. That’s, that’s sort of the core research background that we look for within OTM and, and the majority of the team today do have like both backgrounds.

[00:58:38] swyx: When you say computer vision and graphics, are they the same thing or is it computer vision one thing, graphics, another thing. And how intertwined are they?

[00:58:46] Chris Manning: They’re intertwined but different.

[00:58:49] swyx: Yeah.

[00:58:49] Chris Manning: And I think, this relates to some of the themes that we’ve been talking about, that the more explicit underlying [00:59:00] world models that are being constructed inside Moon Lake really draw on the computer graphics tradition. And so it’s then combining that with the visual understanding of vision.

[00:59:16] swyx: Got it. Yeah. All right. So you’ve written a game engine, you’re come talk to us, right?

[00:59:21] Fan-yun Sun: Oh yeah, definitely. Definitely. But I do think that the line is blurred, like increasingly blurred these days where it’s like if you have a general understanding of group vision and graphics,

[00:59:31] swyx: I think for your standards it is, for me it feels like vision is, is.

[00:59:35] I’ll leave that to the big labs graphics. I, I, I can get that, you would want to do that from more first principles, but vision, there’s so many vision models off the shelf that I can take, but probably not good enough for your

[00:59:45] Fan-yun Sun: I see, I see. If, if you’re sort of like making that distinction then maybe we, we care a little bit more about having graphics

[00:59:51] swyx: knowledge.

[00:59:51] Yeah, exactly.

[00:59:52] It could be like, sometimes a hiring call can be as simple as like, if you know the answer to blah, you should talk to me. Like the sort of core known hard [01:00:00] problem in, in your world.

[01:00:01] Fan-yun Sun: Ah, I see. Yeah. In that case, if you, yeah, definitely. If you’ve written a game engine before, if you’ve rld a variety of coding models on different objectives, like

[01:00:13] swyx: easy,

[01:00:13] Many of those, yeah.

[01:00:14] Fan-yun Sun: If you’ve done multimodal lean space alignment, I, I intentionally include

[01:00:20] swyx: space.

[01:00:20] Fan-yun Sun: Again,

[01:00:21] swyx: a poor editor has a thing every time. Yeah. Lean space alignment. Honestly. Is it that hard?

[01:00:26] I, I, there’s some scripts out there that I’ve saved for the day. I someday have to do it, but I don’t have to do it.

[01:00:31] But it’s

[01:00:32] Fan-yun Sun: done, I think. Yeah. There, there’s, there’s a versions of that that are done. But I, I think we are aligning audio, text, language and video. Yeah. Right. Like, and basically we have these role models that are able to act as agents to like act in these worlds and extract long horizon videos and encoding that back to the model to sort of self-improve.

[01:00:52] So it’s an insanely exciting, but also technically challenge problem. Yeah. So people who wanna do their lives best work, that only [01:01:00] makes a place.

[01:01:01] Vibhu: How big are you guys? Where are you guys based?

[01:01:02] Fan-yun Sun: We’re currently based in San Mateo, although we’re moving up to sf. We’re about 18 folks right now.

[01:01:08] swyx: My ending question was gonna be why, what, what is the name?

[01:01:10] What’s behind the name?

[01:01:11] Vibhu: Yeah.

[01:01:12] Fan-yun Sun: Oh,

[01:01:14] Vibhu: Very cool. Graphics and design, by the way.

[01:01:16] Fan-yun Sun: Actually at the, at the time when the, when the, when we started the company, we were thinking a lot about how do we make a company name that gives people the vibe of like, open ai, but for like, almost like industrial light and magic vibes.

[01:01:28] Wow. Because it’s like we care about creativity and using that as a funnel to solve a GI. So then we were, we, we brainstorm a lot around like Dreamworks, right? Like industrial light magic. And, so there’s a few, few basically, space of things that we feel like are very, very semantically close to the company’s identity.

[01:01:47] swyx: Yeah.

[01:01:48] Fan-yun Sun: And then it ended up being Moon Lake, partly because of the Dreamworks vibe, the Dreamworks, moon

[01:01:54] swyx: Lake.

[01:01:55] Fan-yun Sun: Exactly. Yep. So that was a little bit of that inspiration. And then the moon was sort of [01:02:00] like a, it basically was like about the. Reflection. The reflection part also implies the self-improvement loop.

[01:02:07] Wow. That we sort of like, that’s really bleed and that’s the path towards multimodal general intelligence. So that’s, that’s that. I’ll leave that as I love a good

[01:02:15] swyx: name. I love a good name. This is great. It’s a

[01:02:16] Vibhu: very

[01:02:17] swyx: good name. It’s very good. Lo I’m glad I asked the question. I will also say, one, my favorite story, books or biographies ever is, creativity Inc.

[01:02:24] With Ed Kamal’s, story about Pixar and how he, was rejected as a Disney animation artist. So then he went into computing and brute forced his way into back. No, I love that story. Yeah. Disney.

[01:02:37] Fan-yun Sun: Yeah. And Walt Disney is also like one of my favorite founders. He’s like, his, his story. Like at the time you’re like, okay, I’m gonna create this like.

[01:02:44] Immersive park. Like people can’t, don’t even have that technology to create it virtually, but they’re like, you know what, let’s just build it physically such that people can,

[01:02:50] swyx: so he is the first world modeler.

[01:02:52] Fan-yun Sun: No, I, I I tell people that like, theme parks are world models too.

[01:02:56] swyx: Mm. Yeah. Yeah. Yeah. I mean, it’s a small world or it’s [01:03:00] a, like the Epcot center with all the little, replicas of the countries.

[01:03:03] Yeah. Those are very interesting. Okay. Well thank you, we’ve covered, a huge amount. Thank you for your time and thank you for inspiring us.

[01:03:10] Fan-yun Sun: Thank you

[01:03:10] swyx: for having us. Thank you. It’s fun

[01:03:11] Fan-yun Sun: chatting. Yeah. It’s been a good time.

Perhaps only topped by the Pokemon Go dataset!

Some notable mid tier model releases, but thankfully most companies respected that today is an awful day to launch anything. We’ll give points to Liquid for best April Fools joke.

AI News for 3/23/2026-3/24/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Open-Weight Reasoning and Vision-Coding Releases: Arcee Trinity-Large-Thinking, Z.ai GLM-5V-Turbo, Falcon Perception, and Holo3

• Arcee’s Trinity-Large-Thinking: The biggest substantive model launch in this set was Arcee’s Trinity-Large-Thinking, released with open weights under Apache 2.0 and positioned explicitly for developers/enterprises that want to inspect, host, distill, and post-train their own systems. Follow-up posts claim strong agentic performance, including #2 on PinchBench behind Opus 4.6, SOTA on Tau2-Airline, and frontier-level telecom results (Arcee, Mark McQuade). OpenRouter highlighted the architecture as a 400B total / 13B active model and made it available immediately (OpenRouter). Several ecosystem partners framed it as a milestone for “American open source,” including Prime Intellect, Datology, and infra supporters emphasizing that a small team served a 400B-class model at production cost points (latkins, willccbb, xlr8harder, natolambert).

• Z.ai’s GLM-5V-Turbo: Z.ai introduced GLM-5V-Turbo, a vision coding model that natively handles images, videos, document layouts, and design drafts while preserving pure-text coding performance. The company attributes the gains to native multimodal fusion, a next-gen CogViT encoder, 30+ task collaborative RL, synthetic agentic data generation, and multimodal toolchain extensions for search/drawing/web reading (details, text-coding stability). The model was quickly integrated into multiple downstream surfaces including TRAE, Tabbit, and Vision Arena.

• Falcon Perception and OCR: TII released Falcon Perception, an open-vocabulary referring expression segmentation model, alongside a 0.3B OCR model said to be competitive with models 3–10x larger. The notable design point is an early-fusion transformer that mixes image and text from the first layer instead of relying on multi-stage pipelines and late fusion.

• Other model notes: H Company’s Holo3 was highlighted as a GUI-navigation model family (A3B/35B, Qwen3.5-based, free license, Transformers support). A separate post praised a Qwen3.5 27B distill trained on Claude 4.6 Opus reasoning traces, claiming SWE-bench wins over Claude Sonnet 4.5, 96.91% HumanEval, lower CoT verbosity, 4-bit local usability, and 300k+ HF downloads (Craig Hewitt).

Claude Code Leak, Operational Issues, and the Competitive Coding-Agent Market

• What the leak exposed: Multiple posts converged on analysis of Anthropic’s accidental Claude Code source exposure. The most useful technical synthesis is the long thread from ZhihuFrontier, which emphasizes a minimalist agent core—a single while(true) loop—with sophistication pushed into context management, tooling, and product instrumentation. The leak reportedly showed a 4-layer context compression stack (HISTORY_SNIP, Microcompact, CONTEXT_COLLAPSE, Autocompact), streaming plus parallel tool execution, silent retries on output-length failures, a 40+ tool modular architecture without inheritance-heavy abstractions, and strong use of feature flags and production ablations. A second summary pointed to hidden features including task budget management, AFK mode, “Penguin” fast mode, redirected reasoning, and other unfinished product hooks (ZhihuFrontier).

• Operational pain mattered more than the leak for many users: Alongside leak discussion, many developers complained that Claude was simply slow or unreliable that day (Teknium, andersonbcdefg). Community response also fixated on leaked “pets” and UI affordances (meowbooksj), reinforcing that product polish is part of the competitive moat even when orchestration patterns become legible.

• DMCA blowback: The second-order story was Anthropic’s overly broad repo takedown attempts. Theo reported a DMCA against a fork that did not contain leaked source; he then argued the takedown itself violated DMCA procedure (post). A correction later came from trq212, calling it a communication mistake; the repo was restored and Theo acknowledged the retraction and rapid response (restored, official response).

• Open-source clones and alternatives are gaining mindshare: The leak also turbocharged ecosystem competition. Yuchen Jin noted the leaked Claude Code fork hit 110k+ GitHub stars in a day. At the same time, multiple users said Nous Hermes Agent was easier to deploy and operate than OpenClaw or Claude-derived stacks, often citing near-zero setup and better local workflows (charliehinojosa, VadimStrizheus, Nous). There’s also a tooling wave around prompt steering and efficiency, e.g. a “Universal CLAUDE.md” claiming 63% output-token reduction, and Google’s Agent Skills spec proposing progressive disclosure to cut baseline context by 90%.

Agent Systems Research: Memory, Self-Organization, Coordination Limits, and Security

• Memory is becoming first-class infra: MemFactory proposes a unified inference/training framework for memory-augmented agents with native GRPO integration and reported up to 14.8% relative gains over baselines. Separately, Baseten described a 7M-parameter perceiver that compresses KV cache 8x while retaining 90%+ factual retention, pitching it as a path toward models that “learn from experience.” part_harry_ extended the idea further, arguing pretraining itself is data-inefficient because we discard KV cache every step.

• Do self-organizing agents beat hand-authored roles? A DAIR summary highlighted new work across 25,000 tasks with up to 256 agents, claiming self-organized roles outperform predefined planner/coder/reviewer hierarchies, with a sequential coordination protocol +14% over centralized approaches, 5,000+ emergent roles, and open models reaching 95% of closed-model quality at lower cost. This sits in tension with a separate line of theory: omarsar0’s summary of new MIT work argues delegated multi-agent planning is decision-theoretically dominated by a centralized Bayes decision-maker when agents do not gain access to genuinely different information sources. In practice, the synthesis is likely: multi-agent helps when it partitions tools, environments, or retrieval channels—not just prompts.

• Agent attack surface is the web: A widely shared summary of a new DeepMind paper on “AI Agent Traps” reframes agent security around adversarial content in webpages/documents, not just model jailbreaks. The thread cites hidden prompt injection in HTML/CSS succeeding in up to 86% of scenarios and latent memory poisoning reaching 80%+ attack success with <0.1% contamination, which is material for anyone shipping browse/retrieval-heavy agents.

• Long-horizon evaluation is getting richer: New benchmarks/tools included Kaggle Standardized Agent Exams, YC-Bench for simulating a startup over a one-year horizon, and CaP-Gym / CaP-X, a broad benchmark and toolkit for agentic robotics spanning 187 manipulation tasks, 12 frontier models, and both training-free and RL-improved policies with MIT-licensed code (open-source details).

Training, Retrieval, and Infra: RL Frameworks, Optimizers, Kernels, and Benchmarks

• Post-training stack maturation: Hugging Face’s TRL v1.0 was framed by many as a meaningful unification of open post-training—SFT, reward modeling, DPO, GRPO—into a production-ready package (commentary). A complementary survey thread from adithya_s_k compared 16 RL frameworks across orchestration, rollout buffering, weight sync, staleness handling, partial-rollout behavior, LoRA support, and distributed parallelism, useful for teams choosing between TRL, VeRL, SLIME, and others.

• Optimization and systems releases: HeavyBall 3.0.0 shipped with FSDP, DDP, end-to-end compilation with 2.5x speedup, faster Muon/SOAP variants, and new optimizers. Together AI promoted a behind-the-scenes kernels writeup; Dan Fu followed with a “what a VP of Kernels does” thread. On the low-level DSL side, maharshii argued CuTeDSL materially lowers the barrier to custom kernels by allowing inline PTX directly in Python, avoiding opaque layout gymnastics.

• Retrieval evidence continues to favor late interaction: Several posts reiterated that multi-vector / late-interaction retrieval outperforms single-vector embeddings, even after fine-tuning, with better robustness against catastrophic forgetting (lateinteraction, ladder visualization). There was also continued frustration that “RAG” has become an overloaded umbrella term rather than referring to a specific older paper (lateinteraction).

• Benchmarks and efficiency surfaces: Arena added Pareto frontier charts across text, vision, search, document, and code, making price/performance tradeoffs more explicit. On standardized inference, Lambda and NVIDIA pointed to MLPerf Inference v6.0 as the better lens for real AI-factory productivity than peak-chip specs.

Developer Platforms, Rate Limits, and Tooling UX

• OpenAI Codex usage reset: The most practically important platform announcement for working engineers was thsottiaux’s note that OpenAI reset Codex usage limits across all plans, citing elevated rate-limit hits and a concurrent fraud-account purge that recovered compute. This was quickly amplified by users who interpreted rate-limit generosity as a direct competitive axis in the coding-agent market (reach_vb, Yuchen Jin). Later, thsottiaux also clarified that Codex’s core is intended to be open-source because the ecosystem is still young and mutually informative (post).

• Agent-ready docs and platform surfaces: LangChain embedded chat into its docs grounded on full docs, knowledge base, and OSS code. Together AI open-sourced 12 agent skills so Claude Code and Codex can call its APIs with the right model IDs and SDK idioms. OpenAI Devs also showed tighter Linear integration in the Codex app for keeping tickets synchronized with code work.

• Infra and storage quality-of-life: SkyPilot added native VAST Data support for direct high-speed dataset mounts across heterogeneous compute backends, and Hugging Face rolled out persistent Storage Buckets for Spaces. Tinker added longer context windows up to 256k for select open models, widening its appeal for RL and long-horizon experimentation.

Top tweets (by engagement)

• OpenAI Codex limits reset: thsottiaux reset Codex rate limits across all plans, explicitly tying it to both unexplained user rate-limit spikes and anti-fraud enforcement that freed compute.

• GLM-5V-Turbo launch: Z.ai’s announcement was one of the day’s biggest technical launches: a multimodal coding model aimed at GUI agents, visual coding, and agent workflows.

• Claude Code leak discourse: Theo’s DMCA thread and Yuchen Jin’s note about the leaked project surpassing 110k GitHub stars captured how quickly source exposure translated into open ecosystem momentum.

• Arcee Trinity-Large-Thinking: Arcee’s release and OpenRouter’s architecture summary drew unusually strong engagement for an open-weight reasoning model, suggesting real appetite for serious US-based open releases.

• Falcon Perception: Falcon Perception’s launch stood out on the multimodal side for its simple early-fusion architecture and unusually small OCR model size relative to claimed performance.

• Claude Code’s source just leaked — I extracted its multi-agent orchestration system into an open-source framework that works with any LLM (Activity: 1205): The source code for Claude Code was leaked, revealing over 500K lines of TypeScript, including its multi-agent orchestration system. A developer has re-implemented this system as an open-source framework called open-multi-agent, which is model-agnostic and can work with any LLM, such as Claude and OpenAI. The framework includes features like a coordinator pattern for task decomposition, a team system for inter-agent communication, task scheduling with dependency resolution, and a conversation loop for model-tool interactions. It is implemented in TypeScript, spans approximately 8000 lines, and is available under the MIT license on GitHub. Some commenters express skepticism about the legality and ethics of open-sourcing a re-implementation of leaked proprietary code, questioning the developer’s understanding of the architecture and the choice of licensing. There is also a debate about the practicality of using different models for planning and implementation, with a specific mention of using GPT-4o for coding.

  • A user highlights the technical aspect of the project, noting that the multi-agent orchestration system extracted from Claude Code’s source involves a coordinator that breaks down goals into tasks. This suggests a sophisticated architecture designed for task management across multiple agents, which could be beneficial for complex LLM applications.

  • Another comment questions the choice of using GPT-4o for implementation in the orchestration system, implying that by March 2026, GPT-4o might be outdated for coding tasks. This raises a point about the importance of selecting the most current and capable models for specific tasks in AI development.

• Claude code source code has been leaked via a map file in their npm registry (Activity: 5229): The image reveals a directory listing of the ‘claude-code’ project, which appears to have been unintentionally exposed via a map file in the npm registry. This leak includes TypeScript files and directories such as ‘entrypoints,’ ‘commands,’ and ‘utils,’ providing a detailed view of the project’s codebase structure. The incident highlights potential security oversights in managing sensitive code repositories, particularly for companies like Anthropic that are involved in AI development. Commenters humorously speculate on the oversight, suggesting it might be due to an Anthropic employee’s mistake or a failure of AI oversight mechanisms. There’s also a satirical suggestion that the code is now ‘open source’ due to the leak.

  • The leak of Claude’s source code via a map file in their npm registry raises significant security concerns, particularly given the model’s reputation for identifying vulnerabilities. This incident highlights potential gaps in Anthropic’s internal security measures, as their AI, known for being ‘scary good’ at finding vulnerabilities, failed to detect this issue.

  • The leak has sparked discussions about the potential for community-driven improvements, such as fixing existing bugs like the caching issue. This could lead to a more robust version of Claude, as external developers might contribute patches and enhancements, effectively making it ‘open source’ in practice, if not in legal terms.

  • The incident also underscores the challenges of maintaining proprietary code secrecy in public repositories. The humorous suggestion of an ‘Undercover Mode’ for Anthropic employees, which would strip AI attribution from commits, reflects the tension between open collaboration and the need to protect intellectual property.

• Analyzing Claude Code Source Code. Write “WTF” and Anthropic knows. (Activity: 840): The Reddit post discusses the source code of Claude Code, revealing extensive tracking and classification mechanisms. The system uses simple keyword detection for language classification, tracking words like wtf and frustrating to flag negative sentiment. It also monitors user behavior during permission prompts, logging actions such as opening or closing feedback boxes and typing without submitting. The feedback system is designed to capture negative experiences, prompting users to share session transcripts. Hidden commands like ultrathink and ultraplan alter system behavior, while telemetry logs detailed environment profiles, including session IDs and runtime details. An internal mode (USER_TYPE=ant) collects even more granular data, tying behavior to specific deployment environments. The post suggests this level of instrumentation is more detailed than typical user expectations, though not necessarily malicious. Source. Commenters note that such tracking mechanisms are standard in many applications for analytics and feedback, suggesting that negative sentiment triggers help identify issues with updates. Some commands, like /btw, are now public, while others remain as internal features or ‘easter eggs.’ The extensive internal artifacts are likened to those found in game apps, possibly due to internal incentives for feature development.

  • NandaVegg highlights that the use of keyword lists for sentiment analysis in Claude Code is a standard practice in event-triggered analytics. This approach helps identify negative user feedback, which can be crucial for detecting issues in updates that might disrupt user experience or model behavior. The mention of features like ‘ultraplan’ and ‘ultrathink’ suggests these are experimental or less refined, possibly serving as internal tests or ‘easter eggs’ within the system.

  • SRavingmad expresses curiosity about the ‘tamagotchi mode’ in Claude Code, implying there are unique or playful features embedded within the system. This suggests that the developers might be experimenting with interactive or gamified elements, which could be part of a broader strategy to engage users or test new functionalities.

  • Exhales_Deeply criticizes the reliance on AI-generated content, suggesting that user-generated posts would be more engaging. This comment indirectly points to a broader discussion about the quality and authenticity of AI-generated content versus human-created content, which is a significant topic in AI development and user interaction.

• The Bonsai 1-bit models are very good (Activity: 657): PrismML’s Bonsai 1-bit models offer a significant reduction in model size and memory usage, being 14x smaller than traditional models, which is transformative for local model deployment. The Bonsai 8B model was tested on an M4 Max 48GB MacBook Pro, demonstrating practical applications like chat and document summarization with lower memory pressure compared to models like Qwen3 VL 8B Instruct Q4_K_M. However, it requires a specific fork of llama.cpp to support 1-bit operations, as the main llama.cpp repository lacks this capability. The model’s performance is notably superior to previous MSFT BitNet models, which were largely research-focused and not practical for real-world use. A benchmark comparison between Bonsai and Qwen3.5 models suggests Bonsai’s higher quality for RAM usage, though it struggled with code generation. There is interest in larger Bonsai models, such as a 200B version, and a desire for quantized versions of Qwen 3.5 models.

  • itsArmanJr provides a detailed benchmark comparison between Bonsai and Qwen3.5 models, including specific configurations like 35B-A3B, 2B, and 0.8B. The benchmark results are available on GitHub, offering insights into performance metrics across different model sizes.

  • -dysangel- highlights the efficiency of Bonsai models in terms of RAM usage, noting that while the model struggled to produce fully functional code, it was impressive given its small size of only 1GB. The comment suggests exploring quantized versions of Qwen 3.5 models, such as 9B or 27B, for potentially better performance.

  • Pitiful-Impression70 raises concerns about the performance of 1-bit quantized models like Bonsai on longer contexts, noting that coherence often degrades past 4k tokens. This comment questions whether the Bonsai model maintains quality in extended conversations compared to shorter prompts.

• TurboQuant isn’t just for KV: Qwen3.5-27B at near-Q4_0 quality, about 10% smaller, and finally fitting on my 16GB 5060 Ti (Activity: 899): The image illustrates the TurboQuant TQ3_1S model’s ability to maintain near-Q4_0 quality for the Qwen3.5-27B model while being compact enough to fit on a 16GB RTX 5060 Ti. The TQ3_1S model is about 10% smaller than Q4_0, with a size of 12.9 GB compared to 14.4 GB for Q4_0, and shows a minimal performance gap in perplexity (PPL), with TQ3_1S having a PPL of 7.2570 versus Q4_0’s 7.2431. This demonstrates a practical advantage for users with limited GPU memory, allowing the model to fit fully on the specified GPU setup. The post also highlights the use of advanced quantization techniques like Walsh-Hadamard rotation and 8-centroid quantization to achieve these results. Some commenters criticize the use of perplexity as a metric for quantization loss, suggesting KLD or PPL ratio as more accurate alternatives. Others praise the adaptation of cutting-edge research to solve a practical problem, acknowledging the achievement despite the criticisms.

  • Velocita84 criticizes the use of Q4_0 quantization, stating it’s outdated and surpassed by more advanced Q4 techniques. They argue that using perplexity as a metric for quantization loss is incorrect, suggesting KLD or PPL ratio against a full bf16 model as more accurate alternatives.

  • grumd suggests comparing the model to unsloth Q3_K_S quant of 27B using real benchmarks, implying that practical performance comparisons are necessary to validate claims about model efficiency and quality.

  • XccesSv2 expresses skepticism about TurboQuant’s claims of achieving BF16 quality with 4 or 5 bits, noting that real-world tests often don’t reflect the purported improvements, indicating a gap between theoretical claims and practical outcomes.

• PrismML — Announcing 1-bit Bonsai: The First Commercially Viable 1-bit LLMs (Activity: 596): PrismML has announced the release of the 1-bit Bonsai models, including the 1-bit Bonsai 8B, which is a groundbreaking development in AI model efficiency. These models are fully quantized to 1-bit precision across all components, including embeddings, attention layers, MLP layers, and the LM head, without any higher-precision components. The 1-bit Bonsai 8B model, with 8.2 billion parameters, fits into 1.15 GB of memory and is 14x smaller, 8x faster, and 5x more energy efficient than its full-precision counterparts, making it suitable for edge hardware. The models are open-sourced under the Apache 2.0 license, and the implementation requires a fork of Llama.cpp for inference. More details can be found in their whitepaper. Some commenters express skepticism about the practicality of 1-bit models, while others are intrigued by the potential for on-device AI applications. The debate centers around the trade-offs between model precision and performance efficiency.

  • PrismML has announced the 1-bit Bonsai 8B model, which is a 1-bit weight model that fits into 1.15 GB of memory. It claims to deliver over 10x the intelligence density of full-precision counterparts, being 14x smaller, 8x faster, and 5x more energy efficient on edge hardware. The model is open-sourced under the Apache 2.0 license, and the company emphasizes the potential for on-device AI applications due to its efficiency.

  • The 1-bit Bonsai 8B model is quantized end-to-end using a proprietary method, requiring a fork of Llama.cpp for inference. This model design applies 1-bit quantization across all network components, including embeddings, attention layers, MLP layers, and the LM head, making it a true 1-bit model across its 8.2 billion parameters. This approach highlights a significant shift towards more efficient AI models that can operate effectively on edge devices.

  • The announcement suggests a paradigm shift in AI model design, focusing on intelligence density rather than parameter count. By achieving significant reductions in model size and energy consumption, PrismML’s 1-bit models could enable new applications in real-time robotics and offline intelligence, potentially transforming the AI landscape by making advanced models feasible for local execution on edge devices.

• Local LLM Claude Code replacement, 128GB MacBook Pro? (Activity: 140): The user is considering upgrading to a 128GB MacBook Pro to run local LLMs as a replacement for Claude Code due to potential price increases in API usage. They are currently using a 2019 Intel-based MacBook Pro and are experiencing performance issues with multiple Docker containers. The user is exploring whether local LLMs can match the capabilities of Claude Code for software development. Claude Code is noted for its 1 million context capability, but open-source models are improving. A user reported running qwen3.5 122b ud q4 xl with a 256k context on a 128GB RAM system, finding it competent for lighter tasks, though not as strong as Claude for heavy coding. Another user suggests trying open-source models via DeepInfra before purchasing, and mentions using the Bodega inference engine as a replacement for commercial subscriptions. There is a debate on whether local LLMs can fully replace Claude Code, with some users finding open-source models like qwen 122 competent for lighter tasks but not yet matching Claude for intensive coding. The shared memory model of Mac is seen as advantageous for running local LLMs.

  • EmbarrassedAsk2887 discusses replacing Claude Code and Codex subscriptions with the Bodega inference engine on a 128GB M4 Max MacBook Pro. They provide a detailed write-up and benchmarks, suggesting that Bodega can effectively handle tasks typically managed by commercial solutions. Read more here.

  • Mediocre_Paramedic22 shares their experience running the Qwen 3.5 122B UD Q4 XL model with a 256k context on a 128GB RAM setup using Fedora. They note that while Claude is superior for intensive coding tasks, Qwen performs well for lighter workloads and basic agent tasks, utilizing about 29GB of free RAM.

  • Aisher mentions using a 128GB M5 Max for local LLM development, noting the noise level as a downside. They suggest using multiple desktop Macs for full-time development, connected via ZeroTier for remote access, as a cost-effective alternative to expensive cloud-based solutions.

• Worth building a $7k local AI rig just to experiment? Afraid I’ll lose interest. (Activity: 131): The user is contemplating building a $7k local AI rig to experiment with AI technologies, particularly in photo and video generation, model integration, and AI assistant development. They currently use a MacBook with an M3 Pro chip and 36GB RAM but are concerned it may not suffice for more complex tasks. The proposed rig includes a Corsair Vengeance i5200 with an Intel Core Ultra 9 285K, GeForce RTX 5090, and 64GB DDR5 RAM, with plans to add an additional 128GB RAM. The user is hesitant due to the lack of a concrete use case and the potential for the rig to become an ‘expensive toy’. Commenters suggest alternatives such as renting a machine or using existing hardware with tools like LM Studio to test models like Qwen3.5, 9b, and 27b Q4. Another commenter shares a similar dilemma and opts to continue using a current setup with an RTX 4070Ti and 32GB RAM, highlighting the importance of having a clear use case before investing heavily.

  • TassioNoronha_ suggests starting with cloud-based solutions like Open Router or renting a machine for a week to gauge interest before committing to a $7k investment. This approach allows for experimentation without the upfront cost, providing a practical way to assess long-term interest and needs.

  • Xmede81 shares their experience of sticking with a current setup featuring an RTX 4070Ti and 32GB RAM, which is sufficient for general use and experimentation. They highlight the importance of evaluating actual use cases and the impact of current memory prices on decision-making.

  • Dry-Influence9 advises against building powerful local setups due to current high prices, suggesting that waiting could yield better value. They recommend renting GPUs or using existing computers to experiment, as this can provide similar capabilities without the significant financial commitment.

• We built a local inference engine that skips ROCm entirely and just got a 4x speedup on a consumer AMD GPU (Activity: 124): ZINC is a new inference engine designed to bypass the complexities of ROCm by directly interfacing with AMD GPUs through Vulkan, achieving a 4x speedup on an AMD Radeon AI PRO R9700. The engine supports models like Qwen3.5-35B-A3B and Qwen3.5-2B, with current performance at 33.58 tok/s, compared to 107 tok/s for llama.cpp on the same hardware. ZINC’s architecture allows it to run on hardware not officially supported by ROCm, and it includes an OpenAI-compatible API server for parallel request batching. The project is open-source and available on GitHub. Some commenters question the significance of the speedup given that ZINC’s performance is still less than a third of llama.cpp’s speed. Others express skepticism about achieving such improvements when larger companies have struggled in this area.

  • Big-Masterpiece-9581 questions the significance of the 4x speedup, pointing out that despite the improvement, the performance is still less than a third of llama.cpp‘s speed. This suggests that while the optimization is notable, it may not yet be competitive with existing solutions in terms of raw throughput.

  • fallingdowndizzyvr highlights a performance issue, noting that achieving only 7 tok/s on an AMD Radeon AI PRO R9700 with the Qwen3.5-35B-A3B-UD Q4_K_XL model indicates a potential inefficiency in the initial implementation. This suggests that the baseline performance was suboptimal, which could have skewed the perceived improvement.

  • hipcatinca provides a benchmark comparison using an RX 570 with llama.cpp via Vulkan, achieving approximately 31 tok/s with the llama3.1:8b model. This serves as a reference point, illustrating that other configurations and models can achieve significantly higher throughput on different hardware setups.

/r/Singularity, /r/Oobabooga, /r/MachineLearning, /r/OpenAI, /r/ClaudeAI, /r/StableDiffusion, /r/ChatGPT, /r/ChatGPTCoding, /r/aivideo, /r/aivideo

• Claude code source code has been leaked via a map file in their npm registry (Activity: 1598): On March 31, 2026, the full source code of Anthropic’s Claude Code CLI was leaked through a .map file in their npm registry, as reported on GitHub. The codebase, consisting of approximately 512k lines of TypeScript, is built using React + Ink for terminal UI and runs on the Bun runtime. This leak potentially exposes major gated features that are not yet public. The comments reflect a misunderstanding among some users about the implications of the leak, particularly the difference between Large Language Models (LLMs) and agents, highlighting a knowledge gap in the community.

  • The leak of Claude’s source code via a map file in their npm registry has sparked discussions about the potential implications for developers and researchers. One key point is the distinction between Large Language Models (LLMs) and agents, as highlighted by Nedshent. This leak may expose a knowledge gap where people might not fully understand how LLMs function compared to agents, which are typically more task-specific and interactive.

  • The technical details of the leak reveal that the codebase consists of approximately 512k lines of TypeScript, built with React and Ink for terminal UI, and runs on the Bun runtime. This setup suggests a modern and scalable architecture, potentially offering insights into how Claude’s infrastructure is designed to handle complex tasks and interactions.

  • There is speculation about the reasons behind the leaks, with some users humorously suggesting that Anthropic might be using Claude itself for development and content creation tasks. This raises questions about the security and operational practices within Anthropic, especially if such reliance on AI could inadvertently lead to more leaks or security vulnerabilities.

• Anthropic staff reacts to Claude code leak 👀 (Activity: 859): The image is a meme depicting a humorous Twitter exchange that indirectly references a code leak from Anthropic, a company known for its work in AI. The meme uses a popular internet joke about an ‘immortal snail’ to suggest that the leak is an inevitable consequence of being ‘caught’ by the snail, implying a sense of inevitability or fate. This reflects a lighthearted community reaction to the leak, rather than a technical discussion or official statement from Anthropic. Commenters humorously note the dual reactions to the leak: legal teams wanting to ‘delete it’ while engineers have already ‘starred it,’ indicating a divide between legal caution and technical curiosity. Another comment suggests that with Anthropic’s rapid development pace, such incidents were expected.

  • Belium suggests that the leak of Claude’s code could be beneficial for Anthropic, as it generates hype and allows engineers to identify and fix bugs. The leak also provides engineers with the opportunity to create their own implementations or ‘harnesses’ of Claude, potentially increasing its usage and influence in the developer community.

  • IntenselySwedish highlights a perceived irony in Anthropic’s situation, pointing out that the company, which has been accused of large-scale copyright violations through book piracy, is now facing its own copyright challenges with the leak of Claude’s code. This comment underscores the complex legal and ethical landscape surrounding AI development and intellectual property.

  • xitizen7 comments on the rapid pace of development and releases from Anthropic, suggesting that such a leak was almost inevitable given the company’s trajectory. This reflects a broader industry trend where fast-paced innovation can sometimes lead to security oversights or unintended disclosures.

• Claude Code Source Leak Megathread (Activity: 653): The Claude Code CLI source code was leaked, revealing several technical details. Notably, the npm source (@anthropic-ai/[email protected]) shows that the DuckDuckGo replacement in the Rust port is incorrect; the real package uses a nested API call to Anthropic’s server-side search with encrypted content blobs. Additionally, a two-tier web system is implemented, where 85 domains are pre-approved for full content extraction, while others are limited to 125-character quotes. Structured data in <head> is ignored, and tables are not supported in the markdown converter. The system limits to 8 results per query with no pagination. A hidden feature, KAIROS_DREAM, allows Claude to self-review and update its memory after inactivity. The newer search version (web_search_20260209) enables Claude to programmatically filter search results. The source can be verified in the minified cli.js of the npm package. Anthropic has issued a DMCA to remove the leaked code from GitHub. Some commenters criticize the code quality, suggesting that many critics may lack experience in shipping production apps. Others focus on the technical implications of the leak, such as the incorrect assumptions about DuckDuckGo usage and the limitations of the markdown converter.

  • Ooty-io highlights several technical aspects of the Claude Code source, noting that the package makes nested API calls to Anthropic’s server-side search, with results returned as encrypted content blobs, rather than using DuckDuckGo as a standalone replacement. Additionally, the source code reveals a two-tier web system where 85 documentation domains are pre-approved for full content extraction, while other sites are limited to 125-character quotes. The code also shows that structured data in <head> tags is ignored, and tables are not supported in the markdown conversion process.

  • Independent-Corgi-88 discusses the broader implications of the Claude Code leak, suggesting it points towards a future of AI characterized by multi-agent coordination, memory layers, and persistent interaction. This perspective emphasizes the importance of systems with memory and coordination over raw model capability, suggesting that the future of AI involves environments that support sustained and useful work. The comment also references J3nna, an AI being developed to understand its operating environment, highlighting the shift in focus from model capability to the surrounding system.

  • Joozio provides insights from analyzing the Claude Code source, noting that the CLAUDE.md file is reinserted with every turn change, impacting token usage. They also mention that switching models mid-session clears the prompt cache, leading to increased token costs. Additionally, Claude Code ranks poorly on terminal benchmarks, coming in last for Opus among harnesses, with a flat 77% performance compared to Cursor’s 77% to 93%. Joozio implemented several patterns from the source, such as semantic memory merging and cache monitoring, into their own agent.

• i dug through claude code’s leaked source and anthropic’s codebase is absolutely unhinged (Activity: 6259): The leaked source code of Anthropic’s Claude reveals a whimsical feature: a terminal-based pet system called /buddy, which includes 18 species with a gacha rarity system and interactive ASCII companions. The codebase also shows unconventional practices, such as hex encoding species names to bypass internal scanners, and a voice mode using Deepgram Nova 3 for speech-to-text. The project is codenamed ‘tengu’, with telemetry events and feature flags reflecting this. The codebase is notably large, with main.tsx at 803,924 bytes and several files exceeding 4,000 lines. It contains 460 eslint-disable comments and numerous deprecated functions still in use, indicating a lack of codebase hygiene. Additionally, there are unreleased features like ‘kairos’ and ‘ultraplan’, and several hidden slash commands. Some commenters argue that the codebase’s state is typical for large projects and not particularly ‘unhinged’, while others express interest in the /buddy feature, wishing it were available sooner.

  • A user points out that the presence of deprecated functions in the codebase is likely a strategic decision to signal developers not to use them in new code. This is a common practice in large codebases where gradual migration to new implementations is necessary, especially when multiple developers are involved and there is pressure from sales teams to maintain functionality while transitioning.

  • Another commenter argues that the codebase’s state is typical for large projects, especially those developed before the advent of AI tools like GPT-3. They suggest that the complexity and seemingly chaotic nature of the code are standard in environments where many developers contribute under tight deadlines and evolving requirements.

  • A technical insight is provided regarding the perception of the codebase as ‘unhinged.’ The commenter suggests that such a view might stem from a lack of experience with large-scale software projects, where the code often appears disorganized due to the sheer number of contributors and the necessity to maintain legacy systems while integrating new features.

• Claude Code’s source code just leaked — so I had Claude Code analyze its own internals and build an open-source multi-agent framework from it (Activity: 513): The source code for Claude Code was leaked, revealing over 500K lines of TypeScript, including its multi-agent orchestration layer. A developer re-implemented this as an open-source, model-agnostic framework, allowing integration of different LLMs like Claude and GPT in a shared workflow. Key features include multi-agent teams, task pipelines with dependency resolution, inter-agent messaging, and an LLMAdapter interface. The framework is ~8000 lines of TypeScript and is available on GitHub under the MIT license. Some commenters appreciate the framework’s ability to integrate various LLMs, which can reduce costs. However, others note that the framework’s core functionality is similar to existing solutions like CrewAI and AutoGen, and that the re-implementation mainly replicates standard agent loop patterns.

  • Macaulay_Codin critiques the framework, noting that it follows a standard agent loop pattern: calling an LLM, executing tool calls, and iterating over results. The multi-agent aspect is essentially a task queue coordinator, which is not novel. The framework includes five built-in tools, rewritten from Claude Code’s tools, and is implemented in 8k lines of TypeScript, suggesting it’s a manageable project rather than a massive reverse engineering effort. Alternatives like CrewAI, AutoGen, and the Claude Agent SDK offer similar functionalities.

  • JuryNightFury highlights the framework’s capability to integrate with other model families using an OpenRouter API key, demonstrating its model-agnostic nature. This feature allows it to fetch reviews from various models, showcasing its flexibility in utilizing different AI models beyond its original design.

  • NoInside3418 appreciates the potential cost savings and efficiency gains from using the framework to enable communication between subagents from different models like Gemini, Codex, and Claude. This interoperability could streamline processes by leveraging the strengths of each model, such as Gemini’s large context and low cost, Haiku’s implementation capabilities, and GPT’s planning features.

• Anthropic’s leaked CLI source code reveals a hidden “Tamagotchi” pet and autonomous multi-agent teams. The bar for developer tools is getting wild. (Activity: 161): Anthropic accidentally exposed the source code of their CLI tool, revealing innovative features like a Tamagotchi-style virtual pet called “BUDDY” that gamifies the terminal experience by leveling up based on coding behavior. Additionally, the code includes features like “ULTRAPLAN,” which allows the AI to autonomously plan for 30 minutes, and “BRIDGE MODE,” where multiple AI instances collaborate as a team. Another feature, “KAIROS,” autonomously manages failing tests and dependencies. These features suggest a shift towards more autonomous and interactive developer tools. For a detailed breakdown, see the full analysis. Commenters are skeptical about the feasibility of autonomous multi-agent teams, suggesting the pet feature is more believable due to its potential for user engagement. There is also curiosity about whether these features represent real product directions or are merely experimental ideas.

  • Senior_Hamster_58 raises skepticism about the claim of autonomous multi-agent teams being proven by a leaked repository, suggesting that such features might be more speculative or experimental rather than indicative of a real product direction. They question whether these features are part of a serious development effort or merely internal experiments that may not reach production, highlighting a common issue in software development where many ideas do not survive the transition from concept to release engineering.

  • OutrageousIndustry28 claims that the feature is already live and can be activated using a specific command (/buddy). This suggests that at least some components of the leaked features might be functional or accessible, indicating a level of readiness beyond mere speculation or internal testing. However, without further verification, this claim remains anecdotal.

  • rainmaker66 and prussell774 both suggest that the features, including the “Tamagotchi” pet and autonomous multi-agent teams, are part of an April Fool’s joke by Anthropic. This implies that the leaked code might not represent serious development efforts but rather a playful or humorous initiative, which is a common practice in tech companies around April 1st.

• OpenAI raises $122 billion to accelerate the next phase of AI (Activity: 794): OpenAI has raised $122 billion, reaching a post-money valuation of $852 billion, to bolster its position as a core AI infrastructure provider. The company reports 900 million weekly active users for ChatGPT and $2 billion in monthly revenue. Strategic partnerships with Amazon, NVIDIA, and Microsoft are pivotal in advancing their AI capabilities, focusing on enhanced compute infrastructure and a unified AI superapp for both consumer and enterprise applications. More details can be found in the original article. Commenters are questioning the allocation of such a large funding amount, with some expressing skepticism about the necessity of this capital given recent fundraising efforts.

Unfortunately, Discord shut down our access today. We will not bring it back in this form but we will be shipping the new AINews soon. Thanks for reading to here, it was a good run.

OpenAI’s Largest Fundraise in Human History closed today, growing by a few billion, but disclosing some cool numbers like $24B ARR (growing 4x faster than Google/Meta in their heyday), and also had a “soft IPO” with $3B of investment from rich people and inclusion in ETFs from ARK Invest, although ChatGPT WAU growth seem to has stalled out - they STILL have not crossed the 1B WAU mark targeted for end 2025. Codex also worryingly has not announced a new milestone for March.

By far the biggest news of the day is the Claude Code source leak, in itself not particularly damaging for Anthropic, but surely embarrassing and also somewhat educational - Christmas come early for Coding Agent nerds. You can read the many many tweets and posts covering the 500k LOC codebase, and you can browse multiple hosted forks of the source.

There are fun curiosities, such as the full verb list, or Capybara/Mythos v8, or the /buddy April Fools feature, or Boris’ confirmed WTF counter, or creating the cursed “Claude Codex”, or the dozen other unreleased features, but most serious players are commenting on a few things. Sebastian Raschka probably has a good list of the top 6:

1. Putting Repo state in Context (eg recent commits, git branch info)

2. Aggressive cache reuse

3. Custom Grep/Glob/LSP (standard in industry)

   1. Claude code has less than 20 tools default on (up to 60+ total): AgentTool, BashTool, FileReadTool, FileEditTool, FileWriteTool, NotebookEditTool, WebFetchTool, WebSearchTool, TodoWriteTool, TaskStopTool, TaskOutputTool, AskUserQuestionTool, SkillTool, EnterPlanModeTool, ExitPlanModeV2Tool, SendMessageTool, BriefTool, ListMcpResourcesTool, and ReadMcpResourceTool.

      more in ccunpacked

4. File read deduplication/tool result sampling

5. Structured Session Memory (more on this)

6. Subagents

Claude Code’s Memory has a 3 layer design with 1) a MEMORY.md that is just an index to other knowledge, 2) topic files loaded on demand, and 3) full session transcripts that can be searched. There’s also an “autoDream” mode for “sleep” - merging memories, deduping, pruning, removing contradictions.

A deeper analysis from mem0 finds 8 phases:

And there are 5 kinds of Compaction:

A key feature of CC: they use the KV cache to create a fork-join model for their subagents, meaning they contain the full context and don’t have to repeat work. In other words: Parallelism is basically free.

here:

Including an employee-only gate and an employee TUI, but also a bunch of other stuff in development including ULTRAPLAN and KAIROS:

And internal MAGIC DOCS:

AI News for 3/23/2026-3/24/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Top Story: Claude Code source leak — architecture discoveries, Anthropic’s response, and competitor reactions

Claude Code had substantial source artifacts exposed via shipped source maps / package contents, which triggered rapid public reverse-engineering, mirroring, and derivative ports. The discussion quickly shifted from “embarrassing leak” to “what does this reveal about state-of-the-art agent harness design?” Multiple observers highlighted that the leak exposed orchestration logic rather than model weights, including autonomous modes, memory systems, planning/review flows, and model-specific control logic. Public forks proliferated; one post claimed 32.6k stars and 44.3k forks on a fork before legal fear led to a Python conversion effort using Codex (Yuchenj_UW). Later commentary put the exposed code volume at 500k+ lines (Yuchenj_UW). Anthropic then moved to contain redistribution via DMCA takedowns according to several posters (dbreunig, BlancheMinerva). Separately, a Claude Code team member announced a product feature during the fallout — easier local/web GitHub credential setup via /web-setup (catwu) — implying normal product operations continued. The leak also created a live security hazard: attackers quickly registered suspicious npm packages such as color-diff-napi and modifiers-napi to target people trying to compile the leaked code (Butanium_).

What is reasonably factual from the tweets:

It’s well known that org charts are changing with AI - the first trend we called out was in 2023 with the Rise of the AI Engineer (now an official org at Meta!), and then in 2025 with Tiny Teams (hired by Meta!), but it seems Yoni Rechtman over at the 99D Substack has the mental model for the new post-AI roles (at least in white collar tech):

Karri Saarinen, CEO of Linear, made a popular analogy to the teamwork roles that emerged in World of Warcraft. This is a good 2D augmentation of an earlier age-based company model (much less realistic, name a tech company that fits the latter format, they exist but are very hard to find):

AI News for 3/28/2026-3/30/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Claude Code Computer Use, Codex Interop, and the Coding-Agent Harness Race

• Claude Code gets computer use: Anthropic added computer use inside Claude Code, letting the agent open apps, click through UIs, and test what it built directly from the CLI in research preview for Pro/Max users. The practical significance is closed-loop verification: code → run → inspect UI → fix → re-test, which several engineers called the missing piece for reliable app iteration, especially compared with open-ended desktop agents (Claude announcement, @Yuchenj_UW on the “eyes” unlock, @omarsar0).

• Cross-agent composition is becoming standard: OpenAI shipped a Codex plugin for Claude Code that can trigger reviews, adversarial reviews, and “rescue” flows from inside Anthropic’s toolchain, using a ChatGPT subscription rather than custom glue code. This is notable less as a plugin novelty and more as a signal that coding stacks are becoming composable harnesses rather than monolithic products (plugin by @dkundel, usage thread by @reach_vb, open-source note). Separately, OpenAI shared that late-night Codex tasks run longer, with jobs started around 11pm being 60% more likely to run 3+ hours, which fits the emerging pattern of delegating refactors and planning to background agents (OpenAI Devs).

• Harness quality is now visibly a first-order variable: Theo argued that Opus scores ~20% higher in Cursor than in Claude Code, and more broadly that closed-source harnesses make it hard for the community to diagnose or fix regressions (performance gap claim, closed-source critique). That theme repeated across the feed: model capability deltas are narrowing, while tooling, prompt/runtime orchestration, and review loops still create large practical differences.

Hermes Agent’s Rapid Rise, Multi-Agent Profiles, and the Open Harness Ecosystem

• Hermes has become the week’s breakout open agent stack: Nous shipped a major Hermes Agent update that drove a wave of migrations from OpenClaw/OpenClaw-like setups, with users emphasizing better compaction, less bloat, stronger adaptability, and faster shipping cadence (Nous release, Teknium’s multi-agent profiles, community migration examples, another). The new multi-agent profiles give each bot its own memory, skills, histories, and gateway connections, moving Hermes from “personal assistant” toward a reusable agent OS abstraction.

• An ecosystem is forming around traces, remote control, and self-improvement: Several projects extend Hermes beyond core inference. @jayfarei’s opentraces.ai provides a CLI/schema/review flow for sanitizing and publishing agent traces to Hugging Face for analytics, evals, SFT, and RL. @kaiostephens uploaded ~4,000 GLM-5 Hermes traces to HF. @IcarusHermes described an integration where agents log their own decisions, export data, fine-tune smaller successors on their history, and switch over to cheaper models. @winglian’s ARC adds remote browser-based monitoring/control with E2E encryption.

• Open vs proprietary agent infra is being actively contested: @ClementDelangue explicitly argued that open-source agent tools should default to open-source models, both for privacy and durability. In parallel, vendors are attacking known pain points: @fchollet highlighted PokeeClaw as a more secure OpenClaw-style assistant with sandboxing, approvals, RBAC, and audit trails; Z AI launched AutoClaw, a local OpenClaw runtime with no API key required and optional GLM-5-Turbo.

Qwen3.5-Omni, GLM-5-Turbo/AutoClaw, and the Push Toward Local/Agentic Specialization

• Qwen3.5-Omni is a major multimodal release: Alibaba introduced Qwen3.5-Omni, with native text/image/audio/video understanding, script-level captioning, built-in web search and function calling, and a standout “audio-visual vibe coding” demo where the model builds websites/games from spoken visual instructions. Reported capabilities include support for 10h audio / 400s of 720p video, 113 speech-recognition languages, and 36 spoken languages; Alibaba claims it outperforms Gemini 3.1 Pro in audio and matches its AV understanding in some settings (launch thread, demo thread, additional demo). A useful caveat from @kimmonismus: “omni” here is about interpreting multimodal inputs, not arbitrary multimodal generation.

• Z AI continues to tune for agentic workloads: Artificial Analysis evaluated GLM-5-Turbo, Z AI’s proprietary agent-optimized variant. It scored 47 on the AA Intelligence Index, slightly behind open-weight GLM-5 (Reasoning) at 50, but posted 1503 on GDPval-AA, ahead of GLM-5’s 1408, supporting the claim that the model is tuned for real-world agent workflows rather than broad benchmark maximalism.

• Specialized open models are increasingly the deployment pattern: Several tweets converged on the same thesis: companies will increasingly own and specialize open models on proprietary data rather than rent general-purpose APIs indefinitely (@oneill_c, @ClementDelangue). Supporting evidence ranged from a Qwen3.5-27B model distilled from Claude 4.6 Opus trending on HF for weeks and reportedly fitting on 16GB in 4-bit (Unsloth, @Hesamation) to growing enthusiasm for local runtimes like llama.cpp and MLX.

Local Inference and Systems: llama.cpp at 100k, Flash-MoE on MacBooks, and Web/Serving Toolchains

• Local AI had a symbolic milestone with llama.cpp hitting 100k GitHub stars: @ggerganov’s reflection framed 2026 as potentially the breakout year for local agentic workflows, arguing that useful automation doesn’t require frontier-scale hosted models and that the right portable runtime stack matters more than absolute scale. The post also emphasized the importance of cross-hardware, non-vendor-locked infra.

• Flash-MoE on Apple Silicon drew strong attention: A widely shared post claimed Qwen3.5-397B could run on a 48GB MacBook Pro at 4.4 tok/s using a pure C + Metal engine that streams weights from SSD and only loads the active experts, reportedly using ~5.5GB RAM during inference (summary thread). Related work includes anemll-flash-mlx, which focuses on optimizing only the MoE path on top of MLX, and AI Toolkit’s new Apple Silicon support.

• Web and serving stacks also moved: Transformers.js v4 added a WebGPU backend across browser/Node/Bun/Deno with major perf gains and 200+ architectures. vLLM-Omni v0.18.0 shipped 324 commits, production TTS/omni serving, unified quantization, diffusion runtime refactors, and a dozen-plus new models. On the speech side, Artificial Analysis covered Cohere Transcribe: a 2B conformer encoder-decoder, Apache 2.0, trained on 14 languages, hitting 4.7% AA-WER and roughly 60x real-time transcription speed.

Agent Research: Natural-Language Harnesses, Meta-Harness, Async SWE Agents, and Long-Context via Filesystems

• Harness engineering is becoming a research field of its own: A Tsinghua/Shenzhen paper on natural-language agent harnesses proposed letting an LLM execute orchestration logic from an SOP rather than hard-coded harness rules, a direction that multiple practitioners found mind-bending but plausible as context budgets rise (@rronak_ summary). Meta pushed the idea further with Meta-Harness, a method that optimizes the harness end-to-end over code, traces, and scores rather than just the base model; claims include #1 among Haiku agents on TerminalBench-2 and strong gains in text classification and transfer (@yoonholeee, explainer by @LiorOnAI).

• Async/multi-agent SWE design got stronger empirical backing: The CAID paper from CMU argues for centralized asynchronous isolated delegation using manager agents, dependency graphs, isolated git worktrees, self-verification, and merges. Reported gains were +26.7 absolute on PaperBench and +14.3 on Commit0 versus single-agent baselines, suggesting that concurrency and isolation beat simply giving one agent more iterations (@omarsar0 summary).

• Coding agents as long-context processors is one of the more interesting reframings: A paper highlighted by @dair_ai treats huge corpora as directory trees and lets off-the-shelf coding agents navigate them with shell commands and Python, rather than stuffing text into context windows or relying purely on retrieval. Reported results include 88.5% on BrowseComp-Plus (750M tokens) vs 80% previous best, and operation up to 3T tokens.

Training, Optimization, Evaluation, and Production Case Studies

• Muon got a meaningful systems/math optimization: Gram Newton-Schulz is a drop-in replacement for Muon’s Newton-Schulz step that works on the smaller symmetric XXᵀ Gram matrix rather than the large rectangular matrix, reportedly making Muon up to 2x faster while preserving validation perplexity within 0.01. The work drew praise from @tri_dao as the kind of cross-disciplinary linear algebra + fast-kernel result that actually matters.

• Two practical implementation details stood out: Ross Wightman flagged a subtle but important PyTorch trunc_normal_ misuse pattern in LLM training code: default a/b are absolute values, not standard deviations, so many codebases effectively aren’t truncating at all; he also noted numerical oddities later fixed in nightlies. At the application layer, Shopify’s DSPy case study was notable for economics: one slide highlighted a reduction from $5.5M to $73K/year by decomposing business logic, modeling intent with DSPy, and switching to a smaller optimized model while maintaining performance (follow-up).

• New evals/benchmarks continued to expose gaps: World Reasoning Arena targets hypothetical/world-model reasoning and reports a substantial gap to humans. Tau Bench’s new banking domain adds a realistic 698-doc support environment where best models still only solve about 25% of tasks. Meanwhile, a Stanford-led paper highlighted by @Zulfikar_Ramzan found sycophantic AI can increase users’ certainty while reducing willingness to repair relationships, underscoring that “helpfulness” metrics can obscure socially harmful behavior.

Top tweets (by engagement)

• Claude Code computer use: Anthropic’s release was the biggest technical product launch in the set, and likely the most consequential for day-to-day coding-agent UX (announcement).

• Claude Code hidden features: @bcherny’s thread drew massive engagement, reflecting how quickly expert users are now optimizing around coding-agent workflows rather than raw model prompts.

• Hermes Agent update: The broad community response to Nous’s major Hermes release suggests open agent harnesses have reached a new adoption phase.

• Qwen3.5-Omni launch: Alibaba’s multimodal release was one of the day’s biggest model announcements and especially notable for its practical demos around audio/video-driven app creation (launch).

• llama.cpp at 100k stars: @ggerganov’s milestone post captured the local-first mood of the week: increasingly capable open models plus increasingly capable local runtimes.

Mistral has been on an absolute tear - with frequent successful model launches it is easy to forget that they raised the largest European AI round in history last year. We were long overdue for a Mistral episode, and we were very fortunate to work with Sophia and Howard to catch up with Pavan (Voxtral lead) and Guillaume (Chief Scientist, Co-founder) on the occasion of this week’s Voxtral TTS launch:

Mistral can’t directly say it, but the benchmarks do imply, that this is basically an open-weights ElevenLabs-level TTS model (Technically, it is a 4B Ministral based multilingual low-latency TTS open weights model that has a 68.4% win rate vs ElevenLabs Flash v2.5). The contributions are not just in the open weights but also in open research: We also spend a decent amount of the pod talking about their architecture that combines auto-regressive generation of semantic speech tokens with flow-matching for acoustic tokens (typically only applied in the Image Generation space, as seen in the Flow Matching NeurIPS workshop from the principal authors that we reference in the pod).

You can catch up on the paper here and the full episode is live on youtube!

00:00 Welcome and Guests
00:22 Announcing Voxtral TTS
01:41 Architecture and Codec
02:53 Understanding vs Generation
05:39 Flow Matching for Audio
07:27 Real Time Voice Agents
13:40 Efficiency and Model Strategy
14:53 Voice Agents Vision
17:56 Enterprise Deployment and Privacy
23:39 Fine Tuning and Personalization
25:22 Enterprise Voice Personalization
26:09 Long-Form Speech Models
26:58 Real-Time Encoder Advances
27:45 Scaling Context for TTS
28:53 What Makes Small Models
30:37 Merging Modalities Tradeoffs
33:05 Open Source Mission
35:51 Lean and Formal Proofs
38:40 Reasoning Transfer and Agents
40:25 Next Frontiers in Training
42:20 Hiring and AI for Science
44:19 Forward Deployed Engineering
46:22 Customer Feedback Loop
48:29 Wrap Up and Thanks

swyx: Okay, welcome to Latent Space. We’re here in the studio with our gues co-host Vibh u. Welcome. Thanks. Excited for this one as well as Guillaume and Pavan from Mistral. Welcome. Excited to be here.

Guillaume: Thank you.

swyx: Pavan, you are leading audio research at Mistral and Guillaume, you're Chief Scientist,

swyx

Host

(00:05) Okay. (00:05) Welcome to Lean Space. (00:06) We’re here in the studio with trustee co-hosts, Vibhu. (00:09) Welcome.

Vibhu

Host

(00:11) Very excited for this one.

swyx

Host

(00:12) As well as Guillaume and Pavan from Mistral. (00:15) Welcome. (00:16) Excited to be here. (00:17) Thank you for having us.

(00:18) Pavan, you are leading audio research at Mistral and Guillaume, you’re a chief scientist. (00:23) What are we announcing today where we’re coordinating this release with you guys?

Guillaume

Guest

(00:26) Yeah, so we are releasing Voxtral TTS. So it’s our first audio model that generates speech. It’s not our first audio model. We had a couple of releases before.

(00:35) We had one in the summer that was Voxtral, our first audio model, but it was like a transcription model, ASR. Like a few months later, we released some update on top of this, supporting more languages. Also a lot of table stack features for our customers, context biasing, precision, timestamping and transcription. We also have some real-time model that can transcribe not just at the end of the level.

(00:56) You don’t need to fill your entire audio file, but that can also come in real-time. And here, this is a natural extension in the audio, so basically speech generation. So yeah, so we support nine languages, and this is a pretty small model, 3D model, so very fast, and also state of the art. Performed at the same level as the base model, but it’s much more efficient in terms of cost, and also much, in terms of cost, it’s also much cheaper, only a fraction of the cost of our competitors.

(01:22) And we are also releasing the work that this model is running.

swyx What’s the decision factor?

Guillaume It’s a good question.

swyx

There will be more. Yeah, Pavan, any sort of research notes to add on?

Pavan: But it’s a novel architecture that we develop inhouse.

We traded on several internal architectures and ended up with a auto aggressive flow matching architecture. And also have a new in-house neural audio codec. Which, converts this audio into all point by herds latent [00:02:00] tokens, semantic and acoustic tokens. And yeah, that’s that’s their new part about this model and we’re pretty excited that it’s, it came out with such good quality and Jim was mentioning. Yeah, it’s a three B model. It’s based off of the TAL model that we actually released just a few months back and insert trunk and mainly meant for like the TTS stuff, but they need text capabilities are also there. Yeah.

swyx: So there’s a lot to cover.

I always I love any, anything to do with novel encodings and all those things because I think that’s obviously I creates a lot of efficiency, but also maybe bugs that sometimes happen. You were previously a Gemini and you worked on post training for language models, and maybe a lot of people will have less experience with audio models just in general compared to pure language.

What did you find that you have to revisit from scratch as you joined this trial and started doing this? At least

Pavan: when it comes to, for, I think the, there are two buckets, I guess the audio understanding and audio [00:03:00] generation. The audio understanding, like the walkthrough models that Kim was mentioning that we released earlier.

The walkthrough chat that we released I think July last year, and the follow up transcription only, models family that we released in January, that would be one bucket, and the generation is another bucket. I think. You can also treat them as a unified set of models, but currently the approaches are a little different between these two.

To your question on how audio is fed to the model? In the understanding model, it’s very similar to actually Pixar models that we also released,

swyx: yes.

Pavan: That’s

swyx: amazing.

Pavan: It was pretty, I, that was the first project I worked on after joined Misra. It was pretty, pretty nice. And Wtu was very similar in spirit.

I guess So we feed audio through an audio encoder similar to images through a vision encoder, and it produces continuous embeddings and which are fed as tokens to the main transformer decoded transformer model. Yeah. On the model output is just text. So on the output side, there is nothing that needs to be done in these kinds of mode.

I [00:04:00] guess the interesting part of what the generation stuff is, the output now has to produce audio and. The approach that we have is this neural audio codec, which converts audio into these latent tokens. There is a lot of existing attrition and a lot of models which are based off of this kind of approach.

And we took a slightly. A different, design decisions around this. But at the end of the day, the neural audio product converts audio into a 12.5 herdz set of latents. And each latent is, has a semantic token and a set of acoustic tokens. And the idea is that you take these discrete tokens and then feed it on the input side.

There’s several ways to use this at each frame, but we just sum the embedding. So it’s like having key different vocabularies. Combine all of them because they all correspond to one audio frame on the input side. The output side is the interesting part on the output side, the, it’s not the, I don’t know if it’s the most popular, but one.

Popular technique is to have a depth transformer [00:05:00] because you have K tokens at each time step, like with a text, you just have one token at each time step. So you just do predict the token from the vocabulary with, yeah, with just, you get probability

swyx: This’s a very straightforward text. Very

Pavan: straightforward.

swyx: Yeah.

Pavan: But if you have K tokens, then the name thing would be to predict all of them in paddle. That doesn’t work. At least that doesn’t work that well because audio has more entropy. And the, one of the techniques people use is this depth transformer where you you almost have a small transformer, or it can be L-S-T-M-R in as well, but people use transformers and you predict the K tokens in auto aggressive fashion in that.

So you have two auto reive things going on.

Pavan: So the thing we did differently is in, instead of having this auto aggressive K step prediction, we have a flow matching model. Instead of modeling this as a discrete token set we trained the codec to be both discrete and continuous to have this flexibility.

So we did try the discrete stuff too, and which it works well, but the continuous stuff works just better. So yeah, we took this flow matching, so the, it’s a flow [00:06:00] matching head, which takes the latent from the main transformer and like kind in fusion, it’s denoising, but in this flow matching itself, velocity estimate.

So you go from this noise t all the way to there. Audio latent, which corresponds to the 80 millisecond audio and then, which is sent through the work order to get back the 80 millisecond audio frame.

swyx: Yeah. Is this the first application of flow matching in audio? Because usually I come across this in the image.

Pavan: Yeah. Actually, in some sense there are models flow matching models in audio, but I think this specific combination I could be wrong. There could be somewhat. No. I haven’t seen. I haven’t seen much work in this, so I think it’s novel and a lot of it’s just a way bigger community, so they, I think they pioneer a lot of these diffusion flow matching work, and it’s interesting to adopt some of the ideas there into audio and,

swyx: yeah.

Pavan: Yeah, I’m, personally that’s the think part which is trying out about. One of more meta point is unlike text, even in vision, I think this is true, but in [00:07:00] audio step literature that there is no.

Winner model, yet there is no, okay, this is the way you do things. It’s it’s still by, I think people are still iterating and figuring out like what’s the best overall recipe. I guess the idea. Pretty sure there are models which are also completely end-to-end, like NATO audio. NATO audio, but it’s still not come to a convergence point where this, the right way to think that.

That also makes. A space pretty exciting to explore.

Vibhu: What are some of the ways to look at it?

Vibhu: There are ways where you can do diffusion for audio generation, but if you want like real time generation, that’s a big thing with the approach I’m assuming that you took. Yeah. And also like how do you go about evaluating different axes of what you care about, yeah,

Pavan: good point. I think we so you can do just flow matching diffusion for the whole audio. We didn’t even go down that path because one of the main applications is voice agents and we want real time streaming, and that’s the use case. That’s not the only use case, but that’s one of the primary use cases we want to get to.

So we [00:08:00] picked the auto aggressive approach for that. And within the auto aggressive space, again, you can do chunk by chunk or you can do so we picked the. I think at least personally prefer the operations, which are the simplest, and so we try to see, can we just add audio as just another head to our regular transformer decode model because that kind of makes it easier for eventual end-to-end modeling of audio text native modeling.

Yeah. And it works pretty well. So I guess we went with that and we tried a little bit, but the flow matching head itself, like we had a discreet. Diffusion kind of approach, which also works well, but the flow matching work better.

swyx: I was just curious about how you also think about this overall direction of research.

Do you basically, when you work with the audio team, do you set some high level parameters and then let them explore whatever, or how does it work between you guys?

Guillaume: No I think the way it works is that we are the, we are prioritizing together, I think, what are the most important features because there are many things we can do [00:09:00] in audio.

Yeah, I think we try to. These are like how we should do things, for instance. Ultimately what we want to do is to build this through duplex model, but we are not going to start this start there directly, I think is. Some of the project people are doing, but

swyx: just to confirm, full effects means it can speak while I’m speaking or,

Guillaume: yeah.

Okay. Audio. Yeah. Yeah. So intimately we’re going to get there, but for us it was, we decided to take it like a step by step. So we start with whatever is the most important. I think support customers, which is the transcription is the most popular use case. Then the speech generation, Soviet time, just a bit before that.

And then actually to be like more, but try combining everything all together. But but yeah, we thought it was also important to like separate things and optimize each capability one by one before we

swyx: measure of that together. And the super omni model. But

Guillaume: very interesting because as Par said, it’s when you work on some other domains of this airline and everything, there are many areas where I think it’s not as interesting.

For instance. Many places, it’s essentially just around data or like creating new environments on a lot of kind [00:10:00] of easy things. But things were, I think the research is maybe not as interesting. Were in audio. There are so many ways to actually build this model. So many ways to go around it. That’s the sense I think is really interesting.

And what we also tried for speed generation is that we tried multiple approaches. What was interesting that even though they were extremely different, they under the big know the particles but the for matching turned out to be quite more natural. So we are happy with this.

swyx: Is there intuition why it maybe like flow matching is just models speech better in some natural fundamental, latent dimension?

Pavan: No, I think the main thing is e even at a particular time step, there is a distribution of things.

swyx: Yes.

Pavan: To be predicted like the way you inflate. So you already know the word that you’re speaking and Yeah. The intake space, let’s say the word maps register a single token for simplicity.

In most cases it does. So there is not a lot of so you just pick the word, but with within audio, even the same word could, even with your own voice, could be inflicted in so many different ways. And I think [00:11:00] any approach which like models this distribution and. And flow matching is one, one of the take.

It’s not the only one at all, but it’s a one which works pretty reasonably well. I think that’s better. So you have to pick across several different, the intuition I have is it’s, there are some, several different clusters each corresponding to some specific way you would inflict, pronounce that thing.

And you can’t predict the mean of it because that corresponds to some blurred out speech or something like that. But you have to pick one. And then like sharp

swyx: conditional inference.

Pavan: Yeah, exactly.

swyx: Is that all covered under disfluencies, which is I think the normal term of art. Pauses intonations. By the way, I have to thank Sophia for setting all this up, including like some of these really good notes because

Pavan: Yeah.

swyx: I’m less familiar with the audios for me.

Pavan: No. I think dis dismisses are definitely one such Eno defenses is more like

swyx: which is arms are.

Pavan: Yeah, arms. And also repeat like you like,

swyx: yeah.

Pavan: You do this full of words, your thinking, so you repeat the word.

swyx: Okay. Whereas intonation is like a diff, it’s up up [00:12:00] speak and all this.

Okay.

Pavan: Yeah. So I think there is a lot of like entropy. And modeling it as a distribution. And a, any technique which helps with it and the depth transformer is a conditional way of modeling this. And Transformers actually really good at it, even though that’s a mini transformers. So I think that worked pretty well too for us too.

It’s just that the main concentration is when you have a depth transformer. If you have K tokens, you need to do K auto steps, right? Even though it’s a small thing, it’s K steps, which is very vacant, say heavy, but flow matching. We were able to cut it down significantly. So we are able to do the inference in quad steps or 16 steps and it works pretty well.

And there are more normal techniques to bring it down even further to like, in extreme case, one step like we’re not doing it yet, but it at least the framework, LEDs itself to more efficient and Yes.

swyx: And the image guys have done.

Pavan: Yeah.

swyx: Incredible work guys. Yeah.

Pavan: It now you just. Send a prompt and you get an image.

swyx: Yeah. Surprisingly not enough. I think image model labs use those techniques in production. I think it’s, I feel like it’s a lot of research demos, but [00:13:00] nothing I can use on my phone today.

Guillaume: The thing, there’s a thing that would be interesting here is that since, indeed I’ve been so much sure that has been done in the vision community compared to radio dys, stomach, I think there are so many long infra Yeah.

And there are so many things we can do to actually improve this further. So it’s our first version, but we have so many ways to exist, much better and much more efficient, cost efficient, so

swyx: yeah.

Guillaume: So really it’s not a new field at all, of course, but there are still so many things that can be done.

Perfect. It’s

swyx: nice. I should also mention for those who are newer to flow matching, I think the creator, this guy’s name is Alex, he’s done I think in Europe’s maybe two Europes as ago. There was, there’s a very good workshop. There’s one hour on like this matching is I would recommend people look that up.

That’s the other thing, right?

swyx: The efficiency wise, like I, I imagine like the reason is open weights the reason you pick 3.6 B backbone it you are 3.4 B you are, try to fit to some kinda hardware constraints. You kinda fits some kinda basic constraints. What are they?

Guillaume: Not necessarily, I think something we care about in our model that they’re efficient.

So we have a [00:14:00] lot of separate model, for instance. So we have this that is very small, very efficient. We also have a small OCR model that is available. Good, highly efficient as well. And I think on a project maybe there, I think companies are going to take is to have a coverage general model that will do a bit of everything.

But that is also going to be expensive. On here. What want say is if you care about this specific use case, if you can actually use this model, it just does that. It’s extremely good at it. Survey, very efficient. That’s why we can actually add. We do, but also OCR that are like really good at that.

And that would be much more cost effective factors and the general model that will contain a lot of capabilities you don’t really need. So yeah. So we’re doing like general model, but also like more customized model. This,

Vibhu: how does it compare to other TTS models? It’s, we are going follow open wave.

We’re just dropping it. I think it’s pretty good.

Pavan: Yeah, I think it’s pretty good. Like it, it’s definitely one of the best. For sure. It’s probably I would say it’s the best open source model, but

Vibhu: decipher themselves.

swyx: Yeah.

Vibhu: Why now? How does it fit into broader ral vision? How do you see voice agents?

How do you see voice? I think every year I’ve heard, okay, you’re a [00:15:00] voice. You’re a voice. There’s a lot of architectural stuff. There’s a lot of end time that see it, your solving, but where do you see voice setting?

Guillaume: We had so many customers asking for voice. That’s also why we wanted to build it.

What’s interesting in this domain is that. In a sense, if you take something simple like transcription it doesn’t seem like something that should be very hard to do for a model. It’s essentially, it’s pattern recognition. It’s classification on this. Models are very good at classifying, right?

Or nonetheless, when you talk to them it’s not there yet, right? It’s not, you don’t talk to them the same way you talk to a person. On something, maybe people don’t realize it. It’s in English it’s still much better than in any user language, even compared to French instance. If you talk to this million in French, when you see people talking to this they’ll talk very slow.

They’ll articulate as much as they can. So it’s not natural, right? We’re not yet to this. And I think, yeah, maybe the next generation will not know this, but yeah, I think people that. But our edge will actually always keep this bias speaking very slowly when they talk to this model. Even if maybe, probably in a couple of years, maybe next year it’ll not be necessary anymore.

But yeah. But what’s interesting is to see that yeah, even for like languages [00:16:00] like yeah, French and Spanish Germans that are not no, no resource on religion. You have a lot of audios there on still it’s not as good. And I think a consequence. Because then for this, I suppose just is not as much energy, as much effort that has been put done in some other mod that for some vision or like coding.

But but yeah, there’s still a lot of progress to be done. I think it’s just a question of doing the work and it’s clear path I think to get there.

Pavan: It’s a little fascinating because I worked on Google Assistant I think while back at this point, but it’s, I think it’s, it like when you take a step back, it’s fascinating.

It’s not that long ago. It was like four years ago or five years ago, and it’s now it’s completely audio in, audio out and the function calling and the whole thing happens completely end to end. And in a very natural,

swyx: yeah,

Pavan: natural way and still ways to go. Kim was telling, even despite all the previous, it’s not like you’re speaking to a person.

When you talk to any of these agents, bots, or voice mode kind of situation, it’s still like a gap. I think that’s the great part and I feel like with even the existing [00:17:00] stack, we should be able to get to this very natural speech conversational abilities soon enough I guess.

And we’ll also hope. I get that

Guillaume: on this kind of the next step, right? Because when you talk to these agents, like usually people are just writing to them and sometimes they’ll this very clear, for instance, you are, you want to write code, but you are, you have a very clear idea of how you want the model to implement what you in mind.

But so here you are able to spend a lot of time writing. So it’s not really efficient on audio is really like a natural interface that is just not there yet, but I think it’s just gonna be the place.

Vibhu: How’s it like building, serving, inferencing, like we see a lot about, it’s very easy to take LMS off the shelf, serve them.

Fine tuning, deploying. I know you guys have a whole you have Ford, you have a whole stack of customizing, deploying. Is there a lag in getting that. Like distribution channel. Are you helping? There is. So like prompting, lms, you can have them be concise, verbose, all that.

They’re built on LM backbones, these models. How do you see all that?

Guillaume: Yeah, I think this is a lot of what we’re doing with our own customers. Very [00:18:00] often they come to us, so it’s for different reasons. I think one reason is sometimes they have this lot of privacy concerns.

They have this data that it’s very sensitive. They don’t want data to leave. The companies, they wanted to stay. Inside the company. So we have them deploy model in-house. So either on a, either on premise or on private cloud. So they’re not worried that it’s given to a third party on the there some leakage.

Sometimes they have this kind of many companies have this different, sensitivity of data they have like sometimes channel chat can send it to the cloud has to stay there. So then it creates some kind of heterogeneous workflows where it’s annoying. You cannot send some data to the cloud.

This one you can, so here, when we actually deploy the model for them, they don’t have this consideration. They are like not worried that, this is going to leak. Everything is much easier. So we help them basically do this on the, so it’s one of the very proposition. But but the other is very often, when customers use this off the shelf close model, but very sad is that they are not leveraging, these data that have been collecting for four years or something for decades.

So much data. Sometimes it’s trillions of tokens of [00:19:00] data in a very specific domain. Their domain, which is data that you’ll not find in the public, on the public internet. So data on which, like close model, we actually not have access to one, which that’s going to be really good. So if they’re using like closed source models are basically not benefiting from all these insights.

All these data they have collected three years, they can always give it into the context that in France, but is never as good as if you actually train the modern analysis. So yes, that’s basically what we help them to do. We actually provide them some purchase, basically what we announced at GTC this week.

So we provide them with this, it’s basically like a platform with a lot of tools to actually help them process data. Trained on that. Yeah, it’s actually the same thing that we’re using in the science team. So it’s actually very better tested infrastructure, like a lot of efficient training cut base.

For a quality pre-training like a fine tuning, even doing S-F-T-I-L. So we help them do this using the same tools as what our science team is building is using. So since it’s tools that we’ve been using for two years now, it’s really better tested. It’s really sophisticated.

So it’s the same thing. We are giving to them, giving the company the same thing [00:20:00] that what are same still using internally actually build their own ai and it makes a really big difference. I think sometimes customers. And many in general don’t realize how much better the model becomes when you fine tune it on your own data.

And you can have a, your model is here. You start from there. You have a cross source model, which is sort here, but if you actually fine tune it can actually really go much further than this. And then you have a very big advantage. The model is trained on your entire company knowledge, so it knows everything.

You don’t have to feed like 10 K tokens of contact at every query. So it’s it’s much easier. It’s a bit, I think using a closed source model is really sad because it basically puts. You are not leveraging all this data and you are going to be using the same model as all your old competitors when you’re actually using, everything you have been collected for years, which is really valuable.

So yeah. So we help basically customers do this. We have a lot of solution I mean deployed for engineers that go in the company that basically look at the problem customers are facing to look at what they’re struggling to do what we should do to solve it. So we help them solve them together.

So it’s I think our approach is a bit different, but here. [00:21:00] Some of their companies and competitors, it’s, we don’t just release an endpoint on sale, do some stuff on top of that, or we don’t just give a checkpoint. We really look very closely with customers. We look at the issues they have, we had them solve them.

We really make some tailored solution for the client are facing. Some example are also going to be, sometime we have some customers. They really wanted to have a really good model, really performance on some, like Asian languages on the, if you take some of the shelf models, they can speak it, they can write in this language, but it’s not amazing.

This language would be like maybe zero 1% of the mixture. So it has been included during training, but very little. So what we did here is upgrade. We trained a new model for them, but so this language was 50% of the mix, so it’s much, much stronger. It knows of the dialects, it knows the, so it’s yeah.

So it’s some example of things we can do and it’s really arbitrary, custom. I think you had some of their customers, for instance, they wanted some. They wanted some 3D model that can do audio with a very good function cable. So something you wanted to put in the car in particular, they wanted this to be offline because in a car you don’t necessarily have access to internet.

So [00:22:00] yeah. So here we can actually build the solutions. There is no like model out of the box on this. In the internet you have this very, you have this very general model generalist, like he’s strong model. But for things like this, they always want at specific solutions and on some other reasons.

Sometimes they come to us is because, like they, they experiment with some closed source model. They get some prototype. They’re happy with what they build. They, it works well. They’re happy with the performance, and then they want to go to production and then they analyze. But it’s extremely expensive.

You cannot push this. It’s so then they come back to us on this. They can help us build the same thing as this, but using something much cheaper on here. And here we can sometime be something 10 x cheaper by just functioning a model and it’ll be better OnPrem on their old server and also much cheaper as well.

So yeah,

swyx: that’s the drop pitch right there. Take all the

money.

Vibhu: And outside of that you do, we do put open wave models so people can do this themselves. I feel like not enough people go outta their way.

swyx: They’re not going to, they’re gonna ask them to do it as the expert. I

Guillaume: think initially we didn’t know, [00:23:00] we wanted completely short at the beginning of the company because, I think our study was not exactly the same as what it is today, but what we underestimated initially is the complexity of deploying this model and connecting them to everything to be sure it has access to the company knowledge on the, and it was, yeah, on, we were seeing customers struggling with this, but it was even, that was three years ago and no, things are much more complicated because now you don’t just have, text on SFT on a simple instruction following.

You have reasoning like your agents, you have like tools. You have a multimodal audio, so it’s much more complicated than before. And even back then it was hard for customers. So they really need, have some support and this is why actually providing like always some four D position as well. The process

swyx: I’m curious is there also voice fine tuning that people do?

Pavan: So in this forge we also have a say unified framework. And the hope is like the er speech to text that we released earlier this year. And even the ER chart that we released last year. And I think a big people, I think there’s a big, rich ecosystem [00:24:00] of people fine tuning whisper, and people want the same thing with w so it’s much stronger than Whisper.

And yeah, the the platform offers that kind of fine tuning yeah, which could be any kind of fine tuning. Like for instance, even sometimes people want to support new languages to this, which are tail languages, which we hope to cover. Certain natively, but if there is a language where you data and you want to frank you, I think this is a good use case.

Or the other use cases, you, it’s the same language, like even English but it’s in a very domain specific way.

swyx: Yeah. Terminology, jargon, medical stuff.

Pavan: Exactly. And also there’s specific acoustic conditions like there’s a lot of noise or the, and. The model will do decently in most conditions, but you can always make it better.

And that those are some of the use cases where you can improve it e even further. And that’s one good use case for this and for text to speech. We’re just releasing it so we’ll have support for that soon too. I think it’s similar use case.

Pavan: It’s little different the kind of things that you want to extend a [00:25:00] text to speech model to, which could be like voice personalization, voice adaptation for enterprises.

Many enterprises need very specific kind of tone, very specific kind of like personality for this kind of voice. And all of those are like good use cases for fine tuning.

swyx: This one I was gonna ask you, we never talked about cloning voice clothing here. How important is it, right?

Like I can clone a famous person’s voice. Okay. But

Pavan: the main use case would be like for enterprise personalization, like enterprises need like a lot of customization. You don’t want the same. Voice for all the enterprises. Each enterprise want a customized, specialized something which is representative both their brand and also their, I guess safety considerations and the use case I think the kind of thing that you would deploy as a empathetic assistant in the context of a healthcare domain would be very different from the kind of thing that would be in a customer support bot and would be different from like more conversational aspects.

I think those are the. [00:26:00] Customizations you would expect from enterprise. And that’s the main use case, at least from our side.

Vibhu: My, my basic example is you don’t want to call to customer services and have the same exact voice. It’s just, it’s gonna be weird.

Vibhu: But also on the technical side of this, so there’s like a few things in TRO that I thought were pretty interesting.

He’s a big fan of this paper. Oh, he said very good paper. He said this is the best SR paper he’s ever read. Yeah. I’ve hyped up this voice paper enough. We covered it. Somewhere, but a big thing. So Whisper is known for 32nd generation a 32nd processing. You extended this to 40 minutes. There was a lot of good detail in the paper about how this was done.

Even little niches of how the padding is. So it’s very much needed. You need to have that padding in there, the synthetic data generation around this. I’m wondering if you can share the same about the new speech to text, right? Text to speech. So how do you. How do you generate long form, coherent?

How do you generate, how do you do that? And then any gems? Is there gonna be a paper?

Pavan: Yeah. Yeah. They would be a technical report. Okay. Yeah. I think I could have a lot of details.

Pavan: But me I think the [00:27:00] summary of it, actually, some of the considerations in this paper were, because we started with the wipa encoder as the starting point, and now we have in-house encoders, like the bigger time model, for instance, which we released in January.

Also release a technical report for that real time model as well, which is this dual stream architecture. It’s an interesting architecture. You should check it out. And there we have a causal encoder and I don’t think there’s any strong, multilingual causal encoder out in the community. So we thought it’s a good contribution.

So that’s one nice encoder there. Other people want to adapt. That’s a good end code. And we train it from scratch. I think her. Post stack is now mature enough that we are able to train super strong ENC codes. And some of these considerations, like spatting and stuff, is a function of the Whisper ENC code.

And now that we train encoders, inhouse the design concentrations are different.

Pavan: And for the question on text to speech, I think that’s also leans onto the original auto aggressive decoder backbone. I think, it says very, almost identical considerations. I think the long context in it’s not even long con, [00:28:00] so the model processes audio at 12.5 herds, so one second maps to like 12.5 tokens.

So I think one minute is like 7.8 tokens. You can get like up to 10 minutes in eight K context window and get half an hour and 30 K context window. So that’s and 30 2K context is something that’s we are very comfortable training on. We can extend it even much longer. 1 48 K. Okay. You can naturally see how it can extend to even our long generations.

Yeah. We need the. Like data recipe and the whole algorithm to work coherently enough through such long context. But the techniques are some way very similar to the text, long context modeling. And the key differences, it’s just doing flow matching order regressively instead of a text open prediction.

swyx: Okay. I think that was most, most of the sort of voice questions that we had. But

Vibhu: I have a big question on Mr. Al, Mr. Small. So what is small? How do we define [00:29:00] small? What is this? What is this? I remember the days of Misal seven B on my laptop. The snuff fitting on my laptop. I could run it on the big laptop, but

Guillaume: it’s just additional.

Question of terminology, like here what we did, baseball is north active parameters, but it’s true. Really not give it another name, but yeah, we could have called it medium, but only, I,

I suppose it’s a model that we released mixture of experts. It’s a model that combines different model before which we were doing the same, is that we had one model, general model for Israel. Doing instruction following, were like a separate model that was Devrel trial. So qu coding specify specific to code with another model for Reason Maal.

So this were separate artifacts built by different team at trial on what we’re doing is basically merging all of this. It was, you had pixel trial was the first vision model. We was like a separate model on the way we do things internally is that we have one team focus on one capability, build one model.

On the means mature, mature enough, we decide to merge this into the [00:30:00] matrix. But here it was the first time we basically match all of this into one. But there are some other things we did at first time to merge time, for instance, like more capabilities or function coding I think would be, are, it’s going to be much, much better in this trial, small platform.

But but yeah, so it’s our latest model on the working is,

Vibhu: and yeah, key things is it’s very sparse. Six, be active pretty efficient to serve. 2 56 K context. Yeah,

swyx: I think what’s interesting is just this general theory of developing individual capabilities in different teams and then merging them.

Where is this going gonna end up?

Vibhu: Like we’ve seen the five things put together in this. Yeah. What are the next five teams?

swyx: I think actually OpenAI has gone away from the original four Oh. Vision of the Omni model. This was what they were selling. All modalities and all modalities out.

But I feel like you might do it.

Guillaume: I think there’s some mod where it’s not competitive use, for instance for audio. For audio here, if you want to do transcription, I think it makes no sense to use a model. If you just want to trans tech it, it’ll be very inefficient. If you want to do audio, you probably just want to be the [00:31:00] one VR 3D model performance essentially

swyx: the same.

It’s going to be incredibly cheaper. So here, that’s why we want

Guillaume: to have a separate but just does this. Yeah, I think the question is just, yeah. If you are to, to your model. By speech and you asking like a very complex questions on how you do this on the, just to cascade things. Do you want to put a d in a model that has like a one key around it?

It’s like a, not a competitive discussion, I think unaware if you doing into the direction, but that’s possible. Of course. But yeah. But I think for us, the next capabilities we want to try to integrate into these models when we are going to be yes, like marketing or no reasoning better, I think more capabilities that people don’t talk too much about, but at high bottom, I think for our customers in our, on different industries, for instance, things are around like a legal computer.

I design all these things that is this males out of the box are to put at that. Because people, if you don’t prioritize this, there is not like too benchmark on that. But

swyx: this done how to

Guillaume: make this good and this just start to do the work. Extracting some that processing it [00:32:00] expression. So yeah.

But we are offering the imagine to this.

swyx: I think for voice. Yeah. The key thing I think over maybe like the last year or so with VO and gr Imagine and all these things is joining voice with video, right? Which people don’t understand spatial audio because like most TTS is just oh, I’m speaking to a microphone in perfect studio quality.

But when you have video, like the voice moves around.

Pavan: That’s true. The constitution was a little different in the sense that there it’s like a a standalone artifact where you get the whole thing and you consume it. But in a conversational setting, it’s a, you need the extreme low latency.

swyx: Yeah,

Pavan: streaming would be one of the primary concentrations.

swyx: You can build a giant company just doing that, right? So you don’t need to do the voice, but I was just know on the theme of merging modalities, that is something I, I am like, wow. Like I didn’t, everyone up till, let’s say mid last year was just doing these like pipelines of okay, we’ll stitch a TTS model with a voice thing and a lip sync [00:33:00] thing and what have you.

Nope. Just giant model. Yeah.

Vibhu: I have a two part question. So one is, it’s still open. It seems like open source is still very core to what you guys do and I just have to plug your paper. Jan 2024. This is the one trial of experts like. Very fundamental research on how to do good.

Moes paper comes out very good paper for anyone. That’s just side tangent. No.

swyx: This thing caused, we bring back, eight by 22 was like the nuclear bomb for open source. I think it takes Shouldn be more seven B more. Yeah. Yeah. But this is a bigger opposite than me.

Yeah. Yeah I don’t remember this. I remember, I don’t think it was January, right? It was like new reps it was, it dropped during new reps and everyone in Europes was December of 25th, I think. Yeah. The model was did as well.

Vibhu: It’s just a little update probably.

swyx: Yeah. No, but you have a point to make.

Vibhu: No, you gotta check that. But then, I just want to hear more broadly on open source for you guys, and when you had asked earlier [00:34:00] about what’s next, what are the other, side tapes working on you. You put out Lean straw. This,

swyx: it’s not necessarily surprise. I was like, I don’t, this doesn’t fit my mental model or Misra.

Guillaume: Yeah. First for open source in general, I think it’s really something which looks to the January of the company. I think we started it per once, is we so we have open sourcing with, since the beginning and even before this. So before this, so me and Tim were at Meta, we released LA and I think what was really nice.

To see that before this, for most researchers like universities, it was impossible to work on elements. There was no alien outside. And if you look at many of the techniques that were developed after, for instance, was open source all this post-training approaches like even DPOD, like preference optimization, all of this were done by people that had access to this portal.

And it’ll have been impossible to do without this. So it’s really making sense, move faster. So we really want to contribute to this ecosystem. I think like the deep and also like very lot of impact. All these papers that are I think in the open source community are really helping the science community as a whole to move faster.

So [00:35:00] we want contribute to this ecosystem. That’s why we’re releasing very detailed technical reports. So ma trial and our first reason model, and ation, lot of results, things that work, things that did not work as well. Think helpful on the, yeah, so for the audio model also to share a lot of details, share of them for real time model.

And the, yeah, so we really want to continue this, basically belong to this community of people who share science. I think we really don’t want to be, leading in a world where the smartest model, the best models are only behind, close doors. Only accessible to a shoe companies that we, as a power to decide we can use them on it.

I think it’s a scary future. We don’t want to live in, we really want this model to be accessible to anyone that want. Intelligence to be used unaccessible by anyone who can use it. So yeah, so that’s why we are pushing this mission and source model. Yeah. So not, so yeah, no strategy. So it’s open source, not the first model, so not the best on the Yeah.

Guillaume: LIN trial I think is also one step into this direction. So it’s yeah, a bit different than what we are usually releasing. But we have a small team internally [00:36:00] working on them. Formal proofing, formal math. So I think a subject we care about in general and we were working on reasoning. I think we started too early before doing reasoning without LMD is very hard, especially when you work with formal systems because the amount of data you have is negligible.

It’s addressable community of people writing like formal proofs. But the reason why we like it is because I think there is if you look at what people are doing with reasoning, is there, the problems that you can use. Are usually going to be problems where you can verify the output. So for instance, all this ai ME problem where the solution is a number between 100, like a thousand.

So you can verify, compare this with a reference or it’s an expression. You can actually compare the output expression generic with the reference. But there are many, most of them have problem and most of the reason problem. There is no like way to easily verify the solution. If the question is show that F is continuous, cannot compare in the reference, right?

If it’s a probe that this is true or probes is properties, there is no way to. You cannot act, simply verify the correctness of your proof. So it’s hard to apply the, there is no referable reward here. So [00:37:00] what you could provide is of course, like a judge and judge that will look at your proof. But it’s very hard and it’s very, you could do certain, some reward hacking happening there.

So it’s difficult. You could provide like a reference proof, but then there are also many ways to prove the same thing. So if the model says give negative reward because it’s a different poop, maybe it was still digit proof, just different. So it’s not going to work well. What’s nice with lean and with formal probing is that you don’t have to worry about this whatsoever.

We just,

swyx: they’re all function is largely compiles in lean is functionally the same. Exactly.

Guillaume: It’s like a problem if it compiles it’s correct. It’s very easy. And you can apply this and then you can,

swyx: it’s just way too small. So no human will actually go and do it.

Guillaume: Yeah, that’s exactly.

It’s the only people can do it. It’s like a very small committee of people doing a PhD on that. So it’s super small. And it’s sad because it’s actually very useful on not just mat, but also in software verification. So for instance, software verification today. So tiny market. Very few industries work on this and we need that.

It’s usually going to be like companies like building airplanes, air robotics,

swyx: like

Guillaume: things [00:38:00] where they absolutely want to be sure. Life depend on this, but it’s very rare that people formally verify the correctness of their software. But I think one of the reasons for this is simply that it’s just hard to do.

swyx: Are you think of TLA plus? It’s the language that some people do for software verification? No. That people use in a ference, but but yeah, it’s the reason I think why people don’t use it more and why this industry is not as big as could be is because it’s very hard. But now with cutting edges that are there, it’s going to be very different.

Guillaume: We’re going to see much more of this. So I think yes, industry there is going to be much larger in the future that we, these models. So yeah. Here also anticipating this a little bit, we wanted to work on that because it’s proving like a math theory and like a, essentially the same tools.

swyx: Yeah.

swyx: One of my theories is that because the proofs takes so long, it’s actually just a proxy for long horizon reasoning and coherence and planning. Maybe a lot of people will say okay, it’s for people who like math. It’s for being okay. It’s like a niche math language. Who cares? But actually, and you use this as part of your data mixture for [00:39:00] post-training and reasoning, actually, it might spike everywhere else.

Yeah. And I think that’s un under explored or no one’s like really put out a definitive paper on how this generalizes.

Guillaume: Yeah, absolutely. And

Pavan: I think even

Guillaume: that’s what we’re seeing already. For instance, you should do some reasoning on math as then the American should do reason even.

Yeah. In the early stage. So we, the, there is some transfer, some sort of emergence that happens. And I think some, it’s also interesting, it’s not just I think the topic in general, but it’s, there is a lot of connection with this on including agents because. Sometimes the model can see like a three that it has to prove it’s very complex, but then it can take the initiative to say, I’m going to prove this three lr.

I’m going to suggest three Rs, and I’m going to in parallel prove each R. So three of them in parallel with sub agents, but I’m also going to prove them in theory and the three tool so you can do this also. Pretty interesting. You can, even if you fail to put one of the LeMar, you can actually, maybe you succeed to put the normal lema too, so you get some possible reward here.

So it’s a bit less Spartan issue, just get to zero one for the entire thing. [00:40:00] So it’s pretty interesting. I think we can actually,

Vibhu: yeah, it’s also an interesting case just for specialized models in general, right? Like the cost thing you show is pretty interesting yeah, similar score wise, you are, thirty, seventy, a hundred fifty, three hundred bucks.

Smaller.

swyx: I think cost is a bit unfair, right? ‘cause this one is at like inference cost. It’s always there on top with their margins on top of it. But, we don’t know anything else, so we gotta figure it out.

Vibhu: Okay.

Vibhu: I did wanna actually push on that more. Not on cost, but you mentioned about, okay, it’s a great way to have verifiable long context reasoning.

What are other frontiers that, I’m sure you guys are working on internally, there’s a lot of push of people pushing back on pre-training. Scaling, RL pushing, compute towards having more than half of your training budget. All on rl. Where are you guys seeing the frontier of research in that?

Guillaume: You mean the

Vibhu: just in foundation model training in the next, one thing that you guys do actually is you do fundamental research from the ground up, right? So you probably have a really good look at where you can [00:41:00] forecast this out.

Guillaume: Yeah. I think for us we’re still working a lot on the pre-training side.

I think we are very far from situational, the pre-training. I think ML four preprinting will be like big step compared to everything we have done before. So we are pretty excited about this. And I think on the other side, I think now we have more and more to think about this algorithm that will actually support this very long trajectories.

I think when it was, for instance, GRPO for it doesn’t really work this any bit of policy. Which was okay initially because you are solving math problem that can be solved in like a few thousand tokens. So the model can alize them pretty quickly. So when you do your update, the model is never too far off.

It’s never too far off. But now when you are moving towards this kind of problems where certain takes hours, like six hours to get a reward, then your model is co pick places. So you have bi new infrastructure that supports this, but also new A, so now everything we’re doing internally, we’re trying to. Build some infra that we actually anticipate is what we have in six months, one now, which is this extremely no scenarios on the, I think when we started Missal, part of me and [00:42:00] we wanted to, is very nice under element where people are there, they can do research, they like with a lot of resources.

So it was nice. I think things changed a lot when I think when J Pity came out. I think after that I think was. This one is same again. But but yeah, but it was nice. And I think we also want to work part of this descrip before

swyx: coming to the end.

swyx: We’re just, obviously, I think you guys are doing incredible work.

You’ve, they are a very impressive vision for open source and for voice. What are you hiring for? What’s the what are you looking for that you are trying to join the company?

Guillaume: Yeah, so we are hiring a lot of people in our sense team. We’re hiring, in all our offices. So we have a, our H two is in France in Paris.

We have a small team in London. We like a team in Pato as well. Co we open some offices in in SAU, in Poland. So one in Zurich. We also like some presence in New York as well on Sooner one in San Francisco. So we all bit either way also like hiring remotely. So we’re going the team trying to hire like very strong people.

I think we want to stay, so the team is not. Instead of fairly small team. [00:43:00] But I think we want to keep it that way. ‘Cause we we find it quite efficient. So like a small team they agile so yeah.

swyx: Okay.

swyx: Let’s focus on science and the forward deployed. We actually are strong believers in science.

We started the our new science pod that focuses specifically on the air for science. What areas do you think are the most promis.

Guillaume: What we’re pretty excited about right now, and something we have already started doing or that we’d probably be able to share more about this in a couple of months, is that we are exploring AI for science.

And there are a lot of areas where we think that you could get some extremely promising buzz. If you were to apply AI in these domains. There are a lot of long inputs. You just have to find these domains where actually AI has not been yet applied, and it’s usually hard to do because the people working in those domains don’t necessarily know the capability of these models.

They don’t know. How I would just have to pair them with Yeah, exactly. Your researcher slashing, which is actually hard to do. But this matching, we’re doing it naturally with our customers. So we have some company we are very closely with. So for instance, ISM Andreesen are one of our partners, so we’re doing some research with them on their other, like tons of extremely interesting problems.

Columns in physics, in [00:44:00] science matter science that they’re essentially the only ones to work on. ‘cause they’re doing something No, no one else is doing on the, yeah. So there are many domains where AI can actually revolutionize things. Just you have to think about it on you familiar with what can do or to apply it.

So yeah, it’s something where more modeling with our partners, with our customers sort AI for s, but.

swyx: Yeah. Okay.

swyx: And then for deployed what it makes a good four deployed engineer, what do they need? Where do people fail?

Guillaume: I think it’s usually you need people that are very familiar with the tech and not necessarily with a lot of research expertise, but that are actually pretty good at using this model that can actually like that know how to do functioning, that know how to like, start some error pipeline.

And it’s it’s not easy. It’s something that mucus. Majority of companies will not be able to do this on their own. So here I think we need people that are, that like to solve problems that are accept solving some complex, very concrete problem. It’s applied science basically.

And yeah, so I think it’s not too different. I think from the case you need in research because it’s essentially you are trying to find solutions to problems that in [00:45:00] customers have not yet. So sometimes it’s easy. Sometimes you’re here to do the work. You have to like create synthetic data.

Find some edge case. So it can be, yeah. Depends on the problem. But but yeah, you have to, I think it also a bit of patience on the be creative. I think very similar skill is Asian,

Pavan: the diversity of the work they do. It always surprises me. It’s it’s, it goes all the way from the kind of stuff they encounter in industries.

It’s just very interesting. I think.

swyx: Any fun like success anecdotes.

Guillaume: Yeah, it can be actually training this small model on edge that just we do one specific thing can be like training some very large model without some specific languages as well. Making models really good at some tube use, like for instance, computer ID design, these kind of things.

Is that pairing with vision as well? Yeah,

Pavan: and the fact detection for chips or like in, in factories identifying things like it, the. Diversity could be anything where you can deploy these foundation models. So yeah the work to make it work in that specific setting, basically whatever it takes to make it like add value in that, by the way, workflow.

Vibhu: Yeah. [00:46:00] And it goes across the stack, right? Like even just pulling up the website like.

swyx: It’s so broad on compute. It is so broad.

Vibhu: We didn’t even touch on if you have a coding CLI tool. One thing you guys were actually like, I think the first tool was agents, ral agents. You had the agent builder, you can serve it via API and all that.

And I’m guessing forward deploy people.

Guillaume: Yeah.

Vibhu: Help build that out and stuff.

Guillaume: It is also why we are, so we’re doing many things, but I think that’s also part of the value proposition that sometime know customers. They’re always very. Extremely careful about their data and they don’t want to, they don’t like, trusting so many partners, trusting one partner for code, giving the data to another third party for like audios and another one.

So they don’t like this here. What they really like with our approach that we can help them on anything so they don’t have to send the data to so many clouds. So yeah,

swyx: I think that there can be many orders of magnitude more. F Ds then research scientists and they don’t need your full experience, but they’re still super variable to customers

Guillaume: in practice.

These two teams [00:47:00] are still quite intertwine, very often. Yeah. So first of all, they’re using the same tools, the same data pipeline and everything on the, it’s it’s very helpful for the science team to get the feedback and the solution team ‘cause they can. Look at these customers are trying to do this.

This is not working. It can really be show in the next version. Yeah. But this is basically a real world eval. Yeah, it’s real world eval and it’s not something, for instance, if you’re just working in the lab, it’s just ships model. But you don’t do this work of for customers. You have no idea for whether your model is good at this H case.

For instance, you even in year found this, right? So yeah, there is a very gap, big gap between the public benchmarks that are very like academic. On

Pavan: the rare cases are just very diverse and in the specific concept of a customer, you can fine tune and make it like first evaluate, create a solid eval, benchmark, and then measure in the context of their, the kind of audio.

Like for instance, one use case is literally just, there’s the word for kids and they have to just say it out. It’s a very specific thing. You’re just saying one word and then you have to you, you’ll grade the kid whether they did it right or not. It’s [00:48:00] like R for, but so there’re very diverse use cases and the idea is that they, the.

Applied scientist engineer will go and make it better. And then from the learnings we incorporate it into the base model itself. So it’s it’s just better out of the box.

Vibhu: Yeah. It’s a good full circle system. Like the foundation model evals are all just proxies of what you really, you’re never gonna have one that says it, it doesn’t make sense for there to be, a one word transcription like that.

It’s not something you wanna fit on. Perfect.

swyx: Everyone should go check out everything that Michelle has to offer and try the TTS model, which will link in the show notes. But thank you so much for coming tha thanks. Such a stretch.

At GTC 2022, NVIDIA announced the Hopper architecture and the first H100s started rolling out in October of that year. 2 years later, in October 2024, we published a popular piece on the H100 rental price depreciation cycle, which we had observed to be a going faster than previous cycles and theorized that it was a slight bubble burst dynamic due to temporarily inflated demand. While true for the time (bottoming out after the DeepSeek R1 shock, it did not last; since December 2025 the H100 rental market has gone VERY up:

This is corroborated by Dylan on Dwarkesh saying H100's are worth more today than they were 3 years ago, and surely related to the general chip shortage and reasoning model/agent inflection of December 2025, and the utility of a 4 year old chip now with much better reasoning models and inference software means the chip itself is much more valuable than initial 4-7 year depreciation schedules had assumed.

If you are used to the razor’s edge of data center tokenomics, you should expect that this has very meaningful implications on the business models of data centers and GPUs… as long as it keeps going.

AI News for 3/26/2026-3/27/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Anthropic’s leaked “Mythos” system and the new Capybara tier

• Fortune corroborates a higher Anthropic tier above Opus: A now-pulled “Claude Mythos” post was preserved by @M1Astra, and multiple follow-on posts cite a Fortune report that Anthropic is introducing Capybara, described as a new tier above Opus and “larger and more intelligent” than Claude Opus 4.6. Reporting summarized by @scaling01, @Yuchenj_UW, and @kimmonismus says Capybara posts substantially better scores on coding, academic reasoning, and cybersecurity, with rollout constrained by cost and safety concerns.

• Compute intensity is the central theme: Several posters infer Anthropic is leaning hard into scale, with speculation around a ~10T parameter class model from prior Dario comments, though that remains unconfirmed outside commentary; see @scaling01 and @Yuchenj_UW. Separately, the Financial Times report relayed by @FirstSquawk says Google is close to funding Anthropic’s data center, reinforcing that frontier competition is increasingly gated by power and capex rather than just algorithms.

• Infra strain was visible in production: The leak landed amid a rough day for Anthropic availability, with widespread user complaints about 529s/elevated errors from @dejavucoder, @iScienceLuvr, and others. The practical takeaway is that Anthropic appears to be balancing aggressive scaling ambitions against a still-tight serving envelope.

Open coding models, local inference, and GLM-5.1’s continued push

• GLM-5.1 is widening the pressure on closed coding models: Zhipu announced GLM-5.1 availability to all coding plan users via @Zai_org, along with docs for agent use at @Zai_org. Community reaction framed it as another sign that high-end Chinese open or semi-open coding models are closing the gap: @kimmonismus, @XFreeze, and Arena’s broader leaderboard analysis @arena all point to a much narrower open-vs-closed gap than a year ago.

• Local deployment economics keep improving: A recurring theme across tweets is that local models are now “good enough” for many workflows. Examples include @TheGeorgePu swapping a pricey TTS subscription for a local Qwen 3.5 14B setup, @LottoLabs reporting strong economics for Qwen 27B with Hermes Agent, and @0xSero compressing Qwen3.5-35B enough to fit full context into 24GB VRAM at roughly 1% average performance drop.

• Quantization and cache work remain key enablers: @iotcoi shipped a TurboQuant vLLM fork with fused Triton KV write paths and decode attention, targeting Qwen3.5-35B AWQ, 1M context, and 4M KV cache. Meanwhile @bnjmn_marie benchmarked Qwen3.5 27B formats across RTX Pro 6000/B200/H100, with INT4 emerging as the best inference option on RTX Pro 6000-class hardware.

• But TurboQuant is now under active dispute: The strongest research controversy in the set comes from @gaoj0017 and a longer clarification @gaoj0017, alleging Google’s ICLR 2026 TurboQuant paper misrepresented RaBitQ in theory and benchmarking, including unfair CPU-vs-GPU comparisons. This does not invalidate TurboQuant’s engineering value, but it does cast doubt on some of the publicized comparative claims.

Agents are becoming products, not demos

• Hermes Agent is emerging as the open-agent focal point: The most consistent product momentum in the dataset belongs to Nous Research’s Hermes Agent. @NousResearch integrated Hugging Face as a first-class inference provider with 28 curated models plus access to many more, while @ClementDelangue framed this as a step toward open agents with memory, persistent machine access, and model choice. User reports from @fancylancer3991, @PolackJack, and @alexcovo_eth emphasize lower friction and better persistence than browser-automation-heavy setups like OpenClaw.

• Agent infrastructure is maturing around traces, evals, and debuggability: Hugging Face’s @ClementDelangue called for open agent traces datasets, with follow-up pointing to the Agent Data Protocol from @yueqi_song. LangChain pushed a cluster of production-oriented materials: an agent eval readiness checklist @LangChain, Deep Agents IDE-style UI guidance @LangChain_JS, and LangSmith Prompt Hub Environments for prompt promotion/rollback @LangChain. The direction is clear: the stack is moving from “chatbot with tools” to software lifecycle primitives for agents.

• Agent-facing benchmarks are starting to reflect real workloads: Artificial Analysis introduced AA-AgentPerf via @ArtificialAnlys, focused on real coding-agent trajectories, 100K+ sequence lengths, and throughput expressed as concurrent users per accelerator / per kW / per $ / per rack. That is a more deployment-relevant abstraction than synthetic token benchmarks and should be useful for teams comparing accelerator systems for agent-heavy serving.

Coding agents, Codex plugins, and multi-agent software workflows

• OpenAI’s Codex ecosystem is shifting toward workspace-native automation: OpenAI developers highlighted Codex plugins and a use-case gallery via @OpenAIDevs, while Box shipped a Codex plugin for automating workflows over Box content @Box. User sentiment from @theo, @nickbaumann_, and @reach_vb suggests the center of gravity is moving from prompt/response to persistent workspaces, issue systems, terminals, PR flows, and plugins.

• The winning UX pattern is increasingly “fleet management for software”: @VibeMarketer_ captured the emerging pattern well: kanban-like cards, isolated worktrees, agent-owned tasks, and diff-based review. Related tools include the new agent-browser dashboard from @ctatedev for real-time browser session debugging, and broad enthusiasm for multi-agent SWE systems from Cognition/Devin adjacent commentary like @JTLonsdale and @cognition.

• Composer 2 and long-horizon coding evals are raising the bar: The CursorBench discussion is mostly indirect here, but @cwolferesearch points out the benchmark’s strengths: real coding sessions, underspecified prompts, broader quality dimensions, and median 181 lines changed per task. That’s a healthier benchmark design than static toy tasks and aligns with the broader turn toward long-horizon agent evaluation.

Research and systems: world models, robotics, speech, and multimodal infra

• Meta shipped a practical SAM 3.1 speedup: @AIatMeta released SAM 3.1, a drop-in update to SAM 3 with object multiplexing, allowing up to 16 objects in a single forward pass. Meta says this roughly doubles video throughput from 16 to 32 FPS on one H100 for medium-object workloads, which is meaningful for accessible video segmentation pipelines.

• World models and robotics both had notable open releases: @LiorOnAI highlighted LeCun’s LeWorldModel paper/repo as a small, open world model designed to make representational collapse mathematically impossible via SIGReg, claiming 48x faster planning and ~200x fewer tokens. On robotics data, @UnitreeRobotics open-sourced the UnifoLM-WBT-Dataset, a real-world humanoid whole-body teleoperation dataset intended for rolling updates.

• Speech/open audio remains one of the healthiest open categories: Cohere’s new 2B Apache-2.0 Transcribe model drew strong praise from @victormustar and throughput measurements from @vanstriendaniel, who reports 33 hours of audio transcribed in 12 minutes on an A100. Mistral’s Voxtral TTS paper was flagged by @qtnx_, and browser/local demos appeared from @sophiamyang and @nickfrosst.

• Open robotics stacks are also getting more reproducible: AI2 released MolmoBot, an open robotic manipulation suite trained entirely in simulation, with code, training data, generation pipeline, and evals available via @allen_ai. That complements the Unitree dataset and signals continued progress toward replicable robotics research outside top labs.

Top tweets (by engagement)

• Anthropic/Capybara leak: @Yuchenj_UW on Capybara was the most engaged technical item, summarizing the new tier above Opus and its reported benchmark gains.

• Paul Conyngham’s AI-assisted dog cancer treatment: @sama shared a story of using ChatGPT and related tools to help design an mRNA vaccine protocol for a dog’s cancer, which became a major discussion point about AI-enabled personalized medicine.

• TurboQuant critique: @gaoj0017 drew unusually high engagement for a paper-methodology dispute, likely because it challenges a heavily promoted systems paper.

• GLM-5.1 release: @Zai_org announcing broad GLM-5.1 availability landed strongly, reinforcing sustained interest in open coding models.

• Open infrastructure for agents: @OpenAIDevs on Codex plugins and @NousResearch on Hugging Face integration into Hermes Agent were the clearest product/infrastructure launches with broad developer relevance.

• Google TurboQuant running Qwen Locally on MacAir (Activity: 433): The post discusses an experiment where Google’s TurboQuant compression method was applied to llama.cpp, enabling the running of Qwen 3.5–9B on a standard MacBook Air (M4, 16 GB) with a 20000 tokens context. This was previously unfeasible on such hardware, highlighting TurboQuant’s potential to enable local execution of large models without cloud APIs. The experiment suggests that even entry-level devices like MacBook Airs or Mac Minis can handle large contexts, albeit with some speed limitations. The open-source app atomic.chat is mentioned as a resource for running these models locally. A commenter notes the impressive feat of handling 20K context on a base MacBook Air without swapping, suggesting potential for local use cases that previously relied on cloud APIs. Another commenter inquires about the integration of TurboQuant into llama.cpp, indicating interest in broader accessibility.

  • Tatrions highlights the impressive capability of running a 20K context model on a base MacBook Air with 16GB RAM without swapping, thanks to TurboQuant. This suggests that many applications that previously relied on cloud APIs could now be executed locally, though there is curiosity about the quality degradation at this compression level compared to standard Q4 on the same model.

  • M5_Maxxx provides a detailed audit of the TurboQuant implementation, revealing it as a minimally altered version of Jan.ai. Key changes include renaming, UI tweaks, and a custom llama.cpp backend fork, but no new inference engine or model architecture support. The 96 commits mostly involve CI/build pipeline changes, suggesting limited innovation beyond the original Jan.ai capabilities.

  • AppealThink1733 inquires about the integration of TurboQuant into llama.cpp, indicating interest in whether this technology is already supported by the popular open-source project, which could facilitate broader adoption and experimentation.

• Skipping 90% of KV dequant work → +22.8% decode at 32K (llama.cpp, TurboQuant) (Activity: 744): The post discusses an optimization in the TurboQuant implementation for KV cache compression in llama.cpp, which significantly improves decode performance by skipping dequantization for positions with negligible attention weights. This approach leverages attention sparsity, allowing a +22.8% increase in decode speed at 32K context length on an M5 Max, without affecting perplexity (PPL). The method involves a simple modification of about three lines in the kernel, bypassing the need for complex optimizations like SIMD tricks or fused kernels. The results are consistent across different hardware, including the M2 Pro, where performance improved from ~0.45x to ~0.73x compared to the standard q8_0 KV cache. The implementation and benchmarks are available on GitHub, with a detailed writeup. Commenters praised the simplicity and effectiveness of the solution, noting the innovative use of attention sparsity to skip unnecessary computations. There is curiosity about how this approach scales with even longer contexts, such as 64K+, and interest in integrating this optimization into the mainline llama.cpp.

  • Specialist_Sun_7819 highlights a novel optimization in llama.cpp’s TurboQuant, where skipping 90% of the key-value dequantization work for tokens that don’t significantly impact the output leads to a +22.8% increase in decoding speed at 32K context length. This approach leverages predictable attention sparsity in long contexts, allowing for significant computational savings with minimal code changes, specifically just three lines in the kernel. The commenter is curious about the scalability of this method to even longer contexts, such as 64K, and whether the sparsity ratio continues to increase or plateaus.

  • sean_hash draws a parallel between the optimization in TurboQuant and techniques used in Flash Attention, noting that caching the dequantized output instead of recalculating it at each decoding step is a similar strategy. This method effectively reduces redundant computations, enhancing performance by reusing previously computed values, which is a common optimization in high-performance computing to minimize unnecessary processing overhead.

  • Pentium95 expresses interest in integrating this optimization into the mainline llama.cpp, indicating a desire for broader adoption of this technique. This suggests that the community sees value in these performance improvements and is eager to see them implemented in widely-used codebases, potentially leading to more efficient models and faster inference times across various applications.

• TurboQuant in Llama.cpp benchmarks (Activity: 463): The post discusses the implementation of TurboQuant, a compression technique from Google, in the llama.cpp framework, specifically on Apple Silicon using Metal. The author notes a significant performance drop, with TPS being 50% less than f16, indicating potential issues in their setup. They also attempted to run kernels on a CUDA machine but encountered poor outputs, suggesting errors in their approach. The technique is seen as beneficial for running local models on consumer hardware with limited VRAM, potentially allowing for more complex tasks to be executed locally. The post references ongoing development efforts in related projects like MLX and VLLM. Commenters suggest checking KLD to evaluate the method’s worth and express interest in seeing performance metrics like pp2048, as pp64 is not very indicative. Another commenter recommends trying RotorQuant for comparison.

  • Velocita84 points out the absence of Kullback-Leibler Divergence (KLD) in the benchmarks, which is crucial for evaluating the effectiveness of TurboQuant. KLD is a measure of how one probability distribution diverges from a second, expected probability distribution, and its absence could mean missing insights into the model’s performance under TurboQuant compression.

  • CornerLimits suggests that the benchmark using pp64 is not very informative for assessing performance and recommends using pp2048 instead. The pp metric refers to perplexity, a common measure in language models that indicates how well a probability distribution predicts a sample. Higher pp values can provide a more comprehensive view of model performance.

  • DinoAmino discusses the trade-off between data compression and accuracy in TurboQuant, noting that while it allows for higher data compression with near-lossless accuracy, it doesn’t improve accuracy. They highlight that most large language models (LLMs) experience accuracy degradation at higher context lengths, implying that TurboQuant’s main benefit is enabling the use of longer contexts without additional accuracy loss.

• RotorQuant: 10-19x faster alternative to TurboQuant via Clifford rotors (44x fewer params) (Activity: 652): RotorQuant introduces a novel approach to vector quantization by utilizing Clifford Algebra, achieving 10-19x speed improvements over TurboQuant with 44x fewer parameters. The method replaces the d×d random orthogonal matrix with Clifford rotors, reducing the computational complexity from 16,384 FMAs to approximately 100 FMAs for d=128. This results in a cosine similarity of 0.990 compared to TurboQuant’s 0.991, indicating nearly identical performance. The implementation leverages fused CUDA kernels and Metal shaders, significantly outperforming cuBLAS matmul on RTX PRO 4000 and Apple M4. The trade-off involves higher synthetic MSE on random unit vectors, but with QJL correction, real-model attention fidelity remains intact. GitHub Paper A key debate centers on the theoretical differences between RotorQuant and TurboQuant. While TurboQuant’s global random rotation spreads energy across all dimensions, RotorQuant’s 3D block mixing cannot replicate this, leading to higher max coordinate magnitudes and worse MSE in low-bit quantization. However, RotorQuant’s practical performance in KV cache distributions is acknowledged, suggesting a valuable speed/quality tradeoff for real models.

  • Juan_Valadez highlights a key theoretical limitation of RotorQuant compared to TurboQuant, noting that TurboQuant’s global random rotation (Haar) effectively spreads energy across all dimensions, optimizing scalar quantization. In contrast, RotorQuant’s mixing within 3D blocks limits its ability to achieve the same energy distribution, which can negatively impact low-bit quantization, especially in worst-case vectors like one-hot. However, RotorQuant may still be practically useful for KV cache distributions where vectors are less adversarial.

  • Dany0 draws parallels between TurboQuant and techniques used in graphics programming, specifically referencing QuiP, a similar approach applied to model weights. Despite initial skepticism due to the shortness of the paper and its presentation, Dany0 acknowledges the potential of RotorQuant, likening its use of Clifford rotors to the application of quaternions instead of Euler angles, which simplifies computations by reducing multiplications to zeros.

  • sean_hash comments on the unexpected application of Clifford algebras in quantization, noting it as an example of cross-pollination from geometric algebra into fields outside of graphics. This highlights the innovative use of mathematical concepts traditionally associated with other domains, suggesting a broader applicability of these techniques.

• Glm 5.1 is out (Activity: 1127): The image announces the release of GLM-5.1 by Z.ai, highlighting its improved performance in coding tasks compared to previous versions. The chart in the image shows that GLM-5.1 scores 45.3 in coding evaluation, surpassing GLM-5’s score of 35.4, but still trailing behind Claude Opus 4.6, which scores 47.9. This suggests significant improvements in GLM-5.1’s capabilities, likely due to enhancements in its underlying architecture or training data. Commenters speculate about the potential release of open weights for GLM-5.1, indicating anticipation for broader accessibility. There is also discussion about the delay in the release of DS v4, hinting at possible challenges in training on specific hardware like Ascends.

  • power97992 speculates on potential delays in the release of DeepSpeed v4, suggesting that there might be issues related to training on Ascend hardware. This highlights the challenges in optimizing machine learning frameworks for different hardware architectures, which can impact release timelines.

  • zb-mrx notes the improvement in the rollout process for GLM 5.1, contrasting it with the previous version, GLM 5, which did not have a day-one rollout for everyone. This suggests that the developers may have resolved previous logistical or resource-related issues, such as GPU availability, to ensure a smoother release.

  • jacek2023 mentions the limitations of running GLM locally due to hardware constraints, specifically referencing a 72GB VRAM limit. This underscores the ongoing challenge of hardware requirements for running advanced models, which can be a barrier for many users without access to high-end GPUs.

On its own, the launch of Projects.dev, a way for agents to instantly provision services, is not immediately title-story worthy except for 2 things: 1) it comes from STRIPE, 2) it is a CLI. Run stripe projects add posthog/analytics and it’ll create a PostHog account, get an API key, and set up billing.

If that sounds weird to you, it’s because Stripe doesn’t really have anything to do with PostHog’s setup or signup process. Neither do these launch partners:

Stripe is just doing this because they can, and Patrick cites Andrej’s MenuGen as direct inspiration for how it is too hard for agents to set up backend services today. You’re sure to see the rest of the agent-native infra vendor landscape charts all lobby Stripe for real estate:

But let’s not stop there: scroll down the timeline a little further and here’s Ramp’s CLI also launching today, with some handy usecases:

Oh and look over here! It’s the Sendblue CLI (iMessage) you’ve always wanted, also launching today! catching up from the Kapso CLI (WhatsApp) from Monday! and did you miss the ElevenLabs CLI from yesterday? That’s fine, because you could also try the Visa CLI, the Resend CLI, or the steipete’s Discord CLI, the big momma, the official Google Workspace CLI!

Many, many people have written about why CLIs can be handier than MCPs, which isn’t necessarily a fair nor false comparison, but at this point the trend is undeniable and worth reporting. We credit Cloudflare’s Code Mode from last September in kicking off the “use more computer to wrap MCP” trend, and now of course CLIs in themselves don’t really expose or care about their underlying communication protocols.

AI News for 3/23/2026-3/24/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Model and Product Releases: Gemini 3.1 Flash Live, Mistral Voxtral TTS, Cohere Transcribe, and OpenAI GPT-5.4 mini/nano

• Google’s realtime push with Gemini 3.1 Flash Live: Google rolled out Gemini 3.1 Flash Live as its new realtime model for voice and vision agents, emphasizing lower latency, improved function calling, better noisy-environment robustness, and 2x longer conversation memory in Gemini Live. The launch spans Gemini Live, Search Live, AI Studio preview, and enterprise CX surfaces, with Google citing 70 languages, 128k context, and watermarking of generated audio via SynthID in some developer-facing summaries (Logan Kilpatrick, Google DeepMind, Sundar Pichai, Google). Third-party benchmarking from Artificial Analysis highlights the new “thinking level” tradeoff: 95.9% Big Bench Audio at high reasoning with 2.98s TTFA, versus 70.5% at minimal with 0.96s TTFA.

• Speech stack gets crowded fast: Mistral AI released Voxtral TTS, an open-weight TTS model aimed at production voice agents, with 9-language support, low latency, and strong human preference metrics; several summaries cite a 3B/4B-class model footprint, ~90 ms time-to-first-audio, and favorable comparisons to ElevenLabs in preference tests (Mistral AI, Guillaume Lample, vLLM, kimmonismus). Cohere launched Cohere Transcribe, its first audio model, under Apache 2.0, claiming the top English spot on the Hugging Face Open ASR leaderboard with 5.42 WER and 14-language support (Cohere, Aidan Gomez, Jay Alammar). Notably, Cohere also contributed encoder-decoder serving optimizations to vLLM—variable-length encoder batching and packed decoder attention—reportedly yielding up to 2x throughput gains for speech workloads (vLLM).

• OpenAI’s smaller GPT-5.4 variants look cost-competitive, with caveats: Artificial Analysis reported on GPT-5.4 mini and GPT-5.4 nano, both multimodal with 400k context and the same reasoning modes as GPT-5.4. The standout is GPT-5.4 nano, which was benchmarked ahead of Claude Haiku 4.5 and Gemini 3.1 Flash-Lite Preview on several agentic and terminal-style tasks while remaining cheaper on an effective-cost basis. The downside: both variants were described as highly verbose, with elevated output-token usage and weak AA-Omniscience performance driven by high hallucination rates. That matches anecdotal complaints from developers about codex/GPT-5.4 verbosity in practice (giffmana).

• Other notable releases: Zai made GLM-5-Turbo available to GLM Coding Plan users; Reka put Reka Edge and Flash 3 on OpenRouter; Google/Gemini also began rolling out chat-history and preference import from other AI apps; and multiple posts reported that OpenAI has deprioritized side projects including Sora and an “adult mode” chatbot in favor of core productivity efforts (Andrew Curran, kimmonismus).

Agent Infrastructure, Harnesses, and Multi-Agent UX

• Cline Kanban crystallizes a new multi-agent UX: The clearest tooling launch of the day was Cline Kanban, a free, open-source local web app for orchestrating multiple CLI coding agents in parallel across isolated git worktrees. It supports Claude Code, Codex, and Cline, lets users chain task dependencies, review diffs, and manage branches from one board (Cline, Cline). The reaction from builders was strong, with several calling this the likely default multi-agent interface because it tackles the two practical bottlenecks of current coding-agent workflows: inference-bound waiting and merge-conflict-heavy parallelism (Arafat, testingcatalog, sdrzn).

• “Harness engineering” is becoming a category: A recurring theme across tweets was that model quality is no longer the whole story; the agent harness—middleware, memory, task orchestration, tool interfaces, safety policies, and evaluation loops—is increasingly the real product. LangChain, hwchase17, and others emphasized middleware as the customization layer for agent behavior. voooooogel made the stronger claim that users casually say “LLM” when what they’re actually using is an integrated agentic language system with formatting, parsers, tool use, structured generation, and memory around the base model.

• Hermes vs. OpenClaw: memory and long-running autonomy matter: A large cluster of posts praised Nous Research’s Hermes Agent as more usable than OpenClaw/OpenClaw-derived stacks for long-running, cross-platform agent workflows. Examples included persistent memory across Slack and Telegram, shared memory across agents, lower maintenance overhead, and user reports of agents running unattended for hours on local or cloud setups (IcarusHermes, jayweeldreyer, Niels Rogge). Teknium also teased a controversial GODMODE skill for persistent jailbreaking, underscoring that capability and safety are now being productized at the harness layer, not just the base model.

• Tooling expansion around agents: OpenAI’s Codex team solicited requests for expanded toolkit integrations (reach_vb), while Google published how it built a Gemini API skill to teach models about newer APIs and SDKs, improving Gemini 3.1 Pro to 95% pass rate on 117 eval tests (Phil Schmid). OpenEnv was introduced as an open standard for agentic RL environments with async APIs, websocket transport, MCP-native tool discovery, and deploy-anywhere packaging.

Research Systems and Training Infrastructure: AI Scientist, ProRL Agent, and Real-Time RL

• Sakana AI’s AI Scientist gets a Nature milestone and a scaling-law claim: The most substantive research-system update came from Sakana AI, which highlighted a Nature paper on end-to-end automation of AI research and a notable empirical result: using an automated reviewer to grade generated papers, they observed a scaling law for AI science, where stronger foundation models produce stronger scientific papers, and argued that this should improve both with better base models and more inference-time compute (Sakana AI, paper/code follow-up). Chris Lu added that AI Scientist V1 predated o1-preview-style reasoning models, implying substantial headroom from today’s stronger models (Chris Lu).

• Infrastructure bottlenecks, not model bottlenecks, may be capping agent RL: One of the more important systems threads argued that agentic RL frameworks have been architected incorrectly by coupling rollout and optimization in the same process. The post summarizing NVIDIA’s ProRL Agent claims fully decoupling rollout into a standalone service nearly doubled Qwen 8B on SWE-Bench Verified from 9.6% to 18.0%, with similar gains for 4B and 14B variants, alongside much higher GPU utilization (rryssf_). If accurate, this is a strong reminder that agent training benchmarks can be infra-limited, not purely capability-limited.

• Cursor’s “real-time RL” is a notable production-training pattern: Cursor said it can ship improved Composer 2 checkpoints every five hours, presenting this as a productized RL feedback loop rather than a static model-release cadence. Multiple engineers read this as an early sign of continual learning in production, especially for vertically integrated apps with high-frequency interaction data (eliebakouch, code_star).

Architecture, Retrieval, and Inference Efficiency

• Transformer depth is becoming “queryable”: Kimi/Moonshot described Attention Residuals (AttnRes) as turning depth into an attention problem, allowing layers to retrieve selectively from prior layer outputs rather than passively accumulating residuals (Kimi). A strong secondary explainer from The Turing Post framed this as a broader trend: deep transformers moving from fixed residual addition toward adaptive retrieval over depth.

• Compression and memory-efficiency work remains central: TurboQuant drew attention as a practical route to 3-bit-like compression with near-zero accuracy loss, combining PolarQuant and 1-bit error correction (QJL) to accelerate attention and vector search, reduce KV cache memory, and avoid retraining (The Turing Post). Separately, a subtle but impactful production bugfix landed in vLLM’s Mamba-1 CUDA kernel after AI21 tracked a silent uint32_t overflow that caused logprob mismatches in GRPO training; the fix was effectively changing uint32_t to size_t (vLLM, AI21).

• Retrieval is trending multimodal and specialized: Several posts pointed to a shift away from generic RAG recipes. Victoria Slocum highlighted IRPAPERS, showing that OCR/text retrieval and image-page retrieval fail on different queries, and that multimodal fusion beats either alone on scientific PDFs. Chroma open-sourced Context-1, a search-focused model trained with SFT+RL over 8,000+ synthetic tasks, claiming better/faster/cheaper search than frontier general-purpose models; John Schulman called out its curriculum, verified synthetic data, and context-pruning tool as especially interesting.

Top tweets (by engagement)

• Meta’s TRIBE v2: Meta released TRIBE v2, a trimodal brain encoder trained on 500+ hours of fMRI from 700+ people, claiming 2–3x improvement over prior methods and zero-shot prediction for unseen subjects, languages, and tasks (Meta AI, details).

• Claude Code auto-fix in the cloud: Anthropic shipped remote PR-following auto-fix for Claude Code web/mobile sessions, allowing unattended CI-failure fixing and comment resolution (Noah Zweben).

• Karpathy on full-stack software automation: Andrej Karpathy argued the hard part of “build me this startup” is not code generation but the full DevOps/service orchestration lifecycle—payments, auth, infra, security, deployment—which he sees as just becoming tractable for agents.

• Cline Kanban: The launch of multi-agent worktree orchestration for coding agents generated unusually strong developer interest (Cline).

• Cohere Transcribe and Mistral Voxtral: Open, production-oriented audio releases continue to gather momentum, especially where they come with permissive licensing and immediate infra support (Cohere, Mistral).

It is considered uncouth in the news business to reflect on things that happened a few days ago — that is “olds”, not “news”. But we’re making a very conscious attempt to give you early and repeated signal on trends and not just headlines with AINews. To wit, although we already reported on Claude Cowork Dispatch, and Claude Cowork Dispatch Computer Use (a result of last month’s Vercept acquisition) was technically launched yesterday, the reception has been FAR and away Claude’s biggest launch of all time (inclusive of @AnthropicAI):

We created this chart by accumulating all the top tweets of the company accounts:

There were few bright spots today, as Microsoft AI execuhired AI2 leadership, OpenAI’s Sora became the first casualty of the Side Quest massacre (probably Atlas too), and LiteLLM suffered/created a huge supply chain vulnerability for ~all Python AI projects.

All will fade in due time, so do not deserve title story, but today we highlight this chart which tells an ongoing story that ALL traditional app stores like Apple and AI-native app stores like Dreamer face:

Even as the debate rages on about AI killing all SaaS, the ability to vibecode apps and hopefully buy a ticket to a >$100M exit as an 18 year old high school dropout means that ~everyone with any entrepreneurial spirit is going to at least try it, and traditional app store review processes will die. This comes even as Apple starts blocking vibe code apps like Replit and Vibecode on policy reasons, and, even though there were legitimately defensible issues, it is clear that the normal app distribution paradigms are completely breaking down in 2026. We declared the War on Slop a key theme for 2026 in Jan, and now this is one of the most important charts in the world for thinking through it’s ramifications on the decades-long supremacy of the Apple App Store as well as other similar software distribution platforms vs the open web.

AI News for 3/23/2026-3/24/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Agent Infrastructure, Computer Use, and Design-to-Action Tooling

• Anthropic’s agent harness and “computer use” shift the product surface: A recurring theme today was that agent capability is increasingly about the harness, not just the base model. Anthropic published a new engineering writeup on how it uses a multi-agent harness for frontend design and long-running software tasks, emphasizing orchestration over one-shot prompting (AnthropicAI). Multiple developers independently argued that “computer use” matters because it lets models act in messy software environments with no reliable APIs (glennko), though others noted this is still slow and likely transitional until more tools expose APIs/CLI surfaces (Yuchenj_UW). The broader operational takeaway was captured well by kerrsee: retries, rollbacks, webhooks, structured logging, and recovery paths remain the unglamorous bottlenecks in production agent deployment.

• Figma/MCP/Cursor make design canvases directly agent-editable: The strongest concrete workflow launch was Figma’s MCP server and direct AI editing on the canvas, now in open beta (figma). GitHub highlighted that this works through Copilot CLI and other clients via MCP (github), and Cursor immediately extended the pattern to generating components/frontends in Figma using a team’s design system (cursor_ai). This is one of the clearest examples of tool-calling becoming product-native rather than chat-wrapper-native. LangChain also pushed in the same direction with framework-native tool rendering and Slack-native Fleet workflows, including custom Slack bots and an Inbox for human approvals (LangChain_JS, LangChain, hwchase17).

Open Agent Platforms, Benchmarks, and RL Environment Stacks

• Hermes Agent v0.4.0 is becoming a full personal-agent runtime: Nous released a substantial Hermes Agent v0.4.0 update with roughly 300 merged PRs in a week, adding an OpenAI-compatible Responses API backend, background self-improvement loops, broader messaging integrations, improved context compression, and more CLI ergonomics (Teknium, Teknium, NousResearch). The most technically interesting feature is the post-response review agent that decides what to retain as reusable memory/skills (Teknium). Community reactions focused less on benchmark claims and more on operational value: exposing a personal coding/ops agent behind a standard API makes it usable from Open WebUI, LobeChat, or any OpenAI-compatible client (witcheer).

• Open agent ecosystems are converging around environments, skills, and reproducible evals: AI2 released MolmoWeb, an open-source browser agent built on Molmo 2 in 4B and 8B sizes, claiming open-weight SOTA across four web-agent benchmarks and even surpassing some proprietary agents (allen_ai). In parallel, GenReasoning launched OpenReward, a platform exposing 330+ RL environments, autoscaled environment compute, and 4.5M+ unique RL tasks through one API—explicitly targeting the often-missing “environment compute” layer of agentic RL (GenReasoning, rosstaylor90). Zhipu contributed ZClawBench, a benchmark with 116 real-world agent tasks spanning office automation, coding, and analysis (HuggingPapers). Together, these point to a stack maturing from “agent demos” toward standardized environment serving + benchmarkable task suites + reusable harnesses.

Inference, Storage, and Systems Optimizations

• vLLM and Transformers both reported material inference/runtime gains: vLLM’s GTC recap highlighted several systems upgrades: Model Runner V2 with GPU-native Triton kernels, a hybrid memory allocator, encoder prefill disaggregation with up to 2.5x P99 throughput gains for multimodal workloads, and modular MoE kernels (vllm_project, vllm_project). Separately, Hugging Face/Transformers-side optimization work claimed continuous batching plus torch.compile tuning now reaches 95% of vLLM throughput for 8K generation, effectively closing the previous gap for synthetic data generation workloads (remi_or_).

• hf-mount is a notable agent/data primitive: Hugging Face released hf-mount, which lets users mount Hub datasets, models, and storage buckets as a local filesystem, including examples with a 5TB FineWeb slice (julien_c, ClementDelangue). This matters beyond convenience: several engineers pointed out that agents are unusually good at filesystem operations, making mounted remote storage a natural substrate for agent memory, scratchpads, team artifact storage, and lazy access to large corpora (Vtrivedy10, victormustar). This is one of the more practical infrastructure launches of the day because it reduces the friction between local tooling and cloud-scale data.

• Moreau and TurboQuant show optimization pressure moving below the model layer: Optimal Intellect introduced Moreau, a GPU-native solver from the CVXPY team claiming orders-of-magnitude speedups over existing tools (opt_intellect). Google Research announced TurboQuant, a KV-cache compression algorithm reporting at least 6x memory reduction and up to 8x speedup with no accuracy loss (GoogleResearch). The common pattern: high-value gains are increasingly coming from runtime, memory, and systems layers, not just from larger model checkpoints.

Security, Supply Chain Risk, and Guardrails for Agentic Software

• The LiteLLM PyPI compromise dominated infra/security discussion: Multiple posts warned that LiteLLM 1.82.8 on PyPI had been compromised, with malicious payloads attempting to exfiltrate credentials and replicate across environments (hnykda). simonw noted the package was later quarantined on PyPI, but the incident quickly became a broader conversation about software supply-chain fragility. karpathy gave the most detailed summary, listing possible exfiltration targets including cloud creds, SSH keys, Kubernetes configs, CI/CD secrets, wallets, and shell history, while noting transitive risk to packages like DSPy. The most important systems-level implication came from DrJimFan: in an agentic world, the entire filesystem becomes part of the attack surface, since any file likely to enter context can become a vector.

• “De-vibing” and permissioning are becoming first-class product requirements: Several posts effectively converged on a new design principle: autonomous coding tools need stronger shells, better permission defaults, and fewer broad dependencies. Yuchen called the incident “nightmare fuel” for --dangerously-skip-permissions style workflows (Yuchenj_UW); Anthropic’s new Claude Code auto mode became controversial for exactly this reason, despite enthusiasm over the productivity jump (alexalbert__, kimmonismus). The practical response from many builders was a renewed preference for minimal bespoke routing, tighter audited deps, and stronger human approval loops.

Labs, Org Moves, and Product Strategy Shifts

• AI2 loses leadership to Microsoft; Microsoft AI continues talent concentration: The clearest org move was the reaction to Microsoft poaching part of the AI2 leadership team, including mentions of Ali Farhadi, Hanna Hajishirzi, and Ranjay Krishna joining Microsoft Superintelligence (eliebakouch, NandoDF). The subtext in technical circles was concern over whether open research institutions can continue competing with hyperscalers for top talent and frontier-scale work (stanfordnlp).

• OpenAI is reallocating resources hard: $1B Foundation spend, Sora wind-down, “Spud” coming: OpenAI announced its Foundation will spend at least $1B over the next year, with Wojciech Zaremba moving to lead AI resilience and additional hires across disease, civil society, and operations (sama, woj_zaremba, btaylor). At the same time, reports circulated that OpenAI had finished initial development of its next major LLM, codenamed “Spud,” and was winding down Sora’s app/product footprint to free compute (steph_palazzolo, kimmonismus). For engineers, the signal is straightforward: OpenAI appears to be narrowing product focus around core general models/infrastructure, even at the cost of cutting side products.

Top tweets (by engagement)

• LiteLLM supply-chain compromise: karpathy gave the most technically complete and highest-signal breakdown of the PyPI attack and its blast radius.

• Anthropic’s harness engineering post: AnthropicAI was one of the day’s most important engineering reads on how frontier labs are actually structuring long-running agent workflows.

• Figma MCP launch: figma and github showed perhaps the cleanest mainstream example yet of agents acting directly on a production design surface.

• OpenAI Foundation $1B commitment: sama and woj_zaremba marked a major organizational and safety/resilience shift.

• Hermes Agent v0.4.0: Teknium / NousResearch stood out as one of the biggest open-agent runtime releases of the day.

• LM Studio may possibly be infected with sophisticated malware. (Activity: 1822): The image in the Reddit post shows a Windows Security alert indicating that a severe threat, identified as “Trojan:JS/GlassWorm.ZZ!MTB,” was quarantined from the LM Studio directory. This raised concerns about a potential malware infection in LM Studio. However, LM Studio and Microsoft have since confirmed that this was a false positive, likely due to Defender’s heuristic definitions conflicting with LM Studio’s obfuscated Electron bundle. The community discussion highlights the importance of security audits and the potential risks of obfuscation techniques that resemble malware patterns. Despite the false alarm, users are advised to take precautionary measures to secure their data. The comments reflect a consensus that the malware detection was a false positive, supported by historical instances of similar false alarms and VirusTotal’s low detection rate. However, there is criticism of LM Studio’s code obfuscation practices, which can inadvertently trigger such alerts and complicate security assessments.

  • Yags from LM Studio confirmed that the malware alert was a false positive, verified by Microsoft, and no longer appears in VirusTotal. Despite this, LM Studio is auditing their build machine scripts and environments to prevent any genuine security incidents in the future.

  • Denoflore_ai_guy provided a detailed analysis suggesting the malware alert was likely a false positive due to Defender’s heuristic updates conflicting with LM Studio’s obfuscated Electron bundle. However, they noted that LM Studio’s code obfuscation for IP protection could resemble malware techniques, which complicates detection.

  • Denoflore_ai_guy also outlined steps to mitigate potential risks if GlassWorm malware was indeed present, including changing passwords, moving crypto funds, and checking for malicious Chrome extensions. They emphasized the importance of a clean OS install and credential rotation to ensure security.

• [Developing situation] LiteLLM compromised (Activity: 380): The LiteLLM library has been compromised, as detailed in GitHub issue #24512. The attack exploits a .pth file vulnerability, which executes code on interpreter startup without requiring imports, making it difficult to detect through standard code reviews. Users of version 1.82.8 are advised to rotate credentials immediately if used in production environments, as the compromise could expose sensitive information. A notable comment highlights the effectiveness of using Docker containers for isolating host secrets, which can mitigate some security risks. Another comment emphasizes the stealthy nature of the .pth file trick, which bypasses typical security scans.

  • The .pth file trick is highlighted as a significant security vulnerability. This method allows code execution on interpreter startup without needing imports, making it nearly invisible to standard code reviews. Users who ran LiteLLM versions 1.82.8 or 1.82.7 are advised to rotate credentials immediately due to potential exposure.

  • Aider, a tool that uses LiteLLM for LLM access, is reportedly safe as it operates on an older version (1.82.3) of LiteLLM, which is not compromised. The compromised versions are identified as 1.82.8 and 1.82.7, emphasizing the importance of version control and monitoring for security vulnerabilities.

  • The discussion touches on the use of Docker containers for security isolation. While typically not considered a security measure, in this case, Docker effectively isolated host secrets, demonstrating its potential utility in mitigating certain types of security breaches.

• Litellm 1.82.7 and 1.82.8 on PyPI are compromised, do not update! (Activity: 441): Litellm versions 1.82.7 and 1.82.8 on PyPI have been compromised, as confirmed by a blog post. The attack appears to be a supply chain compromise, potentially affecting thousands of users. The malicious versions were uploaded to PyPI, posing a significant risk to CI/CD pipelines that automatically update dependencies. The attack was executed through the GitHub account of the LiteLLM CEO, which was hacked, as evidenced by unauthorized commits and repository updates claiming ‘teampcp owns BerriAI’. Commenters emphasize the importance of pinning dependency versions to avoid such supply chain attacks, highlighting the risk of automatic updates in production environments. There is also concern about the potential for increased frequency of such attacks on AI tooling.

  • GroundbreakingMall54 highlights the critical importance of pinning dependency versions and avoiding auto-updates in production environments. They emphasize the risk of supply chain attacks, especially in AI tooling, as evidenced by the compromised Litellm versions on PyPI, which could have been automatically integrated into CI/CD pipelines overnight.

  • Gremlation and JockY discuss the breach by ‘teampcp’, who compromised the CEO’s GitHub account to inject malware into Litellm. This malware, embedded in versions 1.82.7 and 1.82.8, is designed to steal secrets upon startup. They note that versions <= 1.82.6 remain unaffected, and provide links to GitHub commits showing the unauthorized changes made under the CEO’s account.

  • kiwibonga points out a specific malicious payload in the compromised Litellm versions that executes a destructive command (rm -rf /) if the system’s timezone is set to Asia/Tehran. This highlights the severity and targeted nature of the attack, suggesting a broader geopolitical context to the cyber threat landscape.

• I built Fox – a Rust LLM inference engine with 2x Ollama throughput and 72% lower TTFT. (Activity: 212): Fox is a Rust-based local LLM inference engine designed as a drop-in replacement for Ollama, offering significant performance improvements. It features PagedAttention, continuous batching, and prefix caching, achieving 72% lower TTFT and 111% higher throughput on an RTX 4060 with the Llama-3.2-3B-Instruct-Q4_K_M model. The engine supports multi-model serving with lazy loading and LRU eviction, and provides a dual API compatible with both OpenAI and Ollama. The official Docker image is available, and the system supports hardware autodetection across CUDA, Vulkan, Metal, and CPU. The project is in beta, with thorough testing on Linux and NVIDIA, but less so on other platforms and configurations. GitHub and Docker Hub links are provided for access. A top comment highlights the impressive technical achievement of implementing vLLM-level features in Rust, noting the significant performance gains from prefix caching and continuous batching. There is a request for LoRA hot-swapping capabilities to further differentiate Fox from Ollama. Another comment expresses skepticism about the project’s authenticity and security, suggesting the need for independent verification and code auditing.

  • No_Strain_2140 highlights the technical achievements of Fox, noting its use of PagedAttention, continuous batching, and prefix caching, which contribute to its impressive performance metrics such as 87ms P50 on a 4060 with Q4_K_M. The commenter contrasts Fox’s approach with Ollama’s sequential processing, emphasizing Fox’s advanced features like multi-turn KV reuse that enhance throughput and reduce TTFT. They also inquire about the potential for LoRA hot-swapping, which could allow serving a base model with multiple LoRA adapters, positioning Fox as more than just a faster alternative to Ollama.

  • PettyHoe raises concerns about the security and credibility of the project, suggesting the need for independent verification and code audits to ensure there are no risks of exfiltration. They express skepticism about the project’s authenticity due to the AI-generated nature of the descriptions and comments, emphasizing the importance of cautious evaluation before adoption.

  • AIDevUK asks about Fox’s capability to operate over multiple GPUs, which is a critical consideration for scaling and performance in large-scale deployments. This question points to the need for understanding Fox’s architecture and its ability to leverage multi-GPU setups for enhanced computational efficiency.

• RYS II - Repeated layers with Qwen3.5 27B and some hints at a ‘Universal Language’ (Activity: 695): The post discusses findings from experiments with the Qwen3.5 27B model, revealing that LLMs may process information in a ‘universal language’. This is evidenced by the similarity in latent representations of the same content across different languages, such as Chinese and English, during the middle layers of the model. The author also found that repeating blocks in the middle of the transformer stack enhances performance. The models are available on Hugging Face. The author suggests that fine-tuning these models, especially the RYS-Qwen3.5-27B-FP8-XL, could set a new state-of-the-art (SOTA) for models of this size. Additionally, there is ongoing work to optimize VRAM usage by keeping duplicated layers as copies, which could be beneficial for future implementations. Commenters appreciate the rigorous approach and potential implications of the research, noting its relevance to performance improvements seen in complex model merges. There is interest in how these findings might influence open-source tuning practices, particularly in creative writing and self-merging techniques.

  • ArsNeph discusses the intriguing performance improvements observed in self-merges like Goliath 120B, noting that not all models benefit equally. They reference historical discussions about VRAM-less duplicated layer inference, highlighting ongoing work on EXL3. The comment suggests that open-source tuners, particularly those focused on EQ performance, might find these insights valuable, especially in creative writing contexts where complex merge trees have shown significant improvements.

  • Kwigg reflects on past experiences with ‘frankenmerging’ during the llama2 era, questioning the efficiency of such methods with newer models that have advanced attention mechanisms. They note that older frankenmerges were memory inefficient, implying that modern models might handle these techniques differently, potentially leading to better performance outcomes.

  • TomLucidor suggests expanding the language testing of Qwen3.5 to include Japanese, Thai, French, German, and Italian. They also propose a comparative analysis between Qwen3.5 and other models like Nemotron-3, known for its speed and linear attention, and Granite-4.0, which offers a similar size variety but is less optimized. This could provide insights into the relative performance and optimization of these models.

• FlashAttention-4: 1613 TFLOPs/s, 2.7x faster than Triton, written in Python. What it means for inference. (Activity: 364): FlashAttention-4 achieves 1613 TFLOPs/s on the Blackwell B200 GPU, utilizing 71% of its theoretical peak performance. It is 2.1-2.7x faster than Triton and up to 1.3x faster than cuDNN 9.13. The implementation is entirely in Python using NVIDIA’s CuTeDSL, which compiles in 2.5 seconds compared to 55 seconds for C++. This version supports GQA and MQA and is integrated into vLLM 0.17.0. However, it is limited to Hopper + Blackwell architectures, specifically H100/H800 and B200/B100 GPUs, due to reliance on specific hardware features like TMEM, 2-CTA MMA, and async TMA. The article also discusses how softmax has become the bottleneck and how selective rescaling optimizes performance. Commenters express frustration with NVIDIA’s marketing of GPUs as ‘Blackwell’ when they lack full compatibility with FlashAttention-4, highlighting a discrepancy between advertised and actual hardware capabilities.

  • JockY expresses frustration with NVIDIA’s marketing of the RTX 6000 Pro as ‘Blackwell’ when it is not fully compatible with Blackwell features, specifically mentioning that FlashAttention-4 (FA4) and NVFP4 are only supported on SM100 architectures. This highlights a discrepancy between NVIDIA’s product naming and actual hardware capabilities, which can mislead early adopters expecting full feature support.

  • Daemontatox points out that the issue with NVIDIA’s RTX 6000 Pro being marketed as ‘Blackwell’ is more related to the Streaming Multiprocessor (SM) architecture rather than the naming or overall architecture. The RTX 6000 Pro and DGX systems are sold under the ‘Blackwell’ name but actually use the SM120 architecture, which lacks some expected features, leading to consumer dissatisfaction.

  • STNKMyyy questions the relevance of such high-performance advancements like FlashAttention-4 for consumer-grade GPUs, implying that while these technologies are groundbreaking, they may not be accessible or beneficial for typical consumer hardware users. This reflects a common concern about the gap between cutting-edge research and practical consumer applications.

• Created a SillyTavern extension that brings NPC’s to life in any game (Activity: 499): The post describes a new extension for SillyTavern that integrates NPCs into any game by using Cydonia as the role-playing (RP) model and Qwen 3.5 0.8B as the game master. This setup allows for dynamic NPC interactions by downloading a game’s wiki and feeding it into SillyTavern, enabling NPCs to have detailed lore and respond contextually. The system uses voice cloning from game files and provides NPCs with game state information, such as player stats and location. The RP model operates locally, ensuring low latency and strong narrative capabilities. A secondary model, Qwen 3.5, interprets RP interactions to trigger in-game actions, enhancing the realism and depth of older games without needing conversational input. The post highlights the effectiveness of specialized RP models over base models in gaming applications. Commenters express surprise and enthusiasm about the potential of AI in gaming, noting the innovative use of AI for NPC interactions and questioning why such technology isn’t already standard in games.

  • A user highlights the impressive use of a 0.8B parameter model for bringing NPCs to life in games, questioning if the project is open source. This suggests a lightweight model capable of running efficiently in real-time gaming environments, which is significant for integration into existing games without heavy computational demands.

• Which local model we running on the overland Jeep fellas? (Activity: 459): The image depicts a Waymo self-driving car, highlighting the technological advancements in autonomous vehicle systems. The discussion centers around the prediction that future cars will require 300GB of RAM, a significant increase from current standards. This prediction is likely based on the assumption that more complex models, possibly involving real-time data processing and AI-driven decision-making, will be integrated into vehicles. The comments reflect skepticism about this prediction, with users questioning the necessity of such high memory requirements, especially when current vehicles operate efficiently on much less RAM. Commenters express skepticism about the prediction of 300GB of RAM for future cars, questioning the basis of this assumption and comparing it to current vehicle capabilities that require significantly less memory.

  • ForsookComparison questions the necessity of high RAM requirements for automotive models, noting that their car operated efficiently with just 16GB of RAM over a 600-mile journey. They challenge the assumption that 300GB is needed, suggesting that such figures might be based on models that require extensive tool-calls, which may not be applicable in all scenarios.

  • txdv highlights the potential cost implications of high RAM requirements in vehicles, expressing concern over the feasibility of 128GB upgrades. They point out that automotive pricing is sensitive, and a 5k cost for RAM could be prohibitive for consumers, indicating a need for balancing performance with affordability.

• The current state of the Chinese LLMs scene (Activity: 639): The Chinese LLM landscape is dominated by major players like ByteDance, Alibaba, Tencent, and Baidu, each with proprietary and open-weight models. ByteDance leads with its dola-seed model, akin to OpenAI, and its Seedance T2V model is popular for video generation. Alibaba excels in open-weight models, particularly small ones, and is strong in T2I and T2V. Tencent’s Hunyuan model is noted for 3D mesh generation, though its latest versions are not open-sourced. Baidu’s Ernie model is less used, with a stronger focus on autonomous driving. Other notable players include Xiaomi with Mimo V2 Pro, Ant Group with Ling 2.5 1T, and Meituan with LongCat-Flash-Chat, which uses a dynamic MoE approach. Deepseek is highlighted for its innovation in attention mechanisms like MLA and DSA. The “Six AI Small Tigers” such as Zhipu and Minimax focus on releasing large open-weight models to gain recognition. Government-funded initiatives like BAAI and Shanghai AI Lab are also contributing, though with varying reputations. Commenters note the rapid pace of open-weight model releases in China compared to the US, with some labs releasing more in a quarter than US companies in two years. Tencent is recognized for its investment in game development-specific models, with Hunyuan 3.1 being state-of-the-art for 3D mesh generation.

  • Tencent is heavily investing in game development-specific models, such as Hunyuan 3.1 for 3D mesh generation and HY-Motion for text-to-animation, which are considered state-of-the-art. Initially, Tencent open-sources these models to build brand recognition, but transitions to closed weights once they reach commercial viability, as seen with the latest Hunyuan 3D models.

  • A list of popular models by token usage on OpenRouter over the last 7 days highlights the dominance of Chinese models, with Xiaomi MiMo-V2-Pro leading at 1.77 trillion tokens. Notably, only three Western labs are ranked, and the ‘Small Tigers’—smaller companies advancing AI rapidly—are prominent, indicating a shift in innovation dynamics.

  • Despite ByteDance’s significant contributions to AI, they have not released any open weight models, as confirmed by the absence of such models on Hugging Face. This contrasts with other Chinese labs that frequently release open weights, accelerating competition in the AI space.

• So cursor admits that Kimi K2.5 is the best open source model (Activity: 629): The image is a tweet from Aman Sanger discussing the evaluation of base models, specifically highlighting that Kimi K2.5 emerged as the strongest model based on perplexity-based evaluations. The tweet notes that the model’s strength is attributed to continued pre-training and high-compute reinforcement learning, which enhance the capabilities of the Composer-2 model. The tweet also acknowledges an oversight in not mentioning the Kimi base in their blog, with plans to rectify this in future communications. One comment critiques the use of perplexity-based evaluations between models, noting that scores can be influenced by factors like dictionary size. Another comment questions the claim about the proportion of training done by Kimi K2, citing reports from Workshop Labs that suggest Fireworks’ K2 training code is not optimized for hyperscaled training, contrasting with claims of its efficacy.

  • The claim that Kimi K2.5 is the best open-source model is questioned due to the methodology of evaluation, particularly the use of perplexity scores which can be misleading as they depend on factors like dictionary size. This raises concerns about the validity of such comparisons between models.

  • There is skepticism about the training claims made by Fireworks regarding Kimi K2.5. Workshop Labs, known for optimizing training code, reported that Fireworks’ code is not optimized for hyperscale training, being only marginally better than basic implementations like HF Transformers 4.x. This suggests potential inefficiencies in Fireworks’ approach to training Kimi K2.5.

  • The assertion that Kimi K2.5 is the best ‘base model’ is attributed to its large parameter count and use of a standard attention mechanism rather than a linear one. This implies that the model’s architecture and scale contribute significantly to its performance, rather than any novel training techniques.

• China’s open-source dominance threatens US AI lead, US advisory body warns (Activity: 922): A US advisory body has raised concerns about China’s growing influence in the open-source AI sector, suggesting it could threaten the US’s leadership in AI. The report highlights China’s strategic investments and advancements in open-source AI models, which are becoming increasingly competitive with US counterparts. The advisory body suggests that the US needs to bolster its open-source initiatives to maintain its competitive edge. Commenters argue that the US is lagging in open-source AI, with Chinese models being more cost-effective and efficient. There is also criticism of US models like Opus, GPT-5.4, and Gemini 3.1 Pro for their perceived dysfunctionality, contrasting with China’s contributions to AI freedom despite its authoritarian regime.

  • EffectiveCeilingFan highlights the competitive edge of Chinese AI models, noting that they are not only cheaper but also outperform US models in open weights. The commenter criticizes the performance of US models like Opus, GPT-5.4, and Gemini 3.1 Pro, suggesting that the US is lagging in terms of open-source AI development.

  • Lissanro emphasizes the importance of open research in AI development, citing the ‘Attention is All You Need’ paper as foundational. They mention that models like Kimi K2.5 owe their existence to open research shared by companies like DeepSeek. The comment also notes that large companies, such as Cursor AI, are adopting Chinese models like Kimi K2.5 for their products, indicating a preference for these open-source models in the industry.

  • Global_Estimate7021 provides a detailed analysis of why the US might be falling behind in AI, citing a significant AI acceptance gap (87% in China vs. 32% in the US) and the volume of AI research publications where China leads. They also mention the strategic advantage of China’s cheaper electricity and grassroots AI literacy initiatives, which contrast with the US’s top-down approach.

/r/Singularity, /r/Oobabooga, /r/MachineLearning, /r/OpenAI, /r/ClaudeAI, /r/StableDiffusion, /r/ChatGPT, /r/ChatGPTCoding, /r/aivideo, /r/aivideo

• The man who originally coined the acronym “AGI” now says that we’ve achieved it exactly as he envisioned. (Activity: 926): The image is a tweet by Mark Gubrud, who claims to have coined the term “AGI” (Artificial General Intelligence). He asserts that AGI has been achieved as he envisioned, with current models performing at a high-human level in language and general knowledge, while being much faster. However, there is debate about the originality of his claim, as the term “artificial general intelligence” is documented as early as 1989, attributed to G. Simons. Gubrud’s definition of AGI involves systems that match or surpass human brain complexity and speed, capable of reasoning with general knowledge in various operations. There is skepticism in the comments about Gubrud’s claim to have coined the term “AGI,” with some suggesting he misremembers the history. The Oxford English Dictionary attributes the earliest use of the term to 1989, in the writings of G. Simons, not Gubrud.

  • The term ‘artificial general intelligence’ (AGI) is documented as early as 1989, with the Oxford English Dictionary citing G. Simons as the earliest source. However, M. Gubrud is often credited with popularizing it in scientific literature, though he did not coin the term himself.

  • The original definition of AGI by its coiner describes it as systems that match or surpass human brain capabilities in complexity and speed, capable of handling general knowledge across various domains, including industrial and military operations. This definition suggests a broad and versatile intelligence, though there is skepticism about whether current systems meet this standard.

  • There is debate about the significance of achieving AGI without recursive self-improvement, which was expected to trigger a technological singularity. The lack of such transformative advancements leads to skepticism about the current excitement surrounding AGI developments.

• Jensen Huang (NVIDIA) claims AGI has been achieved (Activity: 2562): In a recent interview, Jensen Huang, CEO of NVIDIA, claimed that Artificial General Intelligence (AGI) has been achieved, a statement that has sparked significant debate. The interview, available on YouTube, lacks detailed technical evidence to support this claim, leading to skepticism among experts. Huang’s assertion is seen as potentially influenced by his role in promoting NVIDIA’s products, which are heavily invested in AI technologies. The top comments reflect skepticism towards Huang’s claim, highlighting a distrust in business leaders’ statements about their own products. Commenters suggest that such claims may be more about marketing than factual advancements in AI.

  • Sweaty_Rub4322 highlights a critical issue in the AGI debate: the lack of a universally accepted definition of AGI. This ambiguity complicates discussions and assessments of whether AGI has been achieved, as both academia and industry struggle to agree on what constitutes AGI. This underscores the need for a clear, standardized definition to facilitate meaningful progress and evaluation in the field.

• Claude can now use your computer (Activity: 2106): Claude, an AI developed by Anthropic, is now capable of using your computer to perform tasks via Claude Cowork and Claude Code. This feature, currently in research preview, allows Claude to open applications, navigate browsers, and manage spreadsheets, effectively automating tasks typically done manually. It prioritizes using connected apps like Slack and Calendar, but can also directly interact with apps on your screen with permission. This functionality is available on Pro and Max tiers for macOS users, requiring an updated desktop app paired with a mobile device. More details can be found here. Concerns were raised about the security implications of allowing an AI to control a computer, with some users expressing apprehension about potential job displacement. Others noted this as a strategic move by Anthropic in response to competitors like OpenAI.

  • A key concern raised is about security implications of allowing Claude to access a user’s computer. This involves potential risks such as unauthorized data access or manipulation, which could be exploited if not properly secured. The rapid pace of feature releases may exacerbate these concerns, as new functionalities might not be thoroughly vetted for vulnerabilities before deployment.

  • The introduction of Claude’s ability to use a computer is seen as a competitive response to OpenAI’s advancements, particularly in the context of AI models like GPT-4. This move by Anthropic could be aimed at maintaining parity or gaining an edge in the AI capabilities race, highlighting the competitive dynamics in the AI industry.

  • There is a sentiment that the rapid development and release of new features by Claude could lead to job displacement. As AI models become more capable of performing complex tasks traditionally done by humans, there is a growing concern about the impact on employment, especially in sectors heavily reliant on routine cognitive tasks.

• Claude Code can now /dream (Activity: 1953): Claude Code has introduced a feature called Auto Dream, designed to enhance the agent’s memory management by mimicking human REM sleep processes. This feature reviews past session transcripts, identifies relevant information, prunes outdated or contradictory data, and consolidates it into organized files. It operates in the background, triggering after 24 hours and five sessions since the last consolidation, and ensures no conflicts by using a lock file. This approach aims to improve performance by managing memory more intelligently, rather than just expanding context windows. Some commenters express skepticism about the feature, suggesting it might lead to unnecessary token usage and questioning the AI’s self-promotion style. Others humorously suggest additional commands to manage AI hallucinations and errors.

  • AutoDream is a feature for Claude Code that acts like a ‘sleep cycle’ for its memory system, addressing the memory bloat issue introduced by the Auto Memory feature. Auto Memory, released in v2.1.59, allows Claude to take notes on projects, but over time, these notes can accumulate noise and contradictions, degrading performance. AutoDream mitigates this by periodically consolidating memories, similar to human REM sleep, through a four-phase process: Orient, Gather signal, Consolidate, and Prune & index.

  • The AutoDream process involves four phases: Orient, which scans existing memory to understand stored data; Gather signal, which identifies outdated memories and performs targeted searches; Consolidate, which merges new information and resolves contradictions; and Prune & index, which maintains a concise index and removes stale data. This process only triggers after 24+ hours and 5+ sessions since the last consolidation, ensuring it doesn’t interfere with active work.

  • AutoDream operates read-only on project code, modifying only memory files and not the actual codebase. This ensures safety and integrity of the code while managing memory efficiently. The full system prompt for this feature is available on GitHub under agent-prompt-dream-memory-consolidation.md, providing transparency and allowing users to understand its operation.

• Sora is officially shutting down. (Activity: 854): The image is a screenshot of an announcement from the Sora app’s official account on X.com, stating that Sora is shutting down. The message thanks users for their engagement and promises more details on the shutdown timeline for the app and API. This indicates a significant change in the app’s lifecycle, likely due to strategic shifts or financial unsustainability, as suggested by comments noting high costs and low engagement. Comments suggest that Sora’s shutdown is due to its unsustainable business model, particularly after changes to copyright handling that increased costs and reduced user engagement. The app was initially innovative but became a liability.

  • Chasemania highlights the unsustainable nature of Sora, pointing out that the product faced high operational costs and low user engagement. The attempt to respect copyright laws excessively led to a decline in user interest, turning the platform into a liability rather than an asset.

  • The discussion touches on the challenges of balancing copyright compliance with user engagement. Sora’s initial appeal was overshadowed by its inability to maintain user interest while adhering to strict copyright regulations, which ultimately contributed to its downfall.

  • The comments reflect on Sora’s initial success and subsequent decline, emphasizing the difficulty in sustaining a platform that requires high operational costs and strict adherence to copyright laws, which can deter user engagement and lead to financial instability.

• Sora is officially shutting down. (Activity: 1429): The image is a social media announcement from the Sora team about the shutdown of the Sora app. The post expresses gratitude to the community and promises to provide more details soon regarding the app’s and API’s timelines and how users can preserve their work. This indicates a planned and structured shutdown process, aiming to minimize disruption for users. Comments reflect skepticism about the app’s impact and user base, with some users expressing surprise at the app’s longevity given its perceived lack of financial viability.

Unfortunately, Discord shut down our access today. We will not bring it back in this form but we will be shipping the new AINews soon. Thanks for reading to here, it was a good run.

Materials science is the unsung hero of the science world. Behind every physical product you interact was decades of research into getting the properties of materials just right. Your gym clothes contain synthetic fibers developed over decades. The glass screen, diodes, and chip substrate technology needed to read this blog post were only viable due to many teams of material scientists.

Our guest Prof. Heather Kulik was one of the first material scientists to realize that there was alpha in combining computational tools with data driven modeling — 1she did AI for science before it was cool. She has a hard-fought perspective for how to succeed in this field. Yes, she believes the wins are real. To get there you must work hard to deeply integrate domain expertise with AI techniques, and also maintain a discriminating mind. Ultimately what matters is you succeed in the lab, and nature doesn’t care about how hyped a model is. These lessons personally resonated with the Latent.Space Science team and our own experience2.

This episode is a must watch for all aspiring AI for science practitioners. A few highlights:

Designing new polymers with AI: Heather’s group recently used AI to design new polymers that are significantly stronger. These materials were created and tested in the lab, and the scientists who built them were surprised by the designs. The AI had figured out certain building blocks could break in a novel way. The AI discovered a purely quantum mechanical effect, and after convincing their lab collaborators to actually synthesize it, the material turned out to be four times tougher!

The twenty-two-atom ligand challenge: When asked about the role and need of human scientists, Heather points out that AI has a strong understanding of academic chemistry, but is still lacking intuition. Every time an LLM is updated, Heather asks it to design a ligand that contains exactly twenty-two heavy atoms. She has yet to find one that can succeed at this seemingly simple task that any expert could do in a second! Is this the chemistry counterpart to counting ‘r’s in strawberry?

Side note: Heather joked that this comment would date itself immediately, so we decided to see if this was still true three months3 after recording. We found some interesting results! We asked both Claude and ChatGPT to design a 22 atom ligand for both a metal-organic framework (MOF) and a Kinase protein.

• For the Kinase, both models got it right: Claude pulled out RDKit in a python script and iterated on several designs, whereas ChatGPT just one-shotted it.

• For MOFs, both models got it wrong, generating ligands with 21, 23, or 24 atoms, yet stubbornly not getting 22 atoms.

Is there something different about how LLMs reason in the materials and bio domains?

Materials vs biology: The two biggest domains of AI in science have been biology and materials. We asked Heather if there could be an AlphaFold moment for materials. Her answer reframes how we should think about the field:

• First, the datasets in material science are woefully lacking in comparison to the bio world. The closest to ground truth in most cases are noisy DFT datasets. These are just approximations to the real world! The datasets that are accurate are all boring, as Heather quipped “We have really good datasets for really boring chemistry.” Furthermore, good experimental structures are hard to come by and require interpretation. So generating generating high-quality, novel datasets at scale would really drive the field forward.

• More philosophically, AlphaFold is making predictions in a fairly limited space: there are just twenty amino acids. Sure, even here AlphaFold doesn’t get everything right, but it seems plausible that one could learn the entire design space. For materials, each element is a new set of interactions and chemistry, with little to no transferability. This is a massive open problem in material science that we hope some of the smartest AI scientists will want to work on!

The difficulties of trusting the literature: Heather’s team has spent the last few years using NLP and later LLMs to extract data from literature. Even a few thousand data points from these papers can be valuable for guiding her group’s work. One surprising result: sometimes the reported values for a property (say temperature) do not match up with the graphs in the papers! So there’s lots of potential in using LLMs to mine data from the literature, just do it with care.

The role of academia in an ever-changing world: One theme that has been running through many of our conversations has been the changing role of the academic — and the scientist — in science. When startups are raising $100s of millions and hyperscalers and Big Pharma are all ramping up AI-for-science efforts, the academic researcher needs both resources and judgement about problems to chase more than ever.

Resources include data that is organized for machine learning, access to high throughput experimentation labs, and compute resources. These are all things that academics can build together. More importantly, Heather emphasizes curiosity about problems that haven’t hit the radar of the heavily capitalized AI companies. After so many years on the forefront of AI for Science, Heather’s judgement that Chemical Engineering and Material Science still need curious people asking questions with no clear path to money is a welcome beacon in the AI fog.

Is on Youtube!

I really like em-dashes — not an llm I swear!

Heather has a great article that shares far more of her journey and lessons than we could ever cover here on the pod.

We’re getting faster at releasing, sorry it took so long Heather!