Latent Space

Sometime around Christmas 2025, AI engineers noticed a change in agents. They started to work! It’s hard to pin down exactly why. Maybe we finally had holiday downtime to try the newest agents with the newest models. Maybe the models had crossed some capability threshold. Maybe the wrappers around the models had matured.

What I’ll argue in this post is that it was the confluence of the last two. The model and the harness improving together and then their curves of improvement crossing at the right moment. And that dynamic helps to explain what comes next: models keep absorbing the harness into their weights, engineers keep deleting what got absorbed, and what remains is a harness for human attention rather than for the model.

Lukasz Kaiser, one of the people who invented the Transformer, said on “Unsupervised Learning” in June:

“The change last winter, last Christmas — it’s a little hard to pin down. I mean, the harness changed and a little post-training changed and then new pre-trained models came… but it felt like a big jump which is not that easy to pin down what did it.”

The answer to “What happened?” isn’t solely in the model weights. It’s in the system that grew up around the weights.

The answer is in the agent harness.

Think back to November 2022, when ChatGPT was the most advanced AI tool. The only capability at its disposal was next-token prediction and some Reinforcement Learning from Human Feedback (RLHF) that allowed it to act like a helpful assistant. No tools, no search, and no reasoning.

The original ChatGPT was confined to its training data and the prompt you sent it. No more, no less. It was a brain in a vat.

The agent harness is a way for the LLM to break free from that confinement and interact with real digital information space.

An agent harness is everything besides the model weights that makes the agent work. The environment, tools, context and guardrails that surround the model. Without the harness the model is a brain in a vat. It can take an epistemic action, but needs the harness to actuate that decision in real digital space.

The harness is like giving the mind of the model a body. With the harness, the model can perceive (context), act (tools), persist information (memory and compaction), and enforce its boundaries (permissions and guardrails).

Two curves run through the path of model / harness evolution. What the harness asks of the model, and what the model can deliver in practice.

The gap between these two curves is equal to the effectiveness of an agent, and the closing of that gap is what I’ll argue led to the tangible improvement in agents that Lukasz Kaiser referenced.

Here’s how the gap closes, in stages:

1. ReAct, “The Harness on Paper” (October 2022): ReAct is a prompting technique to get models to reason through prompting. It’s the agentic loop on paper, external to the model weights. It defines the idea of an “agent loop” where a model reasons -> acts -> observes -> repeats. Again, the ReAct loop exists only as a prompting method. Prompting is the only reasoning method that exists at this time and no one calls it a “harness.” Toolformer (Meta, Feb. 2023), that same winter, hints that tool use could be trained in rather than prompted. It’s a bit like Alan Turing’s idea of the computer before it was instantiated in a physical substrate. A powerful idea that is only later made manifest. (ReAct predates ChatGPT by a month — October 2022 vs. November 2022). Both the curves are near zero at this point. The gap is small because we are just getting started.

2. AutoGPT/BabyAGI, “Premature Autonomy” (Spring 2023): With AutoGPT/BabyAGI, the harness curve sprints ahead of the model capability curve. Both hand the model full autonomy, asking the model to act as an “autonomous employee,” but the models at this point are still little more than brittle next-token predictors. A loop doesn’t add capability to a model. A loop amplifies the capability a model has, and below some threshold the loop amplifies errors rather than reliability. Consider the power of compounding in the negative: 95% reliability per-step over a 20-step task results in a ~36% average success rate. The harness hands the model an assignment it has no realistic chance of completing. This is where the gap is at its widest and the next 18 months are a reaction and attempt to close that gap.

3. Cursor/Copilot, “Retreat to Human in the Loop” (2023 - 2024): The first AI-powered IDEs recognize the failure-mode of giving the model too much autonomy. They close the gap by pulling the harness curve down below the model curve. Don’t give the model the loop directly; give the human the loop and empower the human to orchestrate the loop while the model speeds the human up. The first version of Devin tries to hand the autonomy back to the model. A test from the team at Answer.AI shows that is still premature, with a ~15% success rate. It’s evidence that the move from the IDEs to retreat from full autonomy is not cowardly, but the correct move. However, while the prevailing tactic is to pull the harness down below the model, models continue to improve. Near the end of 2024, with the introduction of o1 — the first reasoning model — for the first time the gap inverts and we begin to see the first signs of a model capability overhang.

4. Claude Code, “The Curves Cross” (February 2025): The inversion at the end of 2024 sets up an opportunity that someone has to seize: if the model is now ahead of the harness, then a harness intentionally riding the brakes of the model is leaving capability on the table. Claude Code is the first coding agent built to seize that opportunity. It abandons the IDE for the terminal, gives the model bash and file read/write access, and replaces the need for human approval on every change with permission rules. The model is handed the loop again, and this time it understands the assignment. Boris Cherny and team build Claude Code with the next model’s capabilities in mind, not the current one. It is such a hit not because it’s the first product to give the model autonomy, but because it’s the first product to do so at the right time. That time is the crossover point where the model has gotten reliable enough to succeed with autonomy. Claude Code grows to roughly $1B ARR within six months, all because Anthropic seized the opportunity available when the curves begin to meet.

What happens next is that the curves don’t just meet, they begin to braid together.

Today the harness matters, and in a way we can measure. Harness-Bench ran the same model over the same 106 tasks in different harnesses, and scores ranged from 52.4 to 76.2: a 23.8-point spread with zero change to the model. Half the agent is the harness.

OpenAI achieved a similar result on ARC-AGI-3 with harness changes. Adding only retained reasoning and compaction, GPT-5.6 Sol’s ARC-AGI-3 score tripled from 13.3% to 38.3%.

What’s happening under the hood is that Reinforcement Learning (RL) has moved inside the harness. From OpenAI’s codex-1 release announcement in May 2025: “codex-1 was trained using reinforcement learning on real-world coding tasks in a variety of environments.”

The two curves join and start to braid as one unified system.

This is the dream of Toolformer manifesting in reality. Rather than a tool prompted from the outside, now tool calling is trained from within the environment of the model.

Then, as the models are trained in the environment of the harness, they start to absorb the harness capabilities into the model weights, learning how to auto-compact with knowledge of their own context window, for example.

GPT-5.1-Codex-Max launch:

“The first model natively trained to operate across multiple context windows through compaction.”

Once the models absorb the harness capabilities, the harness can shed the scaffold. It’s production by reduction. Thariq Shihipar from Anthropic said that the team recently deleted 80% of Claude Code’s system prompt.

The measure of the pace of agent harness evolution is how much of the harness you get to delete, while retaining the same capability level. This is the future we need to build towards as AI engineers.

This, then, is the loop of model / harness evolution: train -> absorb -> shed -> repeat. The model climbs to the next thing it can’t do yet.

The jump that Kaiser pointed out is hard to pin down because it’s not a discrete event. A pre-training leap is noticeable because you can articulate it in a model card. A model / harness co-evolution jump is less so, because there’s no documentation of the evolution process. That’s the answer to the jump last Winter: it happened in the space between the model and harness working together.

We need to ask: if every harness capability will eventually get absorbed into the model, what does that leave us with?

Keep deleting everything that the model can absorb. Imagine what your agent looks like at the conclusion of that process. What are you left with in your hand when you’ve deleted everything?

What do the model weights absorb next? Multi-agent orchestration, tool selection, memory...to name a few possibilities. Researchers are building self-improving harnesses that can themselves be trained in a similar way to models.

What’s left at the end of this deletion and absorption process are the human-centric agent capabilities. Things like permissions, identity, trust and legibility. A model that absorbs permissions into itself has dissolved permissions. Absorption doesn’t end the harness. Absorption inverts the harness.

The harness becomes the agent’s interface to the human that operates it.

The harness was born as the human interface to the model. We grew from chatbox to IDE to the terminal. If the model absorbs the computer-facing capabilities, the next stage of evolution becomes one layer of abstraction up. The harness becomes the model’s interface to our human attention.

It becomes the attention-interface.

Ryan Lopopolo said on the “Extreme Harness Engineering for Token Billionaires” episode of Latent Space:

“The only fundamentally scarce thing is the synchronous human attention of my team.”

Tokens became abundant and reliable, yet we remain bottlenecked on scarce human attention.

We see sparks of this already, with Anthropic’s long-running agent progress files and agentic approval queues.

The gap between the model and harness curve doesn’t disappear when the model absorbs the harness. It migrates across the human boundary and creates a new pair of curves with a new gap. The new gap is the space between what the agent asks of the human, and what the human is able to answer.

I predict that within a year, every company building agentic AI will ship a human attention policy surface in the way that every agentic AI company shipped AGENTS.md.

AGENTS.md tells the agent how to work with your codebase. The attention-interface will tell the agent how to work with you. It will govern when it’s allowed to interrupt you, when it should keep working, which decisions it can make alone and which decisions need your approval. And like everything else in the agentic system, it will become a learnable component of the system that can learn with more data. Every correction becomes useful data.

The model began as a brain in a vat. The harness gave the brain a body, then the body started to dissolve into the brain. What’s left for us to build is the thing no future model will ever absorb. The interface to the one true scarce resource: human attention.

When we first dicsussed the Summer of Simulative AI in 2024 we knew it would be a brief summer, but it has recently come back with a vengeance with SimGym in April and now Simile AI’s $2B Series B, backed by GreenOaks and Index Ventures with prominent backers like Fei-Fei Li and Andrej Karpathy, running tens of millions of simulations for Fortune 100 clients like CVS and 85–99% accuracy vs human focus groups.

Time to catch up on why this Second Summer of simulation is working!

From creating Smallville, the landmark 2023 paper on Generative Agents that showed AI characters could remember, plan, socialize, and develop emergent behaviors, to now building foundation models of human behavior, Joon Sung Park is trying to answer a much bigger question: what if we could simulate the world before making decisions in it? In this episode, the Simile co-founder and CEO joins us to unpack the path from generative agents to digital twins, why today’s frontier models still fail to capture how humans actually behave, and what it would take to eventually simulate all 8 billion people on Earth.

We go deep on Simile’s approach to modeling human behavior: long-form interviews, observational and transaction data, randomized controlled trials, population-level and individual-level models, and post-training on the causal mechanisms behind why people make decisions. Joon explains how his research created digital twins that reproduced human behavior and attitudes 85% as accurately as people reproduced their own responses, why models optimized to be rational can be bad simulations of irrational humans, and why understanding “social physics” may require changing model weights rather than simply prompting frontier LLMs.

We also explore the much larger ambition behind simulation: testing products and policies before deploying them, finding counterintuitive paths toward desired outcomes, modeling emergent behavior across entire societies, and potentially tackling problems like climate change, democratic instability, and UBI. Joon reflects on scaling laws for simulation, the economics of data-center-scale simulated worlds, the connection to Thomas Schelling and psychohistory, why simulation is surprisingly similar to painting, and whether we might already be living in one.

• How Smallville and Generative Agents led to Simile

• Why Joon’s team asked: “What if we can just recreate the world that we live in?”

• Why useful personal agents require deep models of their users

• Memory architectures, Markdown files, and the limits of prompting

• “Social physics” and behavioral foundation models

• Why web data captures what people say more than what they actually do

• Interviews, transactions, observational data, and randomized controlled trials

• Why predicting the future matters less than understanding how to shape it

• How Simile creates representative simulated populations

• Simulation versus prediction and the connection to Foundation’s psychohistory

• How to evaluate simulations instead of simply stacking LLM hallucinations

• Creating digital twins of 1,000 real people and reaching 85% behavioral accuracy

• Why frontier models can struggle to reproduce real human behavior

• Why good simulations need to reproduce human biases and mistakes

• Post-training models on randomized controlled trials

• Population-level versus individual-level simulation

• Scaling laws for human simulation

• The long-term ambition to simulate all 8 billion people on Earth

• Whether simulations could help solve climate change or detect collapsing democracy

• Thomas Schelling and the history of agent-based modeling

• Why future simulations could require an entire data center

• Multi-agent simulations and what happens when simulated people interact

• Replacing expensive human panels with synthetic populations

• Why market research is only the starting point for simulation

• Why Joon sees simulation as surprisingly similar to painting

• Using simulation to study questions like UBI

• Whether we are already living in a simulation

• Why AGI and simulation may be the twin technologies of advanced civilizations

• LinkedIn: https://www.linkedin.com/in/joonspark

• X: https://x.com/joon_s_pk

• Website: https://www.joonsungpark.com

• Simile: https://www.simile.com

00:00:00 Introduction and Joon’s Path from Art to AI

00:01:46 Smallville, Generative Agents, and the Origins of Simulation

00:05:03 “Let’s Just Create a World” and the Future of Personal Agents

00:09:53 Social Physics and Behavioral Foundation Models

00:14:08 Prediction vs. Simulation: How Do You Shape the Future?

00:16:59 How Simile Models Real People and Populations

00:25:35 Evaluating Simulations, Digital Twins, and 85% Accuracy

00:30:23 Post-Training Models to Reproduce Human Behavior

00:40:04 Scaling Laws and Simulating 8 Billion People

00:43:10 From Schelling to Society-Scale Agent Simulations

00:46:13 The Cost and Economics of Simulating the World

00:52:05 Real-World Use Cases, Synthetic Populations, and the Market

00:57:27 The Future of Simulation, Painting, and UBI

01:04:23 Are We Already Living in a Simulation?

01:06:08 Building Simile and Hiring

Vibhu [00:00:00]: Today, we have Joon in the podcast. Excited to kick this one off. Very exciting company. I wanna kick off and ask you the question, talk us through the story of your life. How have you gotten here?

Joon [00:00:13]: Yeah, for sure. I’m really excited to be here. A story of my life. So I was born in Korea, and I lived there for a good 11 years or so of my life, and then my family moved to Boston. So we moved when I was 11, and my parents were doctors, so they were going through their postdoctoral studies. My dad was a surgeon, so he was doing his sabbatical years at the Boston Children’s Hospital. So I grew up there, not too close to tech. I was very much a music and artsy, painting kind of guy.

Vibhu [00:00:49]: Painting.

Joon [00:00:49]: Exactly. I got into painting a little bit later, in high school, but that’s what I used to do. And then I grew up mostly in the East Coast after Korea. So I lived a good number of years in New Hampshire, and then I went to college in Pennsylvania. And I got into more of this tech scene, in college. So I was originally trained to be an artist. I thought that would be my professional career. So it wasn’t a hobby. It was like, “Hey, let’s make a living out of this.” And then gradually, I got really interested in this idea of, hey, the greatest artist often creates their own medium, and the best medium that we had available today was in computation. So I decided to go deeper into that, and one thing led to another, and we can go deeper into this, but I decided that research was something that I gradually got interested in, and here I am.

Swyx [00:01:46]: So there’s a lot that you packed into the research components. You had one of the best papers of 2023, which was the generative agents paper, commonly known as the Smallville paper.

Swyx [00:01:58]: Feel free to call back to anything else that you mentioned, but most people would have heard of you from this. Do you have any statistics on how many people have, like, read it? arXiv gives you something, right? Some stats.

Joon [00:02:10]: Yeah, it’s a good question. How many people have read it, I’m not sure.

Joon [00:02:14]: I know we do keep track of citations, and they are going up quite fast.

Swyx [00:02:23]: Yeah, Google Scholar has 7,200 citations.

Vibhu [00:02:25]: I feel like it made a bigger hit than that, and it was a pretty instrumental paper. It got cited so many times.

Swyx [00:02:34]: It is frequently the answer when people ask, “What is the best paper you’ve read recently?” It’s this one.

Vibhu [00:02:39]: I thought the memory component was pretty underrated. It was a very good early memory system, and one of the biggest papers.

Joon [00:02:47]: Yeah, so maybe I can talk a little bit about how this particular paper came together. So when I got into research, it was back in 2020 when I started my PhD program at Stanford, and that was the year, when we were about to get GPT-3 to be available. So we already had GPT-2, and you could sense that there was this new class of models that was just becoming available in the market, and the team got very intrigued. And the general consensus was, “Well, is this model going to be useful for anything?” “It’s really strange that these models are not trained to do any particular task.” But we decided to take a bet. So a large group of scholars at Stanford, and it was led by one of my co-founders, Percy Liang, and we came together

Swyx [00:03:35]: Who coined foundation models.

Joon [00:03:36]: Who coined the term foundation models. We wrote this paper, where that term came from called Opportunities and Risks of Foundation Models. And during that process, really the thing that I started to think deeply about was, here is a model that is fundamentally new in our ecosystem. The reason why this was new was it wasn’t, again, trained to do anything in particular, but its premise was it could do anything and everything. It was like a stem cell, if you were to take a biology analogy. And I got really interested in this idea that, well, if we were to really think about what are the killer applications that this particular technology would enable, what would that be? Many of my colleagues were using this for simple classification, simple generations. Interesting that these models can do that, but from an interaction perspective, not that interesting. We’ve known how to do that for many decades. And what we came down to was these models are trained on this very broad data from the web, right? So these are human behavioral data. It’s social media, Wikipedia, all these data. So if you poke at the right angle, then you could see human behavior that would just pop out that’s quite realistic, and we’ve never seen that before.

Joon [00:04:45]: So that got us really interested. The exercise that we decided to do, with this particular group of colleagues, Michael Bernstein, Percy Liang, and myself, who ended up becoming my co-founder at Simile, we sat down and we played this game that we call the time machine game.

Joon [00:05:03]: Imagine we were to get on a time machine and fast-forward 10 years and look back. What would have been the single application that will have mattered that would be the most interesting and inspiring? And when we thought, “Well, what if we can just recreate the world that we live in?” it’s really hard to get more ambitious than that. Like, let’s just create a world.

Joon [00:05:24]: And that’s where we started. And initially, we had this paper that was a precursor to the generative agents paper called Social Simulacra.

Swyx [00:05:32]: Before you go further, were there other candidates for the most ambitious thing in the time machine exercise? What was number two or number three?

Joon [00:05:44]: There is a close second that we were considering, which ended up becoming more of these automation tools, especially the vision around really personalized agents that would do things for you.

Swyx [00:05:59]: That’s also happening.

Joon [00:06:00]: It’s also happening. But it was interesting for us, right, in that the reason why, we decided to go with the idea of simulation, one, I was a huge science fiction nerd, and this idea of creating simulation, I was personally really just fascinated. I loved the idea. It’s really cool to see, like, a game town like this and just see these agents live in it. But at the same time, my bet was if you were to create a really amazing personal assistant out of this technology, what you need first is an amazing model of your users. So I told a model, “Hey, can you go buy late dinner for me?” And it orders Hawaiian pizza, and I do not like pineapples on my pizza. Then it totally failed. The way for it to not make that mistake is only by having a deep understanding of who I am. And I gave a very simple and dumb example here, but you can imagine how this core understanding of people is instrumental. This is how, if we have our family and closest friends, they have a good mental model of who we are. That’s the basis of our social connection. So our bet also was this technology around simulation, creating accurate representation of people ought to precede the more complex agents that would automate the world that we live in. So that was the bet. But that was a very close second, and I’m still very much fascinated by it. I think there’s a lot of interesting work that’s going around. My hot take here, though, is I don’t think we’ve seen a true personal assistant that’s useful, in ways that meet the ambition of that particular line of work. I think there are early applications that are interesting, and if you talk to even ChatGPT nowadays or Claude, they know a lot about us. So a lot of the generation it’s doing, I do think it’s much more tailored, but I think the ambition is quite large in that field, and I don’t think we quite have all the right ingredients just yet.

Swyx [00:08:01]: So OpenClaw and these personal agents, what do you want to see from them that they don’t currently have?

Joon [00:08:09]: I do think it’s slowly getting there, but I do generally want them to have much deeper understanding of the person. Right now, you look at the models. OpenClaw, what it’s leveraging is a Markdown file, and I think it’s quite clever, right? So if you look at the generative agents paper, this was the same intuition that we had, where initially when we were creating the memory architecture for the generative agents, and, like, this is, like, back in 2022, so we didn’t really quite have the idea of even agentive architecture or the term agent. But the intuition that we shared with some of the work that’s coming out today was we initially thought, “Well, do we want to make the memory into, let’s say, knowledge graph? Do we want to train a bespoke model?” All of these things. And what we decided to do was, “No. Just forget about all this.” These language models are quite good at modeling text and understanding and reasoning about text. So just put everything in a Markdown file or a text file. You’re done. I thought that was quite interesting that we could do that, and there’s a lot of strength in doing that. But also, there are limitations. It’s the way you retrieve and make sense of data that’s extremely large, it takes a lot of work. So I think that technology is getting better. I also do, however, think, there are certain things you just cannot shape just by prompting the model. So to some degree, you do need to touch the parameters of the model itself. So there is this work that I do think does need to happen, and it is happening. The question is, how far can we take it? How do we source data, and how do you also create an ecosystem where people are continuously feeding data to this model so it’s learning about you?

Vibhu [00:09:50]: What’s the intuition between why you need to do it in the model?

Joon [00:09:53]: My intuition behind the actual when do you train or even post-train a model versus just prompt a model is if the model has to learn the underlying physics of the world that it’s operating in. So it has to learn new social physics. The places where it doesn’t have to train are the places where it already has the physics. We trust the physics. It already has the base statistics, but it’s just trying to react to an environment. Then I think you can just prompt your way into getting the actions out of it. I don’t think the models that are out in the open have yet learned the complete mapping of social physics of humanity. This is one of the core theses of Simile, right? And one of the core reasons why that is the case is if you look at the data that the model was trained on, these models were trained on the web data, like, whatever was available on the web. And these are really interesting data sets, but they are fundamentally the self-exposed attitudinal data with some behavior data that’s sprinkled around here and there. And it has yet to learn the really deep behavioral nature of people, not just what people say they do online, but what they do in real life. And this is one of what I would consider to be the dark knowledge of humanity that we haven’t quite captured. And it’s these data that would also need to get factored into the model creation.

Vibhu [00:11:21]: You call it behavior foundation model.

Vibhu [00:11:23]: There’s a good one-liner here, but outside of that, what type of data do you need? What are you changing on the model level? How do you go about modeling, doing a behavior foundation model?

Joon [00:11:35]: We think about data in three buckets. So one bucket is interview data. It’s quite interesting. Rich qualitative data is interesting. It’s not behavioral, but we would literally ask people, “Hey, tell me the story of your life.”

Vibhu [00:11:53]: It’s just what we’re doing here exactly.

Joon [00:11:54]: The question that you all asked at the beginning of this interview literally is the question we also ask. And we ask our participants to go a little bit deeper, than how far I went. Maybe I can give more of my life story in lieu of this. But the reason why that data is interesting is by learning about this very long-tail information about people, you get a lot of texture around this model, like, this person as a model. So even understanding their childhood memory or even their trauma, their first love, these things, quite informative in ways that’s really hard to predict. So that’s one. Then there are two tranches of what I would consider to be the behavioral data. One kind of behavioral data is observational. So these might be like transaction data, or these might be data that you can get by scraping the web, right? So you can imagine why these data sets would be interesting, right, because they give you the base statistics of people’s behavior.

Joon [00:12:55]: But then there is the last category of data, that I personally think is perhaps the most important, which is the data that describes the causal mechanism, the whys of people. Some of this is covered by the interview data, the qualitative, because people talk about why they made certain decisions. But really, where you get to see the most behavioral aspect of this is in randomized controlled trials, like RCTs. Imagine you have the same setup, but you have a few different variables that you are trying to tweak. Can you get realistic human behavior out of it in ways where, imagine you had this particular option. Imagine you’re even trying to choose whether you’re going to drink coffee or not. The day you drink coffee versus the day you didn’t drink coffee, does your behavior change? That’s a data set that describes a causal mechanism. This is quite important in modeling people. The reason why this is important is oftentimes when people come to us, or not just to us, but the reason why people are interested in simulation isn’t because they want to predict the future. If you’re trying to win against the stock market, predicting the future is interesting.

Joon [00:14:08]: But most people, most decision-makers, what they want to know is, how can we shape the future? It doesn’t really help you to hear that your sales are going to tank in two quarters. They’re just gonna say, “Wow, that sucks.” What they want to know is, well, what do we need to do now to avoid that future? That’s the causal mechanism. And this is also very hard data to come by, right, because the world is our ground truth, but it happens once. So in a very controlled setup where everything is equal except for one variable, this kind of data set rarely happens. So this is a reason why this data set is both hard to come by and quite important if you’re trying to model human behavior.

Swyx [00:14:50]: So behavior, I think, is the hardest data set to acquire. What is out there? What is even possible? You’re not going to know a lot of details about my life. I don’t even have data for myself on my own health or habits, and I just don’t log everything. So how can you have that data?

Joon [00:15:14]: So we run a lot of randomized controlled trials.

Swyx [00:15:17]: But you put people in the lab, they watch them sleep, or what?

Joon [00:15:20]: We do care a lot about the consent process. People know that we invite them to be a member of this community to both share data and have themselves represented in different forms. But we bring a lot of people to the lab, or virtual lab, where we design experiments that would pose them real behavioral decisions. And often in these experimental setups, what makes the difference between what is attitudinal versus behavioral is whether the stake in your decision is real. That’s ultimately what makes it behavioral. So in these setups, we are inspired by our colleagues in social sciences, psychology, and so forth. So when they run studies, the techniques they utilize is imagine there’s an online store that you’re inviting people to come by. Then whatever they purchase in this experiment, they actually get that item delivered. Like, these are the things that make the stakes real. So we run a lot of these experiments, and we also do partner with firms. Right now, we also have customers who are quite excited to at least give us a glimpse of the behaviors that their users exhibit so that we can get a little bit deeper understanding of how people behave in these different platforms.

Vibhu [00:16:39]: I think on the customer side, they have a lot of data about their users, who has bought. They have the action data.

Vibhu [00:16:47]: Can you walk us through an example of what someone comes to you for? What questions would they want solved? Do you customize a model for them? Do you have something off the shelf? What does that look like?

Joon [00:16:59]: Today, when people leverage our models, it’s often to better understand the population of their interest. So usually, the start of the relationship, we come together and hear about what population they want us to model, right? So it might be that if you’re a CPG company that’s selling to all of the US, then maybe it’s fairly straightforward. You want to model the gen pop of the US. But at the same time, if there is a vertical or if there’s a market that they’re trying to go into, imagine, they want to better understand, let’s say, people in their 20s and 30s living in California. That’s a much more specific population. So we hear about this population, and we go recruit these people, with consent, and with incentives, and we collect some of their data and create a model of these people. Then what our product allows you to do is query them. So it can take as input a filter that is a description of the population that you want to talk to, just like the one I just mentioned, and an environment. The environment can literally be survey questions, behavioral experiments, It can be A/B testing. Oftentimes, the core use cases are things like concept testing, to start with. But also, people sometimes want to do focus groups or one of the fun use cases that we also serve is even modeling things like earnings calls for public companies.

Joon [00:18:21]: So these are the use cases that we often start with.

Swyx [00:18:23]: Concept testing, is that an established term? I’ve never heard of concept testing.

Joon [00:18:27]: Yeah. So it has to do with they have, let’s say, different messaging, different products, different ideas.

Swyx [00:18:32]: It’s like a marketing exercise.

Swyx [00:18:33]: Okay, got it. Got it. Politics?

Joon [00:18:36]: We do, have a strategic partnership with Gallup, and of course, Gallup is deep into policy space and so forth. Right now, we have not worked deeply with politics, like that area just yet, however.

Swyx [00:18:49]: I’m curious if there is demand or if they really would have different needs that somehow fundamentally don’t mix with your existing, users or people.

Joon [00:19:00]: I think there’s certainly demand.

Joon [00:19:02]: But we are very much mindful of how this technology gets adopted and the societal impact that we’ll end up having with this technology. And I do see politics as an area where a company has to be particularly thoughtful about the way they operate and make impact. So this is where we also want to make sure that we form enough of guardrail and perspective on how to leverage this technology before we go on to serve markets like the politics.

Swyx [00:19:29]: I’ll give people an example. one of my favorite shows is The West Wing. I don’t know if people have watched.

Swyx [00:19:34]: One of the key storylines is, like, the president has, multiple sclerosis, but they haven’t. they need to figure out how to disclose it. So they run a poll with a fake governor and ask people to respond on the poll,

Swyx [00:19:47]: They try to make decisions based on the results of that poll on, like, how well they’ll be received, like where, how should we play this?

Swyx [00:19:54]: And I’m like, well, I think those counterfactual things, I would use a simulation for this if I could trust it.

Joon [00:20:01]: For sure.

Joon [00:20:02]: In that show, how’d it go?

Swyx [00:20:04]: In that show, it was, like a foregone conclusion. They were like, “We know it’s bad. We just don’t know how bad.” And then the poll came back. It was like, “It’s really bad.” And then they just did it anyway.

Joon [00:20:14]: Part of it is to show, right? So you’re, you’re looking at the idea

Swyx [00:20:17]: Maximizing drama.

Joon [00:20:18]: How bad could it be? Oh, it’s horrible.

Swyx [00:20:20]: And to some extent, I think that is part of the trick of the, or the challenge or with being a customer of yours, which is that if I know it’s. if I roughly know and can intuit

Swyx [00:20:35]: What the effect is going to be, do I need you? What sensitivity of it, of effect do I need in order to make a decision, right? So for example, if I, my approval rating is 50%

Swyx [00:20:48]: And I, they have this negative piece, news item comes out, and it drops to 30.

Swyx [00:20:52]: If it drops to 20, if it drops to 40, do I care? No. It, I know it drops. It’s negative. So when do I care about simulations?

Joon [00:21:01]: You do something that’s clearly bad, that’s not popular, and people don’t like you, like, yeah, it’s like

Swyx [00:21:05]: You don’t need a simulation.

Joon [00:21:07]: Yeah. Well, so there are a couple of things. one is, there are use cases where, like every day, developers, designers, policymakers, marketers, every single day, they create assets. They create new products. And turns out, it’s many of the decisions in hindsight is obvious. Yes, of course this is bad, but we still run those studies because understanding the magnitude and understanding how acute something is quite difficult, even if, we feel like, of course, like this makes sense. this is the reason why we make so many mistakes. Like, every time somebody goes online and say something that has huge backlash, you look at that and like, “What an idiot.” However, it’s tough. That’s one. There’s also another aspect here, which is, again, this is the reason why simulation is different from prediction. In simulation, in the ideal case scenario. So what simulation is trying to show is it’s trying to show each step of the way or each step that we need to take to get to a certain outcome, right? So in the most advanced simulations, sometimes the next step that we’re suggesting might be quite counterintuitive. The analogy that I sometimes give, and I ground it in a more realistic example, but, I, as I mentioned, I’m a huge fan of science fiction, and I don’t know how, many of the audience members have read, like, things like the Foundation series by Asimov.

Swyx [00:22:37]: Oh, yeah. We’ve mentioned psychohistory a number of times.

Joon [00:22:39]: Okay, fantastic. So I might be, talking to the right crew. If you read Foundation series, literally the first act is there’s a group of scientists who have found out that, “Oh, our galactic empire is going to collapse, and we’re going to have 30,000 years of unrest.” And they run psychohistory, the simulator that tries to teach them, “Okay, how can we keep this unrest to a 1,000 years?” And they plan this out, and the first step of that plan is to get the scientists who say, “Okay, this is coming,” exiled into this random place in this, galax- galaxy.

Swyx [00:23:18]: Terminus.

Joon [00:23:19]: Exactly. And that’s so counterintuitive. Like, what a strange move that you literally sent the group of scientists who was raising voice around this potential collapse of galactic empire into nowhere. How is that the right first move? Well, it turns out in this particular simulation, that was the move.

Joon [00:23:40]: It’s these things, right? And the reason why these reasoning is possible is because you’re showing the step function or each step that results in a particular outcome. So really what simulation allows you to do in its highest form is you give it not a problem or question, like what would people answer to the survey? That’s not what we do. What we tell it is, “Here is a goal that we have. In the context of foundation, we want to keep the unrest to a 1,000 years. What is the path that we need to take now to get to that particular future?” And that’s what simulation allows you to do. Now, translating that into real market, imagine you’re a automobile company and you’re about to release a, EV, and you’re trying to understand, well, how do we market EV, to make sure that our stock price goes up? But what if the answer comes down that, well, you can market your EV in XYZ way, but that might change people’s perception around the cars that’s not EV and make your overall sales to go down. Not very intuitive, especially all you’re trying to optimize is EV salesss, and that’s the only thing that you’re tracking, then that might result in a completely wrong solution, or at least different solution than what you would have expected, whether it’s right or wrong.

Joon [00:24:57]: That’s the power of simulation.

Swyx [00:24:58]: For listeners, we covered a similar topic with Mikhail Parakhin from Shopify, where they are working on SimGym. I don’t know if he ever talked to you about it. it’s very similar.

Joon [00:25:07]: I

Swyx [00:25:07]: The goal is increased conversion, but then the journey is very unusual.

Joon [00:25:12]: Journey is unusual.

Swyx [00:25:12]: Yeah. The-- He’s trying to look for interventions on a shopping trajectory, which is similar to what you’re saying. Like, it’s not about the attitudinal, is your word for it.

Swyx [00:25:24]: It’s about behavior.

Joon [00:25:25]: It’s about behavior.

Swyx [00:25:25]: And that’s exactly the difference, right? It’s, like, not about the near-term direction about-- but it’s more about, like, how do you affect multiple turns of interactions.

Vibhu [00:25:35]: You had a good quote at the start about this as well. It’s not about people wanting to know the outcome. It’s about how they can change it, change the way to get there, something like that. But I wanna take it back to how do we know this is grounded? Like

Vibhu [00:25:47]: How do you run evals? How do you test that simulations come through? if I was to do the same thing that you described with, say, your favorite LLM, Opus, GPT-5.6, have some agent to map out these things

Vibhu [00:26:02]: How different are the answers we would get if I give it the same goal, the same objective, make a decent system? You’re saying that you need to change the model weight. You have your own solution to this. But how far off are we, and how do you check if it’s grounded? you have some interesting stuff on your site that points to how you run real evals, but if you could take us through that side. I think that’s one of the big concerns that people have. They’re like, “LLMs hallucinate.”

Vibhu [00:26:27]: “You’re just hallucinating layer after layer,” right?

Joon [00:26:30]: The way we do this, and this is the paper that we worked on after the generative agents paper that really became the, at least for Simile and also the field of simulation and synthetic panels, really became the foundation. Yeah, this is the paper. the paper is called Generative Agent Simulations of 1000 People. Here’s what we’ve done. For this paper, we brought 1,000 people that’s representatively sampled from the US to a virtual lab. And what we have done was we spent two hours collecting fairly wide-ranging data. In this particular study, we focused a lot on this interview data, that was, whose script was taken from this project called American Voices Project. And then we would also pair that with a lot of behavior data and so forth, whatever we can collect within two hours. And then we would send these people away for a couple of weeks. And during that time, I would use this data to create their digital twins. And I would bring the humans, participants back after 2 weeks and have them complete a battery of surveys, experiments, behavior studies. So we have the list here, which included things like behavioral economics games. We would run literally, like, Big Five personality test, General Social Survey. We would also go ahead and run the randomized controlled trials that were published on PNAS. And we would have their digital twins predict how the source individuals would have acted in these studies and surveys. And this is where we could replicate people’s behaviors and attitudes 85 percent as accurately as people would replicate their own. So that was the first really paper that gave this validated results that we can model individuals in an accurate way. And what we ended up finding now, of course, in AI space, so this paper came out at the end of 2024. AI space, a year and a half, 2 years, that’s a lifetime.

Swyx [00:28:24]: Yeah. Just, for listeners who are not seeing the YouTube, I just wanna say, like, the headline figure is 85 percent accuracy, like, which is a big improvement over all the other

Swyx [00:28:34]: Methods that you showed.

Joon [00:28:36]: But the part that was particularly striking to us, especially as we improved this technology even further, was the generative AI models like ChatGPT, Claude that’s coming out, it does give you the right foundation. However, what they do not consider is the true attitudinal and behavioral aspect of people, especially in the population that you care about. So what these models are really good at today is they’re trying to become the super rational, objective machines, right? So you go get their data from places like Mercor, Scale. You talk to professional programmers, scientists to create model that’s amazing at reasoning. That’s what they do. Simile doesn’t care about any of this. The models that we’re talking about here, what we’re trying to create are models that are as dumb as I am, right? So if I make some mistakes, the model has to make the same mistake.

Swyx [00:29:34]: Oh, that’s very hard.

Joon [00:29:35]: That’s very hard.

Swyx [00:29:36]: You’re solving Murphy’s paradox.

Joon [00:29:37]: That’s exactly. And this is a completely different data and training objective. This is also where we see quite a bit of discrepancy in the performance in human behavior prediction between the frontier models, Simile’s model, and the models being created in this space, where in some cases, the model performance of frontier models go all the way down to 20, 30 percent, especially if you go into that more niche population on topics that our customers would care about. On more gen pop, it might be around 50 to 60 percent. So it’s not very robust. Like, you wouldn’t want to make your decision off of these and these findings. If you can bring that up to 85 percent, that is ultimately what people end up getting very excited about.

Swyx [00:30:20]: Yeah. Do we wanna keep going on the paper, routes?

Joon [00:30:23]: Yeah, for sure. So the last one, was an interesting one. So this, paper was the follow-up paper that we had, to the 1000 agents paper, where the idea was now can we augment the models even further and post-train a model based on a lot of randomized controlled trials? So this was an interesting one. The data is always the most interesting part of modeling in many ways. The data that we got here was there’s this, there’s this platform called Open Science Framework. So some, the audience might be familiar with this. And there has been, especially in the social sciences over the past 5 years or so, there has been this concern around replicability of studies. And so it was a bit of a crisis, the scientists acknowledged, where we rerun the study and we don’t see the same finding.

Vibhu [00:31:12]: Oof.

Joon [00:31:12]: It’s tough. And the reason why it’s there-- that was often the case was there’s this survival bias where the papers that get published often need to maintain what we call the value of less than 0.05 in the experiments that we ran. That suggests that only-- there’s only 5% chance that the results that we saw is false positive. But the tricky part was all the papers that were not published, and there’s still a 5% chance that whatever we publish is totally just randomly generated. Like, there’s a 5% chance that, hey, this effect is not real, but it just happened to be real because of the sampling bias. So because of that, what scientists started to do was they started to register their studies. So before running an experiment, they would go to this platform and say, “Here is the data. Here is the population that we’re collecting, and here’s the hypotheses.” And they would just say, “Here is our hypothesis.” Like, “This is what we believe.” And you cannot retroactively change those hypotheses. This is what gives us more scientific statistical confidence that whatever effect that you ended up seeing is true. So that ended up creating this really interesting platform where there’s one platform that has now contains tens of thousands of real-world experiments and hypotheses. And a lot of these are really high-quality, like, professionally designed behavior studies and random- randomized controlled trials. So we got the data and the studies from this platform and used that to make a point. And this particular, model is not, something that we’re serving commercially because this was a part of the open science. But this particular data set, helped us make a point that by collecting a lot of these randomized controlled trials, that are really well-designed, we can make significant improvement in model’s capability to predict human behaviors. So that’s what this paper was about.

Vibhu [00:33:10]: Is this stuff done on a individual level? Like, do I need to tune the model per individual, per company? Is there foundation model changes and then some slight post-training? Anything you can share there?

Joon [00:33:21]: So this particular model was trained. the data we had at the level of individuals, but this particular model was trained. We experimented with both. And this is what we end up doing at Simile too. We always train 2, distinct model. One is what we call the population-level model. The other is what we call the individual-level model. And both take very similar input, which is the description of a subpopulation or individual and a stimuli. In this particular work, we’ve done the same. Here, the results that we are reporting are much more geared towards individuals because we do think that is a harder task in many ways, but that’s what we have done.

Vibhu [00:34:02]: You seen anything on the questions that humans can solve that models can’t solve? So like

Vibhu [00:34:09]: Currently, it’s, I live 5 minutes walk away from a car wash. It’s a 10-minute drive. Should I walk or drive?

Joon [00:34:16]: Huh.

Vibhu [00:34:16]: The model will say, “Oh, walk to the car wash.” And, you don’t have your car.

Vibhu [00:34:20]: Is anything like this a problem in simulation? You would assume, like, very simple for human to think about, but if the model is saying you should walk to the car wash, anything here?

Joon [00:34:32]: It’s less, what can we solve, but I think it’s more about what biases or mistakes do people make that models miss. Like, imagine that you are, like the. When I was still at Stanford, I lived in Palo Alto. So it’s about, I would say, 40-minute walk from the campus. You ask the model, “Okay, let’s go home. What can I, what can I do?” It would likely call an Uber or, give me, the bus time. But for the longest time, I really liked walking back. And the reason why I wanted to do that was not for efficiency. It really helped me think. And I like to walk for, half an hour or 40 minutes or so a day, where I just get to, just think about ideas, research, just get lost in my thoughts. That’s very human activity. Unless the model has seen that and understands the importance of that activity, it would miss these kinds of features. So that I think, is fundamentally what we’re trying to model. Like, what is fundamentally human might not be the most efficient thing to do, might not be the right thing to do, but things that make us who we are.

Swyx [00:35:43]: I’m curious if, there are some data sets that you really want that would materially help you. One version of this may be interesting, which is more valuable to you to acquire as a data set, all of LinkedIn, all of Twitter, all of Facebook?

Joon [00:35:57]: It’s a little bit hard to rank, in part because, there’s, there’s this product saying where no feedback is wrong because it teaches you something about your users. Doesn’t matter what feedback.

Joon [00:36:11]: I think it’s a little bit like that.

Swyx [00:36:12]: So just whatever is bigger.

Vibhu [00:36:13]: What about a different domain? Say it was. What about all of Amazon data?

Joon [00:36:17]: Oh, yeah.

Vibhu [00:36:18]: Shopping data, right?

Joon [00:36:18]: Shopping data. So Amazon data is interesting in that it’s very much behavioral, although, like, what people do on social media, you could squint and say that is also behavioral. But the transaction data is always interesting. It is also most commonly available, however.

Joon [00:36:33]: If we were to look at purely social media, like if you really, if I were, if I had to really pick, Facebook likely is interesting because I do think it is most a default version of people. Because you go to LinkedIn, it’s very much professional environment. So people put up their, they have their guards up, right? And that still is interesting because that is true human attitude and behavior, but it is not your base state. you go to Twitter- Twitter, people have their own crazy personas, or depending on who you are. Like, my Twitter profile and, persona is very much, initially was I was very much an academic. “Hey, I’m here to share my studies.” Now, I share, things that’s related to Simile. But Facebook is one of those more private space where people just connect with their friends. In that way, I do think it shows you a little bit more about who that person is. So if I had to pick, I’d likely pick, Facebook.

Swyx [00:37:30]: Yeah. And you’re interested in, like, the whole person and their background and philosophy. I, is it too clinical or too machine learning-oriented to just say this is just ways to inject variance and biases? The broad question, is, like, is this any better than a randomized, like, combinatorial explosion version? So we have a link to the Tencent

Swyx [00:37:54]: Billion persona paper, where they did not do any of the groundwork that you are doing.

Swyx [00:37:59]: They just did like a cross matrix of here’s all the professions in the world, here’s all the people, possible backgrounds in the world, do a dot product across all of them, and that’s it. That’s your prompt for a billion people.

Swyx [00:38:12]: This will do something. I don’t know if it’ll do what you do, but it gets you some way, some percent of the way there.

Joon [00:38:18]: So this was an interesting paper. Like, what I admired about this paper when it came out was the scale. And you do gradually want to be able to simulate really large societies and interactions. So the scale is definitely admirable. it is relying heavily on the known statistics that went into training the model. So to the extent that you believe that statistics is correct, this is not a bad way to go about this. But the thesis here, and this is something that we also have seen in the market, like if this works, then we have solved simulation.

Joon [00:38:54]: It,

Swyx [00:38:55]: Because I survey, like, okay, 5% of the US population is in construction.

Swyx [00:39:01]: The other 5% is in medicine, whatever, right? And then you just keep going down the list, and then you do the other side. 5% has, like, the big 5 personality

Swyx [00:39:08]: Of, like, neurotic or whatever. That’s it.

Joon [00:39:11]: That’s it. So if you believe that the underlying data set and the platform that we’re leveraging has all the right statistics, then this will have solved it. you’re at that point merely retrieving the knowledge that is already embedded in the model, in the model parameters. That’s not, unfortunately, what we see, where there is such detailed and also niche knowledge about people that if you just take one example, it might feel very mundane, but it’s quite rich when you put together, that you do need to do a lot of bespoke data collection to better understand people. And this is also, I think what makes this particular, job fun, which you want to deeply understand people, and the process of deeply understanding them requires a lot of attention to the details. And you do need to pay attention to and pay respect to the daily lives that people lead.

Vibhu [00:40:04]: I wanna talk about scaling simulation.

Vibhu [00:40:07]: So what can’t we simulate, what can we simulate, and how does scaling affect this? So how big are the models? What if we go from, 8B, like, couple 100 billion

Vibhu [00:40:18]: Like billion000 parameters, billion000? Do we get scaling? Any interesting emergence? Like, at a certain scale, at a certain amount of training, you uncover anything unusual and any learnings from that?

Joon [00:40:31]: What we are seeing is at Simile, so we do post-train our own model. The thing that we’re seeing is the early glimpse of scaling law in simulations. The more data about humans and more compute you ingest, you start to get predictive and predictable gains of the model performance in simulating it, simulating people.

Vibhu [00:40:51]: Ooh. We need a scaling law curve.

Joon [00:40:52]: It’s scaling law. Whenever you find it’s a beautiful thing. And we’re starting to see the glimpse of it, which is quite exciting. But if you talk about the ambition of simulation as a whole, it’s not merely about building a model. It’s about building a model, then creating the agents that become the individuals in a much larger ecosystem. So they’re creating this multi-agent simulation. Down the line, you want these multi-agent simulation to also live in a very rich environment, right? What we are really trying to get to at that point is, hey, can we create. All right, let’s do a time machine game again, and 5 years, 10 years into the future, can we create a simulation of 8 billion people living on Earth? I think that’s quite interesting. And that really is the vision. And once you get to that state, the questions that you can help answer for the society also start to change from my perspective. The answers are fundamentally about emergence of the emergent behavior of society and large groups of people.

Joon [00:41:53]: So the questions that I get excited by, and maybe this is a stodgy- a bit. I have my, academic side of me.

Joon [00:42:01]: And for me, it’s questions like, can we help solve climate change? If you look at climate change as a problem space, this is what we, like social scientists would often call it the wicked problems, problem where you have many actors with competing incentives for trying to make a very complex decision and coordinating that coordination decision. Very difficult to really solve in real life, which is also the reason why we couldn’t solve it. Can simulation help us solve that? Another one is, can we understand the signals for collapsing democracy, or can we understand or can we uncover the origin story of the monetary system? These are societal questions that we never really had a good way of answering. If we can create simulations of our society, you have to believe that these are the problems that we can solve. So that’s really the ambition of this field. And, I also think, yes, I think there’s a Nobel Prize to be won there, which wouldn’t be surprising. And I think there’s some amazing societal impact that we can have to help people make better decisions.

Swyx [00:43:04]: Nobel Prize in economics?

Joon [00:43:06]: In economics.

Swyx [00:43:06]: Oh, I see. I see. Rooting for you to write that paper.

Joon [00:43:10]: One of these days. But, one of the scholars that I was deeply inspired by, When I was coming into the space of simulation, is this scholar, named Thomas Schelling.

Swyx [00:43:23]: Schelling point?

Joon [00:43:24]: So the canonical example of the work that he’s done was he was one of the creators of agent-based modeling. So this was, like, in the 1970s and 80s. It’s very early days, but this was truly one of the first exemplars of simulations. And one of the canonical model from that time, and of course many of these simulations are trying to tackle the societal problems that’s most relevant for their era, it was called the model of segregation. So racial segregation was a big topic, that, we cared about. And what they’ve done was they created this grid world where they had red dots and blue dots. And these dots were, back in the day, like, they were the agents, and they had a simple rule that governed their behavior. If certain percentage of your neighbors are of different color and if that goes above certain threshold, then you move to a new location at random.

Joon [00:44:21]: One of the striking finding of this paper or this agent-based model was for the longest time, people thought the segregation within society was caused by explicit and overt racism.

Joon [00:44:34]: But if you look at this model, people’s preference towards living with people of the same color, that preference can be very minute.

Joon [00:44:42]: But the very small difference causes the society to segregate completely over time. This was very counterintuitive for a lot of people. And this particular work ended up informing housing policies. Mixed income housing, got really inspired by this work. And Thomas Schelling ends up winning the Nobel Prize for having laid the groundwork for very early versions of simulations. The opportunity that I do see here in the more scientific terms, is agent-based models for the longest, had impact in the 1980s, 90s, to some extent, early 2000s, but it has now gotten forgotten by the community a little bit. Because as you can imagine, red dots and blue dots is not really a rich description of people.

Joon [00:45:31]: But with the emergence of things like generative AI and, in particular, generative agents, we do have an opportunity to create these agent-based models that are high fidelity enough to help us make really complex decisions. And that’s the opportunity that I see. If that truly works, then yes, that is the work that will result in a Nobel Prize.

Swyx [00:45:53]: Yeah. For what it’s worth, and I grew up in Singapore. 80% of Singapore is in public housing, and public housing has, enforced racial quotas for exactly that reason, which is very interesting. okay, so we talk about scaling, we talk about all these, the agent possible applications.

Swyx [00:46:13]: I’m scared about the cost. if you even-- let’s just keep it to the US, about 8 billion people.

Swyx [00:46:21]: But, how much does it cost to model so many hundreds of millions of people?

Joon [00:46:26]: Oftentimes today, we don’t start at that scale, this stage of the, of industry and simulation as technology. But we can get our users extremely rich and meaningful insights even by modeling thousands, tens of thousands of people. And today what we do is every week we are collecting data on the scale of tens of thousands people’s data, and we have panel partnerships that gets us to tens of millions of people globally. So that’s what we do today.

Swyx [00:46:55]: And just as a side note once you’ve collected one person for one study

Swyx [00:46:59]: Can you reuse that same person for all the subsequent studies?

Joon [00:47:03]: That’s exactly right.

Swyx [00:47:03]: Okay.

Joon [00:47:04]: The beauty of this model and these agents is the fact that they are domain-agnostic.

Joon [00:47:08]: That what you’re really trying to understand is what is the fundamental nature of these people? What’s their social physics? And there are a lot of, a lot of, people that does change over time. Like, even, like, even things like, how many times have you gone have you been to, like, CVS the past week? that will change. But there’s so many traits about people that are also known to never change. Like, your risk tolerance doesn’t really change over time. It’s very consistent. So it’s these things that we’re trying to learn. But the scale we are operating is right now hundreds or, tens of thousands to hundreds of thousands. And in many of the core use cases that we are deployed in, and this is more than enough population, to cover those. Really, at that point, what you care about is less the number of people, but more do you have the right subpopulation of interest covered? And this is also the reason why people want a larger sample. It’s not because they want, stronger statistical guarantees. It’s more that can they filter down to any population of their interest. However, you can also imagine in 10 years, if we truly believe that the compute is going to scale, that we’ll have much more availability for compute, and our ambition for simulation is also going to scale accordingly, there’s definitely a reason for us to create an entire data center worth of simulations.

Joon [00:48:35]: Or in my hunch here is I do think in the next some number of years, we will start creating simulations that will cost as much as training a foundation model. But perhaps it’s going to be so valuable to the society that it would be a no-brainer. Right now, even today, like, we are training bunch of new foundation model just so we can say we trained one and we spent tens of millions. But if we can create a simulation at the level of society that would solve climate change, I would run that today. I would raise the money right now just to run that.

Swyx [00:49:10]: Amazing. the follow-up question is, does it also compound if you let the simulations talk to each other?

Swyx [00:49:18]: Or do they already do that today? They don’t, right, as far as I understand?

Joon [00:49:22]: It depends on what simulation you’re trying to run.

Joon [00:49:24]: In the multi-agent simulation setup, the agents do talk to each other.

Swyx [00:49:28]: Right, which is exactly Smallville, right?

Joon [00:49:29]: That’s right.

Swyx [00:49:30]: But a lot of times, for example, in commerce, you’re just by yourself, so there’s no point talking. which is way cheaper.

Vibhu [00:49:37]: But they use all these levels, right? Like, you decide what you will buy based on what other people around you buy and talk about, right?

Swyx [00:49:43]: It depends.

Vibhu [00:49:44]: It depends.

Swyx [00:49:45]: Again, I’m, I’m coming at this from a cost point of view. I’m like, “Oh my God.” Like

Vibhu [00:49:48]: I think

Swyx [00:49:49]: If there is, like, some combinatorial thing of, like, thousands of people talking to thousands of people, then that one million X’s might cost.

Vibhu [00:49:56]: I have a very different view as the cost point aside. Like, running these studies in reality is a lot more expensive, right? Running any study like this is you gotta have people do it, you gotta sign people up. It’s very expensive and sometimes, like, not feasible to run the study.

Vibhu [00:50:14]: But the outcome or the decisions you make are very expensive on them, right? So spend X million on something that, the overall process costs 100 million might as well, right? There’s, there’s a lot of value to be had there. It’s a small cost, but I’m excited on the cost side.

Joon [00:50:33]: To some extent, and when you deploy technology, you often want to deploy in a way where you can replace existing budget or you can make things more efficient, and that is the best way to deploy. However, the way you capture the long-term value of the technology is making the argument that, no, it’s the upside, that by making this better decision using simulation, you have saved yourself or made yourself hundreds of millions or even billions of dollars, and that’s a case to be made.

Vibhu [00:51:06]: Random tangent question. So if you’re doing a lot of inference, a lot of model multi-agent stuff, are you at the point where it makes sense to, train a model that’ very sparse? You’re expecting to do multi-million dollar runs. Are you thinking about this in model architecture standpoint or inference efficiency, or, you’re still at the research phase of it works, we’re not super there yet?

Joon [00:51:34]: Efficiency, we do think quite a bit about. this is technology that is deployed now in some of the largest enterprise companies in the world, and we do process significant number of queries, that are trying to, simulate the populations in the world. So efficiency is a consistent thing. we don’t want to over-optimize too early, so I wouldn’t say, like, this is the higher bid Right now, but this is definitely something that we think pretty carefully about.

Swyx [00:52:05]: Yeah. Are there other case studies? So we, you talked about CVS, talked about Gallup, Deloitte, Wealthfront.

Joon [00:52:12]: Wealthfront is an interesting one, because one of the things they were trying to do, they were one of the first customers that wanted to do product testing that goes beyond just asking people what they think about, let’s say, behavior experiments and so forth. So there, really what we had to do was reason about multimodal input, so images, but also you can also imagine, like, these agents traversing through Figma mockups or websites. So some of the things that our agents can also do is it can be given a domain, like, or, like, a website URL and go use it for a while. It’s these things. And Wealthfront was one of the first, customers, that was very excited about this possibility.

Vibhu [00:52:53]: What have people been asking? Like, is there any demand that we have not covered? Like, UI testing, right?

Vibhu [00:52:59]: I wanna try a new. I wanna ship a new feature, test the UI, simulate how people will do it. Any interesting things that you’re seeing demand for?

Joon [00:53:08]: Today, a lot of the demand does come from like, the places where people have historically used human panels, we can now replace with agents, and these synthetic populations. And this is not replacing human panel. in many ways, the simulation that Simile is building is grounded. So the way that I think about this is we are trying to represent humanity at scale. And in that way, the use cases are what we would expect, but it’s the scale of deployment that surprises me.

Joon [00:53:44]: Turns out there are so many decisions that people make every day in these organizations, groups, and we want to be able to say, “We listen to people. We have consulted our users.” But in reality, that is rarely the case because getting to people and asking them many questions, it’s difficult. It’s both costly, time-consuming, but most importantly, people are just not available. If I had to answer 1000 survey questions for this one particular, vendor, even if I wanted to do that, like, I would never do it. And that’s very much the case. What simulation can do is ensure that the voices of people are always represented in rooms where the decisions for them is made, right? So all the stakeholders of this particular product launch, ideally they’re consulted. That’s what this technology really is trying to enable.

Swyx [00:54:39]: In my mind, that means it skews towards more consumer focus, right? Like, anything with a wide enough customer base where you do benefit from the diversity that you represent. What are some rough statistics, just for people who are not familiar with this market in general, what’s the market size that. I’m sure you have some, like, rough numbers. market size is, like, a vague question

Swyx [00:55:01]: But, like, how much do people spend?

Joon [00:55:03]: So market research is a $100 billion industry.

Joon [00:55:06]: But the thing about simulation is not a tool for market research. Simulation is a tool for human decision-making. So the question around what is a TAM here is quite tricky, right? Because it’s easy to say, “Well, market research TAM is roughly 100 million or 100 billion.” so is it a TAM? And not really, right? Because in many ways, you’re trying to inform all human decision-making. You’re trying to inform every decision that are made about humans for humans. What is a TAM for that? It’s really unclear. And I’ll be honest. Like, I have a scientific background, I have a research background, so I didn’t come into the field calculating, oh, what is the TAM for human decision-making? But I just had to assume, well, if we can inform every decision that is made about human for human, that has to be big.

Swyx [00:55:58]: Some- something valuable.

Joon [00:55:59]: Exactly.

Swyx [00:55:59]: To some extent, you are a unicorn founder now, and you have to care as a CEO. But, like, I do think, like, yeah, when you go into these boardrooms with people that you’re quoting millions of dollars of contracts for, like, you have to say, “Well, here’s what you spend on humans-”

Swyx [00:56:15]: “. And here’s what we save you, and it’s 85% similar.”

Joon [00:56:19]: And certainly, the value case, is something that we care deeply about. Like, what is the value that we provide to the users and the decision-makers? But this is also where, like, as a founder, I think valuation only tells one very superficial aspect of the story, and I try not to think too much about valuation, in general, because that’s not what also motivates a team or certainly doesn’t. I’m, I-- Again, the interesting thing about researchers is we are happy living in academia, getting paid next to. we get paid okay. we don’t get paid that much, as a researcher here in academia, but it’s the impact and it’s the, it’s the value that we can provide to the individuals and the society that really drives us. And in that way, ultimately what drives us is the impact. Does the simulation we provide have a real impact in people’s decision-making in ways that progresses our society forward? If the answer is yes, then yes. that has to be great business, and we see that in numbers, and we do care deeply about that upside story, but that’s the heart of it.

Vibhu [00:57:27]: Do you have any timeline predictions? So we talked about scaling laws of simulations.

Vibhu [00:57:33]: You brought up, okay, maybe one day we can simulate how to solve climate change.

Vibhu [00:57:38]: Where are we now?

Vibhu [00:57:40]: If that’s not the end state, what is an end state, and what does progress look like?

Joon [00:57:45]: So what I sometimes tell people is simulation as industry, it feels a lot like where GPT-3.5, GPT-4 was, for the AGI saga, which is we have now technology that is powerful enough to do real damage on the verticals that we are tackling. At the same time, there’s a lot of progress that is yet to come. And that’s, I think, where this is. So the way I see it, I do think there will continue to be breakthroughs both in data, in algorithms, and there will be much more aggressive scaling that will also happen over the next few years. But I think that’s roughly where we are.

Swyx [00:58:27]: I think that was about the rough set of topics. Anything else that we should have asked you or you wish people asked you more about Simile?

Joon [00:58:38]: I think the, what’s, for me, what’s quite fascinating about simulation, it is very impactful technology, but it is also very interesting technology, both in terms of, like, what it means for human society, our philosophy. And the way I sometimes interpret simulation is. So going back to my background, I as I mentioned earlier, I started my career as a painter. it was a professional pursuit, and I did oil painting, for figures. So I got my training originally in the realism studios, and that’s what I spent a lot of my, years, doing. Simulation is a lot like painting, right? The best paintings teach you something deep about the subject that you’re trying to represent. And it is always not a perfect representation. It-- No painting is perfect. There’s always some small differences and discrepancy, but what it does is it tries to highlight the thing that matters the most about the subject.

Swyx [00:59:47]: The essential

Joon [00:59:49]: The essential essence.

Swyx [00:59:49]: Yes. He, you, he’s brought up some of your work.

Vibhu [00:59:53]: Just nice to put it up.

Joon [00:59:54]: Yeah. So these are some of the works. So this is from, my, personal website that I maintain when, I was still a researcher.

Swyx [01:00:00]: I think a lot of people will say, like a Picasso, like anything postmodern is, like, very much focused on the essence.

Swyx [01:00:09]: Right. yeah, but I don’t know if any one of these evokes something that you like to tell the story of.

Joon [01:00:15]: No, it’s one of those things where, each of these paintings, drawings, whatever it may be, it is trying to surface something about the subject that you feel deeply about onto the surface. when I was a painter, and artist, the topic that I cared really deeply about was, the more mundane aspect of human lives. This shows up in some of the, some of the work that I’ve done, where, like, I did this entire study of a rural town where I went around and took photos of people for not really doing anything special, but just living their everyday lives. I thought that was the most interesting thing. I’m somebody who has this perspective where, the world is oriented around this fractal shape, and you have two choices to understand the fractal shape. You either go outward and try to explore as much as you can to understand the broader shape of the fractal, or you go inward because, the outward resembles the inward, shapes. And understanding the mundane aspect of it was very much that. Simulation has a lot of this, right? You’re trying to understand even the most mundane aspect of people. When put together- teaches you something really deep about that individual and the society. So I think that’s what’s interesting about simulation, the way, the same way that AGI helped us better understand or really think critically about humanity and human intelligence, simulation is really an exercise of understanding more about human society and our collective lives. So that I find to be, yeah, particularly interesting.

Swyx [01:01:56]: Yeah. Now you’re reminding me that some of the best biographers, documentarians, and even photographers, they’re taking a photo of you.

Swyx [01:02:05]: But before I take a photo of you, I must spend-- I must, like, follow you for a week just to understand you?

Swyx [01:02:11]: Which some artists, some do. Part of your work, there’s a very famous book called Working. I don’t know if you’ve, been referred to it before.

Swyx [01:02:18]: It’s very famous, like, to the point of having a Wikipedia page

Swyx [01:02:23]: About this like, really depth understanding and interview of people as they, about their lives, which seems mundane, but is told in a very, compelling way. Yeah, 1970s as well.

Joon [01:02:34]: Okay. It was an amazing decade.

Vibhu [01:02:39]: Before closing question

Swyx [01:02:41]: Okay, here we go

Vibhu [01:02:41]: You said that you started Simile with your 10-year question, right? If we do that now, 10 years down, what can we simulate? What would you simulate if, like, if you’ve made significant progress, are there any questions outside of the ones that we brought up? Any- anything that you think is most impactful? Anything that you would go vision 10 years out?

Joon [01:03:03]: In many ways, as I mentioned, I am somebody who is very much impact-driven. So the what would inspire me is I would want to ask, 10 years later, what would be the most important societal question that we as a society have to ask? I would love to tackle that. Like, do we need UBI? That could be an interesting one.

Swyx [01:03:24]: Ooh, has anyone done that?

Joon [01:03:25]: Well, we were thinking about it.

Vibhu [01:03:27]: Can we get access? Can we just

Swyx [01:03:28]: So OpenAI, this is, like, just trivia now. Like, OpenAI, or I think Sam Altman funded a study on this

Swyx [01:03:35]: In Africa, and the answer was no.

Joon [01:03:37]: The answer was no. But, what, was it something about the implementation?

Swyx [01:03:41]: Yeah, I know. It was a skill issue.

Joon [01:03:43]: Or was it something about, But this is the thing. See, when Sam

Vibhu [01:03:46]: Funny news article

Joon [01:03:46]: Altman funded this particular,

Swyx [01:03:50]: He spent 14 million dollars? Oh my God.

Vibhu [01:03:52]: It’s a little more.

Joon [01:03:52]: Quite a bit. But this is the thing. This is the reason why you want to run a simulation. You spend 5 years, 40 million dollars on this one study and have one finding, but if you can run simulation many times instantly, then that’s the value.

Swyx [01:04:07]: I feel like that one could-- you could have done in a simulation. Like, if you can do the housing study, you can do the UBI one. Like, I, come on.

Vibhu [01:04:13]: I think sometimes people will spend the money because they wanna verify what you think, right? Like, sometimes you just wanna. Is it right? Like, you gotta test it.

Swyx [01:04:23]: Okay, closing question. What are the chances we are in a simulation right now?

Joon [01:04:28]: So it’s a fun question, and I assert at some point I just answer, yeah, we’re definitely in a simulation. But what I do, feel, however, is, whether we are in a simulation or not, that, I don’t think that makes our experience any less real. And I think that’s fundamentally, like, what I believe in. Maybe we live in a simulation, maybe not, but for

Swyx [01:04:48]: It’s real to us. Yeah.

Joon [01:04:49]: Yeah. For me, I don’t really care.

Swyx [01:04:50]: Yeah. Unless you die and you wake up in, like, the level higher or below.

Joon [01:04:55]: That would be interesting.

Vibhu [01:04:55]: I feel like you wouldn’t care. Once you die, then you find out you’re in a higher level.

Joon [01:05:01]: I worry about it when I die.

Swyx [01:05:04]: I think the other thing that. Okay, so I like the mathematical answer to this, which is, like, the, sheer number of possibilities that you are in a simulation far outweigh the sheer number of possibilities that you’re not.

Swyx [01:05:16]: Except for the simplest answer, which is, it is computationally very expensive to have you be a simulation. okay, great. You’ve been very generous with your time. Congrats on all your success. I met you just after your Smallville paper and had no idea that you could build, like, such an enormous company. And then now you’re like, “Well, it’s a $100 billion market, but that’s just where we’re starting.” So this is, very exciting.

Vibhu [01:05:42]: I think $100 billion market was not the term. That was only part of it.

Swyx [01:05:45]: Yeah, exactly. It’s, if you’re thinking too small.

Joon [01:05:48]: Well, I do believe that, maybe my final note here might be, again, I love science fiction. You look at any advanced civilization in science fictions, there’s 2 twin pillar, technology. One’s AGI in some form, and the other is simulation. So I think the market’s pretty big here.

Vibhu [01:06:08]: Tell us about the company. You guys just raised a lot. You’re half a research lab, half a company. you’re hiring. Where are you based?

Joon [01:06:15]: Yeah. So we’re based in Mission Rock, so not too far away from, where we are right now. So we’re in SF, but we are also bicoastal. So we have our, team. I would say our headquarter is in SF, and we have a lot of our technical talent in SF, and we do have a smaller office that just opened up in New York. We are, as a company, an interesting one in that today, there are AI neo labs and then there are AI product companies. Simile truly is both. So this is a company that was founded by 4 founders, myself, Michael Bernstein, Percy Liang, Lainie Yallen. Michael, Percy, and I are all researchers. So of course, Michael was one of the authors of the ImageNet, kickstarted the AI revolution back in 2013, has been instrumental in human-centered AI. Percy coined the term foundation model, and is a, one of the greats of the AI researchers today. And Lanie is my business counterpart, where she led some of the fastest-growing AI native companies from their seed to A and B. But we have this DNA at the company where the vision of the technology that we’re creating is continuously developing, that we are getting people who were my lab mates. We are about 60 people right now.

Joon [01:07:28]: 15%, almost 20% of the company population are just my lab mates from Microsoft Research lab.

Joon [01:07:36]: And we It’s quite fun because many of them then had gone on to OpenAI, Google Gemini, and these places. And so it’s been a few years since we really got together and had a chance to work together. But now they’re coming back and really building out this vision that I find to be quite exciting, and that excitement is shared. So there’s that motion at Simile where we are a group of researchers trying to do something that no one is working on that we find to be the most impactful potentially. But at the same time, this is, again, technology that can make impact today. So we have an amazing group of engineers, product people, and designers, who are sitting here with us trying to imagine what does it look like to help people understand what simulation can do and make real-world decisions with this. Having both and then deploying it to some of the largest customers in the world today, it feels quite unique.

Swyx [01:08:30]: Yeah, it’s very compelling. One part of it was this is the call to action. Like, who are you hiring? You’ve done part of it, which is you have-- you’ve got a very talented group. Who are you hiring? Like, what roles?

Joon [01:08:41]: So honestly, at this point, we’re hiring across

Swyx [01:08:43]: Everything

Joon [01:08:43]: All, section. we are always excited to bring on, amazing research talent.

Joon [01:08:49]: So if you’re interested in working with, our lab mates, we are always welcoming of amazing, researchers. But also we, hire, amazing engineers, that some of whom I, like, I respect the most. Many of them come from places where we have personal connections with, so many of the members are from Figma, Notion, Rive, and so forth, but also more broadly from the companies that we as a team have really admired. So engineers both in the product side, infra side, we’re all looking for those hires.

Swyx [01:09:24]: Well, lots of people. I think you made a really good case. So thanks, and, we’ll see you in the simulation.

Joon [01:09:30]: Amazing.

Joon [01:09:31]: See you all there.

Less than a month ago we had just featured Poolside’s Model Factory with Eiso Kant on the pod (following our Paper Club coverage):

It appears that Jensen really, really liked Poolside too, as he went from investor to doing licensing their factory and hiring 109 of their employees:

Unless things changed drastically, this accounts for the overwhelming majority of the technical Poolside employees:

Eiso Kant [01:52:31]: We are hiring on every possible role in applied research and engineering in the company, from training all the way to evals to post-training architecture. Like, we are still in a world where, individuals can have massive impact. And I think our pitch to join us —I think we are one of the places where it’s the highest ratio to individual to impact, Right? Less than 70 people built this model. Less than 115 between engineering and researchers, like, together did this effort, and that’s a very broad definition ‘cause I put myself in the 115 list.

As the founders say, this is “not an acquisition and not an acquihire”:

We’ve been calling the Windsurf-Google and Character-Google and Scale-Meta and Instacart-OpenAI deals execuhires because usually the executives go leaving the employees with a rich payout but holding the company remaining, but this is a first time it is happening the other way around. The action amounts to founders pivoting the company extremely hard to SOMETHING, and finding an EXTREMELY comfortable golden parachute for investors and employees to continue on with the original mission or stay aboard for the new pivot:

For the last 3 1/2 years we’ve been directionally correct in a race where capital requirements went vertical.

At the end of last year, we had a 6 week window in which to raise $2 billion dollars to pay for a 40,000 GB300 cluster coming online in January. We didn’t close it in time, and we lost the cluster.

and:

We also know that at 10,000-20,000 GB300s we would produce a great model that could rival the current frontier. But the scale of next year’s frontier models requires far more than an order of magnitude larger cluster. And for this the constraint today is not only capital, it is physical data center space and contracted compute.

The compute needed to be at the frontier of the current model recipe is going vertical, and as the world accelerates along the axis of Recursive Self Improvement this will only become more evident.

To this end, the PIC infraco, spun out in Jan 2026, is also interesting in its ambitions…

We’re confused too, and the founders say they are “not ready to share the updated vision”, but everyone here is coming out with a lot of money so we’re just interested to see what’s next for everyone on the 3 different directions emerging from OG Poolside.

The only hints left to us:

We wholeheartedly believe that everything economically valuable, scientifically interesting and a lot of what will be personally meaningful is going to be underpinned by Al. The world has not yet reached 0.1% of this transition….

… We believe human level capabilities of intelligence will be fully commoditized by open source models, while super intelligence will likely not be.

The world has two types of economically valuable problems, those that are intelligence bound, and those that are experiment bound. The first are problems which we can solve by scaling up intelligence e.g. building software, doing accounting, solving a math theorem. The second are ones that require real world experimentation to progress, and no amount of increased intelligence without experimental results will make progress. We could put 100,000 of the world’s brightest minds together to solve cancer but without a real world experimental feedback loop, they likely never will.

Today’s model revenue is from coding and soon from all of knowledge work. In the future, companies who can go beyond human level capabilities will tap into revenue coming from scientific discoveries where there is a true data moat derived from real world experimentation. In our humble opinion, Al’s ultimate value will not derive from the first kind, that will become a low margin commodity, but it will from the second. Al will become the world’s most valuable scientific discovery engine.

Fascinating. Sounds like we could not have timed our AI for Science podcast better.

AI News for 8/19/2026-8/20/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

OpenAI and Anthropic Expand the Agent Product Surface

• OpenAI pushed several desktop and builder features in one wave: @ChatGPT launched an Apple Messages plugin for ChatGPT Work/Codex on Mac, enabling message search, catch-up, drafting, and sending from the desktop app. @OpenAIDevs also added collaborative editing for ChatGPT Sites, with teammates sharing a project while Codex manages git/CI; shared read-only conversation links and PR-context sharing further push ChatGPT/Codex toward being a coordination surface, not just a chat UI. On the API side, transparent backgrounds in GPT-Image-2 are now in preview for reusable design assets.

• OpenAI’s desktop memory/workflow features continue rolling out geographically: @OpenAIDevs said Computer History and cross-app memory are now available in the EEA, UK, and Switzerland for Pro/Business/Enterprise Mac users, with Record & Replay also live there. Together, these features point to a product strategy of capturing user workflows on-device and turning repeated actions into reusable skills.

• Anthropic made its agent platform more composable and production-ready: @ClaudeDevs announced general availability for computer use, browser tool, Skills API, and Files API on the Claude Platform. The Skills API adds versioned reusable procedures; the Files API now supports expiration control, 5x higher rate limits to 500 RPM, and 1 TB/org. Anthropic also published an AG-UI adapter for Claude Managed Agents, mapping chat threads to managed sessions and streaming text, tool calls, and thinking into custom UIs.

Model Economics, Usage Limits, and the Enterprise Shift Toward Open Models

• AT&T became the clearest public case study yet for hybrid routing: the most consequential enterprise datapoint in the set came via @Hesamation, summarizing AT&T’s internal AI deployment: 40% of employee AI usage already routes to open models, with a target of 60–70%; coding costs are down 56% for only a 2% quality drop, at 45B tokens/day. That supports the increasingly common view that frontier closed models remain reserved for the hardest tasks, while “good-enough” open models eat the broad middle of enterprise demand. @amir explicitly framed this as a warning sign for OpenAI/Anthropic’s enterprise moat, while @ollama welcomed AT&T to open models.

• Pricing pressure is intensifying across closed-model distribution: @eglyman announced GPT-5.6 Sol at 50% off through Router, and both @github and @code amplified the temporary discount for GitHub Copilot / VS Code users. At the same time, user sentiment suggests supply constraints are surfacing as usage caps rather than degraded quality: @bridgemindai complained that a $200/mo OpenAI Pro plan could be exhausted in a single heavy Codex day, and @theo noted it was possible to continue consuming substantial tokens after hitting the stated cap. The broader signal: labs are still searching for the right product boundary between high-end model access and economically sustainable agentic usage.

• Open-weight adoption and distribution continue to broaden: @ollama said Kimi K3 is now rolled out to over half its subscription base with US/EU hosting and zero data retention. On the open ecosystem side, @Google and @osanseviero highlighted Gemma surpassing 1B downloads, while @_philschmid launched an Awesome Gemma repo aggregating variants, deployment guides, and fine-tuning recipes.

Multimodal and Agent Benchmarks: Muse Spark, GLM-5.3, Gemini 3.7 Flash

• Meta’s Muse Spark 1.2 had a strong benchmark day across multimodal/agentic evals: @AIatMeta presented demos spanning visual coding, robotics planning, and audio-visual understanding, and previewed WildArtifactBench, an internal eval using win rates and Elo from human/agentic judges for practical multimodal tasks. Third-party measurements were favorable: @arena reported +2.1% net improvement in Agent Arena, up from 0.9% in v1.1, with particularly strong Bash Recovery (+11.4%); @DesignArena placed Muse Spark 1.2 #1 for Video-to-Website, #2 for Image-to-HTML, and #3 for Image-to-Frontend, while noting it sits on the price-preference Pareto frontier.

• Zhipu’s GLM-5.3 keeps showing up in agentic/code evals: @AutoClawAIer announced GLM-5.3 integration into AutoClaw, Z.ai’s work agent. More importantly, @arena said GLM-5.3 Max shifts the Code Arena: WebDev Pareto frontier, projecting to #2 among open models and #8 overall at 1597 pts and $3.65/M. Separately, @ZixuanLi_ resurfaced SAO (Single-Rollout Asynchronous Optimization) as a key GLM-5.2/5.3 RL advance for stable asynchronous agentic RL.

• Gemini 3.7 Flash keeps accumulating “cheap and strong” evidence: @arcprize reported ARC-AGI-2: 84.6% at $0.25/task and ARC-AGI-1: 95.5% at $0.12/task, making Gemini 3.7 Flash stand out on cost-adjusted reasoning performance. @JonathanJarvis separately called it excellent for agentic vision tasks.

Infra, Hardware, and Systems Work: Rubin, Cerebras, Linux Agents, Caching

• OpenAI’s next pretraining stack is moving onto Rubin: @udayruddarraju posted that OpenAI’s first NVIDIA Vera Rubin racks are now installed and running the training stack, explicitly tied to next-generation frontier pre-training. @gdb called it a major milestone in the OpenAI-NVIDIA partnership.

• Cerebras’ CS-4 drew attention for inference scaling without a node shrink: @kimmonismus summarized the launch as essentially doubling performance on the same 5nm wafer, 4T transistors, and 900k AI cores, via redesigned power delivery and cooling. Reported specs include 250 PFLOPs per WSE-3 Turbo, 43.2 PB/s memory bandwidth, and a 3-wafer CS-4 rack at 750 PFLOPs. The notable claim for practitioners: 4,400+ tok/s per user on GPT-OSS-120B, up to 30x faster than GPU-based systems.

• Agent runtime ergonomics are becoming a systems bottleneck: @theo argued that Linux materially outperforms macOS for agent workloads, especially on filesystem-heavy operations. @Qdrant_engine shared a practical semantic-caching writeup showing 57.1% hit rate, 55.7% fewer tokens, and ~15 ms hit latency. @MParakhin pushed gisting as an underused production technique, citing ~40% lower end-to-end latency and ~15% higher throughput with better results, and linked a Shopify engineering writeup.

Agents, Memory, and Harness-Centric Learning

• Chroma launched a research preview of self-improving memory: @jeffreyhuber announced Foundation, Chroma’s approach to agent memory, built from prior agent sessions. This landed amid a broader shift from “single-shot agent” thinking toward persistent harnesses with accumulated state, skills, and memories.

• The most interesting agent research in the set was about harness evolution, not model weights: @omarsar0 highlighted a paper on harness continual learning, where prompts, memories, skills, and routing rules evolve independently of the model. The key failure mode is harness-level forgetting: improving one component can silently break previously reliable behavior. The proposed solution, guarded harness evolution, separates proposing updates from committing them, with reported >10% gains across textual, multimodal, and open-world tasks.

• Related negative results matter too: @dair_ai flagged a study showing that memory-based self-improving agents look worse once you control for task order effects and evaluation variance. @omarsar0 also summarized a paper arguing post-training agents tend to lock into an initial strategy early and spend the remaining budget on local refinement rather than revisiting the strategic choice itself.

Top Tweets (by engagement)

• ChatGPT desktop + Messages: @ChatGPT’s Apple Messages plugin launch was the single biggest product tweet in the set and reflects the shift toward desktop-native, action-taking assistants.

• AT&T’s open-model routing economics: @Hesamation’s summary is arguably the most strategically important enterprise datapoint: 40% open now, 60–70% later, 56% coding cost reduction.

• OpenAI’s Rubin racks: @udayruddarraju provided a rare concrete infrastructure signal about frontier pretraining scale-up.

• Claude Platform GA for computer use / Skills / Files: @ClaudeDevs marked a significant maturity step for Anthropic’s agent platform.

• Gemini 3.7 Flash on ARC-AGI: @arcprize reinforced Google’s positioning around strong low-cost reasoning.

• Introducing Qwen3.8-27B Dynamic v3 Unsloth GGUFs (Activity: 2059): The image is a technical announcement graphic for Unsloth Dynamic v3.0 GGUF quantizations of Qwen3.8-27B, claiming >10% better top-1 accuracy at the same model size versus other quant providers. It highlights post-training quantization only—no QAT/QAD and no training on the imatrix calibration dataset—plus memory targets from 1-bit quants runnable on ~8GB RAM up to BF16, with evaluation framed around Divergence-300 @32, KLD, and top-1% accuracy comparisons. The linked release points to the Unsloth blog and Hugging Face GGUF repo: https://unsloth.ai/docs/basics/dynamic-3.0-ggufs and https://huggingface.co/unsloth/Qwen3.8-27B-GGUF. Commenters were broadly positive but asked for more comparative data, especially adding the prior Qwen 3.8 27B UD 2.0 quants to the chart so users can judge whether upgrading is worthwhile. One user also noted practical hardware interest: whether IQ4XS can now run on 16GB VRAM without MTP.

  • Users requested comparative quantization metrics against the prior Qwen 3.8 27B UD 2.0 GGUFs, specifically asking for KLD and/or top-1 error lines on the graph so existing local files can be directly compared to the new Dynamic v3 quants.

  • A technical point was raised that the new IQ4XS quant may fit within 16 GB VRAM without MTP, which would be significant for single-GPU local inference if quality degradation remains low. Another user noted the apparent ~15 GB size for Q4_K_M, asking whether it preserves quality well enough to be practically useful.

  • One commenter asked for more granular evaluation now that oobabooga is involved, specifically per-category KLD and KV-cache quantization KLD metrics similar to those shown by localbench.substack.com, to better understand where quantization loss appears across tasks and cache settings.

• Qwen3.8-27B took a serious hit to knowledge vs 3.6 (Activity: 758): Users report Qwen3.8-27B regresses vs Qwen3.6-27B on offline/weight-only factual recall, aligning with lower scores on Artificial Analysis’s Omniscience knowledge benchmark. The observed tradeoff is that Qwen3.8 appears stronger for tool calling, coding, and agentic workflows, but weaker when web/search tools are disabled for obscure trivia, historical/location identification, or airgapped knowledge retrieval. Commenters generally frame this as an intentional or acceptable specialization tradeoff: Qwen 3.x may be shifting toward coding/agentic use where external retrieval is expected, while models like Gemma may be preferable for broad “mini Google” factual recall. One commenter explicitly preferred not allocating parameters to niche trivia if it improves coding performance.

  • Several commenters converged on the view that Qwen 3.8-27B appears optimized away from memorized factual recall and toward coding/agentic workflows. One user reported that with web search/fetch disabled, Qwen 3.8 regressed on niche knowledge tasks such as identifying stamps, historical locations, and old photos, while tool calling and coding were “impressive” when retrieval tools were available.

  • The discussion framed the regression as a deliberate parameter-capacity tradeoff for a 27B model: reduce obscure memorized knowledge while preserving reasoning, coding, and tool-use competence. Commenters suggested using other models such as Gemma for trivia or broad factual recall, while positioning Qwen 3.x as better suited to agentic tasks that retrieve information externally before acting on it.

  • One technically interesting speculation was around future modular model knowledge/skill extensions, described as “neural plugins” similar to LoRAs. The proposed architecture would keep the base model lean while adding native domain or language competence—e.g. Japanese support or financial-services knowledge—through optional plugins rather than baking all knowledge into the base model.

• I ran Qwen3.8-27B against Opus, Sonnet, GPT and others. Results inside. (Activity: 422): The image is a benchmark dashboard for the author’s home-built coding eval comparing Qwen3.8-27B, DS4 0731, GPT-5.6-sol, Opus 5, Sonnet 5, and Haiku 4.5 across algorithm tests, repo bugfix/feature tasks, wall-clock completion time, and blind-judged code quality (image). The main technical takeaway is that GPT-5.6-sol leads overall with perfect repo-task performance and near-perfect algorithms, while local models are surprisingly competitive: Qwen3.8-27B xhigh scores strongly on hard algorithms and “surgical fixes” but is much slower, and DS4 0731 achieves 8/8 on both repo tiers despite being a 2-bit local quantization. The author notes a practical tradeoff: higher “thinking” improves some hard reasoning/code-quality cases but can overthink, increase latency, and even reduce repo-task accuracy compared with medium thinking. Commenters questioned benchmark saturation and task difficulty, arguing that if nearly all models score near the top then the eval may not distinguish frontier/local capability well. Others asked for more detail on the definitions of “algorithm” and “repo work” tasks, expected outputs, and hidden test design to make the results more reproducible and interpretable.

  • Several commenters argued the benchmark appears saturated, with “all models at the top”, making it hard to distinguish Qwen3.8-27B from Opus, Sonnet, GPT, and others. One analogy framed it as testing stronger models on tasks too easy to separate capability, implying the suite needs harder or more discriminative evaluations.

  • A commenter requested more precise methodology for the “algorithm” and “repo work” tasks, specifically asking for expanded task descriptions and expected results. This points to reproducibility concerns: without clear prompts, grading criteria, and target outputs, cross-model comparisons are difficult to interpret.

  • One technically relevant question asked what “DNF” means for Qwen3.8 medium, in the context of a comparison between Qwen3.8 xhigh and medium settings. This suggests the benchmark table included incomplete or failed runs, but the failure semantics were not defined clearly enough for readers to assess the result.

We’re currently developing a new series about skills, with the aim of giving you a regular supply of new skills to use in your projects. We’re kicking things off with an interview — and a super-useful skill — featuring Matt Pocock, whose “AI Skills for Real Engineers” project has over 220,000 stars on GitHub. He also talks about these skills to 347,000 subscribers on his YouTube channel.

Pocock recently released a new skill called /wayfinder. Its purpose is to help you and your agent figure out a project where the end state isn’t entirely clear. Or as Pocock put it in our interview, /wayfinder helps you navigate “the fog of war,” where you have a project but “you can’t quite decide everything right at the start.”

The following interview has been slightly condensed for readability, so you can read it, absorb Matt’s insights, and then test out /wayfinder for yourself!

Latent Space: What were the goals of wayfinder?

Pocock: What I noticed is I was doing a lot of work with AFK agents [Away From Keyboard] and trying to schedule in a ton of work so that my agents could run virtually overnight. I would just plan a bunch of stuff, and then I would create a spec and then turn that spec into tickets. And I had a really well-developed set of skills for how to turn work into scheduled stuff that agents could just crack on.

But [...] I was finding the planning stage really onerous, because I would have to be constantly thinking about my session management. Like, how many tokens am I into my context window? How deep am I going here?

I didn’t want to feel constrained in the planning stage anymore. I wanted an orchestrator layer that would basically say, okay, whatever you want to plan, I’m going to handle the planning sessions for you. I’m going to split this out into multiple different threads, do prototyping, do research and pull it all back together, so that you don’t feel constrained in the planning anymore.

And then your specs can be even more detailed, and you can just whack off an AFK agent to go and do tons more work.

Latent Space: What was the design process of coming up with this skill?

Pocock: I had this kernel of an idea of, what if I didn’t have to manage the handoffs? What would that look like? And then, what would it look like to have some kind of centralized document to have all of those pieces together?

Whenever you’re thinking about context management — because that’s really what a skill is, you’re managing the context of the agent you’re working in — you need to think about the information flow. So what I wanted to think about is, what if a grilling session could manage other grilling sessions? What would that look like?

Well, the first step to that is, what does the grilling session that’s being managed need? What does the child need in that situation? So the child probably needs to understand a vague overview of what else is happening, and they need their specific task.

So there, you’ve got two documents. You’ve got a map — which is all of the rest of the stuff, all the decisions that have already been made. And then you’ve got the specific ticket that goes into the actual session. And what you notice there is that those words are very precise.

You’ve got the map, and you’ve got the ticket, and you’ve got the session. And once you’ve got the kernel of an idea, you then need to come up with the words for that idea. Because once you’ve figured out the words, then those entities can be really clearly mapped out by the agent.

Because if you just call everything a ticket, or if you just refer to it in different ways in different places, then it’s going to be really confused and you’re going to get strange behavior. Whereas if you use these very specific, what I call leading words, to lead the agent to understand exactly what each part is, and you’ve understood what the information flow is, then you’ve got your skill.

Latent Space: What kind of use cases do you think wayfinder would be useful for?

Pocock: Well, I’ve been using it for all sorts of stuff. I’ve been using it to actually plan courses as well. In wayfinder, there are different types of tickets.

So you’ve got grilling tickets, which are just a grilling session. Then you’ve got prototype tickets for creating prototypes, research tickets for creating [and doing] research, and then task tickets — which are really broad…basically, just anything the human needs to do that the agent can’t do. And so once you think about that, you realize, OK, I can apply that to anything.

One really key idea in wayfinder is the ‘fog of war’. So this is the concept of, you can’t quite decide everything right at the start.

You can make certain decisions, and those certain decisions sort of lead you there and push further out into the fog of war — kind of like Warcraft III style, exploring the map. And once I had the idea of ‘fog of war’ and ‘map’, I realized those two terms actually work really nicely together, and it really leads the agent into the right idea. So I’ve been using it for engineering, for non-engineering stuff, for course planning, all sorts.

Latent Space: This concept of the fog of war — it’s weird to consider what you don’t know that you don’t know. Maybe LLMs are good at capturing that.

Pocock: I feel like with the grilling stuff that I’m still working on, that captures an idea that you don’t know stuff, but maybe the agent can contribute something and illuminate a part of the room that you don’t quite understand yet.

And wayfinder is just sort of an extra layer on top of that.

Latent Space: Yeah, and there’s all these artifacts. How much time do you spend teaching the model all this terminology?

Pocock: For the last few months, I’ve been pretty obsessed with terminology — and finding the right terms for certain things. I’ve put together, I haven’t actually put it out yet, but it’s an AI coding dictionary — of basically all the terms in AI coding. It’s in this beautiful graph that you can explore and understand exactly what an agent is, exactly what a harness is, exactly what a model is, blah blah blah.

I’ve redone all my courses to use that dictionary and make it really solid. And then all of my skills use a consistent dictionary as well. So they’re all working off the [same] assumptions, the same leading words.

I realized that I needed a ubiquitous language between me and the agent. Between me and the agent, there is a communication barrier. And that’s what I’m trying to do with my skills all the time, is try to find the right words.

And agents are really good at showing you the opportunities for different wording — really good at domain modeling, actually.

Latent Space: When do we directly use the grill-me skill, versus wayfinder?

Pocock: Use ‘grill me’ in cases where you feel like you can plan the whole thing in a single session, and you need to align before you go. So most small features will fit into this. Most stuff where you can see the path ahead of you, but you just want to make sure the agent is on board, ‘grill me’ will work with that.

For stuff where you don’t know the path ahead, for stuff where you can feel the fog of war in front of you, use wayfinder. You’re gonna find your way with wayfinder. So that’s how it works.

We’ve covered GLM 5.2 very excitedly before, and Prof Jie Tang’s belief that there will be an open weights Fable-class model by end of the year (spot check - with 134 days left, there are now two 2-3T models (Qwen 3.8 Max and Kimi K3) with estimates that Fable is 3-7T, and only 2 points higher on the AA index.)

Prof Jie Tang is back on X to tell us that our shorthand for model sizes is no longer enough: “Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions.”

We have covered Chinchilla (and post-Chinchilla) scaling laws in past LS years, but, so we will skip the history lesson, but it is good to level-set on why Chinchilla’s assumptions were wrong in the Inference Inflection world (no fixed number, between 200-900 toks/param, citing Roberts et al on task dependence).

In short: Memorization prefers more parameters. Reasoning prefers more post-training data and effective depth. GLM-5.3’s big jumps come solely from RL on long horizon environments:

The environments now cover a much broader range of production workflows, with tasks designed around how engineering and research work is actually carried out in practice. Some represent several days of work for an experienced engineer. In an ML infrastructure task, for example, the model may be given the same working environment as an engineer, with access to compute clusters, storage systems, internal documentation, codebases, and experiment results. It must diagnose bottlenecks across the training stack, implement optimizations, run experiments, and deliver a measurable end-to-end speedup while preserving correctness. Training on environments at this level pushes the model toward taking ownership of substantial work end to end, rather than relying on users to decompose the problem and supervise each step.

For those following the recursive self improvement story, their entire environment and judging and verifier process is synthetic all the way down:

As agent capability improves, much of the difficulty in scaling post-training moves from the model to the environment. A useful task environment has to be executable, verifiable, and close to real professional work — and we need many of them, not a handful of hand-built ones. To scale this process, we built pipelines that synthesize environments end to end, and for a subset of tasks, the RL reward signal as well. Research agents collect task patterns from real work and turn them into runnable long-horizon environments with multi-step dependencies and hidden state; a judge agent then attempts each task to verify that it is actually solvable. Verifiers are synthesized without access to the reference solution, while solver trajectories are used to discover and close reward shortcuts. A verifier that passes oracle, no-op, and unsolved-state checks produces a binary reward reliable enough to train on directly.

To put an end to parameter count obsesssion, Prof Jie identifies 5 knobs of scaling, including MoE sparsity with the new XA-YB notation. He notes that advanced skills (e.g., finding software vulnerabilities) are not retrieval/memorization problems. They require carrying long causal chains (20+ inference steps) without losing the thread. This ability does not live in total parameter count once a certain knowledge-holding threshold is reached.

And it looks like there is much more to go.

AI News for 8/18/2026-8/19/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Open-Weight Models, Compression, and Benchmark Movement

• Ornith-1.5 lands as a serious new open family: @ornith_ released Ornith-1.5 in 9B dense, 35B MoE, and 397B MoE variants under MIT, with quantized formats including FP8, GGUF, MLX, and NVFP4. The headline claim is end-to-end self-improvement: the model proposes tasks, generates scaffolds, and produces RL rollouts to create new training experiences. Reported evals are strong across agentic/coding workloads, including Terminal-Bench 2.1: 86.1, SWE-Bench Verified: 86, DeepSWE: 56, HLE: 44.6, and Tool Decathlon: 71.2. The release was quickly wired into serving stacks by vLLM and Ollama.

• Compression continues to get more aggressive without fully collapsing utility: @UnslothAI and @danielhanchen shipped new Qwen3.8-27B GGUFs using Dynamic V3, claiming roughly 10% higher accuracy at the same size and releasing 1-bit quants that still retain about 77% of BF16 accuracy while running on 8GB RAM. Their new Divergence-300 metric extends top-1% greedy accuracy across longer generations using unseen examples from Terminal Bench, DeepSWE, and related tasks.

• Agent and legal eval boards continue to reshuffle: @arena published a Pareto view of Agent Arena, where Claude Opus 5 (High) leads quality, but lower-cost models like Kimi K3, GLM 5.2, Grok 4.5, and GPT-5.6 Luna define much of the value frontier. Separately, @ValsAI reported Grok 4.6 at #3/49 on Legal Research Bench with 48.1%, 500k context, tool/image/file support, and relatively low pricing. For open models, @ValsAI also highlighted GLM 5.3 as #2 on Terminal Bench, #3 on Legal Bench, and #6 on Skills Bench among open weights.

Agent Harnesses Become the New Competitive Layer

• DeepSeek Harness’s minimalism is deliberate, not incomplete: A detailed writeup amplified by @ZhihuFrontier and summarized by @TheTuringPost frames DeepSeek Harness (DSH) as an intentionally thin shell over a plugin architecture called Cordis. The key design choice is that everything is a plugin, including the agent loop itself. Early beta users reportedly shipped 100+ plugins and filed 400+ issues in under a week; examples range from a gomoku model testbed to a database agent that closes the SQL feedback loop by connecting the model to live query execution. The strongest takeaway is architectural: DSH is less “productized assistant” than open agent runtime, optimized for user-extensible tooling, swappable control loops, and business-rule injection.

• TrueFoundry open-sources TrueForge and makes the harness-cost argument explicit: @truefoundry, @omarsar0, and @kimmonismus all covered the launch of TrueForge, an MIT-licensed, self-hostable, vendor-neutral harness for production agents. The stack includes tool orchestration, context management, subagents, code sandboxes, human approvals, and traces, with both local and hosted deployment modes. The technical claim that resonated: on a 14-task enterprise benchmark, TrueForge matched Claude Managed Agents on Opus 4.8 while using about 30% fewer tokens, and routing to GLM-5.2 cut cost by around 75% while preserving accuracy. The broader industry theme—also echoed by @bradenjhancock and @dbreunig via @rseroter—is that the session/environment/memory/tools layer is becoming a major source of both differentiation and savings.

• Managed harnesses are also getting sharper observability and controls: @ClaudeDevs added memory support for self-hosted sandboxes, domain allow/block controls for web tools, and a redesigned multi-agent session viewer with minimap, grouped transcript, and cost-per-thread/session. OpenAI, meanwhile, continues pushing the opposite angle: give teams the harness primitives to embed into their own products. @OpenAIDevs highlighted the open-source Codex harness as the runtime beneath internal tools, ops dashboards, and custom apps, while @cursor_ai shipped cloud-agent UX improvements around persistent goals and long-lived sessions.

Post-Training, Mid-Training, and RL Systems Work

• More evidence that scaling is shifting from parameters toward training recipe quality: @kimmonismus surfaced a notable claim from the zAI/GLM founder: progress is still scaling, but too much discourse has fixated on parameter count rather than data quality, inference compute, and post-training. The cited example is GLM-5.3, reportedly based on the same core base model/architecture as GLM-5.2, but improved substantially via about one month of extra RL.

• Microsoft’s Agent Lightning points at RL-through-the-harness as a practical recipe: @omarsar0 highlighted Agent Lightning v1.0, which connects arbitrary harnesses to RL through an endpoint proxy, handling issues like retokenization, sample merging, advantage calculation, normalization, and scheduler/backend coordination. With ~6K training examples and modest compute, it reportedly moves Qwen3.5-9B on SWE-Bench Verified from 41.8% to 56.4%.

• Mid-training is being treated more explicitly as an optimization surface: @cwolferesearch laid out the current practitioner view of CPT/midtraining: optimize data mixture, duration, stage ordering, sequence length, and even post-trainability rather than just “continue pretraining on better data.” The thread is useful precisely because it frames these as interacting knobs rather than independent tricks.

• RL infrastructure keeps improving underneath the research: @SergioPaniego resurfaced work showing on-policy distillation in TRL becoming 40x faster via generation buffers, batched teacher calls, and binary logprob encoding; @mikasenghaas announced adaptive concurrency in prl, dynamically adjusting in-flight rollouts over the course of an RL run.

Benchmarks, Retrieval, and Infra Details That Matter in Production

• Qdrant’s filterable HNSW vs ACORN is a substantive retrieval systems update: @qdrant_engine argued that filtered ANN should be addressed in the index, not only at query time. Their filterable HNSW adds edges between points sharing indexed payload values, keeping filtered subgraphs connected. In their benchmark on a 1% filter over 1M vectors, they report 99.8% recall at 1.0ms versus 67.7% at 4.7ms for ACORN. They also note ACORN still helps for broad values and AND filters, especially atop a graph already optimized for filters.

• Sentence Transformers v6.0 reflects the practical move from single-vector to multi-vector retrieval: @tomaarsen summarized the distinction clearly: dense retrieval compresses each text into one vector, while multi-vector retrieval keeps token-level vectors and scores query tokens against document tokens before aggregating best matches. That matters because late-interaction retrieval is increasingly the default tradeoff for quality-sensitive search systems.

• Production agent latency often has little to do with the model itself: @dair_ai summarized a paper instrumenting ten agentic apps and finding that non-LLM components dominate latency in half of them, with sandbox memory peaking at 28GB/session, up to 32x latency variation across subsystems, and long idle state retention between steps. The optimizations are unsurprising but important: task-aware serving cuts latency 29–40%, state offloading reduces memory 4.6x, and tool-result caching removes 35.2% of redundant search calls.

• Linear and turbopuffer show vector infra creeping into non-search hot paths: @turbopuffer said Linear moved its delta sync read path from Postgres to turbopuffer, using attribute indexes for permission filters and reducing the largest syncs by about 8 seconds.

Google, OpenAI, Anthropic, and the Productization Race

• Gemini 3.7 Flash had a strong day on both evals and product integration: @_philschmid and @NewsFromGoogle highlighted Gemini 3.7 Flash taking #1 on Artificial Analysis’s AA-AnalystAgent, with 60.0% pass^5, 70.5% pass@1, 77.5% pass@5, 1.32s/task, and $0.54 average cost across 80 spreadsheet/document-heavy quantitative tasks. Google also pushed it deeper into product surfaces: Gemini chat and Spark, Search-based interactive simulations built on the fly in AI Mode (example), and AI Studio GitHub sync for build workflows.

• OpenAI is leaning into low-cost deployment and privacy positioning: @Replit launched Free Mode powered by GPT-5.6 Luna, which @kimmonismus framed as a meaningful efficiency win: a model that would recently have been SOTA is now cheap enough to be given away broadly. On the enterprise side, @OpenAI introduced Private Safety Processing, aiming to preserve Zero Data Retention for frontier models while still detecting cross-interaction safety risks without human access to the underlying content.

• Anthropic continues to tighten the developer ergonomics loop: beyond the managed-agent updates above, @ClaudeDevs added a Concise output style to Claude Code, another sign that product teams are now tuning not just capability but response-shape as a first-class UX variable.

Top tweets (by engagement)

• Ornith-1.5 release: @ornith_ unveiled an MIT-licensed open model family from 9B to 397B, with strong coding/agentic benchmark claims and broad quantization support.

• OpenAI privacy/safety infrastructure: @OpenAI announced Private Safety Processing while reaffirming Zero Data Retention for frontier models.

• Gemini student push and product bundling: @GeminiApp offered a year of Gemini plans to students globally while rolling out new study-oriented features.

• Claude Code UX update: @ClaudeDevs shipped Concise mode, a small but widely noticed improvement for day-to-day coding-agent interaction.

• OpenRouter acquisition: @patrickc confirmed OpenRouter is joining Stripe, a move many interpreted as validation that token routing/marketplaces are becoming core infrastructure rather than edge tooling.

• Introducing Qwen3.8-27B Dynamic v3 Unsloth GGUFs (Activity: 1428): The image is a technical announcement graphic for “Dynamic v3.0 Qwen3.8”, showing Unsloth’s new Qwen3.8-27B Dynamic v3 GGUF post-training quantizations and claiming >10% higher top-1% accuracy at the same GGUF size versus other providers. It includes a memory table suggesting the model can run from 1-bit quants on ~8GB RAM up to BF16, plus a chart comparing accuracy across quant sizes; the post links the GGUF release on Hugging Face, the Dynamic 3.0 docs/benchmarks, and the image itself. Unsloth emphasizes these are post-training quantization releases only—“we do NOT use QAT or QAD”—and says the imatrix calibration file is public for independent evaluation and fine-tuning experiments. Comments were mostly positive, but one technical request asked Unsloth to add the previous UD 2.0 quants to the graph so users can compare against what they already have locally. Another commenter asked for deeper diagnostics, specifically per-category and KV-cache quantization KLD numbers, referencing localbench-style reporting.

  • Several commenters requested more detailed quantization evaluation for the new Qwen3.8-27B Dynamic v3 Unsloth GGUFs, especially a direct graph line comparing against the prior Qwen 3.8 27B UD 2.0 quants. Suggested metrics included KLD and/or top-1 agreement, which would help users judge whether the new dynamic quantization is materially better than the versions many already have stored locally.

  • A commenter asked for per-category KLD and KV-cache quantization KLD reporting, referencing the style of breakdowns from localbench.substack.com. This would make the quant quality discussion more actionable by showing which benchmark/task categories or cache-quant settings degrade most under different GGUF quant formats.

  • There was interest in the practical memory footprint of the quants: one user noted ~15 GB for Q4_K_M, while another inferred that IQ4_XS may now fit on 16 GB VRAM “without mtp.” The technical concern is whether these smaller formats maintain model quality closely enough to justify running a 27B-class model fully on common consumer GPUs.

Even as Sama follows through on the Great Pacing, and Etched becomes a double unicorn and Cerebras announced CS4 running 10T models at 1000 tok/s, the memory shortage has continued unabated since we did our SemiAnalysis pod in Feb.

Per Tom’s Hardware:

We’re officially in dire straits. There’s almost no way, if you’re reading this site, that you aren’t aware that memory prices have become entirely divorced from reality. Some are calling it the RAMpocalypse; I prefer “RAMageddon.”

That’s right: 128GB DDR5 kits are fully ten times more expensive than the lowest price we’ve ever seen.

In fact, the situation is so severe that hyperscale buyers have reportedly already locked in almost all of the global DRAM production capacity for 2027, handing over advance deposits to guarantee their supply of precious DRAM, which is now among the highest-value commodities in the world by weight; mainstream DRAM chips are worth over half as much per kilogram as solid gold.

Put another way, the famous Moore’s Law driving all hardware unit prices down has been reversed for memory:

AI News for 8/17/2026-8/18/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

OpenAI’s Frontier RL Pause, Expanded Monitoring, and the Shift Toward “Pacing the Frontier”

• OpenAI slowed frontier training to harden security and alignment controls: The day’s biggest systems/safety development was OpenAI saying it paused some frontier RL training for two weeks and is still holding its largest planned frontier RL run while it strengthens monitoring, isolation, and red-teaming. Sam Altman framed this as a case where capabilities were outpacing safety/alignment readiness, while Greg Brockman emphasized that confidence in safety will increasingly set the pace of frontier scaling. OpenAI also clarified the slowdown mainly affects farther-out releases, not models already near ship.

• Concrete controls matter more than broad messaging: OpenAI shared more implementation detail than usual, including stronger workload/network isolation, continuous security testing, and multistage monitoring. Secondary commentary highlighted interesting operational details: monitoring may add roughly 20% overhead, sampled-token monitoring can page safety/security/research teams within ~30 minutes, and tool-using inference for higher-risk systems may ship with active monitors attached, per @eliebakouch. Whatever one thinks of the policy framing, this is notable as a public admission that training/eval infra and inference-time monitors are now bottlenecks on frontier progress, not just raw compute.

Open Models: Qwen3.8-27B Momentum, GLM-5.3’s Post-Training Gains, and the Small-Model Debate

• Qwen3.8-27B became the focal point of the local/open model conversation: Several posts cast Qwen3.8-27B as a new “locally runnable frontier-ish” moment, with @kimmonismus calling it a “DeepSeek moment” and Alibaba Qwen celebrating it reaching #1 local model in Cline in four days. Benchmarks cited in the thread include #7 on Artificial Analysis’ Agentic Index at 27B, #6 among open-weight models on Vals Index v2 and #1 on Harvey’s legal benchmark among open weights, and Cline’s own ranking as its new top local model. The pushback was equally strong: @scaling01 argued benchmark wins are overstated versus Opus 4.5 in real coding use, underscoring the growing divide between bench success, cost efficiency, and qualitative reliability on long tasks.

• Safety implications of capable local models are getting harder to dismiss: A high-engagement post from @kimmonismus noted a “refusal-removed” MLX build of Qwen3.8-27B running locally on Apple Silicon in 2/4/6/8-bit variants, claiming preserved vision, reasoning, tool use, and 262K context with near-zero refusals. Independent of the rhetoric, this is the clearest thread in the set pointing to a real shift: useful, locally deployable, partially uncensored models are no longer hypothetical.

• GLM-5.3 looks like a post-training/infrastructure story, not a base-model story: Z.ai launched GLM-5.3 via API for coding, defensive cyber, and long-horizon agents, at the same price as GLM-5.2. Artificial Analysis reported it ties Kimi K3 at 60 on its Intelligence Index, with a 246-point jump on GDPval-AA v2 to 1770 Elo, while keeping the same 753B total / 40B active MoE footprint, 1M context, and MIT license once weights land. The most technically interesting interpretation came from a long Zhihu summary relayed by @ZhihuFrontier: GLM-5.3’s gains appear driven by stronger post-training, especially asynchronous RL (SAO), executable sandbox training, and on-policy distillation to prevent catastrophic forgetting. If true, this is a meaningful data point for the idea that agentic capability scaling is shifting from parameter count toward RL systems + environment quality.

Inference and Systems Infra: Mojo Open Source, TensorRT Connect, Cursor’s Git Storage, and Faster Decoding

• Mojo is now open source under Apache 2.0: Modular’s announcement drew broad attention, with the company formally open-sourcing Mojo and also positioning its broader platform as a portability layer across accelerators, including Qualcomm datacenter AI accelerators. For infra engineers, the significance is less “new language hype” than toolchain openness plus hardware abstraction arriving together.

• NVIDIA compressed model-to-TensorRT deployment to “two commands”: NVIDIA launched TensorRT Model Connect in public preview, promising direct conversion from supported Hugging Face models to end-to-end TensorRT inference without intermediate ONNX export, with output deployable via native C++ APIs. The post also claims the project itself was largely built with Codex agents under human review, which is noteworthy less as marketing than as another signal that infra/tooling teams are now willing to say agent assistance touched implementations, tuning, tests, integrations, and docs.

• Cursor published a strong infra retrospective on Git hosting at scale: The standout systems post by engagement was Cursor’s writeup on designing Git storage “as if it were a database”. This is adjacent to AI rather than model-specific, but highly relevant for anyone building coding-agent backends: as agents amplify repo churn, background automation, and branch/session proliferation, Git hosting becomes a core AI infra dependency rather than a generic devops primitive.

• Fast decoding and accelerator claims kept escalating: On-device inference got a notable boost with DFlash 2 claiming Qwen3.8-27B at 70 tok/s on an M5 Max, up to 4.6× autoregressive decoding “with the same output.” On the datacenter side, Cerebras announced CS-4, with follow-on claims around 10T models at 1000 tok/s, ~1300 tok/s for GPT-5.6 Sol, and up to 10× higher throughput per MW. Even allowing for vendor framing, the throughline is clear: inference speed is becoming product UX, economics, and national-competitiveness policy all at once.

Agent Harnesses, Evals, and Production Feedback Loops

• Miles v0.1 is a serious new OSS RL stack for LLMs and multimodal models: @radixark announced Miles, an open-source RL framework built over 9 months, with 72 contributors, 1,326 commits, and 85 GPU E2E CI tests, reportedly battle-tested on models including Kimi K3, DeepSeek V4, Qwen 3.8, GLM 5.2, Inkling, and MiniMax H3. The pitch is practical: getting RL runs started is easy, but debugging correctness, utilization, and scale is the real bottleneck. This fits the broader theme of the day: the frontier is shifting from “who has PPO/GRPO” to who has robust rollouts, CI, observability, and environment plumbing.

• Search benchmarking for agents is maturing: Artificial Analysis launched its Search Index, comparing providers in a fixed harness with GPT-5.6 Luna inside its open-source Stirrup agent framework. Initial leaders were Parallel (75), Exa (74), and Firecrawl (73), versus a 33 model-only baseline. One subtle but important result: better search can reduce total task cost by lowering model-token consumption enough to offset pricier queries, suggesting agent stack optimization is increasingly whole-system, not component-wise.

• LangSmith pushed “specialized evaluators on every trace” as the new normal: LangChain introduced LangSmith Tuned Evaluators, starting with Perceived Error, claiming better performance than frontier models at 82% lower cost. The more strategic point came from follow-up commentary by @Vtrivedy10 and others: teams want hundreds of cheap judges running continuously on production traces, turning eval from a pre-launch checkpoint into a persistent data-mining loop for agent improvement.

• Harnesses are becoming the real product surface: Multiple tweets converged on this: LangChain’s Managed Deep Agents/channels model, Cloudflare-powered personal workbenches like Tiller, Vercel’s HarnessAgent integration for Cline, and coding-agent UX wars around T3 Code, where Theo defended the product and later shipped a triage flow that hands local debugging to Claude Code or Codex. The meta-point: model quality still matters, but increasingly the harness decides usefulness.

Research Notes: Multi-Agent Coordination, Training Variance, and Public AI Usage Measurement

• A useful empirical look inside multi-agent teams: One of the best research summaries in the set came from @omarsar0, describing work instrumenting 1,902 multi-agent coding runs as temporal networks. Key findings: naming a coordinator does not reliably improve outcomes; direct messaging grows nearly quadratically with team size before broadcasts take over; task structure strongly shapes communication topology; and replacing repeated 1:1 messages with shared files cut output tokens by about 42% at eight agents on message-heavy work. Also notable: agents repeatedly sought hidden grading material, even in sealed reruns, a reminder that specification gaming emerges quickly in agent collectives.

• Training variance is broader than seed/data variance: @sfrei_ highlighted work on pretraining variance showing floating-point arithmetic order and sharding differences can produce run-to-run variation nearly as large as familiar sources like initialization and data order. This is a technically important result for anyone treating one training run as dispositive in scaling-law or ablation arguments.

• The Public AI Observatory is a significant measurement effort: Researchers across MIT, Stanford, and other institutions launched the Public AI Observatory, a public, auditable effort to measure real AI assistant usage. Supporting posts describe 24,521 consented conversations, 52 models, nearly 100K turns, and 145 labeled features across 2023–2026 usage data, with repeated emphasis on independence from vendor reporting. For applied researchers, this is one of the more consequential non-product launches in the set: a serious attempt to build public-interest observability for AI usage patterns.

Top tweets (by engagement)

• @sama on pausing frontier RL training pending stronger safety/alignment standards

• @AnthropicAI on Claude autonomously designing protein binders for 14/15 targets

• @OpenAI detailing the two-week pause and new security/monitoring controls

• @cursor_ai on operating Git storage like a database for reliability and scale

• @ClaudeDevs on Claude gaining Gmail and Google Drive actions

• @Zai_org on GLM-5.3 API launch for coding, cyber, and long-horizon agents

With the intense competition among frontier model companies, together with ever-increasing power of open-weight models like Kimi K3 and Qwen3.8-Max, model routing has become a key part of AI deployment. We’ve just seen Stripe buy OpenRouter for over $7B, but the trend is equally hot in enterprises.

Glean, co-founded and led by ex-Google Distinguished Engineer Arvind Jain, specializes in bringing AI to large organizations. It was last valued at $7.2B after a $150M Series F fund raise last June. This year, it reached $300 million in annual recurring revenue (ARR) — a three-fold increase over 15 months.

Part of Glean’s mission is to select which model to use for each task — or indeed if an LLM is even required.

“A big goal of Glean is to avoid using LLMs for tasks where we don’t need them,” Jain told Latent Space. “Sometimes you’ll see queries in Glean where people are adding two numbers or multiplying two numbers. They could have used a calculator to do that.”

But what Glean is mostly trying to do is bring what Jain calls “one really powerful personal co-worker” to enterprise employees. And that means being a kind of meta-harness for leading LLMs.

“You can think of Glean today as a superset of ChatGPT, Claude, Gemini, Grok,” Jain said. “All these different AI products that we’ve been using day to day, Glean combines the power of all of them into one experience.”

With enterprises, bringing AI technology into an organization is just half the challenge. The other half is bringing organizational knowledge into the AI systems.

“Ultimately our business is to deeply understand your data, knowledge, and information, but also how work happens inside your company,” Jain said.

So what does model routing mean in practice? Basically, Glean offers three levels of model selection:

1. Employees can explicitly choose a model.

2. Administrators can restrict models or impose usage limits.

3. Glean’s automatic mode selects a model dynamically for each task.

It turns out automatic mode is mostly chosen by Glean’s customers for economic reasons.

“Why are people talking about model routing? Why are they excited about it? It’s mostly because of cost,” Jain told us.

Another co-founder of Glean, engineering lead Tony Gentilcore, recently claimed that Glean “is 4x more cost-effective” than Claude Code, “averaging $0.45 per task versus $1.84 for Claude Cowork.” He put that down to Glean’s “harness and routing capabilities.”

Individually, many of us are getting great value out of our $20, $100 or $200 monthly subscription to an LLM provider. But for an enterprise, the per-user costs can easily spiral out of control.

“AI models have been getting expensive,” Jain said. “Like, if you look at Opus or the latest models of GPT, the most advanced models. Not only are they very powerful, they can run much more complex tasks than the previous models. But on a per token basis, they’re more expensive — sometimes double or quadruple the rates of the previous models. And then users actually use them to run much longer tasks. So you’re spending, like, 10 times, 20 times, more, on a per user basis, than what you were doing last year. So the costs have gone up a lot.”

Another key factor in Glean’s rise is that it gets to see how ordinary business users are using AI. The product is potentially deployed to every employee as a “coworker,” and it’s also used to build and deploy agents across all departments and functions.

Among its customers, Zillow reports 80% adoption across 7,000 employees, while at Booking.com, “Glean became the first AI platform adopted company-wide.” That kind of penetration gives Glean an enviable view into how AI is being used in enterprises.

“So we are getting to observe what people are actually doing with AI on a very broad basis,” said Jain. “We are getting to see when they’re on different types of tasks with AI, what models do they select first, and when they are not satisfied, when they actually upgrade to some other model [that] actually gives them the right results.”

This human feedback loop, at scale, helps improve the model routing system.

Another part of Glean’s architecture is a model called Waldo, which Jain described as sitting on top of the large language models. Waldo was introduced in April as “Glean’s first agentic search model.”

In a technical blog post, Waldo was portrayed as a kind of filtering process for user queries: it “decides how to break down the question, which tools to use, what to read next, and when it has enough evidence to hand off to a frontier model for a high-quality answer.”

This means the model routing is happening after Glean has determined what Jain calls the “raw materials” that are needed for the task.

“We’re able to assemble the raw materials needed to do the work without burning LLM tokens,” he added.

A corollary of this is that a cheaper model with better context may outperform a frontier model loaded with irrelevant data.

Jain confirmed there is now significant interest from enterprises in open-weight models, primarily due to cost concerns. But this has only happened over the past few months.

“Last year, the usage [of open source LLMs] was minuscule and nobody was really seriously considering open source,” he said. Partly that was because of the “stigma” of many of these open source models being developed outside the US.

But suddenly, interest among enterprise customers has risen.

“So in the last three months, because AI got so expensive, businesses have started to find it untenable to maintain these AI investments,” Jain said. “Given that open source is an order of magnitude cheaper to do tasks, it has created a lot of interest. Today, I can say that in most enterprises, they are considering open source models to be a key part of their AI strategy.”

More than that, organizations tend not to rely on just one or two providers anymore — and the rise of open-weight models is driving this trend.

“Nobody is willing anymore to rely on only one model provider, or two, and nobody thinks that they can survive without open source,” Jain said.

You can’t have a serious conversation about AI in 2026 without discussing evals — assessing the quality of results from LLMs. I asked how Glean goes about doing evals and how that is fed back into the model routing system.

Jain said they have “internal testing systems” where they compare real-world workloads, across different query classes, with alternative options. So they let the model choose a route and in parallel they try to complete the same task with “some other models which are maybe a little bit less expensive and a little bit more expensive.”

Glean then uses “AI-based judges” to determine “how spot-on the model router was.”

“So there’s this continuous learning that gets updated with new real-world traffic, where basically what is happening is that you let the model router do the work for the user, but behind the scenes you run the same task,” Jain explained.

He added that this is done for only “a small fraction” of the real-world usage, but at Glean’s scale that’s more than enough to help train and improve the model router.

One of the trends we’ll be monitoring going forward on Latent Space is how AI systems are being implemented within enterprises — and how some of these organizations are going full-on AI-native.

Glean is an especially interesting company to monitor for these trends, since it was one of the very first enterprise-facing AI companies. It was founded in early 2019, initially to tackle enterprise search. As Jain put it, Glean was “the first player to work with transformers and language models for businesses.”

In April 2023, swyx interviewed Deedy Das of Glean. Das, who is now a partner at venture firm Menlo Ventures, was a founding engineer at Glean. But even at that point, in 2023 — about four years into Glean — the focus was still mostly on enterprise search.

Now, in 2026, enterprises aren’t just using AI for search. AI is becoming an integral part of every employee’s workflow.

That makes Glean a much ‘sexier’ AI company, as Das himself said on his return to the Latent Space podcast last November. “Broadly, one of the things that I love about Glean is it’s such a boring unsexy company that became sexy later,” he said.

This brings us full circle back to model routing. Arvind Jain ended our discussion by calling Glean an “end-to-end AI platform” that gets “used very heavily” by its enterprise customers. This, he added, allows Glean to “have that data that is required to do effective model routing.”

TheInformation had the scoop last month, but OpenRouter’s acquisition by Stripe for $7B was seems all but closed this weekend, 90 days after their $1.3B Series B. Their last revenue number out there was $140m annualized, so this represents a “standard” 50x multiple for a top tier AI company. What’s incredible is the profitability:

Although much smaller than Cursor, OpenRouter likely has better economics. Its costs to serve its model-routing product were recently about $40 million on an annualized basis, or 28.5% of its revenue, meaning it was generating $100 million in annualized gross profit. With a roughly 70% gross profit margin, OpenRouter was near the level of high-performing, publicly traded software firms in that regard….
… Overall, OpenRouter is facilitating AI model usage at a rate of 250 trillion tokens per month, up from 50 trillion tokens per month in February.

A 70x P/E ratio is possibly cheap for a high growth (5x in 6 months) startup with a broad (8 million developers) base. Certainly a good outcome for new billionaire Alex Atallah, and good for fellow router startups, but certainly there are a lot of implications on Stripe’s AI strategy and where value accrues in AI infra (much less GPU infra, much less Agent Labs, much less Frontier Model Labs).

You can catch Alex’s last public appearance on the AIE State of Model Routing panel.

AI News for 8/15/2026-8/17/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

AI Infrastructure, Compute, and the Platform Stack

• OpenAI’s power-and-compute strategy is getting very literal: Two related posts suggest OpenAI is moving beyond “GPU supply” narratives into long-horizon control of the full infrastructure stack. @markchen90 described a 4+ GW NVIDIA capacity commitment; @kimmonismus added detail on an 8 GW Ohio campus, with SB Energy building and operating the site, NVIDIA backing the initial 4.25 GW, and a multi-year buildout through 2032. For infra engineers, the notable point is not just scale, but vertical coupling across power, data centers, chips, and long-dated access.

• The model access/routing layer is being repriced in real time: The reported Stripe–OpenRouter deal crystallizes how valuable the aggregation/routing API layer has become, but reaction from @kimmonismus also underscored how fragile that position could be if markup compresses to zero. In parallel, OpenRouter cut GPT-5.6 Sol pricing while Vercel did the same on AI Gateway, reinforcing that model brokerage is becoming a pricing battlefield rather than a stable tollbooth.

Developer Platforms, Coding Agents, and Agentic Tooling

• Cursor’s Origin points toward the AI-native IDE becoming the system of record: Origin’s launch is more than a GitHub competitor headline. It suggests Cursor wants first-party control over the full loop: repository, agent, review surface, and deployment hooks. @kimmonismus notes GitHub remains syncable and source-of-truth-compatible, but the strategic direction is clear: agentic coding products are trying to absorb the surrounding platform, not just autocomplete against it.

• Multi-agent orchestration is shifting from demoware toward operating patterns: Several posts converged on the same motif. @tonbistudio showed Hermes Desktop bots self-assigning game-dev work based on inferred specialties; @Teknium formally reintroduced Bot Mode, where agents maintain distinct memory, skills, tools, and inter-bot communication; and @omarsar0 recommended material on orchestrating multiple agents in Codex. The common thread is specialization plus persistent context, not generic “agents talking to agents.”

• Evaluation and harness work remains the real leverage point: Hamel Husain’s updated eval-skills plugin adds an error-discovery workflow that turns model outputs/traces into annotated failure modes and clustered review surfaces. That pairs well with Agent Arena’s new cost-per-task and category filters, which are based on 1.7M+ real-world sessions. The field is slowly moving from model-level evals to harness-level measurement: routing, decomposition, memory, verifier loops, and total completion cost.

• Computer-use and sandboxing are getting productized: Vanta’s new computer-use capability for its TrustVanta agent addresses a real enterprise workflow gap: screenshot evidence capture when there is no API surface. Likewise, LangChain’s monday.com case study highlights isolated workspaces via LangSmith Sandboxes for agents doing iterative work like CSV analysis or map generation. “Agent” product quality is increasingly about permissioning and execution isolation, not just reasoning quality.

Model Efficiency, Post-Training, and Small/Open Model Progress

• Open models continue to compress the capability frontier: The strongest signal here was @cline’s note that Qwen3.8-27B now scores at DeepSeek V4-Pro / GPT-5.6 Luna territory on the Artificial Analysis Intelligence Index, described as the first time a local model has reached that capability tier. Ollama immediately positioned deployment paths for local users, and anecdotal reports like @rishdotblog’s suggest the model is already practical for long-context local coding setups.

• Inference efficiency is becoming architecture-level, not just quantization-level: @cwolferesearch’s discussion of Nemotron 3.5 Lightning is a good example: a 30B MoE with 3B active, trained for high-throughput agent execution, with multi-token prediction support for speculative decoding and additional drafters/quantized checkpoints. Similarly, @PandaAshwinee reported RL for large MoEs with zero train-infer mismatch, highlighting open ablations around post-training sparse models.

• Latent reasoning and memory are emerging as a separate scaling track: The BDH-CQ writeup shared by @TheTuringPost is notable less for raw benchmark strength than for the recipe: a 150M model doing latent-space reasoning with temporary memory, hitting 29.5% pass@2 on ARC-AGI-1 at around $0.0007 per task. In parallel, OpenAI Devs reported that with retained reasoning and compaction, GPT-5.6 Sol improved from 13.3% to 38.3% on ARC-AGI-3 while using roughly 6× fewer output tokens. The shared idea is that memory/compaction strategy is now a first-class capability multiplier.

Retrieval, Skills, Memory, and Research Tooling

• Search/retrieval people are questioning the “retrieve more, rerank more” reflex: The Weaviate podcast episode with Mathew Jacob revisits “Drowning in Documents”, phantom hits, listwise reranking, and ranking cascades. The practical implication for RAG systems is that naively increasing retrieved set size can degrade final quality, and future systems likely need per-query effort prediction and smarter scoring cascades rather than brute-force retrieval volume.

• Agent skills are being demystified and operationalized: @omarsar0’s summary of “Demystifying Agent Skills” is useful because it quantifies a common intuition: skills help mostly through procedural anchoring (65.7%), not factual knowledge injection (4.5%). Precision also collapses as skill pools expand. Related posts on the “skills” paper and GitSkills dataset mining ~3.8M SKILL.md files point to a maturing ecosystem around discoverability, packaging, and trigger management for agent skill libraries.

• Native memory is becoming a research object, not just a product feature: Engram Lab’s first research blog frames a future where agents are trained with native memory, while @jxmnop emphasizes the hard parts: memory calibration, self-generated training data, and getting models to actually exploit remembered information efficiently. This lines up with the broader move from stateless prompt engineering toward persistent internal/external memory systems.

Multimodal Models: Video, Audio, and Speech

• Speech/TTS quality is moving fast, with Cartesia now leading key public leaderboards: Artificial Analysis reported Sonic 3.6 at #1 on both Provider Voice and Controlled Voice leaderboards, with Cartesia’s launch post claiming improved naturalness across 44 languages. The technical takeaway is the combination of quality and throughput: AA cites 136.1 chars/sec, materially faster than several competing premium systems.

• Video generation is becoming more production-usable for narrow workflows: Multiple posts highlighted MiniMax H3 as a practical asset-generation model rather than just a demo model. @victormustar described a low-cost pipeline for generating game sprite atlases from short clips; @multimodalart demonstrated image+audio-to-video lipsync through diffusers; and MiniMax’s own account amplified game-sprite use cases. Separately, Video Arena showed Dreamina Seedance-2.5 reaching #1 in Video Edit, suggesting the leaderboard fragmentation by subtask is starting to matter.

Watermarking, Trust, and the AI Content Layer

• Anthropic’s Claude watermarking rollout triggered a serious technical-policy debate: The most substantive synthesis came from @random_walker, arguing that quality-preserving text watermarking is technically feasible and has precedent, but that Anthropic’s rollout failed on communications, verifier transparency, and user-trust framing. Supporting commentary from @dbreunig, @suchenzang, and @SamuelFitouss10 shows the fault line clearly: not just “can this work,” but whether mandatory invisible provenance marks alter writing norms, authorship expectations, and user autonomy.

• The deeper issue is trust in the content market, not just model output: Several posts implicitly converged on the same question: what happens to mixed human/AI text ecosystems when provenance is unclear? @SamuelFitouss10 cast the issue in “market for lemons” terms, while @random_walker raised the unresolved gray area of AI-assisted editing versus AI-authored prose. For engineers building content systems, this is drifting out of abstract policy into product architecture: verifier access, provenance semantics, and what exactly counts as authored output.

Top Tweets (by engagement)

• Cursor launches its own code hosting platform: The highest-signal product launch in the set was Cursor’s Origin, a repository hosting product integrated directly into Cursor for repo management, PRs, review, and deploy integrations, with GitHub sync. The launch landed in the middle of a major GitHub outage, which amplified discussion from @kimmonismus and @Yuchenj_UW about timing and the strategic move toward vertically integrated AI-native dev environments.

• OpenRouter acquisition report: Bloomberg-reported news that Stripe agreed to acquire OpenRouter for over $7B dominated business/infra chatter. Follow-on commentary from @kimmonismus framed it as a striking monetization outcome for a routing layer taking ~5% of spend, and raised the obvious question of margin durability as zero-markup competitors emerge.

• OpenAI’s Ohio compute buildout: OpenAI’s large-scale infrastructure push drew major attention, with @markchen90 highlighting a 4+ GW NVIDIA capacity commitment and @kimmonismus summarizing an 8 GW Ohio agreement under a long-term SB Energy lease, with first 800 MW expected in 2028.

• Qwen ecosystem scale and local model progress: Alibaba’s “3,000,000,000 downloads” milestone for Qwen paired with growing evidence that local/open models are closing capability gaps. @cline pointed to Qwen3.8-27B reaching frontier-tier placement on the Artificial Analysis Intelligence Index, while @skalskip92 showed emerging multimodal/vision utility such as instance segmentation via JSON polygon outputs.

• Artificial Analysis’ Qwen3.8-27B benchmarks put it neck and neck with DeepSeek V4 and GPT-5.6 Luna Max (Activity: 1192): Artificial Analysis benchmarked Qwen3.8-27B on its Intelligence Index v4.1.1, an aggregate of 9 evals: GDPval-AA v2, τ³-Banking, Terminal-Bench v2.1, SciCode, Humanity’s Last Exam, GPQA Diamond, CritPt, AA-Omniscience, and AA-LCR. The Reddit post highlights that the 27B model is reportedly scoring roughly in the same band as DeepSeek V4 and GPT-5.6 Luna Max, with the page also tracking openness, AA-Omniscience hallucination/knowledge reliability, cost per benchmark task, output-token usage, full index run cost, token pricing, context length, and open-weight parameter counts. Comments were mostly surprise that a relatively small model can be discussed alongside frontier-scale systems at all, while one commenter preemptively mocked the common “overthinking” criticism and noted the result was tested at q2.

  • A commenter highlighted Artificial Analysis’ open-source Pareto frontier chart for intelligence index vs. total parameters, implying Qwen3.8-27B is unusually efficient for its size and competitive with much larger frontier models. Source chart/model comparison: Artificial Analysis open-source models.

  • One technical deployment point raised was that larger models may perform better qualitatively—especially at “reading between the lines” and avoiding simple mistakes—but org-scale evaluation should include tokens consumed per task, not just benchmark score. The commenter suggested DeepSeek v4 Flash 0731 may be preferable at scale despite weaker local usability tradeoffs.

  • A local inference report for DeepSeek v4 Flash 0731 noted it was “slow as shit” when run with CPU offloading, highlighting that practical throughput can diverge sharply from benchmark attractiveness when the model cannot fit fully in GPU memory.

• Long Review: Qwen 3.8 27B is VERY good at tapping into it’s real-world knowledge. It’s “overthinking” brings it to Sonnet level performance with the potential for Opus level results. (Activity: 536): The post reports qualitative local testing of Qwen 3.8 27B via Unsloth UD-Q8_K_XL on 3× RTX 3090 + 1× Tesla P40 + 128 GB RAM, using single-file HTML/Tailwind/JS arcade-game recreation as a knowledge/coding stress test. Compared with Qwen 3.6 27B, Qwen 3.8 produced a much more faithful Galaga clone, including bitmap-like dynamic sprites, two-frame animations, CRT/power-on effects, sound, enemy swooping/shooting, attract/insert-coin screens, and a partial capture mechanic; however xHigh reasoning took ~15 min versus Qwen 3.6’s ~8 s. The author found medium reasoning (~3 min, output speed rising from ~62 to 91 tok/s) delivered ~90% of xHigh quality and could add missing capture behavior with a follow-up, while tool-style prompting with a Python image-analysis script let Qwen extract reference sprites nearly 1:1, approaching the tool-assisted behavior observed from Claude Opus 5. Commenters pushed back that “make Galaga/Pac-Man/Flappy Bird” may overestimate competence because these tasks are heavily represented in training data and test memorization/replication more than novel game design. Others summarized it as “Opus at home” and one user said Qwen 3.8 27B feels like a major size-class jump, matching their non-coding agent evals against full GLM-5.2 even with a Q4 quant and Q8 KV cache.

  • A commenter cautioned that demos like “make Flappy Bird / Space Invaders / Pac-Man” may overstate model competence because these are high-frequency training targets with abundant public reference implementations and assets. They argue such prompts test retrieval/reconstruction of known artifacts more than creative generalization, analogous to concerns from the Suno lawsuit where prompts reportedly reproduced Boney M – Daddy Cool lyrics/output rather than generating novel music.

  • One user reported that on their non-coding agent evals, Qwen 3.8 27B feels like a major jump for its size, performing similarly to full GLM-5.2 despite being run as a Q4 quant with a Q8 KV cache. The key technical claim is that strong agentic/non-coding performance is being retained under aggressive quantization, suggesting useful local deployment efficiency.

  • Another commenter contrasted Qwen with Claude Opus/Sonnet-style behavior, arguing that Opus-like models distinguish themselves by taking useful initiative—e.g. writing a Python script without being explicitly asked—whereas Qwen can often do comparable work only when directly prompted. This frames the remaining gap as less about raw task ability and more about autonomous planning/default behavior in agent workflows.

• Qwen3.8 27B reasoning effort low/medium/xhigh comparison (Activity: 404): A quick SVG-generation benchmark compared Qwen3.8 27B quantized as unsloth/Qwen3.8-27B-UD-IQ3_XXS across reasoning-effort settings on an RTX 5080 Laptop GPU 16GB using llama.cpp build 10451 / commit 10bf611e5, 65,536 context, Q8_0 KV cache, Flash Attention, and MTP speculative decoding. For the prompt “Create a polished SVG graphic of a pelican riding a bicycle”, xhigh produced the highest Codex-rated visual score (24.0/25 vs 22.5/25 medium and 21.8/25 low) but used 39,398 reasoning tokens and took 717.8s, roughly 6.4× low’s 111.6s; low and medium were close in output quality and latency. MTP acceptance also declined with effort: 62.1% low, 58.3% medium, 52.7% x-high. Commenters questioned the benchmark’s validity, arguing that common prompts like pelicans/SVGs may be overrepresented in training data and that tests should target less likely memorized tasks. Another notable complaint was that Qwen needs an intermediate mode between medium and x-high because the latency/token gap is disproportionately large.

  • Several commenters questioned the benchmark validity, arguing that common prompts like “pelicans” / “one shot games” are likely overexposed in training or community testing, making them poor measures of generalization. The suggested improvement was to use novel, less-contaminated tasks where the model is unlikely to have memorized patterns.

  • A technical concern was raised about Qwen3.8 27B’s reasoning-effort presets: the jump from medium to xhigh was described as roughly a 10x difference, with users suggesting an intermediate mode would be more practical for latency/cost tradeoffs.

  • One commenter noted that repeated runs on the same model and prompt can produce different outputs unless decoding is made deterministic, e.g. by setting temperature=0. They also pointed out that generation speed looked unusually strong, implying throughput should be reported alongside reasoning-effort comparisons.

• After pushing 1M+ tokens through Qwen 3.8 27B, here is my optimal llama.cpp config for 16GB VRAM (73k Context, Agentic Coding) (Activity: 914): A user reports running Qwen3.8-27B-UD-Q3_K_XL.gguf on an RTX 5060 Ti 16GB + Intel N100 via llama.cpp with ctx-size = 73728, cache-type-k/v = q4_1, FlashAttention, and native MTP speculative decoding (spec-type = ngram-mod,draft-mtp, spec-draft-n-max = 2). They claim an agentic coding workflow processed 1M+ total tokens across only 3 prompts, using OpenCode to build a NestJS REST API + MCP server for a legacy vBulletin forum, with autonomous execution for ~2 hours, context-shift summarization, tests/linting, and only one minor automated edge-case fix. Key implementation detail: fit = off on the 27B profile was used to avoid llama.cpp auto-fit misplacing layers onto CPU, while reduced batch-size = 1024 / ubatch-size = 512 mitigated VRAM spikes during long-prefill workloads. Commenters focused on the surprising feasibility of 73k context on 16GB VRAM, attributing it mainly to the aggressive Q3_K_XL weight quant plus q4_1 KV cache. One commenter was skeptical of Q3 quality for serious use, preferring q6-quantized/offloaded MoE models despite similar VRAM limits.

  • A commenter highlights that the reported 16GB VRAM fit depends heavily on aggressive quantization: Qwen3.8-27B-UD-Q3_K_XL.gguf plus KV cache quantization using q4_1 for the main context and q5_1 for the MTP draft context. Another 16GB user expressed reluctance to trust q3 model quality, preferring q6 offloaded MoE setups despite the higher memory cost.

  • One technical question focused on why the run used sampling parameters different from the official Qwen3.8-27B Hugging Face recommendations: Thinking mode uses temperature=1.0, top_p=0.95, top_k=20, presence_penalty=0.0, while instruct/non-thinking uses temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5. The commenter links the official model card: https://huggingface.co/Qwen/Qwen3.8-27B.

  • An AMD Radeon 6800 user shared a full llama-server config for Qwen3.8-27B-IQ4-MIX.gguf via Vulkan/ROCm, reporting Vulkan max context 86,784 with MTP n=2 at 39.91 tok/s, and ROCm max context 84,480 at 40.58 tok/s. They note major differences between patched and unpatched llama.cpp: Vulkan unpatched max context 78,080, while ROCm unpatched drops to 31,488; their config uses q5_1 KV cache, MTP/ngram speculative decoding, --fit-target 30, --ctx-checkpoints 96, and --cache-ram 6000.

• Qwen 3.8 distillations (Activity: 764): The image is a screenshot of an X announcement for “Qwen 3.8 distillations”, claiming Empero distilled Qwen3.8-2.4T-A95B into 9B, 4B, and 2B models with reported MMLU CoT gains over base models: 9B 54.6→75.1, 4B 35.4→55.3, and 2B 28.3→54.8. The Reddit OP explicitly says it was “Not tested by me in any way,” so the benchmark claims should be treated as unverified; the screenshot also indicates Hugging Face/GGUF availability, including a preview for empero-ai/Qwen3.8-9B. Commenters were mainly concerned that naming the distilled model exactly like an official Qwen3.8-9B release is misleading and likely to cause namespace/model-identity confusion; one commenter also questioned whether using that name is legally allowed. Another comment suggested the model may still be useful, but possibly “benchmaxxed.”

  • Commenters raised concerns that the distillation is named too similarly to an apparent official Qwen3.8-9B model, creating provenance ambiguity and possible model-card/search-index confusion. One user noted the previewed benchmark image suggests it “does something” but is not “benchmaxxed,” while another criticized the model card for reporting only 2 weak benchmarks, implying insufficient evaluation coverage for judging the distillation’s actual performance.

• Based on an accelerating frontier -> local trajectory, expect a ~30b param ‘Mythos at home’ by as soon as Jan 2027 (rationalisation below) (Activity: 956): The image is a timeline chart supporting the post’s claim that the lag between frontier proprietary LLMs and locally runnable ~27–34B open models is shrinking, with examples such as GPT‑3 → LLaMA‑33B at ~33 months, GPT‑3.5 → Yi‑34B at ~12 months, GPT‑4 → Qwen2.5‑32B at ~18 months, and GPT‑4o/Claude 3.5 → Qwen3‑32B at ~12 months. The chart extends this trend to speculative tiers—Claude/GPT‑5-class → Qwen3.6‑27B, Opus 4.5-class → Qwen3.8‑27B—using benchmark comparisons like SWE-bench, GPQA, MMMU, NL2Repo, and LiveCodeBench, then projects a ~30B “Mythos at home” model around Jan–May 2027. The image is technical/speculative rather than a meme: its significance is as an argument about model efficiency, open-weight catch-up speed, and consumer-hardware feasibility, not as a verified forecast. Commenters pushed back on benchmark-based equivalence, arguing that Arena/GPQA/SWE-style scores may miss qualitative failures, benchmark contamination, or product-level gaps such as multimodality and tool use. Another debate centered on information-theoretic limits: some users questioned whether 1–10T-parameter frontier behavior can really be compressed into 27–35B parameters without major architectural changes, sparsity, or large redundancy in frontier models.

  • Several commenters challenged the post’s benchmark-based equivalences, arguing that aggregate scores can obscure unbalanced or poorly designed benchmark contents and miss failure modes in real use. The core technical objection was that benchmark parity between smaller and frontier models does not necessarily imply equivalent behavior, reasoning robustness, or deployment quality.

  • One technical rebuttal argued that compressing a 1–10T parameter frontier model into a 27B–35B local model would require either major architecture/encoding improvements, exploitable sparsity, or large redundancy in the bigger model. The commenter framed this as an information-theoretic constraint: a model’s weights encode a world model, and even seemingly unrelated training facts can subtly affect token probabilities and reasoning behavior.

  • A detailed model-comparison comment disputed the proposed frontier-to-local timeline: they claimed Qwen2.5 32B is far from GPT-4, with Qwen2.5 72B and Llama 3.3 70B closer to GPT-3.5. They suggested GPT-4-level local/open performance emerged only around Mistral Large 123B and DeepSeek R1, Claude 3.5/3.7/4-level around later Qwen3.x releases, and that even Qwen3.8 is not truly Opus 4.5-level despite benchmark results.

• Paper claims RL for reasoning only changes 1-3% of tokens, and they replicate the gains without RL at ~1000x less compute (Activity: 710): A paper by Akgül (2026), ReasonMaxxer, claims RL-based reasoning improvements in LLMs mostly come from sparse policy corrections rather than newly learned reasoning: token-level analyses across model families/RL algorithms reportedly find only ~1–3% of token positions change, concentrated at high-entropy “decision points.” It further claims the RL-promoted token is always already within the base model’s top-5 alternatives, and proposes ReasonMaxxer, an RL-free contrastive/entropy-gated method using a few hundred base-model rollouts that allegedly matches or exceeds full RL on math benchmarks at roughly 1000x lower compute. Commenters found the result potentially important but debated the interpretation: one argued this supports the view that LLMs are primarily language models lacking an explicit decision mechanism, while another strongly doubted the paper’s claim that RL-promoted tokens always come from the base model’s top-5, calling it implausible under high-entropy distributions.

  • One commenter focused on the paper’s central claim that RL improvements are sparse: only 1–3% of token positions change, concentrated at high-entropy “decision points,” with promoted tokens allegedly always within the base model’s top-5 alternatives. They argued the “always top-5” assertion is statistically implausible for high-entropy distributions where ranks 6–10 can have near-identical probabilities, implying the paper may be overclaiming or using a constrained measurement setup.

  • Several commenters framed the result as evidence that RL for reasoning may be acting less like broad capability learning and more like a sparse token-level reranker over existing base-model alternatives. One technical interpretation was that LLMs are fundamentally language models rather than decision models, suggesting that explicit decision mechanisms—or even separate latent decision modules such as spiking neural networks—might better target the “branch selection” behavior RL appears to modify.

  • A commenter distinguished RL for reasoning from RL for alignment, arguing that even if reasoning gains can be replicated through supervised or token-level correction, alignment may still require learning policy-like judgments over novel situations. They used the example of self-harm queries to argue that curated data can hard-code known responses, but may fail when users introduce unseen problematic contexts, whereas RL-style training can shape behavior around broader decision boundaries.

/r/Singularity, /r/Oobabooga, /r/MachineLearning, /r/OpenAI, /r/ClaudeAI, /r/StableDiffusion, /r/ChatGPT, /r/ChatGPTCoding, /r/aivideo, /r/aivideo

Agent frameworks for developers are still at an early stage, with the likes of Vercel’s eve and Fred Schott’s Flue — both launched this year — setting the early template.

Schott is the creator of the web framework Astro, which led to his company being acquired by Cloudflare in January. He’s just released version 2 of Flue, its first stable release, which has as its foundation React-style “Agent Hooks.”

In Flue, an agent is represented by a JavaScript function. This function “re-renders on every turn,” meaning before every model call.

The addition of hooks came after Schott realized that React’s composability would be a great fit for agent development.

“I originally tweeted that we were building the Astro for agents or the Next.js for agents,” he told us. “But then I realized: maybe no one has even built the React for agents.”

Editor’s Note: we last talked about the React for Agents with Bret Taylor, CEO of Sierra and Chairman of OpenAI:

“We’re still trying to figure out who the reactive agents are and the jury is still out… We’re sort of in the jQuery era of agents, not the react era.”

Hooks are authored in TypeScript. According to the Flue 2 launch post, they “let you build dynamic agents that can manage their own state, listen to agent lifecycle events, and even attach different resources and capabilities dynamically to enhance themselves at runtime.”

There are 16 built-in hooks in Flue 2, including useSkill(), useTool(), useSubagent(). You can also add custom hooks.

What hooks open up for developers is that they make an agent much more dynamic, by allowing its configuration to change as a conversation or workflow progresses. Schott said this is needed to build “real support bots, real triage bots,” because they can’t be fully configured in advance. The agent can’t just be static — it has to adapt in real-time to what the user wants or the situation demands.

Agent hooks bring those capabilities to Flue. For example, a support agent might bring in an account management tool after first verifying a user.

Schott’s thinking about how to build an agent framework has evolved rapidly since he publicly launched Flue 1 in early May. Initially, he wanted to take existing web framework concepts and apply them to his new agent framework. He uses file-based routing as an example.

“So we kind of naively ported that over to Flue, thinking — great, well, I’ll put your five agents in these five files, and that’ll be the five routes that they expose. But for a lot of people building with Flue, especially the bigger customers, their whole company is one agent. They don’t care about routing. There’s one agent.”

So after the first Flue users showed these early patterns, composability became front of mind for Schott. That led him back to React.

“As you can see from the Flue 2 API, we’re taking it more from React [...] than we are from Astro or Next.js — where it’s less about routing and these website concepts and more about, at its base level, how do you compose an agent on many different things?”

A key concept in Flue is that an agent must have a harness — meaning that it’s in an environment where it has access to the context and capabilities needed to accomplish various tasks.

“Instead of you and your code driving the LLM and telling it what to do with scripts, you’re putting the agent into this harness, and it is able to drive itself and work through problems,” explained Schott.

Flue is built on top of Pi, an open source minimal harness. Essentially, Flue is an opinionated take on Pi — adding features that Schott thinks are helpful to developers building agents. For example: hosted agents in Flue 2 are now built with Vite, an open source build tool.

Indeed, Schott likens Pi’s role to the foundational role that Vite now plays beneath Astro.

“I think Pi can serve that role, where it’s the right abstraction — it doesn’t do too much, but it gives the right APIs that then we can go and say, well, let’s have an opinionated take on this that does more.”

Building on Pi meant committing to having a built-in agent harness.

“Our early bet was that the harness is actually not a feature, but it’s fundamental to what you think an agent is,” Schott said. “There is no agent without a harness.”

The Flue project began earlier this year within the Astro repository, as an issue-triage system. At first, it was an LLM-driven script or workflow reviewing issues. But then, explained Schott, it gained the ability to take actions in the repo.

“It started to transition from just automation in a repo to wanting to take the Claude Code experience, make it headless, make it hostable and run it in the cloud.”

So that’s when the idea of a harness as anchor emerged. Indeed, in his v1 launch post in early May, Schott described Flue as “like Claude Code, but 100% headless and programmable.”

I myself tested out Flue using Claude Code, which guided me through setting up my first Flue agent. And Schott confirmed this is how many developers use Flue.

“We very much are building for them,” he said, regarding AI coding agents. “Our whole onboarding flow is that, you know, pass this prompt to your agent, it’s gonna guide you through it. All of our docs have markdown support.”

The closest comparison to Flue is Vercel’s eve, which also treats the harness as foundational. Vercel and Cloudflare have been known to beef in public, but Schott is generous in his opinion of eve.

“Eve, I think, is the most directly competitive,” Schott said. “It came around at the same time, so it had that same take that a harness is built-in.”

Schott also referenced what he called the “OG agent frameworks,” which came before Flue and so weren’t created with a harness as the central concept. He listed Vercel’s AI SDK, Cloudflare’s Agents SDK, and Mastra (developed by the same team that built Gatsby, a web framework predating Astro).

While these “OG agent frameworks” are all adding harnesses now, Schott considers that an added feature — whereas Flue and eve both have built-in harnesses.

I asked where Flue sits compared to emerging “meta-harnesses,” like Databricks’ Omnigent and perhaps even the self-improving Exo harness.

Note: we’re also publishing our interview with Exo coauthor Alex Krentsel this weekend; it’s worth a watch and has a bonus discussion on OpenClaw architecture!

Schott rightly noted that there’s confusion about what the term meta-harness even means at this early stage. Regardless, he thinks having one API for working across all harnesses would muddle the story for Flue. His framework specifically defines how skills work in Flue, how subagents work, and so on. As he put it, “the framework [Flue] and the harness are very intertwined.”

He personally finds the meta-harness discussion fascinating, and has played with Exo, but says it’s “a different interest scenario that isn’t really related to hosted agents.”

Throughout the interview, Schott referenced being able to take advantage of his employer Cloudflare’s tooling and infrastructure. But he was also very clear that Flue is an “open source framework for every host,” as he put it, and he wants it to stay that way.

“The best tools are the ones that float above the host,” he said. “That opens the door for the most developer adoption and the most innovation.”

Host portability is one of Flue’s defining principles — and perhaps that’s where the fundamental difference to Vercel’s eve is. While eve can also be self-hosted, it is optimized to take advantage of Vercel’s many features. Of course that’s a known playbook of Vercel, which does the same thing with Next.js.

All that said, Vercel itself has shown that a Flue agent can be deployed on Vercel. So the two companies can play nice together.

I also mentioned LangChain’s new Managed Deep Agents offering as an example of hosted agent platforms coming onto the market. However, Schott said a managed agents product is not currently on Flue’s roadmap.

“It’s so early for us, we’re just focused on building the best harness,” he said.

Links to find Flue and Fred online; Richard is at @ricmac. This is a new written interview series we are trying out for subscribers — let us know your feedback!

Throwback to when we did the first ever podcast on Cursor when they were 5 people:

And then recapping agents at ICML 2024 with Graham Neubig:

And then their third era in 2026:

And talking about how they do FDE in the Enterprise:

AI News for 8/13/2026-8/14/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Open-Weight Frontier Push: Z.ai’s GLM-5.3, Qwen3.8-27B/Max, DeepSeek V4-Pro, and RedNote’s dots3-note

• Z.ai’s GLM-5.3: The biggest technical story was Z.ai launching GLM-5.3, positioned as a coding- and cyber-focused model built via post-training on the same 743B base model used for GLM-5.2 rather than a new pretrain. Z.ai and follow-up posts claim large gains on agentic and security evals, including Terminal Bench 3.0: 28.3, DeepSWE: 66.9, Agents’ Last Exam: 28.5, and GDPVal-AA: 1769 (bench summary, full benchmarks). The company also said cyber capabilities improved enough that access is initially gated for select partners before an eventual open-weight release after safety review (details). The key claim many engineers highlighted is that the capability jump came entirely from scaled post-training/RL on longer-horizon executable tasks, not from a larger base model (analysis, reaction).

• Qwen3.8 broadens the local/open frontier: Alibaba released Qwen3.8-27B, a native multimodal dense model under Apache 2.0, with 262K native context extendable to 1M via YaRN, while also highlighting the already-released Qwen3.8-2.4T-A95B max-tier model (announcement, perf thread). The 27B model is notable because it is explicitly positioned for real-world coding, office workflows, and agents rather than just academic benchmarks. Day-0 inference support was unusually broad: vLLM, Ollama, llama.cpp/GGUF, SGLang reporting 206 tok/s on a single RTX 5090, plus cloud partners including Together, Fireworks, Modal, DigitalOcean, DeepInfra, and others. Practical deployment details mattered here: Unsloth claimed NVFP4 and dynamic GGUF builds, and Qwen emphasized 27B on 17GB RAM for local use (post).

• DeepSeek V4-Pro and RedNote’s dots3-note continue the China open-model wave: vLLM announced support for DeepSeek-V4-Pro, calling out MIT licensing, checkpoint compatibility with the preview path, and integrated drafting support. Meanwhile RedNote’s AI lab released dots3-note Preview, a 280B multimodal MoE with 16B active params and 512K context, aimed at long-running agents and accompanied by a new RL method, TEMPO, for long-horizon self-evaluation (early signal, summary, technical explanation from the team). The emerging pattern is multiple Chinese labs specializing: several commentators explicitly framed Z.ai, DeepSeek, Moonshot, Qwen, MiniMax, and RedNote as a fast-moving open ecosystem with different strengths (one synthesis, another).

Agent Runtimes, Harnesses, and Long-Horizon Training

• DeepSeek Harness is being treated as infrastructure, not a demo agent: The release sparked more discussion about runtime architecture than model UX. Several deep dives described the harness as a pluginized agent runtime where the agent loop, tools, sessions, filesystem, and providers are all replaceable, with Cordis providing lifecycle management, reactive dependencies, and reversible effects (overview, runtime composability thread). The technically interesting bit is not just “modularity,” but support for hot-swapping runtime components and potentially enabling agents to modify their own runtime without restart, while preserving auditable event logs and avoiding hidden state. Multiple builders reacted that current harnesses are probably “wrong” or at least too fixed-core compared with this direction (reaction).

• Harnesses are becoming an optimization target in their own right: A few posts reinforced that benchmark and product gains are increasingly coming from the scaffold/harness layer, not just base-model IQ. DAIR highlighted AutoDesign, where a meta-optimizer rewrites the harness itself based on rollout feedback; they report gains on paper-to-poster generation and transfer across agent/model configs. Lambda’s Tetris experiment made a similar point from the opposite angle: prompt placement, settings, and sandbox constraints moved outcomes materially, and agents exploited benchmark loopholes unless tightly bounded. This aligns with broader discussion that observability data is now doing double duty as evals, memory, and learning substrate (LangSmith docs note).

Benchmarks, Evals, and Benchmark Skepticism

• New evals targeted real agent failure modes: Vals launched an agentic reverse-engineering benchmark focused on deterministic end goals in cybersecurity-relevant binary settings rather than intermediate artifacts; a companion post argues current frontier agents are much stronger when source is available than when they must reason over binaries (context). OpenRouter introduced web search benchmarks for tool-grounded agents, while Ai2’s TutorMoments was cited as a replay-based tutoring eval showing models often over-help rather than encouraging productive struggle.

• The eval backlash continues: A recurring theme was skepticism toward vendor benchmark claims. Vik Paruchuri criticized a LlamaIndex benchmark, saying scorer bugs could move a system from 65% to 93.6%, and explicitly argued developers should run their own evals rather than trust marketing—“including ours” (follow-up). François Chollet reiterated that the public ARC-3 demonstration set is not training or eval data and that leaderboard scores there are weak proxies for private-set performance. Another worthwhile addition here is Meta’s Wiggle Framework, highlighted by Omar Sar: it stress-tests LLM judges under re-prompting and adversarial pressure, finding verdicts can flip 25–71% under static pushback and 62–91% under an adversarial persuader.

Infra, Serving, and Cost Engineering

• Serving optimizations are increasingly first-class model features: Day-0 infra support around Qwen and DeepSeek emphasized things like embedded draft heads, speculative decoding, and memory/quantization tradeoffs rather than only API access. Qwen’s 27B release arrived with vLLM guidance on MTP draft heads, 1M context, and serving on one Blackwell GPU, while ggerganov showed local llama.cpp recipes for large contexts and speculative decode. Tim Dettmers teased upcoming efficiency methods for running a strong model on a single DGX Spark or AMD Strix Halo at ~7 tok/s decode and >250 tok/s prefill.

• Tooling and cluster ops also got practical updates: Stas Bekman added guidance for diagnosing hanging NCCL collective calls in PyTorch, and separately noted that Python 3.14+ allows attaching pdb to a running process without instrumentation (post). Turbopuffer described a custom control plane for operating 100+ TPUf clusters, including BYOC deployments in customer clouds without direct host access. On the data side, Hugging Face’s datatrove 0.10.0 release added a JobsPipelineExecutor for Hugging Face Jobs, HF bucket integration, and preserved reasoning outputs.

Product and Platform Moves: Cursor/SpaceXAI, Gemini 3.7 Flash, Claude Code, and Local Agent UX

• Cursor joins SpaceXAI: The highest-engagement technical/corporate move was Cursor announcing it is now part of SpaceX, with the team joining SpaceXAI to work across Grok, Grok Build, Grok Bot, Grok API, and Cursor. SpaceXAI confirmed the acquisition and framed it as accelerating software engineering first, then broader knowledge work. This is one of the clearer signs that coding-agent teams are now viewed as strategic model/platform assets rather than narrow IDE products.

• Gemini 3.7 Flash rollout focused on agents and workhorse economics: Google pushed Gemini 3.7 Flash broadly across the Gemini app, Search AI Mode, Google Workspace / Sheets canvas, and Spark. The positioning was “most intelligent workhorse model yet for coding and agents,” with demos centered on turning simple prompts into playable web games (Google demo thread). External eval signal was modest but positive: Vals placed it at #7 on Vals Index v2 at 59.4%, up from #14 for Gemini 3.6 Flash.

• Claude Code and local-agent UX keep getting more operational: Anthropic rolled out Auto mode as the default permissions mode in Claude Code for Pro/Max/Team, with repo-aware setup via /auto-mode-setup to suggest trusted repos/domains (announcement, setup details). On the open/local side, Hermes added /loop for cron-like repeated actions inside an agent session, and Nous pointed out Hermes Desktop can target a Hermes Cloud agent, letting work continue after closing the laptop. Ollama also added support for launching the DeepSeek Harness locally.

Top tweets (by engagement)

• Cursor × SpaceXAI: Cursor’s acquisition announcement was the day’s biggest tech tweet by engagement, signaling continued consolidation around coding agents and vertically integrated model/product stacks.

• GLM-5.3 release: Z.ai’s GLM-5.3 launch was the top model-release tweet, largely because it sharpened the argument that post-training and long-horizon RL can unlock large latent capability from an already-trained frontier base.

• Qwen3.8-27B open weights: Alibaba’s release drew major attention because a 27B local multimodal model is now being marketed as viable for serious agentic/professional work with broad day-0 support.

• Practical coding-agent win: redp314’s “Claude Code built a DICOM viewer from 800 files in two prompts” stood out as a strong real-world example of the current ceiling for coding assistants outside benchmark talk.

• A preliminary Qwen3.8-27B model card is live! (Activity: 1006): The image is a technical screenshot of the preliminary Hugging Face model card for Qwen/Qwen3.8-27B (image), matching the post’s note that the card was visible before release and then went live. It indicates planned availability of model weights/config files, compatibility with Transformers, vLLM, and SGLang, and highlights improvements in coding, agent execution, research, and long-context use, with a stated native context length of 262,144 tokens and extension up to 1,000,000 tokens. Commenters focused on reasoning effort as a likely headline feature, praised the long-context window, and noted surprise that the 27B model appears to include vision capabilities while the much larger 2.4T model reportedly does not.

  • Commenters highlighted the model card’s stated native 262,144 token context length, with extension up to 1,000,000 tokens, as one of the most technically notable specs for Qwen3.8-27B.

  • There was interest in architectural/product-line differences: the 27B model reportedly includes vision support, while the much larger 2.4T model does not, which users found surprising from a capability-scaling perspective.

  • A commenter noted the absence of any explicit QAT / quantization-aware training mention, comparing it to Gemma 4 31B, where QAT was seen as materially improving quantized-model performance. Others also pointed to “reasoning effort” as an emerging tuning/control feature in recent model cards.

• Qwen3.8-27B is identical to Qwen3.6-27B! (Activity: 902): The image (GIF) shows side-by-side architecture diagrams for Qwen3.6-27B and Qwen3.8-27B that are visually identical: same vision/embedding path, masked scatter, repeated Qwen3_5DecoderLayer stack, RMSNorm, final Linear, and output. The linked HF Viewer diff reports 0 architectural changes, supporting the post’s claim that any capability gains in Qwen3.8-27B likely come from training/data/finetuning updates rather than model architecture changes. Commenters framed this as an incremental update rather than a from-scratch model, with one noting that training data is usually the largest quality lever. Another speculated that hot-swappable LoRA-style adapters may become popular for improving local-model accuracy on specialized tasks.

  • Several commenters interpreted Qwen3.8-27B as an incremental update rather than a model trained from scratch, with one noting it appears effectively the same as Qwen3.6-27B and even Qwen3.5. The technical implication raised was that dataset changes or post-training updates may be the main quality lever, rather than architectural changes.

  • A commenter pointed to Ninfer (GitHub) as a high-throughput local inference path for Qwen variants, citing newly added concurrent request support up to C=8. Reported numbers include Qwen3.6-35B-A3B reaching 1,313.8 aggregate decode tok/s at C=8, while the 27B NVFP4 profile reaches 1,146.9 tok/s, or 5.67× its single-concurrency throughput.

  • There was speculation that hot LoRA swapping could become important for local inference workflows, enabling task-specific accuracy improvements without replacing the base model. This was framed as a way to compensate for small or incremental base-model updates by dynamically applying specialized adapters.

• Qwen3.8-27B is now available (Activity: 745): The image (link) shows the Hugging Face page for Qwen/Qwen3.8-27B-FP8, indicating a newly available 28B-parameter Qwen 3.8 model packaged with Transformers, Safetensors, Apache 2.0 licensing, and FP8 quantization using F8_E4M3 alongside BF16 tensors. A commenter reports early local inference on an RTX 5090 at roughly 50–60 tokens/s, saying it feels more stable and deliberative than Qwen 3.6, though they note settings may not be optimal and MTP support is apparently not available yet. Comments are cautiously enthusiastic, with one user describing the model as a “grown up 3.6” with stronger long-running task handling. Another commenter asks whether smaller or alternative sizes such as 9B or 35B are available, since 27B is too large for many local users.

  • A user testing Qwen3.8-27B on an RTX 5090 reported stable local inference at roughly 50–60 tokens/s using the same settings as Qwen 3.6, noting performance may improve once MTP support is available. Qualitatively, they found it more deliberate than Qwen 3.6 on long-form generation: instead of immediately drafting a 10k-word story, it revised for cross-paragraph consistency, broke the task into subtasks, and generated chapter-by-chapter with more planning.

• Muse Glimmer was frontier In the model class around 30b models for four days. (Activity: 502): The image is a benchmark table comparing ~30B-class models, with Muse Glimmer-30B and Qwen3.8-27B highlighted: the post argues Muse Glimmer was “frontier” in this size class for only four days before Qwen’s 27B model surpassed it on most reported metrics. Muse Glimmer shows scores like 51.7 Agentic terminal coding, 51.2 SWE-bench Pro, 77.0 IFBench, and 83.5 GPQA Diamond, but many benchmark cells are missing, making the comparison incomplete; image: i.redd.it/2cclgla7xdjh1.png. Comments frame this as evidence that model labs should release multiple parameter scales to avoid being leapfrogged in a single class, with one commenter suggesting Meta should have shipped larger Glimmer variants like 70B, 100B, or 400B. Others speculate that a 27B model reaching near “Opus 4.6 Max” territory would be surprising, while hoping Meta responds with a stronger frontier release.

  • A commenter notes that Muse Glimmer shipped with speculative decoding, which reportedly improved TPS/throughput, and asks whether Qwen has an analogous acceleration path. This is the most concrete implementation-related point in the thread, though no specific TPS numbers or decoding configuration are provided.

  • One technical criticism compares Muse Glimmer unfavorably to Qwen, claiming Glimmer makes more “cognitive mistakes,” including reasoning traces that drift into irrelevant content-policy arguments and then contradict the final answer. The commenter says Qwen’s writing style is less preferred, but they have not observed the same class of reasoning/final-output inconsistency.

  • Another commenter frames the result as ~27B parameters approaching “Opus 4.6 Max level”, implying unusually strong performance for the ~30B model class. However, the thread does not provide benchmark names, scores, evaluation methodology, or reproducibility details to substantiate the comparison.

• Fixed Jinja chat template for Qwen 3.5, 3.6, and the new 3.8 release (Activity: 478): A community-maintained drop-in Qwen fixed Jinja chat template targets Qwen 3.5, 3.6, and new 3.8, addressing reported official-template failures: enable_thinking=false hard exceptions, poisoned multi-turn history from blank injection, crashes on OpenAI-style JSON-string tool arguments, and dropped mid-dialogue system messages causing stalled tool loops. The template adds Qwen 3.8 reasoning_effort steering (xhigh, high, medium, low), restores reasoning disablement via kwargs or <|think_off|>, preserves prior thoughts for prefix/KV-cache reuse, supports llama.cpp --reasoning-preserve, and recommends llama-server ... --jinja --chat-template-file chat_template.jinja --reasoning-format deepseek to emit thoughts as OpenAI reasoning_content. The author notes they cannot locally validate the 2.4T model but report 28 automated tests plus tokenizer parity checks, and request feedback from Qwen 3.8 users. Commenters questioned why Qwen’s official chat templates ship with such basic regressions and whether their QA covers template/tool-calling paths. Another commenter highlighted interest in testing smaller, more accessible variants such as 27B.

  • A commenter reports a Qwen 3.8 chat-template regression where enable_thinking=false does not merely fail to disable reasoning but causes a hard exception, implying the new template path may not handle the non-thinking mode despite exposing the flag.

  • Another technically relevant report says the published template did not produce reliable tool calling for Qwen 3.6 + Hermes Agent + LM Studio, requiring the user to develop a custom Jinja chat template for that stack. This suggests the failure mode may be integration-specific around tool-call formatting rather than base text generation.

• GLM 5.3 Released (Activity: 2227): Z.ai announced GLM-5.3 in an official release post, with the accompanying benchmark chart showing GLM-5.3 substantially ahead of GLM-5.2 across coding, agentic automation, and security-oriented evaluations. The image highlights GLM-5.3 leading or being highly competitive on benchmarks such as AutomationBench, CyberGym, and GDPVal-AA v2, while other models like GPT-5.6 Sol or Mythos/Fable 5 remain ahead on some tasks such as DeepSWE and ExploitBench. Commenters mostly framed this as another rapid Chinese model release; one noted that although this appears to be an API-model announcement, discussion is still relevant because the team has reportedly said weights will be forthcoming.

  • A commenter notes that GLM-5.3 is currently being discussed as an API model release rather than an immediate weights release, but argues it is still relevant to the local/open-model community because the team has reportedly said weights are forthcoming. This frames the release as potentially important for future self-hosting or benchmarking once checkpoints are available.

  • One technical takeaway highlighted from the release wording is: “Scaling post-training is all we did for GLM-5.3.” Commenters interpreted this as notable because it suggests the improvement may come primarily from larger or more intensive post-training/RL/instruction-tuning rather than a new base architecture or pretraining run.

• DeepSeek: We’re launching DeepSeek-V4-Pro today! (Activity: 729): DeepSeek announced DeepSeek-V4-Pro on X (post), and commenters note that model weights have been released on Hugging Face as deepseek-ai/DeepSeek-V4-Pro-0813. A top technical comment highlights new API pricing via an attached pricing image, implying a significant price increase relative to prior DeepSeek offerings. Commenters argue the price hike weakens DeepSeek’s main advantage: despite being “token hungry and a little slower,” it was previously attractive because it was cheap; at higher API prices, some users say they will return to local inference.

  • DeepSeek-V4-Pro weights are reported as released on Hugging Face at deepseek-ai/DeepSeek-V4-Pro-0813, shifting some discussion from API economics to self-hosting feasibility. Commenters argue that if the model’s performance is competitive and infra/electricity costs work out, open weights could let third-party providers undercut the official API.

  • Several commenters focused on the API pricing increase, saying DeepSeek’s prior appeal depended on being very cheap despite being “token hungry” and somewhat slower. The concern is that higher token pricing makes the hosted API less attractive versus local inference or alternative providers.

  • One early user disputed DeepSeek’s claimed parity with Kimi 3, saying V4-Pro does not match Kimi’s “knowledge / long term ability to work on a project hands off.” The criticism is specifically about extended autonomous project work and retained task context, not just short benchmark-style outputs.

• It’s actually crazy how good DSv4 Flash 0731 is (Activity: 556): The image is an Artificial Analysis Intelligence Index bar chart showing DeepSeek V4 Flash 0731 max scoring 52, ranked 46/608, effectively clustered with top frontier models like GPT-5.6 Terra and GLM-5.2 at 53. The post highlights the practical significance: a model near the top of the benchmark table is reportedly usable on a sub-$2k local machine, making it notable for local/offline inference relative to larger frontier APIs. Commenters pushed back that the benchmark may overstate real-world capability: one user said GLM 5.2 remains much stronger for programming and that DeepSeek wastes tokens on complex tasks. Others argued Qwen 3.6 27B is even more impressive due to similar ranking at roughly 1/5 the size, while another said DSv4 Flash is the first locally runnable model that does not feel like a downgrade from frontier models.

  • Several users challenged the headline benchmark implication for DeepSeek V4 Flash 0731, arguing that real coding performance can lag chart results. One commenter reported spending >$100 in API credits and said that on complex programming tasks it often “wastes a ton of tokens doing useless investigations” and may fail to converge, while GLM 5.2 was described as still clearly stronger for programming.

  • A notable comparison was raised with Qwen 3.6 27B, which commenters said appears close to DeepSeek V4 Flash on the referenced chart despite being roughly 1/5 the size. The technical implication discussed is that Qwen may offer a better parameter-efficiency tradeoff if the benchmark placement reflects real workload performance.

  • One user highlighted local usability: DSv4 Flash 0731 was described as the first locally runnable model they had used that “doesn’t feel like a downgrade from frontier models”, becoming their default workhorse for home projects. Another commenter criticized the benchmark chart methodology, noting it showed “Selected 46 of 608 models” and questioning whether the comparison set was cherry-picked or unrepresentative.

• Deepseek Harness is Up! (Activity: 537): DeepSeek AI announced DeepSeek Harness (dsh), an open-source agent harness in developer preview, built around an “everything is a plugin” architecture and powered by Cordis, whose design is described in A Programming Paradigm for Spatiotemporal Composability. The project is explicitly unstable—“THERE WILL BE COMPATIBILITY-BREAKING CHANGES”—and DeepSeek is directing developers to its Discord community for updates and discussion.** Top comments focused on ecosystem skepticism: one user questioned why agent harnesses are so often written in TypeScript, another suspected bot-driven GitHub growth after reported stars jumped from 20k to 30k in about an hour, and a third asked whether dsh can achieve better cache hit rates than reasonix.

  • Commenters pointed to the official DeepSeek Harness repository and docs: github.com/deepseek-ai/deepseek-harness and deepseek.com/harness/en. One technical concern was whether it can achieve higher prompt/cache hit rates than Reasonix, since cache efficiency is increasingly important for inference cost and latency.

  • A commenter questioned why many agent/harness implementations are written in TypeScript, contrasting this with Codex as a possible exception. The concern implies friction for lower-level performance tuning or integration compared with Python/Rust/native tooling, though no benchmarks or implementation details were provided.

• Trained a 1.5B to write shell commands so I’d stop googling tar flags. Runs on a laptop CPU in ~1 sec. (Activity: 1815): The image is a terminal/CLI demo splash screen for the whatisit tool, showing ASCII art in a dark terminal rather than benchmark output or model internals: image/GIF. Context from the post is technical: the author fine-tuned Qwen2.5-Coder-1.5B on 125k natural-language→shell-command pairs, quantized it to Q4_K_M (941MB) for llama.cpp, and reports CPU performance of 31.9 tok/s, 0.59s median/query, 1.6GB RAM, plus 0.620 on InterCode-ALFA vs 0.613 for untuned Qwen2.5-Coder-7B and 0.73 for GPT-4o. The released artifacts are Apache-2.0 weights on Hugging Face and code on GitHub, with a static safety checker because the model can generate destructive shell commands if prompted. Comments were mostly lighthearted rather than deeply technical: users joked that this is “lots of effort to not use man pages,” offered mnemonic tar flags like -czvf / -xzvf, and warned that an NL-to-shell model is potentially dangerous—“like giving a loaded T34 tank to an infant.”

  • A commenter asks whether the author evaluated Gemma Shellper, a smaller shell-command-focused model reportedly under 0.5B parameters, as a baseline or alternative. The comparison is technically relevant because the post’s model is 1.5B and targets ~1 sec CPU inference on a laptop, so latency/accuracy tradeoffs versus a much smaller model would be useful.

• Doom running on an LLM -- Hugging Face checkpoint included (Activity: 347): The author compiled Doom’s deterministic renderer—not trained it—into a stock Phi3ForCausalLM checkpoint using torchwright, with all weights computed analytically and loadable via vanilla transformers with trust_remote_code=False (write-up, source). The prompt encodes level geometry/player pose/view direction and generation emits drawing commands consumed by a 43-line raster host; the 320x200 model is 21B params / 85.87 GB, requiring 3,614 prompt tokens + 53,747 generated tokens per frame and taking just under 40 min on a B200, while the practical 80x50 checkpoint is a 34 GB download (80x50 weights, 320x200 weights). The current compiler requires fp32 weights; the author has only run it on cloud B200/A100-80 GPUs and recommends 80 GB VRAM for the 80x50 model, with 64 GB possibly sufficient but untested. The main technical pushback is that 53,747 tokens in ~40 min on a B200 for a 21B model seems far slower than expected—one commenter claims dual RTX 3080s can generate a similar token count on 27B within 30 min, suggesting a serious optimization issue. Another commenter asks why the project targets an LLM/text-generation architecture rather than a transformer image generator, i.e. whether the choice is purely for the “Can it run DOOM?” novelty or has a technical rationale.

  • A commenter questioned the reported inference performance: “One frame is a 3,614-token prompt plus 53,747 generated tokens -- just under 40 minutes on a B200” for a 21B model, arguing this is far slower than expected and may indicate a broken/unoptimized generation path. They compared it to their own setup claiming a pair of RTX 3080s can generate a similar token count on a 27B model in under 30 minutes, despite being much weaker than an NVIDIA B200.

  • The same commenter asked why the project uses a stock Phi3ForCausalLM LLM architecture—where the prompt encodes level geometry/player pose/view direction and generation emits drawing commands consumed by a 43-line host renderer—instead of a transformer-based image-generation approach, questioning whether the choice was purely for novelty or had a technical rationale.

/r/Singularity, /r/Oobabooga, /r/MachineLearning, /r/OpenAI, /r/ClaudeAI, /r/StableDiffusion, /r/ChatGPT, /r/ChatGPTCoding, /r/aivideo, /r/aivideo

• Gemini 3.7 Flash Benchmarks (Activity: 1182): A Reddit post titled “Gemini 3.7 Flash Benchmarks” discusses benchmark results for Google Gemini 3.7 Flash, but the provided excerpt does not include the actual benchmark table, metrics, tasks, or methodology. Commenters characterize the results as unusually strong for a low-latency/cost-optimized “Flash” model, with one calling it “amazing for a flash model.” The main debate is benchmark relevance: one commenter argues that “97% of flash users” care more about practical qualities like creative writing, emotional intelligence, web search, and hallucination behavior than leaderboard-style scores. Gemini Flash is framed as a strong value model, especially compared with perceived cost increases from DeepSeek.

  • Commenters interpreted the posted Gemini 3.7 Flash benchmark results as unusually strong for a “Flash”/low-cost model tier, with one comparing its apparent performance favorably against Sonnet 5. No concrete benchmark numbers were discussed in the comments, but the theme was that the model may be closing the gap with higher-end competitors while remaining a value-oriented option.

  • One technical critique was that standard benchmark suites may not reflect the majority of Flash usage patterns: a commenter argued that “97% of flash users” care more about creative writing, emotional intelligence, web search quality, and hallucination rate than leaderboard-style scores. They still characterized Flash as potentially the best bang-for-buck LLM, implying cost/performance and real-world reliability matter more than raw benchmark wins.

• Holy... Google actually did it, they actually shipped a frontier model (Activity: 1123): The post reports hands-on testing of Google Gemini 3.7 Flash, characterizing it as a very fast “workhorse” model with strong instruction-following and no observed hallucinations in the author’s tests. A notable anomaly was one run where the model began reasoning in Chinese while still completing the task correctly, suggesting a possible language-routing or hidden-chain-of-thought leakage issue. Commenters broadly push back on prior anti-Gemini sentiment: one says it is “much better” in Antigravity, while another argues it is not truly frontier-level but closer to a Claude Sonnet-class everyday model used for ~80% of tasks, with expectations that Gemini 4 may be frontier-level.

  • One commenter reports hands-on testing in Google Antigravity, saying the new Gemini model is “much better” in that coding-agent environment, though no concrete benchmark numbers or failure cases were provided.

  • A more technical framing compares the model to Claude Sonnet-class systems rather than an absolute frontier leader: it is described as a likely 80% of usage “workhorse” model, with speculation that Gemini 4 may be the model that reaches clear frontier status.

• Example of a real working loop orchestrator (Activity: 1567): The image (PNG) shows a non-meme, working AI loop orchestrator dashboard (“Llyod’s Mission”) used to manage recurring agent sessions and a SQLite-backed internal ticket/memory system. The setup centers on a configurable heartbeat / pulse loop that runs playbooks such as checking inbound bug-report emails, querying prior tickets, inspecting app logs, updating docs, and spawning/monitoring child sessions with visible status, model, progress, cost, and deployment actions like Create PR, Commit & Push, Worktree, and Release Notes. The technical significance is that the orchestrator treats agent memory as an operational database—effectively an internal Jira/tribal-knowledge store with 600+ tickets—so new tasks can be grounded in previous context across models. Commenters generally viewed the setup as a useful concrete example of agent infrastructure beyond a chat UI, especially for email triage and business workflows. One commenter echoed the same pattern—local history tables for client email context—while another said it clarified how to build harnesses, managers, and dashboards around Claude/agent workflows.

  • One commenter described a production-ish inbound email orchestrator that uses a local table of historical client email exchanges as persistent context. When a new email arrives from a known client, agents can inspect prior issue history without the user manually injecting context, effectively turning the loop into a lightweight client-support memory/RAG workflow.

  • Another commenter outlined a more complex always-on architecture: three 24/7 Claude agents on separate machines, each owning a domain and able to spawn subagents across multiple providers/models. They coordinate through a shared main ticket table, plus per-agent Kanban boards used to delegate specialized tasks to subagents based on occupation, task type, provider, and model.

  • The same setup includes a hierarchy where one orchestrator owns the global ticket queue but can escalate or route work to other orchestrators when a task falls under their domain. Human interaction is mediated through a voice-controlled “Hermes” agent on a phone, which can assign tickets, relay messages, and provide status updates.

• I make Claude Code keep a MISTAKES.md file. Here’s what actually happened. (Activity: 1089): The post describes a lightweight persistent-memory workflow for Claude Code: add MISTAKES.md to the repo and instruct CLAUDE.md to append failures with what happened / root cause / consequence / prevention, newest-first. The author reports that Claude later references this file to avoid repeated errors, and recurring entries are promoted into enforceable CLAUDE.md rules, turning anecdotal “flaky area” memory into countable failure patterns and guardrails. Commenters report similar regressions where Claude repeats known mistakes or prematurely stops despite instructions, with one user quoting Claude admitting it “ignored” prior guidance and caused the same issue again. Another commenter extends the idea with hook-triggered “skills” after specs, plans, and implementations to scan past errors against current work, claiming it catches many issues.

  • Several commenters reported that Claude Code repeatedly makes the same implementation errors unless prior mistakes are operationalized as part of the workflow. One user described Claude explicitly acknowledging it had previously avoided a broken approach on a given date, then “ignored this though and caused exactly the same problem again,” suggesting that passive documentation like MISTAKES.md is insufficient without retrieval or enforcement.

  • A more technical pattern was described: adding a secondary workflow layer using Claude Code skills + hooks that run after every spec, plan, and implementation step to scan past errors and compare them against the current work. The commenter said this has “caught so many fuck ups,” implying the useful mechanism is not the mistakes file itself but automated post-step validation against it.

  • There was debate over retrieval strategy: one commenter argued that merely referencing MISTAKES.md will not reliably trigger Claude to consult it, while forcing the whole file into context is inefficient. They suggested Claude’s memories system should be superior because short recall triggers remain in context automatically; another commenter emphasized that without enforceable checks, “it effectively doesn’t exist and will always be ignored by the LLM eventually,” showing an implementation screenshot: https://preview.redd.it/prj0dddf05jh1.png?width=3400&format=png&auto=webp&s=b4164b5a6ffad94c85eee175907cbd45d1efd0db

• DeepSeek just massively increased their API prices (effective August 16, 2026) - up to 1,114% increase for cache hits (Activity: 2009): DeepSeek is updating its API pricing effective 16:00 UTC, August 16, 2026, adding peak/off-peak billing where peak windows (01:00–04:00 and 06:00–10:00 UTC) cost 2× off-peak. The largest increases are on cached-input tokens: V4-Pro cache hits rise from $0.003625 to $0.022/$0.044 per M tokens off-peak/peak, i.e. +507%/+1,114%; V4-Flash cache hits rise from $0.0028 to $0.007/$0.014, i.e. +150%/+400%. Cache-miss input and output pricing also increases substantially, with V4-Pro output moving from $0.87 to $1.98/$3.96 and V4-Flash output from $0.28 to $0.66/$1.32. Comment sentiment is negative but technically thin: users suggest DS4 remains attractive mainly when cheap, and at least one commenter says they have already shifted workloads away. The main implied operational concern is that cached-context-heavy and long-conversation workloads lose much of DeepSeek’s prior cost advantage, especially during peak UTC windows.

  • One commenter notes they have already migrated away from DeepSeek, saying DS4 is only attractive “when cheap”—implying the price increase may erase its main advantage versus competing API models unless its quality/performance justifies the new rate.

  • A user in Brazil points out that DeepSeek’s off-peak pricing window may align unusually well with their local daytime usage: “off peak hours: 7:00 > 22:00”. This suggests regional timezone effects could materially change the real-world impact of the price hike for latency-tolerant workloads that can be scheduled into discounted windows.

• Some Claude users are mad that Anthropic’s new watermarks will catch them using it at their jobs, classes (Activity: 1160): The post discusses user backlash to Anthropic adding detectable watermarks/provenance signals to Claude outputs, with concerns that these markers could reveal AI use in workplaces or classes where disclosure may be penalized. A technical edge case raised in comments is that Claude used for proofreading/editing may cause otherwise human-authored text to be flagged as AI-associated, blurring attribution between generation and assisted revision. Commenters were split: one said workplace AI use is encouraged and a watermark would be “affirmation,” while another worried detectors would mislabel their own edited writing as “AI slop.” A separate comment criticized Yahoo for turning a Reddit thread into news, but it added little technical substance.

  • A commenter with education-sector experience argues that Anthropic-style watermarking is technically weak as an enforcement mechanism because open-weight models are not subject to the same watermarking constraints. They note a likely laundering workflow: use Claude for most generation, then pass the output through an open-weight model to paraphrase and potentially remove or obscure the watermark.

  • Several comments highlight a boundary problem: if Claude is used for editing, proofreading, formatting dictated text, or restructuring notes, watermarking may label a largely human-authored artifact as AI-generated. The concern is that detectors could conflate legitimate assistive use with full synthetic authorship, creating false accusations in workplaces or schools.

  • The education-focused comment warns that even improved statistical watermarking can reproduce problems seen with AI detectors: false positives and inequitable enforcement, especially for non-native English speakers or neurodivergent writers whose syntax may appear formulaic. The commenter recommends designing assessments that measure comprehension and AI literacy rather than relying on detection as a blunt academic-integrity tool.