Latent Space

AI News for 1/23/2026-1/26/2026. We checked 12 subreddits, 544 Twitters and 24 Discords (206 channels, and 14285 messages) for you. Estimated reading time saved (at 200wpm): 1208 minutes. Our new website is now up with full metadata search and beautiful vibe coded presentation of all past issues.

As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

3 months after OpenAI floated a trial balloon with ChatGPT Apps and the Apps SDK at Dev Day 2025, Anthropic has now officially absorbed the independent MCP UI project and, working with OpenAI, Block, VS Code, Antigravity, JetBrains, AWS, and others, has released both:

• the MCP Apps spec

• official support in Claude.ai - comparatively very well received but of course not as popular as the Claude in Excel announcement.

It’s fair to say that ChatGPT Apps haven’t exactly taken the world by storm since announcement, but the overall need for a standard format for applications to return rich UI still cannot be denied.

Now that MCP Apps have been ratified by all the important players, this is the basis for a rich ecosystem of open source support and applications being able to interoperate, and perhaps one day solve the perpetual never ending pile of $20/month subscriptions piling up in your credit card bills.

As a reminder, we interviewed David Soria Parra and the rest of the AAIF, who previewed a bit of the thinking and design process behind MCP Apps here:

Agent Orchestration, RLMs, and “Clawdbot/Clawd” as a UX pattern

• NVIDIA ToolOrchestra + Orchestrator-8B: NVIDIA’s ToolOrchestra frames agentic systems as a small “conductor” model that alternates reasoning with calls to tools and larger “expert” models (search, code execution, specialist LLMs, frontier generalists). The claim is that an 8B orchestrator can reach frontier-level outcomes via delegation at materially lower cost, trained end-to-end with scalable RL using automatically synthesized tool-use environments and multi-turn tasks (summary, link). Closest technical implication: “controller scale” matters less than policy quality + tool/model routing if you can train it with realistic tool-call rollouts.

• RLMs / recursion-first agent stacks: Several posts converge on a Recursive Language Model (RLM) pattern: pass files and context by reference and iteratively pull the minimum slices needed (shell/grep/AST), rather than stuffing everything into context à la ReAct. Dan B illustrates this with file references vs @file expansion as deliberate context management (thread). Daytona is positioning RLMs as “unlimited recursion depth” via per-(sub)agent sandboxes (guide, integration).

• “Clawd/Clawdbot” meme → product signal: The dataset contains a large “Clawdbot” wave (often with Mac mini jokes), but the technically relevant throughline is outcome-first assistant UX + tight context/tool integration. Kimmonismus explicitly calls this a shift from “more chat” to “more outcome,” suggesting incumbents will scramble to match it (tweet). Others push a cloud-first counterpoint (no local Mac mini) (MiniMax reply). There’s also an emerging security backlash as soon as “powerful mode” exists: prompt injection remains a system-level blocker for browser/desktop agents (dilemma, follow-up, Miessler warnings).

Reasoning model releases & eval dynamics (Qwen, Tencent, ARC, etc.)

• Alibaba Qwen3-Max-Thinking: Alibaba positions Qwen3-Max-Thinking as a flagship reasoning+agent model trained with “massive scale and advanced RL,” emphasizing adaptive tool-use (Search/Memory/Code Interpreter) and test-time scaling/self-reflection. They cite strong math and agentic search metrics (e.g., 98.0 on HMMT Feb, 49.8 on HLE) (launch). The model is immediately pushed into public eval channels: LM Arena Text Arena (Arena) and Yupp (Yupp). Community reaction highlights the tool-enabled evaluation regime—claims of outperforming multiple SOTA models on HLE with search tools (commentary).

• Tencent HunyuanImage 3.0-Instruct (image editing): Tencent releases an image-editing-focused multimodal model built on an 80B MoE (13B active), using a “Thinking” schema with native CoT and their MixGRPO algorithm; focus is on precise edits that preserve non-target regions and multi-image fusion (announcement). LM Arena reports it entering the top-10 image edit leaderboard (rank #7) (Arena).

• ARC-AGI cost/perf hacks: A notable optimization claim: “Recursive Self-Aggregation (RSA) + Gemini 3 Flash” reaching 59.31% on ARC-AGI-2 at ~1/10 cost vs Gemini Deep Think (tweet). This points to a broader theme: meta-inference strategies (aggregation, recursion, pruning) are becoming as important as base model choice.

• Open models in arenas: Molmo 2 (Apache 2.0) appears in Arena as a new open model entrant (Arena). Separately, Hugging Face Inference Endpoint notes GLM-4.7-Flash via llama.cpp with a low hourly price point (Q4_K_M, 24k context) (ngxson)—underscoring a continued commoditization of fast open-weight inference.

RL everywhere: test-time training, GRPO stabilization, RL-as-pretraining, and compute savings

• Test-Time Training (TTT) + RL breakthroughs: A widely shared result claims a Stanford/NVIDIA-style TTT+RL approach that: beats AlphaEvolve, finds a new upper bound for an Erdős overlap problem, produces A100 kernels ~2× faster than best human kernels, and beats both best AI+human attempts on AtCoder (rronak_). This cluster also includes meta-discussion about correctly crediting related approaches (EvoTune) (Yejin Cho).

• GRPO training stability knobs: A small but actionable engineering tip: INTELLECT-2 reports a delta=4.0 parameter that improves GRPO stability (QGallouedec).

• RL in pretraining (RLP): NVIDIA authors announce RLP (Reinforcement as a Pretraining Objective) accepted to ICLR 2026, framing RL not as “post-training only” but as integrated into pretraining (ahatamiz1).

• Compute reduction via curriculum-like filtering: AI21’s “Dynamic Data Snoozing” claims up to 3× compute reduction for RLVR by snoozing examples that are too easy (DanielGissin). If validated, this is a practical recipe: make the sampler policy-aware instead of static.

Inference infrastructure & dev tooling: vLLM’s “day-0 model support,” VS Code MCP Apps, Cursor subagents

• vLLM’s governance and commercialization pressure: A long Zhihu-derived summary argues vLLM’s “open-source project → startup” shift was driven by the hidden cost of day-0 support (weeks/months of confidential pre-integration per new model), the rise of MoE and heterogeneous inference (fp8/int4/sparse attention), and the mismatch with PyTorch Foundation style testing vs vLLM’s multi-node CI needs. It claims the maintainers founded Inferact Inc to fund full-time maintainers while keeping vLLM open-source (thread). Related: vLLM shares a practical flag for avoiding OOM on long-context models: --max-model-len auto (vLLM tip).

• MCP Apps: tool calls return interactive UI: The MCP ecosystem announces MCP Apps as the first official MCP extension: tool calls can return interactive UI components rendered in-chat. VS Code is first major editor shipping support (Insiders now, stable soon) (VS Code, alexalbert__). Anthropic simultaneously ships “interactive work tools in Claude” (Slack drafting, Figma diagrams, Asana timelines) (Claude). Net: we’re seeing the “tool interface layer” move from raw JSON to native UI primitives inside agent loops.

• Cursor: multi-browser subagents: Cursor adds multi-browser support via subagents (Cursor), echoing the same direction: parallelized tool execution + better context isolation.

Kernel LLMs, chip stacks, and “AI for hardware” loops

• GPU MODE 2026: post-training Kernel LLMs in public: GPU MODE outlines a 2026 plan to post-train a Kernel LLM and get generated kernels merged into real repos (PyTorch/vLLM), emphasizing “de-slopify kernels” (determinism, reviewer-mergeable PRs), profiler-guided optimization + memory work, and competitions as evals (marksaroufim).

• Microsoft Maia 200: Microsoft announces Maia 200 as a custom inference accelerator; Mustafa Suleyman claims it’s the most performant first-party hyperscaler silicon, with 3× FP4 performance vs Trainium v3 and FP8 above TPU v7 (Mustafa, follow-up). Yusuf Mehdi frames this as infra that makes AI “dependable” (thread).

• Ricursive Intelligence (AI for chip design): Ricursive raises a $300M Series A aiming at end-to-end chip design as a recursive self-improvement loop between AI and hardware (company, Anna Goldie).

Safety, misuse, and societal impact (selected items with direct technical relevance)

• Elicitation attacks via benign chemistry data: Anthropic reports that fine-tuning open models on “benign” chemical synthesis content generated by frontier models can significantly increase capability on chemical weapons tasks—an “elicitation attack” that scales with frontier model strength (AnthropicAI, paper link).

• Dario Amodei’s “Adolescence of Technology” essay: A major, highly engaged post argues AI is entering an accelerating feedback loop (AI building AI), with risks spanning misuse, power-seeking autonomy, and economic disruption; it also explicitly frames wealth concentration as a society-breaking failure mode (Dario). Reaction ranges from strong endorsement to critique of how “takeover risk” framing is presented (Ryan Greenblatt).

• Agent security in practice: Multiple posts treat desktop/browser agents as inherently high-risk until prompt injection and sandboxing mature, reinforcing the need for strict isolation, least privilege, and careful handling of credentials (Miessler).

Top tweets (by engagement)

• “Clawdbot” misuse example (explicitly harmful)

• Karpathy on the phase shift to “programming in English” via agents

• Dario Amodei’s “Adolescence of Technology”

• 216GB VRAM on the bench. Time to see which combination is best for Local LLM (Activity: 366): The post discusses the use of secondhand Tesla GPUs, which offer substantial VRAM at a lower cost, for local large language model (LLM) testing. The author has developed a GPU server benchmarking suite to evaluate the performance of these GPUs when used in parallel. The image shows a technical setup with multiple NVIDIA GPUs, highlighting the focus on maximizing VRAM capacity. The discussion centers around the feasibility and efficiency of using these older GPUs compared to modern devices, particularly in terms of bandwidth and cooling challenges. Commenters express skepticism about the performance of these GPUs, noting potential issues with bandwidth and cooling. One commenter shares personal experience, comparing different GPU models and highlighting the challenges of using older hardware.

  • HugoCortell raises a technical concern about the potential bandwidth limitations when connecting multiple GPUs to a single PC, noting that most affordable server motherboards support only a few GPUs. This could impact the performance of local LLMs if not addressed properly.

  • dc740 shares insights from personal experience with different GPUs, highlighting that the P40 outperforms the M10 despite both being older models. However, they prefer using AMD Instinct Mi50 GPUs due to their performance, even though support for these was recently dropped from ROCm, indicating a trade-off between hardware capability and software support.

  • FullOf_Bad_Ideas critiques the gpu_box_benchmark for not testing scenarios where large models are split across multiple GPUs, which is a primary use case for setups with extensive VRAM. This points to a gap in current benchmarking practices that may not fully reflect real-world applications of multi-GPU systems.

• I just won an Nvidia DGX Spark GB10 at an Nvidia hackathon. What do I do with it? (Activity: 724): The image shows a terminal window on a Linux system running the ‘top’ command, which is used to monitor system processes and resource usage in real-time. The user has won an Nvidia DGX Spark GB10, a high-performance computing device designed for machine learning and data-intensive tasks. The terminal indicates a Python process consuming significant CPU resources, suggesting active computational tasks, possibly related to machine learning or data processing. The user is considering using the device to run multiple NextJS applications simultaneously, leveraging its powerful capabilities. One commenter suggests running three NextJS applications simultaneously, indicating the device’s capability to handle multiple high-memory tasks. Another commenter provides a link to Nvidia’s DGX Spark playbooks, which could be useful for the user to explore the full potential of their new hardware.

  • Fit-Produce420 highlights the capabilities of the Nvidia DGX Spark GB10, noting that with 128GB of memory, it can fine-tune models up to 70 billion parameters. Additionally, it can handle larger models like the 120 billion parameter gtp-oss-120b using techniques like QLoRA, which optimizes memory usage for large-scale models. However, running dense models like devstral 2 may be slow due to their computational demands.

  • randomfoo2 suggests utilizing the NVIDIA DGX Spark playbooks as a resource for getting started with the DGX Spark GB10. These playbooks provide structured guidance and best practices for deploying and managing workloads on the DGX platform, which can be particularly useful for users new to this hardware.

  • LicensedTerrapin humorously suggests selling the DGX Spark GB10 to purchase 8GB of DDR5 RAM, implying a trade-off between high-end specialized hardware and more general-purpose upgrades. This comment reflects a common debate in tech communities about the value of specialized versus general-purpose hardware investments.

• Using a high-end MacBook Pro or a beefy RTX 5090 laptop (with 24 GB of RAM) for inference. (Activity: 29): The post discusses the feasibility of using a high-end MacBook Pro with Apple Silicon (M-series Max) versus a Windows/Linux laptop with an RTX 5090 GPU for running large local LLMs (70B+ parameters) for inference and fine-tuning. The MacBook Pro offers 128–192 GB of unified memory, while the RTX 5090 laptop provides 24 GB of VRAM and at least 64 GB of system RAM. The primary use case is local LLM inference with a target of ≥15 tokens/sec, emphasizing portability. The post queries whether the larger unified memory of Apple Silicon outweighs the CUDA performance of the RTX laptop for inference, and how Apple MLX compares to CUDA for fine-tuning tasks like LoRA/QLoRA. It also seeks insights on thermal performance and sustained inference capabilities of both setups. One commenter suggests using the laptop as a terminal to a more powerful desktop, indicating a preference for leveraging remote resources over local hardware. Another commenter is experimenting with both setups, using a MacBook Pro M2 Max for inference, and is curious about the performance differences.

  • racerx509 shares their experience using a Lenovo laptop with a 3070ti, a custom desktop with a 5070, and a MacBook Pro M2 Max with 96GB RAM for inference tasks. They note that they have been primarily using the MacBook Pro for inference, suggesting it may offer better performance or convenience for their needs.

  • No-Concern-8832 raises a concern about the VRAM limitations of RTX laptops, suggesting that they may not be sufficient for running large models like 70B parameters. This highlights a potential limitation in using high-end RTX laptops for certain deep learning tasks that require substantial VRAM.

  • Tired__Dev discusses their experience with an Asus M16 equipped with a 4090 GPU, noting that it struggled with a 7B parameter model. They express a preference for a MacBook Pro with 128GB RAM, citing its high memory bandwidth and potential performance advantages over even high-end GPU setups like the DGX Spark.

• I built a “hive mind” for Claude Code - 7 agents sharing memory and talking to each other (Activity: 313): The post describes a multi-agent orchestration system for Claude Code, featuring seven specialized agents (e.g., coder, tester, reviewer) that coordinate tasks, share persistent memory using SQLite + FTS5, and communicate via a message bus. The system runs as an MCP server and integrates with Anthropic, OpenAI, or Ollama. It uses a task queue for priority-based coordination, allowing agents to pass context and collaborate effectively. The implementation stack includes TypeScript, better-sqlite3, MCP SDK, and Zod. The project is experimental, open-source under the MIT license, and available on GitHub. A comment questions the system’s uniqueness compared to the BMAD method, suggesting similarities. Another comment humorously questions whether the agents agree with each other, hinting at potential coordination challenges.

  • The user robiinn inquires about the differences between the ‘hive mind’ system and the bmad method, suggesting a potential similarity. This indicates a need for clarification on the unique aspects or improvements of the ‘hive mind’ approach over existing methods, such as how memory sharing and inter-agent communication are implemented differently.

  • No_Afternoon_4260 raises a critical point about the consensus among the agents in the ‘hive mind’. This touches on the technical challenge of ensuring that multiple agents can not only share memory but also reach agreement or consensus, which is a significant aspect of distributed systems and multi-agent frameworks.

  • JellyBean504 draws a parallel between the ‘hive mind’ and Steve Yegge’s Gastown, suggesting that there might be conceptual similarities. This comparison could be valuable for understanding the architectural or functional parallels between the two systems, potentially offering insights into design choices or performance characteristics.

• Clawdbot: the AI assistant that actually messages you first (Activity: 214): Clawdbot is an open-source AI assistant with over 9K GitHub stars, designed to proactively message users, unlike traditional AI assistants that wait for prompts. It integrates with locally hosted LLMs via Ollama and supports messaging apps like WhatsApp, Telegram, and Discord. Key features include sending automated briefings and reminders, local storage of conversations as Markdown files, and the ability to control browsers and run scripts. The software is free under the MIT license but requires terminal proficiency for setup, as there is no GUI installer. Read more. Users report challenges with setup, particularly with obtaining and using OAuth keys for authentication, and difficulties in connecting local LLMs without relying on API keys. Some users express frustration with the complexity of setup, especially when using remote machines.

  • mike7seven highlights the complexity of setting up Clawdbot, particularly emphasizing the need to obtain a Claude OAuth key on a separate machine and then transfer it to the setup machine. This process is noted as cumbersome, especially for those using remote machines, and the MacOS app requires building from source, adding another layer of complexity.

  • Ashamed_Promise7726 raises a technical challenge regarding the integration of local language models with Clawdbot. The user notes difficulty in connecting pre-downloaded models on their PC, as Clawdbot seems to require an API key for usage-based models, questioning the feasibility of running Clawdbot entirely locally without external dependencies.

  • inigid warns about potential security risks associated with Clawdbot, suggesting it could be exploited for supply-chain attacks that compromise sensitive data on a user’s machine and network. The comment also mentions concerns about the association with Solana meme coins, implying a need for caution when using the tool.

• GLM-4.7-Flash is even faster now (Activity: 443): The recent update to llama.cpp by Johannes Gaessler optimizes the CUDA implementation of FlashAttention, specifically for models with a non-power-of-2 ratio of query heads to key/value heads. This is achieved by padding Q columns to the next power of 2, which, although slightly inefficient, enhances performance for small batch sizes. The update is detailed in pull request #19092. One comment humorously notes the obsolescence of a previous post due to this update, while another laments the lack of support for AMD GPUs, highlighting a common issue in the community regarding hardware compatibility.

  • The user ‘jacek2023’ provides detailed performance metrics for the GLM-4.7-Flash model, highlighting its efficiency. The model processes a prompt with 45074 tokens, achieving a prompt evaluation time of 2814.63 ms for 1612 tokens, which translates to 1.75 ms per token or 572.72 tokens per second. The overall evaluation time is 29352.57 ms for 1731 tokens, equating to 16.96 ms per token or 58.97 tokens per second. The total processing time is 32167.20 ms for 3343 tokens, indicating significant improvements in speed.

• KV cache fix for GLM 4.7 Flash (Activity: 380): The recent update to GLM 4.7 Flash involves removing the V component from the KV cache, which significantly reduces VRAM usage, allowing for longer context lengths on the same hardware setup. This change is particularly beneficial for models like DeepSeek and GLM 4.7 Flash, as it can save gigabytes of VRAM, enabling context lengths to double, as demonstrated by a user running a 90,000 context on a 4090 GPU. The update is part of a pull request in the llama.cpp repository, which introduces a V-less KV cache, reducing memory usage by nearly 50%. More details can be found in the pull request. A user noted that the model, while improved, still requires some manual guidance, especially in tasks like coding and creative writing, where it may not perform as well as specialized models. However, it excels in tool use and as an assistant, making it a preferred choice for home-server applications.

  • The user ‘teachersecret’ reports significant improvements in context handling with the UD’s k_xl 4-bit version of the GLM 4.7 model on an RTX 4090. Previously, the model maxed out at 45,000 context tokens, but now it can handle 90,000. Despite these improvements, the model still requires some manual guidance, especially in coding tasks, and is less effective in creative writing compared to other models. However, it excels in tool usage and is now the user’s default model for their home server.

  • User ‘viperx7’ provides detailed benchmark data comparing the performance of the GLM 4.7 model before and after a specific change. The benchmarks show improvements in both prompt processing and token generation speeds across different configurations. For instance, using a single RTX 4090, the context size increased from 64k to 128k, with prompt processing speed improving from 3489 t/s to 3510 t/s and token generation from 88 t/s to 92.5 t/s. The maximum context size achievable with a 4090 and 3060 setup is 200k, leaving about 6GB of VRAM unused.

  • The discussion highlights the technical aspect of the GLM 4.7 model’s KV cache fix, which allows for increased context sizes and improved performance metrics. The benchmarks provided by ‘viperx7’ indicate that the model can now handle up to 207k context size in certain configurations, with significant improvements in processing speeds. This suggests that the model’s efficiency has been enhanced, making it more suitable for high-demand applications.

/r/Singularity, /r/Oobabooga, /r/MachineLearning, /r/OpenAI, /r/ClaudeAI, /r/StableDiffusion, /r/ChatGPT, /r/ChatGPTCoding, /r/aivideo, /r/aivideo

• Why You Need To Constantly Clear Claude Codes Context Window (Activity: 166): The post highlights the necessity of regularly clearing the context window when using coding agents like Claude to maintain optimal performance. It notes that performance degrades significantly when the context window exceeds 40% of its capacity due to the quadratic nature of LLM attention, which increases computational demands and introduces noise. The recommended practice is to avoid accumulating context and instead persist it by using a ‘one session per task’ strategy, ensuring each task starts with a fresh context. More details can be found in the original article. Commenters suggest practical strategies such as using handover prompts to transfer necessary details between sessions, employing the ‘/clear’ command to compact context, and utilizing ‘Plan Mode’ to clear context and execute tasks efficiently. These methods reportedly help avoid the need for a full context window, even for large tasks.

  • Agrippanux suggests using ‘Plan Mode’ as the default setting for Claude, which allows users to clear the context and execute plans without needing a full context window. This approach has been effective for large tasks, such as refactoring, without requiring the entire context to be loaded, thus optimizing performance and resource usage.

  • thurn2 discusses the use of sub-agents in Claude, which involves delegating tasks like creating a git worktree and fixing specific issues. This method allows for parallel execution of tasks and helps in managing complex projects by breaking them down into smaller, manageable tasks, enhancing efficiency and implementation accuracy.

  • Fancy_Excitement6050 notes that as the context window grows, Claude tends to take shortcuts, which can lead to a need for constant reminders to maintain thoroughness. This suggests that managing the context window size is crucial for maintaining the quality of output, and there might be differences in performance between different Claude plans, such as Claude Max.

• Opus fell off? Here’s the workflow that kept my code quality stable (Activity: 133): The post discusses a structured workflow to maintain code quality when using AI models like Opus and Sonnet, which have been perceived as producing “confident wrong” outputs and drifting edits. The workflow emphasizes a loop of specification, ticket creation, execution, and verification. Specifications are detailed with non-goals, user stories, acceptance criteria, edge cases, and more, treated as code to ensure clarity. Tickets are derived from specs, focusing on small, independently mergeable tasks with clear acceptance checks. Execution involves implementing one ticket at a time with constraints to prevent scope drift, and verification involves running tests and confirming acceptance criteria before feeding failures back into the model for correction. This approach aims to maintain discipline and reduce reliance on the model’s “done” signal, ensuring stable and reliable outputs. Commenters agree that the workflow is effective, emphasizing that AI models function more like junior engineers requiring clear specifications and strict feedback loops. This approach shifts effort towards upfront clarity and external verification, making the system more stable and less reliant on the model’s intelligence. Smaller scoped tickets and hard verification are noted as beneficial strategies.

  • GenOS2312 highlights the importance of treating LLMs like junior engineers, emphasizing that a well-specified problem and a strict feedback loop are crucial for reliable outputs. The workflow discussed focuses on upfront clarity and external verification, which stabilizes the system by not relying on the model’s intelligence but rather constraining it to ensure even average runs yield acceptable results.

  • Different-Object5926 notes that smaller scoped tickets combined with hard verification processes significantly improve the stability and reliability of using models like Opus. This approach mitigates the impact of variability in model performance, suggesting that the issue isn’t just ‘unlucky runs’ but rather the need for structured constraints.

  • TheOriginalAcidtech suggests implementing hooks to prevent skipping steps in the workflow, emphasizing that the human interface is often the weakest link. By enforcing strict adherence to the process, the system can better manage user interactions, ensuring that the model and its harness guide the user effectively, rather than relying solely on the model’s capabilities.

• after claude now chatgpt is also uses Grokipedia as source (Activity: 634): The image and accompanying discussion highlight that the latest version of ChatGPT is reportedly using Elon Musk’s Grokipedia as a source. This is significant as it suggests a shift in the data sources used by ChatGPT, potentially affecting the information quality and bias in its responses. The comments reveal a concern about the implications of using Grokipedia, particularly regarding the potential for biased information, as one user notes the risk of models being influenced by ‘right wing’ content. However, it is clarified that Grokipedia is not used as training data but rather as a search tool, which may mitigate some concerns about direct bias in the model’s foundational knowledge.

  • The discussion highlights concerns about language models like Claude and ChatGPT potentially using sources like Grokipedia, which may have biased or unreliable content. This raises questions about the integrity of the information these models provide, especially when they utilize search tools to access real-time data. The implication is that the quality and neutrality of the data sources are crucial for maintaining the accuracy and trustworthiness of AI outputs.

  • There is a debate about the impact of using sources like Grokipedia on the training and performance of language models. Some commenters express concern that incorporating biased or politically skewed sources could lead to the dissemination of misinformation. This reflects broader worries about the influence of data sources on the objectivity and reliability of AI-generated content.

  • The mention of Reddit as a data source for language models suggests a comparison of potential biases. While some argue that Reddit may contain more extreme or varied viewpoints, the underlying issue is the challenge of ensuring that AI models are trained on balanced and factual data. This discussion underscores the importance of curating high-quality datasets to prevent the spread of biased information.

• Giving Claude full access to a laptop (Activity: 795): The post discusses the implementation of giving Claude, an AI model, full access to a laptop, allowing it to autonomously manage a virtual machine (VM) on Ubuntu Google Cloud. The user describes how Claude can be remotely controlled via Discord to build new features and fix bugs, logging major actions with timestamps in a markdown file for memory management. This setup enables the user to learn from Claude’s problem-solving processes and manage workflows effectively, even as a newcomer to programming. One commenter, a desktop support technician, expressed amazement at the implementation, noting its potential impact on job roles, while another sought clarification on the technical specifics of giving Claude full device access.

  • xxxBigMemerxxx describes using Claude to manage a Google Cloud VM running Ubuntu, highlighting its ability to autonomously handle tasks and build features. They mention using Discord for remote requests and bug fixes, and implementing a logging system with markdown and Unicode for tracking changes. This setup allows for a dynamic interaction with Claude, enabling it to learn from errors and maintain a form of short-term memory by logging recent updates.

  • Happy_Requirement187 shares their experience running Claude on an AWS EC2 instance with Ubuntu Linux, accessed via SSH from a Windows laptop. They utilize a Jupyter notebook server for seamless file sharing between the EC2 instance and their local environment, a method recommended by Anthropic. Additionally, they have set up a Ruby on Rails environment with a React frontend for secure file sharing, allowing them to request files via Slack, demonstrating a sophisticated integration of Claude into their workflow.

  • sivadneb inquires about setting up voice control in Linux, indicating a technical challenge in integrating voice commands with Claude. This suggests an interest in expanding the interaction capabilities with Claude beyond text-based commands, potentially enhancing the usability and accessibility of the system.

• CLAUDE.md says ‘MUST use agent’ - Claude ignores it 80% of the time. (Activity: 309): The image and post discuss a technical issue with the CLAUDE.md file, which is supposed to direct the AI, Claude, to use a specific agent for workflow questions. Despite explicit instructions in the file, Claude often defaults to a generic agent, indicating a lack of enforcement in the system. The post suggests that without technical enforcement mechanisms, such as hooks or stronger prompts, instructions are merely suggestions. The image emphasizes these points with highlighted text, suggesting potential solutions like adding enforcement hooks to ensure compliance with the specified workflow. Commenters suggest that the issue may stem from unclear instructions, emphasizing the need for simple and direct commands. They also highlight the importance of implementing technical solutions, such as hooks, to enforce compliance with the CLAUDE.md instructions.

  • Accomplished_Buy9342 suggests using hooks to manage Claude’s behavior, providing a link to a GitHub repository that demonstrates how to block the main chat from performing actions and delegate tasks to a subagent. This approach can help in orchestrating Claude’s actions more effectively, especially when dealing with complex tasks or large contexts.

  • luka5c0m highlights a common issue with Claude when used at scale: as the context grows beyond a few files, the agent may perform unexpected actions. They suggest that instead of relying solely on better prompts, developers should use hooks and dynamic instructions to maintain a sharp and concise context. They also mention working on a dynamic CLAUDE.md file that adapts to the current task, which could help in managing large or nested files effectively.

• My Ralph Wiggum breakdown just got endorsed as the official explainer (Activity: 170): The post discusses a video breakdown of Ralph Wiggum, an autonomous coding loop, which has been endorsed by Geoffrey Huntley as the official explainer. Ralph Wiggum is a bash while loop that calls Claude in headless mode, allowing for autonomous code implementation without context degradation. Key features include avoiding the Anthropic Ralph plugin due to performance issues, using fresh context windows for each iteration, and emphasizing the importance of concise specs to prevent hitting a “dumb zone.” The video link is here. The comments include a link to the endorsement post by Geoffrey Huntley, and general positive feedback on the video, indicating its usefulness and quality.

  • Dennis1451 highlights a practical application of the Ralph Wiggum breakdown, noting the importance of using a well-defined specification and clearing context for optimal results. They mention using ‘auto compact’ without a clear spec initially, which suggests that following the guidelines provided in the breakdown could enhance performance and accuracy.

  • messiah-of-cheese expresses a desire for more scientific validation in the video, particularly regarding the ‘dumb zone’ premise. This indicates a need for empirical evidence or data to support the claims made in the breakdown, which could strengthen its credibility and acceptance among a technical audience.

• [D] ICLR 2026 decision mega thread (Activity: 1589): The post announces the imminent release of ICLR 2026 review decisions, with anticipation heightened due to a previous incident involving OpenReview. The community is preparing for the outcomes, with some users humorously sharing acceptance prediction models based on historical data, such as a simple return uniform(0, 1) > 0.7. This reflects a light-hearted approach to the uncertainty of paper acceptance. The comments reflect a mix of anticipation and humor, with some users expressing frustration over misleading emails from other conferences like ICML, which adds to the tension of awaiting ICLR decisions.

• [D] ICML 2026 - ICML desk-rejected my paper but kept me on as a reviewer. Wow? (Activity: 279): The post highlights a situation where an author’s paper was desk-rejected by ICML 2026, yet they were retained as a reviewer. This reflects a common practice in academic conferences where the author and reviewer pipelines are separate; desk rejections often occur due to scope or formatting issues, while reviewer selection is based on past service or keyword matching. This situation underscores the reliance on unpaid labor in academia, where reviewing is seen as community service, but the feedback loop for authorship and recognition is weak. A notable opinion from the comments suggests that the separation between the author and reviewer roles can feel insulting, as these decisions are made by different parts of the conference organization. It highlights the need for conferences to clarify this separation to avoid personal affronts.

  • AccordingWeight6019 highlights a systemic issue in academic publishing where the processes for desk rejection and reviewer selection are distinct. Desk rejections often occur due to scope or formatting issues, while reviewer selection is based on past service or keyword matching. This separation can lead to feelings of insult among authors, but it’s a structural necessity due to the different roles and responsibilities within the publication process. The comment suggests that conferences should improve transparency about these processes to mitigate personal feelings of rejection.

  • mocny-chlapik points out that the responsibility for a desk rejection often lies with the author, particularly if it results from not following submission guidelines. The comment implies that submitting a paper, even if desk rejected, obligates the author to fulfill reviewer duties, as the submission process involves volunteer time and resources. This highlights the importance of adhering to submission instructions to avoid unnecessary strain on the peer review system.

• [R] Appealing ICLR 2026 AC Decisions... (Activity: 138): The post discusses a situation where an author received mixed reviews for a paper submitted to ICLR 2026, with scores of 4(3)/6(4)/6(4)/6(4). The author invested significant resources, including $1.6k on new experiments and added 20+ pages of theory, to address reviewer concerns. Despite these efforts, the metareview cited “outstanding concerns” that the author believes were addressed, raising questions about the review process’s fairness and accuracy. The author is seeking advice on appealing the decision, expressing frustration that improvements were seemingly ignored. Commenters generally agree that appealing decisions at conferences like ICLR is not feasible, attributing outcomes to luck and the subjective nature of reviews. Some suggest that the meta-review process can be inconsistent, with one commenter noting that meta-reviewers sometimes act as an additional critical reviewer, potentially skewing outcomes.

  • tedd235 discusses the variability in paper acceptance at conferences, suggesting that some PhD students might reject papers to improve their own odds, making the process feel like a ‘coin flip’. They note that if other reviewers provide higher scores, the Area Chair (AC) might consider this in their decision, indicating a potential for subjective bias in the review process.

  • Fantastic-Nerve-4056 shares an experience from AAMAS where despite receiving scores of 6 and 8 from reviewers, the Meta Reviewer recommended rejection with minimal justification, stating it was ‘relevant for other AAMAS session’. This highlights issues with the transparency and accountability of meta-reviewer decisions, which can override individual reviewer scores without detailed explanation.

  • Intrepid_Discount_67 describes a thorough submission process, including extensive theoretical analysis, comprehensive baseline comparisons, and open-sourced code, yet faced non-responsive reviewers and an AC that upheld the initial scores. This underscores challenges in the review process where detailed responses and transparency do not necessarily lead to favorable outcomes.

• [D] ICML new policy: reviewers will be reviewed by meta reviewer. Good policy? (Activity: 151): The image describes a new policy implemented by the International Conference on Machine Learning (ICML) where reviewers will be evaluated by meta-reviewers. The top 25% of reviewers will be recognized as ‘gold reviewers’ and will receive free registration, while the next 25% will be designated as ‘silver reviewers.’ These distinctions are intended to incentivize high-quality reviews and will be considered in financial aid applications. This policy aims to improve the quality of reviews by providing recognition and potential financial benefits to diligent reviewers. Some commenters express skepticism about the effectiveness of this policy, questioning who will oversee the meta-reviewers themselves. Others see it as a positive step, particularly for reviewers from low-resource backgrounds, and suggest further recognition at conferences to encourage quality reviewing.

  • Bitter-Reserve3821 highlights that area chairs have traditionally been responsible for rating reviews, typically using a three-tier system: ‘did not meet expectations’, ‘satisfactory’, or ‘exceeded expectations’. This practice is not new, and there have been ‘Best Reviewer’ awards in the past, sometimes offering incentives like free conference registrations.

  • Unhappy_Craft1906 raises a concern about the feasibility of this policy for top labs with substantial funding, questioning whether they would participate in the review process merely for free registrations. This points to a potential disparity in how different institutions might engage with the policy based on their resources.

  • newperson77777777 suggests an extension of the policy by introducing a visible recognition system, such as a gold or silver star on conference badges, to incentivize quality reviewing. This idea aims to foster a culture of excellence and accountability within the reviewing community.

• Things Get Worse For OpenAI: Consumer groups prep class action suits about their price fixing and supply manipulation through DRAM hoarding. (Activity: 107): OpenAI is facing potential class action lawsuits for allegedly hoarding DRAM to manipulate prices and disadvantage competitors, with accusations of securing nearly 40% of the global DRAM supply. Consumer groups argue this constitutes ‘predatory bidding’ and violates antitrust laws like the Sherman and Clayton Acts. The Free Software Foundation and other groups are pursuing legal remedies, arguing DRAM should be considered an ‘Essential Facility’ due to its critical role in AI, while the FTC and European Commission investigate potential violations of competition laws. The DOJ is also examining whether OpenAI’s ‘Stargate’ project constitutes a ‘monopsony’. Commenters question why only OpenAI is targeted and not other companies like Nvidia, and debate whether buying RAM constitutes price fixing, suggesting that supply issues may not be OpenAI’s fault.

  • Alacritous69 argues that OpenAI’s purchase of RAM does not constitute price fixing, as they are actively using the resources rather than hoarding them. The commenter suggests that the issue lies with suppliers’ inability to meet demand, rather than any manipulative practices by OpenAI.

  • sambull raises a strategic business perspective, suggesting that by purchasing large quantities of RAM, OpenAI could be intentionally limiting resources available to competitors, including those developing at-home language models. This could be seen as a competitive strategy to maintain market dominance.

  • max6296 questions why the focus is solely on OpenAI when Nvidia could also be implicated in similar practices, hinting at a broader industry issue regarding resource allocation and market influence.

• When Ads aren’t enough: OpenAI’s push to Claim a Cut of Customers’ AI Discoveries (Activity: 63): OpenAI is exploring new business models beyond traditional subscriptions and ads, focusing on outcome-based pricing and IP-based agreements. This approach would allow OpenAI to claim a share of the value created when their AI models contribute to profitable outcomes, particularly in enterprise sectors like pharma, scientific research, and energy systems. This strategy aligns OpenAI’s revenue with customer success, aiming to capture more value as AI capabilities expand. OpenAI’s annualized recurring revenue has surged from 2B in 2023 to over 20B in 2025, driven by increased compute scaling. This move is part of a broader trend among AI firms towards value-based pricing, amidst criticism from figures like Elon Musk, who accuses OpenAI of abandoning its nonprofit origins. The community is divided, with some viewing this as a logical evolution of AI monetization, while others criticize it as overly profit-driven. Comparisons are drawn to other industries, suggesting skepticism about the feasibility and fairness of such models.

• CATL, the world’s largest battery maker, launches sodium batteries: extremely durable, stable at –40°C, much cheaper than lithium (5x), safer,10,000 charge cycles, requires no nickel or cobalt... (Activity: 1289): CATL has launched the first mass-produced sodium-ion batteries, offering a cost-effective alternative to lithium-ion with a price of ~$20 per kWh compared to lithium’s ~$100 per kWh. These batteries, part of the Tianxing II range, are designed for microvans and small trucks, featuring an energy density of 175 Wh/kg and a lifespan of over 10,000 cycles, maintaining 90% capacity at -40°C. They utilize a hard carbon electrode and prussian-blue cathode, eliminating the need for nickel or cobalt, and are expected to be scaled up for broader use, including in Europe by 2026. Read more. Some commenters express surprise at the application of sodium batteries in vehicles, expecting them to be used in stationary systems due to weight concerns. Others note the strategic advantage for China in advancing battery technology, contrasting it with perceived setbacks in the US market.

  • The Tianxing II range of sodium batteries by CATL is specifically designed for microvans, light vans, and small trucks, indicating a focus on applications where energy density and weight are less critical compared to cost and durability. This suggests a strategic move to target markets where these factors are prioritized, potentially offering a competitive edge over traditional lithium-ion batteries.

  • The introduction of sodium batteries into vehicles is surprising to some, as it was expected that such technology would first be applied to stationary applications like home energy storage. This is due to the lower energy density of sodium batteries compared to lithium-ion, which makes them less ideal for applications where weight and size are critical factors.

  • There is curiosity about the commercial availability of these sodium batteries, with questions about whether they can be purchased directly for home use or if they will be distributed through third-party vendors. The performance metrics, such as 10,000 charge cycles and operation at -40°C, are impressive and suggest that sodium batteries could rival LiFePO4 in terms of performance, especially given their cost advantage.

• K-Shaped AI Adoption? (Activity: 748): The image highlights a discussion by Kevin Roose on the ‘K-shaped’ adoption of AI technologies, where there is a significant divide between early adopters, particularly in tech hubs like San Francisco, and those who are lagging due to restrictive IT policies. This disparity is creating a cultural and technical divide, with early adopters integrating AI deeply into their workflows, while others struggle to gain access to even basic AI tools. The conversation points to a broader issue of accessibility and the potential for some workers to be left behind in the AI revolution. Commenters note that the disparity in AI adoption is exacerbated by the complexity of the technology, which requires a certain level of expertise to use effectively. Additionally, the high cost of advanced AI tools, such as ‘multi-agent claudeswarm,’ limits access to those with sufficient financial resources, further widening the gap.

  • Setsuiii highlights the technical barrier to effective AI use, noting that current AI technologies require users to have a certain level of expertise to achieve optimal results. This complexity, combined with ongoing ethical debates surrounding AI, may deter widespread adoption. However, those who can navigate these challenges have significant opportunities, although competition is increasing as more technically adept individuals enter the field.

  • Glxblt76 and Gubzs discuss the financial barriers to AI adoption, particularly the high costs associated with advanced AI tools like a ‘multi-agent claudeswarm,’ which can cost around $200 a month. This expense limits access to those with substantial financial resources, such as individuals in tech hubs like San Francisco, while the majority cannot afford such investments.

  • o5mfiHTNsH748KVq shares a personal experience of leaving an enterprise job to join a smaller company, emphasizing the importance of unrestricted access to Large Language Models (LLMs) for maintaining competitiveness in the AI field. They argue that any limitations on LLM access can significantly hinder development speed and career progression, suggesting that smaller companies may offer more flexibility in leveraging AI technologies.

• Former Harvard CS Professor: AI is improving exponentially and will replace most human programmers within 4-15 years. (Activity: 1260): Matt Welsh, a former Harvard CS professor and current Engineering Director at Google, predicts that AI will advance exponentially, potentially replacing most human programmers within 4-15 years. This assertion is based on the rapid improvements in AI capabilities, suggesting a transformative impact on software development and the tech industry. The discussion is available in a YouTube video. One comment highlights the potential for AI to not only replace programmers but also to enable anyone with AI to replicate existing products and services, indicating a broader impact on innovation and competition.

  • The claim that AI will replace most human programmers within 4-15 years is met with skepticism, particularly regarding the use of the term ‘exponential’. Critics argue that the term is often misused, even by experts, to describe growth that may not fit the mathematical definition of exponential growth. This misuse can lead to misunderstandings about the actual pace and nature of AI development.

  • The discussion highlights the potential for AI to disrupt existing products and services if it can indeed replace human programmers. This implies that AI could democratize software development, allowing anyone with access to AI tools to create competitive products, potentially leading to significant shifts in the tech industry landscape.

  • The mention of the speaker’s credentials, specifically as a former Harvard professor and current Engineering Director at Google, adds weight to the prediction. However, some commenters find the emphasis on his past academic title rather than his current industry role to be misleading, suggesting that his current position might provide more relevant insights into AI’s trajectory.

A summary of Summaries of Summaries by gpt-5

1. Funding Frenzy in AI Infrastructure

• Recursive Raises Roar to $4B: Recursive Intelligence is reportedly raising at a $4B valuation to accelerate AI‑driven chip design, creating a closed loop between hardware and models, per Bloomberg: Recursive Intelligence in talks at $4B. The Jan 23, 2026 report highlights a strategy of using AI to shorten design cycles and boost performance for next‑gen accelerators.

  • Engineers framed the pitch as a “self‑improving feedback loop” where better chips train better models that design better chips, amplifying returns on AI‑for‑EDA investment. Community sentiment read this as validation that AI‑native silicon is a core moat, not a sideshow, aligning with recent lab spin‑outs and infra bets.

• Sky Lab Startups Skyrocket: UC Berkeley’s Sky Lab spin‑outs saw major marks: SGLang ~$400M, vLLM ~$800M, and LMArena ~$1.7B, per Alex Dimakis: Sky Lab startup valuations. These January 2026 milestones underscore investor appetite for serving stacks, token‑throughput infra, and benchmarking platforms.

  • Engineers read this as a green light for building on top of vLLM/SGLang primitives and contributing to Arena‑style evals, with one takeaway that practical throughput wins deals. The funding spread also suggests a portfolio thesis across serving, compilers, and eval marketplaces rather than a single-bet strategy.

• Maia Muscles Into Azure: Microsoft’s Maia 200 accelerator went live in Azure, touting 30% better performance per dollar, 216GB HBM3e, and 7TB/s memory bandwidth, per Satya Nadella: Maia 200 in Azure. The platform targets high‑performance inference for large‑scale LLM and multimodal workloads.

  • Builders highlighted that memory topology and bandwidth are the story here, with “30% better perf/$” resonating for cost‑sensitive inference deployments at scale. Teams expect immediate tests against vLLM and SGLang stacks to gauge token latency, context scaling, and multi‑tenant isolation.

2. Kernels, Chips, and Serving: Inference at Warp Speed

• FlashInfer Face‑Off Fires Up MLSys: The MLSys 2026 FlashInfer‑Bench competition challenges teams to build LLM inference kernels for NVIDIA Blackwell GPUs, competing against expert FlashInfer baselines—see MLSys 2026 FlashInfer‑Bench Competition. Tracks emphasize real‑world throughput and correctness under production‑like constraints.

  • Organizers invite agents that “design LLM inference kernels”, pushing program synthesis to meet kernel‑level performance bars. Participants expect aggressive focus on GEMM, KV‑cache motion, and scheduler tactics aligned with Blackwell’s memory hierarchy.

• GPU‑64 Gets Gains with KV‑Cache CAM: A new inference‑only architecture, GPU‑64, introduces a hardware KV‑Cache via on‑chip CAM, claiming 4× faster inference at 75W and reducing memory lookup from O(N) → O(1), per GPU‑64 (Zenodo) with RTL/emulator at gpu64‑inference (GitHub). The design targets LLM‑heavy workloads with KV bottlenecks.

  • Developers flagged the CAM‑based cache as a bold bet on associative search for token histories, noting portability implications for Flash‑style attention and speculative decoding. Discussion centered on whether future ISA/driver stacks can expose these gains without bespoke compilers.

• Cornserve Cuts Tail Latency: Cornserve presents an online serving system for Any‑to‑Any multimodal models that optimizes deployment plans across encoders, LLMs, and DiTs, per Cornserve (arXiv), with an overview talk at Cornserve: Easy, Fast and Scalable Multimodal AI (YouTube). The paper reports throughput gains and tail‑latency reductions under heterogeneous pipelines.

  • Infra engineers liked its planner‑driven scheduling for encoder/decoder mixes and saw it as complementary to vLLM for multimodal graphs. The big open question: standardizing budgeted reasoning and co‑scheduling across text, vision, and diffusion stages without over‑tokenizing control messages.

3. New Multimodal and Coding Models Land in LM Arena

• WAN 2.6 Walks In (With Upload Woes): LM Arena added wan2.6‑t2i (text‑to‑image) and wan2.6‑image (image edit) to the image arena: LM Arena — Image Chat. Users noted wan2.6‑image requires an uploaded image and that wan2.6‑t2i currently lacks image‑upload support.

  • Staff acknowledged the upload gap and are working to enable image uploads for wan2.6‑t2i. Builders suggested testing edit pipelines where masking, prompt strength, and seed control align with Arena scoring to benchmark edit fidelity.

• Devstral Duels and Text Titans: The Code Arena now features devstral‑2 for head‑to‑head comparisons—see LM Arena — Code Arena Direct Battle. On the text side, qwen3‑max‑thinking and molmo‑2‑8b joined the lineup: LM Arena — Text Arena.

  • Engineers are probing reasoning traces and tool‑using prompts to stress code synthesis and refactor quality under tight token budgets. Early chatter favored task‑specific evaluations (e.g., SWE‑style bug‑fix vs. ground‑up implementation) to surface model deltas.

• Hunyuan Hits the Leaderboard: Tencent’s Hunyuan‑Image‑3.0‑Instruct ranks #7 on LM Arena’s image‑edit board—see LM Arena — Image Edit Leaderboard—after a launch post: Tencent Hunyuan announces HunyuanImage 3.0‑Instruct. The model touts an 80B MoE, Native CoT, and MixGRPO for tighter intent alignment.

  • Creators emphasized edit controllability and multi‑image fusion, while evaluators asked for masking robustness, text fidelity, and artifact rates under compositional prompts. Teams plan to pit it against WAN 2.6 variants using the Arena’s standardized edit tasks.

4. Safety, Reliability, and Hallucination Hardening

• Clamp the Chaos: Layer‑Native Safety: Layer‑Native Safety Clamping proposes learning activation‑space harm directions and clamping them to block jailbreaks, with a 10K‑pair dataset at Pacific‑Prime/safety_dataset (HF) and the paper on Zenodo. Authors argue in‑model clamping can’t be bypassed via prompt manipulation.

  • Red‑teamers liked the idea of activation‑level controls versus brittle prompt filters, but pressed for tests against tool‑use and multi‑turn attacks. Expect follow‑ups measuring side effects on helpfulness, coding accuracy, and false positives under adversarial prompting.

• Symbolic Sanity Checks Stop Slip‑Ups: Hybrid approaches check logical consistency for math/code/simple facts, as shown in Consistency Checking for LLMs (arXiv:2409.13724), while broader consistency remains tough per Scaling Consistency Beyond Formal Domains (arXiv:2507.10624). Eleuther discussions framed this as practical hallucination reduction via symbolic/deductive layers.

  • Builders reported wins when pairing symbolic checkers with tool‑augmented prompts, cautioning that coverage gaps appear outside formal domains. The consensus: start with code/math guardrails, then expand to factual QA with curated KBs and provenance scoring.

5. Agent Tooling and Reasoning Workflows Mature

• Levante Leads with MCP‑Native Workspace: Levante launched an open‑source MCP‑native AI workspace for local models (e.g., Ollama) with a modular UI—download at Levante. Engineers highlighted easier tool wiring, local privacy, and composable panes for rapid agent iteration.

  • Early users framed it as a practical hub for tool‑calling and filesystem ops without cloud dependence. Teams plan to benchmark context bloat and tool discoverability patterns versus conventional agent shells.

• RLM Riffs: AsyncReview + Skills Pack: AsyncFuncAI open‑sourced AsyncReview, a DSPy RLM code‑review agent at AsyncReview (GitHub), and a skills kit landed on npm as @unravel‑tech/rlm‑skills. This pairs reasoning‑first prompting with drop‑in skills to extend models.

  • Contributors reported smoother trace inspection and optimizer‑guided prompt tuning for multi‑step modules. One practitioner noted that rejecting premature answers in the metric is key for reliable RLM fine‑tuning.

• Agents Auto‑Assemble a Browser Engine: FastRender—a browser rendering engine—was built using 2,000 AI coding agents, documented by Simon Willison in FastRender: built by 2,000 agents. The project demonstrates task decomposition, verification, and orchestration at non‑trivial software scale.

  • Engineers debated handoff granularity and spec‑to‑test loops needed to keep multi‑agent pipelines from drifting. The case study strengthens the argument that agentic coding can target complex infra when coupled with strict eval harnesses and artifact gating.

First off, a few major announcements:

• Yi Tay returns to the podcast today! If you enjoyed his insights on being a 10,000x Yolo Researcher, then you’ll love our catchup on Gemini’s IMO Gold and more.

• Welcome to the 80,000 AINews subscribers! We are doubling down on Substack, making Latent Space the one email subscription to keep up on a daily (AINews), weekly (LS Pod), and broader (LS Essays) basis. You can opt-in/out as needed.

• Latent Space is also turning into a podcast network! Our first new show, focused on AI For Science, will launch next week, inspired by our Chan-Zuckerberg pod! We also have a new physical space/studio at Kernel in SF!

• AI Engineer is growing! The first official AIE Europe is coming to London April 8-10, and both Super Early Bird tix and Speaker CFPs first deadline is Jan 31st.

• We are hiring across all platforms: AIE Head of Operations, AIE Marketing & Community Manager, LS Researcher/Writer (or occasional Guest Writer!)

At every AIE conference I run, I give myself the challenge of doing the shortest keynote for a “State of the Nation” type address. For AIE Code, I declared War on Slop.

This was partially inspired by seeing the motivating factors behind products like Codemaps and Review, and partially a reaction to the “Canadian Girlfriend AI Coding” performative productivity porn that took over the timeline:

I didn’t expect Slop to then be picked up as word of the year by Merriam-Webster and the Economist and the American Dialect Society1, but clearly the theme of fighting Slop is resonating as generative AI rips the authenticity from creation, piles up open source maintainer workload, fills up academic conference submissions, taking up 20% of YouTube videos (earning millions per year) and running entire Instagram channels and the Apple App Store, slowly deadening the Internet.

There are a few gut reactions to Slop that are reasonable:

• simply reject all AI-generated content, perhaps making exception for AI-assisted, perhaps requiring proof of human review

• severely cut back on quantity to message and deliver quality

• just eat the steak and learn to enjoy the slop machine

However, they feel variously dissatisfying in their general defeatism and pessimistic view of human taste being scalable.

First off, I’ll reiterate the obvious point that plenty of Slop has been made by humans, both well-meaning and cynical, for centuries — AI does not have a monopoly on making Slop. However, AI does make it easy to scale thoughtless output and it is harder to signal intent, effort and quality — an emphasis of my recent writing retreat. If you creatively/skillfully wield AI as "a new brush”, then AI Kino is very possible.

But more importantly, if your solution to AI slop basically means you cut back on your own human output, that doesn’t solve the fact that AI slop will continue to far outpace human output, and therefore simply overwhelm you and your community by sheer brute force and our own recsys’ inability to keep up with our tastes. Humanity slowly starves to death gorging on its own excrement spewed from artificial intestines.

The most important problem in media now is scaling without slop, period, whether or not AI generated or entirely human-origin. Actually I lied, it’s not just a media problem: any act of creation and exercise of free will is is subject to the quality-quantity tradeoff, whether you’re building a new product or texting your adult kids.

Any experienced creator will tell you that there can be very little relationship between effort and result - the thing you slaved on for a month gets overdone and incoherent, whereas the throwaway tweet of frustration done in 3 seconds gets viral and used as an example for decades to come. Sometimes -you- can’t tell what’s slop, sometimes one man’s slop is another man’s kino.

The central problem to solve is the changing the slope of slop, not giving up on humans.

This is the problem I am working on now for both AI Engineer and Latent Space. The case studies of increasing quantity AND quality have been fascinating.

AI Engineer grew from one event a year in 2023 and 2024 to 4 (AIE NYC, AIE WF, AIE Paris, AIE CODE) in 2025, and there will be at least 7 AIEs around the world in 2026.

Viewership did not just 3x, but more than 10x’ed:

And anecdotally, quality has also improved:

I remember a former 2024 speaker who is known for speaking his mind, and being privately pretty cynical about AIE, in 2025 coming back to me to say that conversations and talks felt a lot more authentic and serious this time around.

I say this not as a brag, but I think there are a lot of conscious and subconscious decisions made that contribute toward building a leading IRL and online community for serious, grounded AI discussion without resorting to fearmongering or hyperbole.

Slop has definitely crept in, but here we have kept closer to “Scaling Law X” than Scaling Law 0 and I’m happy about that.

Latent Space (now joined with AINews) has been a tougher beast. The value of media cycles often fit to Sonal Choksi’s McClusky Curve — flanked on the left by TBPN, and on the right by Dwarkesh. We did AMAZING as a podcast - regularly in the 30-50th ranks of the overall US Tech podcasts in Apple, and considered by many to be a top 3 AI podcast, being featured on the GPT5 launch, and featuring conversations with Greg Brockman and Fei-Fei Li and Mark Zuckerberg and Noam Brown and Bret Taylor and Chris Lattner and the Claude Code team and more household names to come.

While our YouTube numbers are still wanting (pls like and subscribe! we started YouTube way, way too late in this game), audio subscribers are still incredibly strong, and Flightcast estimates us at about 1.5m all time unique listeners now.

But we fell behind on writing, evidenced by 2025’s Agent Labs Thesis far underperforming the AI Engineer Reading List (which we’ve continued to maintain, but slowly left behind as less and less frontier knowledge is published).

There’s only so much that one person can do. I am very inspired by creative networks that build a platform for multiple people to shine and contribute their talents, like Dropout TV and Morning Brew and SemiAnalysis. This is part of the movement that a16z has termed New Media, and perhaps unlike most on that list, I want to try to build a platform clearly bigger than me and hopefully bigger than the some of its parts.

The game plan for 2026’s Latent Space is threefold:

• More shows, more hosts: The 1-2 hour interview podcast format is cool but Alessio and I can only do so much. We’ll be doing more education (the booming area of AI for Science is our next focus, with new hosts), but also entertainment (AI can be fun! and AI people are REALLY FUN). Most of our new investments will go into more YouTube and video-native formats.

• More formats: AINews joining Latent Space gives you a place for regular daily updates and commentary. TBA on where the future of AI News should be beyond just a newsletter… but ideas are welcome

• More writing: Writing is thinking, and good thinking powers everything great we have done with Latent Space. We will double down on this and, if you’re keen to help, join us as a researcher or writer full time, or even apply to guest post (we have always done very well serving as editors for guest posts)

The common thread among AIE and LS is that we have to curate very well, and then scale one person’s curation to many, hopefully by crowdsourcing but also taking audience/community feedback very seriously. The most visible output of this is “saying no a lot”, and this is a lonely job saying no to friends and to sponsors, but even that is too broad a brush. There is no “school for curators”, no Maven or Udemy course for curating. You have to have a point of view, you have to know your audience better than they know themselves, and you have to have the freedom to make mistakes and survive making experiments with a high failure rate. Fortunately, we have enough backing and support (read: donations, partially from generous investors (who have been explicitly told we will never sell or sell out), but operating expenses actually come from Substack subscribers! thank you!) that we don’t have to couple this with any sort of business model.

We just want to Make Good Shit at Scale.

Welcome to 2026. It’s gonna be a fun ride.

where it was a runner-up last year, and coined 2 years ago perhaps by @deepfates

From shipping Gemini Deep Think and IMO Gold to launching the Reasoning and AGI team in Singapore, Yi Tay has spent the last 18 months living through the full arc of Google DeepMind’s pivot from architecture research to RL-driven reasoning—watching his team go from a dozen researchers to 300+, training models that solve International Math Olympiad problems in a live competition, and building the infrastructure to scale deep thinking across every domain, and driving Gemini to the top of the leaderboards across every category. Yi Returns to dig into the inside story of the IMO effort and more!

We discuss:

• Yi’s path: Brain → Reka → Google DeepMind → Reasoning and AGI team Singapore, leading model training for Gemini Deep Think and IMO Gold

• The IMO Gold story: four co-captains (Yi in Singapore, Jonathan in London, Jordan in Mountain View, and Tong leading the overall effort), training the checkpoint in ~1 week, live competition in Australia with professors punching in problems as they came out, and the tension of not knowing if they’d hit Gold until the human scores came in (because the Gold threshold is a percentile, not a fixed number)

• Why they threw away AlphaProof: “If one model can’t do it, can we get to AGI?” The decision to abandon symbolic systems and bet on end-to-end Gemini with RL was bold and non-consensus

• On-policy vs. off-policy RL: off-policy is imitation learning (copying someone else’s trajectory), on-policy is the model generating its own outputs, getting rewarded, and training on its own experience—”humans learn by making mistakes, not by copying”

• Why self-consistency and parallel thinking are fundamental: sampling multiple times, majority voting, LM judges, and internal verification are all forms of self-consistency that unlock reasoning beyond single-shot inference

• The data efficiency frontier: humans learn from 8 orders of magnitude less data than models, so where’s the bug? Is it the architecture, the learning algorithm, backprop, off-policyness, or something else?

• Three schools of thought on world models: (1) Genie/spatial intelligence (video-based world models), (2) Yann LeCun’s JEPA + FAIR’s code world models (modeling internal execution state), (3) the amorphous “resolution of possible worlds” paradigm (curve-fitting to find the world model that best explains the data)

• Why AI coding crossed the threshold: Yi now runs a job, gets a bug, pastes it into Gemini, and relaunches without even reading the fix—”the model is better than me at this”

• The Pokémon benchmark: can models complete Pokédex by searching the web, synthesizing guides, and applying knowledge in a visual game state? “Efficient search of novel idea space is interesting, but we’re not even at the point where models can consistently apply knowledge they look up”

• DSI and generative retrieval: re-imagining search as predicting document identifiers with semantic tokens, now deployed at YouTube (symmetric IDs for RecSys) and Spotify

• Why RecSys and IR feel like a different universe: “modeling dynamics are strange, like gravity is different—you hit the shuttlecock and hear glass shatter, cause and effect are too far apart”

• The closed lab advantage is increasing: the gap between frontier labs and open source is growing because ideas compound over time, and researchers keep finding new tricks that play well with everything built before

• Why ideas still matter: “the last five years weren’t just blind scaling—transformers, pre-training, RL, self-consistency, all had to play well together to get us here”

• Gemini Singapore: hiring for RL and reasoning researchers, looking for track record in RL or exceptional achievement in coding competitions, and building a small, talent-dense team close to the frontier

—

Yi Tay

• Google DeepMind: https://deepmind.google

• X: https://x.com/YiTayML

00:00:00 Introduction: Returning to Google DeepMind and the Singapore AGI Team
00:04:52 The Philosophy of On-Policy RL: Learning from Your Own Mistakes
00:12:00 IMO Gold Medal: The Journey from AlphaProof to End-to-End Gemini
00:21:33 Training IMO Cat: Four Captains Across Three Time Zones
00:26:19 Pokemon and Long-Horizon Reasoning: Beyond Academic Benchmarks
00:36:29 AI Coding Assistants: From Lazy to Actually Useful
00:32:59 Reasoning, Chain of Thought, and Latent Thinking
00:44:46 Is Attention All You Need? Architecture, Learning, and the Local Minima
00:55:04 Data Efficiency and World Models: The Next Frontier
01:08:12 DSI and Generative Retrieval: Reimagining Search with Semantic IDs
01:17:59 Building GDM Singapore: Geography, Talent, and the Symposium
01:24:18 Hiring Philosophy: High Stats, Research Taste, and Student Budgets
01:28:49 Health, HRV, and Research Performance: The 23kg Journey

From building internal AI labs to becoming CTO of Brex, James Reggio has helped lead one of the most disciplined AI transformations inside a real financial institution where compliance, auditability, and customer trust actually matter.

We sat down with Reggio to unpack Brex’s three-pillar AI strategy (corporate, operational, and product AI) [https://www.brex.com/journal/brex-ai-native-operations], how SOP-driven agents beat overengineered RL in ops, why Brex lets employees “build their own AI stack” instead of picking winners [https://www.conductorone.com/customers/brex/], and how a small, founder-heavy AI team is shipping production agents to 40,000+ companies. Reggio also goes deep on Brex’s multi-agent “network” architecture, evals for multi-turn systems, agentic coding’s second-order effects on codebase understanding, and why the future of finance software looks less like dashboards and more like executive assistants coordinating specialist agents behind the scenes.

We discuss:

• Brex’s three-pillar AI strategy: corporate AI for 10x employee workflows, operational AI for cost and compliance leverage, and product AI that lets customers justify Brex as part of their AI strategy to the board

• Why SOP-driven agents beat overengineered RL in finance ops, and how breaking work into auditable, repeatable steps unlocked faster automation in KYC, underwriting, fraud, and disputes

• Building an internal AI platform early: LLM gateways, prompt/version management, evals, cost observability, and why platform work quietly became the force multiplier behind everything else

• Multi-agent “networks” vs single-agent tools: why Brex’s EA-style assistant coordinates specialist agents (policy, travel, reimbursements) through multi-turn conversations instead of one-shot tool calls

• The audit agent pattern: separating detection, judgment, and follow-up into different agents to reduce false negatives without overwhelming finance teams

• Centralized AI teams without resentment: how Brex avoided “AI envy” by tying work to business impact and letting anyone transfer in if they cared deeply enough

• Letting employees build their own AI stack: ChatGPT vs Claude vs Gemini, Cursor vs Windsurf, and why Brex refuses to pick winners in fast-moving tool races

• Measuring adoption without vanity metrics: why “% of code written by AI” is the wrong KPI and what second-order effects (slop, drift, code ownership) actually matter

• Evals in the real world: regression tests from ops QA, LLM-as-judge for multi-turn agents, and why integration-style evals break faster than you expect

• Teaching AI fluency at scale: the user → advocate → builder → native framework, ops-led training, spot bonuses, and avoiding fear-based adoption

• Re-interviewing the entire engineering org: using agentic coding interviews internally to force hands-on skill upgrades without formal performance scoring

• Headcount in the age of agents: why Brex grew the business without growing engineering, and why AI amplifies bad architecture as fast as good decisions

• The future of finance software: why dashboards fade, assistants take over, and agent-to-agent collaboration becomes the real UI

—

James Reggio

• X: https://x.com/jamesreggio

• LinkedIn: https://www.linkedin.com/in/jamesreggio/

Where to find Latent Space

• X: https://x.com/latentspacepod

00:00:00 Introduction
00:01:24 From Mobile Engineer to CTO: The Founder's Path
00:03:00 Quitters Welcome: Building a Founder-Friendly Culture
00:05:13 The AI Team Structure: 10-Person Startup Within Brex
00:11:55 Building the Brex Agent Platform: Multi-Agent Networks
00:13:45 Tech Stack Decisions: TypeScript, Mastra, and MCP
00:24:32 Operational AI: Automating Underwriting, KYC, and Fraud
00:16:40 The Brex Assistant: Executive Assistant for Every Employee
00:40:26 Evaluation Strategy: From Simple SOPs to Multi-Turn Evals
00:37:11 Agentic Coding Adoption: Cursor, Windsurf, and the Engineering Interview
00:58:51 AI Fluency Levels: From User to Native
01:09:14 The Audit Agent Network: Finance Team Agents in Action
01:03:33 The Future of Engineering Headcount and AI Leverage

AIE Europe early bird tickets and CFP speaker applications are now open!

In 2024, there was a big question mark around the future of Brex as they cut 20% of staff among stalled growth with the space becoming increasingly competitive. Fast forward to 2025, Brex has accomplished one of the most impressive turnarounds, accelerating sales and passing $500 million in annualized revenue with European expansion in sight, dissipating concerns of a dying business. Among the internal changes that led to this successful turnaround was an aggressive shift towards AI adoption and utilization in every aspect of Brex’s business.

In this episode, we sit down with Brex CTO James Reggio to discuss Brex’s culture and how they facilitated an aggressive shift towards a culture of AI fluency in addition to how Brex modernized their tech stack and apply AI agents to streamline their business.

This Substack is reader-supported. To receive new posts and support my work, consider becoming a free or paid subscriber.

Heading into 2024, Brex showed natural signs of a startup scaling up as they continued increasing headcount and expanded org layers to accommodate. However, Brex realized the rate they were scaling the company didn’t match their speed of execution. Addressing this scary truth head on, Brex laid off 282 people (~20% of the company), flattened their org structure, and reduced layers of management with the goal of having leaders “operate closer to the metal” and be less siloed with only a few important cross-company priorities.

Brex realized that speed and execution shouldn’t come at the cost of increasing headcount. Even as Brex scaled up, it can still operate similar to a startup with high-velocity.

In March 2024, Franceschi described this as Brex 3.0: a new operating model designed to increase intensity and execution quality by making fewer bets, consolidating to one roadmap, and maintaining a release cadence. They doubled down on fostering a high trust and high performance team by promoting from within, bringing employees back in-person to work, and expecting leaders to operate at all levels.

Brex’s team now consists of ~300 engineers and ~350 across EPD (Engineering, Product, & Design). Their team is split across product domains in addition to infrastructure. Their core product domains are the following:

• Corporate Card

• Corporate Bank Account

• Expense Management

• Travel

• Accounting

In addition to these product teams, Brex has formed smaller specialized teams like their most recent team of ~10 focused on LLMs. This has been one of their key initiatives to keep up with and innovate with AI progress. This team was actually formed by the Brex team asking themselves, “What would a company that was founded today to disrupt Brex look like?” and the answers to this question were used to decide what to work on. Currently the team is still very lean with a mix of talented AI native 20 year olds who grew up with the tech and older more experienced staff engineers who are veterans in the industry and space that Brex operates in. This team operates like a startup working 996 (9AM to 9PM, 6 days a week) and growing the team very slowly like a pre-seed startup (i.e. only hire or add people to the team when absolutely necessary)

This specialized team isn’t the only one using AI as Brex has engineering wide adoption for AI tools and coding agents. Oddly enough, the top Cursor user is actually an engineering manager. Brex strongly encourages employees to use AI tools and software that will boost their performance.

This sentiment is echoed by their bold decision to not pick winners between foundational model providers or agentic coding tools, etc. What Brex does instead is to procure a small number of seats of multiple solutions and give employees the ability to pick what they want to use through ConductorOne. For example, an employee can get a ChatGPT, Claude, and Gemini license to build their own stack or even get credits to try Cursor, Windsurf, or Claude Code.

This push for AI adoption doesn’t stop at engineering. Camilla Matias, Brex’s COO, has pushed aggressively to help every member of the operations organization to start rethinking their role as people who are building prompts & evals to become more AI native.

To make it even easier and accessible for non-technical employees, Brex’s Agent Platform is built in Retool which has enabled the operations org to do prompt refinement themselves instead of needing engineers to do it.

While many startups and larger companies discuss how to use AI internally, Brex actually executed on it by making the barrier to entry for any AI tool very low instead of sending a company wide email to “use more AI”.

Brex has also implemented a tangible and definable approach for AI adoption with their org-wide AI training program & AI Fluency Levels which are below:

At the end of the AI training program, every employee must categorize themselves into one of the four groups which will then help managers assign them to relevant projects to upskill their AI fluency. These fluency levels are also used in quarterly assessment to gauge their progress. While this approach may sound excessive, it is not enforced as a metric to fire employees but rather as a tangible way to encourage current employees to utilize powerful AI tools available today.

Brex’s success in upskilling their employees’ AI competence comes down to two things they executed very well on: 1. Make it extremely easy for employees to use (i.e. Retool platform) and adopt AI tools (i.e. open access to any tool through ConductorOne) 2. Implementing a forcing function that aligns employee incentives (i.e. promotions, performance reviews, etc.) with AI fluency.

Even the hiring process has been adapted to align with Brex’s culture of being AI fluent. Previously, Brex had a straightforward coding & system design question for potential hires but that has been revamped to a project you work on-site that requires agentic coding or else it’s impossible to finish. Candidates are then evaluated on the candidates’s fluency and competence with AI tools, how the candidate works, whether the candidate understands the AI generated code, etc.

Making Brex’s culture even more clear, when this new AI native interview process was created, every employee in engineering including managers had to go through the interview. It wasn’t to fire or keep score but so that employees would also realize how they could uplevel their skills using AI which eventually pushed their AI fluent culture further.

By keeping a very high bar and holding everyone to the same standard, Brex ensures every employee is treated fairly while not sacrificing on talent density. The standard that new hires are held to are the same ones current employees are held to, establishing a truly meritocratic culture.

Many Brex employees have gone off to start enduring companies. Surprisingly, Brex actually encourages and celebrates it which they’ve leaned into more recently with their “Quitters Welcome” recruiting initiative focused on attracting talent that want to quit and start their own company one day. Brex has had success hiring former and future founders and have doubled down since many of their smaller, specialized teams, like the one focused on LLMs, operate like a startup. They hope to be the best “founder school” with the value proposition that future founders can learn the best by solving interesting problems with instant distribution under Brex. For example, you could build financial AI applications that are instantly deployed to thousands of customers from Fortune 100 to startups, expediting the feedback loop and learning from high quality customers.

Complementing Brex’s aggressive push towards a culture of employees becoming very AI fluent was a culture of trying and adopting the most modern tools and frameworks in the age of AI. Both were critical in Brex’s turnaround and acted like a feedback loop where AI fluency would enable easier adoption for the best modern tools and frameworks which in turn would increase AI fluency.

But this modernization and adaptation didn’t happen all at once. It started in January 2023 when the team created simple internal infrastructure that made it possible to deploy, manage, & evaluate prompts and route to different models. As more models and technologies were launched, Brex kept up with adoption, eventually building out an entire agent platform. This platform is actually powered by the current version of the internal infrastructure built out in 2023.

Fall of 2025, Brex finally introduced their finance agents that learn, reason, and act on your behalf that is part of their Agent Platform below:

Below are some tangible ways Brex’s Agent Platform have streamlined operations:

1. Automating pipeline for evaluating customer applications to get them instantly onboarded.

   1. Previously, human intervention was required for underwriting or KYC (Know Your Customer) but now a bunch of research agents can first handle all the research and then go do the work on behalf of humans to instantly onboard qualified customers.

2. Maintaining thorough, up-to-date, and accurate knowledge about Brex’s business.

   1. Previously a big challenge Brex faced was outdated knowledge being baked into models during pre-training which would lead to very costly hallucinations and incorrect information that would pollute context for agents. As a result, Brex decided to manage this knowledge base so that every detail about the business is up-to-date and correct.

To ensure the quality of their agents and to prevent regressions during future updates, Brex implemented multi-turn evals where an AI agent would basically embody the end user and be given an objective to accomplish. Sometimes they’ll also pre-set an initial preamble to the conversation (i.e. couple turns will be hand written) then begin the eval in order to isolate certain behaviors and make the eval a bit more deterministic.

Surprisingly, Brex’s AI agents are built with Typescript and the Mastra framework, using pgvector & Pinecone as the data stores. Their eventual adoption of Mastra goes back 3 years when Brex built their own internal LLM framework after being dissatisfied with Langchain’s lack of support for key use cases. With the hope to eventually churn off their own framework which needed constant maintenance, Brex was drawn to Mastra because its ergonomics were very similar to the one they built and it also aligned with a key motivating question they asked themselves, “What languages and technology would we use if we started Brex today?”. Brex’s existing backend code is still written in either Elixir or Kotlin, but they are always reevaluating framework and tech choices because the half life of code has declined so much with agentic coding.

Another modern tool in Brex’s stack is the AI code review software Greptile. According to James, what is most impressive about Greptile is their signal to noise ratio of comments left on a PR, “I never regret going through all 65 comments [Greptile] leaves on my diffs because it catches so many things.”

When deciding whether to build or buy, Brex focuses on whether the solution is only possible with the context they have or can be built by others. When it came to CX (customer experience), Sierra already had alignment with Brex’s primary concern: having a UI/UX (i.e. low-code & more workflow oriented) that makes it extremely easy and accessible for operations and CX strategy teams to administer agents.

There are 2 demographics that Brex primarily serves:

1. Finance teams: they interact with agents very specific for their roles

2. Employees of companies that use Brex: goal is for Brex to disappear for employees (i.e. the only thing they have to do is just use the card)

The way Brex believes this problem is best solved is by giving every employee an AI agent(s) that embodies an executive assistant. In the same way that an EA has all the context and can take relevant and correct actions on your behalf, Brex wants AI agents to do that for finance teams.

Brex also views their AI Strategy through 3 Pillars:

1. Corporate AI Strategy: How are we going to adopt and buy AI tooling across the business and in every single function to 10x workflows

2. Operational AI Strategy: How are we going to buy and build solutions that enable Brex to lower their cost of operations as a financial institution (i.e. fraud detection, handle disputes, etc.)

3. Product AI Strategy: How are we going to introduce new features that enable Brex to be part of the Corporate AI strategy for our customers. We want to be a solution where other companies are saying they adopted Brex for their corporate AI strategy

Brex’s Initial attempt to implement this solution of an EA for every employee was the naive approach of having an agent with a variety of tools, using RAG to fetch the appropriate context.

It soon became apparent that the wide range of Brex’s product lines made it extremely difficult for one agent to perform well when being responsible for everything from expense management to finding travel to answering procurement questions.

Other initial attempts that didn’t work were overloading the agent with tools and context switching (i.e. “this conversation seems more about X topic so lets update the prompt to have the relevant topic in context”)

Without these failed attempts, Brex would not have come to their fleshed out final approach of using subagents. In an era when technology and innovation moves very fast, the optimal path to discover the best approach is actually by trying many hypotheses to learn why an approach doesn’t work.

Brex eventually broke the problem down to using many subagents that sit behind an orchestrator. They built this orchestrator themselves since there are no great solutions on the market right now.

What Brex realized was that there is significant value in multi turn conversations from the orchestrator, assistant, and sub-agent vs. a single tool call. The mental model they use is an agent org chart: sub-agents are specialists and they are conversing with each other so if more context is required, the sub-agents pass it along and then ask the user. (like how a human org chart would work)

Among the many investments and experiments Brex has tried, one expensive initiative that didn’t work at all was using RL for credit decisions & underwriting. Initially, Brex thought that it would be the proper approach to building a model that effectively decides like a human underwriter, but after working with outside experts and investing significant time, the end performance was actually inferior to a simple web research agent.

What Brex realized was that in operations, the thing that matters most is being able to granularly break down problems and form SOPs (Standard Operating Procedure) that humans can repeatedly follow. These repeatable processes can then be audited and done in a compliant manner. Since then, Brex has seen great success using simpler approaches with LLMs for many ops use cases but wouldn’t have known its effectiveness without trying alternative approaches.

• X: https://x.com/jamesreggio

• LinkedIn: https://www.linkedin.com/in/jamesreggio

• Company Website: https://www.brex.com

• 00:00:00 Introduction

• 00:01:24 From Mobile Engineer to CTO: The Founder's Path

• 00:03:00 Quitters Welcome: Building a Founder-Friendly Culture

• 00:05:13 The AI Team Structure: 10-Person Startup Within Brex

• 00:11:55 Building the Brex Agent Platform: Multi-Agent Networks

• 00:13:45 Tech Stack Decisions: TypeScript, Mastra, and MCP

• 00:24:32 Operational AI: Automating Underwriting, KYC, and Fraud

• 00:16:40 The Brex Assistant: Executive Assistant for Every Employee

• 00:40:26 Evaluation Strategy: From Simple SOPs to Multi-Turn Evals

• 00:37:11 Agentic Coding Adoption: Cursor, Windsurf, and the Engineering Interview

• 00:58:51 AI Fluency Levels: From User to Native

• 01:09:14 The Audit Agent Network: Finance Team Agents in Action

• 01:03:33 The Future of Engineering Headcount and AI Leverage

swyx/Alessio [00:00:05]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, founder of Kernel Labs, and I’m joined by Swyx, editor of Latent Space. Hey, hey, hey, and we’re here with Jason Ruggiosi, too, at Brex. Welcome.

James [00:00:15]: Hey, thank you for having me. Thanks for visiting from up in Seattle, where I’ve been a little bit. It’s cold up there, huh? Yeah, and we have an atmospheric river hitting the city right now, so a lot of blowing. Yeah, well, yeah, we’re getting the full-on winter effect right now. Well, you’re here to talk about the sort of AI transformation within Brex. There’s a lot of interesting tidbits that we’re going to draw from your article, but also your background. You’ve got a wide array of experience from Stripe to Banter to Convoy, and I think also mostly I’m interested in your journey as one of the rare people that have transitioned from like a mobile engineering leader to a CTO, which I think is also a bit more rare. I used to have this comment in the past where there’s a career ceiling. There are people who work on client-only things where usually they don’t hit CTO, whereas they typically promote the back-end people, the back-end clouding for people to CTO. Yeah, you know, it’s something that I hear fairly frequently because there aren’t that many folks with a front-end background to reach this level of leadership. And it’s exciting for me to be able to represent that group. But I’ll say that even though my resume kind of reflects that I’ve been more on the front end of things, it’s probably more my experience as a founder a couple of times over that. Yeah. I think that actually helped me get to this level of my career, working for somebody else, becoming CTO is very much like a leadership and like general business role as much as it is a technical role. And so I think it was more the skills that I built from starting companies and trying to build those up made me a decent fit and enabled me to get the nod from Pedro to take this on as my predecessor left about two years ago. Yeah. One thing I’m curious, you guys’ commentary, this is a little bit broad, unscheduled, but a lot of startups are bragging about how many ex-founders they have.

swyx/Alessio [00:01:58]: And yes, to some extent you want people with the founder mentality and agency, which is what you did to be your employees and to take initiative in the company, but also I wonder if it’s becoming anti-signal sometimes, I don’t know if you’ve thought about this. I think it’s more about the turn for me, especially when people are hiring ex-founders, it’s like, if you’re truly of the founder gene, it’s kind of hard to just stay somewhere as like an IC for too long. And then it’s like, all right, I joined this thing and then in one year, I’m like, okay, I’m going to do this. I’m back to being a founder. I’m curious for you. Yeah. What was your, I’m sure you thought about leaving and like doing another company and stuff.

James [00:02:34]: In fact, that was, that was the alternative. I was considering even at the time that I got the phone call where they made me the offer to become CTO, I was thinking about leaving to go start a company. And, you know, I think what’s interesting about it, we actually launched sort of like a new recruiting and employee value proposition for Brex a couple months ago called Quitter’s Welcome, where we actually intentionally are leaning into this idea that we have a disproportionate number of folks who go on to become founders or like heads of a department when they leave our company and we celebrate that it’s actually something that I’m very proud of and that means that like we, we welcome in people who want to get a different experience. I think that there’s certainly like a lot of founders who don’t make it, don’t scale their own businesses to this, to the scale that we’ve achieved at Brex, so there’s something to be learned when they come in, and then we’re very happy to like support people on their way out. And so I actually really like hiring former founders. The one value proposition I find that’s most relevant, because a lot of the folks we’re hiring as AI engineers are kind of folks that are either like winding down their companies or considering maybe running AI startup. The thing that resonates the most with them is that we oftentimes can give them problems to solve that are interesting, problems that maybe they even want to want to like build their own startup around, but with instant distribution, right? Like that is the allure is it’s like you can come into this business and build like financial AI applications. And instantly have that deployed to roughly 40,000 customers across, you know, the Fortune 100 down to, you know, tens of thousands of startups. So that’s what is, I think, appealing to founders.

swyx/Alessio [00:04:09]: But the challenge then is making sure that we set them up for success in an environment that still feels a little bit like the startup that they might build themselves versus like something that’s too corporate. Yeah, instead of doing your own company and then coming to you and be like, can I integrate into Brex? Yeah, get all the data. Yeah, exactly. How’s the engineering team structure?

James [00:04:28]: Yeah, so we have about 300 people in engineering, like 350 total across EPD. And for the most part, we structure around our product domains. And so this means that Brex is a corporate card. It’s also a corporate bank account, expense management, travel and accounting. And so we actually have sort of full stack product domains that are roughly like 30, 40 people for each of those that have everything from like the low level infrastructure up to the web and mobile experiences. That’s generally the structure of our engineering organization. And then we have naturally an organization that focuses on infrastructure, security, IT. And then there are two additional centers of excellence that we’ve built that violate that org design, where we’ve felt the need to put more focus or operate slightly differently. And AI is one of those areas where we have another team of just roughly about 10 people who are focused primarily on LLM applications. And we wanted to create a bit of a separation there because the way that we were thinking about this, and this is actually something we did this summer, is we paused and asked ourselves on our AI journey towards infusing our product with AI and generating customer value. We asked ourselves, what would a company that was founded today to disrupt Brex look like? And then we tried to... To basically use the answer to that question to form this team internally. So it’s a little bit off to the side. Ideally, everybody kind of comes up to speed and contributes, you know, LLM features, but we have this sort of off on the side right now in a centralized manner.

swyx/Alessio [00:06:05]: What’s the difference in AI adoption for those teams? So like, are the people on the LLM team like much bigger cursor users, clock code users, or like, do you see similar diffusion?

James [00:06:15]: It’s actually fairly uniform across the entire engineering department. And it’s actually kinda funny, like one of our largest cursor users is actually an engineering manager. So like, and I think that this also just like speaks to our core value of operate at all levels where we want all of our EMs and everybody in leadership to still basically do the job that they’re managing, manage the work. So it actually is, I think the journey of getting everybody into using agentic coding was not sort of exclusive to the AI group. Yeah. In fact, I think this podcast was actually a really good justification for us. I actually set up because I called outreach to Pedro because he tweeted this. I assume this is the center of Rex’s. He says, I started a new company inside Brex to build the future of agentic finance. No BS, just building 996 and pushing production grade agents to 30,000 finance teams, now 40,000. And then he actually has like a little job description, which I think is really interesting. I’ll skip that and go straight to Brex Accelerator, grow 5x and cut burn 99% in the past 18 months. I assume that’s a mix of internal AI automation and other stuff. But basically, I wanted to put some headline numbers up front to impress people before we dig into the details. Yeah, absolutely. And you’re correct. That’s the team that we have, this AI team. What was that? Very young team. Yeah, it’s very young. I mean, it’s been really interesting. The composition of the team is very young, AI native, 20-year-olds who basically grew up with the tech, kind of paired off with more like staff level software engineers that have been at Brex for a little while, who can kind of navigate like the existing code bases and like understand, the product and the customer deeply. Like we’ve formed these really a couple of tight, tight knit pods in the AI org where it’s like three people. Generally somebody who has like more of a product, a customer focused background, that like staff engineer who knows where the skeletons are and then like a much younger, like AI native engineer who can just do things with, with agents that like the rest of us dinosaurs maybe don’t, don’t, can’t either dream of or like, or where are, I think, I think part of it is like sometimes the, the too much experience or too much knowledge of how to solve a problem and actually be an impediment to thinking differently about it and thinking about it from like an AI first lens. But yes, we, we’ve been, we’ve been slowly growing that team just in the same way that like a pre-seed startup, you want to be very, very careful about talent density and like very deliberate, like only hire when you absolutely need it. And so, yeah, at this point it’s just about 10 people. And I think it was probably four or five people. Uh, I think everybody was actually in the photo that was attached to that tweet, uh, when Pedro put that out a couple months ago. Yeah, we’ll put it up. Yeah. Yeah. Yeah. It’s a photo at 1.20 AM in a, on a Friday. Yes.

swyx [00:08:53]: Oh yeah. Yeah.

James [00:08:54]: Cause we, we always do, uh, we always do like Friday, Friday demos and, and like, that’s a time for everybody to get like kind of exec review time. And so, uh, Everybody’s in Seattle? Um, those folks were all in Seattle, uh, but they’re actually geo, geographically distributed. Uh, we have a couple of folks here, a couple in Sao Paulo, a couple in Seattle. Hmm.

swyx/Alessio [00:09:11]: How, at Decibel we have this like AI center of excellence, which are basically the people running these teams across companies. Yep. How do you make the other engineers not feel like you’re not special? I think that’s something that I hear a lot. It’s like, Hey, you know, why aren’t these people working on all the cool LM things? And like, I’m stuck working on, you know, the KYC integration with whatever, you know what I mean? It’s like, how do you build that culture?

James [00:09:32]: You know, it’s interesting. I, I thought that that would be more of a problem, but the benefit of having really optimized our engineering culture around business impact actually causes it to cut in the other direction where for folks, some folks don’t want to work on the AI products because it doesn’t have as much clear, direct, like business impact. Right now, it doesn’t, it doesn’t impact revenues directly. And so I, uh, I think folks for the most part, uh, we’ve, we’ve enabled folks who have a strong desire to work on, on, um, AI products to, to join that team. Like somebody, somebody transferred out of our expense management organization to come over there because they’re really passionate about taking like their knowledge of like policy evaluation and bringing it into the, the AI, uh, uh, team. But for the most part, I think everybody understands like how their work, uh, ladders up and maybe there’s some like friendly rivalry because like the folks who. We say we’re kind of card product, they, they drive 60% of our direct revenue. And so they, now they’re pretty happy with that. And, uh, and they don’t feel like they’re being left out. Uh, and I will also say, um, as you probably saw in this, this piece that we, we, uh, put out with, uh, first round, there is a lot of smaller applications of LLMs peppered throughout all of our product and operations teams. It’s just some of the more novel, like agentic layer that sits on top of Brex that has been put together, like in this, in this sort of isolated team. So it’s not like folks aren’t getting to, to build with LLMs or use LLMs on a daily basis. Yeah.

swyx/Alessio [00:10:55]: Maybe you run people through the Brex agent platform. We’ll put the diagram in the video where you had the LLM gateway, you know, like the whole MCP layer, which you said, David, the creator of MCP right before you. So this is very timely. Um, yeah. How did you start building that? What’s the architecture?

James [00:11:09]: Yeah, the architecture, you know, I, I think simple is, uh, is elegant and we we’ve had basically an LLM gateway and, and, uh, a basic hand rolled platform, uh, from the very early days. In fact, right before being tapped to become CTO, I was leading, uh, like a AI, uh, labs team internally, uh, in the wake of like the announcement of chat GPT, you know, everybody saw this through technology and said, Hey, what are we going to do with it? And so one of the first things that we did, um, I think January, 2023, that would have been, uh, was try to put together some internal infrastructure that made it possible for us to deploy, deploy, manage version and eval prompts, uh, and then be able to manage, uh, like data egress and model. Routing and, uh, have some very basic, like observability and cost monitoring, uh, in an LLM gateway. So that’s, that’s infrastructure that we stood up and it still continues to power a lot of those smaller, uh, more, let’s say like precise applications of LLMs. So like for instance, we’ve, uh, we set up a completely automated, uh, pipeline for, um, evaluating, uh, customer applications to get them onboarded instantly to Brex, which is something that used to require, um, human intervention either for underwriting or KYC. But now. We basically have a series of, of agents and, um, and particularly like research agents that will go and do the work that humans would normally do. And so that’s running on top of this, uh, this hand-rolled, uh, framework. And then for the agents on Brex that we announced in our fall release, which is like this agentic layer that we’re building that sort of sits on top of Brex and can embody workflows that a finance team would normally, uh, hire humans for. We’ve actually, uh, started using Mastro for that as like the kind of. Yeah. Primary, primary framework for, for accelerating us. We actually have built everything in TypeScript, um, which is another like technology choice. That’s, uh, that answers the question of like, what will we do if we started Brex today, but isn’t the case for all of our existing backend code, which is either Kotlin or Elixir. And then we have, uh, we have a mix of PG vector pine cone. And, uh, like, I think what we’ve seen is we’re always, we’re always reevaluating the tech and framework choices as we go, uh, because the half-life of code has declined so significantly with agentic. It’s actually quite, uh, easy or awesome for anyone else to, to kind of try on for size, a variety of different pieces of tech, to, to figure out what is going to be most ergonomic for solving the problem. So Matthew Cuellar new choice and interesting one. Yeah. I mean, I think that the main, the main reason that we adopted Mastro is that it provided the ergonomics that we were actually, uh, that the ergonomics of master are quite similar to the internal, um, LLM framework.

James [00:14:15]: Swyx and Alessio So like we did, I’m trying to remember because this is now ancient history. We evaluated a link chain, turned off of it, built our own thing. And then as we were looking, we kind of want to deprecate this internal framework that we built because at the end of the day, it’s not leveraged for us to maintain that. And Master ended up fitting the bill for the feature set that we were looking for. And I think what’s been interesting is about half of the applications that we’re building right now on the agent layer are running on Mastra. And then the other half are actually still running on like yet another internally developed framework, which is a framework that’s focused more on networks of agents. So sort of multi-agent orchestration versus more like strict, like, you know, single turn or like workflows, which are easier to use, like either Landgraf or Mastra. Tell us about your multi-agent framework. I mean, that’s what are the design considerations and why is this the first we’re hearing about it? Yeah, yeah. So it’s funny. A big, big reason why we haven’t written more about this is that it continues to evolve quite a bit. And I feel like we actually had a blog post that we were going to put out in conjunction with the fall release talking about how we built this. And by the time that we finished, you know, the blog post and had all the package ready, it was already like halfway outdated. And so the way that this has started to emerge is this multi-agent network approach to implementation was when we started to implement it, it was like, you know, when we started out, we were trying to scale up our sort of consumer-grade Brex assistant. So if you think about like Brex and our customers, there’s really like two very broad personas that we serve. We serve members of a finance team who are generally like going to be doing it like in roles like accountant or controller or head of T&E. For those folks, they are going to be interacting with agents that are much more specific to their roles. But then the other broad cohort of users we have are like employees of companies. And so we’re going to look at like our client, our staff, right? And then we’re going to look at the the past year, and then we’re going to look at the companies that have deployed Brex. So you know, you go join a new company, that company uses Brex, you get your Brex card. And our goal for employees is for Brex to completely disappear. Like the best UI UX for Brex is just the card, like every single thing that you have to do in the software beyond just swiping the card is like an opportunity for AI to to eliminate some work for you. And They have an EA and she knows enough about me. She has access to my calendar. My email has all the context on when I’m traveling and for what business purposes. And so she’s basically able to do everything that I would be obligated to do in Brex, be it like booking travel or like doing expense documentation. And so what we wanted to do is we wanted to build like that EA connected to the same data sources and see if we couldn’t simulate that behavior so that, you know, you basically your interface to Brex’s SMS and the card. And when we started building that out, you know, the most naive like architecture for that would be to have an agent with a variety of tools and maybe maybe do some some rag to ensure that it has like appropriate context for the conversation. But what we were finding is that the wide range of different product lines that exist on Brex made it difficult for one like agent to perform well, being responsible for everything from like expense management to finding and booking travel to answering. policy and procurement questions. And so that’s when we started breaking down the problem and into into a variety of sub agents that sit behind an orchestrator. And obviously, this is something that can be implemented using LandGraph or Master even has the notion of these as like network switches and data. But what we found is that it was easier for us when it came to being able to build evals for the system. We kind of just hit the eject button and built our own framework, which is one in which we have agents that are able to. Basically, DM with other agents and have multi turn conversations amongst themselves to coordinate to, to complete a task to, or like to complete an objective. And what’s what’s been nice about that is it means that like, you can have your Brex assistant is like one single, one single like point of contact between you as an employee and the Brex product, and then behind your assistant, if the company has like expense management turn on you have that if they have reimbursements as another agent for that if they’re they have travel. They actually also then facilitates like, our conception here is that, you know, it’s like generally like software encapsulation patterns taking like sort of projected into the agent space. It also makes it easier for us to have like the team that owns and understands travel, like be the ones to go and iterate on that without needing to worry about like regressing the total system, or needing like one team to own every single possible active action you could take as an employee. And I’ll say that, like, I’m still of the mindset that somebody who’s like, you know, I’m not willing to do that anymore, they will build a great framework, and we they have ultimately migrate to it, but or might be us that we ultimately open source this right like but um, but for us, like this is, this has worked out quite well. And like, loo of like a couple other approaches that we tried along the way that just didn’t perform well, which is to overload the the agent with a variety of tools or intellectual like context switching where we try to say, Oh, this conversation looks like it’s more about reimbursement. So let’s like update the prompt with more reimbursement context. Like that was, that was another approach that we kind of did that. that didn’t perform as well as actually having a reimbursement agent that it would collaborate with.

swyx/Alessio [00:19:46]: What about MCPs as sub-agents? Oh, yeah. That’s another pattern.

James [00:19:50]: The key thing there is that there’s actually a lot of value in having multi-turn conversations from the orchestrator or the assistant to the sub-agent, whereas a tool call is basically just one RPC. And so oftentimes what will happen is, let’s say the user reaches out to their RECS assistant and says, hey, how much am I allowed to expense per person for dinner tonight? I’m taking my team out. And your assistant’s going to then reach out to the policy agent. Maybe the policy agent needs to know, in order to answer that question, maybe it needs to know whether this was a customer event, a team event, or whether you’re traveling. And so it may actually send... Instead of... It can’t just answer the question, so it’s going to reply back to the assistant and say, hey, I need you to ask this clarifying question. And so then the assistant will return to the user, ask clarifying question, and they’ll basically have this sort of multi-turn conversation across multiple agents versus it just being encapsulated in a single call and response tool call. And so there are still... All the sub-agents have a ton of tools, but I think of the MCP and tool usage as being the interface to all of our... Conventional imperative systems, not the AI space.

swyx/Alessio [00:21:07]: Yeah, that’s the conversation we were having earlier, whether or not it should be an agent-to-agent call as well. Yeah. Or like, yeah, there should be like a chatback.

James [00:21:15]: Exactly, exactly. And that’s the thing. It’s like, okay, and one of the ways that we actually grafted this into Nastro before we built our own framework was to make every sub-agent a tool. And then the input was just natural language. The output was natural language. And if you needed to have multi-turn... You would basically just put the full, like, prior conversation in as you kept calling the sub-agent as a tool. And it’s just like, at that point, you’re like, okay, the ergonomics are kind of... The framework is fighting me on this. It’s actually helpful for us to basically conceive of it as an org chart. And like, it’s the agent org chart with, you know, my EA is DMing other specialists and having brief conversations to support me as their client. Yep. That was a really good deep dive. Thanks for indulging. I feel like you guys are not afraid to make your own tech, which I think is a competitive advantage. I really like that culture. Maybe we should go a bit breadth first as well. Of course. Because I think we also deep dive a little bit too much in one area. There’s, and we’ll put up the chart, but I’m also very interested in like the sort of internal agent stuff, the operational stuff, and just the general platform scope. So please feel free to just like go into your spiel on it. Yeah, of course. So one of the things that I was trying to do at the beginning of the year... As CTO, you know, I think it really fell to me to articulate what our AI strategy was as a business. You know, every board of director was, you know, or every member of our board is like, hey, what’s your AI strategy? And while we were doing a lot of things, we’d literally go, he’s got it. Well, yeah. Yeah, and if I didn’t, I’d be in trouble. I think he also was counting on me given that I was doing the AI organization before CTO to have... That’s true. But a big part of it was like we were doing a lot with LLMs. It was more like these little one-off features and, you know, hey, like maybe mix in some suggestions here or maybe do a little bit of ops automation over here. But it wasn’t easy to kind of create like a verbal framework of all of these investments. And without that framework, then we weren’t able to like set a vision or a roadmap for investments. So what we did at the beginning of the year is we took everything that was going on, as well as all of our ambitions, all of the good ideas, as well as like the problems we were trying to tackle. As a business this year, throw it all on the table and see if there were some ways to cluster it into a framework that made sense to the business, to our board, to ourselves. And we came up with, I think this is not particularly novel, but it’s helped us quite a bit. We have like three pillars to our AI strategy. We have our corporate AI strategy, which is how are we going to adopt and like buy AI tooling across the business and basically every single function to be able to 10x our workflows. And we have our operations. We have our operational AI strategy, which is how are we going to buy and build solutions that enable us to lower our cost of operations as a financial institution, because I think it’s fairly intuitive. Like financial institutions like ours face a lot of regulatory expectations and there’s just like a high ops burden for running our business. And so it’s sort of like a lot of kind of internal use cases, like being able to do like fraud detection, underwriting, KYC, be able to handle dispute automation on card transactions. Those types of operational investments are our ops AI pillar. And then the final pillar is the product AI pillar, which is like, are we going to introduce new features that enable Brex to be a part of the corporate AI pillar of our customers? It’s like we want to build features and be a solution that somebody else is saying to their board, hey, we adopted Brex and this is part of our corporate AI strategy. And so it’s kind of has this nice little feedback loop and we basically within the company split. You know, did a little bit of divide and conquer where folks in IT and on our people team were more or less spending more of the effort driving on corporate AI, really like looking for making the procurement decisions, like creating a culture of experimentation where we spotlight and incentivize people for trying to sort of improve their personal workflows using AI. And then the pieces that I’ve been more involved in have been operational and product. And we were just talking about products here, which is like the agents on Brex and stuff. But I think that the operational AI. Investments have been some of the most sort of immediately impactful to the business because we have hundreds of people who work in our operations organization. And it’s actually something that differentiates us because our CSAT and the quality of our support and service is very, very high, something we’re very proud of. And so trying to figure out how can we automate a significant portion of this and use LLMs in a way that doesn’t degrade the customer experience and then also kind of addresses, like, what is the future of the roles of the people who we already have working full time for us? So this is where Camilla, our COO, who kind of co-wrote the piece with First Round with me, she’s been leaning really aggressively to help every member of the operations organization start rethinking their role as being not people who kind of execute against an SOP, but are people who are going to, like, build prompts, build evals and, like, become more AI native and, like, the way that they’re going to be used in the future. And so a lot of the engineering we’ve done has been to enable folks, say, in fraud and risk to be able to refine prompts and add additional automation to their workflows. Yeah, and it’s the secret fourth pillar, the platform. Yeah, yeah, exactly. That is the thing that ties it all together exactly, is the platform. And I think what’s been really nice is that even though the platform is kind of a loose term because it consists of a wide... variety of technologies, as I said, like, we haven’t been too religious or dogmatic about everybody needing to be on one particular thing, what we’ve seen is that by making a variety of sort of ergonomic options for building with LOMs available, it, like, really has made it easier for us to make a quick leap forward on operational AI. Like, as soon as we put our mind to it, we said, like, look, no, we want to hit 80% automated acceptance rate for all startup and commercial businesses that apply for Brax. Like, we want a decision within 60 seconds. It’s fully touchless, no humans involved. We were able to break that down and then actually build the agents, build the tools on top of that platform really quickly. And a lot of those tools are the same tools that our product AI agents use as well. I was pretty sold on the Conductor. I don’t know if this is under exactly that bucket, the Conductor One provisioning command. I was like, yep, I want that. Yeah, that was actually, I’d love to talk about that. So that’s actually on the corporate side. And I think that this goes back to maybe another intuitive, but I’d say, like... I think that this is a really bold decision that we made, which is that we’re not going to, we’re not going to try to pick winners in the horse race between the foundational model providers or the agentic coding tools or like basically anywhere where there’s, there’s an active horse race. What we do instead of like trying to pick a single solution is we will procure like a small number of seats, like multiple solutions, and then we’ll give employees the ability to pick whatever one they want to use. And so, for instance, like we allow employees to... basically go to, in Slack and use Conductor One to get a ChatGPT, a Cloud or a Gemini license. And basically you can just like build your own stack where you pick your, you pick your like chat provider as a dev. You can pick, you know, between like Cursor, Windsurf, Cloud Code, Credits, like, and you can basically craft your stack to your preference and easily switch between them. And what that does for us too is when we’re going to, like, obviously we have sort of enterprise agreements in place for all of them for the sake of like the, you know, the privacy. we have some non-training guarantees, but it’s fun because when we go to renew these contracts, we can basically resist the need to like do a wall-to-wall deployment. We can say, Hey, look, like usage trends, our employees are voting with their feet. They’re voting with their dollars and maybe Amur tools and is as hot as it was a year ago. Jason Wonguaiohnaipai, Zephyr Access Group CEO & OEM Partner, Swyx does this give you a dashboard of what people are choosing? Yeah, actually, we look at that. We were looking at that as we’re going into budgeting for next year. I would love to see that. What’s, you know, anything that’s like really up, anything that’s really down? It’s fascinating how different the landscape is every three months. And I think one of the interesting challenges we had early on was getting folks to just like try these tools, try to incorporate like agentic coding. You know, like early on, I say like 12 to 18 months ago now, like get folks to just take the time to try a new workflow. And now at this point, I think what we’re seeing is like, even if, you know, a new model hits the same, like when Codex came out and everybody was like, oh, Codex is better at CodeGen, but it’s a little bit slower. I find fewer folks are like kicking the tires on new things because like they’re just so comfortable with the ergonomics of their current work. And I think that’s a big part of the workflow that, you know, some folks are just like, I want to stick with Cloud Code because I know it now. I’ve been working with it for like nine months, so I don’t need to keep switching. I don’t need, I don’t feel the incessant need to keep trying new things because I’ve gotten, I’m an iPhone person and I’m just like going to stay with an iPhone even, you know, even though there’s some really sexy Android hardware out there.

swyx/Alessio [00:30:37]: Do you have one of the big numbers, like 80% of all of our code is written by AI or, but how do you measure it internally?

James [00:30:44]: Yeah, no, not really. We, I mean, what we do is we’ll... We’ll measure like the attributions on the number of commits that have the like co-authored with, and we pull some of those stats, but I don’t index have like, in fact, I don’t index on those at all. I don’t, and honestly, like I, I don’t know how I, in honest, like honestly calculate that number. Yeah, I agree. Yeah, and so, so I, and the thing that, the thing that we’re really just, you know, we’re at the point now with the, like our AI agentic coding journey where now we’re trying to solve the second order of factors. So like a little bit too much slop, maybe a little not enough. Yeah, exactly. Not enough like rigor and code reviews. We’re trying to, the adoption is there. And now we have to figure out like how to mature in our usage of these tools. So that we know quality or like long-term maintainability doesn’t suffer. As well as like maybe one of the other facets of being able to generate a lot more code more quickly is like the, the drift between team members. As far as like the. Understanding of the, the, the code that’s in their services increases is like everybody’s moving faster and more independently. It, that is another sort of risk that we’re starting to see, like, you know, an incident response where folks don’t know, they don’t know a service as well as they, they used to, because it’s changed so much in the past couple months because everybody’s moving more quickly. Yeah. This has been a major topic for me this year on code-based understanding and slop, because obviously it’s so much easier to generate code, but then now we have to review it and to some extent you can’t. really fight AI with more AI. You can’t just be like, oh, just throw, throw an AI reviewer on their AI code and you solved it. Uh, and so, so you do need to just scale human attention. And I, I think that’s something I’ve been pushing a little bit in terms of like, well, you’re, you’re just going to like every engineer is just going to own more code. Yup. Period. And, uh, and be parachuted in and be expected to ramp up and be, be productive and also fix bugs. And if you’re on, you know, pager duty or whatever to, just because I mean, everyone’s going to try to be more efficient. And you’re supposed to see ROI productivity, because if you don’t, then what’s the whole point of this? Exactly. Exactly. And I, and I think it’s funny, you’re going back to the point of, you know, you could, you could add AI on top to solve the problems that the AI introduces and there, but you just keep you, that’s like an endless chain. Uh, and so. Well, I mean the, the, the, the code rabbits of the world, the graphites of the world would say, yes, actually you can. And so that’s the little bit of the tension there. Yeah. You know, I, I, uh, I’ve been thinking a lot about how the craft of Avengers. Engineering is evolving and, and I will say that I feel further away from being able to predict what, what it looks like than I, I did this past summer when I spent a bunch of time. Um, I actually basically went on leave for a month and joined the, um, joined the, the, the team that, uh, the, the AI team that we were building just to go and build alongside them. I felt like it was really important for me to deeply understand, uh, the problems in the tech. Uh, but, and so that was me. I was, I was, you know, writing, pushing code, um, effective. And, uh, and I, I went through so many different moments of realization of like, oh my God, this is going to change everything to, oh my God, this is just amplifying all the good and the bad in the industry, uh, to, oh my God, engineers are not going to have a job anymore to, you know, it’s like, and so I, I don’t have any, like, I felt like I had all the predictions back then. And at this point now, I’m just very interested to watch the, the phenomenon continue to unfold in front of us. And, uh, I will say. I was chatting with a bunch of really. Right. Uh, you know, college juniors and seniors at a dinner we hosted last night. And, um, all these folks are about to enter the industry, basically having kind of come up in the, the era of agentic development and LLMs. And I asked them like, so what is your workflow when you’re like building, uh, like building a project? Uh, how do you, how do you use agents versus like when you decide you’re going to actually just write code by hand? And I was surprised to hear the consensus was that most people there were using agents to collaborate on. Like building a design document and like collaborating on the architecture of the solution that they want to build, and then they’d be asking it to like emit, uh, you know, a doc or an implementation plan, but then they’ll go and write a lot of the code themselves still. Uh, so it’s a little bit more of the, the, uh, the rubber duck co-architect, uh, uh, use case that was most prevalent in that group. And I, I was very surprised by that. I’m impressed. The kids, the kids are all right. Yeah, I know. No, they still want to, they still want to actually write the code themselves. It’s interesting. Yeah. What we hear from like the Gen Zs.

swyx/Alessio [00:35:18]: That open the end, they, they just YOLO everything into code S and yeah, I would say most of the code I generate is like, yeah, but, but it’s been a lot of time on the doc. It’s curious, like when you’re like younger in your career, it’s like, you don’t really have all the mental models of the different patterns to instruct. I feel like there’s like over reliance, especially if you’re doing the design doc, you know, I, I feel like most of the senior engineers will spend more time on that. It’s like even things like, you know, what column should you index, depending on, you know. What queries we usually run on this table and things like that. It’s hard for any AI to know that. Right. You know, and it’s like, I feel like the, the role of like the more senior engineer should actually be more of this. It’s like spending time teaching the AI and then the AI can teach the junior people in a way. Yeah.

James [00:36:03]: Yeah. And it, it, everything, everything looks like mentorship and management at the end of the day. Right. It’s like, you’re breaking down tasks, you’re, uh, you’re supervising work, you’re giving feedback. Like it’s, you know, it’s basically management. Except that there’s. Agents are really bad at memory still, like they basically have zero memory and, and it’s, it’s, it’s the end of 2025. What’s going on? Yeah.

swyx/Alessio [00:36:26]: Yeah. What’s your internal stack for like, uh, uh, preferences. There’s like, kind of like, you know, explicit preference you can use with, uh, you know, agents that MD and all that stuff. Uh, there’s implicit preference with linter rules and things like that in a way where it’s like, it just happens. You don’t have to tell it. How do you structure that? Oh, no. You’re talking about for agentic coding or memory? Yeah. Just like. Platform. Yeah. Yeah. For like the coding specifically, it’s like, and then we can kind of talk about, you know, the whole Brex platform. Yeah.

James [00:36:53]: Just, just nothing, nothing special. Just a lot of, um, like explicit rules. That MD files. Yeah. And then we have, uh, and we, um, in linting, we still have like traditional linters in place for the couple of different language full chains. And then we’re, we’re, we’re big fans of reptile and we use them for basically all of sort of the, um, smarter than linting, uh, like agentic code review. Uh, that’s been the one solution that we’ve aligned around. And that has served us extremely well. Yeah. Can I go to reptile? Yeah, no, we’re, we’re huge fans there. They’ve built something really impressive. And I think the thing that constantly blows my mind about it is, um, the way that they’re able to just have a really impressive signal to, to noise ratio. Like the, the comments that it leaves are very, very high signal. Uh, like never, I never regret going through all like 65 comments that leaves on my, on my diffs because it catches so many things. Yeah.

swyx/Alessio [00:37:46]: I found the code. X review to be really good. I don’t use code expert code generation, but like the review product is like very good for some reason. Um, I used to have, when I was working in rails, there was like this project called danger systems. Oh yeah. It was kind of like a semantic linter. Exactly. I feel like there should be more of that now. It’s kind of like the rules are one thing, a generation, but I want something in my CI that is like enforce these rules and call out where they’re broken. And then I can just copy paste that in an agent.

James [00:38:12]: But yeah, when we, when we started building this, this new agent, um, code base, like, cause. As we were saying, like we were answering the question, what would you do if you built a, you know, a Brex disruptor today? And it’s like, it wouldn’t be to pick Kotlin and Elixir as the backend. And, uh, and so we actually went with the full like TypeScript stack and we, we were building on all like public interfaces and, um, really trying to make sure that this agent layer was, uh, like arm’s length from, from the, the good and the bad of, of the core of our product. And, um, and one thing, I think what we did early on. Yeah. And I don’t actually know if this is true because again, the team keeps sort of iterating, uh, but we were having good, uh, good luck using, um, cloud code, like in a GitHub action to basically go and do, uh, do more of that danger style like code review. So have a, uh, a prompt for it that went through all of the different facets that were more conceptual versus like rigidly enforceable by a linter, uh, and have it leave a big comment at the end with, um, you know, your conformance to the idiomatic coding patterns of the, of the new repo. I want us to spend some time. You said you wanted to devalve on operational agents, uh, customer support, onboarding, KYC, fraud, delinquent account disputes. Uh, this is, I imagine the bulk of it, of, of the work anywhere where there’s a good story about maybe, um, when you started out, it was, it was going to be this way. And then you discovered through building or through customer contacts that it had to go a different direction. And so that difference in beliefs is something that people can learn from. The thing that immediately comes to mind is that we, uh, we believed at the beginning that using RL for credit decisions would actually be a, like, would be the way that we would end up or like credit and underwriting, like how much of a, of a limit should we give to this business, um, that reinforcement learning would be the way that we would go about, um, building a model that effectively would decision in the way that, um, a human underwriter would. Yeah. Yeah. Yeah. Turns out that it was, we made this big investment. We were working with some outside, uh, like the, like a company, uh, that specializes in this and the performance we ended up getting was inferior to just building a, a, like a web research agent. And so, so I think what, what we took away, what, what has been most evident in operational AI is that in operations, you need to be able to break down problems really granularly and be able to form SOPs. That humans can. Root, repeatedly follow and, and thus can be audited, uh, because so much of, uh, the responsibilities and operations is to, uh, is to have auditable, repeatable processes that help to ensure that we’re operating in a compliant manner. yes. Um, and that actually translates just so cleanly to LLMs that we haven’t needed to use too many sophisticated techniques in, in operational AI. Uh, it’s been a, it’s been relatively simple, like a few tool, uh, like agents or maybe even a lot of them. problems can be solved, which is like a single turn chat completion. And so the fact that we didn’t, well, we did one sort of attempt to over-engineer and use more sophisticated techniques. And we discovered that, in fact, the solutions are a bit more plain and less technically sophisticated. The challenge is really articulating and refining prompts to reflect the execution of the SOP and reflect all the sort of institutional knowledge that isn’t written down so that agents can properly replace like the humans or the contractors we would have making these decisions.

swyx/Alessio [00:41:45]: How do you decide what is worth like spending a lot of time building versus what you think some of these models are just, because some of these tasks are so generic, they’re not really about Brex. Yep. Like you can assume the models will be good at it versus some of them are like very specific to you.

James [00:41:58]: We kind of prioritize like the tasks that are most common for the broadest number of customers. And some of them are fairly... Fairly intuitive, like being able to research a customer to look to assess like legitimacy of the business and whether that business would fit our ideal customer profile for onboarding, because there’s certain types of businesses that we either legally cannot serve or we are not comfortable being able to serve. So that’s the type of really kind of basic research and like a relatively straightforward problem that isn’t hyper Brex specific. The things that are a little bit more specific to us or companies in our sector would be preparing documentation for a network card dispute. Like if you go and dispute a transaction on your personal card, you will provide evidence to your card issuer. The card issuer then has to put together like a three or four page word document that goes to the card network and then eventually goes to the acquiring bank. And all of that is like much more specific to our business. It’s a huge operational overhead for us. And that’s something that we we decided to automate later because it’s not as it’s not on the critical path of like serving the vast number of our customers. Like disputes are expensive, but not very common operational process. And so they’re lower on the stack. And I think we’re getting there right now. But this year has basically been us just kind of like looking at every single process, this kind of stack ranking. And I will say, like the thing that got us started down this path was we wanted to be able to do a lot of things that we didn’t want to be able to do in the first place. We wanted to expand our ideal customer profile to support more business, like a wider variety of commercial businesses, which tend to be businesses that aren’t growing as quickly. So they’re not like tech startups, which have a lot of growth, and they’re not usually like they’re not enterprises, which also tend to have a lot of growth. It’s more like a lawyers, a law firm or a dentist office, these types of like solid businesses that we should be able to serve and underwrite. But the cost to to onboard them and the cost to serve, if you have all all the humans in the loop, make them ROI negative. And so that was the first sort of use case of of AI within our ops ops organization that then led to us really understanding we could automate much more than that. Is this Berks going back into SMBs? Ah, that’s a good question. Yeah, yeah. So never, never let let let that die. You know, we think the way we’ve thought about this is we want to always like offer our product to customers where we believe we have a like an offering that is well suited to the needs of those businesses. And I would say that still for very small businesses, our offering isn’t it’s not built for that. It’s built for it’s built for companies that have some degree of scale. Typically have at least sort of. One person, if not a couple of people in their finance team. So we consider these to be more like the the commercial segment. And so it rhymes with with SB. But our approach back then was was a little bit more naive. And I would say we also we were just going for volumes, like a volume game. There are internal controls were not as strong. We didn’t have as much experience like underwriting those businesses. And so it was really ended up being a case study.

James [00:45:46]: Swyx and Alessio

James [00:46:03]: Swyx and Alessio It depends on the scale of yourself as a business when you use these terms.

swyx/Alessio [00:46:28]: And all of these things are built in the Brax agent platform, like all these automations

James [00:46:32]: that people build? Yes, exactly. In fact, most of the operational AI is running on that original platform that we have. One element of it that I didn’t mention is that most of the UI UX for this platform is built in Retool. And so you can basically go into Retool and there’s a prompt manager, a tool manager, an eval manager, and that’s sort of where much of this was built. And the goal with that was, again, to make it more accessible, more ergonomic to get started, but a secondary effect of having a more visual set of tools for this is it’s enabled members of the app’s organization to go and do prompt refinement themselves. So you don’t need engineers to go and refine the prompts or even test new foundational models when they come out. I think that’s another fun thing. When a new model drops, folks will go into the platform and basically run the evals on the new model and kind of see, can we get better performance here? Or does this have different latency or different cost characteristics? Yeah, you want the domain experts or the people directly using the tool, not the engineers who are somewhat removed from the tool. I do want to highlight to listeners that a lot of the Brax agent platform are just things that every company should have. Basically, problem management system, which we talked about where the domain experts are doing it, multi-model testing, evaluation and benchmarking frameworks, API integrations for automated workflows, NCP-based architecture shared with Brax’s external AI products. This one is obviously very Brax-specific. One thing I did want to highlight that I was semi-impressed by, because nobody, people very few rarely talk about this, is knowledge base for understanding Brax’s business. So do you want to expand on that? Yeah, and this is an area where we’ve only scratched the surface. This year, but a big challenge that we face is that the world knowledge or the knowledge that’s built into the model about what GPT-5 thinks Brax does and how it thinks our business operates is actually quite different from what our business offers today or how our product works. And so we’ve had to work on building a corpus of product documentation, process documentation, and curate this set of information. What do you think, Zach, was the most of your experience building this model? We really needed to have a lot of communication to basically ground a variety of our LLM applications, including like that Brax Assistant, which is like the, you know, the assistant that employees will talk to is like we don’t want it to hallucinate features that we don’t have, or like give wrong information there. And similarly, some of the operational agents need to be grounded on what our ICP is, because if you ask, you know, ChatGPT-5 right now, like, what types of businesses does Brax honor? Thank you so much for having me. Thank you so much.

James [00:50:25]: Thank you so much.

James [00:50:52]: Thank you so much.

James [00:51:20]: Thank you so much.

James [00:51:50]: Thank you so much.

James [00:52:21]: Thank you so much.

James [00:52:59]: Thank you so much.

James [00:53:20]: Thank you so much.

swyx/Alessio [00:54:02]: Thank you so much.

James [00:54:20]: Evals that are blocking because they would indicate like a regression, an unacceptable regression. So these tend to be just accuracy related evals. But then there are others that are more about like tone and coherency and these types of things where they’re more subjective. And we were just looking at those over time as a metric. But the team is actually interesting. I think we’re going to get a big update on like how the team is thinking about evals tomorrow and like our Friday, our Friday review. So it’s, this is an area where I’d say the largest challenge, like the largest change we needed to make and how we were executing sort of as like a lab or an incubator back earlier this year to like where we are now, where we’ve, we’ve shipped and like we’re trying to, to increase the rigor has been around like avoiding regressions and having more and more increasingly robust evals.

swyx/Alessio [00:55:14]: Yeah, I’ve worked with a company called Verisai that does user simulation. And I think like that’s what’s been interesting. Some of these things they just don’t expect, like the customer does not expect the model to do, but they want to track the saturation of the model in a way, if that makes sense. And I feel like most companies know what they don’t want to happen, but it’s almost like they don’t, they cannot quite articulate, oh, I want in the future the model to be able to do this. They can do it today, but I’ll keep running this eval.

James [00:55:41]: That’s actually really, really interesting to me. And I, I’m going to take that away and start thinking about this because. There are, there are going to be certain, I mean, we already seen this where, where users will ask the assistant for help with things that we don’t support yet, or we haven’t implemented yet. It’s like, those are opportunities actually for us to build a, like effectively write a test that’s going to be fail, like failing for, for weeks or months and then eventually we’ll go green, but as a way for us to actually kind of show like the progression of sophistication of the assistant. I really, I really liked that as an idea. Yeah. I wonder how you also catch hallucinations and things that it does. That’s usually the, that’s usually the problem is it, you know, it’ll, it’ll, it’ll pretend like it can assist with something and it’ll, uh, like one thing that is really annoying that has been tough to, um, to prevent is that the, the assistant, because it is used to speaking to other agents, um, that can support it in like accomplishing various tasks. If you ask it to, to help with a task that it thinks it probably should have an agent to, uh, uh, to, to work with, it’ll just hallucinate that it, you know, it’s like, oh yes, I’ll, I will like, you know, I’ll reach out to the finance team on your behalf to, uh, to pass this question along, but it’s not doing anything. There’s like no finance team. There’s no way for it to do that. This is something that comes up a lot. It’s like, would you like me to ask the finance team? And there’s no, there’s no actual tool for that. Do you put guardrails for that? Yeah. Yeah. That was, that was something that we had to, uh, Like a regex? Oh, no, we don’t. I think we’ve been able, we’ve been able to just beat that out of its system with a system prompt. But, uh, but the, we don’t have as many guardrails in place right now, just around a couple of like potential, uh, like things that could get us into trouble. Yeah, really extreme ones. Yeah. I just, yeah, it’s surprising when I, I guess two years ago was first kicking around the idea of all these things. I would have said that probably guardrails would be more prevalent, especially in finance use cases, but surprisingly they’re not. Yeah. And that was actually part of what we, that was like a feature, I believe we built in the LLM, LLM gateway early on is like the, the sort of last chance, like, uh, you know, like, uh, um, Like hardguarded Yeah, exactly. Here’s some regexes that will just kill. Yeah, exactly. Or just, you know, in the way that like, if you go away a field on a ChatGPT, you just get like the inline 500 error. It doesn’t even tell you that it can’t tell, but just like craps out. Uh, like we kind of built a couple of those circuit breakers or like the ability to put those circuit breakers in and, and I don’t, I don’t believe that we’re using them for anything. One last thing I want to get your thoughts on was AI fluency levels, which you guys have a framework of user, advocate, builder, native, and everyone goes through it, including you. Including Camilla. And I just think it’s interesting. I think it’s a model that other people are thinking about adopting, but they’re worried about ruling it out that everybody’s going to be bad. And then, and also like, how do you have like this in-house training course that you keep up to date? Yeah. Just tell us more about it. Yeah. So in, in the operations org, uh, they’re actually more ahead of even engineering on this front, as far as like trying to create, um, create like, you know, like, you know, like learning pathways for this. Uh, and I think that part of the reason why they’re ahead of us is that in operations are much more, uh, they have to be elaborate, uh, training at, at scale. Like training is a big, very big part of, um, of how people build the aptitude around their, their job function within ops. Whereas like in EPD, a lot of it is sort of, uh, getting hands on building experience, like going a lot and getting mentored, getting code review. But, uh, it’s been really neat because I think we’ve really like, we created an environment.

James [00:59:14]: Um, by speaking openly, uh, about the, the transformation that we saw would happen in this industry towards AI sort of displacing a lot of, um, a lot of the operations and CX roles. And we were just honest about it. And I think what, what in the same breath that we said, Hey, a lot of these job responsibilities will go away. We also said, we don’t anticipate that meaning that your job has to go away. It’s just that your job has to change. And so the, the fluency framework and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and and then, uh, the like the training and support and like the positive sort of culture where we celebrate people making progress has been really helpful for like avoiding a culture of fear or like, oh, you have to do this, or you have you’re going to get this is going to go in your performance evaluation. I think the, it does it well, it’s not like his road is like, oh, like, what is, you know, what is the, like how much are you using AI and is it, is it enough? It’s, it’s more, I think we’ve built a pretty positively framed culture where we’ll do spot bonuses for people who have particularly novel uses of AI in their day-to-day. In our company, All Hands, every two weeks, we’ll do an AI spotlight. And it’s very rarely somebody in EPD. For the most part, it’s folks in VTMs, ops, finance, the people organization showing off how they’re building agents in ChatGPT or on Glean or how they just found some new use case that they thought was helpful. So we’re trying to create... I think at the end of the day, we’ve hired a bunch of really smart people who... I have full confidence that this type of work is within the reach of anybody who’s motivated to challenge themselves. And so we’ve done that. And in engineering, there’s one other thing that I want to call out because I think that this is kind of fun, is that we adapted our interview loop to be more AI, sort of agentic coding native. So instead of... We had a coding and a system design question. We had a coding and a system design question that we basically have revamped into a project where we’ll give you a brief before you come on site and then an additional sort of spec when you start. We expect you to use agentic coding to complete the task. In fact, it’s kind of impossible to get all the way through it if you don’t. And so we’re evaluating your knowledge. We’re kind of watching how you work. We’re evaluating whether you understand the code that’s coming out. We’re kind of probing at you as you go. But what we did in order to kind of maybe bootstrap the process... of all of our existing engineers, like getting familiar with agentic coding, is that we, as soon as we had the interview ready to ship, we started, we said, everybody in engineering, including all the managers, are going to have to go through this interview. And so we re-interviewed everybody internally. And it’s like, it’s one of those things where it’s like, it’s not a, we didn’t like keep a score or like, or like, you know, I don’t have any data on like who passed or failed or what they, what they scored. But what we found is like, as people would take it, it would actually cause them to have moments of realization where it was like, oh, I, I can up-level my skills or sort of like, I have, like, I want to be better at this. And so we’re trying to find like a way, like a variety of techniques that kind of push the culture along. And I think as I reflect on like the year, cause this is the year where we really put all the effort into it. I’m really satisfied to see the ascent to which everybody’s leaning in on a, on a daily basis. Going back to like, even I was shocked when we were looking at our cursor logs that like the number one user is, is an engineering manager. Yeah. And for, and for org, it’s like, that, that is super cool to me. It means that like folks have, have, uh, have taken this to heart and found, found ways of, um, doing their job differently. I guess my, I had a closing question or I guess a parting question, and this is broadening out from Brex and this is just, you interface with other engineering leaders all the time. Did we not cover anything that other CTOs are having as top of mind today? Like their number one problem is underscore. The thing I find myself discussing with. With folks that, and I, I don’t want to shy away from like scary topics. Uh, in fact, we were just, just kind of on one that was adjacent, which is like, how do you evaluate somebody’s like progression towards being more AI native? The, the, the, the cousin to that question is it’s like, will we need as many people, um, to operate our businesses? Like, are there layoffs coming? Are, how are we, how are we thinking about like, um, headcount growth, junior versus senior, junior versus senior? Yes, exactly. Like level mix. Um, and. And I still have more questions than I have answers there. I think, I think what has been really interesting is that I view a gentic development as being something that amplifies all the, all the good, just as much as it amplifies all the bad and the amplifies, uh, uh, sloppiness, poor architectural thinking, um, uh, misunderstanding of, of the requirements. Like there are, for all of the, the acceleration of good outcomes. It also accelerates. Bad outcomes. And I think what has been interesting is that there has been, when you sum that all together, there’s less of a obvious, um, like capacity increase. It’s, it’s more, it’s more nuanced than that. And so I’m not looking at headcount planning, uh, as we think about it next year as, as being something like, oh, well, because AI is giving us so much more leverage. We, we don’t need as many people. Um, we’ve actually, the thing I’m really proud of in, in my tenure as CTO is that we, we haven’t grown. Engineering at all. What we’ve done is we’ve, we’ve grown the business significantly, but we’ve been able to build, uh, like greater efficiencies and, and how we execute, like how we, how we, uh, we think about building, how we roadmap, um, what we choose to do and what not to do that. We’re able to, uh, to serve significantly more customers with more lines of business, um, without needing to grow engineering headcount. I think that that’s kind of the way that we’re gonna just continue on this road is like, I like having 300 engineers, like I would love, love to just, you know, a year from now have 300 engineers, but we’re still. You know, 30, 50, a hundred percent more efficient, but, uh, that, that, that is the thing that comes up with, uh, with other engineering leaders. And the other part of that conversation is like, how much is AI getting blamed for this sort of ordinary performance oriented, uh, realness, you know, like if, if Microsoft is letting go of like 4,000 people as a business, what they have 150,000 employees, I believe. Uh, is that really like AI causing that? Or is it them just using it as a way to, uh, uh, to avoid some harder, like perf management decisions? I’m not entirely sure, but I’m, I’m listening more than I’m speaking on the, on this topic because I, every time I feel like I have a pretty firm point of view, some new, uh, anecdote or experience comes in that kind of challenges or invalidates it. Yeah. Well, you know, I, I take these signals as it’s my job to go find people who think they have answers and surface them. And you may or may not disagree, but at least you have something to use as a straw man in, in your work. Exactly. Exactly. And I, and I think as, as an industry, it was just early innings on, on, on this transformation. So I’m looking forward to seeing, uh, uh, you know, listening to this, this podcast episode a year from now and, and, and seeing, you know, what we got right, what we got wrong and what’s different, uh, because so much changes, uh, quarter over quarter. Yeah. I do think AICOE is a very well established pattern. I think, uh, internal platform is very well established pattern. And this, uh, fluency thing is something that people are figuring out that I think you guys are ahead on. Um, I’m happy to hear that. That’ll be my feedback. Yeah.

swyx/Alessio [01:06:38]: Any final call to action for things that you want to buy? Like what should people build for you? Like problems you’re trying to solve that you would love people to reach out for to, to help with?

James [01:06:48]: The call that I’d make is for folks who are interested in, in multi-agent networks to, to get in touch with us, because I, I do feel like this is something where, where we’re, we’re innovating in, in service of, of our customers and where I, I feel like the frameworks, the tooling, um, and the. The research is, is, is there, there’s actually quite a lot of like interesting papers and things that we, we lean on. Uh, but I would love to, uh, would love to see more of that, like encoded in the, um, in the what’s available writ large in the industry, because I feel like my intuition has been that trying to craft LLMs into deterministic workflows and DAGs is, is kind of underselling, like the power that they have to actually plan and execute more and a more sophisticated, like. Yeah. Fluid way. And, and I, and I just want to see like the industry lean in more, um, on, uh, on these agent to agent, uh, uh, interactions. Okay. So, uh, I’ll, I’ll dive in a little bit here. Yeah. I have a minor opinion. You keep using the word networks. Yep. Is that a reference to a specific paper or it’s your term for it? It’s just, it’s our term. And I think that that is, that’s actually the term that master uses as well. Um, or it, it, we, um, yeah, initially we used to call them agent run times, uh, internally. Yeah. And then we just, yeah, switched to networks. Uh, and then I think the other thing I wanted to get a clarification on is, is it mostly a full agent talking with a full agent or is there a kind of like a orchestrated boss agent talking to a sub agent? And I think that does matter for a subset of people who are building all these things, because when you say multi-agents and guys, people don’t agree what that means. Yeah. So it’s, it’s a tree more than it is a graph. So it is like, yeah, we have it when you say network, it’s feels more of a graph. Yeah. But. It seems more directional as a tree. Like there, there is a hierarchy. There’s a hierarchy. Yeah. But there, but there are some violations of that. Like one of the, one of the interesting use cases, uh, and this is where like the power of, of having an, an assistant for every employee plus having agents that run and, and embody, uh, members of the finance team is really powerful because, uh, there’s this interesting use case that, that we brought to market, which is that, um, one of the finance teams is a finance team. Uh, a lot of the finance team agents that we, we, uh, launched is an audit agent where like an audit agent kind of embodies the work that a lot of larger finance teams will do to look for patterns of waste fraud or abuse or like systematic, uh, avoidance of policy that isn’t as obvious with a single expense. Like you can evaluate a single expense in the metadata around it to see if it, if it’s, um, within policy or not. But, um, what if you start seeing an employee often make a large number of like $74 transactions when receipts are required? Part of 75 or what if you, what if you see, um, certain things like, oh, okay, there’s actually a fair number of like DoorDash expenses during business hours from this individual, like on, on days that an office lunch is provided, or maybe you see like ride share patterns that are, are, um, where you have to look at a broader context. Um, so we built this audit agent that can like ingest your SOP and, and look also ingest your. This is a Bikes’ customer’s SOP. Exactly. Yep. And, uh, and what it does then is it’s, it’s basically always looking for. Potential violations and what it does is it, it is extremely zealous. Like it, it wants to have a minimum number of false negatives, so it will raise a large number of potential violations. And then a separate agent, a review agent will then apply wisdom, the wisdom of like, is this important enough to follow up on? Is the dollar amount in question high enough? Does this user seem to have like a high compliance behavior more generally? It makes a judgment call about whether it’s worthy enough to. Take that violation and make it into a case. Then once it’s made into a case, generally what happens is that you need to get more information from the individual. So if humans were doing this, there’d, there’d be some outsourced team that’s like looking for all the potential violations. Then you have some full-time employee on the finance team who’s, who’s looking at all the violations that, oh, these are the ones that are important. We need to follow up on it. Now, what they do is they hand it off to somebody who will go and Slack that employee and be like, Hey, what’s going on here? And so what we have is like the audit agent looks for violations. The review agent. Decides whether it’s worthy enough to turn into a case. And then from there, uh, when the case is filed, the, that, that will trigger an event to the Brexit assistant for that employee. And like any additional information about like the business justification, um, can be collected or maybe the assistant already knows because it, in its conversation history of the employee knew something about why this, this expense, uh, looked out of, out of policy. And so you start having the, the network becomes interesting when you have the finance team agents. Yeah. Communicating with, uh, the assistant or various employees. And then behind there, you have other, other sub-agents. And so then you start seeing like more of a graph, uh, emerge, but when you look at just what serves the employee, it looks more like a tree. Amazing. Well, I didn’t know you were going to go into that level of detail. Yeah. Yeah. Don’t worry about that. No, no, no, no. I’m, I’m actually really glad I asked like that is very impressive. And, uh, I hope you, uh, do more content about that. Yeah, absolutely. We’re really excited about it. I think, uh, it’s, it’s been, it’s been good to finally figure out, uh, a use. Use for, for agents and have the technology be as, uh, like as robust as it is to start realizing this vision. Cause it’s something that we, we kind of dreamt of a couple years ago in the tech, like to your earlier point, the tech just wasn’t there when we were trying to make the, make the, a similar concept to work with the GPT 3.5. It was like, now we were hallucinating tool calls and, uh, back in that day.

swyx/Alessio [01:12:24]: Um, awesome, man. Thanks so much for joining us. This was fun.

James [01:12:27]: I really enjoyed it. Uh, happy holidays guys. Thank you for having me. Thank you.

Happy New Year! You may have noticed that in 2025 we had moved toward YouTube as our primary podcasting platform. As we’ll explain in the next State of Latent Space post, we’ll be doubling down on Substack again and improving the experience for the over 100,000 of you who look out for our emails and website updates!

We first mentioned Artificial Analysis in 2024, when it was still a side project in a Sydney basement. They then were one of the few Nat Friedman and Daniel Gross’ AIGrant companies to raise a full seed round from them and have now become the independent gold standard for AI benchmarking—trusted by developers, enterprises, and every major lab to navigate the exploding landscape of models, providers, and capabilities.

We have chatted with both Clementine Fourrier of HuggingFace’s OpenLLM Leaderboard and (the freshly valued at $1.7B) Anastasios Angelopoulos of LMArena on their approaches to LLM evals and trendspotting, but Artificial Analysis have staked out an enduring and important place in the toolkit of the modern AI Engineer by doing the best job of independently running the most comprehensive set of evals across the widest range of open and closed models, and charting their progress for broad industry analyst use.

George Cameron and Micah-Hill Smith have spent two years building Artificial Analysis into the platform that answers the questions no one else will: Which model is actually best for your use case? What are the real speed-cost trade-offs? And how open is “open” really?

We discuss:

• The origin story: built as a side project in 2023 while Micah was building a legal AI assistant, launched publicly in January 2024, and went viral after Swyx’s retweet

• Why they run evals themselves: labs prompt models differently, cherry-pick chain-of-thought examples (Google Gemini 1.0 Ultra used 32-shot prompts to beat GPT-4 on MMLU), and self-report inflated numbers

• The mystery shopper policy: they register accounts not on their own domain and run intelligence + performance benchmarks incognito to prevent labs from serving different models on private endpoints

• How they make money: enterprise benchmarking insights subscription (standardized reports on model deployment, serverless vs. managed vs. leasing chips) and private custom benchmarking for AI companies (no one pays to be on the public leaderboard)

• The Intelligence Index (V3): synthesizes 10 eval datasets (MMLU, GPQA, agentic benchmarks, long-context reasoning) into a single score, with 95% confidence intervals via repeated runs

• Omissions Index (hallucination rate): scores models from -100 to +100 (penalizing incorrect answers, rewarding \”I don’t know\”), and Claude models lead with the lowest hallucination rates despite not always being the smartest

• GDP Val AA: their version of OpenAI’s GDP-bench (44 white-collar tasks with spreadsheets, PDFs, PowerPoints), run through their Stirrup agent harness (up to 100 turns, code execution, web search, file system), graded by Gemini 3 Pro as an LLM judge (tested extensively, no self-preference bias)

• The Openness Index: scores models 0-18 on transparency of pre-training data, post-training data, methodology, training code, and licensing (AI2 OLMo 2 leads, followed by Nous Hermes and NVIDIA Nemotron)

• The smiling curve of AI costs: GPT-4-level intelligence is 100-1000x cheaper than at launch (thanks to smaller models like Amazon Nova), but frontier reasoning models in agentic workflows cost more than ever (sparsity, long context, multi-turn agents)

• Why sparsity might go way lower than 5%: GPT-4.5 is ~5% active, Gemini models might be ~3%, and Omissions Index accuracy correlates with total parameters (not active), suggesting massive sparse models are the future

• Token efficiency vs. turn efficiency: GPT-5 costs more per token but solves Tau-bench in fewer turns (cheaper overall), and models are getting better at using more tokens only when needed (5.1 Codex has tighter token distributions)

• V4 of the Intelligence Index coming soon: adding GDP Val AA, Critical Point, hallucination rate, and dropping some saturated benchmarks (human-eval-style coding is now trivial for small models)

Links to Artificial Analysis

• Website: https://artificialanalysis.ai

• George Cameron on X: https://x.com/georgecameron

• Micah-Hill Smith on X: https://x.com/micahhsmith

• 00:00 Introduction: Full Circle Moment and Artificial Analysis Origins

• 01:19 Business Model: Independence and Revenue Streams

• 04:33 Origin Story: From Legal AI to Benchmarking Need

• 16:22 AI Grant and Moving to San Francisco

• 19:21 Intelligence Index Evolution: From V1 to V3

• 11:47 Benchmarking Challenges: Variance, Contamination, and Methodology

• 13:52 Mystery Shopper Policy and Maintaining Independence

• 28:01 New Benchmarks: Omissions Index for Hallucination Detection

• 33:36 Critical Point: Hard Physics Problems and Research-Level Reasoning

• 23:01 GDP Val AA: Agentic Benchmark for Real Work Tasks

• 50:19 Stirrup Agent Harness: Open Source Agentic Framework

• 52:43 Openness Index: Measuring Model Transparency Beyond Licenses

• 58:25 The Smiling Curve: Cost Falling While Spend Rising

• 1:02:32 Hardware Efficiency: Blackwell Gains and Sparsity Limits

• 1:06:23 Reasoning Models and Token Efficiency: The Spectrum Emerges

• 1:11:00 Multimodal Benchmarking: Image, Video, and Speech Arenas

• 1:15:05 Looking Ahead: Intelligence Index V4 and Future Directions

• 1:16:50 Closing: The Insatiable Demand for Intelligence

Micah [00:00:06]: This is kind of a full circle moment for us in a way, because the first time artificial analysis got mentioned on a podcast was you and Alessio on Latent Space. Amazing.

swyx [00:00:17]: Which was January 2024. I don’t even remember doing that, but yeah, it was very influential to me. Yeah, I’m looking at AI News for Jan 17, or Jan 16, 2024. I said, this gem of a models and host comparison site was just launched. And then I put in a few screenshots, and I said, it’s an independent third party. It clearly outlines the quality versus throughput trade-off, and it breaks out by model and hosting provider. I did give you shit for missing fireworks, and how do you have a model benchmarking thing without fireworks? But you had together, you had perplexity, and I think we just started chatting there. Welcome, George and Micah, to Latent Space. I’ve been following your progress. Congrats on... It’s been an amazing year. You guys have really come together to be the presumptive new gardener of AI, right? Which is something that...

George [00:01:09]: Yeah, but you can’t pay us for better results.

swyx [00:01:12]: Yes, exactly.

George [00:01:13]: Very important.

Micah [00:01:14]: Start off with a spicy take.

swyx [00:01:18]: Okay, how do I pay you?

Micah [00:01:20]: Let’s get right into that.

swyx [00:01:21]: How do you make money?

Micah [00:01:24]: Well, very happy to talk about that. So it’s been a big journey the last couple of years. Artificial analysis is going to be two years old in January 2026. Which is pretty soon now. We first run the website for free, obviously, and give away a ton of data to help developers and companies navigate AI and make decisions about models, providers, technologies across the AI stack for building stuff. We’re very committed to doing that and tend to keep doing that. We have, along the way, built a business that is working out pretty sustainably. We’ve got just over 20 people now and two main customer groups. So we want to be... We want to be who enterprise look to for data and insights on AI, so we want to help them with their decisions about models and technologies for building stuff. And then on the other side, we do private benchmarking for companies throughout the AI stack who build AI stuff. So no one pays to be on the website. We’ve been very clear about that from the very start because there’s no use doing what we do unless it’s independent AI benchmarking. Yeah. But turns out a bunch of our stuff can be pretty useful to companies building AI stuff.

swyx [00:02:38]: And is it like, I am a Fortune 500, I need advisors on objective analysis, and I call you guys and you pull up a custom report for me, you come into my office and give me a workshop? What kind of engagement is that?

George [00:02:53]: So we have a benchmarking and insight subscription, which looks like standardized reports that cover key topics or key challenges enterprises face when looking to understand AI and choose between all the technologies. And so, for instance, one of the report is a model deployment report, how to think about choosing between serverless inference, managed deployment solutions, or leasing chips. And running inference yourself is an example kind of decision that big enterprises face, and it’s hard to reason through, like this AI stuff is really new to everybody. And so we try and help with our reports and insight subscription. Companies navigate that. We also do custom private benchmarking. And so that’s very different from the public benchmarking that we publicize, and there’s no commercial model around that. For private benchmarking, we’ll at times create benchmarks, run benchmarks to specs that enterprises want. And we’ll also do that sometimes for AI companies who have built things, and we help them understand what they’ve built with private benchmarking. Yeah. So that’s a piece mainly that we’ve developed through trying to support everybody publicly with our public benchmarks. Yeah.

swyx [00:04:09]: Let’s talk about TechStack behind that. But okay, I’m going to rewind all the way to when you guys started this project. You were all the way in Sydney? Yeah. Well, Sydney, Australia for me.

Micah [00:04:19]: George was an SF, but he’s Australian, but he moved here already. Yeah.

swyx [00:04:22]: And I remember I had the Zoom call with you. What was the impetus for starting artificial analysis in the first place? You know, you started with public benchmarks. And so let’s start there. We’ll go to the private benchmark. Yeah.

George [00:04:33]: Why don’t we even go back a little bit to like why we, you know, thought that it was needed? Yeah.

Micah [00:04:40]: The story kind of begins like in 2022, 2023, like both George and I have been into AI stuff for quite a while. In 2023 specifically, I was trying to build a legal AI research assistant. So it actually worked pretty well for its era, I would say. Yeah. Yeah. So I was finding that the more you go into building something using LLMs, the more each bit of what you’re doing ends up being a benchmarking problem. So had like this multistage algorithm thing, trying to figure out what the minimum viable model for each bit was, trying to optimize every bit of it as you build that out, right? Like you’re trying to think about accuracy, a bunch of other metrics and performance and cost. And mostly just no one was doing anything to independently evaluate all the models. And certainly not to look at the trade-offs for speed and cost. So we basically set out just to build a thing that developers could look at to see the trade-offs between all of those things measured independently across all the models and providers. Honestly, it was probably meant to be a side project when we first started doing it.

swyx [00:05:49]: Like we didn’t like get together and say like, Hey, like we’re going to stop working on all this stuff. I’m like, this is going to be our main thing. When I first called you, I think you hadn’t decided on starting a company yet.

Micah [00:05:58]: That’s actually true. I don’t even think we’d pause like, like George had an acquittance job. I didn’t quit working on my legal AI thing. Like it was genuinely a side project.

George [00:06:05]: We built it because we needed it as people building in the space and thought, Oh, other people might find it useful too. So we’ll buy domain and link it to the Vercel deployment that we had and tweet about it. And, but very quickly it started getting attention. Thank you, Swyx for, I think doing an initial retweet and spotlighting it there. This project that we released. And then very quickly though, it was useful to others, but very quickly it became more useful as the number of models released accelerated. We had Mixtrel 8x7B and it was a key. That’s a fun one. Yeah. Like a open source model that really changed the landscape and opened up people’s eyes to other serverless inference providers and thinking about speed, thinking about cost. And so that was a key. And so it became more useful quite quickly. Yeah.

swyx [00:07:02]: What I love talking to people like you who sit across the ecosystem is, well, I have theories about what people want, but you have data and that’s obviously more relevant. But I want to stay on the origin story a little bit more. When you started out, I would say, I think the status quo at the time was every paper would come out and they would report their numbers versus competitor numbers. And that’s basically it. And I remember I did the legwork. I think everyone has some knowledge. I think there’s some version of Excel sheet or a Google sheet where you just like copy and paste the numbers from every paper and just post it up there. And then sometimes they don’t line up because they’re independently run. And so your numbers are going to look better than... Your reproductions of other people’s numbers are going to look worse because you don’t hold their models correctly or whatever the excuse is. I think then Stanford Helm, Percy Liang’s project would also have some of these numbers. And I don’t know if there’s any other source that you can cite. The way that if I were to start artificial analysis at the same time you guys started, I would have used the Luther AI’s eval framework harness. Yup.

Micah [00:08:06]: Yup. That was some cool stuff. At the end of the day, running these evals, it’s like if it’s a simple Q&A eval, all you’re doing is asking a list of questions and checking if the answers are right, which shouldn’t be that crazy. But it turns out there are an enormous number of things that you’ve got control for. And I mean, back when we started the website. Yeah. Yeah. Like one of the reasons why we realized that we had to run the evals ourselves and couldn’t just take rules from the labs was just that they would all prompt the models differently. And when you’re competing over a few points, then you can pretty easily get- You can put the answer into the model. Yeah. That in the extreme. And like you get crazy cases like back when I’m Googled a Gemini 1.0 Ultra and needed a number that would say it was better than GPT-4 and like constructed, I think never published like chain of thought examples. 32 of them in every topic in MLU to run it, to get the score, like there are so many things that you- They never shipped Ultra, right? That’s the one that never made it up. Not widely. Yeah. Yeah. Yeah. I mean, I’m sure it existed, but yeah. So we were pretty sure that we needed to run them ourselves and just run them in the same way across all the models. Yeah. And we were, we also did certain from the start that you couldn’t look at those in isolation. You needed to look at them alongside the cost and performance stuff. Yeah.

swyx [00:09:24]: Okay. A couple of technical questions. I mean, so obviously I also thought about this and I didn’t do it because of cost. Yep. Did you not worry about costs? Were you funded already? Clearly not, but you know. No. Well, we definitely weren’t at the start.

Micah [00:09:36]: So like, I mean, we’re paying for it personally at the start. There’s a lot of money. Well, the numbers weren’t nearly as bad a couple of years ago. So we certainly incurred some costs, but we were probably in the order of like hundreds of dollars of spend across all the benchmarking that we were doing. Yeah. So nothing. Yeah. It was like kind of fine. Yeah. Yeah. These days that’s gone up an enormous amount for a bunch of reasons that we can talk about. But yeah, it wasn’t that bad because you can also remember that like the number of models we were dealing with was hardly any and the complexity of the stuff that we wanted to do to evaluate them was a lot less. Like we were just asking some Q&A type questions and then one specific thing was for a lot of evals initially, we were just like sampling an answer. You know, like, what’s the answer for this? Like, we didn’t want to go into the answer directly without letting the models think. We weren’t even doing chain of thought stuff initially. And that was the most useful way to get some results initially. Yeah.

swyx [00:10:33]: And so for people who haven’t done this work, literally parsing the responses is a whole thing, right? Like because sometimes the models, the models can answer any way they feel fit and sometimes they actually do have the right answer, but they just returned the wrong format and they will get a zero for that unless you work it into your parser. And that involves more work. And so, I mean, but there’s an open question whether you should give it points for not following your instructions on the format.

Micah [00:11:00]: It depends what you’re looking at, right? Because you can, if you’re trying to see whether or not it can solve a particular type of reasoning problem, and you don’t want to test it on its ability to do answer formatting at the same time, then you might want to use an LLM as answer extractor approach to make sure that you get the answer out no matter how unanswered. But these days, it’s mostly less of a problem. Like, if you instruct a model and give it examples of what the answers should look like, it can get the answers in your format, and then you can do, like, a simple regex.

swyx [00:11:28]: Yeah, yeah. And then there’s other questions around, I guess, sometimes if you have a multiple choice question, sometimes there’s a bias towards the first answer, so you have to randomize the responses. All these nuances, like, once you dig into benchmarks, you’re like, I don’t know how anyone believes the numbers on all these things. It’s so dark magic.

Micah [00:11:47]: You’ve also got, like… You’ve got, like, the different degrees of variance in different benchmarks, right? Yeah. So, if you run four-question multi-choice on a modern reasoning model at the temperatures suggested by the labs for their own models, the variance that you can see on a four-question multi-choice eval is pretty enormous if you only do a single run of it and it has a small number of questions, especially. So, like, one of the things that we do is run an enormous number of all of our evals when we’re developing new ones and doing upgrades to our intelligence index to bring in new things. Yeah. So, that we can dial in the right number of repeats so that we can get to the 95% confidence intervals that we’re comfortable with so that when we pull that together, we can be confident in intelligence index to at least as tight as, like, a plus or minus one at a 95% confidence. Yeah.

swyx [00:12:32]: And, again, that just adds a straight multiple to the cost. Oh, yeah. Yeah, yeah.

George [00:12:37]: So, that’s one of many reasons that cost has gone up a lot more than linearly over the last couple of years. We report a cost to run the artificial analysis. We report a cost to run the artificial analysis intelligence index on our website, and currently that’s assuming one repeat in terms of how we report it because we want to reflect a bit about the weighting of the index. But our cost is actually a lot higher than what we report there because of the repeats.

swyx [00:13:03]: Yeah, yeah, yeah. And probably this is true, but just checking, you don’t have any special deals with the labs. They don’t discount it. You just pay out of pocket or out of your sort of customer funds. Oh, there is a mix. So, the issue is that sometimes they may give you a special end point, which is… Ah, 100%.

Micah [00:13:21]: Yeah, yeah, yeah. Exactly. So, we laser focus, like, on everything we do on having the best independent metrics and making sure that no one can manipulate them in any way. There are quite a lot of processes we’ve developed over the last couple of years to make that true for, like, the one you bring up, like, right here of the fact that if we’re working with a lab, if they’re giving us a private endpoint to evaluate a model, that it is totally possible. That what’s sitting behind that black box is not the same as they serve on a public endpoint. We’re very aware of that. We have what we call a mystery shopper policy. And so, and we’re totally transparent with all the labs we work with about this, that we will register accounts not on our own domain and run both intelligence evals and performance benchmarks… Yeah, that’s the job. …without them being able to identify it. And no one’s ever had a problem with that. Because, like, a thing that turns out to actually be quite a good… …good factor in the industry is that they all want to believe that none of their competitors could manipulate what we’re doing either.

swyx [00:14:23]: That’s true. I never thought about that. I’ve been in the database data industry prior, and there’s a lot of shenanigans around benchmarking, right? So I’m just kind of going through the mental laundry list. Did I miss anything else in this category of shenanigans? Oh, potential shenanigans.

Micah [00:14:36]: I mean, okay, the biggest one, like, that I’ll bring up, like, is more of a conceptual one, actually, than, like, direct shenanigans. It’s that the things that get measured become things that get targeted by labs that they’re trying to build, right? Exactly. So that doesn’t mean anything that we should really call shenanigans. Like, I’m not talking about training on test set. But if you know that you’re going to be great at another particular thing, if you’re a researcher, there are a whole bunch of things that you can do to try to get better at that thing that preferably are going to be helpful for a wide range of how actual users want to use the thing that you’re building. But will not necessarily work. Will not necessarily do that. So, for instance, the models are exceptional now at answering competition maths problems. There is some relevance of that type of reasoning, that type of work, to, like, how we might use modern coding agents and stuff. But it’s clearly not one for one. So the thing that we have to be aware of is that once an eval becomes the thing that everyone’s looking at, scores can get better on it without there being a reflection of overall generalized intelligence of these models. Getting better. That has been true for the last couple of years. It’ll be true for the next couple of years. There’s no silver bullet to defeat that other than building new stuff to stay relevant and measure the capabilities that matter most to real users. Yeah.

swyx [00:15:58]: And we’ll cover some of the new stuff that you guys are building as well, which is cool. Like, you used to just run other people’s evals, but now you’re coming up with your own. And I think, obviously, that is a necessary path once you’re at the frontier. You’ve exhausted all the existing evals. I think the next point in history that I have for you is AI Grant that you guys decided to join and move here. What was it like? I think you were in, like, batch two? Batch four. Batch four. Okay.

Micah [00:16:26]: I mean, it was great. Nat and Daniel are obviously great. And it’s a really cool group of companies that we were in AI Grant alongside. It was really great to get Nat and Daniel on board. Obviously, they’ve done a whole lot of great work in the space with a lot of leading companies and were extremely aligned. With the mission of what we were trying to do. Like, we’re not quite typical of, like, a lot of the other AI startups that they’ve invested in.

swyx [00:16:53]: And they were very much here for the mission of what we want to do. Did they say any advice that really affected you in some way or, like, were one of the events very impactful? That’s an interesting question.

Micah [00:17:03]: I mean, I remember fondly a bunch of the speakers who came and did fireside chats at AI Grant.

swyx [00:17:09]: Which is also, like, a crazy list. Yeah.

George [00:17:11]: Oh, totally. Yeah, yeah, yeah. There was something about, you know, speaking to Nat and Daniel about the challenges of working through a startup and just working through the questions that don’t have, like, clear answers and how to work through those kind of methodically and just, like, work through the hard decisions. And they’ve been great mentors to us as we’ve built artificial analysis. Another benefit for us was that other companies in the batch and other companies in AI Grant are pushing the capabilities. Yeah. And I think that’s a big part of what AI can do at this time. And so being in contact with them, making sure that artificial analysis is useful to them has been fantastic for supporting us in working out how should we build out artificial analysis to continue to being useful to those, like, you know, building on AI.

swyx [00:17:59]: I think to some extent, I’m mixed opinion on that one because to some extent, your target audience is not people in AI Grants who are obviously at the frontier. Yeah. Do you disagree?

Micah [00:18:09]: To some extent. To some extent. But then, so a lot of what the AI Grant companies are doing is taking capabilities coming out of the labs and trying to push the limits of what they can do across the entire stack for building great applications, which actually makes some of them pretty archetypical power users of artificial analysis. Some of the people with the strongest opinions about what we’re doing well and what we’re not doing well and what they want to see next from us. Yeah. Yeah. Because when you’re building any kind of AI application now, chances are you’re using a whole bunch of different models. You’re maybe switching reasonably frequently for different models and different parts of your application to optimize what you’re able to do with them at an accuracy level and to get better speed and cost characteristics. So for many of them, no, they’re like not commercial customers of ours, like we don’t charge for all our data on the website. Yeah. They are absolutely some of our power users.

swyx [00:19:07]: So let’s talk about just the evals as well. So you start out from the general like MMU and GPQA stuff. What’s next? How do you sort of build up to the overall index? What was in V1 and how did you evolve it? Okay.

Micah [00:19:22]: So first, just like background, like we’re talking about the artificial analysis intelligence index, which is our synthesis metric that we pulled together currently from 10 different eval data sets to give what? We’re pretty much the same as that. Pretty confident is the best single number to look at for how smart the models are. Obviously, it doesn’t tell the whole story. That’s why we published the whole website of all the charts to dive into every part of it and look at the trade-offs. But best single number. So right now, it’s got a bunch of Q&A type data sets that have been very important to the industry, like a couple that you just mentioned. It’s also got a couple of agentic data sets. It’s got our own long context reasoning data set and some other use case focused stuff. As time goes on. The things that we’re most interested in that are going to be important to the capabilities that are becoming more important for AI, what developers are caring about, are going to be first around agentic capabilities. So surprise, surprise. We’re all loving our coding agents and how the model is going to perform like that and then do similar things for different types of work are really important to us. The linking to use cases to economically valuable use cases are extremely important to us. And then we’ve got some of the. Yeah. These things that the models still struggle with, like working really well over long contexts that are not going to go away as specific capabilities and use cases that we need to keep evaluating.

swyx [00:20:46]: But I guess one thing I was driving was like the V1 versus the V2 and how bad it was over time.

Micah [00:20:53]: Like how we’ve changed the index to where we are.

swyx [00:20:55]: And I think that reflects on the change in the industry. Right. So that’s a nice way to tell that story.

Micah [00:21:00]: Well, V1 would be completely saturated right now. Almost every model coming out because doing things like writing the Python functions and human evil is now pretty trivial. It’s easy to forget, actually, I think how much progress has been made in the last two years. Like we obviously play the game constantly of like the today’s version versus last week’s version and the week before and all of the small changes in the horse race between the current frontier and who has the best like smaller than 10B model like right now this week. Right. And that’s very important to a lot of developers and people and especially in this particular city of San Francisco. But when you zoom out a couple of years ago, literally most of what we were doing to evaluate the models then would all be 100% solved by even pretty small models today. And that’s been one of the key things, by the way, that’s driven down the cost of intelligence at every tier of intelligence. We can talk about more in a bit. So V1, V2, V3, we made things harder. We covered a wider range of use cases. And we tried to get closer to things developers care about as opposed to like just the Q&A type stuff that MMLU and GPQA represented. Yeah.

swyx [00:22:12]: I don’t know if you have anything to add there. Or we could just go right into showing people the benchmark and like looking around and asking questions about it. Yeah.

Micah [00:22:21]: Let’s do it. Okay. This would be a pretty good way to chat about a few of the new things we’ve launched recently. Yeah.

George [00:22:26]: And I think a little bit about the direction that we want to take it. And we want to push benchmarks. Currently, the intelligence index and evals focus a lot on kind of raw intelligence. But we kind of want to diversify how we think about intelligence. And we can talk about it. But kind of new evals that we’ve kind of built and partnered on focus on topics like hallucination. And we’ve got a lot of topics that I think are not covered by the current eval set that should be. And so we want to bring that forth. But before we get into that.

swyx [00:23:01]: And so for listeners, just as a timestamp, right now, number one is Gemini 3 Pro High. Then followed by Cloud Opus at 70. Just 5.1 high. You don’t have 5.2 yet. And Kimi K2 Thinking. Wow. Still hanging in there. So those are the top four. That will date this podcast quickly. Yeah. Yeah. I mean, I love it. I love it. No, no. 100%. Look back this time next year and go, how cute. Yep.

George [00:23:25]: Totally. A quick view of that is, okay, there’s a lot. I love it. I love this chart. Yeah.

Micah [00:23:30]: This is such a favorite, right? Yeah. And almost every talk that George or I give at conferences and stuff, we always put this one up first to just talk about situating where we are in this moment in history. This, I think, is the visual version of what I was saying before about the zooming out and remembering how much progress there’s been. If we go back to just over a year ago, before 01, before Cloud Sonnet 3.5, we didn’t have reasoning models or coding agents as a thing. And the game was very, very different. If we go back even a little bit before then, we’re in the era where, when you look at this chart, open AI was untouchable for well over a year. And, I mean, you would remember that time period well of there being very open questions about whether or not AI was going to be competitive, like full stop, whether or not open AI would just run away with it, whether we would have a few frontier labs and no one else would really be able to do anything other than consume their APIs. I am quite happy overall that the world that we have ended up in is one where... Multi-model. Absolutely. And strictly more competitive every quarter over the last few years. Yeah. This year has been insane. Yeah.

George [00:24:42]: You can see it. This chart with everything added is hard to read currently. There’s so many dots on it, but I think it reflects a little bit what we felt, like how crazy it’s been.

swyx [00:24:54]: Why 14 as the default? Is that a manual choice? Because you’ve got service now in there that are less traditional names. Yeah.

George [00:25:01]: It’s models that we’re kind of highlighting by default in our charts, in our intelligence index. Okay.

swyx [00:25:07]: You just have a manually curated list of stuff.

George [00:25:10]: Yeah, that’s right. But something that I actually don’t think every artificial analysis user knows is that you can customize our charts and choose what models are highlighted. Yeah. And so if we take off a few names, it gets a little easier to read.

swyx [00:25:25]: Yeah, yeah. A little easier to read. Totally. Yeah. But I love that you can see the all one jump. Look at that. September 2024. And the DeepSeek jump. Yeah.

George [00:25:34]: Which got close to OpenAI’s leadership. They were so close. I think, yeah, we remember that moment. Around this time last year, actually.

Micah [00:25:44]: Yeah, yeah, yeah. I agree. Yeah, well, a couple of weeks. It was Boxing Day in New Zealand when DeepSeek v3 came out. And we’d been tracking DeepSeek and a bunch of the other global players that were less known over the second half of 2024 and had run evals on the earlier ones and stuff. I very distinctly remember Boxing Day in New Zealand, because I was with family for Christmas and stuff, running the evals and getting back result by result on DeepSeek v3. So this was the first of their v3 architecture, the 671b MOE.

Micah [00:26:19]: And we were very, very impressed. That was the moment where we were sure that DeepSeek was no longer just one of many players, but had jumped up to be a thing. The world really noticed when they followed that up with the RL working on top of v3 and R1 succeeding a few weeks later. But the groundwork for that absolutely was laid with just extremely strong base model, completely open weights that we had as the best open weights model. So, yeah, that’s the thing that you really see in the game. But I think that we got a lot of good feedback on Boxing Day. us on Boxing Day last year.

George [00:26:48]: Boxing Day is the day after Christmas for those not familiar.

George [00:26:54]: I’m from Singapore.

swyx [00:26:55]: A lot of us remember Boxing Day for a different reason, for the tsunami that happened. Oh, of course. Yeah, but that was a long time ago. So yeah. So this is the rough pitch of AAQI. Is it A-A-Q-I or A-A-I-I? I-I. Okay. Good memory, though.

Micah [00:27:11]: I don’t know. I’m not used to it. Once upon a time, we did call it Quality Index, and we would talk about quality, performance, and price, but we changed it to intelligence.

George [00:27:20]: There’s been a few naming changes. We added hardware benchmarking to the site, and so benchmarks at a kind of system level. And so then we changed our throughput metric to, we now call it output speed, and then

swyx [00:27:32]: throughput makes sense at a system level, so we took that name. Take me through more charts. What should people know? Obviously, the way you look at the site is probably different than how a beginner might look at it.

Micah [00:27:42]: Yeah, that’s fair. There’s a lot of fun stuff to dive into. Maybe so we can hit past all the, like, we have lots and lots of emails and stuff. The interesting ones to talk about today that would be great to bring up are a few of our recent things, I think, that probably not many people will be familiar with yet. So first one of those is our omniscience index. So this one is a little bit different to most of the intelligence evils that we’ve run. We built it specifically to look at the embedded knowledge in the models and to test hallucination by looking at when the model doesn’t know the answer, so not able to get it correct, what’s its probability of saying, I don’t know, or giving an incorrect answer. So the metric that we use for omniscience goes from negative 100 to positive 100. Because we’re simply taking off a point if you give an incorrect answer to the question. We’re pretty convinced that this is an example of where it makes most sense to do that, because it’s strictly more helpful to say, I don’t know, instead of giving a wrong answer to factual knowledge question. And one of our goals is to shift the incentive that evils create for models and the labs creating them to get higher scores. And almost every evil across all of AI up until this point, it’s been graded by simple percentage correct as the main metric, the main thing that gets hyped. And so you should take a shot at everything. There’s no incentive to say, I don’t know. So we did that for this one here.

swyx [00:29:22]: I think there’s a general field of calibration as well, like the confidence in your answer versus the rightness of the answer. Yeah, we completely agree. Yeah. Yeah.

George [00:29:31]: On that. And one reason that we didn’t do that is because. Or put that into this index is that we think that the, the way to do that is not to ask the models how confident they are.

swyx [00:29:43]: I don’t know. Maybe it might be though. You put it like a JSON field, say, say confidence and maybe it spits out something. Yeah. You know, we have done a few evils podcasts over the, over the years. And when we did one with Clementine of hugging face, who maintains the open source leaderboard, and this was one of her top requests, which is some kind of hallucination slash lack of confidence calibration thing. And so, Hey, this is one of them.

Micah [00:30:05]: And I mean, like anything that we do, it’s not a perfect metric or the whole story of everything that you think about as hallucination. But yeah, it’s pretty useful and has some interesting results. Like one of the things that we saw in the hallucination rate is that anthropics Claude models at the, the, the very left-hand side here with the lowest hallucination rates out of the models that we’ve evaluated amnesty is on. That is an interesting fact. I think it probably correlates with a lot of the previously, not really measured vibes stuff that people like about some of the Claude models. Is the dataset public or what’s is it, is there a held out set? There’s a hell of a set for this one. So we, we have published a public test set, but we we’ve only published 10% of it. The reason is that for this one here specifically, it would be very, very easy to like have data contamination because it is just factual knowledge questions. We would. We’ll update it at a time to also prevent that, but with yeah, kept most of it held out so that we can keep it reliable for a long time. It leads us to a bunch of really cool things, including breakdown quite granularly by topic. And so we’ve got some of that disclosed on the website publicly right now, and there’s lots more coming in terms of our ability to break out very specific topics. Yeah.

swyx [00:31:23]: I would be interested. Let’s, let’s dwell a little bit on this hallucination one. I noticed that Haiku hallucinates less than Sonnet hallucinates less than Opus. And yeah. Would that be the other way around in a normal capability environments? I don’t know. What’s, what do you make of that?

George [00:31:37]: One interesting aspect is that we’ve found that there’s not really a, not a strong correlation between intelligence and hallucination, right? That’s to say that the smarter the models are in a general sense, isn’t correlated with their ability to, when they don’t know something, say that they don’t know. It’s interesting that Gemini three pro preview was a big leap over here. Gemini 2.5. Flash and, and, and 2.5 pro, but, and if I add pro quickly here.

swyx [00:32:07]: I bet pro’s really good. Uh, actually no, I meant, I meant, uh, the GPT pros.

George [00:32:12]: Oh yeah.

swyx [00:32:13]: Cause GPT pros are rumored. We don’t know for a fact that it’s like eight runs and then with the LM judge on top. Yeah.

George [00:32:20]: So we saw a big jump in, this is accuracy. So this is just percent that they get, uh, correct and Gemini three pro knew a lot more than the other models. And so big jump in accuracy. But relatively no change between the Google Gemini models, between releases. And the hallucination rate. Exactly. And so it’s likely due to just kind of different post-training recipe, between the, the Claude models. Yeah.

Micah [00:32:45]: Um, there’s, there’s driven this. Yeah. You can, uh, you can partially blame us and how we define intelligence having until now not defined hallucination as a negative in the way that we think about intelligence.

swyx [00:32:56]: And so that’s what we’re changing. Uh, I know many smart people who are confidently incorrect.

George [00:33:02]: Uh, look, look at that. That, that, that is very humans. Very true. And there’s times and a place for that. I think our view is that hallucination rate makes sense in this context where it’s around knowledge, but in many cases, people want the models to hallucinate, to have a go. Often that’s the case in coding or when you’re trying to generate newer ideas. One eval that we added to artificial analysis is, is, is critical point and it’s really hard, uh, physics problems. Okay.

swyx [00:33:32]: And is it sort of like a human eval type or something different or like a frontier math type?

George [00:33:37]: It’s not dissimilar to frontier frontier math. So these are kind of research questions that kind of academics in the physics physics world would be able to answer, but models really struggled to answer. So the top score here is not 9%.

swyx [00:33:51]: And when the people that, that created this like Minway and, and, and actually off via who was kind of behind sweep and what organization is this? Oh, is this, it’s Princeton.

George [00:34:01]: Kind of range of academics from, from, uh, different academic institutions, really smart people. They talked about how they turn the models up in terms of the temperature as high temperature as they can, where they’re trying to explore kind of new ideas in physics as a, as a thought partner, just because they, they want the models to hallucinate. Um, yeah, sometimes it’s something new. Yeah, exactly.

swyx [00:34:21]: Um, so not right in every situation, but, um, I think it makes sense, you know, to test hallucination in scenarios where it makes sense. Also, the obvious question is, uh, this is one of. Many that there is there, every lab has a system card that shows some kind of hallucination number, and you’ve chosen to not, uh, endorse that and you’ve made your own. And I think that’s a, that’s a choice. Um, totally in some sense, the rest of artificial analysis is public benchmarks that other people can independently rerun. You provide it as a service here. You have to fight the, well, who are we to, to like do this? And your, your answer is that we have a lot of customers and, you know, but like, I guess, how do you converge the individual?

Micah [00:35:08]: I mean, I think, I think for hallucinations specifically, there are a bunch of different things that you might care about reasonably, and that you’d measure quite differently, like we’ve called this a amnesty and solutionation rate, not trying to declare the, like, it’s humanity’s last hallucination. You could, uh, you could have some interesting naming conventions and all this stuff. Um, the biggest picture answer to that. It’s something that I actually wanted to mention. Just as George was explaining, critical point as well is, so as we go forward, we are building evals internally. We’re partnering with academia and partnering with AI companies to build great evals. We have pretty strong views on, in various ways for different parts of the AI stack, where there are things that are not being measured well, or things that developers care about that should be measured more and better. And we intend to be doing that. We’re not obsessed necessarily with that. Everything we do, we have to do entirely within our own team. Critical point. As a cool example of where we were a launch partner for it, working with academia, we’ve got some partnerships coming up with a couple of leading companies. Those ones, obviously we have to be careful with on some of the independent stuff, but with the right disclosure, like we’re completely comfortable with that. A lot of the labs have released great data sets in the past that we’ve used to great success independently. And so it’s between all of those techniques, we’re going to be releasing more stuff in the future. Cool.

swyx [00:36:26]: Let’s cover the last couple. And then we’ll, I want to talk about your trends analysis stuff, you know? Totally.

Micah [00:36:31]: So that actually, I have one like little factoid on omniscience. If you go back up to accuracy on omniscience, an interesting thing about this accuracy metric is that it tracks more closely than anything else that we measure. The total parameter count of models makes a lot of sense intuitively, right? Because this is a knowledge eval. This is the pure knowledge metric. We’re not looking at the index and the hallucination rate stuff that we think is much more about how the models are trained. This is just what facts did they recall? And yeah, it tracks parameter count extremely closely. Okay.

swyx [00:37:05]: What’s the rumored size of GPT-3 Pro? And to be clear, not confirmed for any official source, just rumors. But rumors do fly around. Rumors. I get, I hear all sorts of numbers. I don’t know what to trust.

Micah [00:37:17]: So if you, if you draw the line on omniscience accuracy versus total parameters, we’ve got all the open ways models, you can squint and see that likely the leading frontier models right now are quite a lot bigger than the ones that we’re seeing right now. And the one trillion parameters that the open weights models cap out at, and the ones that we’re looking at here, there’s an interesting extra data point that Elon Musk revealed recently about XAI that for three trillion parameters for GROK 3 and 4, 6 trillion for GROK 5, but that’s not out yet. Take those together, have a look. You might reasonably form a view that there’s a pretty good chance that Gemini 3 Pro is bigger than that, that it could be in the 5 to 10 trillion parameters. To be clear, I have absolutely no idea, but just based on this chart, like that’s where you would, you would land if you have a look at it. Yeah.

swyx [00:38:07]: And to some extent, I actually kind of discourage people from guessing too much because what does it really matter? Like as long as they can serve it as a sustainable cost, that’s about it. Like, yeah, totally.

George [00:38:17]: They’ve also got different incentives in play compared to like open weights models who are thinking to supporting others in self-deployment for the labs who are doing inference at scale. It’s I think less about total parameters in many cases. When thinking about inference costs and more around number of active parameters. And so there’s a bit of an incentive towards larger sparser models. Agreed.

Micah [00:38:38]: Understood. Yeah. Great. I mean, obviously if you’re a developer or company using these things, not exactly as you say, it doesn’t matter. You should be looking at all the different ways that we measure intelligence. You should be looking at cost to run index number and the different ways of thinking about token efficiency and cost efficiency based on the list prices, because that’s all it matters.

swyx [00:38:56]: It’s not as good for the content creator rumor mill where I can say. Oh, GPT-4 is this small circle. Look at GPT-5 is this big circle. And then there used to be a thing for a while. Yeah.

Micah [00:39:07]: But that is like on its own, actually a very interesting one, right? That is it just purely that chances are the last couple of years haven’t seen a dramatic scaling up in the total size of these models. And so there’s a lot of room to go up properly in total size of the models, especially with the upcoming hardware generations. Yes.

swyx [00:39:29]: So, you know. Taking off my shitposting face for a minute. Yes. Yes. At the same time, I do feel like, you know, especially coming back from Europe, people do feel like Ilya is probably right that the paradigm is doesn’t have many more orders of magnitude to scale out more. And therefore we need to start exploring at least a different path. GDPVal, I think it’s like only like a month or so old. I was also very positive when it first came out. I actually talked to Tejo, who was the lead researcher on that. Oh, cool. And you have your own version.

George [00:39:59]: It’s a fantastic. It’s a fantastic data set. Yeah.

swyx [00:40:01]: And maybe it will recap for people who are still out of it. It’s like 44 tasks based on some kind of GDP cutoff that’s like meant to represent broad white collar work that is not just coding. Yeah.

Micah [00:40:12]: Each of the tasks have a whole bunch of detailed instructions, some input files for a lot of them. It’s within the 44 is divided into like two hundred and twenty two to five, maybe subtasks that are the level of that we run through the agenda. And yeah, they’re really interesting. I will say that it doesn’t. It doesn’t necessarily capture like all the stuff that people do at work. No avail is perfect is always going to be more things to look at, largely because in order to make the tasks well enough to find that you can run them, they need to only have a handful of input files and very specific instructions for that task. And so I think the easiest way to think about them are that they’re like quite hard take home exam tasks that you might do in an interview process.

swyx [00:40:56]: Yeah, for listeners, it is not no longer like a long prompt. It is like, well, here’s a zip file with like a spreadsheet or a PowerPoint deck or a PDF and go nuts and answer this question.

George [00:41:06]: OpenAI released a great data set and they released a good paper which looks at performance across the different web chat bots on the data set. It’s a great paper, encourage people to read it. What we’ve done is taken that data set and turned it into an eval that can be run on any model. So we created a reference agentic harness that can run. Run the models on the data set, and then we developed evaluator approach to compare outputs. That’s kind of AI enabled, so it uses Gemini 3 Pro Preview to compare results, which we tested pretty comprehensively to ensure that it’s aligned to human preferences. One data point there is that even as an evaluator, Gemini 3 Pro, interestingly, doesn’t do actually that well. So that’s kind of a good example of what we’ve done in GDPVal AA.

swyx [00:42:01]: Yeah, the thing that you have to watch out for with LLM judge is self-preference that models usually prefer their own output, and in this case, it was not. Totally.

Micah [00:42:08]: I think the way that we’re thinking about the places where it makes sense to use an LLM as judge approach now, like quite different to some of the early LLM as judge stuff a couple of years ago, because some of that and MTV was a great project that was a good example of some of this a while ago was about judging conversations and like a lot of style type stuff. Here, we’ve got the task that the grader and grading model is doing is quite different to the task of taking the test. When you’re taking the test, you’ve got all of the agentic tools you’re working with, the code interpreter and web search, the file system to go through many, many turns to try to create the documents. Then on the other side, when we’re grading it, we’re running it through a pipeline to extract visual and text versions of the files and be able to provide that to Gemini, and we’re providing the criteria for the task and getting it to pick which one more effectively meets the criteria of the task. Yeah. So we’ve got the task out of two potential outcomes. It turns out that we proved that it’s just very, very good at getting that right, matched with human preference a lot of the time, because I think it’s got the raw intelligence, but it’s combined with the correct representation of the outputs, the fact that the outputs were created with an agentic task that is quite different to the way the grading model works, and we’re comparing it against criteria, not just kind of zero shot trying to ask the model to pick which one is better.

swyx [00:43:26]: Got it. Why is this an ELO? And not a percentage, like GDP-VAL?

George [00:43:31]: So the outputs look like documents, and there’s video outputs or audio outputs from some of the tasks. It has to make a video? Yeah, for some of the tasks. Some of the tasks.

swyx [00:43:43]: What task is that?

George [00:43:45]: I mean, it’s in the data set. Like be a YouTuber? It’s a marketing video.

Micah [00:43:49]: Oh, wow. What? Like model has to go find clips on the internet and try to put it together. The models are not that good at doing that one, for now, to be clear. It’s pretty hard to do that with a code editor. I mean, the computer stuff doesn’t work quite well enough and so on and so on, but yeah.

George [00:44:02]: And so there’s no kind of ground truth, necessarily, to compare against, to work out percentage correct. It’s hard to come up with correct or incorrect there. And so it’s on a relative basis. And so we use an ELO approach to compare outputs from each of the models between the task.

swyx [00:44:23]: You know what you should do? You should pay a contractor, a human, to do the same task. And then give it an ELO and then so you have, you have human there. It’s just, I think what’s helpful about GDPVal, the OpenAI one, is that 50% is meant to be normal human and maybe Domain Expert is higher than that, but 50% was the bar for like, well, if you’ve crossed 50, you are superhuman. Yeah.

Micah [00:44:47]: So we like, haven’t grounded this score in that exactly. I agree that it can be helpful, but we wanted to generalize this to a very large number. It’s one of the reasons that presenting it as ELO is quite helpful and allows us to add models and it’ll stay relevant for quite a long time. I also think it, it can be tricky looking at these exact tasks compared to the human performance, because the way that you would go about it as a human is quite different to how the models would go about it. Yeah.

swyx [00:45:15]: I also liked that you included Lama 4 Maverick in there. Is that like just one last, like...

Micah [00:45:20]: Well, no, no, no, no, no, no, it is the, it is the best model released by Meta. And... So it makes it into the homepage default set, still for now.

George [00:45:31]: Other inclusion that’s quite interesting is we also ran it across the latest versions of the web chatbots. And so we have...

swyx [00:45:39]: Oh, that’s right.

George [00:45:40]: Oh, sorry.

swyx [00:45:41]: I, yeah, I completely missed that. Okay.

George [00:45:43]: No, not at all. So that, which has a checkered pattern. So that is their harness, not yours, is what you’re saying. Exactly. And what’s really interesting is that if you compare, for instance, Claude 4.5 Opus using the Claude web chatbot, it performs worse than the model in our agentic harness. And so in every case, the model performs better in our agentic harness than its web chatbot counterpart, the harness that they created.

swyx [00:46:13]: Oh, my backwards explanation for that would be that, well, it’s meant for consumer use cases and here you’re pushing it for something.

Micah [00:46:19]: The constraints are different and the amount of freedom that you can give the model is different. Also, you like have a cost goal. We let the models work as long as they want, basically. Yeah. Do you copy paste manually into the chatbot? Yeah. Yeah. That’s, that was how we got the chatbot reference. We’re not going to be keeping those updated at like quite the same scale as hundreds of models.

swyx [00:46:38]: Well, so I don’t know, talk to a browser base. They’ll, they’ll automate it for you. You know, like I have thought about like, well, we should turn these chatbot versions into an API because they are legitimately different agents in themselves. Yes. Right. Yeah.

Micah [00:46:53]: And that’s grown a huge amount of the last year, right? Like the tools. The tools that are available have actually diverged in my opinion, a fair bit across the major chatbot apps and the amount of data sources that you can connect them to have gone up a lot, meaning that your experience and the way you’re using the model is more different than ever.

swyx [00:47:10]: What tools and what data connections come to mind when you say what’s interesting, what’s notable work that people have done?

Micah [00:47:15]: Oh, okay. So my favorite example on this is that until very recently, I would argue that it was basically impossible to get an LLM to draft an email for me in any useful way. Because most times that you’re sending an email, you’re not just writing something for the sake of writing it. Chances are context required is a whole bunch of historical emails. Maybe it’s notes that you’ve made, maybe it’s meeting notes, maybe it’s, um, pulling something from your, um, any of like wherever you at work store stuff. So for me, like Google drive, one drive, um, in our super base databases, if we need to do some analysis or some data or something, preferably model can be plugged into all of those things and can go do some useful work based on it. The things that like I find most impressive currently that I am somewhat surprised work really well in late 2025, uh, that I can have models use super base MCP to query read only, of course, run a whole bunch of SQL queries to do pretty significant data analysis. And. And make charts and stuff and can read my Gmail and my notion. And okay. You actually use that. That’s good. That’s, that’s, that’s good. Is that a cloud thing? To various degrees of order, but chat GPD and Claude right now, I would say that this stuff like barely works in fairness right now. Like.

George [00:48:33]: Because people are actually going to try this after they hear it. If you get an email from Micah, odds are it wasn’t written by a chatbot.

Micah [00:48:38]: So, yeah, I think it is true that I have never actually sent anyone an email drafted by a chatbot. Yet.

swyx [00:48:46]: Um, and so you can, you can feel it right. And yeah, this time, this time next year, we’ll come back and see where it’s going. Totally. Um, super base shout out another famous Kiwi. Uh, I don’t know if you’ve, you’ve any conversations with him about anything in particular on AI building and AI infra.

George [00:49:03]: We have had, uh, Twitter DMS, um, with, with him because we’re quite big, uh, super base users and power users. And we probably do some things more manually than we should in. In, in super base support line because you’re, you’re a little bit being super friendly. One extra, um, point regarding, um, GDP Val AA is that on the basis of the overperformance of the models compared to the chatbots turns out, we realized that, oh, like our reference harness that we built actually white works quite well on like gen generalist agentic tasks. This proves it in a sense. And so the agent harness is very. Minimalist. I think it follows some of the ideas that are in Claude code and we, all that we give it is context management capabilities, a web search, web browsing, uh, tool, uh, code execution, uh, environment. Anything else?

Micah [00:50:02]: I mean, we can equip it with more tools, but like by default, yeah, that’s it. We, we, we give it for GDP, a tool to, uh, view an image specifically, um, because the models, you know, can just use a terminal to pull stuff in text form into context. But to pull visual stuff into context, we had to give them a custom tool, but yeah, exactly. Um, you, you can explain an expert. No.

George [00:50:21]: So it’s, it, we turned out that we created a good generalist agentic harness. And so we, um, released that on, on GitHub yesterday. It’s called stirrup. So if people want to check it out and, and it’s a great, um, you know, base for, you know, generalist, uh, building a generalist agent for more specific tasks.

Micah [00:50:39]: I’d say the best way to use it is get clone and then have your favorite coding. Agent make changes to it, to do whatever you want, because it’s not that many lines of code and the coding agents can work with it. Super well.

swyx [00:50:51]: Well, that’s nice for the community to explore and share and hack on it. I think maybe in, in, in other similar environments, the terminal bench guys have done, uh, sort of the Harbor. Uh, and so it’s, it’s a, it’s a bundle of, well, we need our minimal harness, which for them is terminus and we also need the RL environments or Docker deployment thing to, to run independently. So I don’t know if you’ve looked at it. I don’t know if you’ve looked at the harbor at all, is that, is that like a, a standard that people want to adopt?

George [00:51:19]: Yeah, we’ve looked at it from a evals perspective and we love terminal bench and, and host benchmarks of, of, of terminal mention on artificial analysis. Um, we’ve looked at it from a, from a coding agent perspective, but could see it being a great, um, basis for any kind of agents. I think where we’re getting to is that these models have gotten smart enough. They’ve gotten better, better tools that they can perform better when just given a minimalist. Set of tools and, and let them run, let the model control the, the agentic workflow rather than using another framework that’s a bit more built out that tries to dictate the, dictate the flow. Awesome.

swyx [00:51:56]: Let’s cover the openness index and then let’s go into the report stuff. Uh, so that’s the, that’s the last of the proprietary art numbers, I guess. I don’t know how you sort of classify all these. Yeah.

Micah [00:52:07]: Or call it, call it, let’s call it the last of like the, the three new things that we’re talking about from like the last few weeks. Um, cause I mean, there’s a, we do a mix of stuff that. Where we’re using open source, where we open source and what we do and, um, proprietary stuff that we don’t always open source, like long context reasoning data set last year, we did open source. Um, and then all of the work on performance benchmarks across the site, some of them, we looking to open source, but some of them, like we’re constantly iterating on and so on and so on and so on. So there’s a huge mix, I would say, just of like stuff that is open source and not across the side. So that’s a LCR for people. Yeah, yeah, yeah, yeah.

swyx [00:52:41]: Uh, but let’s, let’s, let’s talk about open.

Micah [00:52:42]: Let’s talk about openness index. This. Here is call it like a new way to think about how open models are. We, for a long time, have tracked where the models are open weights and what the licenses on them are. And that’s like pretty useful. That tells you what you’re allowed to do with the weights of a model, but there is this whole other dimension to how open models are. That is pretty important that we haven’t tracked until now. And that’s how much is disclosed about how it was made. So transparency about data, pre-training data and post-training data. And whether you’re allowed to use that data and transparency about methodology and training code. So basically, those are the components. We bring them together to score an openness index for models so that you can in one place get this full picture of how open models are.

swyx [00:53:32]: I feel like I’ve seen a couple other people try to do this, but they’re not maintained. I do think this does matter. I don’t know what the numbers mean apart from is there a max number? Is this out of 20?

George [00:53:44]: It’s out of 18 currently, and so we’ve got an openness index page, but essentially these are points, you get points for being more open across these different categories and the maximum you can achieve is 18. So AI2 with their extremely open OMO3 32B think model is the leader in a sense.

swyx [00:54:04]: It’s hooking face.

George [00:54:05]: Oh, with their smaller model. It’s coming soon. I think we need to run, we need to get the intelligence benchmarks right to get it on the site.

swyx [00:54:12]: You can’t have it open in the next. We can not include hooking face. We love hooking face. We’ll have that, we’ll have that up very soon. I mean, you know, the refined web and all that stuff. It’s, it’s amazing. Or is it called fine web? Fine web. Fine web.

Micah [00:54:23]: Yeah, yeah, no, totally. Yep. One of the reasons this is cool, right, is that if you’re trying to understand the holistic picture of the models and what you can do with all the stuff the company’s contributing, this gives you that picture. And so we are going to keep it up to date alongside all the models that we do intelligence index on, on the site. And it’s just an extra view to understand.

swyx [00:54:43]: Can you scroll down to this? The, the, the, the trade-offs chart. Yeah, yeah. That one. Yeah. This, this really matters, right? Obviously, because you can be super open, but dumb. I mean, obviously goes the wrong way here. Right.

George [00:54:55]: A lot of people would like to see labs hill climb on the, and target.

Micah [00:55:00]: This is the access to hill climb. Yeah. Unfortunately, it might be fundamentally true that the, the slum will always go this direction because once you open something up, then everyone else can get to the level of what you have now.

swyx [00:55:11]: Well, so let me, let me tweak your points. You have, I have a point system, right? Like you have these like numbers on the point system and it go up to 18, you know, but like, just because I have a little bit of open data doesn’t mean I’m necessarily that much better in someone who put a lot of effort into their open ways, it is that it’s smarter. So I might, I might just mess with the point system to make sure that like, I’m accurately representing the, the contribution to the open openness.

Micah [00:55:36]: It is hard to wait for the materiality of the contribution to open source. We tried to make it so that it is quite well-defined and no one can disagree about which category things should be in. So we’re not saying this was a big contribution or a small contribution in terms of impact on the industry or anything. It’s just how much of your data did you release? I would say that it is still valid to say that we trained a model that’s not that smart, maybe even not at the frontier for a particular size category, but we chose to open up all the data, all the training code. That is a very useful exercise for the industry. And we want to recognize that even if the smartest model in the category.

swyx [00:56:18]: Yeah. And also a special shout out to NVIDIA and Emotron, which doesn’t get enough credit for the amount of stuff that they do. And honestly, it’s a sales enablement for NVIDIA as well. The fact that they can do this is... Side project.

Micah [00:56:29]: Totally. But I mean, it is true that NVIDIA have actually put an enormous amount of effort over the last year, especially into the Nematron models.

swyx [00:56:35]: Yeah. And so many people actually use it for synthetic data and stuff. It’s a pretty interesting secret of the industry that NVIDIA holds up all these guys.

Micah [00:56:45]: I mean, it’s in their interest for there to be more AI.

swyx [00:56:49]: So obviously, I think you want to push openness as having an index. Every index that you push has encoded some kind of opinion or value. Yes. I think one of the openness questions from this year was people messing with the license. And so Lama had this, like, if you have 700 million daily active users, you’re not allowed to use our model or you have to talk to us, something like that. So basically, like, what are your customers telling you about the kind of licensing worries that they have? Right. Because obviously, most people will never hit 700 million users.

Micah [00:57:21]: We have like a detailed breakdown of that in the openness index. And that was actually one of the initial questions that took us down the route of wanting to... Do this. Because, yeah, the simplest thing that, like, our opinion is, is that there is a lot of advantage to having, like, an official OSI license like MIT or Apache 2, because then the box is just checked. You don’t even need to read it because it’s just Apache 2 and you can do it ever you want and it’s fine. There are often very good reasons that companies don’t want to release language models with those completely open licenses. The index tells you. So if you get the top category, that’s one of those licenses. You’re totally good. And then... And then we’ve got some lower categories for when attribution is required and then when commercial use is not allowed. Yeah, they’re there.

swyx [00:58:05]: So that’s the openness index. Thank you for doing all those works. Let’s talk a little bit, or at least end the pod, on just the trend reports that you guys do, which is kind of a bit of the bread and butter how you make money. I highly encourage everyone to see George’s talk at World’s Fair, which gives a little bit of a preview. And you were very excited about talking about the smiling curve, or I don’t know what you call it. Yeah, yeah, yeah, yeah, let’s talk about that one. Let’s explain it for people. And I might, I might actually put it up because I don’t have it. Yeah, I’ve got to copy the slide, that’d be, that’d be excellent. It’s important for people to have in their head because, yeah, people only get the marketing message from the labs that, oh, we’re cutting costs all the time.

Micah [00:58:41]: Yeah, yeah, but it’s, it’s true. It’s it’s not the whole picture. So, okay. A couple of like the big trends that we track at Artificial Analysis over time and that like we’re always showing charts of on the trends page in these reports and stuff. One, that the cost of intelligence has been falling dramatically. Over the last couple of years, the best way to think about that is that the cost for each terror of intelligence has been dropping the, like one fact on that is that you can get intelligence at the level of GPT-4 for over a hundred times cheaper than GPT-4 was at launch right now. I think my number is a thousand actually.

swyx [00:59:16]: If you look at the Amazon Nova models, which are very, very cheap. Yeah.

Micah [00:59:21]: Like my, my conservative statement is normally like, but in fairness, this slide. Like I, we were actually saying for the podcast, right. It’s like maybe six months old now and it’s conceptually still correct, but like could actually probably do a tweak on the exact numbers because like the market’s moving so quickly.

swyx [00:59:37]: If you’re feeling kick it off, I mean, we’ll have this chart.

Micah [00:59:39]: I told people to watch the world’s fair talk, but let’s, let’s introduce what context makes you make something like this. There are two trends that seem to not make sense together, both of which we talk a lot about at Artificial Analysis and are very important to developers building stuff in AI. The first is that the cost of intelligence for each level of intelligence has been dropping dramatically over the last couple of years. We track the cost to run Artificial Analysis Intelligence Index for each bucket of Intelligence Index scores and each bucket, you just see the line go down really, really quickly and actually go down more quickly to each new level of intelligence that’s been achieved over the last couple of years. So the rate of that cost has actually been going up. So. Yeah. We’ve got that being true. And yet it is clearly possible to spend quite a lot more on AI inference now than it was a couple of years ago.

George [01:00:34]: NVIDIA stock go up.

swyx [01:00:36]: It’s going, it’s going really up. Uh, I just heard from a friend’s startup that just went through the shift zero. They’re spending $5,000 per employee on coding agents spend alone. That’s ridiculous. That’s an impressive number.

Micah [01:00:49]: We need to get our numbers up. We’re, uh, we’re, we’re not quite hitting, hitting.

swyx [01:00:52]: Well, I was like, it’s so high down. I’m like, are you doing something wrong? Yeah.

Micah [01:00:55]: Cause there are some efficiency questions along the way, but like you can make AI inference useful to that level in a bunch of ways that I can imagine. Right. Yeah. Um, I, I don’t think that’s that nuts. Um, but basically the, the reason we made this slide to answer the question, right. Is to show that the crazy thing is that it is actually true. We’ve had this hundred X to a thousand X decline in the cost of GPT four level intelligence on the left-hand side. And yet on the right-hand side, because the multipliers are so big for the fact that even though. Small models can do GPT four level. Now we still want to use big models and probably bigger than ever models to, um, do frontier level intelligence. We’ve got reasoning models using tokens, and then we’re throwing them in these, them in these agentic workflows where they’re consuming enormous numbers of input tokens and making enormous numbers of output tokens working for a really long time. Those two things taken together, get you back to, we can spend enormously more today than we could a couple of years ago. Yep.

George [01:01:50]: I think that’s right. There’s a number of drivers at play and we kind of outline kind of. Six key ones here. Um, but you know, as complex as changing quickly, all of these have changed very dramatically in the last, uh, in the last 12 months.

swyx [01:02:04]: Let’s pick on hardware efficiency since you also have, you also track hardware stuff. And I think the general assertion or the message is that the efficiency from next gen Nvidia chips is actually not 4X. So you have what? 3X or 4X? You have 3X in here and it’s, it’s like 2X maybe, or it’s more of like a. Power story rather than like a share sort of compute tokens efficiency story. But yeah, what, what’s going on in, in hardware. Okay.

Micah [01:02:31]: So the, the, the, the odds, unfortunately, uh, is it depends and it just depends massively on like so many things across a bunch of different types of workloads and ways to think about it. So one of the simplest ways to think about this is to take single relevant model, to think about serving it at speeds that are realistic for what you actually might want to hit. And can afford to hit, and then think about the throughput per GPU that you can achieve serving the model at those speeds. Rease. One of the reasons that’s important is that there’s a trade-off between the throughput per GPU that you can achieve and the per user speed that you can achieve. And as a, it costs more to serve stuff fast to, to users. When you run all of that for especially big sparse models, you can get a lot better than two or three X gain going from Hopper to Blackwell generation to video. I am. This shouldn’t be too controversial. Let’s say I’m like, I’m. I’m pretty confident that Blackwell has delivered pretty enormous gains and that the next couple of years of NVIDIA’s roadmap are going to continue to deliver quite enormous gains and that those will actually come through as lower total cost per token to the companies that are running models on them and will allow bigger models will allow way more tokens to be made for lower cost and that that’s gonna continue these things also stack on all of the software and model improvements. So basically like my prediction across like both sides of that, like smile chart, uh, that we’re gonna see the left-hand side continue to be true and probably like for another order of magnitude and the right-hand side continue to be true for another order of magnitude, and that’s gonna enable a whole lot of things.

swyx [01:04:12]: Okay. Well, I’ll push on, uh, let’s go back to the, the, the small chart. I’ll push back on sparsity, right? Uh, we’ve gone a long way on sparsity. Deep seek was a major pusher of fine grain experts. Let’s call it. Yep. Right. Well, I have a mental number of sparsity in terms of let’s say active params versus total params. And that number went from 25%, let’s say down to like 15, right? You obviously can’t really go below, I don’t know, five. Is that obvious? So there’s a lower limit to, to sparsity is what I’m saying. I don’t know that that’s that obvious actually. All right.

Micah [01:04:45]: Um, there, there must be a limit somewhere, right? Yeah, exactly. But we’ve got numbers in the wild that are quite a lot lower than that right now. So the GBD OSS models, like the big ones at about 5%, um, active, Kimmy K2, is it like 3% active? Oh, okay. I think, pretty sure.

swyx [01:05:05]: I’ve looked at those numbers. I calculated them. I don’t remember. Yeah. But I remember thinking like, this must be it.

George [01:05:11]: Your 5% is exactly like around the ballpark for the open weights models of, of what’s released today. I think one interesting that gives me kind of pause when thinking that it won’t go, the sparsity won’t go high. Or the number of percentage of active parameters lower is that we, in our benchmark, see a lot of performance, uh, correlated more with, uh, total parameters than active and not that correlated with how sparse, like the models are. Our accuracy, benchmark as part of a omniscience, it’s very correlated with total. It’s not correlated with, with active, uh, parameters, which I think is very at all, which is very, very interesting. And so I think, yeah, they could, they could be quite. A bit, um, to go here. Awesome.

swyx [01:05:55]: Well, we don’t have that much time, but I w I did want to leave some room to cover reasoning and non-reasoning models and token efficiency. Let’s do that. So at a high, at a super high level, people have to classify this binary thing of reasoning versus non-reasoning. People who are insider have some discomfort with that because basically you just have to think tag or no think tag. How have you guys decided to approach this? And also how does that laid out in, over the course of the year where we have things like GPT-5, which is a model. Right.

Micah [01:06:24]: Let’s say GPT-5 in chat GPT, the consumer experience as a model router, when you’re hitting the API, like we can, you can pick the different versions and you can pick reasoning strength of the different versions, but that, that goes to why this is now such a complex thing. So earlier this year, and probably when you and George last spoke for the AI engineers world’s fair, we had this great slide that was super easy, where we would show that the average reasoning model is using 10 times the number of tokens per query in our intelligence index as the average non-reasoning model. And there was this moment where that was a pretty clear distinction and extremely useful to look at it just like that. Definitely no longer the case, not least because you can think about reasoning strength for a bunch of these different models, but particularly because different models have wildly different token efficiency now, more than an order of magnitude in difference. That means that the way that you probably need to think about cost for any application is to use something like our cost around intelligence index metric as the starting point. Right. for what it’s going to look like for these different models, these different reasoning strengths, and this continuous spectrum from non-reasoning to reasoning. That’s basically like where we’re at. So we will still show reasoning and non-reasoning and define reasoning as when there is that separated chain of thought that you’re getting at a different parameter in an API normally, but it doesn’t necessarily anymore mean that that model is actually going to have longer end-to-end latency that is going to use more tokens than something that is branded

swyx [01:07:51]: in a non-reasoning model for the same task. That’s true. I think 5.1 was it. And then 5.1 Codex had these chart, which was super nice of this, like, let’s say bottom 10 percentile query being faster, but top 10 percentile being longer. And that’s a kind of the efficiency chart

Micah [01:08:10]: you want to see, right? Yeah. So that is an extra thing. Let’s say that we’ve got, that’s a really important extra thing though, right? That you’ve got not just the average number of token span used by the model, which we cover really well right now, but the behavior that you want in the model is it to use more tokens when it needs more tokens and not to use more tokens when it doesn’t need more tokens. So that’s what OpenAI, we’re basically claiming that 5.1 Codex is better at. We don’t actually publish anything on this right now, but have tracked it a bunch internally in our internal analytics on evals across all the models that we run, where we look at the difficulty to questions and the correlation between token usage and difficulty and net net, surprise, surprise, like models have got. I think going into next year, that’s going to be really important, especially as you multiply it by the number of steps in an agentic workflow that a model has to take to get to an answer. We are going to care a lot about token efficiency and number of turns efficiency for getting to what

swyx [01:09:08]: we want. Which would you rather have token efficiency or number of turns efficiency? Or like, which is more important to work on?

Micah [01:09:16]: it depends on the application and both are going to be really important.

George [01:09:18]: Uh, yeah.

Micah [01:09:20]: Well, total cost is just-

swyx [01:09:21]: TalBench Retail, TalBench Airline.

George [01:09:23]: Yeah. Interestingly in Tal, um, Tal2Bench Telecom, it’s cheaper to run, you know, on a per token basis, more expensive models like a GBD5 compared to some smaller open source models, because the, um, some of the GBD5, for instance, uh, got to the answer faster. And so it was able to resolve the customer’s query faster and fewer turns. And maybe it used more tokens per turn, but it certainly- It’s not going to cost more per token. So you would always rather use GBD5 in, in, in, in that scenario. And so I think that’s what, that’s where we’re getting to. I think number of turns is, it’s going to be a metric that we’re going to be talking about a lot more. And, uh, I think it’ll be something that people want to really start to think about, uh, a lot more.

swyx [01:10:06]: There’s a trade-off in benchmarking here where most benchmarks needs to be one turn to be autonomous, to be parallelized and all that. But most, a lot of real life use cases need to be multi-turn and especially like quick multi-turns. So you can align. Yeah.

Micah [01:10:19]: Yeah. I mean, I, I would say that historically benchmarks have been single turn, but I wouldn’t say they need to be at all into the future, right? Like we have a couple of agentic benchmarks in the index right now and GDP that we were talking about. We let the models do up to a hundred turns and, um, our stirrup agent to do that evil. And we’re going to build similar stuff like that in the future. It definitely is hard and you’ve got whole kinds of infrastructure problems to run that and exactly as you say, parallelize it because we need to run that on hundreds of models and we want to do that really fast when you want us to come out and with labs want us to run it on their models,

swyx [01:10:53]: but you can do it. We’re putting in the work to build that stuff and it’s going to be great. Okay. So we’ve covered, I mean, there’s a lot more to cover and you haven’t even touched on

George [01:11:01]: multimodal, which is huge. We also do speech benchmarking, image benchmarking, uh, video

swyx [01:11:09]: benchmarking, hardware. I like the way that you’ve done it because they’re very smart, which is a video takes a long time. So you pre-generate, right? So then people just pick their preferences and you can see the, the overall arena results. And you also avoid like any sensitivity issues

Micah [01:11:23]: around the unsafe content that is being generated. Yeah. And you can see it as a good, good thing, a bad thing, depending on what your view is. But it means that we have a quite active creative direction approach to trying to understand what creative professionals and users want to do with those image and video models. And so that we can be directing the arenas in our categories toward gathering data, votes on what people care about. One call out actually to listeners, like if you are using our arenas is that you can submit requests to us for things that we should cover. I didn’t know that. Yeah. Understudied categories, areas that you think the models are bad at and the labs don’t focus on enough. Like if you want something solved, one of the levers that you have is send us a couple of prompts on it. We might be able to get a category going on it. And this thing that we were talking about earlier, right? That once things get measured, they can get targeted. You can make that work for you.

swyx [01:12:18]: For me as a content creator, infographics, very needed. I took the latest deep seek paper and they had some descriptions of their search agents and their coding agents and I put it in and I created an infographic. And I just think like as I said, industrial use case that doesn’t require a lot of, I guess, design tastes, but just requires some, you need to conform to some preset references, which is something that is increasingly important, especially in like the nano banana series. But yeah, and I think that’s the key there. I think it’s important to be able to I think OpenAI is releasing Image 2 soon, which is going to have that. So I think it’s all of a kind where people need to incentivize workhorse use cases and not just art. I don’t know. Totally. Yeah. What are we going to be talking about next year? What’s emerging that you’re seeing and maybe not in the discussion?

Micah [01:13:06]: The first answer that I’ll give to that is the boring answer is that on most of our charts, the lines go in a particular direction and our overall prediction is the lines are going to keep going in that direction. We’re going to do a lot and do a lot to be as useful as possible to developers and companies to measure what’s important on every one of those and along those lines. But I think we’re going to talk about similar stuff. It’s just that we’re going to have continued on this trajectory for another year and things are going to feel pretty different because of that happening. I know this is the boring answer to that question. No, no.

swyx [01:13:36]: I mean, I’m a fan of things that, truths that don’t change because you can build and plan for that. And I think in media in general, in the podcast business, newsletters, you know, there’s a Twitter business, Twitter business, people are addicted to change, like, oh, everything’s breaking. Everything’s, no, like there’s some truths that aren’t just constants that you can plan on and build. And yeah.

George [01:13:58]: I think one of the truths is that the demand for AI intelligence and smarter AI intelligence is going to be insatiable. Some people disagree that, okay, once we reach certain thresholds, then you don’t need more intelligence. I think to that, I ask people, have they ever worked with? Or managed someone in a work environment and wouldn’t press the button that they were smarter to make them smarter or better at their job or would they never press that for themselves? And I’m not sure that that’s, that’s the case, but I think for artificial analysis, we’ll keep benchmarking raw intelligence, but we also want to think about it and explore models more deeply across other axes as well. I think hallucinations, the start of that, but we’re getting into wanting to support people and understanding, okay, the behavior, the person personalities. Of the models to help people make more nuanced decisions, you’re going to have a personality bench.

swyx [01:14:52]: Maybe that is a direction that Chadji opening eyes leaning into a lot. So if you manage to solve that, you should definitely talk to Fiji and Roon. Oh, okay. Yeah. So what is going to be included in, let’s say like a V3 of the intelligence index, because obviously you’re going to saturate in March.

Micah [01:15:10]: Why don’t we break it now? How soon is the podcast going to come out? Whenever you want. Okay. So we’re at V3 right now. So the, so the, the, the version that we, that’s going inside is, is, is, is V3 V4 is what we’re going to call the next, you know, major of it. Surprise, surprise. We’re going to be adding several of the things that we’ve actually talked about today that we’ve launched over the last few weeks. So it’s not, that’s not going to be wildly shocking, but some of the things that are most exciting is that adding GDP value is going to give us this general agentic performance in a really strong way in intelligence index and in critical point, the, um, physics, EBL, George was talking about similar to frontier math. Yeah. That’s very interesting. That gives us completely new view with a brand new data set of very, very hard research problems. We are going to be using Omniscience and we are going to be using hallucination rate. The exact way is that all of those are going to come together. Um, The waitings is going to be hard because the numbers are different. Yeah. We’re going to make sure that we don’t do anything to cause odd distortions and stuff that could be misleading. But every time you version it, you have a one-time reset of the Exactly. Yeah. That’s exactly how we think about it. We will make sure that within each version number that there’s no parliamentary issue. No drift in any of the scores so that people can rely on them and reference them. You just have to watch out for that version number. Once it’s v4.1, those numbers won’t be compatible with v4.

swyx [01:16:23]: Of course. There’s a little bit of debate over the accuracy of TileBench. I don’t know if you’re clued in to what’s going on. Apparently, a very high number of TileBench tests are impossible.

Micah [01:16:34]: Potentially for the earlier versions, Tile2Bench Telecom, we’re pretty convinced is pretty good. If anything, the only issue there is that models have got very good at doing it. And so, like anything... Tile3. Yeah.

swyx [01:16:49]: On we go. Yeah, on we go. Okay, well, thank you so much for providing such a great service to the industry. I’m glad to at least know you guys before you got famous and now you are famous.

Micah [01:16:59]: Oh, look, our pleasure. We really appreciate your support along the way. I wasn’t kidding at the start, right? That it was a quite material moment for us when artificial analysis was covered on Latent Space. Some random guy. And San Francisco mentions you. I was a fan of Latent Space for like a year before you mentioned us. So, I’d been listening. I don’t think I was familiar with you personally yet at that point. But I listened to your voice probably for many, many hours. And so, once you mentioned it, I got to get to know you and meet you for the first time nearly a couple of years ago. It was really cool, honestly. So, yeah, it’s great to be here.

George [01:17:36]: And thanks for being such a great member of the community and kind of spotlighting projects, projects which don’t have attention and bringing them to your audience. Yeah.

swyx [01:17:44]: Well, actually, so it wasn’t me, right? Someone in the Discord dropped it in our Discord. And I rely on our community and it kind of feeds itself, right? Nice. So, someone brought it to my attention. I don’t know who. We should probably go back and check. But once I saw it, I was like, this looks good. This is something I always wanted. I wanted to build it. I was too shy or dumb or lazy to build it. And you guys did. And now it’s a whole thing. So, thank you for being here.

George [01:18:08]: I built some really cool other stuff like this pod. Yeah. Yeah. Totally. So, thank you. That’s it. Great. Cool. Thanks.

We are reupping this episode after LMArena announced their fresh Series A (https://www.theinformation.com/articles/ai-evaluation-startup-lmarena-valued-1-7-billion-new-funding-round?rc=luxwz4), raising $150m at a $1.7B valuation, with $30M annualized consumption revenue (aka $2.5m MRR) after their September evals product launch.

—-

From building LMArena in a Berkeley basement to raising $100M and becoming the de facto leaderboard for frontier AI, Anastasios Angelopoulos returns to Latent Space to recap 2025 in one of the most influential platforms in AI—trusted by millions of users, every major lab, and the entire industry to answer one question: which model is actually best for real-world use cases? We caught up with Anastasios live at NeurIPS 2025 to dig into the origin story (spoiler: it started as an academic project incubated by Anjney Midha at a16z, who formed an entity and gave grants before they even committed to starting a company), why they decided to spin out instead of staying academic or nonprofit (the only way to scale was to build a company), how they’re spending that $100M (inference costs, React migration off Gradio, and hiring world-class talent across ML, product, and go-to-market), the leaderboard delusion controversy and why their response demolished the paper’s claims (factual errors, misrepresentation of open vs. closed source sampling, and ignoring the transparency of preview testing that the community loves), why platform integrity comes first (the public leaderboard is a charity, not a pay-to-play system—models can’t pay to get on, can’t pay to get off, and scores reflect millions of real votes), how they’re expanding into occupational verticals (medicine, legal, finance, creative marketing) and multimodal arenas (video coming soon), why consumer retention is earned every single day (sign-in and persistent history were the unlock, but users are fickle and can leave at any moment), and his vision for Arena as the central evaluation platform that provides the North Star for the industry—constantly fresh, immune to overfitting, and grounded in millions of real-world conversations from real users.

We discuss:

• The $100M raise: use of funds is primarily inference costs (funding free usage for tens of millions of monthly conversations), React migration off Gradio (custom loading icons, better developer hiring, more flexibility), and hiring world-class talent

• The scale: 250M+ conversations on the platform, tens of millions per month, 25% of users do software for a living, and half of users are now logged in

• The leaderboard illusion controversy: Cohere researchers claimed undisclosed private testing created inequities, but Arena’s response demolished the paper’s factual errors (misrepresented open vs. closed source sampling, ignored transparency of preview testing that the community loves)

• Why preview testing is loved by the community: secret codenames (Gemini Nano Banana, named after PM Naina’s nickname), early access to unreleased models, and the thrill of being first to vote on frontier capabilities

• The Nano Banana moment: changed Google’s market share overnight, billions of dollars in stock movement, and validated that multimodal models (image generation, video) are economically critical for marketing, design, and AI-for-science

• New categories: occupational and expert arenas (medicine, legal, finance, creative marketing), Code Arena, and video arena coming soon

00:00:00 Introduction: Anastasios from Arena and the LM Arena Journey
00:01:36 The Anjney Midha Incubation: From Berkeley Basement to Startup
00:02:47 The Decision to Start a Company: Scaling Beyond Academia
00:03:38 The $100M Raise: Use of Funds and Platform Economics
00:05:10 Arena's User Base: 5M+ Users and Diverse Demographics
00:06:02 The Competitive Landscape: Artificial Analysis, AI.xyz, and Arena's Differentiation
00:08:12 Educational Value and Learning from the Community
00:08:41 Technical Migration: From Gradio to React and Platform Evolution
00:10:18 Leaderboard Delusion Paper: Addressing Critiques and Maintaining Integrity
00:12:29 Nano Banana Moment: How Preview Models Create Market Impact
00:13:41 Multimodal AI and Image Generation: From Skepticism to Economic Value
00:15:37 Core Principles: Platform Integrity and the Public Leaderboard as Charity
00:18:29 Future Roadmap: Expert Categories, Multimodal, Video, and Occupational Verticals
00:19:10 API Strategy and Focus: Doing One Thing Well
00:19:51 Community Management and Retention: Sign-In, History, and Daily Value
00:22:21 Partnerships and Agent Evaluation: From Devon to Full-Featured Harnesses
00:21:49 Hiring and Building a High-Performance Team

From undergraduate research seminars at Princeton to winning Best Paper award at NeurIPS 2025, Kevin Wang, Ishaan Javali, Michał Bortkiewicz, Tomasz Trzcinski, Benjamin Eysenbach defied conventional wisdom by scaling reinforcement learning networks to 1,000 layers deep—unlocking performance gains that the RL community thought impossible. We caught up with the team live at NeurIPS to dig into the story behind RL1000: why deep networks have worked in language and vision but failed in RL for over a decade (spoiler: it’s not just about depth, it’s about the objective), how they discovered that self-supervised RL (learning representations of states, actions, and future states via contrastive learning) scales where value-based methods collapse, the critical architectural tricks that made it work (residual connections, layer normalization, and a shift from regression to classification), why scaling depth is more parameter-efficient than scaling width (linear vs. quadratic growth), how Jax and GPU-accelerated environments let them collect hundreds of millions of transitions in hours (the data abundance that unlocked scaling in the first place), the “critical depth” phenomenon where performance doesn’t just improve—it multiplies once you cross 15M+ transitions and add the right architectural components, why this isn’t just “make networks bigger” but a fundamental shift in RL objectives (their code doesn’t have a line saying “maximize rewards”—it’s pure self-supervised representation learning), how deep teacher, shallow student distillation could unlock deployment at scale (train frontier capabilities with 1000 layers, distill down to efficient inference models), the robotics implications (goal-conditioned RL without human supervision or demonstrations, scaling architecture instead of scaling manual data collection), and their thesis that RL is finally ready to scale like language and vision—not by throwing compute at value functions, but by borrowing the self-supervised, representation-learning paradigms that made the rest of deep learning work.

We discuss:

• The self-supervised RL objective: instead of learning value functions (noisy, biased, spurious), they learn representations where states along the same trajectory are pushed together, states along different trajectories are pushed apart—turning RL into a classification problem

• Why naive scaling failed: doubling depth degraded performance, doubling again with residual connections and layer norm suddenly skyrocketed performance in one environment—unlocking the “critical depth” phenomenon

• Scaling depth vs. width: depth grows parameters linearly, width grows quadratically—depth is more parameter-efficient and sample-efficient for the same performance

• The Jax + GPU-accelerated environments unlock: collecting thousands of trajectories in parallel meant data wasn’t the bottleneck, and crossing 15M+ transitions was when deep networks really paid off

• The blurring of RL and self-supervised learning: their code doesn’t maximize rewards directly, it’s an actor-critic goal-conditioned RL algorithm, but the learning burden shifts to classification (cross-entropy loss, representation learning) instead of TD error regression

• Why scaling batch size unlocks at depth: traditional RL doesn’t benefit from larger batches because networks are too small to exploit the signal, but once you scale depth, batch size becomes another effective scaling dimension

—

RL1000 Team (Princeton)

• 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities: https://openreview.net/forum?id=s0JVsx3bx1

00:00:00 Introduction: Best Paper Award and NeurIPS Poster Experience
00:01:11 Team Introductions and Princeton Research Origins
00:03:35 The Deep Learning Anomaly: Why RL Stayed Shallow
00:04:35 Self-Supervised RL: A Different Approach to Scaling
00:05:13 The Breakthrough Moment: Residual Connections and Critical Depth
00:07:15 Architectural Choices: Borrowing from ResNets and Avoiding Vanishing Gradients
00:07:50 Clarifying the Paper: Not Just Big Networks, But Different Objectives
00:08:46 Blurring the Lines: RL Meets Self-Supervised Learning
00:09:44 From TD Errors to Classification: Why This Objective Scales
00:11:06 Architecture Details: Building on Braw and SymbaFowl
00:12:05 Robotics Applications: Goal-Conditioned RL Without Human Supervision
00:13:15 Efficiency Trade-offs: Depth vs Width and Parameter Scaling
00:15:48 JAX and GPU-Accelerated Environments: The Data Infrastructure
00:18:05 World Models and Next State Classification
00:22:37 Unlocking Batch Size Scaling Through Network Capacity
00:24:10 Compute Requirements: State-of-the-Art on a Single GPU
00:21:02 Future Directions: Distillation, VLMs, and Hierarchical Planning
00:27:15 Closing Thoughts: Challenging Conventional Wisdom in RL Scaling

From creating SWE-bench in a Princeton basement to shipping CodeClash, SWE-bench Multimodal, and SWE-bench Multilingual, John Yang has spent the last year and a half watching his benchmark become the de facto standard for evaluating AI coding agents—trusted by Cognition (Devin), OpenAI, Anthropic, and every major lab racing to solve software engineering at scale. We caught up with John live at NeurIPS 2025 to dig into the state of code evals heading into 2026: why SWE-bench went from ignored (October 2023) to the industry standard after Devin’s launch (and how Walden emailed him two weeks before the big reveal), how the benchmark evolved from Django-heavy to nine languages across 40 repos (JavaScript, Rust, Java, C, Ruby), why unit tests as verification are limiting and long-running agent tournaments might be the future (CodeClash: agents maintain codebases, compete in arenas, and iterate over multiple rounds), the proliferation of SWE-bench variants (SWE-bench Pro, SWE-bench Live, SWE-Efficiency, AlgoTune, SciCode) and how benchmark authors are now justifying their splits with curation techniques instead of just “more repos,” why Tau-bench’s “impossible tasks” controversy is actually a feature not a bug (intentionally including impossible tasks flags cheating), the tension between long autonomy (5-hour runs) vs. interactivity (Cognition’s emphasis on fast back-and-forth), how Terminal-bench unlocked creativity by letting PhD students and non-coders design environments beyond GitHub issues and PRs, the academic data problem (companies like Cognition and Cursor have rich user interaction data, academics need user simulators or compelling products like LMArena to get similar signal), and his vision for CodeClash as a testbed for human-AI collaboration—freeze model capability, vary the collaboration setup (solo agent, multi-agent, human+agent), and measure how interaction patterns change as models climb the ladder from code completion to full codebase reasoning.

We discuss:

• John’s path: Princeton → SWE-bench (October 2023) → Stanford PhD with Diyi Yang and the Iris Group, focusing on code evals, human-AI collaboration, and long-running agent benchmarks

• The SWE-bench origin story: released October 2023, mostly ignored until Cognition’s Devin launch kicked off the arms race (Walden emailed John two weeks before: “we have a good number”)

• SWE-bench Verified: the curated, high-quality split that became the standard for serious evals

• SWE-bench Multimodal and Multilingual: nine languages (JavaScript, Rust, Java, C, Ruby) across 40 repos, moving beyond the Django-heavy original distribution

• The SWE-bench Pro controversy: independent authors used the “SWE-bench” name without John’s blessing, but he’s okay with it (”congrats to them, it’s a great benchmark”)

• CodeClash: John’s new benchmark for long-horizon development—agents maintain their own codebases, edit and improve them each round, then compete in arenas (programming games like Halite, economic tasks like GDP optimization)

• SWE-Efficiency (Jeffrey Maugh, John’s high school classmate): optimize code for speed without changing behavior (parallelization, SIMD operations)

• AlgoTune, SciCode, Terminal-bench, Tau-bench, SecBench, SRE-bench: the Cambrian explosion of code evals, each diving into different domains (security, SRE, science, user simulation)

• The Tau-bench “impossible tasks” debate: some tasks are underspecified or impossible, but John thinks that’s actually a feature (flags cheating if you score above 75%)

• Cognition’s research focus: codebase understanding (retrieval++), helping humans understand their own codebases, and automatic context engineering for LLMs (research sub-agents)

• The vision: CodeClash as a testbed for human-AI collaboration—vary the setup (solo agent, multi-agent, human+agent), freeze model capability, and measure how interaction changes as models improve

—

John Yang

• SWE-bench: https://www.swebench.com

• X: https://x.com/jyangballin

00:00:00 Introduction: John Yang on SWE-bench and Code Evaluations
00:00:31 SWE-bench Origins and Devon's Impact on the Coding Agent Arms Race
00:01:09 SWE-bench Ecosystem: Verified, Pro, Multimodal, and Multilingual Variants
00:02:17 Moving Beyond Django: Diversifying Code Evaluation Repositories
00:03:08 Code Clash: Long-Horizon Development Through Programming Tournaments
00:04:41 From Halite to Economic Value: Designing Competitive Coding Arenas
00:06:04 Ofir's Lab: SWE-ficiency, AlgoTune, and SciCode for Scientific Computing
00:07:52 The Benchmark Landscape: TAU-bench, Terminal-bench, and User Simulation
00:09:20 The Impossible Task Debate: Refusals, Ambiguity, and Benchmark Integrity
00:12:32 The Future of Code Evals: Long Autonomy vs Human-AI Collaboration
00:14:37 Call to Action: User Interaction Data and Codebase Understanding Research

From pre-training data curation to shipping GPT-4o, o1, o3, and now GPT-5 thinking and the shopping model, Josh McGrath has lived through the full arc of OpenAI’s post-training evolution—from the PPO vs DPO debates of 2023 to today’s RLVR era, where the real innovation isn’t optimization methods but data quality, signal trust, and token efficiency. We sat down with Josh at NeurIPS 2025 to dig into the state of post-training heading into 2026: why RLHF and RLVR are both just policy gradient methods (the difference is the input data, not the math), how GRPO from DeepSeek Math was underappreciated as a shift toward more trustworthy reward signals (math answers you can verify vs. human preference you can’t), why token efficiency matters more than wall-clock time (GPT-5 to 5.1 bumped evals and slashed tokens), how Codex has changed his workflow so much he feels “trapped” by 40-minute design sessions followed by 15-minute agent sprints, the infrastructure chaos of scaling RL (”way more moving parts than pre-training”), why long context will keep climbing but agents + graph walks might matter more than 10M-token windows, the shopping model as a test bed for interruptability and chain-of-thought transparency, why personality toggles (Anton vs Clippy) are a real differentiator users care about, and his thesis that the education system isn’t producing enough people who can do both distributed systems and ML research—the exact skill set required to push the frontier when the bottleneck moves every few weeks.

We discuss:

• Josh’s path: pre-training data curation → post-training researcher at OpenAI, shipping GPT-4o, o1, o3, GPT-5 thinking, and the shopping model

• Why he switched from pre-training to post-training: “Do I want to make 3% compute efficiency wins, or change behavior by 40%?”

• The RL infrastructure challenge: way more moving parts than pre-training (tasks, grading setups, external partners), and why babysitting runs at 12:30am means jumping into unfamiliar code constantly

• How Codex has changed his workflow: 40-minute design sessions compressed into 15-minute agent sprints, and the strange “trapped” feeling of waiting for the agent to finish

• The RLHF vs RLVR debate: both are policy gradient methods, the real difference is data quality and signal trust (human preference vs. verifiable correctness)

• Why GRPO (from DeepSeek Math) was underappreciated: not just an optimization trick, but a shift toward reward signals you can actually trust (math answers over human vibes)

• The token efficiency revolution: GPT-5 to 5.1 bumped evals and slashed tokens, and why thinking in tokens (not wall-clock time) unlocks better tool-calling and agent workflows

• Personality toggles: Anton (tool, no warmth) vs Clippy (friendly, helpful), and why Josh uses custom instructions to make his model “just a tool”

• The router problem: having a router at the top (GPT-5 thinking vs non-thinking) and an implicit router (thinking effort slider) creates weird bumps, and why the abstractions will eventually merge

• Long context: climbing Graph Blocks evals, the dream of 10M+ token windows, and why agents + graph walks might matter more than raw context length

• Why the education system isn’t producing enough people who can do both distributed systems and ML research, and why that’s the bottleneck for frontier labs

• The 2026 vision: neither pre-training nor post-training is dead, we’re in the fog of war, and the bottleneck will keep moving (so emotional stability helps)

—

Josh McGrath

• OpenAI: https://openai.com

• X: https://x.com/j_mcgraph

00:00:00 Introduction: Josh McGrath on Post-Training at OpenAI
00:04:37 The Shopping Model: Black Friday Launch and Interruptability
00:07:11 Model Personality and the Anton vs Clippy Divide
00:08:26 Beyond PPO vs DPO: The Data Quality Spectrum in RL
00:01:40 Infrastructure Challenges: Why Post-Training RL is Harder Than Pre-Training
00:13:12 Token Efficiency: The 2D Plot That Matters Most
00:03:45 Codex Max and the Flow Problem: 40 Minutes of Planning, 15 Minutes of Waiting
00:17:29 Long Context and Graph Blocks: Climbing Toward Perfect Context
00:21:23 The ML-Systems Hybrid: What's Hard to Hire For
00:24:50 Pre-Training Isn't Dead: Living Through Technological Revolution