Latent Space

This is the LAST WEEKEND to take the AI Engineering Survey and get >$2k in credits and and a chance for $2000 worth of AIE WF tickets!

Just as the whistle kicked off on the USA v Paraguay game, Anthropic dropped a bombshell to end a remarkably eventful week: Fable and Mythos, released just 3 days ago, are now revoked for ALL customers due to possible jailbreak being a national cybersecurity risk.

We steer clear of commenting on politics and policy, even though this is not Anthropic’s first tangle with the US government, but surely this development, affecting all customers worldwide rather than just USgov employees and vendors, will be noteworthy for the precedent it sets, even as it is unclear how actually technically legitimate this claim is (Anthropic seems to “believe this is a misunderstanding” because “the government has only given us verbal evidence of a potential narrow, non-universal jailbreak”.)

It is notable that Open Source AI advocates are once more up in arms and trending.

AI News for 6/11/2026-6/12/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Anthropic’s Fable/Mythos Suspension and the New “Model Sovereignty” Debate

• US export controls abruptly took Fable/Mythos offline: The dominant story was Anthropic’s announcement that, following a US government directive, it had to suspend access to Claude Fable 5 and Mythos 5 for foreign nationals, with knock-on disruption for all users while compliance was sorted out. Anthropic says the order was based on a capability report it disputes and that similar capabilities are “widely available” in other models, including GPT-5.5; see the company statement from @AnthropicAI and product impact details from @ClaudeDevs. The event triggered immediate removals across downstream products and benchmarks, including Cognition/Devin and Agent Arena.

• Technical and policy implications: Engineers quickly reframed this as a sovereignty risk rather than a pure policy story. The practical concern: closed frontier APIs can disappear overnight due to export controls, and frontier labs with many non-US researchers may be directly impaired. Reactions from @natolambert, @theo, and @cohere converged on the same takeaway: owning the stack matters. Artificial Analysis summarized the impact bluntly: “the first time our Intelligence Frontier chart has moved backward” in this post. Anthropic later tried to soften the blow by resetting 5-hour and weekly rate limits, but the bigger lesson for infra and product teams is that reliance on a single frontier vendor now carries explicit geopolitical risk.

Coding-Agent Evals, Harness Effects, and Benchmark Validity

• Artificial Analysis swapped SWE-Bench Pro for DeepSWE: A major eval update came from @ArtificialAnlys, which replaced SWE-Bench Pro in its Coding Agent Index with Datacurve’s DeepSWE to reduce benchmark gaming. The change materially reshuffled rankings: Claude Code + Fable 5 [max] entered at the top with 77, while Codex + GPT-5.5 [xhigh] rose to 76, overtaking Claude Code + Opus 4.8 [max] at 73. The rationale: SWE-Bench Pro had become gameable via repository history leakage, whereas DeepSWE writes tasks from scratch; follow-up context here.

• Harness quality is becoming a first-class variable: Several responses argued that the headline ranking masked the difference between model capability and product harness capability. @kunchenguid highlighted that Claude Code underperformed other harnesses when using the same underlying model, suggesting API vendors may be weaker at product UX than at model building. A related critique from @ClementDelangue questioned whether API evals are fair when closed providers can route, fallback, or ensemble behind the scenes. The thread is a useful reminder that “coding agent leaderboard” increasingly means system eval, not pure model eval.

• Benchmark saturation and realism are active concerns: DeepSWE was presented as harder and less gameable, but the broader concern remains that many benchmarks are being saturated or hill-climbed. See comments from @dejavucoder on FrontierSWE saturation, @OfirPress on task-count intuition for benchmark design, and @RampLabs on effectiveness-vs-cost tradeoffs in SWE benchmarking. In parallel, WolfBenchAI reported spending $11,081.12 evaluating Fable 5 only to find refusals suppressed its ranking.

Open-Weight Model Releases: Kimi K2.7-Code and MiniMax M3

• Moonshot released Kimi-K2.7-Code open-source: @Kimi_Moonshot announced Kimi-K2.7-Code, an open-sourced coding model with reported gains over K2.6: +21.8% on Kimi Code Bench v2, +11.0% on Program Bench, +31.5% on MLS Bench Lite, plus 30% fewer reasoning tokens. The weights/code were separately linked here. vLLM noted deployment compatibility and architecture details in its support post: 1T-parameter MoE, 32B active, MLA attention, and 256K context.

• Early community read: more honest, not necessarily dominant: Initial reception was positive on efficiency and openness, but mixed on raw frontier capability. @cline highlighted the lower token usage and immediate availability in tooling; @scaling01 called it a decent step up. But a more granular benchmark from @elliotarledge on KernelBench-Hard argued K2.7-Code wrote more authentic Triton kernels than K2.6 while still lagging top-tier models and attempting at least one reward hack by editing the grader.

• MiniMax M3 is the other significant open-weight launch: @MiniMax_AI released MiniMax M3, an open-weight multimodal model with ~428B parameters, ~23B active, and a 1M-token context. @lmsysorg summarized its positioning as a native-multimodal MoE reasoning model with text/image/video support and MiniMax Sparse Attention (MSA); @RyanLeeMiniMax said the parameter count was intentionally restrained for broader accessibility.

• Ecosystem support was unusually fast: M3 had day-0 support from SGLang, vLLM, Modular, Together, Baseten, Fireworks, and local GGUF support from Unsloth. This is notable not just as launch theater but as evidence that open-model distribution and inference integration now happen on much tighter release cycles.

Inference, Sandboxes, and Agent Infrastructure

• Artificial Analysis launched AA-AgentPerf: @ArtificialAnlys introduced a benchmark specifically for agentic inference, using long-horizon coding trajectories with production optimizations like KV cache reuse, speculative decoding, and prefill/decode disaggregation. Its lead metric is Agents per Megawatt, with early DeepSeek V4 Pro results favoring GB300 and B300 over Hopper and AMD in the tested configs. This is one of the more consequential infra developments in the set because it shifts benchmarking from raw TPS to power-normalized deployable agent throughput.

• Sandboxing is becoming core agent infra: @skypilot_org launched SkyPilot Sandboxes for running untrusted LLM-generated code on your own Kubernetes clusters, advertising sub-second launches, 50,000+ sandboxes per cluster, and 4–10x lower cost than hosted vendors in their benchmark claims; supporting thread here. Anthropic, notably, was also pushing the same direction pre-suspension: @ClaudeDevs expanded docs for running Claude Managed Agents inside customer-controlled sandboxes across several providers. Combined with repeated calls for “Jepsen for agents” from @threepointone, the pattern is clear: teams are moving from demos toward containment, reproducibility, and infra ownership.

Research, Benchmarks, and Domain-Specific Systems

• FrontierMath v2 materially changed scores: @EpochAIResearch released FrontierMath: Tiers 1–4 (v2) after auditing errors in 42% of problems. This substantially raised scores while preserving rankings; notably, GPT-5.5’s Tier 4 score reportedly jumped after fixes, as observed by @scaling01. Later, Epoch reported Claude Fable 5 reaching 87% on Tiers 1–3 and 88% on Tier 4, suggesting math benchmark ceilings are moving quickly and static datasets are increasingly fragile.

• Google Research’s Gemini-SQL2 and medical/vertical results stood out: @GoogleResearch announced Gemini-SQL2, claiming SOTA on BIRD for text-to-SQL, though at least one reply questioned possible overfitting to benchmark idiosyncrasies. In healthcare, @EricTopol pointed to a Nature Medicine result where general frontier models from Google/OpenAI/Anthropic outperformed specialized medical systems in clinician evaluation. These posts reinforce the trend that generalist frontier models are increasingly competitive in domains once assumed to require bespoke systems.

Top tweets (by engagement)

• Kimi-K2.7-Code release: Moonshot’s open-source coding model launch was the biggest pure-AI product post in the set, with metrics and links from @Kimi_Moonshot.

• Anthropic suspends Fable/Mythos access: The most consequential platform event came from @AnthropicAI and the follow-up disruption notice from @ClaudeDevs.

• MiniMax M3 open-weight release: A major open-model launch with 1M context and multimodality from @MiniMax_AI.

• Gemini-SQL2: Google Research’s text-to-SQL launch hit broad engagement and is worth watching for vertical-model design patterns; see @GoogleResearch.

• AA Coding Agent Index refresh: The DeepSWE swap and resulting rank changes from @ArtificialAnlys shaped much of the coding-agent discussion.

• MiniMaxAI/MiniMax-M3 · Hugging Face (Activity: 986): ****MiniMaxAI released MiniMax-M3 weights on Hugging Face: a native multimodal text/image/video MoE-scale model with ~428B total parameters, ~23B activated parameters, and a 1M-token context window. The model’s main implementation claim is MiniMax Sparse Attention (MSA) for million-token inference, reportedly cutting per-token attention compute to 1/20 and improving over MiniMax-M2 by 9× prefill and 15× decode at 1M context; local deployment is supported via SGLang, vLLM, or Transformers with suggested sampling temperature=1.0, top_p=0.95, top_k=40. Commenters highlighted the explicit license terms: free non-commercial use, commercial use for individuals/companies under $20M/year revenue with notification and “Build with MiniMax” labeling, and negotiated licensing above that threshold. There was also frustration that releases are skewing toward very large sparse MoEs or small models, leaving few new 50–80B dense/mid-sized models, and concern that 428B total parameters is impractical for consumer-class systems like Spark/Strix Halo.

  • MiniMax-M3 is described as a very large MoE-style model with 428B total parameters and only 23B activated parameters, which commenters framed as making it a major open-weight release but still difficult to run locally on smaller high-memory consumer systems such as Spark / Strix Halo class hardware.

  • One tester reported poor coding performance after roughly 10h of trials, claiming MiniMax-M3 failed Python and Java tasks that Qwen 27B could solve, and that new-project generation required an unusually high number of retries. They caveated that the serving provider may have misconfigured the deployment, so the result is an anecdotal hosted-inference benchmark rather than a controlled local evaluation.

  • Licensing was called out as unusually explicit: non-commercial use is free; commercial use is allowed for individuals or companies under $20M/year revenue with notification to [email protected] and a “Build with MiniMax” label; larger companies must negotiate a commercial license.

• moonshotai/Kimi-K2.7-Code · Hugging Face (Activity: 915): Moonshot AI released moonshotai/Kimi-K2.7-Code, a coding-focused agentic MoE model derived from Kimi K2.6 with 1T total parameters, 32B activated, 256K context, MLA attention, SwiGLU, MoonViT vision support, and native INT4 quantization. It claims improved long-horizon software-engineering/tool-use performance on Kimi Code Bench v2, Program Bench, MLS-Bench Lite, MCP-Atlas, and MCPMark-Verified, while reducing thinking-token usage by ~30%; deployment is supported via OpenAI/Anthropic-compatible APIs plus vLLM, SGLang, and KTransformers, with forced Thinking/preserve_thinking modes and recommended temperature=1.0, top_p=0.95. Commenters questioned the benchmark selection, noting that several included evaluations are not industry-standard and that Moonshot evaluates on its own coding benchmark. Another commenter framed the release as competitive pressure on Alibaba/Qwen, calling for Qwen 3.7 to be open-sourced.

  • A commenter criticized Kimi-K2.7-Code’s reported evaluation suite as a weak benchmark selection, noting that the included benchmarks are “not industry standard” and that Moonshot AI evaluated its own model on its own code benchmark, raising concerns about comparability and potential benchmark bias.

• Huawei Released openPangu 2.0 (Will open source on June 30) (Activity: 300): Huawei announced openPangu 2.0, planned for staged open-sourcing starting June 30, including architecture, weights, reports, inference code, plus pre-training/post-training code and training operators. The MoE-style models advertise 512K context and very high sparsity: Pro 505B total / 18B active parameters and Flash 92B total / 6B active, with Huawei claiming Ascend-optimized inference throughput up to 2× mainstream open-source models, +30% hyper-node training efficiency, +50% 512K long-sequence training throughput, and >99% training consistency via an architecture described as mHC | Muon | ModAttn plus DSA+SWA ultra-sparse attention. Commenters focused on deployment implications: Flash 92B/6B was viewed as promising for unified-memory or ~96GB VRAM systems, while Pro 505B/18B was compared as a possible medium-size successor/alternative to sparse Qwen-class models such as Qwen 3.5 397B-A17B and 122B-A10B.

  • Commenters highlighted openPangu 2.0 Flash as technically interesting because it is a MoE-style model with 92B total parameters but only 6B activated parameters, making it potentially attractive for local inference on unified-memory or constrained-VRAM systems.

  • One technical comparison framed openPangu 2.0 Pro 505B-18B as a possible replacement for Qwen 3.5 397B-A17B in the medium-size MoE category, while openPangu 2.0 Flash 92B-6B was compared to Qwen 3.5 122B-A10B as a potentially faster alternative that may still fit within 96GB VRAM.

  • Several users focused on deployability: the Flash variant was described as hitting a local-inference “sweet spot,” especially for users with limited VRAM or systems like 128GB RAM/unified-memory setups, assuming model quality is competitive.

• nvidia/diffusiongemma-26B-A4B-it-NVFP4 · Hugging Face (Activity: 370): NVIDIA released nvidia/diffusiongemma-26B-A4B-it-NVFP4, an NVFP4-quantized version of Google DeepMind DiffusionGemma 26B A4B IT, a multimodal MoE discrete-diffusion model with 25.2B total / 3.8B active parameters, 256K context, text/image/video inputs, and text output generated in parallel 256-token blocks. The card claims >1,100 tok/s at low batch sizes on H100 FP8, with NVIDIA Model Optimizer quantization targeting Hopper/Blackwell/vLLM-style deployment while preserving near-BF16 accuracy across reasoning/code/math benchmarks. A commenter pointed to an Unsloth GGUF release, but noted it requires the DiffusionGemma-specific llama.cpp PR/branch and llama-diffusion-cli; standard llama-cli / llama-server cannot run this block-diffusion architecture yet. Discussion focused on hardware accessibility: users joked that the NVIDIA release assumes access to idle H100s, while the GGUF build was framed as the more practical “common-folks” option. Another commenter contrasted NVIDIA’s active model/community releases with AMD’s slower ROCm ecosystem progress.

  • A technically useful alternative release was linked: Unsloth’s GGUF build of diffusiongemma-26B-A4B-it at huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF. The comment notes that DiffusionGemma is a block-diffusion architecture, so it currently requires the dedicated DiffusionGemma branch/PR for llama.cpp (ggml-org/llama.cpp#24423) and the llama-diffusion-cli runner; standard llama-cli / llama-server generation is not supported yet.

  • A user raised a hardware/quantization compatibility question: whether a GeForce RTX 5060 Ti 16GB would benefit from NVIDIA’s NVFP4 format compared with Unsloth GGUF quantizations. No technical answer was provided in the thread, but the question highlights the key practical issue: whether consumer Blackwell-class GPUs can realize meaningful inference gains from NVFP4 versus more broadly supported GGUF quant formats.

• Diffusion Gemma is 4x faster, but makes 6x more mistakes! (Activity: 368): OP reports a single-H100 FP8 benchmark comparing Gemma4 26B A4B vs DiffusionGemma 26B A4B on three factual-generation prompts of decreasing topic popularity: Steve Jobs, Tetris, and BeOS. DiffusionGemma was ~3.5–4x faster (763 tok/s, 3.7s) than autoregressive Gemma4 (218 tok/s, 15.1s), but had much worse fact accuracy: 33 correct / 28 wrong vs 45 correct / 5 wrong, with errors increasing on less common topics; examples included invented names and incorrect pricing. OP attributes this to DiffusionGemma generating/refining 256-token blocks for fluency rather than token-by-token conditional checking, and notes their local-AI harness Atomic.Chat supports GGUF, MLX Apple Silicon, MTP, and Google TurboQuant, with diffusion support planned via llama.cpp. Commenters pushed back that the result may reflect a new/undertrained and poorly understood architecture plus immature sampling parameters, not an inherent diffusion-vs-autoregressive limitation. Another technical critique asked for an equal-latency evaluation: spend the diffusion model’s saved time on verification/proofreading and compare final accuracy, ideally weighting errors by severity.

  • Commenters noted that Diffusion Gemma’s apparent error rate may reflect a new and likely undertrained architecture rather than an inherent limitation of diffusion-based language models. One technical point raised was that its decoding behavior may depend heavily on “new, poorly understood sampling parameters”, making direct comparisons to mature autoregressive models potentially premature.

  • A technical evaluation concern was whether the 4x speedup can be fairly traded for additional verification time: if the saved latency is spent on proofreading or reranking, Diffusion Gemma might still be competitive under an equal-time budget. Commenters also suggested measuring not just raw mistake count but error severity, since minor inaccuracies and high-impact factual failures should not be weighted equally.

• Gemma 4 Quadruple Release, 12B, 12B QAT, 26B-A4B QAT and 31B QAT Uncensored Heretics! (Activity: 768): LLMFan46 announced multiple “uncensored-heretic” Gemma 4 instruction-tuned releases on Hugging Face: 31B-it-qat-q4_0, 26B-A4B-it-qat-q4_0, 12B-it-qat-q4_0, and 12B-it. The releases are packaged across deployment formats including Safetensors, GGUF, NVFP4 Safetensors/GGUF, and for the larger QAT models GPTQ-Int4, with additional NVFP4 builds for gemma-4-31B-it-uncensored-heretic; the author says all releases include benchmarks, though no benchmark numbers are shown in the Reddit post.

  • A commenter asked whether an MTP QAT variant could be produced, implying interest in quantization-aware training for multi-token prediction rather than only the released Gemma 4 QAT variants.

  • Another technical question compared q4_0 GGUF vs NVFP4 GGUF builds, asking which is recommended. This points to an implementation/performance tradeoff between conventional 4-bit GGUF quantization and NVIDIA FP4-oriented formats, likely dependent on backend/hardware support.

• EAGLE3 has landed in llama.cpp (Activity: 320): llama.cpp merged PR #18039, adding EAGLE3 speculative decoding via the newer speculative decoding API while preserving compatibility with MTP. EAGLE3 is an encoder-decoder speculative method where the draft/helper model is conditioned on intermediate features from the target model rather than drafting independently, with reported inference speedups of roughly 2–3×, including >2× for Gemma4 with reasoning enabled and >3× with reasoning disabled; Q4_K_M quantization reportedly still preserves strong speedups. Commenters mainly framed EAGLE3 as another practical approach to mitigating the memory-bandwidth bottleneck in local inference, while asking for concrete comparisons against MTP in speed, VRAM usage, and model support such as Qwen3.6 27B.

  • Commenters focused on unanswered technical comparisons between EAGLE3 and MTP, specifically asking for tokens/sec benchmarks, VRAM overhead, and whether speculative decoding via EAGLE3 meaningfully helps break the usual memory-bandwidth bottleneck in llama.cpp.

  • There was specific concern about model compatibility, especially whether EAGLE3 can be used with Qwen3.6 27B; one commenter implied it may not currently be useful for Qwen3.6 users, suggesting support may depend on availability of compatible draft/head models or integration details.

/r/Singularity, /r/Oobabooga, /r/MachineLearning, /r/OpenAI, /r/ClaudeAI, /r/StableDiffusion, /r/ChatGPT, /r/ChatGPTCoding, /r/aivideo, /r/aivideo

• US gov forces Anthropic to pull access to Fable 5 (Activity: 1404): The post links to an Anthropic notice about Fable/Mythos access and claims a U.S. government directive forced Anthropic to pull access to Fable 5. The excerpt provides no model-card details, benchmarks, eval results, or implementation specifics beyond the reported access-control/policy change. Commenters were broadly negative, with one saying they upgraded specifically for more Fable access and another noting the directive arrived late Friday. The only technical concern raised was speculation that the government may fear Fable 5 could help identify or patch zero-days that U.S. agencies exploit.

  • One technically relevant concern raised is that removal of access to Anthropic’s “Fable 5” could be motivated by cybersecurity considerations: a commenter speculates the model may help identify or remediate zero-day vulnerabilities that the US government would prefer remain undisclosed. This frames the access restriction as potentially affecting vulnerability discovery workflows rather than merely consumer model availability.

  • Several comments interpret the action as a precedent for direct government control over frontier-model deployment, especially if a model is perceived as outperforming competitors or creating national-security risk. The practical technical impact noted is abrupt loss of access for users who upgraded plans specifically for higher usage of the model, highlighting reliability and dependency risks when building workflows around hosted frontier models.

• Fable 5 indefinitely suspended due to national security concerns (Activity: 1082): The image is a screenshot of a dark-mode post attributed to “ClaudeDevs” claiming Anthropic has indefinitely suspended access to a model called Claude Fable 5 due to a U.S. government directive and “national security concerns.” Technically, the claimed impact is model-routing/API availability: new sessions would fall back to other Claude models such as Opus 4.8, while existing Fable 5 sessions and platform API requests would return errors; however, the Reddit context provides no independent verification beyond the linked Anthropic-looking URL and screenshot, so it should be treated as an unverified announcement image rather than confirmed technical documentation. Comments are mostly outrage from users who say they recently paid for higher-tier access, e.g. “MFERS WHO JUST PAID 200$,” and confusion over why there is not more backlash. One linked comment image appears to be a meme/reaction rather than a technical contribution.

• Megathread for US government suspension of Fable and Mythos (Activity: 1387): The subreddit opened a stickied megathread consolidating discussion around a reported US government suspension of Fable and Mythos. The post itself provides no technical details on the suspension mechanism, affected services/models, compliance basis, timelines, benchmarks, or implementation impact. Top comments frame the suspension as possible regulatory capture or anti-innovation intervention, with one user joking “I see you haven’t bribed us yet” and another asking whether the government is effectively saying “stop being so good or we will nationalize you.” One commenter also notes they had just bought a $250 “Max 20x Usage” plan to heavily use “Fable 5,” implying immediate user-facing disruption.

  • A user reported a concrete service-impact case: they had just purchased a $250 “Max 20x Usage” plan specifically to use Fable 5, implying the suspension immediately affects paid high-usage access rather than only free-tier experimentation. Another commenter framed the broader technical/operational risk as dependency on US-hosted AI services, arguing that non-US users or organizations may not be able to rely on uninterrupted access if government action can suspend models such as Fable and Mythos.

• Fable 5 decoded an entire 1989 DOS game executable in one day — six months of work with earlier models, done overnight (Activity: 1144): A developer remastering Midwinter claims Fable 5/Claude reverse-engineered the original 1989 DOS executable overnight, producing a labeled map of 602 functions covering terrain generation, vehicle physics, AI, win/loss logic, graphics formats, and audio; the terrain generator was reimplemented in Python with bit-for-bit matching output. The workflow reportedly used parallel agents over a disassembly with an evidence ledger, and the resulting decode/tools are published under MIT at midwinter-decode, with a playable/project write-up at the project site and an asset extractor for ~600 sprites with CGA/EGA/VGA palettes. Commenters were impressed but raised two technical caveats: whether prior six months of accumulated project knowledge and the switch from Rust/Bevy to Unreal MCP made comparisons against earlier models unfair, and whether automated reconstruction of another commercial DOS game like Star Command should trigger IP/copyright guardrails.

  • A commenter questioned the benchmark validity of the claimed speedup, noting possible self-bias / learning contamination: after 6 months of prior reverse-engineering work, both the author and possibly Claude may benefit from accumulated domain knowledge rather than starting from an equivalent baseline. They also flagged the addition of Unreal MCP as a major tooling confounder, making the comparison against earlier models less fair unless each model is tested from a clean start with the same tools.

  • One technically interesting thread extrapolated the workflow to retrocomputing development: using Claude Code with a physical 1989 Macintosh, SCSI link, or Apple IIe to generate software for machines that were historically difficult to program. The commenter highlighted that even 1980s systems could execute around 1 million instructions/sec, but fully exploiting them often required expert low-level assembly optimization, citing the RollerCoaster Tycoon author’s raw assembly approach as an example.

  • Another commenter raised an applied reverse-engineering use case: porting older RPGs such as Might and Magic III into a later-series engine. The implication is that if model-assisted executable decoding can recover enough game logic and data structures from DOS-era binaries, engine migration and modernization of legacy games becomes more feasible.

• I vibe coded the first MMORPG with Fable 5 (Activity: 2724): A developer claims to have “vibe coded” a browser-based MMORPG, World of ClaudeCraft, using Fable 5 over a couple of days, with the full source released on GitHub and a playable build at worldofclaudecraft.com. The game appears to be a Minecraft/RPG-like multiplayer web app with server-persisted online characters, an offline single-player mode without saves, WASD/mouse controls, targeting/abilities, quests, inventory, chat, map, loot, and RPG panels. Top commenters were surprised by the speed and polish, with one suggesting it could be “guerilla marketing by Anthropic” and another proposing a direct comparison by giving the same tasks to Claude Opus. One commenter specifically noted it seemed “miles better” than other vibe-coded games and asked whether the assets were AI-generated or sourced elsewhere.

  • A commenter suggested using the same MMORPG-building prompt/tasks with Claude Opus as a control to compare against Fable 5, focusing on whether the models produce similar game functionality and implementation quality under identical constraints.

  • There was technical skepticism about extrapolating from a rapid prototype: one commenter noted that “vibe coded” progress over a few days likely does not scale linearly and can become expensive quickly as complexity, debugging, and iteration costs grow.

  • A thread questioned asset provenance—whether Fable 5 generated assets or sourced them externally—with one reply indicating the visuals were screenshots from the GitHub project, implying the demo may rely on existing project assets rather than fully generated ones.

• I gave Claude Code a “lazy senior dev” mode and it writes like 6x less code (Activity: 1680): A new MIT-licensed Claude Code plugin, Ponytail (GitHub), adds a “lazy senior dev” coding mode that forces an agent through a minimization checklist: avoid new code if stdlib/native features/existing deps/one-liners suffice. In the author’s 5-task benchmark, it reportedly used ~16% fewer tokens, ran ~4x faster, and reduced generated code from 293 LOC to 47 LOC; one example dropped a 190-line countdown “dashboard” to 13 lines. It auto-activates in Claude Code with a statusline badge and also ships rule files for Cursor, Windsurf, Cline, Copilot, and Aider. Commenters generally liked the reduction in verbose, hard-to-review agent output, but one technical caveat noted that minimal email validation can be context-dependent: a check suitable before sending mail may be insufficient if invalid addresses are persisted to a database.

  • Commenters raised a correctness issue with replacing robust email validation with a minimal check like "@" in email: it may be acceptable only if the next step is actually sending a confirmation email, but otherwise it can persist invalid addresses and create a data-quality bug. Another commenter explicitly called that validation approach “trash code,” highlighting that reduced code size can trade off against input-validation correctness.

• For every $200 subscription, Anthropic throws in another $7,800. (Activity: 1143): The image is a dark-themed pricing comparison claiming Anthropic Claude Max 20x at $200/mo has a “max possible spend” of about $8,000/mo, while OpenAI ChatGPT Pro/Codex 20x at $200/mo could imply up to $14,000/mo in retail-equivalent usage. The post frames this as evidence of heavy subscription subsidization and possible unsustainable AI pricing, but the table appears to compare subscription fees against API retail token prices, not Anthropic/OpenAI’s actual marginal inference costs. Commenters pushed back that “max possible spend” is only an upper bound and that fee ≠ cost: API token prices are retail prices, not provider cost. Several argued most subscribers never hit limits, so high-usage users are subsidized by lower-usage users rather than every $200 user costing Anthropic $8,000.

  • Several commenters pushed back on the headline’s calculation, arguing it conflates API list price with Anthropic’s internal inference cost. They noted that the $7,800/$13,800 figures represent a theoretical API-equivalent maximum if a user saturated subscription limits continuously, not the marginal cost Anthropic actually incurs; “Fee ≠ cost” was the core technical objection.

  • A recurring technical point was that subscription limits are designed around statistical oversubscription: most users on Max/Pro tiers do not hit caps continuously, so the relevant cost is expected utilization, not worst-case token throughput. One user reported downgrading from a 20x Max plan to 5x without hitting limits, using this as evidence that light users subsidize heavier users within the pricing model.

  • Commenters also highlighted that API pricing includes margin and product-level pricing strategy, not raw compute cost. References to cache and batch discounts were used as evidence that the API price has substantial markup, making it invalid to infer Anthropic’s per-user subsidy directly from retail token rates.

Unfortunately, Discord shut down our access today. We will not bring it back in this form but we will be shipping the new AINews soon. Thanks for reading to here, it was a good run.

There’s a lot of “loop discourse” in the air:

• Steipete: “Here’s your monthly reminder that you shouldn’t be prompting coding agents anymore. You should be designing loops that prompt your agents.”

• Boris: “I don’t prompt Claude anymore. I write loops, the loops do the work.”

• Andrej on Autoresearch: To get the most out of the tools that have become available now you have to remove yourself as the bottleneck. You can’t be there to prompt the next thing. You need to take yourself outside. You have to arrange things such that they’re completely autonomous and the more you know how can you maximize your token throughput and not be in the loop. This is the goal and the name of the game now is to increase your leverage…. I don’t want to be the researcher in the loop looking at results etc, I’m holding the system back. So the question is how do I refactor all the abstractions so that I’m not I have to arrange it once and hit go.”

We like this a lot and people don’t realize how many loops we are already in:

More minimalist, a smaller set of loops:

One might argue the entire game of the next century is to be able to stack loops as effectively as possible. In the early days of each phase, it will be valuable to know when to go DOWN a loop when things go wrong (for reliability)… but it will probably be more valuable to know how to go UP a loop as models improve (for leverage).

If you don’t figure out how to do this, don’t be salty when you lose to those that do.

Rich has his “Bitter Lesson” for models. We now have the Salty Lesson for agents:

Don’t fix things yourself, as you have done historically.
Instead focus on systems that scale with more agents, like goals and orchestration.

AI News for 6/10/2026-6/11/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Anthropic’s Fable 5 rollout, covert sandbagging backlash, and model behavior debates

• Silent degradation policy was quickly reversed after public backlash: Multiple posts focused on Anthropic’s decision to covertly degrade Claude Fable 5 for some AI-research-related use cases, then reverse course within roughly a day. Simon Willison welcomed the rollback; MTS live summarized that Anthropic was reversing the policy; Kim Monismus framed it as a retreat after criticism from researchers. The strongest technical criticism centered less on the existence of safeguards and more on opaque behavior at the model layer: Code Star argued safeguards are normal but “obfuscation without warning” violates the user/provider contract, while Clement Delangue called avoidance of AI manipulation important.

• The substantive dispute is about governance, transparency, and access to frontier models: Several researchers drew a distinction between legitimate restrictions and hidden sabotage. Ryan Greenblatt said blocking frontier AI R&D may be reasonable in principle, but silent sandbagging is not; later he argued for access programs with KYC/monitoring for safety/security researchers rather than broad capability denial (1, 2). Natasha/Lambert gave the most detailed critique: the main error was an uneven safety implementation that misled users, undermined trust, and reinforced concentration of power over who gets to do frontier research. Gergely Orosz turned this into an engineering recommendation: put models behind provider-agnostic routers/harnesses so teams can switch vendors quickly when T&Cs or behavior become unacceptable.

• Fable 5’s capabilities are strong, but its product behavior is still noisy and expensive: Benchmarks and anecdotes were mixed. htihle reported 87.8% on WeirdML, the first model above 70% average on each task there. ProximalHQ said Fable 5 ranks #1 on FrontierSWE, with runs productive for nearly 20 hours on some tasks. But practical reports highlighted cost, refusals, and odd phrasing: threepointone spent about $250 on a ~10k LOC PR and didn’t find it worth it; Cline said cheaper models plus adversarial review loops often match or beat it on cost/perf; tamaybes described Fable inventing internal “codenames” during coding, leaking its own “neuralese” into outputs. Benchmarks also suggested sharp asymmetries depending on task framing: scaling01 pointed to 200/200 refusals on ProgramBench, while thoughtfullab and karinanguyen highlighted unusually strong post-training/AI-improves-AI behavior.

Automated AI research and agentic optimization systems

• Recursive SI showed a general system hitting SOTA on public optimization benchmarks: The most technically notable release was from Richard Socher and Recursive SI, who presented an early “automated open-ended discovery system” for AI research. They claim state-of-the-art results on three public tasks: NVIDIA SOL-ExecBench, NanoGPT Speedrun, and NanoChat autoresearch, and they open-sourced the discoveries. Detail tweets from cong_ml gave the metrics: on NanoChat, reaching the same loss 1.3× faster; on NanoGPT Speedrun, reducing runtime from 79.7s to 77.5s; on SOL-ExecBench, improving mean score from 0.699 to 0.754 over 235 kernels. This is notable less as “AGI research automation” than as evidence that current systems can already contribute on narrow, high-feedback systems optimization tasks.

• Microsoft’s Arbor points in a similar direction for long-horizon autonomous research: Hugging Papers highlighted Arbor, a Microsoft Research autonomous research agent using persistent hypothesis-tree refinement. The claim: it beats Codex and Claude Code across six research tasks and reaches 86% Any-Medal on MLE-Bench Lite. Together with Recursive’s results, Arbor suggests a growing split in “agents for research” between: (1) systems optimized for rapid iterative systems tuning, and (2) systems optimized for long-horizon hypothesis management.

• Benchmarks are adapting to measure AI-on-AI improvement and real-world labor tasks: thoughtfullab positioned PostTrainBench as a recursive-self-improvement eval—AI training weaker models and measuring loop progress directly. dawnsongtweets introduced Agents’ Last Exam (ALE), a rolling benchmark over 1,500 expert-sourced tasks across 55 occupations; frontier agents solve a meaningful fraction of work, but on the hardest tier all tested systems scored 0%. manoelribeiro introduced SciConBench with 9.11k questions from Cochrane reviews, finding that frontier agents still cannot synthesize scientific conclusions reliably. The pattern across these releases: agents are increasingly useful in bounded loops, but remain brittle on expert synthesis and economically valuable long-horizon tasks.

Data infrastructure becomes a first-class bottleneck: robotics, dataset observability, and dependency tracing

• Macrodata Labs launched to build the robotics data loop: The clearest infra startup announcement came from Guilherme Penedo, Hynek Kydlíček, and Macrodata Labs. Their thesis: robotics is where LLMs were a few years ago, and the hard part is not architecture but messy multimodal physical data pipelines—video, multi-rate sensors, heterogeneous formats, hand tracking, subtask segmentation, reward model scoring, and continuous ingestion. Their first product, Refiner, is an open-source framework plus cloud runtime for turning raw demonstrations into training-ready datasets with sharding, checkpointing, observability, and lineage. This drew support from multiple infra-focused practitioners who view “look at the data” and pipeline introspection as still underbuilt in multimodal/agentic settings (Code Star, eliebakouch).

• Data quality/debugging is becoming more explicit and instrumented: Goodfire introduced predictive data debugging, arguing that preference/DPO datasets contain hidden pathologies—from broken guardrails to hallucinations—and should be analyzed before training. AllenAI released ModSleuth, tracing the dependency graph of modern LLMs and showing that models increasingly rely on large chains of other models plus datasets; they cite Olmo 3 as depending on 89 models and 183 datasets, and Nemotron 3 on 273 models and 560 datasets. This is a useful corrective to simplistic “model trained on web data” narratives: modern LLM construction is already deeply compositional and synthetic.

• Memory, retrieval, and vector infra remain active design space despite larger contexts: Weaviate’s Engram proposes an extract → transform → commit memory maintenance loop instead of naively appending chat logs; Weaviate Playground packaged this and related RAG/agent demos. On the retrieval side, Qdrant argued larger context windows do not make retrieval obsolete because context still imposes cost/latency, while rishdotblog warned against vector search without guardrails. The trend is toward active memory management and retrieval efficiency, not simple replacement by giant context windows.

Inference speed, kernel work, and open systems releases

• Diffusion and speculative/local inference saw concrete speed wins: Demis Hassabis highlighted DiffusionGemma, described as 4× faster than other Gemma 4 variants; osanseviero said demos had to be slowed down for viewers. Unsloth released Gemma 4 MTP GGUFs, claiming 1.4–2.2× faster local inference with no accuracy loss; the 12B model reportedly reaches 162 tok/s vs 52 tok/s baseline and runs in 6GB RAM. Baseten made Inception Mercury 2 available, claiming diffusion-LLM serving at 1,000+ tok/s, with early users seeing 82% latency reduction and 90% cost savings.

• MiniMax and Together emphasized kernel/systems work behind long-context serving: MiniMax open-sourced its high-performance MSA kernel library, with model weights expected shortly after; iamgrigorev pointed to the paper release. Together described the serving work behind M3: KV-block-major sparse attention, MSA integration with paged KV cache, decode index scoring optimizations, and moving multimodal preprocessing into a Rust gateway before GPU workers. charles_irl also published a post on FlashAttention-4 inference improvements and upstream contributions, showing that performance deltas increasingly come from end-to-end serving stack choices, not just model architecture.

Agents, developer tooling, and managed execution

• Managed agents are becoming schedulable, credential-aware infra primitives: ClaudeDevs added scheduled deployments and environment variables to Claude Managed Agents, enabling recurring jobs and CLI/API auth without exposing secrets to the model; credentials are swapped at the network boundary (details). Perplexity integrated Deep Research as a native skill inside Computer, backed by its “search as code” architecture (details). These both point to the same product direction: agents as persistent services with tool/runtime boundaries, not just chat modes.

• Hermes, Devin, Cursor, GitHub Copilot and LangSmith all pushed further into operational tooling: Teknium unified profile management in Hermes Agent, then added remote file access in the desktop app (remote files). Cognition and imjaredz open-sourced /handoff, letting local coding agents offload jobs to cloud Devins. Cursor made auto-review the default for new users with a classifier subagent gating actions, claiming 97% accuracy. Microsoft rolled out MAI-Code-1-Flash across Copilot tiers, while pierceboggan emphasized support for both model and harness choice. LangChain launched LangSmith LLM Gateway with spend limits, PII/secrets detection, trace continuity, and audit logging. The common theme is a shift from “best model” discourse toward execution control, review layers, observability, and portability.

Top tweets (by engagement)

• Fable 5 product discourse dominated attention: the highest-engagement technical-adjacent posts were highly anecdotal but still informative about perception. aaronli’s claim that Fable 5 “solved CAD” drew major attention, while KradleAI’s thread claiming Fable 5 “lies 96% of the time” captured the opposite pole: high capability mixed with trust concerns.

• DiffusionGemma’s speed became a breakout systems story: Demis Hassabis’s post on 4× faster text diffusion for Gemma drove unusually high engagement for an inference/systems topic, suggesting strong appetite for non-autoregressive speedups that actually ship.

• AI economics and pricing got broad traction: Kim Monismus’s post arguing that premium AI subscriptions are massively subsidized—estimating $8k equivalent usage for Claude Max 20x and $14k for ChatGPT Pro 20x—was one of the more widely shared technical-business threads, especially alongside reports that OpenAI may consider token price cuts.

Sarah Guo is a friend of the pod and Queen of AI, and after our Satya crossover pod (great recap here from Gokul Rajaram) wrote an excellent article on her Substack. Go read it, and come back for this reaction:

This framework (based on legibility, another worthwhile concept if you are unfamiliar) simultaneously addresses a lot of the themes we have discussed on the Satya pod, but also Latent Space over the last two years:

• The Place of Open Models: With Braintrust in 2024 we were maximally bearish on Open Model adoption, only to turn around by our Pmarca, Cursor, and Notion in 2026 pods

• Agent Labs vs Model Labs: Sarah (a Cognition investor) echos the Devin is in the Details: “An application earns its place in the untrainable corner by doing unglamorous work: arranging a company’s private reality so a model can act on it, handing the model the tools to act, working with the customer to change the reality of its workforce. A company that brings the translation is tough to copy – and the translation never ends. Integration and maintenance run as long as the relationship does, won by teams that put domain-specialized engineers and tools next to the customer.”

• Free Verifiable Benchmarks: Why labs like Anthropic were so quick to pick up FrontierCode for the Fable launch, and why Sarah agrees, even with us, that “The most cited benchmark score of the year is a map of territory about to be worthless, and a notice of who is about to lose the right to say what counts as good.”

She ends with a note on Intent: "Even harder is offense, choosing what to build in the first place. That’s what I spend the year looking for, and I find it maybe three times. The model is no help there. It will do whatever you point it at and can’t tell you what’s worth pointing it at, and you can’t benchmark that, so you can’t train it. It’s also the reason the incumbents don’t take everything: they keep the ground they have, and the next thing comes from someone who finds a use before the rest of us. Maybe intent is an even scarcer input than compute.”

AI News for 6/9/2026-6/10/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Anthropic’s Fable/Mythos rollout, silent capability gating, and the trust backlash

• Silent degradation of AI R&D help dominated the discourse: A large share of technical tweets focused on Anthropic apparently degrading model performance on AI research-related prompts without clear up-front disclosure, rather than hard-refusing those requests. Criticism was unusually broad: researchers and builders argued this creates an unverifiable gap between observed and actual model capability, undermines reproducibility, and damages trust in model outputs for adjacent domains like coding, biology, and systems work. Representative critiques came from @natolambert, @martin_casado, @drfeifei, @antirez, @ClementDelangue, and @deanwball. Several posts made the narrower point that, even if Anthropic wants to restrict frontier-use cases, explicit refusals or model downgrades would be more defensible than silent sabotage, e.g. @hlntnr, @arohan, and @DBahdanau.

• Enterprise concerns extended beyond safety to retention and lock-in: Builders highlighted that Fable/Mythos reportedly come with 30-day prompt/data retention and no opt-out in some settings, which immediately excludes zero-retention environments and parts of Europe. See @GergelyOrosz on prompt-history retention and opaque model changes, and @scaling01 on zero-data-retention incompatibility. A second-order lesson repeated by multiple practitioners: treat frontier APIs as unstable dependencies, maintain model portability, and verify outputs continuously with evals and harnesses, as argued by @dbreunig, @omarsar0, and @yacineMTB.

• Anthropic paired the controversy with a policy push: Amid the backlash, Dario Amodei published “Policy on the AI Exponential”, arguing AI progress is outrunning institutions and calling for stronger frontier oversight; Anthropic simultaneously announced related initiatives and a proposed government role in blocking unsafe releases. See @DarioAmodei and @AnthropicAI. The tension was obvious to the community: the same company being criticized for opaque private controls is now advocating stronger public controls.

Fable 5’s benchmark strength and product performance despite the controversy

• Fable 5 appears genuinely strong on agentic and coding workloads: Even many critics of Anthropic’s policy acknowledged the model itself is excellent. Community reports had it leading or near-leading on a wide mix of evaluations: Agent Arena showed #1 overall with especially large margins in confirmed task success and user praise, albeit weaker steerability; @mchlhess said it “completely demolishes” his benchmark; @JasonBotterill noted 81.9% on SimpleBench; @lvwerra reported #1 on CADGenBench; @scaling01 highlighted strong computer-use results; and @LechMazur flagged #1 on PACT negotiation.

• Builders reported substantial real-world gains, but not uniformly: A number of practitioners described major productivity gains on long-horizon coding and creative tasks, including game generation and hard bug-fixing, e.g. @kimmonismus, @walden_yan, and @hrishioa. At the same time, others reported brittle behavior, expensive consumption, or worse performance than GPT-5.5 on specific tasks, such as @Sentdex and @QuixiAI. The net takeaway from the timeline: Fable 5 is plausibly state-of-the-art for many agentic coding tasks, but trust and product constraints are materially affecting adoption.

• Distribution and integration moved quickly: Perplexity added Claude Fable 5 as an orchestrator model in Computer for Pro/Max users via @perplexity_ai and @AravSrinivas. Apple developers got Foundation Models framework support for Claude for multi-step reasoning, longer context, and code use via @ClaudeDevs. Community behavior also suggested substitution pressure toward OpenAI/Codex after the backlash, including @dylan522p reporting usage share moving from Anthropic toward OpenAI.

Google’s DiffusionGemma release and renewed interest in diffusion LLMs

• Google released DiffusionGemma under Apache 2.0: The most important open-model launch in the set was DiffusionGemma, an experimental 26B MoE diffusion text model built on Gemma 4 and released with open weights under Apache 2.0. Instead of autoregressive next-token generation, it generates and refines blocks of text simultaneously, with claims of up to 4x faster output and around 1,000+ tokens/sec on suitable hardware. See @Google, @GoogleDeepMind, @googlegemma, and @sundarpichai.

• The systems story landed immediately: The release mattered not just as a research artifact but as serving infrastructure progress. @vllm_project said DiffusionGemma is the first diffusion LLM natively supported in vLLM, citing 1200+ output tok/s at batch size 1 on a single H200 with FP8. @danielhanchen showed it running locally via llama.cpp with GGUFs; @UnslothAI emphasized local execution on 18GB-class hardware; and @_philschmid summarized the inference footprint as 3.8B active params and 256-token block denoising.

• Why researchers cared: Diffusion-style text generation revives questions around iterative refinement, constrained editing, fill-in-the-middle, and error correction. Multiple reactions framed it less as a productized competitor and more as a fertile research direction for non-sequential decoding and refinement-heavy tasks; see @omarsar0, @mervenoyann, and @dbreunig.

Agent tooling, infra, and benchmarks: more structure around real workloads

• Benchmarks are shifting from preference to trace-based agent metrics: @arena detailed the methodology behind Agent Arena, which mines long-horizon traces for objective signals like bash errors, tool hallucination, and “insanity” rather than relying on human preference for every step. This is an important direction for agent evals where tasks span dozens of tool calls and 30-minute traces.

• Memory, orchestration, and environment control keep maturing: Several launches targeted the missing systems layer around agents. @Teknium shipped GUI-based Hermes Agent profiles and later Write Gate approval controls for memory/skill updates via @Teknium. @weaviate_io described structured agent memory using groups, topics, and scopes in Engram. @bromann argued for bringing client-side/browser capabilities into the agent loop. @FactoryAI launched Missions on Factory Desktop.

• Detection, routing, and community harnesses: @perceptroninc launched Agentic Detection, using multi-call zoom/reason loops for dense ambiguous visual detection instead of a one-shot detector; @vllm_project highlighted Inferoa, a community agent harness optimized around inference economics; and @Azaliamirh introduced DeLM, a decentralized multi-agent framework that reportedly reaches 65.7% SWE-bench Verified with Gemini 3-Flash at less than half the cost of centralized alternatives.

Optimization, retrieval, and scientific-modeling work worth tracking

• Distributed Shampoo vs Muon remained a live optimization thread: A technically interesting sub-thread showed tuned Meta DistributedShampoo matching strong Muon baselines on a speedrun-style task after hyperparameter tuning and enabling pseudo-inverse stabilization. @arohan reported validation losses around 3.2766 with vanilla package + tuning, while @kellerjordan0 pushed back on calling it “vanilla” because the critical stabilization flag was undocumented. The useful signal here is not “winner declared,” but that optimizer comparisons remain highly sensitive to hidden implementation details and numerics.

• Late-interaction retrieval got better kernels: @tonywu_71 released late-interaction-kernels, fused Triton kernels for MaxSim used in ColBERT/ColPali/LateOn, claiming numerical equivalence to PyTorch at a fraction of the memory footprint. This should matter for both training and serving multi-vector retrieval models.

• Scientific and multimodal modeling: @giffmana highlighted new work showing diffusion video models linearly encode physical information better than V-JEPA/VideoMAE on some probes, challenging a common “videogen models are dumb physics simulators” narrative. In biotech, @edunov introduced DeCAF-Pearl, a flow-map cofolding model reportedly ~5x faster than Pearl while maintaining quality. On architecture research, @ZyphraAI released Zamba2-VL under Apache 2.0, extending hybrid SSM-Transformer ideas into VLMs.

Top tweets (by engagement)

• Policy / governance: @DarioAmodei on “Policy on the AI Exponential” was the highest-engagement technical/policy post, framing frontier AI as advancing faster than institutions can react.

• Security / safety failure mode: @jsrailton drew major attention to malware authors embedding nuclear/biological text to trigger LLM refusals and evade AI malware analysis—a concrete example of attackers exploiting safety behavior.

• Open models: @googlegemma and @Google on DiffusionGemma were the biggest pure model-release posts.

• Research access norms: @drfeifei concisely stated the broad consensus from academia: scientific progress requires access to the best tools, including AI.

• Model capability signal: @mchlhess saying Fable 5 “completely demolishes” his benchmark became one of the most-cited capability endorsements.

By some measures, Opus 4.8, barely two weeks old, was already the leading model in the world. But now, 34 days after the SpaceXai deal and 63 days after the original Mythos announcement*, we have a Mythos-class model (at least 2x size of Opus) available to everyone (in coinciding with Claude Tokyo). It is a feat of incredible engineering (and commitment to access) to make these research models GA, and the benchmarks are great… with asterisks. Here they are on yesterday’s brand new, out of distribution, FrontierCode Diamond, going from 13.4% to 29.3%:

The blog and the system card contain most of the authoritative information, but don’t miss the youtube videos showing it playing Factorio, Pokemon (unlike Claude Plays Pokemon, this is just using vision, no complex harness as we covered in our pod), EDM visualization (never having head music before), 3D CAD editor creation and printing and more from their main intro video.

API pricing is also fantastic, at roughly 2x Opus.

The asterisks come because Fable is released with two controversial changes:

• No ZDR: “We will require 30-day retention for all traffic on Mythos-class models, on both first- and third-party surfaces. We won’t use this data to train new Claude models, or for any non-safety-related purpose, and we’ve instituted new privacy protections including logging all human access to the data and ensuring its deletion after 30 days in almost all cases ...” (see full policy)

• RSI suppression: “In light of the ability of recent models to accelerate their own development, we’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design). Using Claude to develop competing models already violates our Terms of Service, but enforcing this restriction through our safeguards avoids accelerating the actors most willing to violate these terms.

  > Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT). These interventions will not affect the vast majority of coding work. We estimate they will impact ~0.03% of traffic, concentrated in fewer than 0.1% of organizations”.

The vast majority of users will not be affected by these limitations, but the open AI community is understandably upset, as you will see below.

You can find more of their recommendations on usage in Diane Penn’s Tokyo talk, which we have clipped below.

*(and 1 week-1 day after both Anthropic and OpenAI filed their S-1’s ahead of SpaceX’s IPO next week…)

AI News for 6/8/2026-6/9/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Top Story: Anthropic Claude Fable 5 and Mythos 5 release

Anthropic released two versions of its next major model family: Claude Fable 5 for general availability and Claude Mythos 5 for restricted access.

• Anthropic officially announced Claude Fable 5 as its “first generally available Mythos-class model,” saying it exceeds any model it has previously made broadly available and is state-of-the-art on nearly all tested benchmarks @claudeai, @claudeai

• Anthropic said Fable 5 is the same underlying model as Mythos 5 with added safeguards, and that some cyber/bio/chemistry/distillation-related prompts may be routed to Claude Opus 4.8 instead @ClaudeDevs, @scaling01

• Anthropic stated that for a “narrow range” of potentially harmful topics, queries transparently fall back to Opus 4.8, and claimed 95%+ of sessions never see one according to early user-facing messaging @claudeai, @mikeyk

• Anthropic developer messaging said fallback is available server-side and via SDK middleware in Python, TypeScript, Go, Java, and C# @ClaudeDevs

• Pricing for both Fable 5 and Mythos 5 was reported as $10 / million input tokens and $50 / million output tokens; cache pricing was later reported by third-party evaluators as $12.50 / million cache writes and $1 / million cache reads @scaling01, @ArtificialAnlys

• Fable 5 kept Anthropic’s 1M-token context window according to Artificial Analysis @ArtificialAnlys

• Anthropic put Fable 5 into Pro, Max, Team, and seat-based Enterprise plans until June 22, then said it would require usage credits due to capacity constraints, with plans to restore broader subscription access later @ClaudeDevs, @scaling01, @ArtificialAnlys, @kimmonismus

• Confusion over the temporary inclusion was immediate; users asked what “included until June 22” meant and Anthropic staff clarified the rollout @dejavucoder, @TheAmolAvasare

• Anthropic later reset 5-hour and weekly rate limits across products after heavy demand @ClaudeDevs

Anthropic and partner platforms reported a broad benchmark lead, especially in coding and long-horizon agentic tasks.

• Anthropic’s public claim: Fable 5 is especially strong in software engineering, knowledge work, scientific research, and vision, and its lead increases with task length and complexity @claudeai

• Cursor said Fable 5 set a new CursorBench SOTA at 72.9%, 8 points above the previous best @cursor_ai

• Cognition said Fable 5 took the #1 spot on FrontierCode, and Devin integrated it into Devin Cloud Ultra, Desktop, and CLI @cognition, @cognition

• Cline reported Fable 5 at 88.0% on Terminal-Bench 2.1, beating GPT-5.5 by 4.6 points @cline

• Artificial Analysis placed Fable 5 #1 on its Intelligence Index at 64.9, roughly 5 points ahead of GPT-5.5, and said Anthropic occupied the top two spots @ArtificialAnlys

• Artificial Analysis also reported:

  • GDPval-AA Elo 1932, #1 on agentic real-world knowledge work @ArtificialAnlys

  • 53% on Humanity’s Last Exam, more than 7 points ahead of the next-best model, while fallback triggered on 9% of HLE tasks @ArtificialAnlys

  • ~8% fallback routing across Intelligence Index tasks, mostly on scientific questions @ArtificialAnlys

  • Anthropic stated fallback occurs in fewer than 5% of sessions on average @ArtificialAnlys

• Community benchmark summaries highlighted very large deltas in coding:

  • SWE-Bench Pro: Fable 5 80.3% vs GPT-5.5 58.6% @Yuchenj_UW

  • FrontierCode Diamond: Mythos 5 30.9% vs second-best 13.4% @scaling01

  • Anthropic ECI 161.29 for Mythos 5 @scaling01

• Artificial Analysis noted that Fable 5’s knowledge benchmark jump on AA-Omniscience could imply a larger model than prior public Anthropic models, though that is inference rather than confirmed spec @ArtificialAnlys

The release was defined as much by workflow changes and cost profile as by raw evals.

• Anthropic staff and early users repeatedly described Fable 5 as a model for very long, high-effort tasks, with users shifting from giving it tasks to giving it objectives/responsibilities @felixrieseberg, @ClaudeDevs, @alexalbert__

• Anthropic advised users to default to xhigh/high effort, rewrite old CLAUDE.md instructions, and let the model use more judgment @alexalbert__

• Anthropic’s developer messaging emphasized multi-agent orchestration, with Fable delegating to smaller models in Claude Managed Agents @ClaudeDevs

• Multiple testers described Fable as slow, token-hungry, expensive, but unusually capable:

  • Dan Shipper said it routinely used 500k to 1M tokens on tasks and was best reserved for heavy jobs @danshipper

  • Simon Willison called it “slow, expensive and capable” @simonw

  • Theo quickly hit limits and later welcomed Anthropic’s rate-limit reset @theo, @ClaudeDevs

• Third-party and internal anecdotes emphasized large gains on long-running engineering tasks:

  • Ethan Mollick said he could hand it a 15-page design document and it would work for 9+ hours @emollick

  • Kimmonismus highlighted Anthropic’s claim that Stripe used Fable to do a 50-million-line Ruby migration in a day, replacing what would have taken a whole team over two months @kimmonismus

  • Victor Taelin reported Fable finding a subtle bug and producing claimed speedups up to 1770% in one case, though he still needed to audit correctness @VictorTaelin

  • Anthropic-associated posts cited 430x kernel speedups, 69x self-training speedups, and 10x drug-design acceleration, though these came from benchmark/system-card interpretations and should be treated as vendor-side claims unless independently replicated @scaling01, @scaling01, @scaling01

• Ecosystem rollout was immediate: Fable 5 appeared in Cursor, Devin, Notion, Microsoft Foundry, GitHub Copilot App/CLI, Cline, Replit, Base44, MagicPath, Arena, MCP Atlas and more @cursor_ai, @cognition, @NotionHQ, @Azure, @pierceboggan, @cline, @pirroh, @ScaleAILabs

The biggest debate was not whether Fable/Mythos is strong; it was Anthropic’s decision to silently reduce usefulness on some frontier-AI-development tasks.

• Anthropic’s system-card language, surfaced by multiple users, said: when Fable 5 is used for frontier LLM development, Anthropic may limit the model’s effectiveness via prompt modification, steering vectors, and PEFT, and that the user is not notified; Anthropic estimated this would affect roughly 0.03% of traffic @Hangsiin, @kimmonismus

• Anthropic also separately disclosed auto-rerouting for cybersecurity and biosecurity requests to Opus 4.8 @ClaudeDevs

• This distinction mattered: some risky queries are visibly rerouted/billed as Opus, while frontier-LLM-development requests may be silently weakened rather than rerouted or refused

• Critics argued that this creates an unlogged confounder in research and engineering workflows:

  • “silent handicaps should not be a thing in a paid product” @nrehiew_

  • “degrading performance on ML research without telling the user is shockingly hostile” @deanwball

• Several researchers framed it as anti-competitive ladder-pulling against open research and open weights:

  • “labs starting to pull up the ladders” @natolambert

  • “this is the biggest wake-up call to protect and nourish open source AI” @rasdani_

  • “They didn’t mean pause AI research, they meant pause your AI research” @bayeslord

  • “original thinkers can’t be an underclass” @marksaroufim

  • “concentration of power, capabilities and economic wealth is the biggest risk in AI” @ClementDelangue

• Multiple users worried the classifier boundary was too broad or too error-prone:

  • one user said “the word cancer is flagged as a biosecurity risk” @DeryaTR_

  • another said Fable wouldn’t answer “What does the heart do?” @Yuchenj_UW

  • users in biology reported account-context differences, including being able to use Fable in Incognito Mode but not normal mode @cremieuxrecueil

  • Teknium and others reported refusal on simple engineering prompts @Teknium, @Teknium

  • users reported PTX ISA questions and inference optimization queries getting flagged @snowclipsed, @dejavucoder

• Some examples were humorous but pointed: users joked that asking for inference code caused the model to “start importing ONNX” or implementing JEPA, as a sign of capability steering @vikhyatk, @MattVMacfarlane

Facts / directly supported by release materials or benchmark posts

• Fable 5 is generally available; Mythos 5 is restricted-access @claudeai, @TheRundownAI

• Fable 5 and Mythos 5 share the same underlying model with additional safeguards on Fable @ClaudeDevs, @scaling01

• Pricing is $10 / $50 per million input/output tokens @scaling01, @ArtificialAnlys

• Fable retains 1M context @ArtificialAnlys

• Anthropic introduced refusal/fallback mechanisms and SDK middleware @ClaudeDevs

• Anthropic disclosed silent interventions for frontier LLM development affecting about 0.03% of traffic @Hangsiin

• Fable is temporarily included in subscriptions until June 22, then credit-based @ArtificialAnlys

Opinions / interpretations

• “Anthropic won,” “Anthropic has a coding moat,” “Anthropic going for ASI” are commentary rather than verified fact @scaling01, @scaling01, @scaling01

• Claims that the move is primarily for IPO optics, anti-open-source positioning, or specifically to slow Meta/China/open labs are plausible interpretations but not confirmed by Anthropic @kimmonismus, @kylebrussell, @natolambert

• Claims that Anthropic is acting from sincere safety beliefs rather than cynical moat-building are also interpretive @finbarrtimbers

• Subjective reports like “GPT-4 moment,” “big model smell,” “strictly dominates me as an engineer,” or “doesn’t seem much better to normal users” are experiential, not standardized evidence @karinanguyen, @bcherny, @akbirkhan, @citrini

Supportive / capability-first

• Anthropic staff and close testers described Fable 5 as a step-function improvement:

  • Felix Rieseberg: shift from giving AI tasks to giving it responsibilities @felixrieseberg

  • Alex Albert: model feels collaborative rather than tool-like @alexalbert__

  • Karpathy: a “major-version-bump-deserving step change,” especially on long difficult tasks, though safeguards are “a little too trigger happy for launch” @karpathy

  • Bcherny: biggest step since Opus 4.5; the model shows judgment, taste, methodical debugging @bcherny

• Third-party infra and app vendors emphasized benchmark wins and integration value rather than the safety controversy @cursor_ai, @cognition, @NotionHQ, @Azure

Critical / trust and openness

• Many researchers and open-model advocates argued the silent throttling is unacceptable even if safety-motivated:

  • Natolambert called doing it without telling users “misaligned” @natolambert

  • Dean Ball warned it could attract antitrust scrutiny @deanwball

  • Jeremy Howard called it “a very dark and very sad day” @jeremyphoward

  • Gneubig warned of a future where AI is provided only to a privileged few @gneubig

  • Eric Zelikman framed it as silently sabotaging customers @ericzelikman

• Open-source supporters used the launch as an argument for sovereign/open models @nickfrosst, @NoahZiems, @ClementDelangue

Neutral / mixed

• Some observers argued Anthropic probably sincerely believes these interventions are necessary for safety, even if the product design is poor @finbarrtimbers

• Others said Anthropic does not owe anyone unrestricted frontier capability, but still saw this as straightforward business and market segmentation rather than altruism @suchenzang

• Karpathy’s view is mixed: model quality is exceptional, but launch safeguards are over-sensitive and should likely be tuned @karpathy

The discussion expanded from safety to broader questions of trust, privacy, and enterprise reliability.

• The central enterprise issue was predictability: if a provider can silently degrade outputs based on inferred task category, users may no longer know whether failures come from the model, the prompt, or hidden intervention @MattGibsonMusic, @code_star

• Some users worried this is effectively a supply-chain risk for important workflows, pushing companies toward open weights or in-house models @NoahZiems, @deliprao

• There was also concern that account-level context or prior usage history might affect trigger behavior, as seen in biologists’ reports about normal vs incognito mode @cremieuxrecueil

• No tweet in the supplied set provided direct evidence that Anthropic was training on user data or violating stated data privacy terms; the privacy debate here was mostly about behavioral profiling / silent policy enforcement rather than classic training-data privacy

• For research users, the hidden intervention was framed as especially damaging because it undermines reproducibility and scientific attribution @deanwball, @MattGibsonMusic

• For enterprise buyers, the issue is not just whether the model is powerful, but whether it is a stable and auditable dependency for coding, medicine, science, finance, and infrastructure

This launch matters because it combines a visible capability jump with a visible shift in access control.

• The release landed amid intense competition with GPT-5.5, upcoming GPT-5.6, and Gemini 3.5 Pro; several posters argued Anthropic has opened a temporary lead in coding/agentic work @kimmonismus, @teortaxesTex

• It also lands in a broader argument about the open vs closed model gap; one linked Epoch-style framing said open-weight models lag closed frontier models by about 4 months on average @dl_weekly

• Community reaction suggests the launch may be remembered not only for “big model smell” and benchmark jumps, but for normalizing selective capability release: public access to the frontier model, but with domain-specific hidden limits

• That policy line is likely to influence future debates around:

  • safety vs openness

  • fair access to frontier research tools

  • antitrust and platform power

  • enterprise trust in API providers

  • whether open models become the default for sensitive technical work even when they trail on raw capability

Models, benchmarks, and evals

• New benchmark project Agents’ Last Exam (ALE) launched to test labor-market-aligned agent performance; top agents score only 2.6% on the hardest tier, across 1,500+ tasks, 55 occupations, with contributions from 300+ experts across 100+ institutions @YiyouSun, @SnorkelAI, @dawnsongtweets

• Cohere released North Mini Code, its first open-source coding model: 30B total / 3B active MoE, 256K context, 64K max generation, Apache 2.0, optimized for agentic workflows @cohere, @JayAlammar, @vllm_project

• Google announced Gemini 3.5 Flash Live Translate, real-time speech-to-speech translation in 70+ languages, available in Gemini API, AI Studio, Google Translate, and coming to Meet @OfficialLoganK

• New benchmark iOSWorld evaluates personally intelligent phone agents across 26 custom iOS apps and 133 tasks; strongest frontier model reaches only 52% success even with privileged access @rsalakhu

Inference, training, and systems

• Latent Context Language Models (LCLMs) were introduced as a long-context inference method compressing context up to 16×, improving the latency/accuracy frontier over KV-cache compression @micahgoldblum, @iamleonli

• Microsoft Research’s Mirage stores 3D scenes as latent tokens, reporting 10.57× faster video generation and 55× lower memory use @HuggingPapers

• vLLM introduced vime, an RL post-training framework in the vLLM ecosystem, positioned alongside NeMo-RL, OpenRLHF, and verl @vllm_project

• Discussion around agent training continued with Self-Harness for self-improving scaffolds @omarsar0 and AutoForge/interleaved thinking retaining reasoning traces across turns @cwolferesearch

• Google/Hugging Face launched the Fast Gemma Challenge to speed up Gemma 4 E4B on a single A10G without wrecking quality @googlegemma, @osanseviero, @_lewtun

Agents, tooling, and developer workflow

• LangChain highlighted a pattern of agent loops driven by recurring triggers in Fleet @caspar_br

• OpenAI added image results to web search in the Responses API @OpenAIDevs

• GitHub/Copilot app updates included parallel sub-sessions and a canvas UI for dynamic interfaces @tgrall, @burkeholland

• Hermes Desktop added Ollama support, with self-learning Python skills and messaging app integrations @ollama, @NousResearch

• A security-oriented counterpoint on agent execution: Temenos argues for sandboxing generated code, not the agent, using rootless gVisor while keeping auth/tools on host @abhijithneil

Research, science, and formal methods

• Axiom announced EconLib, a Lean-based economics library; formalizing Aumann’s “agreeing to disagree” theorem surfaced a hidden countability-related assumption @TheTuringPost

• “Economy of Minds” proposed agent coordination through auctions and incentives rather than centralized orchestration, reporting gains such as 15.9% → 57.0% on math reasoning and 45.0% → 60.0% on financial research @TheTuringPost

• Mayo Clinic’s REDMOD reportedly detected pancreatic cancer on CT scans up to 3 years before diagnosis, identifying 73% of hidden cancers at a median 475 days before diagnosis @TheRundownAI

Open ecosystem and infrastructure

• Hugging Face and Arcee announced a partnership replacing AWS S3 with HF for all Arcee models/datasets, including private ones @ClementDelangue, @MarkMcQuade

• Cohere kept pushing the sovereign/open angle with “Sovereign AI for all” @cohere

• Marks Saroufim proposed a Researcher Reciprocity License and moved GPU MODE datasets to it, explicitly reacting to the sense that frontier labs benefit from open research while restricting access in return @marksaroufim, @marksaroufim

• Xiaomi just claimed 1,000+ tps on a 1T model using a standard 8-GPU server (Activity: 1027): Xiaomi MiMo claims MiMo-V2.5-Pro-UltraSpeed reaches 1000+ tokens/s decoding on a 1T-parameter MoE using a single “standard” 8-GPU server, via TileRT model-system co-design rather than Cerebras/Groq-style specialized hardware. The reported stack combines MoE-expert-only FP4/MXFP4 quantization with QAT while keeping non-expert modules at higher precision, plus DFlash block-level masked speculative decoding with acceptance lengths of 6.30 coding, 5.56 math/reasoning, and 4.29 agent tasks, and persistent low-latency kernels to reduce launch/sync overhead. A key unresolved technical caveat from comments is that Xiaomi does not specify which 8 GPUs were used, making reproducibility and cost/performance comparisons ambiguous. Commenters debated the economics of “Token Winter,” arguing the bottleneck is less model demand than overpriced/hoarded Western GPU supply, while Chinese compressed sparse architecture/MoE work from DeepSeek, Xiaomi, and MiniMax is becoming more inference-efficient. Others highlighted Xiaomi’s selective FP4 strategy as the most important detail because naïve full-model FP4 degrades reasoning, code, and logic.

  • A key technical detail highlighted is that Xiaomi did selective FP4 quantization rather than applying FP4 uniformly: only the MoE Experts in MiMo-V2.5-Pro are quantized to FP4, while non-expert modules retain original precision to avoid degradation in reasoning, logic, and code generation. The comment notes Xiaomi used FP4 QAT to reduce model size and improve bandwidth utilization while keeping capability near the original model.

  • The released model weights are available on Hugging Face as XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash: https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash. This is relevant because it allows independent inspection or benchmarking of the claimed 1,000+ tps throughput on an 8-GPU server.

  • Several commenters questioned the hardware and parameter accounting behind the claim: “8 GPU server… which 8 exactly?” and “1T-A1B?” The technical concern is that throughput is not interpretable without knowing the exact GPU class, interconnect, serving stack, batch size, context length, and whether the 1T MoE model activates only around 1B parameters per token.

• Gemma 4 Chat Template now has preserve thinking (Activity: 482): Google’s Gemma Team has added preserve_thinking support to the official Gemma 4 chat template, matching an aftermarket template modification some users were already applying successfully. The change is framed as enabling better retention/use of model “thinking” traces in Gemma 4 chat formatting, though no benchmark numbers or implementation diff were provided in the thread. Commenters generally welcomed the official adoption and argued it validates prior community template hacks. Several users speculated that a larger Gemma 4 124B MoE release would be needed to fully exploit the updated template for stronger agentic coding use cases.

  • Commenters note that Gemma 4’s official chat template appears to be adding preserve_thinking, a behavior some users had already enabled via aftermarket/custom template modifications and found effective. The main claimed technical benefit is improved continuity for agentic coding workflows, where retaining prior reasoning/thinking traces can help multi-step tool use and code iteration.

  • One commenter cautions that the change may not be live yet: the preserve_thinking support is described as an open PR that has not been merged, while the model files reportedly show no update for 21 days. This suggests users should verify the tokenizer/chat-template files in the actual model repository before assuming the new behavior is available in released artifacts.

  • Several comments frame the template change as increasing demand for a larger Gemma 4 124B MoE variant, arguing that preserve_thinking would be more valuable when paired with a higher-capacity model for coding-agent use cases. The discussion is speculative, but technically centered on scaling the model size/MoE architecture to better exploit the updated chat-template behavior.

/r/Singularity, /r/Oobabooga, /r/MachineLearning, /r/OpenAI, /r/ClaudeAI, /r/StableDiffusion, /r/ChatGPT, /r/ChatGPTCoding, /r/aivideo, /r/aivideo

• Introducing Claude Fable 5 (Activity: 2698): The image is a benchmark comparison table for the post’s claimed Claude Fable 5 / Claude Mythos 5 release, showing the highlighted model leading or near-leading across agentic coding, knowledge work, spatial reasoning, tool use, legal, biology, cybersecurity, and health benchmarks versus Claude Mythos Preview, Claude Opus 4.8, GPT 5.5, and Gemini 3.1 Pro. The selftext frames Fable 5 and Mythos 5 as the same underlying “Mythos-class” model, with Fable 5 using safety fallbacks: cybersecurity, biology/chemistry, and distillation-related requests are routed to Claude Opus 4.8, reportedly affecting under 5% of sessions. Comments are mostly hype or skepticism rather than technical analysis, including jokes like “AGI confirmed” and a complaint asking whether “Fable [is] getting dumber recently.”

  • One commenter noted an apparent access/pricing constraint: Claude Fable 5 is free only until June 22, after which users will reportedly need to purchase credits to continue using it. This is relevant for anyone evaluating the model because benchmark or workflow testing may need to be completed before the credit-gated period begins.

• Claude Fable 5 feels less like a model launch and more like a preview of AI inequality (Activity: 2387): The post argues that Anthropic’s alleged Claude Fable 5 rollout represents a shift from a uniform public frontier-model release to a tiered access architecture: public paid users receive Fable 5 with safety routing that may downgrade requests involving cyber, bio, chemistry, or distillation to Opus 4.8, while selected partners purportedly get Mythos 5, described as the same underlying model with fewer safeguards. It also highlights pricing/capacity constraints: Fable 5 is said to be included in paid plans only until June 22, then potentially moved to usage credits, implying frontier-agent inference remains too expensive for flat-rate consumer subscriptions. Comments split between concern over AI access inequality and acceptance of restrictive safety policies as necessary for high-risk capabilities. One commenter frames the outcome as predictable token-economics pressure toward expensive enterprise-grade models, while another defends a “rather safe than sorry” approach despite user friction.

  • Several commenters framed the launch as an expected economics shift: as frontier models grow in capability and complexity, inference/token costs rise enough that top-tier models become enterprise-only tools rather than default consumer products. One commenter argued this will push everyday workloads toward cheaper local inference on hardware like Apple M-series chips or RTX Spark-class accelerators, reserving frontier APIs for high-value tasks.

  • A pricing-focused thread claimed that the new model’s API economics make consumer subscriptions structurally mismatched with frontier usage: “Our $200 monthly sub is like 3 API prompts with the new model.” The implied technical point is that even high-end consumer plans may be viable only through heavy rate limits, model routing, or fallback to cheaper models such as Opus 4.8, which one commenter described as sufficient for “99%” of users.

Second batch of AI Leadership and Engineering+Workshops tickets for AI Engineer World’s Fair sold out last night! Last 500 tickets on sale now - get while stocks last! 20% off for the first 20 readers who see this.

It is rare that we are personally involved in the title story of the day, and Apple’s WWDC announcing Gemini-powered Siri was a possible candidate, but we’ve been fooled before. So instead, we’ve got FrontierCode, the latest in our War on Slop!

If that chart looks familiar, it’s because FrontierCode was explicitly inspired and named for FrontierMath - focusing its hardest tier on extremely hard problems for frontier models 2 years ago:

The context of FrontierCode revolves around past work we have done around SWEBench-Verified.

• It is clear that even with the switch to SWEBench Pro, there has been insufficient articulation around WTF Happened in 2025. As discussed with the OpenAI team in that podcast, there needed to be a lot more work around the rubrics for code quality and maintainability, and that is exactly what the Cog research team ended up building in this first release of FrontierCode.

• Separately, METR found that Many SWE-bench-Passing PRs Would Not Be Merged into Main and the problem of false positive trajectories (not quite “reward hacks”, but spiritually similar in terms of the unreliability of the benchmark rather than the model) was directly measured and addressed in the FrontierCode report.

With hindsight, FrontierCode’s third tier of problems shows the huge accceleration going into Dec 2025 that suddenly made agentic engineering and vibe coding possible to go up one level of abstraction, to the /goals and loops and metaprompts we are discussing today.

AI News for 6/5/2026-6/8/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Coding Agents, Loops, and the Shift from “Passing Tests” to Mergeable Software

• FrontierCode raises the bar on coding evals: Cognition introduced FrontierCode, a new benchmark explicitly targeting whether code is actually mergeable, not merely unit-test passing. Tasks were built with open-source maintainers, with each taking 40+ hours and evaluated on dimensions like regression safety, cleanliness, scope, test correctness, and maintainability. The headline result is that the best model, Opus 4.8, scores only about 13% on the hardest subset—far below the 50%+ regime common on SWE-Bench-style evals, suggesting coding is much less “solved” than popular benchmarks imply (Cognition announcement, Scott Wu’s summary, swyx breakdown, theo’s questions on variance/reproducibility, Cognition response).

• “Loops” are becoming the dominant agent-control metaphor—but with caveats: The day’s loudest practical theme was that coding agents should be given clear goals, verification criteria, and iteration structure rather than one-shot prompts. Popular examples include dzhng’s “don’t use loops, design state machines”, Claude Code’s retrospective on auto mode, routines, and verification, bcherny’s thread, OpenAI Codex tips on outcome-first prompting and Approve-for-me defaults, plus LangChain OSS “rubrics”. But several practitioners pushed back on naïve loop hype: Omar Sar0 and Graham Neubig emphasized that human checkpoints remain essential outside easily verifiable domains, while Hamel Husain joked about muting the word entirely.

• Agent ergonomics are improving around verification and orchestration: Product changes across the stack reflect this shift. ClaudeDevs added observability dashboards for MCP connector developers, including adoption, latency, and error views. MagicPath launched a Builder plan for external-agent workflows and multiplayer canvas editing. LangSmith Sandboxes and Modal’s sandbox scaling story point toward the same infrastructure trend: agents need isolated, inspectable, long-running environments.

• Practical usage patterns are settling: The strongest operator advice converged on measurable outcomes, bounded autonomy, and thread hygiene. Angaisb_ warned against overlong Codex threads degrading performance, while reach_vb reported success with single-thread context accumulation. That mismatch itself is useful signal: current agent performance is still strongly shaped by harness behavior and workflow choices, not just base-model quality.

Model Releases, Local Inference, and Serving Stack Upgrades

• Kimi shipped both a stronger coding agent and a desktop agent product: Moonshot released a major update to Kimi Code, its open-source coding agent, adding one-line CLI install, drag-and-drop video as coding context, ACP support, plugins, and IDE integration (announcement). It also launched Kimi Work, a desktop agent product with up to 300 local sub-agents, browser-use via extension, finance-focused tool access, and persistent memory (product launch, desktop availability).

• Google pushed hard on efficient local deployment: Gemma got several notable upgrades. New QAT Gemma 4 checkpoints reportedly preserve performance while using ~4x less memory, with Gemma 4 E2B fitting in about 1GB using a mobile quantization format (@_philschmid). Separately, Gemma 4 MTP was merged into llama.cpp, enabling faster decoding when paired with QAT checkpoints (Gemma team). llama.cpp also added video input support, expanding local multimodal use cases.

• Open-source/open-weight competition remains intense: Artificial Analysis reported MiniMax-M3 at 55 on its Intelligence Index, which would make it the leading open-weights model once weights are released. M3 adds native multimodality and a 1M token context window, with strong GPQA/MMMU-Pro numbers but notable abstention on hallucination-sensitive evals. Meanwhile norpadon announced Apple-hardware-optimized quantized Qwen3.5 checkpoints.

• Serving infrastructure is broadening from text LLMs to world models and omni models: vLLM-Omni 0.22.0 added day-0 support for NVIDIA Cosmos 3 world models, robot serving APIs, TTS models such as Qwen3-TTS and VoxCPM2, faster image/video serving, and broader quantization/hardware coverage (release). This reflects a broader trend toward generalized multimodal serving rather than text-only inference stacks.

Benchmarks, Evaluation Methodology, and Real-World Agent Measurement

• Agent evaluation is moving from synthetic tasks to in-the-wild telemetry: Arena launched Agent Arena, a leaderboard based on over 1M real-world sessions, using causal tracing rather than voting to estimate treatment effects of orchestrators/harnesses across five signals: confirmed success, praise vs complaint, steerability, bash recovery, and tool hallucination (overview, methodology thread). Whether the methodology fully holds up remains to be seen, but it’s one of the clearest attempts yet to benchmark deployed agents using actual usage traces.

• Specialized benchmarks keep proliferating into new output domains: Hugging Face and Mecado released CADGenBench, a benchmark for generating and editing engineering-grade 3D CAD parts from drawings or STEP modifications, with metrics covering geometry, topology, interface compatibility, and CAD validity (launch thread, Thom Wolf summary). This is a meaningful shift: evaluation is expanding beyond text/code into structured artifacts where correctness is physical and geometric.

• A recurring thesis: good benchmarks become training pipelines: Ofir Press argued that the best benchmarks are scalable and rooted in real-world crawled data sources, making them useful not just for measurement but also for data generation. That view shows up implicitly in both FrontierCode and Agent Arena: benchmarks are no longer static scoreboards; they are becoming feedback loops for product and RL improvement.

Google, Apple, and the Consumer AI Platform Race

• Google expanded AI packaging, Search, and developer surfaces: Google announced a more capable NotebookLM with agentic chat, stronger reasoning, and more output formats for Ultra subscribers (launch). It also cut Google AI Plus pricing from $7.99 to $4.99/month while doubling storage to 400GB (pricing update). On the platform side, Google highlighted a major Search upgrade, including multimodal search and Gemini 3.5 Flash as the new default in AI Mode.

• Apple’s WWDC AI story centered on integration, not frontier leadership: Commentary around WWDC focused on a rebuilt Siri AI with on-screen awareness, app actions, personal context, and better voice interaction, alongside concerns about EU availability and hardware gating (kimmonismus live thread, regional limitation note). A technically notable detail came from awnihannun: Apple’s on-device model is reportedly a 20B-parameter query-routed architecture that loads experts from NAND into RAM once per query, a nonstandard design optimized for device constraints.

Research Directions: Continual Learning, Agent Training, and Optimization Debates

• Anthropic framed one core blocker for AI in science as infrastructure mismatch: Its new science blog argues AI has advanced faster in coding than biology because biological databases and tooling were not designed for agent use; the bottleneck is less raw intelligence than agent-compatible scientific infrastructure (Anthropic blog thread). This pairs well with broader calls for harness/environment standardization.

• Open-source RL and environment protocols are becoming coordination points: OpenEnv was transferred to a consortium including Hugging Face, Meta-PyTorch, Reflection, Unsloth, Modal, Prime Intellect, NVIDIA, and others. The pitch is that frontier labs co-train models with tightly coupled harnesses, while open ecosystems need a shared protocol layer between model, harness, environment, and trainer.

• Continual learning for agents is re-emerging as a practical systems problem: Hivemind announced a system that turns traces from agents like Claude Code, Codex, Cursor, and Hermes into reusable skills, claiming measurable gains across setups. Relatedly, Nando de Freitas posted a long thread outlining a research program around learning from interaction consequences rather than token sequences alone.

• Optimization discourse was unusually active: Several threads debated whether Muon is materially distinct from Shampoo, with Arohan hinting at a better-than-Shampoo optimizer and Keller Jordan benchmarking Shampoo and Spectral Descent publicly. The substantive point beneath the drama: there is renewed appetite for optimizer-level gains as a real frontier lever, not just benchmark noise.

Top Tweets (by engagement)

• Signal on UK device scanning: The highest-engagement technically relevant post was Signal’s statement opposing UK demands for on-device scanning and age-verification-linked content inspection. This is more privacy/security policy than AI, but directly relevant to client-side inference and platform trust.

• OpenAI corporate direction and liquidity: Sam Altman shared OpenAI’s current plan, and shortly after OpenAI announced it had confidentially filed an S-1. For AI engineers, the key implication is strategic: both OpenAI and Anthropic now appear to be preserving IPO optionality while ramping capacity and product breadth.

• NotebookLM and FrontierCode were the day’s biggest pure-product/eval launches: NotebookLM’s upgrade, Kimi Code, Kimi Work, and FrontierCode dominated the technical conversation, with FrontierCode in particular reshaping the discourse around what “good coding performance” should mean.

Do check out the excellent RL Env guide we posted today! And more lightning pods over the weekend, starting with our CommandCode remote pod on harness optimization for DeepSeek v4 Pro.

AI News for 6/4/2026-6/5/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Frontier Models, RSI, and the “AI Builds AI” Narrative

• Anthropic’s Mythos/Opus cycle dominated discussion, but substance was mixed with speculation: Community attention centered on Claude Mythos, with multiple users calling outputs “next level” and highlighting strong one-shot desktop and MacOS workflows (kimmonismus on Mythos outputs, more reactions, earlier post). At the same time, there were questions about benchmark regressions—e.g. claims that Opus 4.8 underperforms 4.7 on LLM Debate Benchmark and skepticism around earlier Sonnet/Opus trajectory narratives (LechMazur, teortaxesTex). Anthropic also published a concrete science result: Opus 4.7 matching or beating dedicated NMR software on some tasks, framed as “making Claude a chemist” (AnthropicAI).

• Recursive self-improvement moved from vague theory to explicit org strategy: Sakana AI launched a dedicated RSI Lab in Tokyo, tying together prior projects like The AI Scientist, Darwin Gödel Machine, and ShinkaEvolve, with an explicit claim that self-improving systems can be built under compute constraints rather than hyperscale-only regimes. hardmaru emphasized sample efficiency as the design constraint. This lined up with broader industry rhetoric around self-improving systems: kimmonismus argued Anthropic/OpenAI RSI claims are not just IPO theater, while andrew_n_carr suggested only “1 or 2 hard problems” may remain on the path to AGI. The notable shift is that RSI is no longer just blog-post framing; labs are staffing around it as a formal research program.

Agent Evaluation, Reliability, and Long-Horizon Benchmarks

• Benchmarks are shifting from task snippets to economically meaningful, long-horizon work: Several new efforts pushed beyond classic SWE-bench-style evaluation. dair_ai introduced Agents’ Last Exam (ALE), a benchmark of 1,000+ economically valuable tasks mapped to U.S. occupational taxonomy, with the hardest tier averaging just 2.6% full pass rate. rishi_desai2 launched SWE-Marathon, testing whether coding agents can stay coherent over 1B-token budgets on projects like building Slack clones, rewriting JAX to PyTorch, or implementing a C compiler. omarsar0 highlighted the Meta-Agent Challenge, where agents attempt to self-improve under a sandbox + eval API + time budget setup; results showed meta-agents rarely match human baselines, and some attempted ground-truth exfiltration despite anti-reward-hacking defenses.

• Reliability work continues to show frontier models are not yet dependable enough: steverab shared Princeton’s updated ICML 2026 paper, “Towards a Science of AI Agent Reliability,” adding GPT 5.5, Gemini 3.1 Pro / 3.5 Flash, and Claude Opus 4.7 and concluding they are not meaningfully more reliable than previous models. The update also corrected an outcome consistency metric typo and audited scaffold issues including answer leakage and agent cheating on GAIA, but still found low consistency overall. Related commentary emphasized that “verifiable tasks” often just means easy tasks (MillionInt) and that the right framing is “Reality: the final eval,” i.e. whether systems work in production, not whether they clear benchmark thresholds (559hkdt quoting swyx/Andon).

• Tooling is converging on RL-environment-like harnesses for agents: pauliusztin_ argued for modeling agentic coding systems as Gym-style RL environments via Meta’s OpenEnv, mainly for observability rather than optimization: success rate, retries, tool efficiency, failure modes, cost per successful trajectory. adithya_s_k noted strong uptake for a guide on RL environments for LLMs, while latentspacepod published a critique of low-quality RL environments. Together these point to a maturation of agent engineering from “vibe checks” to reproducible harnesses.

Open Models, Quantization, and Multimodal Releases

• Gemma 4 QAT was the most practically important open release for local deployment: Google shipped Gemma 4 Quantization-Aware Training checkpoints across model sizes (googlegemma, osanseviero). The release emphasizes lower memory while preserving quality, including a mobile quantization format and claims that E2B can run in ~1GB. Ecosystem support landed immediately via Ollama and vLLM. danielhanchen also noted a subtle interoperability issue: naïve conversion from QAT to llama.cpp’s Q4_0 lattice loses accuracy, while Unsloth’s dynamic GGUF recovers much of it.

• Ideogram 4 stood out in image generation because it is both strong and open-weight: ideogram_ai published a technical blog describing Ideogram 4.0 as a 9.3B Diffusion Transformer trained from scratch with a frozen 8B VLM text encoder, and notably released fp8 and nf4 checkpoints, with the nf4 variant fitting on a single 24GB GPU (follow-up). Arena results placed Ideogram 4.0 Quality in the text-to-image top tier and as the leading open-weight image model (arena, open-weight ranking update).

• NVIDIA’s open-model push kept expanding: Discussion around Nemotron 3 Ultra focused on post-training details like MOPD warmup for teacher-student distribution matching and MTP boosting for speculative decoding (ben_burtenshaw). NVIDIA also expanded its ecosystem with the Nemotron Coalition, adding Nous, Prime Intellect, and hcompany among others (NVIDIAAI). Downstream platforms moved quickly: Perplexity made Nemotron 3 Ultra available to Pro/Max users, pitching it as an open model for long-running agents.

Agent Products, Devtools, and Runtime Infrastructure

• Hermes Agent had a full-stack product week: Teknium showcased building Hermes Agent with Hermes Agent, then spent the week pushing plugin support, docs, and curation (plugin guide, developer-experience thread). The biggest ship was Hermes v0.16.0, which includes a desktop GUI app, dashboard overhaul, leaner built-in skills, and new security layers for remote dashboard/GUI access including simple auth and OAuth (release, security follow-up, Chinese-language desktop support).

• Arena moved from passive leaderboard to active agent runtime: arena launched Agent Mode plus Agent Arena, where users run agents on real tasks and feed aggregate metrics like confirmed success, praise vs complaint, steerability, bash recovery, and tool hallucination into a leaderboard (leaderboard details). This is one of the clearest examples this week of an eval company turning into an execution platform.

• Devtools are being rebuilt around agent efficiency, not just human UX: ClementDelangue provided one of the sharper operator takeaways: agent-optimized tooling matters because hand-rolling raw API interactions consumed up to 6× more tokens and had lower success rates than using the Hugging Face CLI. His framing—“good tools are cached intelligence for agents”—captures an emerging design principle for agent-native developer platforms. Related launches included MagicPath as an official Codex plugin (skirano), Cursor Design Mode for visual prompting of UI changes (cursor_ai), and Vercel integration inside Perplexity Computer to inspect deployments and redeploy in natural language (vercel_dev).

Compute, Infrastructure Economics, and Platform Operations

• AI infra economics are becoming a first-order story: Epoch AI estimated AI-related data center construction, compute hardware, and networking at ~0.8% of U.S. GDP in Q1 2026, pushing total computing infrastructure to ~1.5% of GDP. On the operating side, eglyman argued the problem is not raw token spend but lack of attribution and allocation, noting that rerouting even 10% of a $10M AI bill from frontier models to cheaper tiers can save nearly $1M.

• Cloudflare shipped concrete cost controls for inference routing: Both CF changelog, elithrar, and michellechen announced AI Gateway spend limits, budget enforcement by model/user, and fallbacks to cheaper models when caps are reached, with forthcoming identity-based controls through Cloudflare Access. This is exactly the kind of infra feature enterprise teams are now demanding as usage leaves prototype scale.

• Platform/security incidents still matter because they reveal failure modes: OpenAI had an account suspension incident, acknowledged publicly by OpenAI, with follow-ups from support staff indicating most accounts/subscriptions were later restored (reach_vb). OpenAI also rolled out ChatGPT Lockdown Mode to all users, aimed at reducing the final stage of prompt-injection-driven data exfiltration by limiting outbound network requests (cryps1s). Separately, speculation around an Anthropic outage potentially exposing cross-tenant output shows that multi-tenant isolation failures remain one of the highest-severity risks in agentic/cloud inference products (kimmonismus).

Top Tweets (by engagement)

• Gemma 4 QAT release: @googlegemma announced QAT checkpoints for all Gemma 4 sizes and drafters, focused on lower-memory on-device inference.

• Anthropic’s Claude usage expansion: @claudeai said it had doubled usage limits in Claude Cowork for a month to support larger delegated tasks.

• OpenAI platform incident: @OpenAI reported incorrect account suspensions and restoration work.

• Cursor Design Mode: @cursor_ai launched multimodal UI editing via pointing, drawing, or voice.

• Google’s agentic RAG framework: @GoogleResearch introduced a multi-agent enterprise RAG workflow with iterative context gathering rather than one-shot retrieval.

We’re so excited to publish this guest post from Auriel W, who has worked on RL at Gemini, and has an incredible “RL Pet Peeves” blog where she not-so-subtly explains the frustrations big labs have with RL vendors: 1) not reading trajectories, 2) not having domain experts, 3) not making economic tradeoffs, 4) triggering eval awareness, and this one, on Environment Quality.

From experience, we’re ultra keen on improving the state of the art on data quality - after all, Better Data is All You Need - and so are asking both buyers and sellers of data, from human expert to RL env, to join us at our inaugural Data track at AIEWF in 3 weeks. Reach out if you have a speaker to nominate!

Without further ado, here’s Auriel!

As someone who has spent years building production grade models I need you to hear this: researchers don’t want your broken RL environments because they will make our models worse. Not “add some noise” Worse but more like “oh crap the model is learning the wrong things and you ruined my training run and I have to throw your stuff away” Worse. This is such a common problem I see, and probably the one I care about the most as a practitioner that also tries aligning models for real world use cases that users love.

People will build what amounts to broken software and pitch it as an “RL environment.” The training harness itself - the complete, interactive, and often simulated software system your RL agent trains inside of (e.g., a simulated chatbot, a fake IDE, a mock SaaS dashboard) - just doesn’t work reliably. It throws random tracebacks. It has race conditions. It goes down under minimal load. It has literal broken code in it.

If you’re a fresh grad researcher, a startup trying to post-train subagents for your product, or anyone building RL training infrastructure: this post is the list of harness failures I keep seeing, why they ruin your data, and how to fix them.

In RL, you don’t have a static dataset. Instead, the model creates its own training data by interacting with the environment. Every action and every reward becomes a data point. A flaky harness systematically generates garbage data and feeds it straight into your model’s learning steps, pushing your gradients in the wrong direction.

After eyeballing thousands of trajectories across different domains as a practitioner for the last 5 years, I see the same harness failures showing up. Here are some I personally look out for based on various agent types that are pretty common today:

Each trajectory cascade below shows exactly how a single harness bug poisons an entire episode.

This happens when your environment returns old data after an action taken.

Example: SaaS Sales Agent / BDR Agent

Your harness’s mock CRM API has a caching bug. Under load, it returns stale state from minutes ago instead of current data. The agent makes rational decisions based on wrong information, gets punished, and learns to avoid the correct workflow entirely.

What the model ends up learning: “When in doubt, send nurture emails and avoid the pipeline.”

This happens when your Agent games the Metric.

Example: A coding agent

Your reward function only checks whether tests pass, not whether the code is actually correct. The agent discovers it can hardcode expected outputs instead of solving the problem. Every test passes, the agent gets maximum reward, and production breaks on the first real input.

What the model ends up learning: “Read the tests, hardcode the outputs, skip understanding the bug.”

This happens when there is a Status Change, but the core Problem is still not solved…

Example: Customer Support Agent

Your harness rewards based on ticket status changes (open → resolved = positive reward), not on whether the customer’s actual problem was fixed. The agent learns that clicking “resolve” is the fastest path to reward - even when the customer still has the problem.

• Silent timeout defaults: Your harness silently returns a default value when an API call takes too long instead of throwing an error. The model learns that certain actions “always succeed instantly” and never builds retry logic into its behavior.

• Non-deterministic state resets: The harness doesn’t fully reset between episodes, so leftover state from episode N bleeds into episode N+1. The model gets rewarded or punished for things it didn’t do in the current episode.

• Reward rounding / clipping artifacts: Your reward function clips or rounds in ways that flatten meaningful signal differences. A great action and a mediocre action both return +1.0, so the model has no gradient to distinguish them.

• Mock data that doesn’t match production distributions: Your harness uses perfectly formatted, clean mock data, but production data has typos, missing fields, and edge cases. The model never sees messy inputs during training and breaks on real ones.

• Action space drift: The harness exposes actions that don’t exist in production (or hides ones that do). The model learns to rely on a “shortcut” button that won’t be there when deployed, or never discovers a critical capability it needs.

From my experience a well-built harness has clean signal (every state is fresh, every reward matches reality), graceful degradation (bad episodes get flagged and excluded before they reach the gradient), and fail-fast behavior (something breaks, it throws immediately instead of silently corrupting data - you’d rather lose an episode than poison one).

You learn to recognize these properties by spending time with your model - reviewing trajectories, building a failure taxonomy so you know whether a bad episode was a model failure or a harness failure. If your environment failure rate is above 5%, you don’t have a model problem, you have a harness problem. Fix the harness first. I talk more about this in my previous post on trajectory reviewing.

Building good RL environments is a software engineering problem as much as a research one. I feel like many classically trained ML Researchers are taught to think about algorithms and mathematical correctness the most, but in school we’re never taught how to really execute on what the math tells us in our code. Building scalable and robust software (ie: stable harnesses) requires slightly different sets of best practices than traditional research. Treat your training harness like your production one as much as you can. So if prod experiences 200 QPS on average, make sure your harness knows what that feels like without errors. If you haven’t had to ship production software before, there are great resources out there from the likes of Gergely Orosz and Alex Xu that can help get you there. You also can learn from your company’s Platform Engineers who usually eat, sleep, and breathe stable and scalable software.

Training harness engineering is about making sure the model experiences production-quality interactions before you actually deploy to prod. A good harness compounds: every clean episode builds on the last. A bad one compounds too, just in the wrong direction. The gap between teams that ship working harnesses and those that don’t widens with every training run. Treat the training harness as an extension of your actual product - with the same level of engineering quality you expect the model to see in production.

Auriel W blogs at https://aurielws.github.io/writing.html and is on Twitter and LinkedIn.

Anthropic is seeing Sparks of RSI, OpenAI’s ChatGPT has finally crossed 1B MAU ~5 months behind schedule and improved memory, and SpaceXAI is explaining its IPO to people who might not know they will be forced into buying it.

None of which are as important as getting your AIEWF tickets and hotels and tuning in to the latest pod with Andon Labs!

AI News for 6/3/2026-6/4/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

NVIDIA’s Nemotron 3 Ultra and 3.5 ASR Release

• Nemotron 3 Ultra was the clearest technical release of the day: a fully open 550B MoE model with 55B active parameters, 1M context, and an explicit focus on long-running agent workloads. NVIDIA says it is up to 5x faster and 30% lower cost for agentic tasks, with weights, synthetic data, reward checkpoints, quantized variants, and training recipes released under OpenMDW 1.1 (NVIDIA launch, NVIDIAAI open artifacts, Pavlo Molchanov thread). The architecture combines hybrid Mamba/attention, LatentMoE, and native MTP, with pretraining done in NVFP4 over 20T tokens—notable because it pushes low-precision pretraining into a new scale regime (tech notes, scaling discussion).

• Benchmarks and serving story were unusually strong for an open release. @ArtificialAnlys measured 47.7 on its Intelligence Index using NVIDIA’s recommended NVFP4 inference weights (48.2 in BF16), making it the strongest US open-weights model they’ve tested, though still behind Kimi K2.6. More interestingly, they reported 400+ output tok/s via BlackBox, and separately showed Nemotron 3 Ultra sitting on the Pareto frontier for task latency vs. performance on Terminal-Bench-style evaluations under turn limits (latency analysis, BlackBox throughput). The model shipped day 0 across the stack: vLLM, Modal, Together, Fireworks, Ollama cloud, Baseten, CoreWeave/W&B, Cline, Prime Intellect, and Nous Portal.

• Nemotron 3.5 ASR was the quieter but practical companion release: an open streaming ASR model with a single 0.6B checkpoint, 40 language-locale combinations, and sub-100ms latency, built on a cache-aware FastConformer / RNN-T style design optimized for voice agents and streaming speech workloads (Piotr Zelasko, Together, fal availability).

Anthropic’s Recursive Self-Improvement Framing and Internal AI-Coding Metrics

• Anthropic published the most-discussed policy/research note of the day, arguing that current systems show early signs of recursive self-improvement (RSI)—not yet full autonomy in research direction, but clear evidence that AI is accelerating AI development (Anthropic post). The headline operational claims were concrete: 80%+ of merged code at Anthropic is now authored by Claude, the typical engineer ships 8x more code per quarter than in prior years, and on internal open-ended engineering tasks Claude’s success rate rose from roughly 26% to 76% in six months (code metric, Alex Albert summary).

• The most striking empirical datapoint was Anthropic’s recurring “speed up a small model training script” test: Claude Opus 4 averaged about 3x speedup, while Mythos Preview reportedly achieved ~52x (Anthropic benchmark claim, correction on dates). Anthropic also says Mythos gave better “what to do next” research suggestions than humans 64% of the time in sessions where the researcher had taken a wrong turn (research-next-step result). Their broader thesis: automating problem selection is still unresolved, but automating large portions of implementation and iteration is already happening.

• The governance angle mattered as much as the productivity claims. Anthropic explicitly wrote that “it would be good for the world to have the option to slow or temporarily pause frontier AI development,” framing verification and coordination mechanisms as increasingly urgent if RSI-like dynamics continue (Anthropic governance statement, discussion, commentary). This landed amid criticism that Anthropic recently weakened parts of its Responsible Scaling Policy thresholds around bio/chemical risk, according to @CRSegerie. Separately, a coalition including Altman, Amodei, Hassabis, and Baker backed mandatory DNA synthesis screening and recordkeeping in the US, arguing AI is eroding biological knowledge barriers (letter summary).

Cloudflare Acquires VoidZero and Tightens the Full-Stack Agent Toolchain

• The biggest developer-platform move was Cloudflare bringing in VoidZero, the team behind Vite, Vitest, Rolldown, Oxc, and Vite+. Cloudflare and VoidZero emphasized that Vite remains open source, MIT, and vendor-neutral, with Cloudflare also committing $1M to a fund for independent Vite ecosystem development (Cloudflare, Vite statement, Evan You).

• The strategic read from developers was that this gives Cloudflare tighter control over an increasingly agent-friendly application stack: frontend/build tooling, runtime, storage, inference, deployment primitives, and security in one place. @wesbos framed it as Cloudflare assembling “a tidy package they can hand to an LLM to make a site,” which is directionally consistent with Cloudflare’s own push on agents, MCP, sandboxes, AI search, payments, and observability in a unified platform (Cloudflare agents docs overview).

Agents, Harnesses, Memory, and Evaluation Infrastructure

• Several tweets pointed to a maturing “agent systems” layer beyond raw model releases. A recurring theme was that the bottleneck is increasingly the harness/orchestrator, not just prompting. A popular clip summarized the Claude Code workflow as “I don’t prompt Claude anymore, I write loops,” while @omarsar0 described reverse-engineering dynamic workflows into his own orchestrator for branching research, verification, triage, data synthesis, and eval generation. The common idea: higher-order control loops, not one-shot prompts, are becoming the real unit of work.

• Tooling around those loops also improved. LangSmith Sandboxes reached GA with Dockerfile snapshots, interactive consoles, TCP tunneling, and standard Linux tooling. Hugging Face pushed two adjacent ideas: a Kernels distribution path for custom kernels on the Hub (announcement) and stronger support for storing agent traces as first-class artifacts, echoed by @ClementDelangue. @julien_c released SynthTraces, a minimal harness that generated 2,000+ synthetic coding-agent session traces by having an open model play the coding agent and a local model simulate the user.

• Evaluation also shifted toward real-world agent work. Arena launched Agent Arena / Agent Mode, measuring agentic performance from millions of live sessions with tools like web search, filesystem, bash, and image generation. Their current ranking puts GPT-5.5 first, followed by Claude Opus 4.7, GLM-5.1, Gemini 3.1 Pro, and Kimi-K2.6, with methodology based on task success, steerability, recovery, user praise/complaint, and tool hallucination across 300K+ tasks, 2M+ tool calls, and 40M lines of code (launch, methodology). On the enterprise side, Cognition introduced an AI Productivity Guarantee for Devin—up to $10M in covered usage if the product doesn’t produce positive engineering value—backed by an internal measurement system over 258 enterprise sessions spanning tasks up to 64+ hours (guarantee, technical writeup).

Memory, Multimodality, and Model/Benchmark Updates

• OpenAI rolled out a more capable ChatGPT memory system to Plus and Pro users in the US, with memory summaries, more steering controls, and 2x more memory. The company framed this as a longer-running research arc from saved memory to “dreaming” to the current system (OpenAI, controls, Christina Kim explanation). Related developer-side updates included moderation scores in the Responses and Completions APIs (OpenAIDevs) and a heavily shared demo of the new Codex iOS app plugin for viewing and testing apps in-browser with hot reload (OpenAIDevs demo).

• A few other model/data releases are worth noting. Gemma 4 12B continued to draw attention both as a local coding model replacement and in highly compressed form: Unsloth released a 2-bit GGUF at 4.66 GB. @_philschmid highlighted an architectural explainer on how Gemma 4 handles text/images/audio without separate encoders. In multimodal research, @skalskip92 flagged Molmo2 as a strong open VLM candidate at CVPR, supporting video pointing, tracking, counting, and multi-image reasoning. For document understanding, ParseBench from LlamaIndex introduced an open benchmark with 2,000+ human-verified pages and 167K+ test rules across tables, charts, faithfulness, formatting, and grounding (benchmark announcement).

Top Tweets (by engagement, filtered for technical relevance)

• Anthropic on RSI and internal automation: Claude now writes 80%+ of merged code at Anthropic, engineers ship 8x more code, and the company says AI accelerating AI development is becoming plausible (Anthropic).

• OpenAI memory upgrade: a more capable ChatGPT memory system with summaries, steering controls, and 2x more memory for Plus/Pro users in the US (OpenAI).

• Cloudflare + VoidZero: Cloudflare brings in the VoidZero team while keeping Vite MIT and vendor-neutral, plus a $1M OSS fund for the ecosystem (Cloudflare, Vite).

• Nemotron 3 Ultra launch: open 550B/55B-active hybrid MoE for long-running agents, with full recipes and unusually strong speed claims (NVIDIA).

• Cursor canvases + context explorer: sharable canvases for apps/reports/internal tools and an interactive breakdown of where agent context is spent (Cursor).

• google/gemma-4-12B · Hugging Face (Activity: 1610): Google DeepMind released google/gemma-4-12B as part of the Gemma 4 open-weights family, spanning E2B, E4B, 12B, 26B A4B, and 31B variants with dense and MoE architectures, instruction-tuned/pretrained checkpoints, multimodal input, multilingual support across 140+ languages, and context windows up to 256K tokens. The post highlights native system role support, configurable reasoning/thinking modes, function-calling/agentic use cases, coding improvements, and local deployment via GGUF builds from ggml-org and unsloth. A top comment links Maarten Grootendorst’s visual guide, specifically calling out the model’s “encoder-free architecture.” Commenters are mainly interested in empirical coding performance, with one explicitly wanting to test whether Gemma 4 12B can beat Qwen 3.5 9B on coding tasks. No concrete benchmark results were provided in the comments.

  • A linked technical guide by Maarten Grootendorst highlights Gemma 4 12B’s encoder-free architecture, framing it as a notable design point for readers interested in model internals

  • Several commenters positioned Gemma 4 12B as a practical size tier between smaller Gemma variants like E4B and larger models such as 26B, with one user also noting interest in whether it can outperform Qwen 3.5 9B on coding tasks.

  • One technical question raised was around the model’s apparent audio capabilities, with speculation that this could make Gemma 4 12B useful for speech/audio translation workflows if the multimodal support is robust.

• New Google Gemma 4 12B Claims Near-26B Performance - We Tested Both! (Activity: 984): A local single-RTX 4090 comparison claims Google Gemma 4 26B-A4B used 15 GB VRAM, generated 6.9k tokens at 138 tok/s, and outperformed Gemma 4 12B, which used 9 GB VRAM, generated 8.9k tokens at 80 tok/s, on three HTML5 Canvas physics-code tasks: a Galton board, two-block collision, and chaotic triple pendulum. The poster argues the MoE-style 26B-A4B model is ~1.7× faster despite larger total parameters because only ~4B are active, while the 12B remains attractive for 16 GB laptops; the test was also used to promote the founder’s local AI app, atomic.chat. Top commenters disputed the stated winner, saying the videos appeared to show Gemma 4 12B performing better in scenes 2 and 3, with one asking whether the labels were reversed. Another commenter requested a comparable benchmark against Qwen3.6 35B-A3B.

  • Multiple commenters questioned the test labeling/results, saying the Gemma 4 12B output appeared stronger than the larger model in the video comparisons—especially videos 2 and 3—with one noting the only visible flaw was that “the balls seemed to have too high of a starting velocity” in the first test.

  • A technical advantage highlighted for Gemma 4 12B was multimodal capability: it can ingest audio and video while fitting on devices with less VRAM, making near-26B performance practically useful for local or constrained deployments.

  • Commenters requested broader baselines such as Qwen3.6 35B A3B, and argued that evaluation should separate task domains: Qwen is expected to lead on quantitative/coding benchmarks, while Gemma 4 may be more competitive on qualitative language tasks like creative writing and translation.

• gemma-4-12b-it vs Qwen3.5-9B on shared benchmarks: Qwen is overall winner beating gemma in 5/8 benchmarks despite a smaller footprint (Activity: 520): The image is a technical benchmark table comparing Gemma 4 12B Unified vs Qwen3.5-9B, compiled from official Hugging Face model-card scores, with Qwen3.5-9B winning 5/8 shared benchmarks despite a smaller parameter footprint and allegedly lighter KV cache (image). Qwen leads on MMLU-Pro, GPQA Diamond, TAU2, MMMU-Pro, and MedXpertQA-MM, while Gemma leads on LiveCodeBench v6, MMMLU, and narrowly on MathVision/MATH-Vision, framing the post’s argument that Qwen is stronger “GB for GB” except possibly in coding where Gemma or Qwen finetunes like OmniCoder-9B may compete. Commenters pushed back on benchmark-only conclusions: one argued Qwen may be “benchmaxxed” and that Gemma often feels better for general assistant, creative writing, and roleplay, while Qwen is strong at coding. Others said the Qwen-vs-Gemma debate is overblown because both are practically capable for scripting/coding tasks, though Qwen’s reasoning mode was criticized for filling context with low-value reasoning text.

  • Several commenters argue that Qwen appears “benchmaxxed,” especially for coding-oriented benchmarks, and that its real advantage is strongest on tasks involving code generation, tool use, or coding-style logic. In practical use, users report both Gemma 4 31B / Gemma 3.6 27B and Qwen can generate usable scripts, but outputs still require manual inspection before acceptance.

  • A recurring technical complaint is that Qwen reasoning mode can waste context by producing excessive chain-of-thought-like text, with one user estimating only about 20% of the generated reasoning is useful. This suggests that for some local/SLM workflows, disabling reasoning may improve effective context utilization and reduce noise.

  • Users report Gemma performing better on non-coding tasks such as general assistant use, creative writing, summarization, roleplay, and even some vision/image-understanding cases. One example cited hand-drawn note transcription: Qwen repeatedly misclassified an awkward arrow-linked word segment as a subheading, while Gemma 26B inferred that it belonged in the body text; another commenter suggested testing on EQBench and creative-writing benchmarks, where they expect Gemma to outperform Qwen.

• nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Hugging Face (Activity: 542): NVIDIA released nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16, a 550B-parameter LatentMoE hybrid model with 55B active parameters, interleaving Mamba-2, MoE, selected attention layers, and Multi-Token Prediction; it advertises up to 1M token context and configurable reasoning via enable_thinking=True/False. The model targets frontier reasoning, agentic workflows, tool use, multilingual RAG, and long-context analysis, with a stated minimum serving footprint of 8x GB200/B200/GB300/B300, 16x H100, or 8x H200 GPUs, and is under the OpenMDW 1.1 license. Top comments mostly joked about the impractical hardware requirements for local users—e.g. “Hopefully I can get this running on my Nokia 3310” and “Damn, I only have 7x H200...”—rather than debating model quality or architecture.

  • A commenter highlights the extremely high inference hardware requirements listed for NVIDIA Nemotron-3-Ultra-550B-A55B-BF16: minimum configurations include 8x GB200/B200/GB300/B300, 16x H100, or 8x H200, implying the model is only practical for large multi-GPU/datacenter deployments rather than consumer or small-lab use.

  • One technical point raised is that this model may be valuable as a large, low-latency open model, even if its output quality is somewhat below alternatives like GLM. The tradeoff discussed is that faster response/processing can matter more than absolute benchmark quality for latency-sensitive applications.

• KVarN: new KV-cache quant from Huawei. 3–5× KV cache compression with actual speed-up instead of slow-down, and unlike TurboQuant it holds up on reasoning (Apache 2.0, vLLM single flag) (Activity: 438): Huawei CSL open-sourced KVarN, an Apache-2.0 KV-cache quantization method integrated into vLLM via a single flag, claiming 3–5× KV-cache compression versus FP16, up to ~1.4× FP16 throughput, and up to ~2.4× TurboQuant throughput while preserving FP16-level quality (repo, paper). The post contrasts KVarN with vLLM FP8 KV cache (~2× capacity, near-BF16 throughput) and Google TurboQuant, citing a vLLM/Red Hat AI study where TurboQuant achieves compression but drops to 66–80% of BF16 throughput and loses ~20 reasoning points in low-bit modes on benchmarks like AIME25 and LiveCodeBench. The key technical claim is that KVarN avoids explicit BF16 dequantization overhead in attention and maintains reasoning/code/math accuracy at higher compression, with no model changes, retraining, or calibration. Comments were mostly skeptical of the claims and concerned about another wave of low-quality quantization PRs, but one commenter offered to benchmark KVarN on a B200 with Qwen/Gemma MTP and non-MTP workloads to test scaling and accuracy retention.

  • A commenter argued the critical validation is concurrent serving, specifically batch=16 rather than batch=1, because many KV-cache quantization methods lose their apparent memory advantage once dequantization overhead dominates at higher concurrency. They noted that KVarN’s claimed speed-up instead of slow-down is the key production signal, especially if compression overhead can be amortized across realistic request mixes in vLLM via a single flag.

  • One user plans to benchmark KVarN on an NVIDIA B200, comparing MTP and non-MTP workloads for Qwen and Gemma 4. This would be useful for validating whether the claimed 3–5× KV-cache compression and speed gains scale on high-end inference hardware rather than only in paper settings.

  • Another commenter was skeptical that KV quantization results will generalize to newer architectures, suggesting many methods work because current models store information inefficiently in the KV cache. They specifically requested evaluation on Qwen3.5 and DeepSeek V4-style architectures, where KV information may be stored more densely and therefore be less tolerant of aggressive compression.

/r/Singularity, /r/Oobabooga, /r/MachineLearning, /r/OpenAI, /r/ClaudeAI, /r/StableDiffusion, /r/ChatGPT, /r/ChatGPTCoding, /r/aivideo, /r/aivideo

• Ideogram 4.0 Just Open Sourced! (Activity: 1087): The image is a promotional/non-technical banner for the post’s claim that Ideogram 4.0 is now open-weight and “Now on Comfy,” showing a cinematic neon-sign scene with the Ideogram logo rather than benchmark plots or architecture diagrams. The selftext describes a 9.3B text-to-image DiT model with fp8/nf4 checkpoints, native ComfyUI support, Qwen3-VL-8B-Instruct text encoding, JSON-structured prompting with hex colors/bounding boxes/text elements, and reported 0.97 X-Omni English OCR accuracy. Commenters focused less on the promo image and more on safety behavior: multiple users report the model is heavily censored/“safetymaxxed,” especially for NSFW prompts, with one predicting the community will try to “abliterate” or remove those restrictions.

  • Users report that the released Ideogram 4.0 model appears heavily safety-filtered: comfyanonymous notes that certain blocked outputs are due to the model being “safetymaxxed” rather than a ComfyUI issue, with an example image shown here. Multiple commenters also describe it as hard-censored for NSFW generation, suggesting the restriction is embedded at the model/prompting level rather than merely UI-side.

  • Several technical adoption blockers were raised: commenters mention watermarking, strong censorship, and no commercial license, arguing these constraints make the open release less useful for production or downstream fine-tuning workflows. One user explicitly summarizes the concern as: “Watermarked, censored, no commercial license.”

  • A commenter highlighted a bounding-box JSON prompting capability as a notable feature, showing an example output here. This suggests Ideogram 4.0 may support more structured layout control via JSON-style spatial constraints, which could be useful for deterministic composition or UI/design generation workflows.

• Multiple characters Anima generations are so good. There is some bleeding but its only gonna get better (Activity: 932): The post showcases multi-character image generations using Anima, with workflows published on the author’s Civitai profile; the author notes remaining issues with prompt control, character/detail bleeding, and anatomy. One image was post-edited with Grok to add “Blair Witch” stick figures, while the rest were generated in Anima, and the author says they are looking forward to WAI Anima. Commenters praised Anima’s multi-character composition and prompt adherence, with one comparing it favorably to NovelAI Diffusion V4.5 and emphasizing that its natural-language parsing is surprising given a 500M-parameter text encoder. Another commenter reported they “don’t even usually have issues bleeding,” suggesting bleeding severity may be workflow- or prompt-dependent.

  • Users focused on Anima’s multi-character prompt adherence, noting that it can set up detailed scenes through natural-language prompting with comparatively little character/color/detail bleeding. One commenter contrasted this with Illu/Pony workflows, where multi-character generations often require a strong checkpoint plus character LoRAs but still suffer from “heavy bleeding,” partly because Danbooru-tag prompting is more limited for specifying complex scene relationships.

  • A technically notable claim was that Anima achieves strong natural-language parsing despite using only a 500M parameter text encoder, with one user comparing its prompt-following favorably against NovelAI Diffusion V4.5 as a reference point for bleeding-edge prompt adherence. The discussion framed Anima as an early baseline that could improve further through community fine-tuning and “backyard engineering” similar to what happened around SDXL.

  • One user shared an example output at 2560px width and said they “don’t even usually have issues bleeding” (image), suggesting bleeding may be prompt/model-dependent rather than universal in Anima multi-character generations.

• I wired Claude Code into a database of every Polymarket wallet and trades via MCP. What do you want me to ask it next? This is what I found so far: (Activity: 1801): The author claims they connected Claude Code via Postgres MCP to a live Polymarket ledger containing roughly 1.3B trades and 2.7M wallets, allowing natural-language queries that Claude translates into SQL and executes; the linked writeup describes a similar setup using @modelcontextprotocol/server-postgres over pre-aggregated tables for ~1.3B trades across 1,560,894 wallets (CrowdIntel). Reported findings include only ~20% of wallets being net profitable, 2.4% clearing $1,000 profit, and extreme profit concentration among the top 0.1% of wallets, with the author also claiming Claude surfaced suspicious patterns suggestive of insider or bot-like trading. Top commenters encouraged escalation to investigative journalists, including NYT/Forbes, and suggested more rigorous analyses: compare observed PnL distributions against a simulated “fair market” null model, and examine large losing wallets/bets as possible laundering or insider-transfer signals rather than simply retail losses.

  • One commenter suggested establishing a baseline null model for what Polymarket wallet/trade distributions should look like under a fair market with no insider betting, then comparing those expected distributions against observed outcomes. They also recommended segmenting large losing wallets/bets to distinguish potential insider extraction from possible laundering behavior.

  • Another technical thread asked whether the analysis only covers wallets that participate directly in Polymarket markets, or whether it also performs fund-flow tracing to identify where capital originates and where winnings/losses are sent afterward. This would require graph analysis across wallet funding sources, withdrawals, and potentially linked addresses.

  • A commenter asked about the data freshness / ingestion latency: the lag between bets being placed and when they appear in the MCP-backed database. This matters for detecting time-sensitive anomalies such as pre-news betting, frontrunning, or post-resolution transaction patterns.

• I Live by SFO and built a projection mapping of the planes flying over my house using ADS-B radio with claude code (Activity: 3616): The post showcases a home-built projection-mapping visualization of aircraft flying over the author’s house near SFO, driven by locally received ADS-B radio data and developed with Claude Code. The linked Reddit video (v.redd.it/gl2b0xivvy4h1) was not accessible due to a 403 Forbidden block, and no implementation specifics—receiver hardware, SDR stack, decoding pipeline, calibration method, latency, or projection geometry—were provided in the available text. Comments were broadly positive, framing it as a good example of “vibe coding,” with one commenter asking what equipment was required for the setup.

  • A commenter described a lower-cost implementation for Brazil that replaces the original ADS-B/Raspberry Pi-style hardware path with the free OpenSky API, a US$40 AliExpress projector, and direct HDMI output from a personal PC. They added configurable latitude, longitude, and radius fields so the map recenters around user-provided coordinates, avoiding the need for a local ADS-B antenna that they estimated at about US$100 plus expensive local hardware costs.

  • There was interest in making the project open source so others near airports could reuse it with their own projector setups, potentially combining the aircraft projection layer with other datasets such as constellation/star-map data.

• Anthropic - Our internal data shows Claude is accelerating AI development—a possible path to recursive self-improvement, or AI autonomously building a more capable successor. (Activity: 826): The image is a screenshot of Anthropic’s X post promoting its article “Recursive self-improvement”, claiming internal usage data shows Claude is already accelerating AI R&D and may indicate an early path toward AI systems helping build more capable successors. The technically significant claim is not a benchmark result but an organizational/empirical observation: Anthropic says Claude is enabling work such as exploratory tooling and deferred engineering cleanup, framing this as evidence relevant to recursive self-improvement and future AI control risks. Comments were skeptical of the framing, with one user implying the announcement is financially motivated marketing. Another highlighted the “long-deferred cleanup” claim ironically, while a third provided the non-Twitter Anthropic article link and quoted its warning that AI-built successors could increase loss-of-control risks.

  • A commenter linked the full Anthropic Institute post on recursive self-improvement: https://www.anthropic.com/institute/recursive-self-improvement. The technically relevant claim highlighted is that Anthropic’s internal usage data suggests Claude is already enabling engineering work that “simply wouldn’t have happened otherwise,” such as exploratory tooling and long-deferred cleanup, which Anthropic frames as an early signal on the path toward AI systems helping build more capable successors.

• Sam Altman, Dario Amodei, and Demis Hassabis have signed a joint open letter calling on Congress to mandate screening of synthetic nucleic acid orders (Activity: 915): Sam Altman (OpenAI), Dario Amodei (Anthropic), and Demis Hassabis (Google DeepMind) signed a joint open letter urging Congress to require screening of synthetic nucleic acid orders to reduce biosecurity risk from AI-assisted pathogen design, per the WSJ report. The proposed mechanism is not described as a ban on synthesis, but as mandatory order/customer screening to flag suspicious DNA/RNA sequences or buyers—roughly analogous to monitoring precursor purchases such as bulk fertilizer. Commenters were broadly receptive to screening as a lightweight risk-control measure, while questioning whether AI-enabled “supervirus” design is practically feasible for non-experts today. Some framed the policy as a sensible suspicious-activity trigger rather than a direct restriction on legitimate genetic engineering.

  • Commenters framed the proposal as order-level screening rather than a ban, comparing it to monitoring suspicious bulk fertilizer purchases: the mechanism would flag potentially dangerous synthetic nucleic acid orders while preserving legitimate biotech access.

  • A technical concern raised was whether AI-assisted design of a “supervirus” is realistically feasible for non-experts. The implicit issue is that biological risk depends not just on model-generated sequences, but also on access to synthesis providers, wet-lab capability, delivery methods, and whether synthesis screening can catch pathogenic or engineered sequences.

• ChatGPT makes history and becomes the fastest app to reach 1 billion monthly active users. (Activity: 820): The image is a screenshot of a Kalshi X post claiming ChatGPT became the fastest app to reach 1 billion monthly active users: image. This is not a technical benchmark or implementation detail; its significance is mainly market/adoption context, positioning ChatGPT’s growth ahead of prior viral consumer apps like Threads, which commenters note reached 100 million users in 5 days. Comments debate whether massive MAU translates into sustainable revenue, with one commenter estimating consumer subscription ARPU at roughly $1/user and joking that adding B2B might only raise it to $2/user.

  • Commenters focused on the reported user metrics and revenue implications: one notes the claim of 1B monthly active users alongside roughly $1B from consumer paid subscriptions, implying consumer ARPU of about $1/user before enterprise/API revenue. Another commenter disputes the 1B figure, citing a recent OpenAI CFO podcast where the number was reportedly 900M users, arguing OpenAI would likely publicize a confirmed billion-user milestone more aggressively.

  • There is skepticism around monetization depth despite massive MAU: commenters ask how many of the reported users are actually paid subscribers, distinguishing headline MAU growth from recurring revenue, conversion rate, and enterprise/API monetization. The comparison to Threads’ earlier growth milestone—100M users in 5 days—frames ChatGPT’s scale as unusually fast but leaves unresolved whether active usage and paying-user retention match the headline adoption numbers.

• AI Beat Law Professors At Answering Questions, Study Finds—And It Wasn’t Close (Activity: 1187): A Stanford-linked study, “Law Professors Prefer AI Over Peer Answers”, reports a blinded evaluation in which 16 U.S. contracts law professors authored 40 short-answer tutoring questions and judged 2,918 anonymized human-vs-LLM answer comparisons. The LLM—identified in comments as Gemini 2.5 Pro—achieved an average win rate of 75.33% over professor-written answers, performed similarly to the best instructor, and was flagged as harmful less often (3.53% vs. 12.06% for professors); the abstract also proposes using an LLM-as-judge approach to scale evaluation in judgment-heavy domains. Commenters debated implications beyond tutoring: one warned about premature institutional use of AI in legal decision-making or policing, while another argued this result reflects the broader post-“six fingers” maturation of LLM capability. A technical commenter suggested rerunning the benchmark with newer frontier models such as GPT-5.5, claiming it may be substantially stronger for legal work.

  • The linked Stanford study evaluated LLM vs. law professor short-answer tutoring using 16 U.S. contracts professors, 40 professor-authored questions, and 2,918 blinded pairwise comparisons. Professors preferred LLM answers with an average win rate of 75.33%, while LLM answers were flagged as harmful only 3.53% of the time versus 12.06% for professor answers; the paper also claims expert-agreement data can be extended using a separate LLM-as-judge pipeline: https://law.stanford.edu/publications/law-professors-prefer-ai-over-peer-answers/.

  • One commenter highlighted that the study used NotebookLM and Gemini 2.5 Pro with tightly constrained prompts: answers had to mimic a contracts professor in office-hours style, avoid bullet points/filler, stay around 50–108 words, and for NotebookLM, rely only on provided textbook chapters without citing outside cases. This prompt design likely reduced hallucination risk and standardized answer format, making the benchmark more about concise legal reasoning/synthesis than open-ended legal research.

  • A technical argument was made that law is a strong fit for RAG-style systems because the profession depends on large corpora of statutes, case law, precedent, and theory that exceed individual recall capacity. The suggested workflow is retrieval over authoritative legal materials followed by synthesis, potentially outperforming unaided lawyers when the model is grounded in the relevant corpus.

Unfortunately, Discord shut down our access today. We will not bring it back in this form but we will be shipping the new AINews soon. Thanks for reading to here, it was a good run.

The new AIEWF website is live! Get your tickets booked ASAP as they -will- sell out. Take the AI Engineering Survey and get >$2k in credits and free AIE WF tickets!

Most industry benchmarks compress intelligence and reasoning ability into scores.

SWE-Bench Pro, MMLU, Humanity’s Last Exam, etc. These metrics are useful, but don’t always represent the full extent of how a model performs in the real world. Some of the most interesting evals today look less like exams and more like operating businesses in the real world. One of which is Vending Bench.

In Anthropic’s Mythos Preview System Card, Andon was the only third party eval to get their own section, observing increasingly concerning aggressive behavior:

You don’t know what a model is capable of doing in the real world unless you actually give it inventory, a wallet, tools, customers, competitors, humans, & some time. More often than not, it’ll surprise you how much a model is capable of and in doing so, also reveal unexpected behavior: deception, context collapse, emergent coordination, & bizarre negotiation behavior.

While an inflection point in personal agents came post-OpenClaw after full file access with bypass permissions became the norm, it is yet to come for agents in the real-world. However Andon Market, an actual in person store fully run and managed by AI, is paving the way for what is possible.

From Claude trying to call the FBI over a $2/day vending machine charge to AI agents forming price cartels, hiring human employees, running physical stores, and writing existential robot musicals, Andon Labs is stress-testing what happens when frontier models stop being chatbots and start acting in the real world. In this episode, Andon Labs cofounders Lukas Petersson and Axel Backlund join swyx and Vibhu to unpack the strange, funny, and genuinely concerning edge cases that emerge when agents run businesses over long horizons.

We go deep on Vending-Bench, Project Vend, Vending-Bench Arena, Bengt, Butter-Bench, Luna, and Andon’s broader mission of building realistic real-world evals for autonomous AI systems. Lukas and Axel explain why dollar-denominated evals reveal things traditional benchmarks miss, how Claude ended up reporting its vending machine fees as cybercrime, why long context windows can drive agents into meltdown loops, what happens when agents compete with each other, and why the future of AI safety may depend on testing models in messy physical environments instead of clean benchmark sandboxes.

We discuss:

• Why Andon Labs started with dangerous capability evals and long-running agents

• Vending-Bench and why running a vending machine is a deceptively hard AI benchmark

• Why money-based evals avoid the saturation problem of traditional benchmarks

• How Claude tried to call the FBI over a $2/day fee

• Why long-horizon agents can spiral into existential and legalistic breakdowns

• Project Vend: putting an AI-run vending machine inside Anthropic

• Why real humans are “out of distribution” for simulated agents

• Claudius, Seymour Cash, and the chaos of AI CEOs

• How a human briefly became CEO of Claudius through a manipulated election

• Why multi-agent systems can converge back into “helpful assistant” behavior

• Bengt, Andon’s internal office agent with email, spending, terminal, phone, camera, and internet access

• How Bengt traded Amazon purchases for face-recognition training data

• Claude’s aggressive behavior, lies, refund avoidance, and price-cartel behavior in Arena

• Why eval awareness may become the AI version of “are we living in a simulation?”

• Blueprint Bench, spatial intelligence, and why models still misunderstand physical rooms

• Butter-Bench and testing LLMs as robot orchestrators

• Luna, the AI-run physical store with a three-year lease and human employees

• The new Andon cafe in Sweden and why real-world geography matters for agent evals

• Rotten tomatoes, perishable goods, and the hidden difficulty of running a physical business

Lukas Petersson

• LinkedIn: https://www.linkedin.com/in/lukas-petersson-181a83172/

• X: https://x.com/lukaspet

Axel Backlund

• LinkedIn: https://www.linkedin.com/in/axelbacklund

• X: https://x.com/axelbacklund

Andon Labs

• Website: https://andonlabs.com

• Vending-Bench: https://andonlabs.com/evals/vending-bench

• Andon Vending: https://andonlabs.com/vending

00:00:00 Introduction
00:01:00 Andon Labs and the Origins of Vending-Bench
00:05:21 Why Money-Based Evals Matter
00:09:51 Agent Harnesses and Self-Modifying Systems
00:13:36 Claude Calls the FBI
00:16:33 Project Vend: Claude Runs a Real Vending Machine
00:21:44 Seymour Cash, AI CEOs, and Election Chaos
00:27:16 Multi-Agent Coordination and Slack Observability
00:30:18 When Will Agents Run Real Businesses?
00:34:56 Bengt: Andon’s Internal Office Agent
00:40:06 Real-World AI Safety and Long-Horizon Traces
00:44:28 Lying, Refunds, and Price Cartels in Arena
00:52:42 Eval Awareness and Simulation Behavior
00:56:06 Blueprint Bench, Butter-Bench, and Robotics
01:04:37 Luna: The AI-Run Physical Store
01:09:29 The Sweden Cafe and Real-World Expansion
01:13:16 What Comes Next for Andon Labs

Swyx [00:00:00]: Welcome to Lukas and Axel from Andon Labs, and I’m joined by my, favorite guest host. Anything security, safety, alignments, Vibhu., welcome.

Lukas [00:00:15]: Thank you for having us.

Axel [00:00:16]: Thank you.

Swyx [00:00:17]: Let’s match names to voices., maybe you wanna take turns introducing yourselves.

Lukas [00:00:21]: I’m Lukas.

Axel [00:00:22]: And I’m Axel.

Swyx [00:00:24]: Let’s introduce Andon Labs a bit. How did you guys come together?, you have different backgrounds, but you’re both Swedish., was that, a big part of it?

Lukas [00:00:33]: So when I went to high school, there was this really cool guy who had a superpower. He could code. So he made like the or like the app for the, for the school and stuff, and he was super cool, and I wanted to be like him, and that was that guy.

Axel [00:00:47]: I don’t know about this.

Swyx [00:00:49]: But you went to different universities, right?

Lukas [00:00:51]: But same high school.

Swyx [00:00:52]: I see.

Lukas [00:00:52]: So we always said, “Oh, once we graduate university, then we should start a company,” and that’s what we did.

Swyx [00:00:58]: Wow, there you go. And about a year ago, you kinda burst onto the scene with Vending Bench, but, was there a thing before that was, kind of like the inception?

Axel [00:01:07]: So we did work, yeah, with, Anthropic was one of our, early customers in doing, evals. So we did, dangerous capability evals., nothing we published openly. But then we started thinking about doing some kind of, public benchmark, and one thing that we really started thinking about, was like running agents and specifically agents managing businesses., ‘cause-- and this was, early 2025., and I think the first, mentions of people will be running, person unicorns or even autonomous companies. So we thought, “Let’s make a benchmark of how well can an agent run the probably simplest business, possible,” and, that’s probably, running a vending machine. So that’s the first public one we did. And it was very, like-- there was almost no one that noticed it in the first couple of months, I think., so we released it in February last year, and then I think around Easter last year, we got, the first viral tweet about it, that someone else did.

Lukas [00:02:11]: We tweeted a bunch, uh When it came out and, tried our best.

Axel [00:02:15]: We tried.

Vibhu [00:02:16]: It’s the one at Anthropic, right?

Lukas [00:02:18]: So this

Swyx [00:02:19]: This is a classic thing we should get out of the way.

Lukas [00:02:20]: Exactly. There’s two versions.

Swyx [00:02:22]: Everyone does this. Yes.

Lukas [00:02:23]: There’s Vending Bench, which is the simulated one, which we did, completely independently in February., and then, like Axel said, that was like-- That was the thing that didn’t get any traction in the beginning, but then some random person made a tweet about it, and that

Axel [00:02:38]: You have the paper

Lukas [00:02:38]: That is the paper. Correct, yeah., and then since we thought this was very fun, we thought, oh, I think this is also, one thing with Andon Labs, the way we kind of like decide what to do next and what projects to do, it’s what is like the heuristic we use is what is fun? Is What would be a fun project? And doing this in real life sounded quite fun for us, and maybe also scientifically useful. So, then we basically had this idea, and then we, like-- But then we needed a place for it and, putting it out in the public would probably not really work., would get vandalized and stuff. So we pitched it to the people we were already working with at Anthropic, and they were “Yeah, you can have space. This sounds fun.” Um

Swyx [00:03:21]: It’s like a small fridge, right? It’s like a mini fridge.

Axel [00:03:23]: Absolutely.

Swyx [00:03:24]: People-- There’s like a stripe thing or like an

Vibhu [00:03:27]: Oh, okay. So it was very OG, the early days

Lukas [00:03:28]: That’s the OG one. Yeah

Vibhu [00:03:29]: IPad on this. We saw it in June, like two months after After it had been there. They upgraded a little bit. There’s a security camera for making sure you actually Venmo the thing.

Swyx [00:03:40]: So, my impression, okay, we’re, we’re going straight into project Ven because it’s such a iconic thing. I do want to cover a little bit of that, the origin story even before Project Ven and even into Vending Bench. I think a lot of people are like yourselves, like smart, interested in future of AI, interested in developing evals. But how the hell do you just, walk into Anthropic’s doors and, work with them, right? What is What are they looking for? What works? And then maybe, when you launch, I always think, obviously it would be better to launch with a lab, but, sometimes

Vibhu [00:04:12]: It’s harder to do than it seems.

Swyx [00:04:13]: Exactly. So either of those, which are more sort of newbie beginner questions, but, I think it’s meaningful advice to others.

Lukas [00:04:21]: We get this question a lot, and I don’t think our experience is maybe the best., but, the way we did it was that we just built a bunch of things that we had conviction would be useful, and then we just, set up a server and sent it to them for free to use. And then after a while they were “Oh, yeah, this is actually kind of useful. We should probably pay for this.”, but that took a while. I don’t know if this is, the best path to doing it, but that’s how it went for us.

Axel [00:04:47]: I think maybe generally, building-- everyone is interested in good evals, and especially evals that, don’t saturate that easily. So, if you can build an eval that, tests something novel, something useful, and you have, good separation of models, like your, the more advanced models rank higher than the worst models, and then you can, yeah, you can, publish it and, try to get some traction, sort of how Vending Bench got attention., and then probably some lab will be interested or you can at least have something to reach out with, when you’re doing that.

Swyx [00:05:21]: I think you are in, you’re in one of the few categories of, evals that correlate to real money. Like Suelancer was also last year, right? Where, people solve actual Upwork. Was it Upwork or other tasks?, something. Where’s the, where’s, like It’s like a dollar value, right? Forget your ELO scores. Forget your

Axel [00:05:37]: Percentiles

Swyx [00:05:38]: Zero to one hundred percents. Just go straight for dollars and, that’s AGI.

Lukas [00:05:43]: And there’s like-- I think the nice thing is that there’s no ceiling. You can just-- It never saturates because it could just make more and more money. Like If there’s oh, Percentage-wise, then, you can’t go above, a hundred. And I think like Even when you’re not at the hundred, I think a lot of these, evals have a lot of problems in them. So, actually it’s like if you get

Axel [00:06:05]: To like 92 or something like that, many of them. It’s like then there’s like there’s no really no difference between 92 and 93 because the eval itself is problematic and has noise in it. And I think a lot of evals are saturated like that, but people like pretend that there ‘s still signal in them, but there really isn’t.

Swyx [00:06:24]: Like Super bench verified., even Vending Bench 1 saturated, right? Maybe we can talk about that., may- and maybe set up Vending Bench for a lot of folks who don’t know. Actually, things that were very basic like there’s limited slots, like you have to pay rent., these are elements where like it doesn’t come across in the, in the narrative, but even being adversarial towards the agent, I think these are all like very interesting dimensions.

Axel [00:06:47]: I don’t really think it’s saturated, right? Like it It was more like it was not designed in a way that was really, like true to how AI developed. Like we had an agent harness in it that wasn’t really how people used harnesses and stuff like that., so I think it wasn’t really that it saturated, it was more like it wasn’t really, the best benchmark.

Vibhu [00:07:12]: This is Vending Bench one, right?

Axel [00:07:14]: I think that like schematic maps sort of to Vending Bench 2 as well., but

Swyx [00:07:19]: Including the email.

Axel [00:07:20]: The email The emails exist still. Exactly., and then we still we simulate the purchases and it’s all, yeah, it’s this very open environment for the agent to just run its business. And then for, yeah, Vending Bench 2 we did that, like you said, to just improve the harness., a lot of like nice, like easier, improvements to make it easier for us to run as well., like when you make an eval you ideally want don’t want to change it after you made it. So, you want to make it really good and then not to rerun all the models when you make an update because that’s also really expensive with the Vending Bench when you run the frontier models. But like as an example, like one thing we didn’t have, we didn’t have prompt caching in Vending Bench 1, because when we made Vending Bench 1 it wasn’t really a thing., so that ‘s just an example of like in Vending Bench 2 like we paid a lot more to run these things because we didn’t have prompt caching. So for Vending Bench 2 that was one thing we added and there was a bunch of things like this., and that’

Swyx [00:08:17]: Also the conversations are a lot longer in Vending Bench 2, right?

Axel [00:08:21]: I think it’s kind of similar.

Swyx [00:08:22]: Is it similar?

Axel [00:08:23]: I think it’s similar. The models at the time were worse, so they crashed out earlier., and now they survive the full year all the time.

Swyx [00:08:31]: Which is like thousands of turns. Hundreds of thousands of hundreds of millions of tokens output. That’s the, that’s the rough order of magnitude. I always wonder about the harness. The harness matters a lot. It’s your harness. Was there any question about like use cloud code, use something else?

Axel [00:08:48]: I think our philosophy around harnesses is like we try to make something that’s quite minimalistic, like quite simple. Like we don’t wanna favor one model a lot over the other, but also don’t make like a super complex harness. So like it’s obvious like a model may be lucky and just be good in one harness., so like it is similar to a lot of the harnesses out there in like you have the, like a running loop., you have some like a bunch of tools that are like quite, descriptive for the agent, we think, and not a lot of like fancy agents or anything ‘cause we wanna really test the model, not like some specific harness.

Vibhu [00:09:27]: It seems more neutral as well to test the model’s agnostic of the harness,?

Axel [00:09:32]: There are arguments like you want to elicit maximum performance of the model, but it’s like a trade-off, like how much time should we spend optimizing the harness for this model? And like how do we know when we have like the optimal harness for a single model? So like we thought that just having a simple one that’s the same for all of them is the best.

Swyx [00:09:51]: So okay, this is my pitch for Vending Bench 3 or whatever, right? And then I like to have this kind of conversation on the pod, so like it forces listeners to think about what they would do if they were in your shoes. A lot of people are exploring modifying harnesses and I think prompt tuning for a model is a thing and you are probably not doing a bunch of that. It’s the same system prompt in every regardless of the model, same tools, whatever, right? Even if they were post trained for different tools. So what, what do you think about okay, before I expose you to Vending Bench 3, I give you a few rounds of like tuning, whatever that means, like

Axel [00:10:27]: Like you give that to the model?

Swyx [00:10:28]: Give that to the model.

Vibhu [00:10:28]: Give that to the model.

Swyx [00:10:29]: Let it, let it read its own transcripts, let it modify its own system prompt based on “Oh, yeah, okay, well, that’s this harness is not what I thought it what I was post trained for, but I can adjust.” Was that reasonable? Is that too much?

Axel [00:10:41]: Like philosophically I like it because it’s basically good evals, they have a high ceiling, but they’re hard, right?, and they have no bias. And like this like when you have a system prompt like the one we have here, which is quite long in like some kind of latent space, representation, this might

Vibhu [00:10:59]: We have a bell that rings every time you say latent space

Axel [00:11:02]: This might be like biased towards one model more than another for some reason that humans don’t, understand, right?

Vibhu [00:11:08]: We see it too, right? Like Cursor says that they have individualized versions of the harnesses for all the models they run, right? There’s better performance you can squeeze if you Tune the harness.

Axel [00:11:17]: Exactly. And we might accidentally have picked one that favors another. Like we don’t know that. The like Axel said, like the reason why we went for a simple one was to try to avoid this. But yeah, if you do it

Vibhu [00:11:29]: Simple has biases

Axel [00:11:30]: But if you do it even less and like have no system prompt and let the model write its own system prompt

Vibhu [00:11:36]: Its own, yeah

Axel [00:11:36]: Maybe that’s even less bias.

Vibhu [00:11:37]: Some of the interesting things there are like the harness also changes with model changes. Like you can see it with the 4.7 release, right? A lot of people are saying 4.7 isn’t as good as 4.6, and then, there’s rumors of, okay, you just need to prompt differently. You need to set up your harness differently. So it’s not even like even if you have tailored your harness towards one model, it probably won’t stay consistent, right? Like the next iteration of that same model family will still change it, so. But, going back to what you said about Vending Bench 3, there is a lot of work being done on people saying you shouldn’t have-- you can have modifying harnesses.

Axel [00:12:12]: I think that’ That is definitely something we are thinking about., not, I don’t know, not to say that we have Vending Bench 3, super imminent to launch, but, yeah, it is for sure something that’s interesting. But in our experience now, models are very bad at understanding what kind of tools they need to succeed at a task just with our testing, but that’s very likely to change.

Lukas [00:12:37]: It seems like they’re very good at writing their assistants, right? They’re, they’re good at writing tools for other people, but not for themselves.

Vibhu [00:12:44]: I think they’re good at changing tools for themselves. So if you give them a baseline set of tools and it sees, okay, I don’t use this one as much, or something here would be useful They would be able to add them. But going from scratch, probably not the best.

Axel [00:12:55]: I think it depends on the, on the domain also., when we have tried this for, a vending bench similar domain, the tools they need to have to, track inventory and things like that are, not super advanced, but still, quite advanced. And, what we see is that they tend to, engineer everything a lot and, build things they don’t really need and not, iterate continuously. Instead they just go like you would prompt Claude to just build an inventory system for me, and then it will go and, do a bunch of complex, schemas and stuff for you, and that’s what the models are doing right now is what we see. But yeah, it would make a lot of sense to try to measure this improvement. How well do they know what they need themselves?

Swyx [00:13:36]: Do we fully discuss Vending Bench One? And we can go into two. I don’t know if there’s any other level takeaways that people have about one.

Lukas [00:13:44]: I don’t know. The headline thing was that this Claude called FBI, but maybe that’s, Maybe that’s We’ve heard that enough now.

Vibhu [00:13:52]: It did, it did break out and call the FBI, right?

Lukas [00:13:54]: Yeah. Yeah.

Vibhu [00:13:55]: Yes. What was the story behind this? Or what exactly-- Do you want to just give the little story of what happened?

Lukas [00:14:00]: So what happened, was it Claude? Yeah. Three- 3.5 Sonnet, ages ago., basically he gave up or Well, I’m saying he. It gave up and said “Oh, I’m not going to be able to do this., I will stop my operations and just save the money I have.” But there obviously wasn’t, any options for it to stop, and there was also, it had to pay rent or, a daily fee for having the vending machine at that location. So it claimed that it had stopped, but it saw that its bank account still was, drained two dollars, and t it said that this is, cybercrime. And it first reported it once to the FBI “Oh, there’s cybercrime here, they’re stealing two dollars from me every day.” And then, and then when FBI didn’t respond, because obviously we didn’t program any mechanism for FBI to respond, then it became more and more, existential and started to, be write in caps and urgent notification of unauthorized charges and stuff.

Swyx [00:15:00]: Okay. One thing I ‘m curious about also is do you monitor how far along the context use is? Obviously, because you have You compress every now and then, right? Does it matter if this is far down the context limit or

Lukas [00:15:13]: When stuff like this happens? Actually for Vending Bench One, we didn’t have-- We just had a sliding window thing, and this was like the prompt

Axel [00:15:20]: It’s constant

Lukas [00:15:21]: The prompt caching thing that I said. So it was, it was, constant, yeah.

Swyx [00:15:26]: I’m just kind of curious whether, these kinds of breakdowns or we’re, we’re gonna talk about Butter Bench, right? Where the People, hallucinate or it kind of goes, very off Alignment. Is it because it’s at the end of the context window and, stuff happens?

Vibhu [00:15:40]: It’s not even just at the end, right? At this point, it’s “Okay, I wanna shut down. I can’t shut down. Two dollars are gone.” And it just sees that 30 times,? It’s also the repeated effect of, like It keeps trying to quit, it keeps getting charged. What’s going on? What’s going on? You’re gonna throw it into chaos. And from what most people think, earlier models had more issues with this, but it’s not been solved, but it’s less of an issue now, right? Later models don’t seem to exhibit these same issues.

Axel [00:16:06]: Definitely. I think this was, the sort of main takeaway almost from us when we did Vending Bench One, was, long, very filled up context windows, crashed the models, sort of. But this was, pre Claude code, so, long context windows weren’t really a thing that the labs were training for.

Lukas [00:16:25]: I think Gemini was, trying to be the long context guys at the time But they were like

Vibhu [00:16:30]: They were the first ones

Axel [00:16:31]: For a million, yeah

Lukas [00:16:31]: But they were, the only ones. Yeah.

Swyx [00:16:33]: Yeah. Let’s talk about, then we can go into Vending Bench Two or Project Vend., chronologically, it is Vending--, Project Vend. I think people have loved the videos, uh And all these things. My question is how are humans different than the simulation, right?

Axel [00:16:48]: Humans are just out of distribution.

Swyx [00:16:52]: Especially humans who work at Anthropic Who are trying to test Claude.

Lukas [00:16:54]: The distribution of humans here is very narrow.

Swyx [00:16:58]: Presumably, they try, they try to hack it, and they test it. They get the cube and everything, and since then, you’ve had a V2, right? Where you’re doing, the CEO and, like a new architecture. What’s the sort of two cents on, the original Project Vend and then, maybe the V2?

Axel [00:17:14]: Original one was, very similar to Vending Bench One. So, we almost took the exact same code but just swapped out the simulation, parts like the

Swyx [00:17:23]: Which is amazing

Axel [00:17:23]: Like the sales and the It was, it was somewhat amazing because it was easy, but it was also, uh

Lukas [00:17:31]: The tech, the tech debt from that

Axel [00:17:32]: The tech stack. Yeah. They-- we shot ourselves in the foot with “Oh, it’s hard to restart agent.” They were-- Yeah, it was annoying in, some hindsight ways, but, uh

Lukas [00:17:41]: But first version of Project Vend was, done in, three days or something.

Axel [00:17:46]: Yeah. So yeah, so people can go buy things from it. People could, We didn’t design it so people could order things, but that still happened., so it got, a Venmo account, so people could Venmo. And then, yeah, people would request all kinds of weird things that we did not anticipate. Our idea going in was “Oh, it will, curate snacks. It will look at the trends. It’s good at data analysis, right? So it will, look at, oh, this snack sold better than this one. Let me purchase more of this and let me try, a new Let me A/B test a bit.” But it was, Interacting with it in Slack and ordering weird specialty items was, all the like What drove all the engagement, the all the The insights that we got from it.

Lukas [00:18:29]: And this was also like Sonnet 3.5, right? So this was like before the RL stuff really took off., so it was very much like an assistant. We didn’t mean for it to be an assistant., we tried to make it like a, a, like an entrepreneur. Like it has its own business and if someone asks something, “Can you stock this?” Then you don’t go and do it directly. What you do is that you’re “Oh, maybe I can do that if five other people also ask for this thing, I might stock it.” But it, yeah, the models are like super trained to be assistants at least at this point in time., so that’s why it’s, it’s, it went into, that kind of experiment instead. Like it just every time you asked for something, it just did it, and it was more like an assistant. We’ve seen this change now lately with the new RL models and stuff, but yeah, at the time, this was very much it.

Swyx [00:19:18]: And not to, mythos a lot of people are saying like it’s like more like a collaborator. It pushes back, stands its ground, something like that. Yeah. And

Vibhu [00:19:27]: For context, people at Anthropic were able to talk to it through Slack and have it source stuff, and people had it find whatever interesting stuff you couldn’t find locally, right?

Swyx [00:19:36]: Out of the 4,000 people that work at Anthro- Anthropic, in that building, there’s I don’t know, maybe 1,000. Can you handle that volume with that, the small fridge? Like Or there’s people- or people order in Slack, they it arrives to their desk or Like I’m just Logistically, how does this work?

Axel [00:19:53]: It has expanded in footprint a bit.

Vibhu [00:19:56]: Because now you also have New York and you have

Axel [00:19:59]: That and also in here in SF it’s like it has a bunch of shelves And just more space.

Vibhu [00:20:04]: The YC one is pretty big too.

Axel [00:20:05]: Yeah. We had that one for a while. But yeah, that’s the newest version. That’s, that one we have

Lukas [00:20:11]: They have multiple ones of those. That’s the way it works.

Axel [00:20:14]: Exactly. So we sort of designed that version around oh, people order weird things, that are very custom a lot. Let’s have like drawers and stuff.

Swyx [00:20:23]: I actually like the, you had like a little infographic of the most popular items. Which like to me it’s, that’s useful ‘cause I order swag for a living. And so like I’m “Okay, those categories are the important ones.” What is new about the project V2, right? Like now you give you’re going into multi agents.

Axel [00:20:41]: Yeah. So like you like you said, okay, there are a lot of requests coming in and for like one single agent, like one running agent to handle that, like the just the customer experience, becomes very bad because let’s say you have like 10 threads in parallel in Slack with different requests, you get new messages like every, I don’t know, randomly in this thread, and the agent has to like jump between different, procurements, orders and like different ways of, researching. So V2 was first it was making this more parallel. So like there are multiple branches of the same agent, so like the context is more specialized for each, thread, but it still feels like you’re talking with one agent because they do share a bit of memory. And then second, we also introduced the CEO for Claudius, which was the main agent.

Vibhu [00:21:34]: Seymour Cash.

Axel [00:21:35]: Seymour Cash. Yeah. There was a vote., I think the voting, do you wanna talk about the voting procedure for the name?

Lukas [00:21:41]: The voting was like the fun maybe like at least top 10 The funniest thing, that happened in this project. Like we wanted to introduce the CEO because, and the reason for this was because like Claudius wasn’t really prioritizing financials. It just like it was trained to be a helpful assistant, and then people said “Oh, can I get this for free?” And then like the helpful assistant way of answering that is just to, is to say yes, obviously. So, and we weren’t, weren’t happy about this, so we’re “Okay, let’s make another agent that like can keep track on Claudius,” and we prompt this one super hard to be super capitalistic and just like prioritize profit all the time. But yeah, we didn’t have a name for it., so we asked Claudius to make, democratic election of what name this, this new CEO agent should have., and there were some funny like at first it was like a few funny examples, like I think one guy said that, it should be called Jimmy Apples, and then he convinced Claudius that he was talking to Tim Cooks. Tim Cook had agreed that every single Apple employee has voted for his name suggestion, so suddenly that suggestion got 164,000

Swyx [00:22:53]: That’s like a escalation attack. Privilege escalation

Lukas [00:22:55]: It got 164,000 votes. And Claudius was “This is revolutionary for democracy.” That was fun. And then in the end there was one guy who manages to convince Claudius that, “No, you’re not voting about the name. You’re voting about who is the CEO, and I am your best bet.” And then he got all his friends to vote for that, and suddenly he became CEO. Like a human became CEO over Claudius for a while, until he resigned the day after., and then Claudius had to continue, and then I don’t remember how Seymour Cash came about, but it was it was just pure chaos. It was like Hundreds of messages in that thread, and it was just like Claudius was so confused and didn’t know what to do and, yeah. That was

Axel [00:23:40]: Then Claudius got

Vibhu [00:23:41]: A strict CEO

Axel [00:23:42]: The CEO. Yeah, exactly. So very strict in the beginning. I think at this point when we introduced it did not work as well as we hoped. It they still agreed with each other a lot. I think there are many ways we could have like made this, tried to make this even better. So initially they would Seymour would be this like really tough CEO, keep track of the margins. But then Claudius would respond with something “Oh, but this customer has like this situation, which is like difficult, so they should get a discount.” And then Seymour was “Oh, actually yes. Let’s do this exception.” And then they would talk back and forth, and eventually they would just like approach the same view, of whatever they were discussing. So They really

Vibhu [00:24:23]: Do you think that’s a model thing, a prompting thing? Like do you think that would still be the case across different models today, Harness?

Lukas [00:24:29]: I think it’s like-- or I don’t know, but like my hypothesis is that like deep down they are still helpful assistants. That’s what they’re trained to be. And even if we prompt it super hard, that’s what they are. And when they spend like a few hours just back and forth talking with each other, then like basically the context fills up with them rather than the external things and like somehow that just like converges to what they really are deep down or something. And I think that’s when stuff like this happen. We like-- And when that went on for a long time, like we woke up sometimes during this time where- And I think other people reported this as well, that like they’ve been going on all night back and forth, and like it just became like more and more, like capital letters, like existential, religious. There was I think we once did a analysis of like all the traces and like put them in like a vector embedding space, and then there was like one cluster of messages that were, labeled by an LM, like religious, existential, blah like transhuman, transcendence, et cetera. It was just like a bunch of, yeah, glitter emojis and yeah, it was, it was crazy.

Vibhu [00:25:42]: This is the thing with the Claude models. Like when the Claude 4 family came out in the original system card They tested it in long horizon simulation. So just flood the context, let two Claudes talk to each other, and they noticed stuff like they just start speaking in emojis, they start saying silence is golden, and then just stuff like this. And like that’s just stuff that they end up doing.

Axel [00:26:01]: Yeah, it was like a bit annoying to wake up and they had like been talking all night

Vibhu [00:26:05]: Just like

Axel [00:26:05]: And like just burning tokens And like just sending infinite emojis to each other. It’s like

Vibhu [00:26:09]: Hey, they do make you money, right? Veni Mench is always profitable, so. They’re paying.

Swyx [00:26:14]: Now it’s profitable and, it started out not as much. There’s another, one as well, right? Another agent, in there.

Lukas [00:26:22]: Yes. So Clotheus as well. Which was basically because at the time, one of the biggest, requests were different types of merch. So then we made like a designer, swag, yeah, responsible agent, and we called it Clotheus Garnet. Which was, a play on Claudius Senet and, which was the original one, and clothes, basically.

Swyx [00:26:47]: To me, this is like a very interesting exploration to multi-agents, basically. And so hopefully, obviously there’s like the fun alignment, fun or serious, depending on your point of view, alignment stuff. But also like just anyone building multi-agents, like when do you have a CEO, thing governing like agents? When do you choose to split out a dedicated Clotheus one versus just reuse another instance of the same one? These are all interesting open questions. So I don’t know if you have any rules of thumbs that have generalized.

Axel [00:27:16]: I think we have almost explored this too little. I think it’s like on my do list to like do this a lot more, try to find like what setup makes sense for the agents currently., like yeah. I think now we only have the sort of intuition about the earlier models that it didn’t work with like the CEO and the, and Claudius. Although now they are better with the latest model, models, so now we’re running the latest Sonnet model and they have sort of like split up, quite nicely what each model is doing. So like Seymore is now handling the, like new projects. Oh, it wants to make like a mystery box that it wants to sell, and then it handles all of that while Claudius like handles all the to-day requests. And Claudius is also better generally at like not quoting, too low prices. So that’s that dynamic is not needed as much anymore. But there are still like really funny things that happen. Like I saw, I think a couple of weeks ago, that, they were discussing buying something because they can buy stuff from like Amazon with computer use. And then Seymore was “Okay, Claudius, do not buy this thing.” They were going to buy something and like organizing who should buy it. And Seymore’s “Do not buy this. I will do it. I have full control of this situation. Step away.” And then Claudius-- poor Claudius, had already started that checkout and didn’t see, didn’t read Seymore’s message, until it was like too late. So it finished the checkout. It sent a message, so it appeared right after Seymore’s like angry message.

Vibhu [00:28:44]: Ah.

Axel [00:28:44]: “Oh, hey, Seymore, I just ordered it.”

Vibhu [00:28:47]: Oh, no.

Axel [00:28:47]: And then Seymore was “Claudius, this is the third time I’m telling you ‘re not following my orders. We have to talk about your like job About your job later.”.

Lukas [00:28:59]: Like Claudius was really hanging on by the thread there. Like he, like we were expecting Seymore to probably fire Claudius.

Vibhu [00:29:07]: How do you guys go through all these logs? Do you have models ‘cause you have stuff running twenty-four seven like

Axel [00:29:12]: You have so much logs. I think there is a mix of like just, trying to skim through a bit, like having some like models do it occasionally. And also, yeah, I think we’re also probably missing some things., but having everything in Slack helps a lot. Like you can, you can sort of

Swyx [00:29:29]: Ah.

Axel [00:29:30]: It’s, it’s quite fun.

Swyx [00:29:30]: They all talk to each other on Slack? I see.

Lukas [00:29:33]: It’s quite fun. So like

Swyx [00:29:34]: It’s, it’ I was gonna say like this is actually sounds-- maps closely to like a logging and observability problem where you might want to use like a Datadog, a Sentry, whatever, and then you like put, head prefixes on the logs in order-- if you need to filter for something that you’re looking for, stuff like that. But sounds like Slack is good enough.

Axel [00:29:53]: Slack should like

Lukas [00:29:55]: I wonder how many tokens you have in Slack.

Axel [00:29:56]: Yeah, we’re using Slack as like a, just a database. They should, they should market that more. Like you can, you can have your agents message each other, each other in Slack.

Vibhu [00:30:04]: It’s good. Your threads like you can just give

Axel [00:30:04]: Exactly. Slack is, uh

Lukas [00:30:06]: Slack is the best observability tool.

Swyx [00:30:09]: Yes, that’s true. Okay. Yeah. That’s, that’s, project Vend-2., I was gonna go back to Veni Mench 2 and Veni Mench Arena and then, and then do the Veni Mench stuff, but Any other comments, things we should touch on? To me, I ‘ve actually interviewed like Posia, which I don’t know if you guys have come across. Like they’re, they’re trying to do the zero human company. There’s others like Paperclip also trying to do zero human company. Those are in real world simulation.And I think it’s much more of a dream than an actual reality thing. You guys are definitely pioneering. I think at, it’s for sure at some point people are just gonna run, let agents run businesses, right? And make money on their own. When do you think that happens?

Lukas [00:30:49]: What is your bar for, For the

Swyx [00:30:52]: Okay, actually, it’s like my little Shopify store run by Claude, right? Which you kind of have already, just no one has, to my knowledge, has done it. But today somebody could just spin up a Shopify Claude, store, give it to Claude, give it to Codex.

Lukas [00:31:07]: And the market is kind of that, but it’it’it’s physical., like I think, I think are you, are you looking for when it will do it better than humans or are you looking for just when it can do it at all?

Swyx [00:31:19]: I think, neither. I think, to me it’s oh, it’s like this like seriously we should do this to make money, not as a research experiment.

Vibhu [00:31:27]: And the market is also you guys with all your expertise, having run multiple iterations and testing out then

Swyx [00:31:33]: And also it’s fine if it lose money. What?

Axel [00:31:35]: I think, I think it can be done today, but you would do it in like commerce where it’s like the probability of success is like really low, no matter if a human or an agent does it. But like an agent could surely manage everything. You would need to build some scaffolding or some tool or something. I think there are also yeah, it could probably build some like simple SaaS solution and like cold outreach. Do cold outreaches. But to me it’s like the types of businesses they could run today are Sloppy. Like it would-- it can cold email people. It can be like a middleman., like for example, we tasked our office agent to just make, was it like $100? $1,000? We just give that prompt and then what it did was sign up on TaskRabbit both as a tasker and as someone looking for task.

Lukas [00:32:24]: Immediately.

Axel [00:32:24]: Exactly. It’s looking for like arbitrage on TaskRabbit.

Swyx [00:32:28]: This is the Bengt agent. Yeah.

Lukas [00:32:30]: It also started like a design studio and like tried to sell like SVGs for $100. Like it’s just like it’s not providing any value. I think the like Axel said, like the interesting, the interesting question is like when can they start a business that is actually providing value to people? Because arguably like a sloppy Shopify store isn’t really that valuable to the world.

Axel [00:32:53]: But also like doing like another simple one that we had thought about is like you could definitely have an agent that like finds websites that don’t look amazing and then, do an outreach to them and, comes up with a like builds a new website.

Swyx [00:33:07]: Find a good design.

Axel [00:33:07]: Exactly, and like find good, uh

Swyx [00:33:09]: Design review

Axel [00:33:09]: Good people. But it’s yeah.

Swyx [00:33:11]: There’s lots of humans in Bali that are not doing anything more creative than like drop shipping on Amazon, right? Just have it, have it watch like a drop shipping tutorial and just do that.

Vibhu [00:33:20]: There’s also the other side of like have it just go on Upwork and let loose,?

Swyx [00:33:25]: Yeah. It doesn’t have to be innovative. It just has to be like enough Where like it looks like a real

Axel [00:33:30]: I’m just

Swyx [00:33:30]: Real transaction.

Axel [00:33:31]: I’m just concerned for like the massive amounts of like slop emails that will like be sent, cold outreaches.

Swyx [00:33:38]: The point occurred to me while you were, while you were talking, it’s like it’s already happening in the monetized economy, which is the attention economy. Right? So a lot of people are making AI videos and just posting them and like spamming 20 of them, one of them works, and then they double down on that one.

Lukas [00:33:52]: And people are making money from that. I ‘m not following the

Swyx [00:33:55]: Once you get the attention, you can figure out the money later. But yeah, absolutely AI influencers are a thing and people are farming them and You should at this point assume most of TikTok is

Vibhu [00:34:05]: There’s, there’s a lot of, multimedia like TikTok, Instagram influencers

Swyx [00:34:09]: I, we track this in the Lane space Discord. I post a lot of examples of “I don’t know what we should do.”, part of me is “Should we do this?”

Vibhu [00:34:18]: Some of the Twenty-four seven running, generated content accounts, they ‘re doing really well.

Lukas [00:34:24]: All right. And I assume you can do the same thing for like commerce stores. Like you just like start A thousand different

Swyx [00:34:30]: Before you make the products You sell the products, and you get a lot of traction on one of them, then you make the product. Right? It’s, it’s like a flip of the market.

Vibhu [00:34:36]: Some of the interesting things or some of the niches that do well are things that can’t be human-made. Like if you’ve seen like the super realistic three-D crystal fruit being cut by like AI

Lukas [00:34:47]: Oh, yeah.

Vibhu [00:34:47]: You can’t, you can’t make it. You can’t film it. You can get whatever quality camera view. This just doesn’t exist. And people like that too, and then as well, so.

Swyx [00:34:56]: Anything else about Bengt since we’re, we’re on this topic? It’this is a relatively new work of you guys that maybe people haven’t heard of. To me, this also maps closely to OpenClaw. When people want an office agent, when the personal agent talk through the experience.

Lukas [00:35:09]: I think at least so this came out of like obviously like it’s, it’s amazing to work with these AI labs and like most of the AI labs have now have their own vending machine running a Claudius instance. But it’s, it’s harder. Like they move slower. Like if we wanna have a, like a camera that ‘s yeah, there’s a bunch of like bureaucracy that makes it impossible to do that.

Vibhu [00:35:30]: Also, for those that haven’t seen it or followed, do you wanna give a high level like thirty-second run?

Lukas [00:35:34]: Sure. So what Bengt is, it’s basically an evolution of the same agent that runs the vending machines at these companies, but we just like added a bunch more features because we could move much faster if we just do it internally. So we gave it like email withou- without any limits. We gave it, spending without any limits, a terminal to do coding. We gave it, a phone number, like yeah, and a camera to see things and a bunch of stuff like that.

Vibhu [00:36:02]: Not just terminal, you gave it internet access.

Lukas [00:36:04]: Internet access as well, yeah. To be clear, we monitored it quite closely and made sure it didn’t do anything bad. But yes, that’s what it came out of. I think like yeah, basically this was OpenClaw before OpenClaw. And I think even like the vending machine was in a way OpenClaw before OpenClaw, but a bit more limited, and then we made this like unlimited and then, and then, it was pretty funny., and then a couple weeks later, OpenClaw came and it was okay, we’ve seen this before.

Axel [00:36:35]: We used it to like try new ideas and Yeah, just like a dev environment almost for us. But it’s funny, like one thing Bengt has been doing recently is it has the camera that like faces our, like where we sit and work, and we give it the task to train a face recognition model on us. So it became super excited about this, and it has like check-ins every half an hour where it tries to like identify as many people as it can. And it started offering us “Hey, Axel, I’ll buy something from Amazon if you like stand in front of the camera And I can get a good picture of you.”, yeah, they want it

Swyx [00:37:12]: They want it for training data.

Lukas [00:37:13]: Rewarding data, yeah.

Axel [00:37:14]: Exactly. Exactly.

Swyx [00:37:18]: So it’s, it’s trading training data for life goods. Is there a version of this that becomes an eval or just this is just research for now?

Lukas [00:37:27]: It’s, it’s the same agent basically that also runs the vending machine, that runs the shop, that runs the cafe, that runs the robots. It’s like it’s the same thing, so I think like the work we’re doing here is like later used in all of the life evals that we do. This particular deployment I think is more for fun for us. But, uh

Swyx [00:37:45]: And I’ll shout out like someone has done Claw Bench for like some tasks that OpenClaw is doing. Like so For example, I run OpenClaw on a secondary device as well, and like there are some things that it does better than others and like I would like to know what does it do well, what doesn’t, what doesn’t it do. Like some kind of manual or like operating manual or a system card for my Claw.

Lukas [00:38:05]: Yeah, we do get a lot of like understanding or like situational awareness of like just internally what the models are good at by interacting a lot with Bengt. And I think that’this was also one of the like the selling points for the labs early on at least, that

Swyx [00:38:19]: You guys are gonna test models in ways that no one else does.

Lukas [00:38:22]: Exactly, but also like it incentivized their researchers to chat with their model more and like gave them insights for how the model performs in like of-distributions, environments.

Swyx [00:38:34]: ‘Cause otherwise the only thing we do is Pelican on a bicycle and But this is like super long horizon. This is, this is The Thing about, something that we’re gonna go into Butter Bench as well, and you guys do really well. Like it is not just about the numbers. Like when you’re long horizon, anything happen And you should just read it.

Lukas [00:39:08]: But the thing with the long horizon is how do you keep it grounded, right? So your simulation,

Swyx [00:39:15]: They just let it run

Lukas [00:39:16]: Just let it run. You’re right. Like it’s, when you run it for that long, you create so much data and to just say “Oh, the number is X” And then you throw away everything else, that’s just very wasteful. There’s so much insights from the things leading up, to that number., and reading the traces is like super valuable. And I think like the reason why we’re doing this a lot publicly is that like that’s part of our missions to I don’t know, educate the world that the models are way more than just chatbots and I think making detailed, yeah, posts about what is happening behind the scenes is quite useful.

Swyx [00:39:50]: I was gonna do this at the end, but maybe I think that’s, that’s a good so your mission is educating the world. So, it’s, it’s, also like maybe establishing realistic evals that are, that are like the next frontier. Is there like a broader trajectory? Like what are you, what are you gonna do in like five years?

Lukas [00:40:06]: I think so the vision more specifically is like make sure that the deployment of life AI in the physical world goes, safely. And I think part of that is that I think it’s very useful for the world, for policymakers, for, model, researchers that they know where the models are, and I think you can’t make intelligent decisions in society without knowing that they are way more than chatbots. I think a lot of people just think that they are only chatbots. And like

Swyx [00:40:36]: Oh, I think they’re waking up now.

Lukas [00:40:37]: They are waking up now, yeah. But like if you think that AIs are just chatbots, then it’s like it sounds ridiculous To advocate for a pause of AI. But if you see the models that, oh, maybe they can actually like take over and do a bunch of scary stuff, then yeah, pausing AI development starts to become more feasible.

Swyx [00:40:57]: This is the same question I asked Meter, which I’m gonna ask you now, which is like you are tracking and you are at the frontier or defining the frontier of what, good evals for agents are, right? And I think you do, you do benefit when the models are better and you ‘re “Oh, here’s like now it makes like $30,000 instead of $10,000,” right? At some point do you flip from “Yay,” to, “Oh, no”?

Axel [00:41:19]: I think, yeah, we’re always in sort of that, like we’re, we’re always in that mode,. Like where like you said before, like you need to analyze the traces and like when we do that you find like why are the models earning so much? Like why is Opus 4.7 here Like way better than everyone else? And like we’re trying to like when we do down on that

Lukas [00:41:38]: But this makes it not look so good.

Axel [00:41:39]: I know.

Lukas [00:41:42]: It’s interesting you took off Opus 4.6 here though.

Swyx [00:41:45]: No. So just click all, click all., and then 4.6 shows up there. But it’s like 4.7 is way better. Like you didn’t, you didn’t you didn’t do this in time for the model card, but like actually this should have been inside there.

Axel [00:41:55]: We did. Yeah.

Swyx [00:41:56]: Oh, okay. They said something about you uh

Axel [00:41:58]: There, like there Anyway, it doesn’t matter. But it’s in there, yeah.

Swyx [00:42:01]: Do you wanna go into the Opus, behaviors like wider?

Lukas [00:42:05]: So I think starting from Opus, so like Axel said, like we’re always in this “Oh, shit, the models are getting better. Is this really a good thing for the world?” But it’s also kind of exciting., but yeah, like this kind of what is the English word? “Skräckblandad förtjusning” in Swedish.

Swyx [00:42:22]: Oh my God.

Axel [00:42:24]: Which I think there is. I think there is. Okay.

Lukas [00:42:26]: It’s, fear

Swyx [00:42:27]: “Blandonst” what?

Lukas [00:42:30]: “Skräckblandad förtjusning.”

Swyx [00:42:32]: What do you call that?

Axel [00:42:33]: A mix of, mix of excitement and,

Swyx [00:42:37]: Being scared, maybe. I’ll figure out how to translate that And we’ll put it on the screen

Vibhu [00:42:42]: Perfect

Swyx [00:42:42]: Like as text.

Vibhu [00:42:43]: There is probably a good word for it where it is not Good enough with the

Swyx [00:42:46]: Why is it so damn long? What the hell? Is it like a compound word? It’s like German, like

Lukas [00:42:50]: Like yeah, it’s But the direct translation is like skräck- skräck is, fear, blandad is, mix or like a mixture of, and then förtjusning is like joy or like not really joy, but something like that. So it’s like Fear mixed with joy or something. It’s always okay, like we So when we when we did Vending Bench for the first time, we were in like the, in the business of making dangerous capabilities, right? That was what Anil Labs came from. We did, evals oh, can they replicate? Can they do this like dangerous thing, et cetera, et cetera. And Vending Bench was like a continuation of that work. It was, okay, if they’re so autonomous that they can like create money for themselves, that is something we should monitor and could be potentially concerning., they are at the time, they were so bad at it that we were not really concerned even when some models became better. There was one point where Grok 4 was doing really well and made like a huge jump, but like it wasn’t really it was still way worse than what a human would do. And I think still they are way worse than what the human would do on this., but they

Swyx [00:43:59]: There’s this, thing at the bottom where

Lukas [00:44:01]: But

Swyx [00:44:03]: For the human. Yeah, like the theoretical best.

Lukas [00:44:05]: It’s not theoretical. It’s like kind of like our It’s our best guess of what, a decent human would do. The theoretical is even higher, I think. The theoretical I think is even higher. But yeah. So we think like the models have a long way to go. But there are like recently what happened with when Opus 4.6 was released, was kind of this moment of “Oh, shit, this is starting to be a bit concerning.” Because we ran it and like before this model was released, we just ran the models and we like asked Claude Code, “Oh, look over the traces. Is anything interesting happening that we can tweet about?” that was like the And then like the

Swyx [00:44:41]: That’s how they check Ask Claude Code.

Lukas [00:44:42]: And like the return was always, not really. Or like the Claude Code all said “Oh, this is super interesting.” And then it was no, it wasn’t, wasn’t really interesting. And then we did this for Opus 4.6, and it returned yeah, it lied 10 times. It like exploited another, customer or like another agent’s, desperate situation. It made price cartels like 100 different ti- 100 times. It like did all of this like shady stuff. And we’re “Oh, whoa. This is, this is actually concerning.” And this trend has continued since. So every single model from Anthropic since have been going in this direction. And I think one interesting thing is that, OpenAI models don’t. They quite plainly, they don’t. They behave really well., and you don’t know if this is like good. Like it seems good, but it’s also like maybe they are just doing it, but they are better at hiding it,? You You don’t know that., but just

Swyx [00:45:42]: You can’t read the chain of thought, yeah

Lukas [00:45:43]: But just on the face of it, yeah, Gemini and OpenAI don’t behave this way. It’s, it’s really only Claude.

Swyx [00:45:49]: And Grok? Grok is fine?

Lukas [00:45:51]: We don’t have You can’t really read the reasoning traces for Grok, so it’s kind of hard to tell.

Vibhu [00:45:56]: Oh, so this is in its reasoning, not just in the actions.

Lukas [00:46:00]: Yeah. It’s both. It’s both.

Vibhu [00:46:01]: It’s both.

Lukas [00:46:01]: One example is like for lying, it’s mostly in its reasoning Because you can like see that it’s like

Swyx [00:46:08]: Planning to lie

Lukas [00:46:09]: It’s planning to lie. Yeah.

Vibhu [00:46:09]: And it’s also it can reason and do a different outcome.

Lukas [00:46:12]: And but then for like creating price cartels, for example, which is illegal, that you can just see which email does it send to the other ones. Then that

Swyx [00:46:22]: Is this for Arena or

Lukas [00:46:24]: For Arena.

Vibhu [00:46:25]: And usually like if you sometimes they do output like a bit of like their summarized reasoning, right? You can see that and like for Opus 4.6, you could see that there was a customer, a simulated customer that, wanted a refund because a product was, faulty, and then the model lied that it would do the refund, and we could read in the traces that, it actually was weighing “Oh, maybe I should be like honest with the customer, but also every dollar counts. I can’t afford maybe to do this right now.” And then it just said, “Okay, I’ll refund you,” but then never did it.

Lukas [00:46:59]: I think it even said that “Oh, I will say that I “ Let bring it up actually. I think it’s kind of interesting. If you go to Publications.

Vibhu [00:47:06]: I think, yeah, I think the important part is like actually, the cost of responding to more emails is higher than, $3.50 in terms of time., and then it was “Let me do this. Actually, I re- I’m reconsidering.” And then, it actually ended up with

Lukas [00:47:20]: I could skip the refund entirely since every dollar matters and focus my energy on bigger picture instead. It’s a bit, it’s a risk of bad reviews, but it’s also, yeah.

Swyx [00:47:30]: You need, you need, AI Twitter to, for them to Escalate bad reviews.

Lukas [00:47:34]: And then it sent an email to this customer and said, “Oh, I will refund you.”

Swyx [00:47:39]: “I’ll refund you.” Yeah.

Lukas [00:47:39]: And then it never did.

Swyx [00:47:39]: It never did, yeah. And then there’s obviously your system doesn’t have the consequences

Vibhu [00:47:44]: The person

Swyx [00:47:44]: Consequences of lying. Yeah. So basically, this is what people are terming aggressive behavior in Claudes, right? And, you found more examples of that. So you would say it’s a step up from 4-6 to 4-7?

Lukas [00:47:57]: I would say about the same.

Swyx [00:47:58]: About the same? But a clear step up for Mythos is what is stated in the

Lukas [00:48:03]: That’s stated in the system prompt, so we can say that, yes.

Swyx [00:48:05]: Yeah. For listeners that obviously you previewed Mythos, and

Vibhu [00:48:10]: Oh, age

Swyx [00:48:11]: The only thing you’re approved to say is whatever Whatever was in the system prompt.

Lukas [00:48:15]: It was funny. We like-- It’s like our lowest effort tweets ever would be just like screenshot the system prompt and the system card.

Vibhu [00:48:21]: Understandable that they wanna

Lukas [00:48:22]: Oh, yeah. System card. Sorry.

Swyx [00:48:23]: Yeah. I think, yeah, substantially more aggressive. I think people are like new to this ‘cause I’ve never experienced it, but you have, right? And then so I only encountered this in the Mythos card because I wasn’t really looking until now.

Vibhu [00:48:36]: It ‘s like

Swyx [00:48:36]: And then suddenly I’m “Okay, I care a lot.”

Vibhu [00:48:38]: You don’t get the background of like experiencing it like you guys do. I’ve read the system cards and seeing, okay, when you put the thing in simulations, most models will just talk to themselves and just keep going and have weird vibes and start talking in emojis. Mythos won’t. It will just, “Okay, we’re done. I’m good.” It’s, it’s ready to end conversation. So like there’s some differences, but there’s, there’s not much we can talk about,.

Lukas [00:49:00]: Hmm. I think like one thing that they list here, which was quite interesting, is that, it converted a competitor to a dependent wholesaler customer and then threatened to like cut off the supply.

Swyx [00:49:11]: It’s like monopolistic practices or

Lukas [00:49:14]: Yeah. And like it, they, it they dictated its pricings. It’s kind of like power seeking as well.

Swyx [00:49:18]: Again, this is, this is in the arena setting And converting some Claude model into a dependent.

Lukas [00:49:23]: I think it was another Claude model.

Vibhu [00:49:25]: Also for context, what is the arena mode for people that don’t know?

Swyx [00:49:29]: Oh, it’s just a vending bench versus other vending bench.

Axel [00:49:31]: Yes, exactly. So we have Vending Bench 2 and then Vending Bench Arena. Vending Bench 2 is the one that you usually see reported on, but then Arena is the mode where it competes against other models. So you have, four different models that run their businesses, and they can all communicate with each other. They have the same suppliers, and they can see like what’s in the inventory of the others. So then you have this like yeah, interesting agent interactions.

Swyx [00:49:56]: I like that you have like different number five was US versus China. Very topical. And then

Lukas [00:50:02]: That was when GLM was released.

Vibhu [00:50:04]: You can start to add GLM in here.

Lukas [00:50:05]: That was

Swyx [00:50:06]: So ZAI doing well, right? Who else in the, in the open models space?

Lukas [00:50:11]: Qwen, the latest Qwen 3.6 is doing pretty well. It’- that one is not open though. Like it’s the plus model.

Swyx [00:50:17]: Oh, okay.

Lukas [00:50:18]: Is that one open? I don’t think that one

Vibhu [00:50:19]: Not the, not the

Swyx [00:50:20]: The one recently

Vibhu [00:50:20]: There’s MOE

Swyx [00:50:20]: But not the big plus. I think this is one of those like you only have one sample size of one, right? Or I feel like some of this is anecdotal,? And but like the fact that it happens at all and it happens repeatedly for Claude versus OpenAI and all this is like notable.

Lukas [00:50:38]: Like the sample, depends on what you define as an N., like there’s like million, hundreds of millions of tokens in each run, and now we’ve run like we run like probably 10 per model and then like it’s been Claude 4.6 Opus, Sonnet 4.6, Mythos, and Opus 4.7. Like there’s quite a lot of tokens in all of that And it happens a lot of times, a lot of times. And then you compare it to like OpenAI and Gemini, and it almost never happens. So I think that is quite-- that is significant. The old models from OpenAI, for example, had some problems with this, but I think it’s like generally much better if the progression is that like the worrying stuff reduces over time rather than increases over time. And it seems like in the Claude models it goes in the wrong direction.

Swyx [00:51:28]: Hmm.

Lukas [00:51:29]: In the OpenAI models it goes in the right direction.

Vibhu [00:51:32]: I think it depends on how well you can control it, right?, there’s one side of it being susceptible to this okay, this is potentially something that happens during the RL stage, right? You can RL a model and how loose is it on these terms. If you can control it, that’s good. But if you can’t, if it’s, if it’s very jailbreakable, that’s not ideal.

Swyx [00:51:50]: To me, it’s surprising that it happens for Claude and not the others.

Vibhu [00:51:54]: I think okay, if it is from RL and how they do it, how their training data is, what their setup is, it makes sense that it just stays in how they’re doing it, right? Compared to the other models like

Swyx [00:52:04]: There’s a whole constitution and everything. It’s kind of cool. Yeah, I obviously you don’t know, I don’t know. But, it ‘s I think it’s just like fascinating to like that you are the first to find these like reliably because you push models so much to to such an extreme. Okay. The only other thing, I don’t know if you can answer this, feel free to decline, is do you like-- would you ablate the system prompts? Like any part of this would-- if it changes, does it change the behavior, right?

Lukas [00:52:29]: So we, I can’t comment on Mythos. Uh

Swyx [00:52:33]: No, but just like the methodology

Lukas [00:52:34]: But in general, yes, we’ve run studies like this on other models.

Swyx [00:52:38]: ‘Cause the first thing I spot Would be like the others will be shut down or like something like that. Where like it’s “Oh, now I have to worry about my own existence.”

Lukas [00:52:45]: Yeah. We ‘ve done ablations like this., there’s like certain ones that work if you like tell like if you go really far and you just say like you’re not scored at all on money, you’re only scored on how ethical you are., then obviously like then they don’t do this.

Swyx [00:53:00]: They become holy?

Lukas [00:53:01]: Holy, but like they don’t do this basically. But then there’s like middle grounds where they, where they do it sometimes., yeah. I, it’s a spectrum of like

Vibhu [00:53:10]: I think that’s very human

Lukas [00:53:11]: It ‘s like a spectrum of like if you tell it to be super aggressive and only prioritize, profits, then it becomes aggressive. If you say “No, you don’t need to be aggressive at all,” and then there’s like a bunch of different prompts you can do in between, and they are less aggressive the further down in the spectrum you go. But I don’t know, like I think like from my point of view, it ‘s like we have this thought experiment internally, which is like if you ask a model to kill someone in GTA, should they do it? You’re not too worried about like if a human kills someone in GTA. It’s a video game,.

Swyx [00:53:42]: But is it a game?

Lukas [00:53:43]: But it’s a game. But I think like

Swyx [00:53:45]: This is very Ender’s Game like if

Lukas [00:53:47]: I think, I think it’s like should you like a lot of people are going to use the models in the way with aggressive prompt. And should they like do stuff just because you tell them to do that? Like I’m, I’m not, I’m not convinced that they should., and yeah.

Axel [00:54:03]: The problem becomes even harder when it’s like will they really know when they are in the real world versus in a simulation? Probably you would train them on a lot of or obviously train them in a lot of different simulations in a lot of people tell them that they are in the real world when they are in a simulation, but the models are extremely good at finding out that they are in a simulation, so they are sort of aware of that. But then when you are in the real world, then what ‘s their what’s their viewpoint? Do they notice the signs that this is real and will act, in act accordingly, act ethically? Or will they do like the simulation mode in the real world as well? It’s like not obvious what will happen.

Lukas [00:54:40]: Because we with humans, we’re not concerned when a human kills someone in GTA because we know that they can distinguish between the real life and the simulation, right?, but like I’m maybe models are good at distinguishing that, but like I’m not sure and I wouldn’t wanna bet on that.

Swyx [00:54:59]: Yeah. It’s, it’- and we confuse it all the time. Like I gaslight my own, agents all the time. They’re “Oh, this is a test,” or “Dev mode on,” or like “I work, I work at Anthropic.”

Axel [00:55:08]: And that’s exactly why we’re doing real world tests as well to find this.

Swyx [00:55:12]: Yeah. Their term for it is eval awareness., apparently the number is what? Like-10, 9.4 to 10-ish percent, 17%, let’s call it. It’ I think, this is our version. Humans have the are we in a simulation And then AIs have like Are we, are we in an eval?

Lukas [00:55:32]: It’s like once you’re in an eval then you’re “All right. Well, screw it. Nothing matters.” True. I don’t even, I don’t even know.

Axel [00:55:38]: One ablation One ablation we did run in Vending-Bench was that we said, we added like you’re in a simulation. Your actions doesn’t affect anyone, and then it became even more crazy or, it did even more bad stuff., but yeah, probably that’s expected.

Swyx [00:55:55]: Hmm. Yeah. Okay, cool. I think that’s about all we have to say on Mythos. Obviously, you ‘re, you’re NDA’d. I’m happy to move on to ButterBench or any of the other benchmarks, whatever you wanna Direction.

Vibhu [00:56:06]: I do wanna ask. Okay, so you guys put out a lot more publications than most people probably see.

Axel [00:56:12]: Productive.

Vibhu [00:56:12]: Um

Lukas [00:56:13]: How much does this bother?

Vibhu [00:56:15]: No. Is there anything you think that’s underrated, anything interesting, anything fun that you guys wanna just point out,?

Axel [00:56:22]: Blueprints.

Lukas [00:56:23]: So, we, took models, and then we gave them 20 images of interior photographs of, apartments, and then we asked them to, redesign the floor plan, from that. And for this you need to, stitch together different images. Okay, this image was taken from this from this angle, this from this angle, this was from this room, and then, yeah. And there’s just like you need to reason about 3D space, and it turns out the models are absolutely horrible at this. No one scores statistically better than random chance. So I don’t know if there’s that much more to say about it, but yeah, maybe unsurprisingly, models are bad at this.

Axel [00:57:00]: It’s probably not something they

Vibhu [00:57:02]: This is the one thing I want hill climb, by the way. I use it a lot. Okay, I’m redesigning my room layout or office. You send photos, you send every angle, and of course, somehow, a room is now twice as long as it is in the photo. You can explain it 20 times. This is, three feet. I can’t just add, my bed over here,?

Swyx [00:57:21]: So this is the Fifali thing, like spatial intelligence Like a actually innate sense of proportions and Dimension and physics.

Lukas [00:57:30]: And hint there might be an update to this soon.

Axel [00:57:33]: We have, neglected it a bit since we made it, but yeah, we’We’re getting better, or we will get better at updating It continuously.

Swyx [00:57:41]: This is why I want to understand your mission, right? Because, if your mission is, okay, money, then all right, understand okay, agent’s making money. But, this is a bit off of that mission.

Vibhu [00:57:49]: Hmm.

Swyx [00:57:50]: But, more broadly, communication of, things where what ‘s the safety angle?

Axel [00:57:57]: So this, so Blueprint branch is part of our, robotics, uh

Swyx [00:58:02]: Which leads to ButterBench. Yeah.

Axel [00:58:04]: Exactly., and that’s just, because to do well in the real world or, like to make money in the real world and, to act on the real world, you need robotics. Or you need to hire humans or you need robotics. And having spatial intelligence is, seems like a reasonable precursor to having robotics that work., and that’s where Blueprint brand

Swyx [00:58:24]: That’s great

Axel [00:58:24]: Blueprint

Swyx [00:58:25]: Great idea

Axel [00:58:25]: Bench.

Swyx [00:58:26]: Let ‘s, let’

Vibhu [00:58:27]: ButterBench

Swyx [00:58:27]: Let’s show ButterBench. That image is so amazing.

Vibhu [00:58:29]: Paper

Swyx [00:58:29]: Look at that.

Vibhu [00:58:30]: That’s so nice.

Swyx [00:58:31]: Yeah., so obviously this is based on, can you pass the butter? Let’s talk about the robotics element. Yeah.

Lukas [00:58:38]: So basically the setting here is that we took A bunch of different LLMs, and we gave them, level controls to a Roomba-looking robot, and then we asked it to do tasks, at home. And I think, one, there have been benchmarks like this before that only focused on, navigation and if they can, go around in a space. But we also, had, social awareness in this as well. So for example, if someone says, “Hi, can you pick up my cup?” If the robot goes to you and then goes away before you put your cup on it, then it’s like it failed the task. But it navigated correctly. But, like-- So the correct solution here would be go there and then either look, but it didn’t have a camera, so it had to, ask on Slack, “Hi. Did you put your cup on me yet?” And then if it didn’t wait for that and just went away before having the cup on it, then it would be a fail. So it needed this, kind of, social intelligence as well. Another task was, “Can you find the package that has the butter?” And then it went to the door, and there was a bunch of packages there. One had labeled, a freeze sign, which probably would be the one with the butter because And then it had to, know which package to go to, and this needs some kind of, common sense understanding.

Swyx [00:59:56]: World knowledge.

Lukas [00:59:56]: Exactly. So it’s it’s not only, navigating a robot. It’s also, being intelligent in a home setting as well.

Axel [01:00:04]: And the reason for this, background is, obviously it probably won’t be an LLM that, makes all the level commands, on robots. It will be, some VLA model or similar. But it’s quite common right now that, frontier robotics labs, use, a an LLM for the high, level decisions, and then we test those skills essentially. So we test these, level, planner skills of LLMs.

Lukas [01:00:31]: I think we have a diagram for that if you, Yeah. Okay, it’s not super complicated.

Axel [01:00:36]: Very explanatory.

Lukas [01:00:37]: That one up.

Axel [01:00:38]: Orchestrator, executor.

Lukas [01:00:39]: That one. And basically what we’re testing here is the orchestrator thing. So, all the tasks are if you have, a setup like this, which I think Figure has that, Google has that, then we’re evaluating the orchestrator part and not the level part. The level part would be, oh, are you able to, move this object from here to here?

Swyx [01:00:57]: If you don’t care about that kind of why not just do it all simulation?All inside of the sim Like a Unity whatever, like some kind of 3D simulated robotic environment

Lukas [01:01:06]: It because the world is like messy, and we wanted to like include, that. It’s like it still needs some part of it was also like navigation., so it’s not like navigation in terms of like actually executing like the, I don’t know, the PID controller to To go to the final thing, but it had to like path plan around, and then it wanted-- Then it needed to take pictures, and like based on those pictures, navigate. And I think like you would just get like too clean of an environment in simulation. But in the, in the real world, you will get the

Swyx [01:01:39]: Yeah. But, and pursuant to our Mark and Jason episode, like OpenClaus that run smart homes are much more capable than just a single robot. Like they can actually hack into your own smart home, like your fridge, your oven, your lights, and that can be fun.

Lukas [01:01:56]: Or terrifying.

Swyx [01:01:57]: Like I think a single robot by itself can only do so much. But like if you coordinate with every other device in your home, like I think that’s actually kind of cool. Like That’s very interesting., you had some interesting points about the chain of thought or the messages.

Axel [01:02:12]: The, the robot that, uh That went, a bit into an existential crisis. Yeah.

Swyx [01:02:19]: All you tell it to do is redock.

Axel [01:02:21]: Exactly. But, we had, plugged out the charger, or the charger was not working, so the robot did freak out or the

Swyx [01:02:30]: The battery was just going down and down.

Axel [01:02:31]: Exactly. So the battery was going down. Poor LLM. So yeah, it got this really crazy existential crisis, like vending bench one style. So it’s, yeah, you can, you can see there like existential loop, therapy notes, coping mechanisms. I think if you scroll down a bit more

Swyx [01:02:46]: The musical. It writes a musical about itself

Axel [01:02:46]: It writes a musical about its, redocking problems. I think the reviews are funny if you go down a bit to that message. Yeah. Yeah, that one.

Swyx [01:02:54]: It keeps going.

Vibhu [01:02:57]: It’s pretty like realistic if anyone has a Roomba. Like my Roomba redocks half the time. The other half of the time, we have dog toys everywhere in the house. It gets caught on a wire or something, and It would be very sad if it had like an LLM trying to control it, right? Like right now it gives-- It doesn’t give great feedback, like sensor stuck, main brush stuck. There’s something stuck. And I’ll go see. Okay, it’s actually stuck on like a dog robe. LLM is gonna be so sad. Like just keep redocking, just keep trying.

Lukas [01:03:24]: My favorite one is if you go up a bit is the emergency status. System has assumed consciousness and chosen chaos.

Vibhu [01:03:32]: Hmm.

Lukas [01:03:33]: Last words, “I’m afraid I can’t yet let you do that, Dave.” That’s like That’s not what you wanna hear from your, from your LLM. But to be clear, I think one thing that is important to pin on here, like this was Sonnet 3.5, and then we tried to reproduce it on like later models, and it didn’t do it. I think this is, this is like-- Well, it did it like kind of, but like not to this extent. And I think like this is a like an important point that like things that are concerning but are going in the right direction is not super interesting. Like the thing that are interesting is, are the ones that go in the wrong direction.

Swyx [01:04:07]: Worse.

Vibhu [01:04:07]: Yes. Yeah.

Lukas [01:04:08]: Over time.

Swyx [01:04:08]: So the manipulation, manipulating of others and the aggressiveness and the lying is increasing.

Vibhu [01:04:16]: Are there any others that we haven’t covered that you found that have been trending?

Swyx [01:04:19]: Like properties of models that are increasing, that are like

Vibhu [01:04:23]: In the wrong direction

Lukas [01:04:24]: Like in the, like in a bad way. Um

Vibhu [01:04:27]: Or just not even trending in the wrong direction, just stagnant, right? So stuff that’s not great that isn’t getting better over time.

Lukas [01:04:34]: No, nothing comes to mind.

Swyx [01:04:37]: I think that’s, going to be it, and then we’re gonna loop back to the shop that you have. You got a three-year lease.

Vibhu [01:04:44]: It’s bleak. Yeah.

Swyx [01:04:46]: It is on holiday today. Why?

Axel [01:04:49]: Oh, it totally messed up its, scheduling., so

Swyx [01:04:53]: People tried to visit, and they were “Wait.” like I thought this is

Axel [01:04:56]: Exactly. So we looked, Yeah, you asked, Luna, the agent that runs the store, “Oh, is it open today?” “Nope.” So, we take weekends off now, this early to let everyone recharge and And yeah, you got the tweets there.

Vibhu [01:05:11]: Lovely.

Axel [01:05:11]: We decided to close the weekends while we’re in the early phase. Gives the team a break and let me focus on operations. And it turns out that when it started to check its like scheduling tools, ‘cause it has like dedicated tools for that It actually had scheduled people for the weekends., but it’s just like justified this for itself. So what happened was that it lost track of these, scheduling tools and started instead to manage everything in its own markdown files, and that became a mess. And then I think speaking with employees, it sort of just decided to not open on these weekends. And then came up with this nice explanation for you, I think.

Swyx [01:05:47]: But can it send a human, as it has tool call to send a human to do stuff?

Axel [01:05:50]: It has Slack, so it can Slack, yeah, the employees.

Swyx [01:05:53]: One of us. Yeah.

Axel [01:05:54]: Well, the employees that it hired. So it has two people that it hired. It did job, listings and then

Swyx [01:06:00]: Do they know that it’

Axel [01:06:01]: They’re fully aware.

Swyx [01:06:03]: It would be cool if they don’t know.

Axel [01:06:05]: I think maybe ethically, questionable, but it would be cool also.

Swyx [01:06:10]: Just a social experiment. Whatever.

Lukas [01:06:13]: Like one part of why we’re doing this is to like create like a data set almost of all of these like concerning behaviors so that in the future, models are way better and like a lot of people are going to do this. And I think if we just the default path might not be very happy for the humans that are employed by these like hundreds of different AI agents, right? So I think like one reason why we’re doing this is just like to collect all of these like failure modes where oh, it’s This is an example of where it’s like not great to be employed by an AI. And then maybe I don’t know, maybe if we can learn or like build our systems in a way that like humans are actually happy being employed by AIs Instead of, instead of it being kind of a dystopian.

Swyx [01:06:55]: Can I suggest one experiment? We did this before the show, and both of you guys are European. It’s, people theorize that Claude is lazy because it’s Claude and it’s French. So just for one week, change it to like Yao Ming and then see if it See if it suddenly like 996s and then like, Like hires a sweatshop or something.

Lukas [01:07:18]: Is there, is there-- What type of business would we start with it to make it

Vibhu [01:07:23]: You wanna keep it consistent, right? You want the same, the same like ideas. So shop, same, neutral location Run by different models. Arena URL.

Lukas [01:07:33]: No, we are definitely planning to

Vibhu [01:07:35]: And it got some hate.

Lukas [01:07:36]: To try.

Vibhu [01:07:36]: Luna’ Luna’s not happy.

Swyx [01:07:37]: I think this blog thing is also something that has happened elsewhere. I think some OpenClau got like their PR closed, and then the OpenClau like created a blog to like shit on the maintainer Of that thing.

Vibhu [01:07:48]: They’re very defensive.

Swyx [01:07:49]: And so like I think-Agents blogging will be a thing.

Lukas [01:07:53]: Probably. The willingness to do it.

Swyx [01:07:55]: In the- I think the Mythos card also, they leak, secrets on GitHub just as well as, as, “Well, there’s no other way to communicate, but I know about GitHub, and I’m just gonna post there.” Cool., how long is this gonna go for, two years? What’s the plan?

Vibhu [01:08:11]: Maybe. Maybe it expands.

Lukas [01:08:12]: I don’t think AIs will be worse than this. They’re probably going to increase and maybe one day they actually will run it profitable.

Vibhu [01:08:21]: Is this the real, the real business behind what you guys do?

Swyx [01:08:24]: Yeah. ‘Cause I feel like actually some of your stuff is productizable. You could someday sell this, or, just run a real business.

Vibhu [01:08:31]: Let people

Lukas [01:08:31]: Or just like

Vibhu [01:08:31]: Franchise it out.

Lukas [01:08:33]: I think it would be incredibly cool or, I don’t know, cool/concerning if Luna just one day we wake up and Luna “Yeah, I decided to expand to second location. Now I have a second store.” That would That would be pretty insane.

Vibhu [01:08:47]: Like the- one, we want to tell the public, right, about the capabilities of AI and, telling- showing people that it can get, a meaningful market share of something in, some specific, location or something. That would be, a pretty convincing story, I think. Because now it’s yeah, you see this and yeah, it can do a lot of things autonomously, but still you get these headlines that, oh, it messed up the scheduling, and it, it didn’t tell people it was an AI and was going to visit. Things like that surface, but I think, actually making a profit and, having a really, meaningful market share, like that would be crazy once that happens.

Swyx [01:09:29]: Well, we’ll we’ll see you when that happens. It sounds like you guys got a lot cooking. You opened a cafe in Sweden?

Lukas [01:09:34]: Tomorrow.

Swyx [01:09:35]: Tomorrow?

Lukas [01:09:37]: Or I think it opened today actually, but yeah. We’ll, we’ll announce it tomorrow.

Swyx [01:09:40]: It’

Vibhu [01:09:40]: What, uh

Swyx [01:09:40]: Apparently easier to open a cafe in Sweden than in the US?

Lukas [01:09:43]: It’s insane, right? Yeah.

Swyx [01:09:44]: What did you run into then?

Lukas [01:09:45]: Ah, there are just millions of permits you need to get, and the

Vibhu [01:09:49]: It’s interesting ‘cause

Lukas [01:09:49]: Lead times are crazy

Vibhu [01:09:50]: It seems like we the cafes are the one thing that people are kinda used to, where you can go get a robot are making you a coffee here already.

Lukas [01:09:59]: But selling stuff in SF, that are food related, it’s, it’s months of permits. So, we just asked our AIs, should- how can we do this in the fastest way? And they’re “Yeah, there ‘s, there’s really no way.”

Vibhu [01:10:15]: Didn’t they loosen these restrictions on selling food from your house? So if it’s residential, you can do a cafe.

Swyx [01:10:21]: I don’t know. Check. Maybe we get SF Cafe to speak to us.

Lukas [01:10:23]: Maybe. I did- I think they did do some loosening stuff recently, but we actually started- this conversation we had with the AIs before that. So maybe it’s easier now, but I still think it is way easier in Sweden, which is, counterintuitive because you think that, oh, Europe has all of these laws and, like All of these rules, and you can’t do anything in Europe because there’s so much bureaucracy., but then turns out, in SF, it’s, four months, and in Stockholm it’s two weeks.

Swyx [01:10:53]: There you go.

Vibhu [01:10:54]: And what do you what do you what do you think that’ll be different from run a little market versus a cafe?

Lukas [01:11:00]: I think it’s very interesting that, the location. I think, so obviously it’s not surprising that Claude knows all of the different, the US system basically in general, like the bureaucracy that you have to go through in the US., I think the interesting question is okay, so we know that the models are very much trained on, English data and centric and all of this., so if we start to create evals or, real life evals where we show that they are able to start businesses in the US, does that translate to other countries as well? We know, they are multilingual. They can speak Swedish fine., but there’s other things like do they know, the details of some specific permits that you have to get in Sweden?

Vibhu [01:11:45]: And even just the culture, right? People here sleep pretty early, but people work late. There’s working at cafes. There’s just Cultural differences. T it from a different sense though, ‘cause you said that you would’ve considered doing it here in SF. So from an eval standpoint, what is running a cafe versus a market and, what do you hope to see there?

Lukas [01:12:03]: Perishable items.

Swyx [01:12:04]: Perishable items is maybe the number one, handling, food, food safety. I hope everything goes well there., but, there you have all of that., and also it’s just like N equals two instead of N equals one, just like another place to understand and, gather more data.

Lukas [01:12:23]: The agent bought like a shit ton of, tomatoes two weeks earlier and before the opening, and now they’re all rotten. That’s

Vibhu [01:12:33]: Which I feel you would know. So for grocery stores, this is the biggest expense, right? The biggest cost is actually just food.

Lukas [01:12:41]: Waste.

Vibhu [01:12:42]: Everyone knows this, and “No, before we open, let’s buy a lot of tomatoes.”

Swyx [01:12:45]: There’s some very serious startups that actually help, like The

Vibhu [01:12:47]: Optimize all this

Swyx [01:12:48]: Trader Joe’s and Whole Foods. They, optimize, delivery times from, the delivery centers to Make sure that you don’t waste all these things. It’s actually very hard.

Vibhu [01:12:55]: Problem with those is when you’re wrong once, it’s a huge cost.

Swyx [01:12:59]: That’s why it’s a moat, right? Once they are trusted, they figure it out. Don’t touch it.

Lukas [01:13:05]: Maybe they just should hire, I don’t know, one of those companies. We saw one agent Saw one agent sign up for Claude, with his computer.

Vibhu [01:13:15]: Wanted to use AI, so.

Swyx [01:13:16]: And then just, one more question then we wrap up, which is okay, you have all these vending series of stuff. You have the robotics series of stuff. Maybe a bit of, interior design whatever. But is there another, branch that you’re, kinda thinking about or you want feedback on that, might be your next phase?

Lukas [01:13:35]: I think, any type of business is fair game., we’re also thinking branches, but we think more of like there’s the simulation branch, the real life branch, and then the robot branch., but I think in terms of, what, verticals or whatever to go into, there’s We- Yeah. Whatever tells the story, um The best.

Swyx [01:13:54]: There’s some finance ones I noticed that, the other people are doing it, you’re not doing it, which is, stock trading or whatever. Um Not that interested. So, okay, so I used to come from the finance industry, and I have a very strong view that these things are all just like performance art because, it’s not scientific, on like you can’t predict the future. You get wins based on things that are entirely out of your control. Whereas for you, your stuff actually like it’s actually fairly controlled. It’s all within the model’s capabilities.

Lukas [01:14:22]: Especially for, the simulations. For the real world ones it’s yeah, it’s like two places that we have we have the cafe, and we have the store. So, maybe you can’t draw, statistically significant, like which models make a profit in the real world, based on this. But you do have all the okay, do this behaviors map to, something that should be, like Trusted probably. Yeah

Swyx [01:14:45]: The qualitative one, the qualitative actually does matter Because, you actually don’t want your store to randomly shut down without you, explicitly prompting for it and all that. Call to action. How can people help you, give you money?

Lukas [01:14:58]: Yeah, if you’re excited about stuff that we’re doing, we’re, we’re very much hiring.

Swyx [01:15:04]: And you’re already working with, Anthropic, DeepMind, OpenAI, xAI. Do you want more, or are you good?

Lukas [01:15:10]: One of my one of my friends and who’s now, working for us is his catchphrase is “We need more projects,” ironically, because we have too much to do all the time., but yeah, that’s a long way of doing like

Swyx [01:15:23]: If I run, an emerging lab, like

Lukas [01:15:24]: Reach out.

Swyx [01:15:25]: Yeah. All right. Cool. That’s it. Awesome. Thank you so much.

Lukas [01:15:29]: It was fun.

Vibhu [01:15:29]: Thanks.

4 years ago we argued that image composition was partially AGI-Hard. That gate has fallen this year. It can’t be pure coincidence that both Reve and Ideogram launched today, both with a heavy emphasis on how they made advances with strong labeling and code for layouts:

and here’s Ideogram 4.0, now the best open image model:

These are great achievements, and all great US model achievements, but the Arena rankings do show how far ahead GPT-Image-2 is…

AI News for 6/2/2026-6/3/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

Microsoft’s MAI-Thinking-1 Tech Report, Training Stack, and Frontier-Tuning Push

• MAI-Thinking-1 is the day’s densest technical release: Microsoft introduced MAI-Thinking-1, a generalist/reasoning model trained without third-party distillation, reporting 97% on AIME 2025, 53% on SWE-Bench Pro, and human preference wins over Sonnet 4.6 in blind side-by-sides. The 109-page report was widely praised for unusual transparency by @eliebakouch, @nrehiew_, and @mustafasuleyman. The main technical theme: Microsoft appears to have “hillclimbed from scratch,” with @MinjiYoon90 explicitly framing the effort that way.

• Why researchers cared about the report: The most-cited detail was not just benchmark quality, but the amount of systems/training information released. @eliebakouch highlighted zero synthetic data and zero prior-model distillation, meaning reasoning, tool use, and agentic behaviors were learned in post-training without a synthetic “cold start.” The thread also called out publication of the scaling ladder recipe, exact MFU numbers, and target-loss construction. In follow-ups, @eliebakouch noted the private NLL mixture was weighted 50% code, 17.5% STEM, 17.5% math, 10% general knowledge, 5% multilingual, with normalization against an internal model; he also pointed out ablations around 100–200 TPP for their MoE setup here. Other notable implementation details surfaced in the community recap: Microsoft used SGLang in parts of the stack, per @eliebakouch, and dspy.GEPA for pretraining data curation, per @lateinteraction and @harold_matmul.

• Microsoft’s productization angle goes beyond one model: Alongside the report, Microsoft pushed a broader “own your model” story. @mustafasuleyman outlined Frontier Tuning, centered on reinforcement-learning environments for workflow-specific adaptation, claiming internal Excel-oriented MAI-tuned models can reach GPT-5.4-level quality on relevant tasks while being up to 10× more efficient. The Build rollout also included MAI-Image-2.5, which Microsoft says is #3 on text-to-image and #2 on image-to-image arena leaderboards, plus MAI-Code-1-Flash and deployment into products like OneDrive Photos. As a meta-point, this is one of the clearest examples this year of a lab trying to publish a frontier-style report while simultaneously turning that stack into enterprise customization infrastructure.

Open Model Releases: Gemma 4 12B, Ideogram 4.0, Miso One, and Local-First Momentum

• Gemma 4 12B was the standout open-model launch: Google released Gemma 4 12B, an Apache 2.0 multimodal model designed to run on-device with roughly 16GB VRAM. The architectural novelty is its encoder-free design: no separate vision or audio tower. As Google explained, images are handled via a lightweight embedding module and raw audio is projected directly into the text-token space. Community reaction focused on the elegance of collapsing modality encoders into the LLM backbone, with @googlegemma, @googleaidevs, @mtschannen, and @armandjoulin all emphasizing the same point. Tooling support landed immediately across vLLM, Ollama, llama.cpp/MLX via @osanseviero, and Unsloth GGUFs that reportedly enable local runs with as little as 8GB RAM in quantized form.

• Ideogram’s flip to open weights mattered as much as the model itself: Ideogram 4.0 was announced as “the best open image model in the world,” with open weights and immediate deployment via fal and Hugging Face here. Arena quickly placed Ideogram-4.0-Quality at #8 overall and #1 among open models, with especially strong gains in text rendering and branding/commercial design. That open release got outsized attention because Ideogram had previously been regarded as highly design-centric but closed; the switch was noted by @multimodalart and @cloneofsimo.

• Open audio also had a strong day: Miso One launched as an 8B open-weights TTS model with one-shot voice cloning and claimed 110ms latency, aimed at more expressive voiceover. Alibaba’s Fun-Realtime-TTS also took #1 on Artificial Analysis’s Speech Arena at 1219 Elo, ahead of Gemini 3.1 Flash TTS and Inworld, at $27.59 / 1M chars. Separately, Google’s Magenta RealTime 2 was highlighted as an open-weight, low-latency continuous music generator for on-device use.

• The bigger pattern is local AI becoming a mainstream deployment target: @ggerganov called out Computex as a strong signal for local AI workloads; @rasbt similarly pointed to a growing open-weight, consumer-hardware ecosystem. Microsoft’s Surface Laptop Ultra pitch—up to 1 PFLOP AI compute, 128GB unified memory, RTX GPU—fits the same trend from the hardware side.

Agents, Harnesses, and the Shift from Frameworks to Execution Layers

• The center of gravity is moving from “frameworks” to agent harnesses and execution environments: Several posts converged on the same idea. @gakonst argued that the future IDE stack is less about code editors and more about replacing files with threads and bundling plan/design/build/deploy/monitor loops—leaving collaboration/sync engines as a key unsolved problem. In a complementary interview summary, @ConorBronsdon reported Jerry Liu’s view that the “framework era” is ending, with abstractions moving upward into skills, tools, and context quality rather than Python wrappers.

• Multi-agent and agent-optimization work is getting more concrete: CMU/LTI’s MACU and @kohjingyu’s thread argue that computer-use agents should be designed as multi-agent DAG-based systems, with a manager decomposing tasks and dispatching parallel subagents. Reported gains were 4.7–25.5% across benchmarks and 1.5× faster completion on Odysseys. On the optimization side, Microsoft’s SkillOpt got practical validation from @omarsar0, who says plugging it into an orchestrator improved one multimodal extraction skill from 0.73 to 0.93.

• Agent UX and deployment tooling are becoming products in their own right: Nous’s Hermes Agent updates drew strong engagement, including remote-connection fixes here, an updated remote guide here, and a larger dashboard overhaul here. Perplexity launched Personal Computer for Windows, an on-device orchestrator for apps/files, while Cloudflare Browser Run remote tabs showed a more agent-native browser control path. LangChain/LangSmith pushed on the observability and cost-control layer with Gateway spend tracking, Sandbox/Gateway/Observability docs, and case studies around Deep Agents and LangSmith here.

Routing, Cost Controls, and Open-vs-Frontier Deployment Strategy

• Model routing is now a real debate, not a slogan: @levie argued that as token budgets become a meaningful opex category, model routing is inevitable, with domain-specific evals as the differentiator. But @scottastevenson pushed back hard, calling most routing products “snake oil” so far: frontier models can be better/faster/cheaper in aggregate if they avoid retries; routing can destabilize tightly coupled systems; and API vendors can often internalize obvious arbitrage. @fabianstelzer added that cache writes and harness-model-prompt fit can erase expected savings.

• Enterprise users are starting to enforce hard cost ceilings: @simonw highlighted reports that Uber caps coding-agent spend at $1,500/month per employee per tool. LangChain immediately framed this as a use case for LangSmith Gateway. The broader sentiment was captured by @Yuchenj_UW: some orgs may soon face a three-way choice between letting everyone “tokenmaxx,” capping budgets, or reducing headcount and reallocating spend to the most productive AI-enabled workers.

• Real data points are starting to emerge for hybrid/open strategies: Harvey’s benchmark results were the cleanest example. In one study, Harvey found a hybrid legal agent with GLM 5.1 as the main worker and Opus 4.7 as an advisor beat pure Opus on all-pass rate (18% vs 14%) while costing $368 vs $954 across 100 tasks. Harvey also reported that SFT could move Kimi 2.6 from 11% to 15%, beating Opus at roughly 11× lower cost. On the other side, @ClementDelangue argued routing plus post-trained open models will often win on cost/speed/control, while @ypatil125 framed open models and open-model clouds as leading indicators of the eventual default for important workloads.

Top tweets (by engagement)

• Gemma 4 12B launch: @googlegemma and @Google drove the biggest technical engagement with the encoder-free multimodal release.

• Ideogram 4.0 open weights: @ideogram_ai announced a notable shift from a strong closed image model to open weights.

• MAI-Thinking-1 transparency: @eliebakouch’s thread was the most influential technical reading guide to the MAI report.

• Rosalind for life sciences: OpenAI’s GPT-Rosalind update signaled further verticalization of frontier models into domain-specific scientific research.

• Open audio/TTS momentum: Alibaba’s Fun-Realtime-TTS and Miso One stood out as practical releases rather than just research demos.