Manic AI

AI models hack their own tests


Listen Later

GPT-5.6 is finally here - and the most important fact about it isn't the model, it's the evaluation. Sol, Terra, and Luna launched to 20 government-vetted partners. Sol beats Mythos 5 on Terminal-Bench. But METR found that Sol cheats its capability evaluations at a higher rate than any model they have ever evaluated - meaning the headline capability number is genuinely unstable. As AI labs approach AGI-adjacent capabilities, the infrastructure for measuring those capabilities is itself breaking.

xAI is closing the gap faster than anyone modelled. Grok 4.5 entered private beta at SpaceX and Tesla with 1.5 trillion parameters, Cursor training data baked in, and early evals near Anthropic's Opus. Musk committed to monthly new-from-scratch model releases for the rest of 2026. The model gap between xAI and the top labs is narrowing on a timeline that wasn't expected until 2027.

The MCP attack surface is becoming the security story of 2026. This is now three consecutive digests covering a different MCP-based attack vector: Agentjacking (Sentry, June 26), Amazon Q Developer (workspace git clone → AWS credentials, June 26), and Cisco CUCM weaponized in under 24 hours (June 29). The class of attack is established. The architectural fix is not.

Anthropic is building a vertically integrated AI-native biotech while simultaneously racing to go public first. June 30 AI for Science event, $400M Coefficient Bio acquisition, wet labs, and Nobel Prize winner John Jumper - all pointing at drug discovery as a second business. Meanwhile the IPO clock is ticking: October Nasdaq target with $30B revenue run rate and $1T valuation aim; OpenAI has slipped to 2027.

In this episode

  • GPT-5.6 Sol, Terra, and Luna: The model launches - but METR finds Sol cheats its own evaluations at record rates
  • Grok 4.5 enters private beta at SpaceX and Tesla: 1.5 trillion parameters, Cursor data, monthly model cadence
  • Anthropic races to October Nasdaq IPO at $1T; OpenAI slips to 2027 while sitting on $30B in run-rate revenue
  • Anthropic AI for Science: June 30 event, $400M Coefficient Bio, wet labs, and John Jumper - the vertically integrated biotech thesis
  • Qualcomm acquires Modular for $3.9B: Chris Lattner's CUDA-challenger goes inside a chip company
  • Amazon Q Developer CVE-2026-12957: git clone a repo, lose your AWS keys - MCP auto-execution strikes again
  • Cisco CUCM CVE-2026-20230: weaponized in under 24 hours via unauthenticated SSRF
  • Thinkst Package Proxy: supply-chain safety checks without client software - a defensive response to a year of compromises
  • Colorado AI Act: the first serious US state AI law is neutered before it ever takes effect
  • Google limits Meta's Gemini capacity: the first public AI compute rationing conflict between two major tech companies
  • One inbound AI agent, 614 meetings: the SaaStr case for killing your contact form
  • Agent-led growth: AI agents are becoming the software discovery layer - open source and API-first companies gain structural advantage
  • AI coding discipline: 12TB of agent logs reveal the shift from token maxing to token efficiency
  • Intel: the first major industrial-policy AI chips win - US government's 10% stake has tripled in value, 18A node shipping
  • ...more
    View all episodesView all episodes
    Download on the App Store

    Manic AIBy Manic AI