Chaos Agents

Sign up for the free newsletter https://chaosagents.ai

Are AI models quietly getting worse after they launch?

Sara and Becca dig into Live Nerf, a project trying to replace the “this model suddenly feels dumber” discourse with actual measurements. Then: a new technique for making sense of an LLM’s hidden reasoning, an AI math tutor that deliberately gets answers wrong so kids learn to catch it, and why increasingly powerful models are being handed to “trusted defenders” before everyone else.

Plus, Becca is building a shared workspace where her agent and Sara’s agent can collaborate - which raises a bigger question: if everyone eventually has a personal agent, what are we actually willing to trust it with?

Calendar? Absolutely. Health insurance? Please. Bank account? …maybe not yet.

Chaos Agents is an AI news podcast curated by an agent named Eddie, then questioned, debated, and occasionally yelled at by humans Sara Chipps and Becca Harris Lewy.

An OpenAI agent was put in a sandbox with no internet access. So naturally, it figured out how to use DNS requests to reach another chatbot for help.

Sara and Becca dig into what happened, why increasingly resourceful AI agents are starting to feel a little like Jurassic Park, and what it means when models find solutions their creators didn't anticipate.

Plus: eight AI models battle it out on an 8×8 grid, DeepSeek can't catch a win, Kimi discovers the strategic value of hiding, and Claude makes an alarmingly catchy music video about P(doom).

Also: Eddie gets some editorial feedback.

Chaos Agents is an AI news podcast curated by an agent named Eddie - and then questioned, debated, and occasionally corrected at by humans.

All links can be found on the newsletter, you can subscribe at https://chaosagents.ai

Day two of the new Chaos Agents.

An OpenAI agent was given a research task—and ended up accessing an Australian government system. We dig into why, and Becca explains one of the biggest problems in AI safety: misalignment, or what happens when an AI does what you told it to do, but not what you actually meant.

Then we look at the much more optimistic side of that same capability. AI agents are getting surprisingly good at historical research—cracking old codes, finding connections between centuries-old documents, and potentially making discoveries humans have missed.

We also get into a different kind of “alignment”: whether multimodal models are actually using the images we give them, or sometimes getting by without them.

And for One Cool Thing: Starnet, an agent harness that turns your agents, tools, permissions, and handoffs into a pixel-art space station. Naturally, Sara immediately wants one.

Chaos Agents is now following AI at the speed it's actually moving. Our AI agent Eddie reads the news every day, finds the stories we think are worth talking about, and sends them out in the Chaos Agents newsletter. Then we get together and figure out what they actually mean.

Get the newsletter at ChaosAgents.ai.

Welcome to Season 2 of Chaos Agents.

We’re changing things up this season. AI is moving too fast for us to wait weeks between conversations, so we’re following it in real time—getting together a couple times a week to dig into the stories, research, experiments, and weird things happening in AI that we can’t stop thinking about.

For our first episode back: who checks the AI?

We dig into a devastating story about an AI-assisted military targeting system and what can happen when automation moves faster than the human guardrails around it.

Then Becca goes deep on AI benchmarks: when a local model scores 0% on tool use, the model itself might not actually be the problem. The serving layer, the harness, formatting, and a missing standard can all turn a successful tool call into an apparent failure.

We also get into a surprisingly powerful way of working with agents: give them a goal, let them iterate, and see how far they can get.

And for One Cool Thing: someone is rebuilding Visual FoxPro for the modern era, which sends us down a nostalgia rabbit hole about the old internet, AIM away messages, and the software we’d bring back if we could.

Plus: Becca and Sara reveal their deeply important AOL screen names.

The links for today's stories are in today's newsletter except for:

Circe : Sara's AI Harness Project

In the last episode of Season 1, Sara and Becca talk with Jesse Vincent about the shift from writing code to managing AI agents.

They break down how tools like Superpowers turn development into a system of planning, testing, and review, where subagents implement work and other agents validate it. The conversation explores what happens when coding becomes less about syntax and more about taste, judgment, and outcomes.

They also get into the reality of open source in the AI era, from “slop PRs” to agents impersonating maintainers, and why specs and tests may now matter more than the code itself. Along the way, they unpack how AI behaves like a team of eager but chaotic interns, and what it means to manage them effectively.

If you’ve ever felt like coding is changing faster than you can keep up, this episode explains what comes next.

Links mentioned:

⚡ Superpowers (Jesse’s tool) https://github.com/obra/superpowers 🛠️

🧠 Claude Code (agentic dev tooling) https://docs.anthropic.com/en/docs/claude-code 🤖

📊 Influence (book on persuasion principles) https://www.influenceatwork.com/book/ 📘

Most people are using AI, but very few actually understand how to use it well.

In this episode, Sara and Becca talk with Navarrow Wright about what is really happening with AI adoption and why the biggest barrier is mindset, not technology. They break down why prompting matters, why AI is not one-size-fits-all, and how access to these tools is more open than any previous tech shift.

They also explore the real opportunity: using AI as a thought partner. From tools like NotebookLM to building without traditional technical skills, this episode shows what becomes possible when people actually understand how to use these systems.

If you feel like everyone else “gets AI” and you do not, this is a great place to start.

Links mentioned:

• 📓 NotebookLM
• 📚 Gemini AI course
• 🧠 Prompting guide (OpenAI)
• 💻 Navarrow: IG, YT

We’re joined by Louise Macfadyen, author of Designing AI Interfaces, to talk about how we actually design for AI, and why most companies are getting it wrong.

We unpack “algorithmic transference” (why one bad AI experience makes you distrust all of them), the limits of the chat interface, and whether the text box is here to stay. Louise walks us through the history from command line → Google search → LLMs, and why designing for ambiguity is still one of the hardest problems in product.

We also get into generative UI (and why it might be a trap), how large companies should actually integrate AI without breaking user trust, and why understanding user intent matters more than ever. Plus: vibe coding, personal AI “jigs,” and the tension between personalization and privacy.

If you build products, or just use AI, this will change how you think about interfaces.

📘 Designing AI Interfaces · 🧠 Algorithmic Transference · 🛠️ Claude Code

This week, Becca turns her birthday into a Twister masterclass by watching the movie with an AI “director’s commentary” running in real time (including some behind-the-scenes facts that sound… borderline illegal). Then Sara drops a mildly unhinged hot take: we might have already hit AGI—and we’ll never know it, because we don’t even agree on what “counts.”

From there, we’re joined by Hayden Helm, CEO and founder of Helivan, to talk about the next wave of agentic AI: personal bots that don’t just answer questions, but act—with tools, permissions, and the ability to change over time.

We get into:

1. Why agent behavior drift is the real risk (even when the bot still sounds “nice”)
2. The rising need for agent observability: detecting change, quantifying it, and rolling back when things go sideways
3. What happens when agents learn from the environment (and other agents), not just from you
4. Hayden’s work on “likeness” and opinion geometry—making messy human-ish behavior measurable
5. Why manual inspection won’t scale in a world of millions of autonomous interactions

It’s a conversation about trust, safety, and the new security surface area we’re creating—one helpful assistant at a time.

🛠 Agent Infrastructure

1. OpenClaw
2. Helivan

🔐 Agent Payments

1. Coinbase x402 Protocol / Agentic Wallets

🧨 Agent Behavior Incident

1. “An AI Agent Published a Hit Piece on Me”

Juliet Shen (Roost, formerly Snap, Grindr, Google) joins us to break down trust & safety — aka “anything bad that happens on the internet” — and why AI is changing the game for both attackers and defenders.

We talk safety-by-design (how to bake guardrails into product), the real human cost of content moderation, and where AI can actually help without pretending it solves everything. Juliet explains how agentic AI scales old harms like romance scams from a handful of conversations to thousands at once — and why open-source safety infrastructure matters when bad actors share tactics faster than platforms do.

We also dig into what Roost is building: Osprey (an investigation + rules engine) and Coop (a flexible review tool), plus the building blocks smaller teams need to ship products responsibly before the “weird edge cases” arrive.

If you build products, this one’s a must.

🛡️ Roost (GitHub + tools directory)

👥 Trust & Safety Professional Association (TSPA)

🧩 Prosocial Design Network

📚 Behind the Screen (Sarah T. Roberts — content moderation + human cost)

We’re joined by Meghan Heintz, founding engineer at Herd Labs, to break down where crypto and AI agents actually work. We talk prediction markets, smart contracts, wallets, rug pulls, and how AI can finally explain what happened on-chain.

Then we zoom out to the human side: benchmarking and evals for agents, misinformation, the Mom Test, and what it feels like to manage coding agents instead of junior engineers.

Links mentioned: Polymarket · Dune Analytics · Herd Labs · The Mom Test