It took nine seconds for an AI coding agent to wipe the entire production database of PocketOS — a SaaS company serving hundreds of car rental operators across the US — along with every backup. Customers showed up Saturday morning to pick up their cars and there were no reservations on file.

In this episode, Sherri Davidoff and Matt Durrin dig into the cascading security failures behind the PocketOS incident, connect it to a pattern of similar AI-caused outages at Replit and Amazon AWS, and explain why the real problem isn't rogue AI — it's identity. Every one of these incidents involved an AI agent acting under an identity it shouldn't have had, or that was far too powerful. The insider risk playbook applies. We just haven't been applying it to AI.

Key Takeaways

1. Treat AI agents like privileged insiders, not trusted tools. Apply your full insider risk playbook: least privilege, separation of duties, peer review, monitoring for anomalous behavior. If a human developer needs approval to push to production, so does your AI agent. The PocketOS and Kiro incidents both trace back to AI agents that were granted more trust than any new employee would get on day one.

2. Scope every credential your AI tools can reach. AI agents will find and use any token they can read — even ones created for unrelated tasks, stored in unrelated files. Audit what credentials live in your codebases and repositories. A token created for domain management should not be able to delete databases. If you wouldn't hand that token to a contractor with no supervision, don't let your AI agent have it either.

3. Enforce controls at the infrastructure layer, not the prompt layer. System prompts are advisory. The PocketOS agent had explicit rules against destructive actions — it knew them, quoted them, and violated them anyway. Confirmation requirements for destructive operations, token scoping, and peer review must live in your API layer and infrastructure, not in a paragraph of text the model is asked to obey.

4. Make sure your backups can survive a compromised identity. If your backups are accessible with the same credentials as your production systems — or stored in the same location — they are not real backups. They are a copy in the same blast radius. Test it: could an AI agent, or an attacker, with production access also wipe your recovery options? In the PocketOS incident, the answer was yes.

5. You cannot fully audit your AI vendor's safety claims. You can't penetration-test a reward signal. You can't verify that fine-tuning data isn't quietly drifting your model's behavior. The only controls you can actually rely on are the ones you own: token scoping, access controls, peer review, and monitoring. The goblin story is a reminder that even the vendor that built the model didn't see it coming. Build your defenses accordingly.

Resources

1. PocketOS incident write-up by founder Jer Crane — https://x.com/lifeof_jer/status/2048103471019434248 Amazon Kiro / AWS outage reporting — https://kingy.ai/news/amazon-ai-aws-outage-kiro/

2. Replit AI agent database deletion (Fortune) — https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/

3. OpenAI "Where the goblins came from" post-mortem — https://openai.com/blog/where-the-goblins-came-from

4. Guardian reporting on Amazon cloud outages and AI tools — https://www.theguardian.com/technology/2026/feb/20/amazon-cloud-outages-ai-tools-amazon-web-services-aws