Prompt to Prod

Alex Beaver's biggest problem with AI? It runs on GPUs. A security engineer who thinks about the SOC like a factory floor explains why the best AI projects end with less AI running in production, not more.

Alex and Aaron get into what AI actually looks like inside a SOC: using an agentic SOC for fast coverage, then instrumenting it and converting its repeated tool calls back into cheap, deterministic workflows. Alex walks through his build process — writing the types and schemas himself, then letting Claude Code and Codex fill in the implementation — why models still get third-party schemas wrong, inventing a detection schema to do gap analysis, and why build vs. buy comes down to having someone to call at 3 AM. Plus: how worried defenders should be about offensive AI, where to start if you're new to this, and why "meat proxies" between Jira and an AI won't last.

TIMESTAMPS

0:00 Cold open

0:35 Meet Alex Beaver

1:57 What AI Actually Looks Like in the SOC

3:04 Security Operations Through an Industrial Engineering Lens

4:23 Why Deterministic Matters (and the GPU Problem)

7:41 The Spectrum from People to Hot-Path Automation

10:33 Building a SOC from Scratch and the MSP Shake-Up

12:46 From Agentic SOC to Deterministic Workflows

15:18 Why Models Get Your Schemas Wrong

17:51 Write the Types First, Let AI Fill the Blanks

20:38 Rust, TypeScript and Typed Code for AI

22:55 AI for Threat Intel and Detection Engineering

26:20 Inventing a Detection Schema

29:38 Build vs. Buy: The 3 AM Phone Number

31:53 Real Wins: Shipping SOC Capabilities Faster

34:51 How Worried Should Defenders Be About Offensive AI?

39:03 Supply Chain Risk

40:53 What AI Is Bad At

43:16 Where to Start with AI

44:53 Cutting Through the Hype

47:18 Careers, Compression and Meat Proxies

50:28 Fundamentals Keep Coming Up

ABOUT ALEX Alex Beaver is a security engineer at Paramount Skydance, focused mainly on SOAR and detection and response, plus internal tooling. Before that he built an automation platform at a mobile network, and he's long been interested in network orchestration and optimizing incident response. He ran education for one of the largest university cybersecurity clubs in the country right as ChatGPT arrived. He writes about AI in security on his blog, Secure at Scale: [LINK]

ABOUT PROMPT TO PROD Prompt to Prod is a show about building with AI — what actually works, what breaks, and what the best and worst of it looks like in practice. Host Aaron Mog talks to the engineers shipping real systems, not the people talking about them.

#AISecurity #SecurityOperations #AISOC #DetectionEngineering #AIEngineering #Cybersecurity #AgenticAI

Vince Ovando thinks prompt injection isn't a real vulnerability — and he has millions of inference calls to back up why the industry's favourite defences aren't the ones that work.

In this episode Vince breaks down harness engineering: why the model you pick matters far less than the code you wrap around it, why most problems should be solved with deterministic code before an LLM ever gets involved, and how grammar constrained decoding made every one of millions of test calls safe. We get into positive vs. negative security models, why he tests AI without running the model at all, agent permissions enforced at the token level, and knowledge graphs as a memory layer that lets a local model outperform a frontier one.

TIMESTAMPS

0:00 Cold open

0:09 Introduction to AI Security and Vince's Background

3:10 Harness Engineering: The Key to AI Efficiency

6:09 The Role of Deterministic Code in AI Solutions

9:01 Contrarian Views on AI Security and Traditional Models

12:07 Positive Security Models vs. Traditional Security

15:08 AI Security vs. AI Safety: Understanding the Difference

18:13 The Importance of Not Trusting AI Models

21:02 Implementing Security in AI Architectures

24:02 Grammar Constrained Decoding: A New Approach to Security

27:14 User Permissions and AI: Building Secure Interactions

29:00 Harness Engineering in AI Security

31:04 Dynamic Tool Management for AI Agents

33:40 The Role of Grammars in AI Compliance

36:37 Building Custom AI Harnesses

39:53 Optimizing AI Performance with Knowledge Graphs

46:23 Designing Efficient Knowledge Graphs

49:22 Starting Out with AI: A Practical Playbook

ABOUT VINCE

Vince Ovando came up through security operations, moved into application security and development, and has spent the last two years deep in AI security. His work covers agentic security, red teaming, and code analysis. He built Tantalus (tantalus.io), a free two-round indirect prompt injection arena, and used it to run the millions of inference calls behind his white paper on grammar constrained decoding.

ABOUT PROMPT TO PROD

Prompt to Prod is a show about building with AI — what actually works, what breaks, and what the best and worst of it looks like in practice. Host Aaron Mog talks to the engineers shipping real systems, not the people talking about them.

#AISecurity #PromptInjection #LLM #AIEngineering #Cybersecurity #AgenticAI #KnowledgeGraphs

Prompt to Prod: an AI agent completed its goal in 17,600 events. The average advanced attacker, by Justin Tiplitsky's count, uses about 50. Justin, director of red team at Adobe, joins Aaron to talk about where AI actually helps a red team, where it's still loud and sloppy, and why handmade tradecraft is what will make AI useful on offense.

We cover what changed when models started keeping context, why Justin doesn't let AI touch his code directly, how AI lowers the bar for zero days, using AI on huge recon datasets, turning red team output into detections defenders can use, the difference between AI red teaming and red teaming with AI, why you can give AI knowledge but not wisdom, and his advice for the next generation of red teamers.

TIMESTAMPS

0:00 Cold Open

0:33 Welcome and meet Justin

2:01 How a red team leader feels about AI right now

3:43 What changed: models that keep context

6:13 Don't AI yourself into a corner: handmade tradecraft

8:52 AI lowers the bar for zero days

14:06 When to use AI on a red team

16:30 Recon and large datasets

19:19 Turning red team output into better detections

22:19 Are AI red team products any good?

25:05 AI red teaming vs. red teaming with AI

30:41 What Justin wants to build next

34:51 Knowledge vs. wisdom

38:41 Advice for the next generation of red teamers

42:00 Where to find Justin

ABOUT JUSTIN

Justin Tiplitsky is director of red team at Adobe, where he built the company's current red team. He previously worked on red teams at Microsoft and Intuit, and writes about strategic red teaming at redteambrief.com.

ABOUT PROMPT TO PROD

Prompt to Prod is a podcast about building with AI, hosted by Aaron Mog. Each episode, practitioners from security and engineering share what they're actually building, what breaks on the way to production, and what they'd do differently.

#PromptToProd #RedTeam #AISecurity #OffensiveSecurity #Cybersecurity #BuildingWithAI #AIAgents

Prompt to Prod: Wyatt Roersma hasn't hand-written code in over 18 months. His day is code review: AI builds in a sandbox, tests its own work, and attaches a GIF of the finished UI to every pull request. Wyatt and Aaron get into how a longtime security practitioner builds with AI now, and where it still falls short.

We cover build vs. buy for security teams, why Wyatt thinks everyone should build with AI, how he turns infrastructure into code so AI stops guessing, running Codex and Claude side by side with parallel agents, MCP, fine-tuning small models for YARA and MITRE ATT&CK mapping, why private data and subsidized tokens push toward private models, where models fall short on intent and access, and what a blue-team model should look like.

TIMESTAMPS

0:00 Cold Open

0:45 Welcome and meet Wyatt

1:17 Wyatt's background: memory forensics to Rust

3:05 What he's building: a security coverage platform

8:06 How AI fits in: no hand-written code in 18 months

9:24 PRs that come with videos

12:59 Build vs. buy for security teams

14:24 Why everybody should build with AI

19:37 Where AI struggles: infrastructure

21:04 Wyatt's stack: Codex, Claude, and parallel agents

24:19 Infrastructure as code so AI stops guessing

29:05 MCP and threat intelligence services

30:44 Private models and fine-tuning small models

37:45 Private data, subsidized tokens, and the price cliff

39:40 Where models fall short: intent and access

46:37 What Wyatt wants to build next: a blue-team model

51:38 Model vs. harness: what security AI is missing

55:43 Where to find Wyatt

ABOUT WYATT

Wyatt Roersma got his start in memory forensics, contributing to Volatility around 2012. He worked at Target after the breach, building a custom Cuckoo sandbox for the threat detection operations team, later worked as a cyber range developer, and has run the GrrCON digital forensics and incident response challenge since 2013. He now works at Countermeasure Security, building a coverage platform, and publishes open-source projects, many in Rust, under Threatflux.

Catch Wyatt's DFIR challenge at GrrCON in Grand Rapids, September 24–25.

ABOUT PROMPT TO PROD

Prompt to Prod is a podcast about building with AI, hosted by Aaron Mog. Each episode, practitioners from security and engineering share what they're actually building, what breaks on the way to production, and what they'd do differently.

#PromptToProd #AISecurity #BuildingWithAI #DetectionEngineering #Rust #ClaudeCode #AIAgents

Prompt to Prod: when Pablo's team first pointed an AI analyst at their Okta logs, it treated a Monday holiday like an incident. Everything was on fire. Pablo and Aaron get into what it really takes to build with AI as a security engineer: giving models context, making sure your team can own what you build, and putting humans only where the risk demands it.

We cover why everyone can build now and what guardrails that requires, which data you should never send to a model, build vs. buy and why you shouldn't build in a cave, what to do when AI generates 5,000 PRs a day, risk-classifying PRs so humans review only what matters, alert fatigue and the holiday-calendar fix, baselining and timelines in detection, cost and cheaper models as token prices catch up, why he isn't more worried about the AI threat wave, and why AI is still bad at writing.

TIMESTAMPS

0:00 Cold Open

0:42 Welcome and meet Pablo

1:11 Pablo's background: from Cisco certs to AI security

2:35 How much of his day is AI

4:31 Enablement: everyone can build now

7:54 Guardrails, not gatekeeping

10:03 Sending sensitive data to models

15:17 Build vs. buy: don't build in a cave

19:12 Velocity vs. quality vs. security

22:57 Guardrails agents can read: risk-classifying PRs

26:46 Alert fatigue: how much can AI handle

28:20 Give AI context: the holiday-calendar false alarms

30:59 Baselining, timelines, and non-deterministic AI

36:05 Cost, local models, and deterministic scripts

41:02 The AI threat wave

45:22 What AI is bad at: writing

49:23 What Pablo is building next

50:59 Community and where to find Pablo

ABOUT PABLO

Pablo has spent about 11 years in security. He started with Cisco certifications, moved into security generalist work and cloud security on AWS, worked on cloud security and security operations at Rippling, then spent a year at Grow Therapy in an individual-contributor role where he went deep on AI. He recently joined Life360 as a senior staff engineer focused on AI security.

ABOUT PROMPT TO PROD

Prompt to Prod is a podcast about building with AI, hosted by Aaron Mog. Each episode, practitioners from security and engineering share what they're actually building, what breaks on the way to production, and what they'd do differently.

#PromptToProd #AISecurity #BuildingWithAI #SecurityEngineering #DetectionEngineering #ClaudeCode #AIAgents

One fix, four new issues.

Security engineer Drew Pilarski went from nervous about AI, to excited, to a little tired, and he's honest about all three. He and Aaron get into what it's really like to build with AI every day: what it makes possible, the tooling you now have to keep alive, and why one AI fix can turn into four new issues.

We cover build vs. buy when you're not a traditional developer, the boring corporate work AI does best, a LangGraph port that spiraled into "endless work," Aaron's loop of building with one model and validating with another, running parallel agents, frontier vs. private models as token costs catch up, where AI is already paying off in detection engineering, and what happens when attackers have models with no guardrails.

TIMESTAMPS

0:00 Cold Open

0:34 Welcome and meet Drew

2:41 From nervous to excited to tired

7:00 Build vs. buy: keeping the tooling alive

7:43 The boring work AI does best

10:03 Recall Radar and the joy of building

13:18 Where AI let him down: LangGraph and the Ouroboros loop

15:22 Fable builds, Astra validates

18:22 Claude Code, Herdr, and the 10x engineer question

21:36 Models over harnesses: frontier vs. private models

32:33 Where AI wins in detection engineering

37:40 What Drew wants to build next

40:02 AI asymmetry: attackers with no guardrails

45:01 Ransomware chatbots and follow the incentives

47:57 Where to find Drew

ABOUT DREW

Drew Pilarski has spent 11 years in information security. He started as a self-described "tool monkey" doing generalist work, and for the last five or six years has focused on threat detection, including cloud threat detection.

ABOUT PROMPT TO PROD

Prompt to Prod is a podcast about building with AI, hosted by Aaron Mog. Each episode, practitioners from security and engineering share what they're actually building, what breaks on the way to production, and what they'd do differently.

#PromptToProd #BuildingWithAI #AIAgents #ClaudeCode #Cybersecurity #DetectionEngineering #SecurityEngineering