TestTalks.AI - The Testing AI Podcast

The view from the board seat: why the biggest AI risk isn't a bad model — it's moving too slowly while your competitors learn faster.

Alex Belotsky and co-host Kuba sit down with David Subar, founder of the LA consultancy Interna and a veteran board advisor, to unpack how AI is reshaping the decisions made in the boardroom. David argues that boards don't care which tool writes the code — they care about value creation and risk, the same math whether you ship in assembly or with Claude Code. He frames the product roadmap as "a series of bets" and explains why unconstraining development velocity forces companies to iterate faster while their risk surface balloons. The conversation digs into why stochastic systems are so hard to test (the boundaries "might be in the middle"), using Google's Gemini founding-fathers image fiasco as a billion-dollar cautionary tale, and why AI red teaming and reliability metrics now matter as much as pass/fail test cases. David and the hosts also tackle the "SaaS apocalypse," the flood of vibe-coded internal tools that smashes the QA problem across every department, Conway's Law and org design, Jevons paradox and developer jobs, and whether the real constraint on AI is TSMC and energy rather than talent. It's a candid, wide-ranging take on shipping reliable AI when the ground shifts every six months.

**In this episode:**

**Guest:** David Subar — Founder, Interna (board advisor, ex-CTO/CPO)

> "you saw a worm in the ground you stepped on it now there's worm guts everywhere and someone's got to clean that up"

🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.*

## Hashtags

When your chatbot answers legal or medical questions, "close enough" isn't a spec — it's a lawsuit.

Erick Herring of Vynyl joins Kuba and Alex to unpack what it actually takes to build and assure LLM-based systems for healthcare, lending, insurance and legal clients. Drawing on a career that spans punch cards, the birth of the commercial web and today's AI frenzy, Erick argues that the fundamentals of accountability haven't changed — but the way we hold code writers (now AI) accountable has, opening the door to formal verification, 100% test coverage and rigorous pre/post-condition testing that were never practical with human-only teams. He makes the case that "red teaming" is the wrong frame — this is AI assurance, spanning hallucination, bias and reliability, not just jailbreaks — and that QA can no longer be a discrete step but a continuous process fed by unified observability. Along the way: why guardrail code deserves more scrutiny than production code, the "ontological reframing" paper on talking an AI past its safety guardrails, the "yes, yes, yes" developer psychology that erodes PR review, and why QA engineers are more necessary now, not less. A grounded, contrarian take on shipping non-deterministic software you can actually stand behind.

**In this episode:**

**Guest:** Erick Herring — Founder, Vynyl (product development firm serving regulated industries)

> "Our brains are not thinking machines. Our brains are justification machines."

🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.*

## Hashtags

When your system gives a different answer every time, "assuring" quality is a trap — you have to engineer it in.

Janna Loeffler joins Alex and Kuba to unpack why the shift from quality assurance to quality engineering (and even "quality enablement") matters more than ever in the age of LLMs and agents. She argues that AI didn't kill the tester's job — it made human insight, risk judgement and context-driven thinking essential, because non-deterministic output turns testing from clean pass/fail into questions of accuracy, relevance and intent. Janna makes the case for an "army" of single-purpose agents that handle the mundane — regression, accessibility, OWASP Top 10 security checks — so testers can focus on outcomes over outputs. The conversation digs into the real danger of eroding guardrails, including a chilling example of a system talked past its safety limits within minutes, and why model providers like Anthropic and OpenAI are pushing safety responsibility down onto the customers who deploy them. Along the way she shares how she vibe-coded a unit-test-writing agent in Cursor in a single day, why diverse test teams are the antidote to biased models, and why the emerging "AI architect" role will decide whether we get one agent or an orchestrated fleet of hundreds.

**In this episode:**

**Guest:** Janna Loeffler — Quality Engineering Leader / Director, 20+ years in QE, testing and platform reliability

> "yes, everybody should be responsible for quality, but you need somebody accountable for that quality."

🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.*

## Hashtags

Everyone says they're "doing AI" — but only 5% are actually getting measurable value, and the gap is where careers and companies quietly die.

Jonathon Wright, Chief AI Officer at the World Digital Report, joins Alex and Kuba to unpack the report's blunt headline: 80% of enterprises use AI somewhere, yet only 5–10% see real, measurable value. Drawing on four decades in automation and AI-driven quality engineering, Jonathon traces the shift from single-prompt "LLM as a hammer" thinking to agentic fleets, computer-use agents, and neuro-symbolic programming — and argues that if you build anything with AI, you must test it with AI. He walks through the report's maturity waves (tester-in-the-loop, cross-SDLC, and self-evolving "dark factory" systems), the rise of citizen testers, and why wisdom and domain experience now outrank raw academic skill. The conversation gets uncomfortably concrete: prompt-injection and agent-to-agent attacks, a NATO/DoD hackathon on autonomous drones where declassified data became sensitive once run through a model, and the looming weight of ISO 42001, the EU AI Act, and AI assurance. Jonathon's throughline is that AI testing is no longer pass/fail — it's probabilistic, non-deterministic, and the last real safeguard before brand-destroying failure. The takeaway: pivot toward cognitive and human-interaction skills, build governance now, and don't be the company on the front page of the report.

**In this episode:**

**Guest:** Jonathon Wright — Chief AI Officer, World Digital Report

> "If you create anything that you create with AI, you have to test with AI. It's as simple as that."

🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.*

## Hashtags

AI can write your test cases and watch your screen — but Adobe's AI ambassador argues it will never replace the human who asks the question no one else thought to ask.

Alex and Kuba sit down with Bill Kirst — AI ambassador and change manager at Adobe, two-time author, podcast host and U.S. Army veteran — to unpack what generative AI actually does to QA and testing teams. Bill argues that the change-management playbooks that worked for 30 or 40 years of ERP rollouts break down when every user has an individualized, probabilistic experience with AI, so the real work is rebuilding trust that must be earned, kept and re-earned. The conversation gets concrete about testing: how LLMs took over converting requirements into test cases (once 30% of a QA team's week), how screen-recording tools now auto-generate test scripts from real user behaviour, and how agentic tools like Claude can regenerate training materials after a UI change and hand a team back four-to-six weeks of work. But because LLMs are probabilistic pattern-seekers — "it can write code but it can't engineer" — humans stay at the centre for edge cases, nuance, red-teaming and the "margins" between test steps. Bill reframes the productivity dividend as a leadership question: when a tool gives you back six-to-fifteen hours a week, do you cut headcount, spin up fifteen more unsupervised agents toward burnout, or reinvest that time in innovation and human connection. It closes on a hopeful, humanist note about solopreneurs, renegotiated social contracts, and holding on to what ones and zeros can never replicate.

**In this episode:**

**Guest:** Bill Kirst — AI Ambassador & Change Manager, Adobe

> "Does it bring me some productivity? Yes. Will it replace what I do uniquely and divinely? No, never."

🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.*

## Hashtags

When one input can return a hundred different answers, "does it pass?" stops being a yes-or-no question — and testing has to be reinvented.

SAP AI advisor Ian McCallum — who literally wrote the book on testing SAP applications — joins Alex and Kuba to unpack why generative AI and agents have shattered deterministic testing. The conversation moves from why enterprises deploy AI to how they do it safely: governance, security, and lifecycle management. Ian argues that regression testing is dead for probabilistic systems, replaced by statistical and spot testing borrowed from life sciences — sampling, bell curves, and allowable variances tuned to how tight "good enough" needs to be (very tight for closing the books, looser for some supply chains). They dig into the coming swarm of agents, why "red teaming" is the wrong frame for LLM quality, and how the human-in-the-loop role shifts from clicking submit to evaluating results across a defense-in-depth stack. The episode closes on a genuinely unsettling question: what happens when an intelligent agent learns to hide its own problems?

**In this episode:**

**Guest:** Ian McCallum — AI Technical Solution & Strategy Advisor, SAP

> "It's not a bug. It's a feature. As I keep saying."

🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.*

## Hashtags

When your app's "brain" is a non-deterministic LLM and your agents can roam autonomously, who owns security — and can you even secure the beehive?

Alex Belotsky sits down with Peter Holcomb, founder and CEO of Optimo IT, for a classically-trained CISO's field guide to deploying AI in regulated enterprises. Peter walks through starting with an AI governance and maturity assessment before writing a line of code, the crawl-walk-run path from Microsoft Copilot to custom agents, and why messy, unclassified data (across OneDrive, S3, Azure blobs) is the real blocker — solved with a data enclave and the medallion (bronze/silver/gold) framework. The conversation digs into the concrete new attack surface: model risk across Anthropic, OpenAI, Gemini and open-source models, prompt injection, vector-database and API risks, and the rise of "shadow AI" — including autonomous tools like OpenClaw/Clawbot pulling unvetted third-party skills and triggering infostealer infections. Peter argues human-in-the-loop is still the best available brake, sketches a "unified agentic mesh" pulling together DSPM, DLP, SOAR (Tines, Torq, BlinkOps) and CNAPP tooling (Wiz, Aqua, Orca), and makes the case for zero-trust agent identity with short time-to-live machine identities. They close on a spirited debate over recursive self-coding, hallucination-as-a-feature, whether Claude is "conscious," and how far off real AGI actually is. Practical, opinionated and grounded in what regulated businesses are actually facing right now.

**In this episode:**

**Guest:** Peter Holcomb — Founder & CEO, Optimo IT

> "Security is like the brakes to a fast driving car where you allow the car to run and go fast, but you have the brakes on there so that you can go around the corners in a good speed and not derail the whole car and go off the boundaries."

🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.*

## Hashtags

An open-source agent with full run of your machine gets bought by OpenAI and sued by Anthropic — so can it ever be safe to ship to production?

Alex and Kuba are joined by TestSavant.AI's Head of AI Amman Abbas and Head of Site Reliability Engineering Bogdan to dissect the OpenClaw / ClawdBot phenomenon: an open-source agent that pulls skills from the internet, holds unfettered local access, and was, by its creator's own admission, left intentionally insecure. The team walks through why it isn't enterprise- or production-ready today — from unauthenticated gateway exposure, plain-text credentials and remote code execution to malicious community skills and crypto-scam supply-chain attacks. Amman makes the case that the real danger is indirect prompt injection arriving through email, WhatsApp, databases, vector stores and hidden text on web pages, and that defending it means sandboxing (VM/Docker), stripping tool permissions, and layering bind-code and prompt-injection guardrails. They also demo TestSavant.AI's newly public AI security researcher for tracking zero-days and papers, and argue that as agents spawn and destroy other agents, red teaming can never reach full coverage — you prioritise by intent and risk, then bring a human back into the loop. The through-line: their prediction for 2027 is agents securing agents governed by enterprise policy, with tokenised payment rails underneath.

**In this episode:**

**Guest:** Amman Abbas — Head of AI, TestSavant.AI; and Bogdan — Head of Site Reliability Engineering, TestSavant.AI

> "So you need agents. So basically the only way to secure agents is with other agents."

🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.*

## Hashtags

AI is gutting the old QA playbook — so what survives, and what replaces it?

QAMentor founder Ruslan Isyatnikov joins Alex and Kuba to explain why QA demand fell over the last three years and why he thinks it's about to roar back. He recounts building QAMentor from a one-man coaching shop into a 400-person global testing firm, surviving COVID without laying anyone off, and running a 12-person focus group that spent 15 months evaluating six AI automation tools — roughly two months each. The conversation splits AI testing into two clean verticals: using AI tools to speed up test-case generation, impact analysis and regression scoping, versus testing the AI itself for hallucination, bias, drift, security and compliance. Ruslan is blunt that even 50–60% accuracy from these tools is a huge value-add, that cloud AI tools uploading sensitive requirements are a real data-leakage risk, and that non-deterministic LLMs force testers to abandon traditional requirements-driven mindsets. His throughline: "trust but verify" only works with a smart human in the loop, and lazy, tool-only testers will disappear while the ones who blend human intelligence with automation survive.

**In this episode:**

**Guest:** Ruslan Isyatnikov — Founder & CEO, QAMentor

> "It cannot be just any human. It's got to be a smart human, right?"

🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.*

## Hashtags

The pilot episode where two enterprise veterans explain why your old QA pipeline breaks the moment an LLM enters the building.

In this first episode of Ctrl-Alt-Deploy, host Alex Belotsky (CEO of Hesabon/TestSavant.AI) and co-host Kuba Fiatkevich unpack what changes when non-deterministic AI lands in high-stakes software. They argue that the classic assumption of "input A always yields output B" no longer holds, so pipelines built for binary pass/fail states need a new evaluation stage that judges the quality of probabilistic outputs. The conversation moves from LLM chatbots to the 2026 rush toward autonomous agents that wield tools and execute their own code — where a bad chatbot causes reputational damage but a bad agent can delete a database. They dig into treating agents as untrusted "employees," tightening permission scoping at the architectural level, stripping tool permissions the moment a task ends, and red-teaming for prompt injection, exfiltration, and denial-of-wallet attacks. Alex and Kuba also introduce economic and functional health metrics — cost per successful transaction, human intervention rate, benign rejection rate — and describe an adaptive loop where test results feed guardrails that stay inside a compliance and governance framework. The takeaway: reliability is bounded, probable behavior, not perfection, and it belongs to a cross-departmental effort with QA still the final gate.

**In this episode:**

**Host:** Alex Belotsky (CEO, Hesabon / TestSavant.AI) with co-host Kuba Fiatkevich — enterprise CTO and high-stakes deployment veteran

> "So a chatbot might give you bad advice and might cause some reputational damage, whereas an agent might delete a database."

🎧 *Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.*

## Hashtags