In this episode 600 of the TestGuild Automation Podcast, Joe Colantonio talks with Jason Arbon, founder of Testers.ai, Jank.AI and IcebergQA and author of the new book Testing AI: Engineering Confidence in Non-Deterministic Systems. Take Our 2027 Survey Now: <a href= "https://testgld.link/27data">https://testgld.link/27data</a> Jason makes a case most testers have not heard yet. Coding is being absorbed by AI. Specification work is thinning out. Product, development, and test roles are converging into one. And when the music stops, the only seat left belongs to the person who can look at what the machine produced and make an evidence backed call on whether it ships. He calls that confidence engineering, and he argues it is not a rebrand of QA. It is what QA was always supposed to be. Along the way, Joe and Jason get into the containment problem and why alignment, not lockdown, is now the real safety goal. They dig into why testing cost scales quadratically, meaning ten times more generated code creates roughly a hundred times more testing demand. Jason also pushes back hard on skeptics of agentic testing, pointing out that almost nobody has run the obvious experiment of testing a site themselves for a week and comparing their results against what AI finds. You will also hear Jason's most practical piece of advice in the whole conversation. If you are not running the same suite five times against the same build and looking at the actual results, not just flake, you are not testing seriously in an AI world. Plus a detour into grokking, the Chinese Room, and Geoffrey Hinton, because it would not be a Jason Arbon episode without one. Listen up!

In this episode 600 of the TestGuild Automation Podcast, Joe Colantonio talks with Jason Arbon, founder of Testers.ai, Jank.AI and IcebergQA and author of the new book Testing AI: Engineering Confidence in Non-Deterministic Systems. Take Our 2027 Survey Now: https://testgld.link/27data Jason makes a case most testers have not heard yet. Coding is being absorbed by AI. Specification work is thinning out. Product, development, and test roles are converging into one. And when the music stops, the only seat left belongs to the person who can look at what the machine produced and make an evidence backed call on whether it ships. He calls that confidence engineering, and he argues it is not a rebrand of QA. It is what QA was always supposed to be. Along the way, Joe and Jason get into the containment problem and why alignment, not lockdown, is now the real safety goal. They dig into why testing cost scales quadratically, meaning ten times more generated code creates roughly a hundred times more testing demand. Jason also pushes back hard on skeptics of agentic testing, pointing out that almost nobody has run the obvious experiment of testing a site themselves for a week and comparing their results against what AI finds. You will also hear Jason's most practical piece of advice in the whole conversation. If you are not running the same suite five times against the same build and looking at the actual results, not just flake, you are not testing seriously in an AI world. Plus a detour into grokking, the Chinese Room, and Geoffrey Hinton, because it would not be a Jason Arbon episode without one. Listen up!

In this episode 600 of the TestGuild Automation Podcast, Joe Colantonio talks with Jason Arbon, founder of Testers.ai, Jank.AI and IcebergQA and author of the new book Testing AI: Engineering Confidence in Non-Deterministic Systems. Take Our 2027 Survey Now: <a href="https://testgld.link/27data" rel="noopener noreferrer">https://testgld.link/27data</a> Jason makes a case most testers have not heard yet. Coding is being absorbed by AI. Specification work is thinning out. Product, development, and test roles are converging into one. And when the music stops, the only seat left belongs to the person who can look at what the machine produced and make an evidence backed call on whether it ships. He calls that confidence engineering, and he argues it is not a rebrand of QA. It is what QA was always supposed to be. Along the way, Joe and Jason get into the containment problem and why alignment, not lockdown, is now the real safety goal. They dig into why testing cost scales quadratically, meaning ten times more generated code creates roughly a hundred times more testing demand. Jason also pushes back hard on skeptics of agentic testing, pointing out that almost nobody has run the obvious experiment of testing a site themselves for a week and comparing their results against what AI finds. You will also hear Jason's most practical piece of advice in the whole conversation. If you are not running the same suite five times against the same build and looking at the actual results, not just flake, you are not testing seriously in an AI world. Plus a detour into grokking, the Chinese Room, and Geoffrey Hinton, because it would not be a Jason Arbon episode without one. Listen up!

Testing AI: Engineering Confidence in Non-Deterministic Systems with Jason Arbon

Your developers just got supercharged by AI coding agents. Your test coverage did not. So how do you keep quality high when product code is shipping faster than any test team can follow? In this episode Andrew Knight, the Automation Panda and Senior Director of Product and Engineering at Cycle Labs, shares how his six person team is running the biggest release quarter in company history using AI coding agents, spec driven development, and Playwright. You will discover: <ul> <li>How Playwright turned itself into an AI automation platform with the MCP server, the planner, generator, and healer agents, and the new CLI and skills approach that cuts your token usage way down.</li> <li>Why Andy uses Spec Kit to codify his testing strategy once, in markdown, so quality standards get baked into every single pull request instead of being caught in review.</li> <li>How to decide which AI generated tests are actually worth running when compute time and budget are finite.</li> <li>What AI slop looks like from a manager's seat, and how to build a team culture that catches it before it ships.</li> <li>Why Andy believes AI coding tools are the new compiler and markdown is the new programming language, plus my pushback on what that means for everything testers were trained to care about.</li> <li>What Andy really thinks about token costs, subscription tiers, and what happens when the AI subsidies run out.</li> <li>Plus the one piece of advice he gives to any tester still sitting on the sidelines of the AI shift.</li> </ul> Whether you are an automation engineer, a QA lead, or an engineering manager trying to figure out where testing fits in an AI first workflow, this episode gives you a practical playbook you can start using this week.

Your developers just got supercharged by AI coding agents. Your test coverage did not. So how do you keep quality high when product code is shipping faster than any test team can follow? In this episode Andrew Knight, the Automation Panda and Senior Director of Product and Engineering at Cycle Labs, shares how his six person team is running the biggest release quarter in company history using AI coding agents, spec driven development, and Playwright. You will discover: How Playwright turned itself into an AI automation platform with the MCP server, the planner, generator, and healer agents, and the new CLI and skills approach that cuts your token usage way down. Why Andy uses Spec Kit to codify his testing strategy once, in markdown, so quality standards get baked into every single pull request instead of being caught in review. How to decide which AI generated tests are actually worth running when compute time and budget are finite. What AI slop looks like from a manager's seat, and how to build a team culture that catches it before it ships. Why Andy believes AI coding tools are the new compiler and markdown is the new programming language, plus my pushback on what that means for everything testers were trained to care about. What Andy really thinks about token costs, subscription tiers, and what happens when the AI subsidies run out. Plus the one piece of advice he gives to any tester still sitting on the sidelines of the AI shift. Whether you are an automation engineer, a QA lead, or an engineering manager trying to figure out where testing fits in an AI first workflow, this episode gives you a practical playbook you can start using this week.

Playwright With AI: How to Automate Tests Without Shipping AI Slop with Andrew Knight

Dave Haeffner author of the new book The Selenium Guidebook shares his tips and tricks on how to use Selenium, successfully. Learn how to craft automated acceptance tests that you and your team can trust.

Dave Haeffner author of the new book The Selenium Guidebook shares his tips and tricks on how to use Selenium, successfully. Learn how to craft automated acceptance tests that you and your team can trust.

02: Dave Haeffner : The Selenium Guidebook

What do Steampunk and Harry Potter have to do with Behavior Driven Development (BDD)? Join us as Lance enlightens us about BDD.

What do Steampunk and Harry Potter have to do with Behavior Driven Development (BDD)? Join us as Lance enlightens us about BDD.

01: Lance Kind: Discover the Benefits of Behavior Driven Development

TestGuild Automation Podcast (formally Test Talks) is a weekly podcast hosted by Joe Colantonio, which geeks out on all things software test automation. TestGuild Automation covers news found in the testing space, reviews books about automation, and speaks with some of the thought leaders in the test automation field. We'll aim to interview some of today's most successful and inspiring software engineers, and test automation thought leaders.

Share TestGuild Automation Podcast

Sign up to save your podcasts

TestGuild Automation Podcast

TestGuild Automation Podcast episodes

About TestGuild Automation Podcast

More shows like TestGuild Automation Podcast

This American Life

Freakonomics Radio

Planet Money

Hidden Brain

Software Engineering Daily

Frank Off The Radio: The Frank Skinner Podcast

The Daily

Darknet Diaries

Today, Explained

TestGuild Devops Toolchain Podcast

The Stack Overflow Podcast

Huberman Lab

The Startup Ideas Podcast

The Rest Is Politics

The Rest Is Football