Software Testing Unleashed - QA, DevEx & Quality Engineering

Why Your CI Pipeline Is Lying to You - Simon Stewart

Listen Later
Managing Flaky Tests Strategies for Trustworthy CI Pipelines

📘 Free e-book: The 7 success factors of software testing. 25 years of project experience in one 33-page workbook, now also in English 👉 Get it for free

"A flaky test can actually sometimes be a good test because it's highlighting things." - Simon Stewart

In this episode, I talk with Simon Stewart, professional software developer and former lead of the Selenium project for over 10 years, about one of the most frustrating problems in software testing: flaky tests. Simon reveals why a flaky test isn't always a bad test – sometimes it's actually exposing real production risks that your team needs to address. We dive into practical strategies for handling flakiness in CI pipelines, from gatekeeping techniques used at Meta to knowing when it's actually okay to delete tests. You'll learn why assigning ownership to individuals (not teams) is crucial, and how to use test flakiness as valuable signal rather than just noise.

Simon Stewart has been a professional software developer since before the millennium began. He was the lead of the Selenium project for over a decade and is the co-editor of the W3C WebDriver and WebDriver Bidi specs.

As well as browser automation, Simon is also interested in monorepos, blazing fast byte-for-byte reproducible builds, and scaling software development efficiently. He draws on his experience working in Open Source, ThoughtWorks, Google, and Facebook. He was the tech lead of Facebook’s build tool team, and is currently working on projects using Bazel, for which he’s the maintainer of several rulesets.
Simon lives in London with his family and dog.

Highlights:

  • A flaky test breaks the foundational property of repeatability, and once developers stop trusting CI results, the entire value of continuous integration collapses.
  • Flaky tests sometimes expose real production risks: transient network errors, unavailable dependencies, and race conditions in tests are the same failures users will encounter in the live system.
  • Removing a flaky test from CI rotation is necessary but not sufficient; ownership must be assigned to a named individual, not a team, or the fix will never happen.
  • Meta's practice of running every new test a hundred times concurrently overnight before admitting it to CI is a concrete, automated gatekeeping strategy that prevents flaky tests from entering the build at all.
  • Deleting a flaky test is a legitimate resolution, especially when lower-level tests already cover the same risk, because shipping changes quickly and safely matters more than preserving test code.
    • ...more
    View all episodesView all episodes
    Download on the App Store
    Software Testing Unleashed - QA, DevEx & Quality EngineeringBy Richard Seidl | Software Development & Testing Expert