In this episode of TMI, Chip and Tess dive deep into the frustrating world of flaky tests. These are the tests that pass sometimes and fail at other times, even when the code hasn't changed.

We discuss reference research from Trunk, who analyzed over 20 million CI jobs and found that flaky tests are a widespread problem, even among large tech companies like Meta, Google, and Uber.

We examine strategies for managing flaky tests, such as quarantining them. We also cover how to prevent flaky tests by ensuring isolation, using dynamic waits instead of hard-coded timeouts, and utilizing deterministic inputs.

Finally, we discuss the importance of the testing pyramid, controlled test environments, contract testing, robust error handling, retry mechanisms, and clear communication.

References:

• https://www.testrail.com/blog/flaky-tests/
• https://testsigma.com/guides/flaky-test/
• https://trunk.io/blog/what-we-learned-from-analyzing-20-2-million-ci-jobs-in-trunk-flaky-tests-part-1
• https://trunk.io/blog/what-we-learned-from-analyzing-20-2-million-ci-jobs-in-trunk-flaky-tests-part-2