TMI - Testing and Management Insights

In this episode of TMI, Chip and Tess dive deep into the frustrating world of flaky tests. These are the tests that pass sometimes and fail at other times, even when the code hasn't changed.

We discuss reference research from Trunk, who analyzed over 20 million CI jobs and found that flaky tests are a widespread problem, even among large tech companies like Meta, Google, and Uber.

We examine strategies for managing flaky tests, such as quarantining them. We also cover how to prevent flaky tests by ensuring isolation, using dynamic waits instead of hard-coded timeouts, and utilizing deterministic inputs.

Finally, we discuss the importance of the testing pyramid, controlled test environments, contract testing, robust error handling, retry mechanisms, and clear communication.

References:

• https://www.testrail.com/blog/flaky-tests/
• https://testsigma.com/guides/flaky-test/
• https://trunk.io/blog/what-we-learned-from-analyzing-20-2-million-ci-jobs-in-trunk-flaky-tests-part-1
• https://trunk.io/blog/what-we-learned-from-analyzing-20-2-million-ci-jobs-in-trunk-flaky-tests-part-2

Explore the three pillars of Agile estimating and planning: embracing change, prioritizing value, and effective communication—principles that resonate beyond software development.

• Learn why multitasking can hinder progress despite feeling productive.
• Understand the pitfalls of setting initial estimates in stone.
• Learn why providing estimation dates and team velocity as ranges -- rather than a definitive number -- helps manage expectations.
• Discover how properly prioritizing features can deliver 80% of benefits with 20% of the effort.
• We discuss using Story Points to estimate complexity, Ideal Days for estimating the time it will take to completer a task, and tips for splitting large user stories.
• Delve into planning poker for estimation and strategies for backlog prioritization.

Finally, we consider a sprint structure that includes dedicated time for technical debt.

More Resources:

• Agile Estimating and Planning

In this episode, explore the Agile philosophy in software development. Learn about using story points and “Ideal Days” for estimation, and the importance of communication and trust in Agile teams.

Discover how to choose optimal sprint durations, understand velocity, and use burn-down charts to spot risks. Dive into Scrum as a framework, including the role of the Scrum Master, sprint planning, daily meetings, and retrospectives.

Understand continuous integration’s impact on quality and the significance of visualization with diagrams. Learn how Google fosters a testing culture and why automation doesn't replace human tester's, but rather enhances their productivity. F

inally, get tips for distributed teams and effective backlog prioritization by the product owner.

More Resources:

• More Agile Testing: Learning Journeys for the Whole Team
• Agile Estimating and Planning
• Succeeding with Agile: Software Development Using Scrum
• How Google Tests Software
• The Feiner Points of Leadership: The 50 Basic Laws that Will Make People Want to Perform Better for You
• The A Word
• Exploratory Software Testing
• The One Thing: The Surprisingly Simple Truth Behind Extraordinary Result
  

In this episode, we dive into exploratory testing, a dynamic technique where tests are designed and executed simultaneously, using insights from previous experiments to guide future ones. We explore various exploratory testing tours, including:

• The Guide Book Tour: Following the user manual step by step.
• The Money Tour: Prioritizing features most essential to users.
• The Saboteur Tour: Intentionally introducing problems like network errors and corrupted files.
• The Anti-Social Tour: Doing the opposite of expected user actions.

We also discuss testing in the small versus testing in the large, scenario testing, and designing tests using states and events. Finally, we share thoughts on the future of testing.

Other Resources:

• Exploratory Software Testing
• Explore it? Explore it! (YouTube Video)

In this episode of the TMI Podcast, we dive into the intriguing world of Testing in Production and how it allows you to observe your software in real-world conditions. Join us as we discuss key techniques to minimize risks while testing in production:

• Feature Flags: Deploy code to production servers without immediate exposure to users. Feature flags allow you to unveil new features only when you’re confident they will function seamlessly.
• Shadowing: Safely test new code by sending a copy of live traffic without impacting the customer experience.
• Staged Rollouts and Canarying: Gradually introduce new code to a small percentage of users, increasing exposure as confidence grows.
• A/B Testing: Compare different versions of features to determine the most effective approach.

We also discuss the innovative practice of chaos engineering, where intentionally causing disruptions builds system resilience and confidence.

Discover why a corporate culture that embraces risk is essential for successful Testing in Production. Tune in for insights on how to leverage these strategies to enhance your software’s performance and reliability in the real world.

More Resources:

https://www.youtube.com/watch?v=n2iE0wUt3bc https://medium.com/life-at-apollo-division/testing-in-production-339dc58dd8cd https://copyconstruct.medium.com/testing-in-production-the-safe-way-18ca102d0ef1 https://copyconstruct.medium.com/testing-in-production-the-hard-parts-3f06cefaf592

Are you ready to navigate the AI revolution and elevate your career? Dive into our latest episode, where AI and testing experts share the must-have skills to thrive in the age of artificial intelligence.

🔍 Key Highlights:

• Essential AI Literacy: Discover why understanding AI is vital for everyone in tech, no matter your role.
• Transforming Testing: Learn how AI is reshaping the testing landscape and what this means for you.
• Human Skills Matter: Explore the crucial role of critical thinking, context understanding, and ethical judgment in an AI-driven world.
• Prompt Engineering Insights: Uncover the skills that truly matter when working with AI.
• New Opportunities: Find out how AI is creating job prospects and the importance of continuous learning.
• Building AI Literacy: Get practical steps to enhance your AI knowledge and stay ahead, featuring resources like Deepseek and Testers.ai.
• AI Assurance and Ethics: Understand the growing importance of responsible AI use and the human element in tech.

Don’t get left behind! Learn to collaborate with AI and position yourself for success in the tech landscape of 2025 and beyond. Tune in for actionable insights and a reassuring perspective on the future of work in the age of AI.

Resources:

• https://www.youtube.com/watch?v=FHS6hlDko4E
• https://www.youtube.com/watch?v=oX625Lm1QoM

In this episode, we explore the four essential metrics that define high-performing teams:

1. Delivery Lead Time: The speed of transitioning code from a developer’s machine to production.
2. Deployment Frequency: The regularity of deployments to production.
3. Resilience: Mean time to restore service
4. Change Fail Rate: The percentage of releases that encounter issues in production.

We challenge the notion that developers shouldn’t be highly engaged in testing. We also delve into the critical role of fast feedback loops.

Other Resources:

• Accelerate: Building and Scaling High Performing Technology Organizations
• The Weasel Speaks Blog
  

In this episode, we dive into the Agile Manifesto, emphasizing individuals and interactions over processes and tools. We explain why quality isn't something you can tack on at the end. We discuss fostering a culture where everyone feels empowered to speak up.

Explore iterative development and user stories, along with the agile testing quadrants. Learn about test-driven development (TDD), acceptance testing, exploratory testing, and usability testing.

We also cover performance, scalability, load, and stress testing. Communication is key, with daily standups, retrospectives, and sprint reviews. Discover the test automation pyramid and strategies for overcoming team resistance during the transition to Agile.

References:

• Agile Testing: A Practical Guide for Testers and Agile Teams
• More Agile Testing: Learning Journeys for the Whole Team
• Agile Estimating and Planning
• Coaching Agile Teams: A Companion for ScrumMasters, Agile Coaches, and Project Managers in Transition
• Succeeding with Agile: Software Development Using Scrum
  

In this episode, we dive deep into integrating testing throughout the entire development process. Discover the nuances of equivalence class testing and embark on exploratory testing tours like The Back Alley Tour and the Rained Out Tour.

We discuss striking the right balance between manual and automated testing and explore how tech giants like Google categorize their test cases. Learn about the concept of test independence at Microsoft and the significance of building testability into your software.

We’ll tackle the dangers of code complexity and how to measure it effectively. Embrace holistic testing and understand why quality goes beyond just functional bugs. Discover the practice of dogfooding, where employees use products before public release, and the strategic use of “User Personas” in planning your testing. 

Tune in for a comprehensive guide to enhancing your testing strategy!

References:

• A Practitioners Guide to Software Test Design
• How Google Tests Software
• Exploratory Software Testing
• How We Test at Microsoft
• Holistic Testing: Weave Quality into your Product
• The A Word
  

Is AI about to revolutionize software testing? This episode dives deep into the future of QA, exploring how AI is changing the game.

We tackle the big questions: Will robots take our jobs? How do we test AI? Can we trust AI systems?

• AI-first testing and what it means for efficiency.
• The rise of the "hybrid tester" and the evolving role of human expertise.
• Autonomous testing and the crucial role of human oversight.
• The challenges of ensuring AI-generated code quality.
• Tool fatigue and how to choose the right AI-powered testing solutions.
• The growing importance of compliance and safety in AI.
• Shift left testing and integrating AI into every stage of development.
• Low code and no code testing solutions.
• Agentic testing and AI as a partner.
• AI-powered analytics and predictive testing.

Learn what skills you need to stay ahead and thrive in the AI-powered future. Adaptability and continuous learning are key.

Tune in to discover how to embrace the possibilities of AI and shape the future of software testing!

Resources:

• https://www.youtube.com/watch?v=FHS6hlDko4E
• https://www.youtube.com/watch?v=oX625Lm1QoM
• https://www.t-plan.com/navigating-software-testing-in-2025-the-predicted-top-10-trends/
• https://kishor-munot.medium.com/predictions-in-2025-testing-and-qa-with-ai-in-next-365-days-f9b150fc0e7f
• https://www.itsutra.com/blog/qa-automation-2025