Martin Tingley, Head of Windows Experimentation at Microsoft and former Head of the Experimentation Platform Analysis Team at Netflix, talks about why humans are the bottleneck in experimentation, and how a five-level maturity framework points the way toward self-optimizing software.
Our conversation traces the path from basic hypothesis testing to a frontier where Generative AI creates, evaluates, and refines product variants in a closed loop. We explore the architectural shift required to move from testing single variants to optimizing entire parameter spaces, and how startups are already using AI to generate production-ready landing pages for Fortune 500 companies in hours rather than weeks. Tingley also shares a strategic lens on "experimentation programs," explaining how plotting the distribution of treatment effects across different product areas can serve as a powerful tool for capital allocation and high-level strategy.
Martin on LinkedInWant Your Company to Get Better at Experimentation? by Iavor Bojinov, David Holtz, Ramesh Johari, Sven Schmit and Martin Tingley (Harvard Business Review)Avoid the Pitfalls of A/B Testing by Iavor Bojinov, Guillaume Saint-Jacques and Martin Tingley (Harvard Business Review)Martin & Co.'s Seven Part Blog Series on Experimentation at NetflixRoberto Medri (Meta) on High Signal: The Incentive Problem in Shipping AI Products — and How to Change ItTim O’Reilly on High Signal: The End of Programming As We Know ItWatch the podcast episode on YouTubeDelphina's Newsletter