Summary

Chess.com ran its first A/B test in 2023. Two years later, the team is on track to run 1,000 experiments in a single year—and they've already shipped 195 in Q1. 

In this episode, Ashley Stirrup sits down with Nafis Shaikh, Director of Product Management at Chess.com, to get inside the experimentation engine powering one of the world's most beloved gaming products. 

Nafis brings experience from Zynga and Prodigy and a refreshingly honest take on what changes when a product built on passion suddenly has to serve a 10-million-DAU user base that spans absolute beginners to rated FIDE players. He and Ashley get into why one-size-fits-all doesn't actually fit anyone, how to measure an AI coach when you can't tell whether users have their volume on, and a game review experiment that completely upended the team's assumptions about how players want to learn. 

Nafis also shares practical advice for product managers trying to introduce experimentation culture to organizations that have never done it before—starting with a simple pre/post test rather than a fancy platform. If you lead product, care about experimentation maturity, or just want to hear how a classic product is scaling its learning loop, this one's worth your time.

Timestamps

• [00:35] – Chess.com's experimentation origin story and the 1,000-test goal
• [05:01] – Designing for a user base that spans beginners to FIDE-rated players
• [07:30] – The four metrics dimensions Nafis uses to evaluate tests
• [12:03] – How do you A/B test an AI coach when you can't tell who's listening?
• [15:49] – Embracing humility and the shift away from "we know what works"
• [20:50] – The game review test that surprised everyone: 80% of users review wins
• [24:06] – Advice for PMs introducing experimentation at a new company
• [29:15] – The onboarding debate and personalization from session zero

Takeaways

• Scale test volume to learning speed, not just shipping speed
• Build hypotheses around user psychology, not just KPI movement
• Accept that being wrong is the point—experimentation only works when leadership embraces humility
• Start simple if you're new to experimentation; a clean pre/post comparison beats a fancy platform you don't use
• Reposition features around how users actually feel, not how you assume they should feel
• Design onboarding around the shortest path to value, not the longest path to personalization

Guest LinkedIn: https://www.linkedin.com/in/nafis-shaikh-20161916/

Company website: https://www.chess.com