Summary

Most e-commerce companies test a handful of features each month. Fanatics runs nearly 100 experiments monthly and delivers a big portion of the company's total annual growth through experimentation alone. Medha Umarji, VP of Growth and Experimentation at the multi-billion dollar sports merchandising retailer, explains how she built a program that scales from 10 tests per month to 100—and maintains enough rigor to spot false positives before they become costly decisions.

The difference isn't tooling or headcount. It's culture. When your CEO reads Excel spreadsheets for fun and actively wants data to prove him wrong, you stop debating whether to test and start debating how to test smarter. Medha shares the frameworks Fanatics uses to balance speed with rigor: a "do no harm" track for brand plays that won't show up in conversion metrics, a small-sample framework for teams that can't hit statistical significance thresholds, and an experimentation Wiki that feeds a continuous iteration flywheel. One surprising test on ad removal initially showed 95% statistical significance—until they replicated it and found the result was a false positive. The lesson: even at scale, you need to double-click on causality.

Timestamps

03:09 How Fanatics scaled from 10 to 100 experiments per month over 10 years

05:25 Why some leadership teams embrace experimentation and others resist it

07:06 How experimentation consistently delivers a big portion of Fanatics' annual growth

08:20 What happens when your CEO consumes Excel spreadsheets and questions everything

10:35 How top-down humility shapes an entire company's testing culture

12:10 The ad removal test that looked like a 95% win—then failed replication

15:55 How Fanatics built an experimentation Wiki that powers their growth engine

22:45 The "do no harm" framework for features that don't measure cleanly in A/B tests

25:20 Why lowering barriers to adoption matters more than statistical perfection early on

26:27 Your odds of winning at experimentation are worse than roulette

Takeaways

• Replication catches false positives: A 95% confidence level still means 1 in 20 results are noise—if a critical test outcome can't be explained through micro-metrics, run it again before committing resources.
• Top-down buy-in shifts the conversation from "why test?" to "how do we test?": When leadership treats data as the tiebreaker, teams stop defending opinions and start building better experiments.
• Frameworks like "do no harm" and "small sample" expand who can test: Not every initiative needs 30,000 orders to ship value—lower the barrier for teams that can't hit statistical thresholds while protecting core KPIs.
• Documenting experiments in a centralized Wiki creates a growth flywheel: Fanatics' Wiki feeds their roadmap with iterations on already-built features, reducing tech dependency and accelerating velocity.
• Micro-metrics establish causality beyond top-line KPIs: If revenue moves but scroll depth, cart adds, and product views don't follow the same pattern, question the result before declaring a win.

Connect with the guest

LinkedIn: https://www.linkedin.com/in/medhaumarji/

Learn more about Fanatics

https://www.fanatics.com/