The Experimentation Edge

Summary
What changes when you take a mature B2C experimentation practice and apply it to a B2B storefront? Anuradha Tempe, Lead Product Manager at Samsung Electronics America, joins host Ashley Stirrup to share what she learned managing both sides of Samsung.com. She explains why B2B is not a sidekick to B2C, how a single bulk order can push a test into a false positive unless you normalize the data, and why not everything deserves an A/B test. She walks through the add-on experiment that nearly doubled attach sales once her team realized business buyers decide on mobile and purchase on desktop, and the Buy Now test that lost because B2B buyers value clarity over faster conversions. The conversation closes with her approach to North Star and guardrail metrics and the AI copilot she built to draft A/B test plans with a human still in the loop. A practical episode for product managers, engineers, data scientists, and growth leaders running experimentation across different customer types.

Chapters
00:45 Meet Anuradha Tempe: from chip design to leading Samsung e-commerce
02:15 Why B2B is not a sidekick to B2C
03:30 Not everything needs an A/B test
05:15 Spreading experimentation practice across a global conglomerate
07:00 Bringing B2C rigor to a fast-paced B2B team
08:45 The add-on experiment that nearly doubled attach sales
11:15 Buyers decide on mobile and purchase on desktop
16:30 The Buy Now test that lost
21:30 North Star metrics, guardrails, and the EPP discount fix
26:05 An AI copilot for A/B test planning

Takeaways
- Treat B2B as its own customer base with its own testing discipline; a single bulk order on one day can inflate a B2B test into a false positive unless order data is normalized before results are read.
- Not everything needs an A/B test; route lower-risk changes through UAT feedback or pre/post comparisons and reserve full experiments for features where being wrong is expensive.
- Map where the decision happens, not just where the purchase happens; Samsung's business buyers decide on mobile and buy on desktop, and surfacing add-ons on mobile nearly doubled attach sales.
- B2B buyers value clarity over faster conversions; a Buy Now button earlier in the flow confused bulk purchasers because returns and cancellations on large orders are costly.
- Align on the North Star before building, whether it is revenue, engagement, NPS, or fewer support tickets, and set guardrails so an engagement feature can never quietly drag sales down.

Connect with the Guest
Anuradha Tempe LinkedIn: https://www.linkedin.com/in/anuradha-tempe/
About the guest: https://www.growthbook.io/podcast/guests/anuradha-tempe
Episode page: https://www.growthbook.io/podcast/episode/1-42
Company Website: https://www.samsung.com/us/business/

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Summary
On this episode of The Experimentation Edge, Ashley Stirrup talks with Joe Yevoli, Director of Growth at Charlie Health, a virtual intensive outpatient program that sits between weekly therapy and hospitalization. Joe explains how removing a page from Charlie Health's intake form produced a winning test that created a new bottleneck further down the funnel, why Teachers Pay Teachers cut about 80% of its market for one feature and roughly quadrupled retention, and how premortems that ask "what went wrong?" before launch make dissent safe and surface safeguards that avoid catastrophe. He also shares the two questions he asks before any experiment, why more top-of-funnel traffic always dents conversion rate, and how AI is flattening the product pod in ways that speed teams up and put them at risk. It's for growth leaders, product managers, and experimentation teams who want to learn as much from a loss as from a win.

Chapters
00:00 Intro
01:00 About Charlie Health
02:05 Joe's role across performance, lifecycle, and experimentation
03:30 Building experimentation rigor across teams
04:40 Even a loss is a win
05:15 The form page test that created a new bottleneck
08:50 Teachers Pay Teachers and the Easel lesson
16:30 Designing experiments that teach you something when they lose
21:00 Premortems for high-risk experiments
24:30 AI is flattening the org

Takeaways
- A winning test is a data point, not a finish line; Charlie Health's form page removal won on top-of-funnel metrics and still exposed a downstream bottleneck that became the next experiment.
- More traffic entering the funnel means conversion rate goes down, even when more people reach the bottom; treat it as a law of physics and plan for it before you read the results.
- Build it for everyone and you build it for nobody; Teachers Pay Teachers cut roughly 80% of the market for Easel, repositioned it around one job, and roughly quadrupled retention.
- Run a premortem before any large, risky experiment: ask the room "it was a massive failure, what went wrong?", have everyone write and share, and build the safeguards while there is still time.
- Before launching, check the quantitative and qualitative data behind the hypothesis, map the full funnel, and measure toward the real business outcome, so a loss still leaves you with a next step.

Connect with the Guest
Joe Yevoli LinkedIn: https://www.linkedin.com/in/joeyevoli/
About the guest: https://www.growthbook.io/podcast/guests/joe-yevoli
Episode page: https://www.growthbook.io/podcast/episode/1-41
Company Website: https://www.charliehealth.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Summary
What do you do when your biggest experiment win turns out to be a loss? Whitney Perez, Director of Product Management at Realtor.com, joins host Ashley Stirrup to share the checkout bundling test that posted a 300% attach rate and still lost revenue, the 30/30/30 rule she uses to set expectations for a new experimentation team, and how AI is turning an English major into an aspirational data scientist. This episode is for product managers, engineers, and data scientists building experimentation programs from the ground up.

Chapters
00:00 Cold open and welcome
01:10 From growth hacker to Realtor.com
02:50 Three foundations for a new experimentation team
04:20 The 30/30/30 rule
05:05 The 300% bundling win that lost revenue
07:10 You don't need a stats degree to experiment
09:20 Cascading North Star metrics
12:50 Do the homework before the experiment
13:50 The wishlist: instrumentation, embedded knowledge, culture
15:50 AI as an aspirational data scientist
18:45 Keeping a human in the loop

Takeaways
- A winning decision metric is not enough. Realtor.com's bundling test hit a 300% attach rate, but funnel fallout from the extra step made it a net revenue loser. Set secondary metrics and their thresholds before launch.
- Expect the 30/30/30 rule: roughly a third of tests win, a third are inconclusive, and a third lose. The math is the math, and the losers carry most of the learning.
- Start a new team on foundations: what a clean test and an A/A test look like, which surfaces should not be tested, and a peer review program that lets people graduate to more complex experiments.
- You don't need a stats background to run good experiments. Teach the simplest definition of a good test, then let people learn by doing.
- AI can make anyone an aspirational data scientist for analyzing results and spotting opportunities, but it can be confidently wrong. Keep a human in the loop and sanity check output the way you'd peek at a freshly launched test.

Connect with the Guest
Whitney Perez LinkedIn: https://www.linkedin.com/in/whitneykperez/
About the guest: https://www.growthbook.io/podcast/guests/whitney-perez
Episode page: https://www.growthbook.io/podcast/episode/1-40
Company Website: https://www.realtor.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Summary
How do you keep 1,000 experiments a year pointed at one North Star? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Mi Tian, Head of Applied Science at Zalando, about running experimentation inside a central economics org that reports to the CFO. Mi shares how Zalando balances safe confirmatory tests with game-changing bets, how the team measured the discovery feeds homepage launch when success had no established metric, and how a KPI tree cascades the company North Star down to the controllable inputs teams ship every day. She also looks ahead to LLM-based agents as a simulation layer for screening hypotheses. A practical conversation for anyone building an experimentation program that wants both rigor and ambition.

Chapters
00:45 About Zalando and its marketplace model
02:00 Mi's path from engineering to experimentation
03:10 Economists and data scientists in one decision-making org
04:15 Running over 1,000 experiments a year
05:55 What makes an experiment high risk
07:10 Sharing learnings through standardization and champions
09:10 The discovery feeds launch and its measurement plan
11:15 Balancing the experimentation portfolio
12:35 Growing a KPI tree from the North Star
16:05 LLM agents and the future of experimentation at Zalando
18:35 New missions for a longstanding business

Takeaways
- Treat experimentation as a portfolio: balance confirmatory tests that protect the business with game-changing bets that can win big.
- Assess risk tiers when building the roadmap so measurement rigor scales with the stakes instead of slowing every decision down.
- When a launch is too new to have a success metric, pair short term A/B tests with long term holdouts from day one.
- Connect every experiment to the company North Star by cascading it down to sensitive proxy metrics and controllable inputs.
- Use LLM based agents as a cheap simulation layer to screen hypotheses, not as a replacement for real A/B tests.

Connect with the Guest
Mi Tian LinkedIn: https://www.linkedin.com/in/mi-tian-941b4767/
About the guest: https://www.growthbook.io/podcast/guests/mi-tian
Episode page: https://www.growthbook.io/podcast/episode/1-39
Company Website: https://www.zalando.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Summary
Rich Liebling, senior director of engineering at Learneo, joins host Ashley Stirrup to explain the practice that came out of growing Shop It To Me from 50,000 subscribers to one million in nine months: test your hypothesis, and test its opposite. Rich covers the page where cutting text lost and adding text won, why the inverse wins more often than teams expect, and why small focused tests are the only ones where "the opposite" means anything. He also walks through translating a million subscriber goal into a target of 300 A/B tests, making a new engineer's second merge request their own experiment, and the different constraints at Course Hero, where competing team metrics were resolved with an early lifetime value model and three-day SQL analyses quietly capped testing velocity. This episode is for product managers, engineers, data scientists, and growth leaders building or scaling an experimentation program.

Chapters
00:00 Cold open and introduction
01:50 Shop It To Me and the first engineering hire
02:55 300 A/B tests as the path to one million subscribers
04:00 Testing as a core value and the second merge request
07:20 Learneo, Course Hero, and two ways to get access
08:50 Modeling lifetime value to stop teams competing
11:10 Testing the opposite of the hypothesis
14:35 A portfolio of small, medium, and large tests
17:15 Multi armed bandits and seasonal traffic
20:35 The flywheel that keeps copycats behind

Takeaways
- Test the opposite of every hypothesis. At Shop It To Me the inverse won surprisingly often, and even when it lost it proved the variable mattered.
- Keep tests small and focused. Redesign a whole page and lose, and you learn that version failed but not what to change next.
- Translate a growth goal into an execution count. One million subscribers is not actionable. 300 A/B tests by year end is, and everyone can influence it.
- Make experimentation part of hiring and onboarding. Every new engineer's second merge request was their own test idea.
- Analysis friction sets the ceiling on testing velocity. Three days of ad hoc SQL per test quietly discourages teams from running more.

Connect with the Guest
Rich Liebling LinkedIn: https://www.linkedin.com/in/richliebling/
About the guest: https://www.growthbook.io/podcast/guests/rich-liebling
Episode page: https://www.growthbook.io/podcast/episode/1-38
Company Website: https://www.learneo.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Summary
What separates a valid A/B test from an expensive guess? Priya Singhee, VP of Enterprise Analytics & Data Science at Early Warning — the bank-owned consortium that fights payment fraud and operates Zelle, which processed a trillion dollars last year — joins host Ashley Stirrup to share the experimentation playbook she built leading storefront analytics at Wayfair. She walks through the four questions to ask before launching any A/B test, why 85 to 90% of tests are supposed to fail, how pre-registration and kill criteria stop p-hacking before it starts, the pitfalls that fake wins (novelty effects, hidden heterogeneity, multiple comparisons), and how to roll out winners with gradual ramps and long-running holdouts. A practical episode for product managers, engineers, data scientists, and growth leaders building rigorous experimentation programs.

Chapters
00:45 Meet Early Warning: fraud detection, Zelle, and a trillion dollars in payments
02:00 Wayfair and optimizing every step of the storefront funnel
03:05 The four questions to ask before any A/B test
05:15 Test setup best practices: hypotheses, guardrails, power, and pre-registration
07:40 Why 85 to 90% of tests fail and why that's a learning agenda
09:25 Novelty effects, hidden heterogeneity, and the multiple comparisons problem
12:55 Pre-registration, kill criteria, and stopping p-hacking
15:10 Rolling out winners: gradual ramps and long-running holdouts
17:15 Causal inference when you can't A/B test
20:05 The case for more A/B testing, not less

Takeaways
- Run the four-question framework before any test: clean randomization, a plausible effect size for your traffic, a reversible and cheap change, and a falsifiable hypothesis.
- Treat A/B testing as a learning agenda: 85 to 90% of tests are supposed to fail, and a suspiciously high win rate is a red flag, not a trophy.
- Pre-register the full analysis plan, including hypothesis, mechanism, primary metric, exact statistical test, and subgroups, so p-hacking can't creep in when a test goes sideways.
- Define kill criteria and success, failure, and guardrail-dip actions before launch, with leadership sign-off, so nobody chases a loss into a fake win.
- Log every test and its learnings where the whole organization can see them; that reinforcement loop is what separates world-class experimentation programs.

Connect with the Guest
Priya Singhee LinkedIn: https://www.linkedin.com/in/priya-singhee/
About the guest: https://www.growthbook.io/podcast/guests/priya-singhee
Episode page: https://www.growthbook.io/podcast/episode/1-37
Company Website: https://www.earlywarning.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Summary
On this episode of The Experimentation Edge, Ashley Stirrup talks with Shan Huang, data scientist on the central experimentation team at Supercell, the Helsinki mobile game company behind Clash of Clans, Clash Royale, Brawl Stars, Hay Day, and Boom Beach. Shan explains how a famously decentralized, creative-first company with 300 million monthly active users runs fewer than 100 A/B tests a quarter and why that restraint is deliberate, how Supercell announces experiments to players in advance and promises make-up events to keep testing fair, and how importable AI skills now let anyone at the company analyze their own experiments, making quality consistency the next big challenge. It's for product managers, data scientists, and growth leaders balancing creative conviction with experimental rigor.

Chapters
00:00 Intro
01:10 About Supercell and 300 million players
02:30 The central team and a decentralized culture
04:25 Fewer than 100 tests a quarter
05:40 Sharing learnings across independent game teams
07:35 Retention as the North Star
08:35 Onboarding experiments with gems and tutorials
12:45 Telling players about A/B tests
15:15 Hypotheses and proxy metrics
20:45 AI and the future of experiment analysis

Takeaways
- Supercell runs fewer than 100 A/B tests a quarter for 300 million monthly players, because the goal is to become more hypothesis driven while staying creative, not to maximize volume.
- In a decentralized company, a central experimentation team earns its impact by providing the platform, partnering on rigor, and making sure learnings travel across independent game teams.
- Announce experiments to users before they run; Supercell's community managers tell players what is being tested and why, which turns a skeptical community into a research partner.
- Promise fairness, not just transparency; players who get the worse variant always receive a make-up event later, because game players come to have fun, not to be disadvantaged.
- AI-powered self-serve analysis means everyone can now run and analyze experiments, so the next challenge is making the quality of AI analysis consistent across the whole company.

Connect with the Guest
Shan Huang LinkedIn: https://www.linkedin.com/in/cnshanhuang/
About the guest: https://www.growthbook.io/podcast/guests/shan-huang
Episode page: https://www.growthbook.io/podcast/episode/1-36
Company Website: https://supercell.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Summary
How do you run an experimentation program when classic A/B testing is off the table? Ben Schein, Director of Product Management at Clover, joins host Ashley Stirrup to explain how a platform serving 300,000+ merchants and processing billions of dollars daily proves every feature through pilots and ground-level testing before rollout, why uncertainty and downside — not feature visibility — decide testing depth, and what his years leading product at Shake Shack taught him about turning a checkout funnel into a brand channel. This episode is for product managers, engineers, and data scientists building experimentation programs where the stakes are real.

Chapters
00:00 Cold open and welcome
01:30 The Clover business model and its scale
03:45 Ben's role and the metrics that matter
07:45 Deciding what gets tested: uncertainty and downside
09:05 Why Clover can't test in production
12:45 Testing the Shake Shack checkout experience
19:35 Advice for PMs new to experimentation
23:45 Context over personalization
25:45 The future of experimentation and the human element

Takeaways
- At Clover's scale, testing happens before rollout: pilots and detailed go-to-market plans replace in-production A/B tests, because a merchant's work tool can never change overnight without warning.
- Uncertainty and downside set the testing depth. High-risk changes like payment authorization flows earn deep, ground-level experimentation, while table stakes features like Apple Pay earn a monitored rollout.
- Testing is the evidence that justifies rollout investment: if the data doesn't show a feature will succeed, the go-to-market dollars never get spent.
- A checkout funnel can carry the brand. At Shake Shack, Ben's team tested prep-time expectations, fixed wrong-location orders, and used loading screens to deliver hospitality digitally.
- Structure experiments for durable business value, not pass-fail verdicts: pair headline metrics with counter metrics and anchor on ground-level measurements like items per check that resist marketing noise.

Connect with the Guest
Ben Schein LinkedIn: https://www.linkedin.com/in/benschein/
About the guest: https://www.growthbook.io/podcast/guests/ben-schein
Episode page: https://www.growthbook.io/podcast/episode/1-35
Company Website: https://www.clover.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Summary
What happens when the most logical feature you've ever built has zero impact? In this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with Edd Saunders, product experimentation manager at JobLeads, to unpack the pizza personalization experiment that cut an ordering flow from 22 clicks to 5 and changed nothing. Edd shares the problem mapping framework he uses to move new experimenters from solution space to problem space thinking, how JobLeads grew from 0.3 to 2.8 experiments per month, and why democratizing experimentation across a whole company comes down to habit change rather than education. A practical conversation for product managers, data scientists, engineers, and growth leaders building experimentation cultures.

Chapters
00:00 Introduction
01:40 Meet Edd Saunders and JobLeads
03:12 How JobLeads uses AI for prototyping
04:05 Building experimentation operations and a knowledge base
05:35 Learning over winning and compounding growth
08:10 The pizza personalization experiment
12:50 Moving from solution space to problem space
17:05 Problem mapping on a 2x2 matrix
22:20 Velocity and democratizing experimentation
24:20 AI automation for the unsexy work

Takeaways
- A one click reorder feature that cut a pizza ordering flow from 22 inputs to 5 had zero impact on purchases, proving that removing friction can also remove the customer's sense of control.
- Exploration is part of the customer's delight; returning customers wanted to browse the menu even though they ordered the same thing every week.
- Moving new experimenters from solution space to problem space thinking raises win rates and produces learnings the whole organization can use.
- Problem mapping on a 2x2 matrix of evidence versus impact turns customer research into a prioritized experiment roadmap, and one validated problem can spring a whole tree of testable ideas.
- Scaling experimentation from 0.3 to 2.8 tests per month is less about education and more about habit change, shared learnings, and giving non specialists the tools to launch their own experiments.

Connect with the Guest
Edd Saunders LinkedIn: https://www.linkedin.com/in/eddsaunders/
About the guest: https://www.growthbook.io/podcast/guests/edd-saunders
Episode page: https://www.growthbook.io/podcast/episode/1-34
Company Website: https://www.jobleads.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Summary
What does a winning A/B test mean when your website serves 1,100 dentist owned offices? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Arie Polycarpou, Senior Manager of Test and Learn at Aspen Dental, about building experimentation programs at Kohl's, Marriott, Total Wine, and now the largest company in The Aspen Group's healthcare retail portfolio. Arie explains why appointment bookings are the North Star but never the whole story, why he deliberately targets a 25% win rate and expects it to fall as the program matures, and why he applies a haircut to every stacked win before it reaches leadership. A practical conversation for product managers, engineers, data scientists, and growth leaders building test and learn programs at multi-location businesses.

Chapters
00:00 introduction and Arie's path into experimentation
02:05 building programs at Kohl's, Marriott, and Total Wine
03:15 healthcare retail and 1,100 dentist owned offices
04:30 the test and learn team and 100 tests a year
07:15 building a culture of shared wins and explained losses
09:50 what a losing navigation redesign revealed
12:30 testing your way into big redesigns
13:55 the case for a 25% win rate
15:50 haircuts, holdouts, and honest math on stacked wins
17:55 metrics beyond conversion and where AI fits next

Takeaways
- Aspen Dental runs experimentation as healthcare retail: with 1,100 dentist owned offices, the office, not just the website visitor, is the real unit of analysis.
- A mature program should target a true win rate around 25%; a 40% win rate usually signals a young program still picking off low-hanging fruit.
- Apply a haircut to stacked wins: lifts depreciate as customers acclimate, and two 5% wins never add up to 10%.
- Losing tests are valuable when they're designed to isolate why: test your way into big redesigns instead of shipping them whole.
- Experimentation culture grows from sharing wins and explaining losses: keep the statistical rigor in the back end and the communication simple.

Connect with the Guest
Arie Polycarpou LinkedIn: https://www.linkedin.com/in/ariepolycarpou/
About the guest: https://www.growthbook.io/podcast/guests/arie-polycarpou
Episode page: https://www.growthbook.io/podcast/episode/1-33
Company Website: https://www.aspendental.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io