The Experimentation Edge

Summary
Mary Alfheim, VP Data and Analytics at Headspace, joins host Ashley Stirrup to explain why more engagement is not always better when your members come to you with real symptoms. She walks through how Headspace experiments on Ebb, its AI companion, by defining inputs, outputs and guardrails for every test, with clinicians in the loop and stop rules that let teams self-serve safely.

Mary also shares two hard-earned lessons from earlier in her career: an 18 month redesign that taught her to test early and in small pieces, and a payment processor test where the randomization was not what it seemed. She closes with how Headspace is scaling experimentation and why every portfolio needs a learning bucket. For data, product and growth leaders experimenting on AI features and long term outcomes.

Chapters
00:00 Cold open
01:02 Headspace and Mary's path from Amazon and HBO
03:26 Why more engagement is not always better
09:25 Predicting long term outcomes with composite metrics
13:29 Testing Ebb: inputs, outputs and guardrails
19:28 A/B testing LLM features
22:35 Office hours and a culture of experimentation
28:46 The redesign that waited too long to test
32:03 Know your treatment: the payment processor test
38:11 Faster tests and the learning bucket

Takeaways
- At Headspace the target is health outcomes, not time in the app, so engagement metrics sit alongside self-reported measures of how members feel
- Every experiment on Ebb, Headspace's AI companion, is defined by inputs, outputs and guardrails, with clinicians doing clinical QA
- Stop rules written before launch let teams self-serve: lift with no guardrail harm ships, guardrail harm rolls back
- Test early and in discrete pieces; an 18 month redesign could not show which change was underperforming
- Keep a learning bucket of experiments that never have to report a win, alongside small loop tests and big bets

Connect with the Guest
Company Website: https://www.headspace.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Arv Ramchandani, Senior Product Manager for Experimentation & Analytics at Walmart Connect, runs ad experiments on pages owned by other teams, from search to the conversational AI shopping agents, each with its own primary metric. He explains how his team negotiates a degradation threshold with every page owner, why relevance and latency sit next to revenue as guardrails, and how months-long ad-free holdouts measure what ads cost in customer trust.

Host Ashley Stirrup also hears the story of a cross-sell test everyone expected to win: showing complementary items on searches like banana encouraged exploration, but that engagement did not consistently translate into the conversion outcomes the team had hoped for. That highlighted an opportunity to better align the experience with shopper intent, so the team rebuilt around the customer's mission. The rebuilt version delivered better CTR and better conversion, and the team submitted a white paper on the work to the AAAI 2027 Innovative Applications of AI conference for consideration. Arv closes with how to design experiments for learnings instead of launches and why self-serve experimentation is the next step beyond the analyst. For product, analytics and retail media teams who run experiments on shared pages.

Chapters
00:00 Intro
01:42 Balancing metrics across search, shopping agents and advertisers
03:08 Guardrails, degradation thresholds and long-term holdouts
05:58 The banana problem: rethinking a cross-sell idea
08:20 Rebuilding around the customer's mission
11:16 Designing experiments for learnings, not launches
13:48 Trustworthy experiments and where experimentation goes next
15:56 Wrap

Takeaways
- Every page owner has their own primary metric; ads teams negotiate a degradation threshold per page and stay inside it
- Revenue is not the only guardrail: relevance and latency are long-term metrics, and ad-free holdouts running for months measure what ads cost in trust
- Customers engaged with complementary recommendations, but that engagement did not always translate into purchases; the fix was reducing diversity and building themes around the customer's mission
- Start every experiment by asking how to make the product more usable, not how to launch it; the hypothesis changes and so do the metrics
- A written process (test plan, no early peeking, end-of-test checks) catches bugs early and builds trust; self-serve tooling is what lets PMs experiment without an analyst

Connect with the Guest
Arv Ramchandani LinkedIn: https://www.linkedin.com/in/arv-ramchandani
Company Website: https://www.walmartconnect.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Sapna Kashyap, CRO Lead at VidaXL, runs the experimentation program for an e-commerce company operating in more than 30 countries, where product, development and business teams all validate their decisions with an A/B test first. She explains the biggest winner of her career: B2B buyers were abandoning a registration page, a survey showed they could not see what the subscription included, and adding that information to the signup flow lifted account creation 38.7%.

Host Ashley Stirrup also hears the story of a localized payment method the whole company expected to win, why it lost to PayPal and cards, and how the team used Stripe data and other sources of truth to prove it. Sapna closes with how VidaXL designs experiments, from guardrail metrics and device segmentation to the 70% of the work that happens before launch. For CRO, product and growth teams building an experimentation program that the business trusts.

Chapters
00:00 Intro
01:35 VidaXL: 30 countries, every team tests
02:28 Sapna's role and the maturity program
04:39 The 38.7% B2B signup winner
08:42 Designing experiments: research, guardrails, device segments
12:07 The localized payment method that lost
15:58 Seven years of maturing
16:53 70% of the work is before launch
18:33 Wrap

Takeaways
- Friction is not always bad: buyers dropped off a registration page because they could not see the subscription details, and adding them lifted account creation 38.7%
- Localization is not a guaranteed win; a country-specific payment method lost to PayPal and cards, and only cross-verified data got the team to accept it
- Guardrail metrics (return rate, latency) are decided with the stakeholder before launch, not after
- Segment by device and traffic source; a change that works on desktop is not shipped on mobile if it did not win there
- About 70% of an experiment's work is done before it goes live: research, clear objectives, data quality, then the analysis is the small part

Connect with the Guest
Sapna leads and manages roadmaps for 155+ growth initiatives and experiments a year, has helped build and scale an experimentation culture across the organization, and has driven CRO and experimentation across both B2B and ecommerce businesses.
Sapna Kashyap LinkedIn: https://www.linkedin.com/in/sapnakashyapx/
Sapna Kashyap Website: https://sapnakashyap.com/

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Michael Mathews, Director of Data and AI Product at FanDuel, runs the teams behind the auto researcher, an agent that takes a data set, an objective function and the production model, then proposes, writes and scores about 300 strategies in an hour for roughly $300 in tokens, handing back the winner as readable code. Host Ashley Stirrup gets the three things that surprised FanDuel as it scaled: the agent builds its own features (756 down to 20), the holdout is enforced so it cannot cheat, and it reframes problems when the scoring rules tell it to.

The conversation covers the churn run that lifted F1 by 6% and is worth about a million dollars over baseline, the two popular parlays tests that became penalty weights and floors in the next experiment, North Star versus OEC in a luck-contaminated business, responsible gaming as a guardrail a machine has to be taught, and Michael's six-level ladder of experiment autonomy, including the shadow mode rung where FanDuel buys trust with six months of evidence. For product, data and growth leaders deciding how much of an experiment a machine should be allowed to run.

Chapters
00:00 Cold open
01:02 FanDuel, prediction markets and the data and AI product team
03:23 The auto researcher: 300 strategies, an hour, $300 in tokens
05:25 Three surprises: it builds features, enforces the holdout, reframes the problem
07:50 Two popular parlays tests and the penalty weight they became
12:38 What humans still own: the hypothesis and the goal
16:00 North Star vs OEC and luck-adjusted margin
20:18 The churn run: 750 features to 20 and a million dollars over baseline
24:18 The six levels of experiment autonomy and shadow mode
36:32 Always measure baseline, and when to stop iterating

Takeaways
- FanDuel's auto researcher reads an objective function, proposes a strategy, writes the model, scores it and repeats about 300 times in an hour for around $300, returning the winner as readable code.
- The agent builds its own features (756 candidates to 20), never sees the validation set, and will reframe a problem when the scoring rules make a wrong guess cost more than a right one earns.
- One churn run cut 750 features to 20, lifted F1 by 6% and tested live across three arms of 47,000 users, worth roughly a million dollars over baseline.
- Losing tests become coded constraints: two losing popular parlays tests turned into a 2.5x penalty weight and a hard floor in the next experiment's envelope.
- Autonomy is earned in shadow mode, where the machine's complete package is recorded against every human decision until six months of evidence settles the trust question.

Connect with the Guest
Michael Mathews LinkedIn: https://www.linkedin.com/in/michael-mathews-jr-0b194224
Company Website: https://www.fanduel.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Grubhub does not run much classic A/B testing. Michal Lenik, Associate Director of Product Design at Grubhub, explains what the company does instead to make sure its big product bets win before they reach millions of users.

She joins Ashley Stirrup to walk through Grubhub's playbook: small cohort rollouts that protect core order metrics, merchant round tables, AI built clickable prototypes, and a concept test in the merchant promotions space that sent the team toward a much simpler dashboard than their research predicted. Michal also explains why she follows losing experiments with user interviews, why "I didn't even notice it" is valuable feedback, and why every company should decide deliberately whether it de-risks upstream or in a fast follow after launch.

Chapters
00:00 Where design is heading in the AI era
00:53 Meet Michal Lenik of Grubhub
02:21 Design as an upstream business lever
03:48 Small cohorts instead of classic A/B tests
05:27 The merchant concept test that changed direction
08:18 Prototyping with AI and merchant round tables
10:16 Success metrics for product, design and engineering
11:56 Pairing experiment data with user interviews
14:26 AI, personalization and the designer's new role
17:45 De-risking upstream versus shipping fast

Takeaways
- Grubhub tests big bets by releasing to internal users or small cohorts first, so it can read business impact without disrupting core revenue and order metrics.
- A concept test with merchants overturned what fifteen research interviews suggested, steering Grubhub to a simpler dashboard and away from a launch that likely would not have won.
- AI lets Grubhub's design teams put clickable prototypes in front of users early, turning each iteration into a micro experiment.
- Pairing quantitative results with five to ten user interviews after a test explains why it underperformed, including whether users even saw the change.
- Both upstream de-risking and fast iteration work. The right choice depends on how quickly engineering can ship, and it should be made on purpose.

Connect with the Guest
Michal Lenik LinkedIn: https://www.linkedin.com/in/michallenik/
About the guest: https://www.growthbook.io/podcast/guests/michal-lenik
Episode page: https://www.growthbook.io/podcast/episode/1-48
Company Website: https://www.grubhub.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

• (00:00) - Where design is heading in the AI era

Summary
Dorothy Crepin runs the A/B testing program at MilliporeSigma, the life sciences business of Merck KGaA, where one global catalog has to serve three very different visitors: scientists hunting for a specific instrument, procurement buyers working off a list someone else wrote, and students doing research with no purchasing power at all. Add a regulated industry where what you can sell changes country by country, and personas stop being a marketing exercise.

She walks Ashley Stirrup through the test that changed how her team thinks about discovery. MilliporeSigma rebuilt its on-site type ahead so it suggests a search term instead of pushing a matching product, and saw double digit increases in feature usage and add to cart. From there Dorothy makes the case for measuring the action a page is actually responsible for rather than revenue alone, for pairing A/B tests with a user testing ambassador panel to get at the why, and for sending product managers back to one question before anything gets built: what problem are you solving for the customer?

Chapters
00:00 Revenue is an outcome, not the metric
01:26 Welcome and what MilliporeSigma does
02:22 The three audiences on a B2B science site
03:53 What an A/B testing analyst coordinates
04:20 Running experiments across product pods
05:31 Building leadership buy in for testing
08:08 Where A/B testing ends and user testing begins
08:52 The type ahead search test that lifted add to cart
11:30 How to frame a test with a product manager
13:34 Why testing the opposite pays off
15:26 Personas, personalization and what AI changes
18:57 Building trust across competing teams

Takeaways
- MilliporeSigma changed its on site search to suggest search terms instead of products and saw double digit gains in feature usage and add to cart.
- Revenue is an outcome, so Dorothy's team measures the next action each page is responsible for, like add to cart on a product detail page.
- A/B testing tells you what happened. MilliporeSigma's user testing ambassador panel supplies the why behind the click.
- Every test starts with one question to the product manager: what problem are you solving for the customer, not what is your hypothesis.
- Dorothy is building personas into agents so browsers, not just buyers, can be tested for a directional read before an experiment ships.

Connect with the Guest
Dorothy Crepin LinkedIn: https://www.linkedin.com/in/dorothycrepin/
About the guest: https://www.growthbook.io/podcast/guests/dorothy-crepin
Episode page: https://www.growthbook.io/podcast/episode/1-47
Company Website: https://www.sigmaaldrich.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

Summary
Gaurav Sethi joins Ashley Stirrup on The Experimentation Edge to explain how PayPal's experimentation program went from 700 to 800 tests a year with four week readouts to 2,587 experiments a year and roughly $180 million in measured impact. Gaurav inherited a reported win rate of 55% to 60%, four times the industry average, and traced it to experiments logging assignment data instead of exposure data, carrying 25% to 30% dilution. The conversation covers the exposure event his team introduced, the instrumentation and metric standards that cut readouts to 24 hours, a carousel test that a multi armed bandit resolved in 51 days instead of a projected 700, and why cost avoidance from losing experiments belongs in the ROI number. It closes on what changes when the thing you are testing is an AI agent rather than a button. Useful for product managers, engineers, data scientists and growth leaders building or defending an experimentation program at scale.

Chapters
00:00 Cold open
00:54 Welcome Gaurav Sethi of PayPal
01:30 Elmo, PayPal's homegrown experimentation platform
03:53 $180 million in revenue impact and cost avoidance
04:28 The win rate that was too good to be true
06:29 Instrumentation standards and the exposure event
08:56 The carousel test: 700 days down to 51
11:29 Designing experiments so every result teaches you
13:45 Building the platform is only half the job
15:07 Education, office hours and executive support
18:28 Exposure events and joining transactional data
24:14 AI for experimentation, and experimentation for AI

Takeaways
- A win rate of 55% to 60% against an industry average of 11% to 14% was a tracking problem, not a performance one: assignment data carried 25% to 30% dilution and reached significance on users who never saw the test.
- The exposure event fixed it. Fire an event as close to render as possible so the platform knows exactly when a user entered the experiment, instead of logging on page load.
- Standardized instrumentation and canonical metric definitions are what let the analysis pipeline run without human cleanup, taking experiment readouts from four weeks to 24 hours.
- A six variant carousel test projected at 700 days was resolved by a multi armed bandit in 51 days, with a winner worth almost five basis points.
- About half of PayPal's $180 million impact in 2025 was cost avoidance from features that tested badly and never shipped, which is why Gaurav argues there are no losing experiments.

Connect with the Guest
Gaurav Sethi LinkedIn: https://www.linkedin.com/in/gauravsethi22
About the guest: https://www.growthbook.io/podcast/guests/gaurav-sethi
Episode page: https://www.growthbook.io/podcast/episode/1-46
Company Website: https://www.paypal.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

Summary
ServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer.

She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline lesson is not a technical one. ServiceNow built agentic skills that worked, and users abandoned them anyway, because a task that ran five to fifteen minutes broke every expectation people had about what a machine should feel like. The fix was not a faster model. It was telling the user what the AI was doing while it did it.

Ashraf also gets into the harder measurement problem underneath all of it: how do you evaluate a nondeterministic system? ServiceNow's answer is an internal LLM judge scored against a golden data set, plus a customer effort score that catches what CSAT reports too late.

Chapters
00:00 What customers actually care about
01:14 Meet Ashraf Karim of ServiceNow
01:39 Owning the entire post-sale experience
02:08 Why ServiceNow runs as its own Customer Zero
03:02 Lessons from Google, PayPal and Verizon
07:11 The agentic skills users kept abandoning
09:29 Latency, expectations and the feedback loop fix
12:29 Users wanted the outcome, not the process
14:49 Designing experiments that can afford to lose
18:13 Using an LLM judge on nondeterministic models
21:48 Customer effort score as a guardrail metric
25:03 Where experimentation at ServiceNow goes next

Takeaways
- ServiceNow is its own Customer Zero, running the platform on itself and testing every feature internally before customers see it.
- The agentic skills worked. Users still abandoned them, because a five to fifteen minute wait broke their expectation of what AI should feel like.
- The fix was a feedback loop that narrates what the AI is doing in the background, which buys the patience a long task needs.
- Users did not want the agentified version of the human process. They wanted the outcome and the next best action, not the steps.
- Nondeterministic output needs its own evaluation layer, so ServiceNow built an LLM judge that scores against a golden data set before anything reaches a customer.

Connect with the Guest
Ashraf Karim LinkedIn: Ashraf Karim on LinkedIn
About the guest: https://www.growthbook.io/podcast/guests/ashraf-karim
Episode page: https://www.growthbook.io/podcast/episode/1-45
Company Website: ServiceNow

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

Summary
Wanli Lau spent eleven years at Expedia Group, most recently running consumer product analytics on a program that shipped more than 1,500 A/B tests a year. Five months ago she moved to Twilio to lead global R&D analytics. In this episode she walks Ashley Stirrup through what carried over and what did not.

The centerpiece is a test that looked like a win. At Expedia, an account sign-up takeover on the front page drove conversion sharply up, and quietly harmed return rate and engagement across every geography. That result is where Wanli's insistence on guardrail metrics and a trade-off decision framework comes from. She also explains the parts of B2B experimentation that have no B2C equivalent: choosing between user-level and account-level randomization, the interference risk when two colleagues at the same company see different pricing, and the many-to-many problem of one developer belonging to several accounts. And because Twilio's customer counts are lower than a consumer business, her teams often ship on signals and team conviction rather than waiting for statistical significance.

Chapters
00:00 Cold open
01:02 Welcome Wanli Lau of Twilio
01:23 Inside Wanli's role at Twilio
04:03 Eleven years and 1,500 tests a year at Expedia
04:58 Why B2B experimentation differs from B2C
05:33 Cross-functional collaboration and good hypotheses
06:39 Teaching teams to experiment rigorously
07:48 The Expedia sign-up takeover that won on conversion
11:09 Guardrail metrics and the trade-off framework
14:52 Designing experiments to maximize learning
17:46 Diagnosing where a feature failed
19:10 North Star metrics and account-level randomization at Twilio
21:40 Learning from signals instead of significance
22:11 Where experimentation goes next at Twilio

Takeaways
- A test that wins on the primary metric can still be a loss. Expedia's sign-up takeover raised conversion and damaged return rate and engagement at the same time.
- Name the primary, secondary and guardrail metrics before the test runs, not at readout. Deciding afterwards turns a result into a debate.
- Conversion should be a do-no-harm guardrail for teams that do not own it, even when it is not their primary metric.
- In B2B the randomization unit is a design decision. User-level bucketing risks two colleagues at one account seeing different prices, and account-level avoids that but costs sample size.
- When customer counts are low, significance is often out of reach. Learning from signals and shipping on team conviction beats waiting for a number that will never arrive.

Connect with the Guest
Wanli Lau LinkedIn: https://www.linkedin.com/in/wanlilau
About the guest: https://www.growthbook.io/podcast/guests/wanli-lau
Episode page: https://www.growthbook.io/podcast/episode/1-44
Company Website: https://www.twilio.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Will Guyeskey is Director of Digital Product at GoPro, where his team owns the e-commerce side of gopro.com. Before GoPro he ran personalization at Gap and cut his teeth at Brooks Bell, testing for brands like Barnes & Noble, Under Armour and Ralph Lauren. In this episode of The Experimentation Edge, he tells Ashley Stirrup why knowing your customer is the through line of every good test program.

Will shares the Barnes & Noble order confirmation test he was sure would lose, and why the same idea never worked for any other client. He walks through a recent GoPro Mission launch test that asked whether a step-by-step configurator adds too much friction, and what a flat result revealed about high consideration buyers. He also explains why GoPro shares interim readouts across the company, why win rate makes a poor North Star for an experimentation program, and how his team plans to use AI for speed without outrunning its own learnings.

Chapters
00:00 Intro
00:52 Will's role running e-commerce at GoPro
01:43 Learning A/B testing across retail at Brooks Bell
05:24 How GoPro runs one to three tests a month
06:35 Sharing learnings and interim readouts across teams
09:20 The Barnes & Noble order confirmation win
13:02 Why the win did not transfer to other clients
14:41 Testing friction on the GoPro Mission configurator
18:51 Why win rate is the wrong North Star
22:19 How AI will shape experimentation at GoPro

Takeaways
- A winning idea rarely travels. The Barnes & Noble recommendation module worked because of that audience's low order values and reading habits, and it failed for every other client that tried it.
- Design every test so it teaches you something whether it wins, loses or ends flat. Losing tests are jet fuel when the learning is built in.
- A flat result is still an answer. GoPro's configurator test showed that buyers of high consideration products accept extra steps when each choice adds value.
- Share interim readouts across the company, and use them to show how volatile results are before a test reaches statistical significance.
- AI can speed up building and running experiments, but a team that runs more tests than it can learn from is not getting better.

Connect with the Guest
Will Guyeskey LinkedIn: https://www.linkedin.com/in/willguyeskey/
About the guest: https://www.growthbook.io/podcast/guests/will-guyeskey
Episode page: https://www.growthbook.io/podcast/episode/1-43
Company Website: https://gopro.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io