The Experimentation Edge

Summary
Delina Ivanova, Director of Data Science at Wealthsimple, joins host Ashley Stirrup to share how Canada's fast-growing financial services company decides what deserves an experiment and what should simply ship. With 4 million clients and a steady stream of new brokerage and banking features, Wealthsimple has spent the last 2 years formalizing experimentation across 3 data science teams.

Delina explains her rules of thumb for skipping a test, why lagging metrics like deposits and revenue force teams to pick the right early signals, and what a margin promotion taught her team about messaging, targeting and combined offers. She also shares how to diagnose a losing test, her plan for meta-learning and uplift modeling, and how AI makes it cheaper to test ideas before investing in a full build. For product, data and growth leaders who want experiments that measure the thing they are trying to drive.

Chapters
00:00 Wealthsimple, 4 million clients and a never-ending experimentation world
02:46 Two years of formalizing experimentation
03:42 What Misplay taught her about when you need a test
06:00 Rules of thumb: when to test and when to just ship
08:21 Ronny Kohavi, Bing and the value of a millisecond
09:26 How Wealthsimple's 3 data science teams run experiments
12:15 Engagement features, lagging metrics and early signals
14:49 Launching margin and finding an education gap
16:57 Why surveys need experiments
19:10 Sharing results and building systemic learning
21:56 The margin promotion that taught a messaging lesson
26:23 Experiments as a way to learn about people
27:39 Messaging, targeting or experience: diagnosing a losing test
30:35 Meta-learning and uplift modeling
33:42 Using AI to test ideas faster and cheaper

Takeaways
- Skip the test when you are removing a clear blocker for clients, and test when the change is optional or the outcome is uncertain
- Deposits and revenue can lag 60 to 90 days in financial services, so pick early metrics that are proven to lead to them
- Combining 2 incentives in one message hides what you are trying to learn; decouple offers so each test isolates one outcome
- When a feature you believe in loses, check whether it is a messaging, targeting or experience problem, and design variants to find out
- Pooling every past experiment into one view turns tests into a mega learning system for pricing, messaging and personalization

Connect with the Guest
Delina Ivanova LinkedIn: https://www.linkedin.com/in/delina-ivanova/
Company Website: https://www.wealthsimple.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Andy Denning, Product Leader, joins host Ashley Stirrup to talk about the judgment that comes before any A/B test. Drawing on his experience with a two-sided jobs marketplace serving job seekers and recruiters across the UK and US, Andy explains why underpowered, badly chosen tests can create false confidence, how experimentation brings organizational honesty, and why his teams start with the riskiest assumption instead of the most comfortable one. He breaks down a US branding test that barely moved applications, how traffic and market shape test design, and why AI makes judgment, not test volume, the real bottleneck. For product and growth leaders who want fewer wasted tests and more learning from the ones they run.

Chapters
00:00 Test the riskiest assumption, not the comfortable one
00:55 A two-sided jobs marketplace in the UK and US
01:53 Andy's product role and AI-powered job matching
02:52 Deciding what's worth testing at a smaller company
04:21 Underpowered tests and false confidence
04:45 Experimentation as organizational honesty
05:49 From prototyping software to Optimizely
06:58 Cross-functional teams that own experiments end to end
08:13 Sharing learnings and popcorn sessions
10:31 Guardrail metrics in regulated industries
11:53 Experimenting across the UK and US markets
13:38 The branding test that didn't move applications
17:04 The judgment layer: is the test worth running?
19:03 Experiment design starts with the riskiest assumption
20:39 What to do when a test will take 6 weeks
21:27 B2C vs B2B: what your traffic can support
22:27 Pre-mortems, non-goals and trade-offs
23:49 AI and the flood of low-conviction tests
27:36 Where to spend human judgment
28:26 The AI apply tsunami in hiring

Takeaways
- Test the riskiest assumption, not the most comfortable one; it is the one the bet rests on
- Underpowered or badly chosen tests can manufacture false confidence that is worse than not testing
- Before you test, ask whether the outcome is uncertain and whether you would ship it anyway
- Run the numbers up front; if a test needs 6 weeks, find a faster or cheaper way to learn
- AI makes variants cheap, so the bottleneck is now the quality of the hypothesis

Connect with the Guest
Andy Denning LinkedIn: https://www.linkedin.com/in/andy-denning-547a0862/

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Tom Gores, Director, Product - Digital Experience & E-commerce at GE HealthCare, joins host Ashley Stirrup to share what changes when you take an experimentation program from B2C retail to B2B healthcare. Tom built his first A/B testing program at Kohl's 12 years ago and now runs a centralized program at the center of GE HealthCare's digital ecosystem, where buying cycles can take a year and each qualified lead carries much higher stakes.

Tom explains why a Contact Us test that lifted clicks also lifted junk leads, how his team turns one-off page ideas into bigger, broader tests across templates, and why planning for failure before launch keeps experimentation framed as learning. He also covers the 3 areas where he sees experimentation growing, people and culture, data, and technology, and the metrics he watches, from CSAT and conversion rate to visibility in AI search. For product, digital and growth leaders running experiments in long, high-stakes B2B journeys.

Chapters
00:00 Welcome Tom Gores of GE HealthCare
00:51 From Kohl's to GE HealthCare: 12 years of A/B testing
02:01 GE HealthCare's digital ecosystem
02:51 B2B vs B2C: longer buying cycles and the long game
03:49 A centralized program with lower traffic and higher stakes
05:52 The Contact Us test: more clicks, junkier leads
07:26 Funneling the right buyers to the right experience
08:47 Designing experiments for the most learning
10:33 Planning for failure and picking leading indicators
13:07 Where experimentation grows next: people, data and technology
15:22 CSAT, conversion rate and AI visibility
17:34 How GE HealthCare measures CSAT

Takeaways
- In B2B healthcare, buying cycles can run months to a year, so experiments need leading indicators and long-term tracking rather than instant conversions
- Pushing a metric in front of people can lift clicks while lowering lead quality; check results with the sales teams who work the leads
- Turn a single-page idea into a bigger, broader test by applying it to a whole template, so thousands of visitors are affected instead of hundreds
- Map out what you will do if a test fails before launch, so results stay framed as learning and avoid wasted builds
- Track CSAT, conversion rate and AI visibility together to measure an ecosystem that serves many audiences and journey stages

Connect with the Guest
Tom Gores LinkedIn: https://www.linkedin.com/in/tom-gores-6925a81a
Company Website - GE HealthCare: https://www.gehealthcare.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Jeremy Echols, Director of Digital Analytics at Glassdoor, part of Indeed, joins host Ashley Stirrup to talk about building an experimentation program around people, not sessions. Jeremy came to analytics from an unusual path, as a former pastor with a master of divinity, and he sees math and statistics as a way to do applied psychology.

Jeremy explains how Glassdoor went from a slow homegrown system and a handful of tests a year to dozens of tests a quarter, with a center of excellence, office hours, Amplitude templates and a shared data language. He breaks down an Indeed account-linking test that exposed variant jumpers across devices, why every test needs a hypothesis and a curious set of secondary metrics, and how some friction is worth keeping. For analytics, product and growth leaders who want faster iterations and deeper learning from every test.

Chapters
00:00 Sessions don't buy anything, people do
00:42 Glassdoor's mission: workplace transparency
01:27 Glassdoor Recruiter: a recruiter that works for the job seeker
02:50 From pastor to analytics: statistics as applied psychology
03:46 Why synthetic users can't replace real people
04:14 Centralizing experimentation at Glassdoor
05:24 Templates, guardrails and an experimentation community
06:49 The account-linking test that exposed variant jumpers
08:10 Why users browse without logging in
09:48 Guest checkout hides the customer journey
11:01 Costco and the case for personalization
11:40 What to learn when a new feature fails
12:11 Write the hypothesis, then ask what else might happen
14:00 Tying experiments to your analytics funnels
14:51 Pre-mortems and testing the opposite
15:56 When removing friction hurts the funnel
17:04 Turning experimentation marathons into sprints
18:19 One data language for every team
19:02 Glassdoor Recruiter and SuperMatch
20:35 Hiring managers and passive candidates

Takeaways
- Measure people, not sessions; users move across devices, so incentivize login to see the whole journey
- Bake best-practice settings and guardrail metrics into test templates so no one has to remember them
- Write a hypothesis for every test, then add the secondary metrics that could teach you something unexpected
- Some friction is good; more leads can mean fewer qualified leads and a weaker downstream funnel
- One shared data language across marketing, email and product makes experiments faster and easier to compare

Connect with the Guest
Jeremy Echols LinkedIn: https://www.linkedin.com/in/jeremyechols/
Company Website: https://www.glassdoor.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Andrew Miller, Director of Product Management, Growth and E-commerce at RealPage, joins host Ashley Stirrup to share how experimentation works inside Loft, RealPage's end-to-end renter app. Andrew has stood up experimentation programs at 3 companies, and today he runs a portfolio of moving services where a winning A/B test ties directly to revenue.

Andrew explains how a small, centralized optimization team ships tests in Optimizely faster than engineering release cycles allow, and how he builds a culture of experimentation with funnels, group ideation, the Opportunity Solution Tree and simple test templates. He also breaks down 2 electricity experiments: one that cut choices from 8 to 3 and lifted conversion 12%, and a price display test that lost. For product and growth leaders who want more ideas, faster launches and tests that actually teach them something.

Chapters
00:00 RealPage, Loft and a B2B2C business
01:17 Why every A/B test in Loft focuses on the renter
03:02 The moving marketplace and how it makes money
04:53 Standing up experimentation programs at 3 companies
07:10 The T-shaped PM: depth in data and experimentation
08:57 A centralized optimization team for all of Loft
11:05 Why tests run in Optimizely instead of engineering
12:42 Building a culture of experimentation with funnels
15:34 The Opportunity Solution Tree and test templates
17:06 How good is your gut at predicting winners?
19:01 Cutting electricity options from 8 to 3 for a 12% lift
21:13 The price display test that lost
24:26 Minimizing variables when traffic is limited
26:04 Why diversity of ideas matters
27:38 PMs submitting PRs, not just hypotheses
30:45 Prototyping tests with AI: from 2 weeks to 1 day

Takeaways
- Start with telemetry and funnels; funnel drop-offs get stakeholders to the insight before you have to say it
- Run the first ideation sessions as a group and organize ideas with an Opportunity Solution Tree; one session can fill a year of backlog
- Require a hypothesis, a metric and a definition of success for every test idea so learning is the baseline outcome
- Fewer choices can lift conversion: cutting electricity plans from 8 to 3 raised conversion 12%
- An estimate shown with too much precision can backfire; round estimated prices so they read as estimates

Connect with the Guest
Andrew Miller LinkedIn: https://www.linkedin.com/in/findandrew/
Company Website: https://www.realpage.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
In this special episode of The Experimentation Edge, host Ashley Stirrup looks back at lessons from more than 50 guests with two people from GrowthBook: co-founder and CEO Graham McNicoll and Head of Customer Success & Strategy Cris Dizon, who ran experimentation programs at The Home Depot, Walmart and Peacock. The starting point is a humbling number: across companies, only about 25% of experiments win, so even your best ideas are wrong most of the time.

Graham shares the untested redesign that quietly cut new-user sign-ups by about 20% for six months, and the bug that set a paywall to zero and overturned years of tribal knowledge. Cris explains how a streaming onboarding flow that failed came back and won once personalization caught up, and why heavy promo codes train customers to wait. Along the way they revisit stories from UPS, JPMorgan Chase, Fanatics, Chess.com, Groupon, Signet Jewelers, Box, Disney, Twitch, Learneo and Charlie Health, and close with five ways executives can build a culture that measures and learns from being wrong.

Chapters
00:00 Welcome to a special episode with Graham McNicoll and Cris Dizon
01:24 Why only about 25% of experiments win
03:28 Building a culture where it's okay to fail
04:31 From Fyxer to UPS: guests who admit their gut is wrong
05:05 The untested redesign that cost millions
06:17 A big-bang homepage redesign that took months to fix
07:48 Spot the million-dollar change in the JPMorgan Chase funnel
08:48 Twitch pricing and the risk of anti-knowledge
09:28 The bug that set the paywall to zero
11:05 The onboarding test that failed, then won
12:59 Capturing learnings so they don't get lost
13:54 UPS, JPMorgan Chase and Fanatics: the size of the wins
14:18 Reframing wins and losses as gains and savings
15:45 Measure your learning rate, not your win rate
16:49 Chess.com: review your wins, not your losses
18:09 Groupon swaps flash sales for scarcity
18:52 How promo codes train customers to wait
20:18 Signet Jewelers and friction that helps buyers
21:11 Box and the wine effect
22:34 The checkbox with more text that lifted conversions
23:28 Test the opposite of your hypothesis
24:15 Disney's painted door test for video support
25:55 Five ways executives can build an experimentation culture
27:58 Could we have learned this faster?
28:32 Share the design and let people pick the winner
29:35 Buddy programs and pre-mortems
30:27 You're already wrong 75% of the time

Takeaways
- Expect to be wrong: even mature programs see only about 1 in 4 experiments win
- Never ship a big-bang redesign untested; a hidden loss can run for months
- Track savings from losses alongside gains from wins
- Measure your learning rate, not your win rate
- Revisit old losers when your product or capabilities change

Connect with the Guests
Graham McNicoll LinkedIn: https://www.linkedin.com/in/grahammcnicoll/
Company Website: https://www.growthbook.io

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Sebastien Polis, Senior Experimentation Manager at Preply, joins host Ashley Stirrup to explain how a learning marketplace with 25 to 30 product teams runs close to 1,000 experiments a year. At Preply, no change reaches customers without a test, OKRs can only be claimed through experiments with an approved target metric, and a public Slack channel puts every launch, result and kill decision in front of the exec team.

Sebastien shares why roughly 80% of experiments lose and 30 to 40% are iterations of an earlier idea, how a global control group shows double-digit growth, and what Preply learned when a control group went negative and a simple wording win turned out to be Twyman's law in action. He also covers why marketplaces randomize tutors instead of learners, the clustered analysis that requires, and his plan to build a learning system that gives AI the context it needs. For product, data and growth leaders building an experimentation culture at scale.

Chapters
00:00 Preply's learning marketplace and Sebastien's path into experimentation
04:52 Tying OKRs to experiments across 25 to 30 product teams
06:39 Why 80% of experiments lose and 30 to 40% are iterations
08:35 Double-digit growth from a global control group
09:46 The control group that went negative and Twyman's law
15:03 How a new PM designs an experiment at Preply
17:57 A public Slack channel where execs question every test
20:48 When clicks go up but learners don't buy
23:23 Building a learning system before feeding AI
25:20 An experiment run entirely by AI, with a human approver
28:13 Randomizing tutors, not learners, in a 2-sided marketplace

Takeaways
- Preply ties OKRs to experimentation: teams can only claim results through experiments that use an approved target metric
- Losing is the norm; about 80% of Preply's tests fail and 30 to 40% of each year's experiments are iterations of an earlier idea
- A surprising win from a simple change deserves a rerun; Preply unscaled a wording test and the lift disappeared
- A public Slack channel where execs question PMs about launches and kills keeps rigor and transparency high across 1,000 tests a year
- In a marketplace, randomize the tutor and measure learner outcomes with a clustered analysis, and ship bug-free code because sample size is scarce

Connect with the Guest
Sebastien Polis LinkedIn: https://www.linkedin.com/in/sebastien-polis-b5aa8256/
Company Website: https://preply.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Vamsi Krishna Kunapuli, Director of Analytics at New York Red Bulls, leads a small team that supports marketing, ticket sales and partnerships for one of the original MLS clubs and its 25,000 seat stadium. He shares how an experiment across five to six seasons changed the club's North Star from average ticket price to gross margin.

Host Ashley Stirrup and Vamsi also discuss why a losing test is not a failure, why segments can never be static, where good test ideas come from, and how identity resolution will open the door to machine learning and personalized fan experiences. For analytics, marketing and product leaders building experimentation into a business where every event is different.

Chapters
00:00 Intro
00:20 New York Red Bulls: an original MLS club
01:32 Data-driven fan engagement 
04:46 Single game buyers outspend season ticket holders
07:21 Testing frameworks 
09:29 Holistic fan value 
11:11 Segments can never be static
12:59 A losing test isn't a failure
14:51 Loser or lever: judging tests against the North Star
17:13 Building a culture with clear end goals
20:25 Ideas populate when there is more data
23:10 Identity resolution, AI and personalization
26:06 Empowering more teams to experiment
27:57 Every game is like hosting a wedding

Takeaways
- We have seen instances where single game buyers at the New York Red Bulls outspend season ticket holders once concessions and the Hype House zone are counted
- The club moved its North Star from average ticket price to gross margin, which changed couponing, fan profiles and retargeting
- A losing test is not a failure; judged against the North Star, a negative result shows what not to do and that a lever exists
- Segments should be dynamic, and the club is building toward a separate segmentation for every event
- Ideas populate when there is more data, and the stakeholders who know the business best should drive the questions

Connect with the Guest
Vamsi Krishna Kunapuli LinkedIn: https://www.linkedin.com/in/vamsikunapuli/
Company Website: https://www.newyorkredbulls.com
Vamsi is a panelist at DMWF North America 2026, "Connected Data: The Foundation of Intelligent Marketing"

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Summary
Shiva Manjunath, Lead Web Strategist at Motive and host of the From A to B podcast, joins Ashley Stirrup to explain why CRO works best as a mindset every team carries rather than a role one team owns. He shares how building bridges to sales and customer service turns everyday complaints into test ideas.

The conversation covers why customers who ask for price are often asking for value, how emotional context earns a CRO credibility, how to make an impact without an executive sponsor, and why experimentation needs the kind of marketing AI gets. For CROs, product managers and growth leaders building an experimentation culture.

Chapters
00:00 Cold open: gifts under $50
01:18 What a true culture of experimentation looks like
04:43 CRO as a mindset across every function
11:27 Become the sales team's best advocate
15:29 Fear, uncertainty and doubt: the window shade story
21:51 Customers ask for price when they want value
24:17 GTA 6 and why button colors cost CROs trust
29:14 Making an impact without an executive sponsor
34:27 Making experimentation as exciting as AI

Takeaways
- CRO works best as a mindset shared by design, marketing and product, not as one team's job.
- Sales and customer service teams hold free qualitative data that makes strong test ideas.
- When customers ask for price, test value messaging such as benefits, ROI and time saved.
- Understanding the emotional context around a purchase earns a CRO more trust than tactical button tests.
- A culture of experimentation needs data-driven decision making first and a safe space to lose.

Connect with the Guest
Shiva Manjunath LinkedIn: https://www.linkedin.com/in/shiva-manjunath/
From A to B podcast: https://open.spotify.com/show/5xDlubE3GZwUDDen64hefz
Newsletter: https://fromatob.beehiiv.com/

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

Summary
Vipul Tyagi, Senior Product Manager at Quill, owns the online experience from add to cart through checkout, with a focus on getting customers to use Quill's loyalty program. Host Ashley Stirrup digs into why a homepage rewards widget drew under 0.5% engagement over 60 days, and how moving the value to the cart with a Lego-inspired points to dollars option lifted conversion by 1% while shrinking the points liability on Quill's books.

The conversation covers Vipul's 8 winners out of 11 loyalty initiatives and the BillionToOne advice behind them, how he coaches new PMs on primary and secondary hypotheses and guardrail metrics, a live test on showing the $5.99 shipping saving at checkout, a semantic layer experiment built for AI crawlers, and how AI has cut his research time from two months to seven days. For product managers, growth leaders and e-commerce teams who want a higher win rate from their experiments.

Chapters
00:00 Cold open
01:06 Quill, office supplies and three kinds of customers
04:20 Owning add to cart through checkout, and 8 winners out of 11
05:32 Know the direction before you run the test
06:54 The homepage loyalty widget and the icing on the cake
10:19 Points into dollars on the cart page: a 1% conversion lift
12:39 Coaching new PMs: ranked hypotheses and guardrail metrics
15:41 Taking credit for free shipping in the order summary
18:10 Copy testing, strategic bets and a semantic layer for AI crawlers
22:24 AI in the PM workflow: research in seven days instead of two months

Takeaways
- Rewards work where purchase intent already exists: Quill's homepage loyalty widget drew under 0.5% engagement, while the same value on the cart page lifted conversion.
- Converting points into dollars at the cart raised conversion 1%, brought small orders back and reduced the unredeemed points liability on the books.
- Know the direction of a test before you run it; the A/B test measures how much, which is how Vipul's loyalty work reached 8 winners out of 11.
- Design every experiment with a primary hypothesis, ranked secondary hypotheses and guardrail metrics so a loss still explains itself.
- Showing the $5.99 shipping saving at checkout raised conversions without changing order value, a sign that transparency builds trust.

Connect with the Guest
Vipul Tyagi LinkedIn: https://www.linkedin.com/in/vipul-tyagi
Company Website: https://www.quill.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

• (00:00) - Cold open