The Experimentation Edge

Summary
In this episode of The Experimentation Edge, host Ashley Stirrup sits down with Raunak Kumar, senior manager of GTM analytics at Fin (formerly Intercom), to unpack how experimentation actually works when the data is messy and the traffic is thin. Drawing on nearly 12 years in marketing analytics across Atlassian, Stripe, and Fin, Raunak explains how AI tools like Claude Code have collapsed analysis from weeks to hours and freed his team to clear its experiment backlog, why declining organic search traffic and a 5x jump in untagged ChatGPT referrals are forcing teams to rethink attribution, and how the most valuable experiments are often the ones that "lose." From a Jira Service Desk bundling test that won on trials but had to be rolled back, to a Stripe contact form that was quietly blocking real buyers, this conversation is a practical guide for product managers, engineers, data scientists, and growth marketers who want to learn more from every test they run.

Chapters
00:45 Welcome and what the show is about
01:45 Raunak's role and 12 years in marketing analytics
02:45 How AI and Claude Code changed the analyst's day
04:15 LLMs, declining organic traffic, and the 5x ChatGPT jump
05:15 Two kinds of experiments at Fin: on page and off page
07:15 The Jira Service Desk bundling experiment
10:45 Why the trial winner became a rollback
11:45 Contextual onboarding turns the loser into a winner
14:45 Reading an experiment that loses
18:45 What's next: incrementality, connected TV, and testing creative

Takeaways
- AI has collapsed marketing analysis from weeks to hours, and the real payoff is a cleared experiment backlog plus analysts who compete on the questions they ask, not the speed they query.
- Organic search traffic is declining as ChatGPT, Gemini's AI mode, and Claude answer buyers in place; Fin saw a 5x rise in ChatGPT referrals, but LLMs don't tag that traffic, so attribution has to be proven through experiments.
- A guardrail metric saved Atlassian from a costly mistake: bundling Jira Service Desk lifted trials more than 50 percent but tanked activation and paid conversion, forcing a rollback.
- A failed test can hold the real winner; contextual onboarding matched to user intent roughly doubled activation and became the default variant after the bundling experiment was rolled back.
- In low-volume B2B, read losing experiments for sub-segment signal; a "failed" Stripe form simplification revealed the form was blocking legitimate small-business buyers using Gmail.

Connect with the Guest
Raunak Kumar LinkedIn: http://linkedin.com/in/raunakkumar1991
About the guest: https://www.growthbook.io/podcast/guests/raunak-kumar
Episode page: https://www.growthbook.io/podcast/episode/1-22
Company Website: https://fin.ai

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Summary
What does experimentation look like inside a $150 billion retailer? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Kim Ting Li, Senior Manager of Experimentation at The Home Depot, where one centralized team tests every major change to a $25 billion online business. Kim explains how 40 people serve 40–50 business teams, why executives join test readouts and ping analysts directly, how every result since 2020 lives in a searchable library, and why scaling beyond hundreds of experiments per year depends on server-side testing capabilities more than AI. For product, data, and engineering leaders building or scaling experimentation programs.

Chapters
00:00 Intro
00:45 From neuroscience research to Home Depot
01:45 A $150B enterprise, a $25B online business
02:45 The centralized experimentation model
03:45 Inside the 40-person team
04:30 Readouts, blast emails, and the experiment library
05:40 Executive visibility and the golden rule
06:15 "If you won't act on a bad result, don't run the test"
11:15 Learning from losing tests
12:30 Scaling up: AI, server-side testing, and what's next

Takeaways
- One centralized team of about 40 people tests every major change to Home Depot's $25B online business, serving 40–50 business teams with consistent hypothesis and analysis standards.
- Executive engagement is real at Home Depot: leaders join 30-minute readouts, search the experiment library, and ping analysts directly because they treat A/B testing as the golden rule for measuring incrementality.
- Institutional memory is infrastructure — every test result since 2020 lives in a centralized, searchable archive so no one re-runs a question the company already answered.
- Kim's stakeholder filter: if you wouldn't do anything differently after a bad result, don't run the test.
- Scaling past low hundreds of experiments per year is a capabilities problem before it's an AI problem — Home Depot is moving from client-side to server-side testing so winners release quickly, end to end.

Connect with the Guest
Kim Ting Li LinkedIn: https://www.linkedin.com/in/kimtingli
About the guest: https://www.growthbook.io/podcast/guests/kim-ting-li
Episode page: https://www.growthbook.io/podcast/episode/1-21
Company Website: https://www.homedepot.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Summary
What does it look like to kill a multimillion dollar feature before anyone builds it? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Crystal Ammari, a digital product optimization and experimentation strategy leader whose career spans Nike and The Walt Disney Company. Crystal shares the "dry test" that used a single fake button to measure demand for video chat (4 million users, 106 clicks), why she reframes experimentation as savings and gains rather than wins and losses, how a misconfigured tool, not bad methodology, made tests take six months, and how a stuck Disney team went from "we don't know where to start" to 110 scored and prioritized test ideas. For product, data, and engineering leaders building or scaling experimentation programs.

Chapters
00:00 Intro
00:45 The mindset shift from shipping to results
02:00 Why testing took six months, a tooling problem
03:15 The dev team that laughed, and the vendor who agreed
04:50 An executive demand for video chat
05:35 Dry testing with a fake button
06:30 106 clicks and a multimillion dollar save
07:30 Savings and gains, not wins and losses
08:45 The Disney team that didn't know where to start
10:30 From low engagement to 110 prioritized ideas
12:45 Just get something live, and where AI fits next

Takeaways
- A "dry test", a fake "Click here to video chat" button that grayed out on click — measured real demand without building the feature. Of roughly 4 million users, only 106 clicked, killing a multimillion dollar build.
- Reframe experiment outcomes as savings and gains, not wins and losses. A "losing" test saves you from a costly mistake, which keeps teams focused on learning instead of fearing failure.
- Slow experimentation is often a tooling problem, not a methodology problem. One program's six month test cycle came from rebuilding every page instead of overlaying changes the way the tool intended.
- Getting a stuck team unstuck starts with data and a workshop. A Disney team went from "we don't know where to start" to 110 scored, prioritized test ideas, using Contentsquare heatmaps to diagnose low engagement first.
- The biggest thing that gets a team testing is to just do it. Stop designing the perfect experiment and get something simple live to take away the mystery.

Connect with the Guest
Crystal Ammari LinkedIn: https://www.linkedin.com/in/crystal-ammari/
About the guest: https://www.growthbook.io/podcast/guests/crystal-ammari
Episode page: https://www.growthbook.io/podcast/episode/1-20
Company Website: https://thewaltdisneycompany.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Summary
How do you know if the thing you just shipped actually worked? On this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with Kevin Yang, Executive Director and Head of Experimentation at JPMorgan Chase, who has spent six years building experimentation across Chase's digital platforms. Kevin shares how his team turned experimentation into more than a billion dollars of estimated value, why the losing experiments matter more than the winners, and the simple chart exercise he uses to prove that a million-dollar change is invisible without a control group. He and Ashley also dig into measuring engagement without chasing vanity metrics, planning for failure to defeat confirmation bias, and why AI is pushing experimentation into a golden era. It's a practical look for product managers, data scientists, and engineers at how a bank operating at massive scale makes better decisions.

Chapters
00:00 Welcome to the experimentation edge
01:45 Kevin's role leading experimentation at chase
04:15 Why chase invested in experimentation
06:45 A billion dollars and the value of losers
12:45 Plan for failure to beat confirmation bias
14:30 The million dollar change you can't see
18:45 Sharing learnings and experimentation wrapped
20:45 Engagement without vanity metrics
22:00 Experimentation's golden era with AI
23:30 Why AI needs more experimentation, not less

Takeaways
- Chase estimates over a billion dollars of value from experimentation, and most of the lasting learning comes from the losing tests, not the winners.
- A control group is non-negotiable: at scale, a change worth millions is invisible under noise and seasonality, and no one can spot it by eye.
- Treat engagement carefully. For a bank, more time in the app isn't a win; trust, fast task completion, and healthy repeat engagement are.
- Plan for failure before you run a test. A pre-built playbook for a loss prevents confirmation bias and keeps teams from gaming the metrics.
- AI is ushering in a golden era for experimentation, because shipping faster only compounds mistakes unless you measure what you ship.

Connect with the Guest
Kevin Yang LinkedIn: https://www.linkedin.com/in/kevintyang
About the guest: https://www.growthbook.io/podcast/guests/kevin-yang
Episode page: https://www.growthbook.io/podcast/episode/1-19
Company Website: https://www.jpmorganchase.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Summary
How do you make a high-stakes product decision when the safe choice is to never test it at all? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Arun Bodapati, director of data science at Twitch, about the discipline behind trustworthy experimentation. Drawing on his experience at Schwab, Uber, and Twitch, Arun explains why false negatives are the most dangerous result a team can produce, what hygiene to nail before you push play, and how Twitch used geo-fenced experiments and causal inference to finally settle a pricing question it had avoided for years. It's a practical conversation for product managers, engineers, data scientists, and growth leaders who want experiments that hold up and earn executive trust.

Chapters
00:00 Welcome and introduction
01:15 Arun's background and marketing experimentation at Schwab
04:15 Uber's mature, experiment-driven culture
06:30 Coming to Twitch: from Python notebooks to a shared standard
08:30 The pricing problem Twitch had long avoided
10:30 Geo-fenced experiments, matched markets, and elasticity
13:15 The gifted-subs surprise and testing promotions
16:15 The discipline that matters before you push play
18:15 Why false negatives are worse than false positives
20:05 Enrollment triggers and broad explore experiments
22:45 AI, the Kiro tool, and what's next for experimentation

Takeaways
- False negatives are more dangerous than false positives — they get institutionalized as "we tried that, it didn't work" and quietly kill good ideas for years.
- The most valuable experiment work happens before you push play: clear enrollment logic, a plain-English hypothesis, and no optimizing ahead of the test.
- If an intervention sounds weak when you write it out in plain English, don't run the experiment — you're just wasting time.
- Run a broad explore experiment first; small, over-narrowed populations lack power and raise the odds of a false negative. Find the responsive segment with heterogeneous treatment effects afterward.
- Twitch used geo-fenced experiments with matched markets and causal inference to measure true price elasticity, turning a feared pricing decision into a measured, accretive one.

Connect with the Guest
Arun Bodapati LinkedIn: https://www.linkedin.com/in/abodapati/
About the guest: https://www.growthbook.io/podcast/guests/arun-bodapati
Episode page: https://www.growthbook.io/podcast/episode/1-18
Company Website: https://www.twitch.tv

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Summary
What do you do when your big launch increases engagement and tanks conversion? On this episode of The Experimentation Edge, host Ashley Stirrup talks with Lina Blackman, Director of Product Analytics at Squarespace, about the blank template launch that flopped — and how its learnings became Blueprint, Squarespace's AI-guided website builder. Lina explains how her embedded analyst team runs 150–200 experiments a year for 3 million customers, the two questions she asks every time a test loses, why teams only need one or two big wins a quarter, how Squarespace calibrates statistical certainty to business stakes, and where AI belongs (and doesn't) in the A/B testing workflow. For product managers, data scientists, and experimentation leaders who want to extract more learning from every test.

Chapters 00:00 Introduction: Lina Blackman, Director of Product Analytics at Squarespace 01:45 Squarespace's business and 3 million website customers 02:30 Decentralized analysts, centralized experimentation program 04:15 150–200 experiments a year: onboarding, mobile, checkout, pricing 04:55 The blank template disaster that became Blueprint AI 07:45 Two questions for every losing test 09:30 Moving ship-first teams up the experimentation maturity curve 12:30 A/B test logs and insights rituals 13:30 North Star metrics and the KPI tree 16:35 AI in the A/B testing workflow — and what stays manual.

Takeaways

• Stated preference lies: users asked for a blank canvas, but behavior demanded guided design — and only the experiment could referee.
• Close every losing test with two questions: did it work for a granular segment, and is the idea worth further investment?
• One or two big wins a quarter is a healthy hit rate when you run 150–200 experiments a year.
• Calibrate certainty to stakes — tight bounds on revenue and pricing tests, wider bounds on engagement tests so teams don't spin on noise.
• Hand AI the mundane parts of the workflow (tracking, assignment setup), but if AI runs the brief and the analysis, ask why you're running the test at all.

  
  

Connect with the Guest
Lina Blackman LinkedIn: https://www.linkedin.com/in/linanguyen
About the guest: https://www.growthbook.io/podcast/guests/lina-blackman
Episode page: https://www.growthbook.io/podcast/episode/1-17
Company Website: https://www.squarespace.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Summary
Craig Kistler, VP of Experience Design, Personalization, and Experimentation at Signet Jewelers (the parent company of Kay, Jared, Zales, Peoples, and Banter), joins host Ashley Stirrup on The Experimentation Edge to unpack how a hybrid online-and-in-store jewelry retailer runs experimentation at scale. Craig shares the counterintuitive "view all" experiment where his team blocked the product grid, added friction on purpose, and grew revenue; why he optimizes for revenue per visitor instead of conversion rate; and how Signet deliberately traded a high-volume testing program for fewer, higher-value experiments. A practical listen for product managers, designers, and experimentation leaders building programs that compound.

Chapters
00:45 What Signet Jewelers actually is (Kay, Jared, Zales, and more)
01:45 From art school to UX to experimentation: Craig's background
04:45 How experimentation is organized: a centralized model across brands
06:45 From 40–50 tests a quarter to 15–25 value-driven experiments
08:45 The "view all" experiment: adding friction to grow revenue
12:45 One product page, many stakeholders: financing, warranties, chat
15:45 Extracting learnings when an experiment loses
18:45 Why revenue per visitor beats conversion as the north star
20:15 Intent-based personalization and "engagement season is every day"
24:45 Bringing the whole org along by tying insights to dollars.

Takeaways
- Friction can increase revenue. Blocking the "view all" grid and forcing a style choice sent shoppers deeper and lifted conversion and revenue, because the extra click added value.
- The three-click rule is conditional. Clicks only hurt when they're empty; a click that narrows thousands of options to dozens is a feature, not a cost.
- Revenue per visitor is the honest north star. Conversion rate can be gamed to 100% by making everything free or cutting bounce-heavy traffic; revenue per visitor can't.
- Fewer, bigger experiments beat high volume. Signet went from 40–50 tests a quarter to 15–25 because complex, value-driven tests produce reusable insights that small tweaks don't.
- Tie every result to dollars. Translating experiment outcomes into revenue is how Craig keeps financing, warranty, and chat stakeholders aligned and gets executives to act.

Connect with the Guest
Craig Kistler LinkedIn: https://www.linkedin.com/in/craigkistler/
About the guest: https://www.growthbook.io/podcast/guests/craig-kistler
Episode page: https://www.growthbook.io/podcast/episode/1-16
Company Website: https://www.signetjewelers.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Summary
RingCentral's Director of Product Management for AI Products, Mayank Agarwal, joins host Ashley Stirrup to dismantle the metrics most teams use to judge AI agents. Drawing on his background founding an AI-first quantitative trading firm and scaling Groupon's bookable marketplace, Mayank explains why accuracy and thumbs-up/down feedback both mislead, and introduces DART — a four-metric behavioral framework (decay, acceptance, relevance, task completion) ported from how he measured trading strategies. He also breaks down a Groupon flash-discount experiment that backfired and the scarcity pivot that fixed it. Essential listening for product managers, engineers, and data scientists building or measuring AI features.

Chapters
00:00 Welcome and Mayank's path from quant trading to RingCentral AI
02:45 Why experimentation has to be owned cross-functionally
04:55 Small experiments that compounded to a 12% lift at Groupon
06:45 Why accuracy and thumbs-up/down fail for AI agents
08:15 The DART framework, metric by metric
12:45 Applying DART to AI-generated smart notes
14:55 The Groupon flash-sale that dropped conversion
16:45 Swapping price urgency for scarcity and social proof
19:45 North Star metrics, guardrails, and Goodhart's law
26:45 The future: experimenting on — and for — AI agents

Takeaways
- Accuracy is a comfortable lie. It grades a narrow test set and can stay high while the agent fails real users.
- Thumbs-up/down feedback is sparse and skewed. Unhappy users rarely rate — they just quietly stop using the product.
- DART measures behavior, not opinions. Four signals read off logs and transcripts: decay, acceptance, relevance, and task completion.
- Acceptance rate is the trust metric. The share of output users keep without editing is the strongest available proxy for trust.
- A losing experiment is paid-for information. Groupon's flash-sale flop revealed the lever was wrong, not the goal — scarcity beat price-based urgency.

Connect with the Guest
Mayank Agarwal LinkedIn: https://www.linkedin.com/in/mayank-agarwal-6223b04a/
About the guest: https://www.growthbook.io/podcast/guests/mayank-agarwal
Episode page: https://www.growthbook.io/podcast/episode/1-15
Company Website: https://www.ringcentral.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Summary
On this edition of The Experimentation Edge, Ashley Stirrup talks with Geoffrey Bell, Experimentation Product Specialist at Ford Credit, about building an experimentation practice inside a captive auto lender. Geoffrey shares the losing test that earned his program credibility, the "experimentation piggy bank" he picked up at Microsoft, and the breakthrough of connecting online experiments to offline dealership receivables. The throughline: a program proves its worth not just by the wins it ships, but by the expensive mistakes it prevents and the revenue it can finally trace. It's for product managers, data scientists, and growth leaders who want experimentation taken seriously by the business.

Chapters
00:00 Intro
01:15 Geoffrey's path: Lowe's, Microsoft, Ford Credit
07:15 How Ford Credit fits with Ford Motor
10:15 The teams behind every Ford Credit page
15:15 The vehicle selector test that lost on purpose
19:15 Why feature placement beats feature ideas
21:15 Personalization and the shrinking-audience problem
25:15 Telling the story when a test loses
30:45 Connecting an online test to an offline car sale
33:55 The experimentation piggy bank

Takeaways
- 1. Losing tests often create more value than winners because they stop expensive mistakes before they ship.
- 2. Measure experimentation two ways: the revenue you earn from wins and the revenue you save by killing bad experiences.
- 3. A feature that fails early in a flow can succeed later; placement and timing often matter more than the idea itself.
- 4. Connecting online experiments to offline outcomes like receivables turns a small lift into a number leadership cares about.
- 5. When you struggle to land a result, lead with the story of what the customer did, then bring the numbers.

Connect with the Guest
Geoffrey Bell LinkedIn: https://www.linkedin.com/in/geoffrey-bell-62a03617/
About the guest: https://www.growthbook.io/podcast/guests/geoffrey-bell
Episode page: https://www.growthbook.io/podcast/episode/1-14
Company Website: https://www.ford.com/finance/

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

• (00:00) - Intro

This episode of The Experimentation Edge explores how A/B testing, feature flags, and user research transformed Atlassian's talent product after it failed with its first users. Andrew Willingham — 11 years at Amazon, now Head of Legal and People Products at Atlassian — shares how product experimentation works when you can't test at scale, why your customer and your user are not the same person, and how the metrics you choose decide which experiments you can even run.

Summary
Andrew Willingham, Head of Legal and People Products at Atlassian, spent 11 years at Amazon before joining Atlassian a year ago. His path from running A/B tests on millions of Amazon shoppers to building talent management software for a few hundred thousand employees forced a fundamental shift: when you can't run tests at scale, you have to sit with your actual users and watch them fail. He shares how building a talent review product for Amazon's HR specialists completely flopped when handed to HRBPs — and why that failure taught him more than any winning experiment. Now at Atlassian, he's applying that same rigor to reimagining hiring processes with AI, testing everything from recruiter screens to interview sequences that the industry has run the same way for decades.

Timestamps
03:09 From marketing Amazon's mobile app to building HR software for 1.5 million associates
08:19 Why a talent review product loved by IO psych experts flopped with actual HRBPs
11:11 How A/B testing helps product managers escape opinion-based politics
15:25 Testing copy that changes behavior: "We'll generate that status report for you"
17:20 The two North Star metrics Andrew optimizes: efficiency and quality
19:05 Khan Academy's metric trap: measuring cognitive engagement, not just completion
21:10 Why product managers resist experimentation — and what changes when you admit you don't know

Takeaways
- Your customer and your user may not be the same person — building for HR specialists instead of the HRBPs who actually run talent reviews resulted in a feature nobody could use.
- When you can't test at scale, desk rides replace A/B tests — sitting with users and watching them struggle reveals failures faster than any dashboard.
- Experimentation short-circuits political debates by removing opinion from product decisions.
- Test metrics before you test features — usage time could signal engagement or just mean your product takes too long to do its job.
- The experiments that fail deliver the most valuable learnings, especially when you expected a slam dunk.

Connect with the Guest
Andrew Willingham LinkedIn: https://www.linkedin.com/in/andrewwillingham/
About the guest: https://www.growthbook.io/podcast/guests/andrew-willingham
Episode page: https://www.growthbook.io/podcast/episode/1-13
Learn more about Atlassian: https://www.atlassian.com/

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Topics: A/B testing, product experimentation, feature flags, user research, talent management, qualitative research, metric design, experimentation at scale, growth experimentation.