In this episode of Experiment Nation, host Jason sits down with Namata Agarwal, a seasoned product designer with over 17 years of experience building digital products, websites, and apps.Together, they explore:- Ethical testing: how to balance user needs with business goals.- Leaks vs. puddles framework: a practical way to identify urgent vs. non-urgent product problems.- The role of design in CRO and experimentation.- Frameworks, guardrails, and collaboration strategies for building an experimentation culture.Whether you’re a CRO specialist, product manager, or experimentation nerd, this episode is packed with actionable insights you can bring to your own team.👉 Don’t forget to like, comment, and subscribe to stay updated on the latest in experimentation and growth!📖 Check out Prove It or Lose It: The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama .⏱️ Chapters with Timestamps0:00 – Why design must be involved early0:22 – Welcome & intro to Jason0:43 – Guest intro: Namata Agarwal0:55 – Topics: Ethical testing & Leaks vs. Puddles1:18 – What is ethical testing?2:22 – Balancing business needs vs. user needs2:54 – Handling stakeholder pushback4:01 – Misaligned metrics and real solutions5:17 – The Leaks vs. Puddles analogy explained6:32 – Why better hypothesis building matters7:48 – Using Airtable & frameworks for test prioritization8:40 – Why experimentation must be collaborative9:24 – Reframing experiments around the user journey10:00 – Unpredictable user journeys & design challenges11:12 – Involving guest collaborators in experimentation11:50 – Mapping user journeys with Miro & qualitative research12:50 – Jobs To Be Done & other qualitative frameworks14:44 – Turning qualitative insights into hypotheses15:32 – How product design elevates CRO strategies16:11 – The importance of forms in experimentation17:36 – Why design should be included from day one18:20 – Design is not just visuals—it’s about impact19:11 – Governance, responsibility, and inclusivity in testing19:46 – Leak vs. Puddle framework deep dive22:58 – Running collaborative workshops25:45 – The “Stinky Fish” workshop format27:21 – Micro vs. macro frameworks for problem-solving28:05 – Balancing bold ideas vs. safe bets29:19 – Planning tests like an investment portfolio30:13 – Testing across 13 sites at scale30:26 – What Namata would A/B test in daily life31:57 – Art, painting, and creative balance32:42 – Martial arts & keeping calm32:56 – Wrap up & final thoughts33:14 – How to connect with Namata

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode of the Experiment Nation Podcast, host Ward Vanestra sits down with Ryan Lux, one of the first team members at Eppo and now Experimentation Evangelist at Datadog.They dive into:What Eppo is and how it helps companies experiment at scaleWhy data warehouses are critical for trustworthy resultsThe recent Datadog acquisition of Eppo and what it means for the futureAdvanced methods like CUPED (noise-cancelling for experiments) and contextual banditsHow to scale experimentation culture and get promoted by driving adoption, efficiency, and trustworthinessWhether you’re a CRO, growth leader, or experimentation enthusiast, this episode is packed with insights from someone who’s been shaping experimentation for over a decade.📘 Check out Experiment Nation’s new book: Prove It or Lose It: The Mostly No-Nonsense Guide to Surviving Experimentation Program Drama → experimentnation.com⏱️ Chapters0:00 – Why win rates don’t really increase0:20 – Introduction to Ryan Lucht & Eppo1:20 – What Eppo does and who it’s for2:18 – Experimentation at smaller companies3:18 – Datadog’s acquisition of Eppo3:59 – The data warehouse advantage6:03 – CUPED explained: noise-cancelling for experiments9:13 – Segments, personalization & pitfalls11:09 – Heterogeneous treatment effects (HTEs)12:06 – Personalization with contextual bandits14:00 – Why Datadog acquired Eppo16:22 – Data observability & product analytics17:52 – Experimentation and site performance19:46 – Content, culture, and experimentation career paths22:10 – Every experiment is valuable (win, lose, or flat)24:17 – OKRs for experimentation leaders25:47 – Three key OKRs: adoption, efficiency, trustworthiness28:14 – Knowledge building and learning plans29:34 – The vision: experiments as the ultimate insight system30:24 – Ryan’s Substack “Everything is an Experiment”31:02 – Closing thoughts & where to connect

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode of the Experiment Nation Podcast, host Richard Joe welcomes Manuel Da Costa, founder of Effective Experiments and now Efestra, to unpack a critical issue in the experimentation industry: the trust gap between practitioners and business decision-makers.🎯 Manuel shares how running more experiments isn’t the goal—making better decisions is. He introduces the Learning Loop, explains why most experimentation programs stall, and gives practical strategies for connecting experimentation insights to executive decisions.🔍 Whether you’re in CRO, product experimentation, or experimentation governance, this is a must-watch for moving beyond vanity metrics and toward real impact.📌 Connect with Manuel:📍 LinkedIn: https://www.linkedin.com/in/manueldacosta/#CRO #Experimentation #ABTesting #EffectiveExperiments #ProductManagement #DigitalMarketingChapter00:00 – Experiments don't matter unless they drive decisions 00:30 – Meet Manuel Da Costa, founder of Effective Experiments 01:45 – Manuel’s journey: from web design to CRO and Lean Startup 03:00 – How experimentation became his product 05:20 – The limitations of spreadsheets in experimentation 06:45 – Building Effective Experiments to streamline the workflow 09:10 – What is the "Trust Gap" in experimentation? 11:30 – Why 88% of experimentation programs aren’t transformative 13:00 – The survival mindset vs. strategic experimentation 16:00 – The echo chamber of experimentation advice 17:40 – Why velocity and win rate are not the real success metrics 19:00 – The problem with how learnings are shared 21:00 – How to communicate insights in one powerful slide 23:00 – Building organizational learning with the “Learning Loop” 25:00 – Step-by-step breakdown of the Learning Loop framework 27:15 – Why experimentation needs executive alignment 29:00 – How to reset a tactical experimentation program 30:40 – Introducing Efestra: a governance platform for experimentation 32:00 – Final thoughts and how to get in touch with Manuel 33:14 – About the book: *Prove It or Lose It*

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode, we dive into Speero’s shift from CRO to Experiment-Led Growth, how they structure their testing systems, where to start with a new client, and what makes a good growth bet. Plus, we discuss the power of insights, the rise of EXL meetups and the future of Speero.Chapters----00:00 - Introduction 00:18 - Meet the Hosts & Guest 01:11 - From CRO to Growth Experimentation 02:55 - Building a System for Experimentation 05:00 - Organizational Buy-in and Culture Shift 06:02 - Client Reactions and Collaboration 08:00 - Applying Experimentation Across the Business 10:00 - Strategy vs. Tactics 11:30 - Why Create a Strategy? 13:02 - Making Better Bets with Experimentation 15:05 - Breaking Down Big Initiatives 16:41 - Landing Pages, Messaging & Paid Spend 18:20 - Irreversible Risks and Thought Experiments 20:15 - Research Frameworks: Qual & Quant 21:47 - Copy Testing and Messaging Insights 23:48 - Combining Qualitative and Quantitative Testing 25:49 - Using Ads to Accelerate Learning 26:33 - EXL Meetups: Community and Format 28:00 - How to Host an EXL Meetup 30:04 - Upcoming Global Meetups 31:35 - How to Connect with Paul Randall 32:40 - Outro#cro #experimentnation #abtesting

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

This episode features Achille Vaulot, a CRO expert, discussing the evolution of conversion rate optimization (CRO) and the steps to building a mature program. He highlights the shift from basic A/B testing to a data-driven, strategic approach encompassing research, thorough analysis (including post-test analysis of both primary and secondary KPIs), and effective client communication. Achille emphasizes the importance of planning for both successful and unsuccessful tests, managing client expectations, and fostering a culture of experimentation within the client’s organization. He stresses the value of qualitative data alongside quantitative results and the importance of aligning CRO strategies with broader business goals. The conversation underscores the crucial role of education and collaboration in scaling a successful CRO program.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Ekaterina (Shpadareva) Gamsriegler, a seasoned growth and marketing expert, shared her insights on mobile app experimentation. Here are 5 key takeaways:

Mobile experimentation is more complex: Release cycles are longer, and dependencies on app versions can slow down learning.
Think beyond the app: Experimentation should encompass ads, store listing pages, and the app itself.
Borrow insights from other areas: Use data from paid advertising to inform store listing optimizations.
Interconnectivity is key: Changes in one area can impact others. Strong stakeholder management is crucial.
Document everything: Detailed documentation helps track results and avoid surprises.
Check out the full conversation on Experiment Nation’s YouTube channel!LinkedIn: https://www.linkedin.com/in/ekaterina-shpadareva-gamsriegler/

#mobilemarketing #appmarketing #experimentation #growthmarketing #experimentnation

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Tracy Laranjo shares learnings from years of growing her career with Suchita Srinivasan, here are some key takeaways:

1. CRO vs Experimentation: CRO focuses on driving performance and results, while experimentation centers around decision-making and validating ideas.
2. Agency vs In-House: Tracy’s journey highlights the benefits of agency experience for rapid skill development and exposure to diverse challenges.
3. Building Your Experimental Brand: Don’t be afraid to be yourself online, even if it’s not the typical thought leader persona.
4. Adapting to Constant Change: Focus on fundamentals and building your skills; stay curious and open to new developments, but don’t fear the future.
5. CRO is for Everyone: While AB testing requires scale, CRO encompasses qualitative research and user interviews, making it accessible for businesses of all sizes.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

This episode of Experiment Nation features an interview with Dr. Else van der Berg, a product management consultant for B2B SaaS startups and scale-ups. Dr. Else van der Berg discusses her process for helping companies become more outcome-driven, focusing on understanding the ideal customer profile (ICP) and its associated value. She also shares insights on common challenges faced during the testing phase, including participant recruitment, data analysis, and differentiating signal from noise.

Highlights
🎯 Dr. Else van der Berg emphasizes the importance of defining a narrow and well-defined ICP, emphasizing the need to focus on a specific customer segment.
🔎 She believes in a layered approach to testing, using a variety of methods, including interviews, landing page tests, and moderated user tests.
🗣️ Empathy plays a crucial role in interpreting user feedback, recognizing that data interpretation can be subjective and require a “gut feeling” alongside evidence.
📈 Dr. Else van der Berg stresses the importance of finding a mentor within the field and learning on the job to master product management techniques.
👩‍💻 She encourages aspiring product managers to reach out to her via LinkedIn to learn more and potentially connect for career advice.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

We recently had the pleasure of chatting with Rishi, a Shopify product expert. Here are 5 key takeaways for optimizing your product pages:

1) Product pages are the Grand Central Station of conversions. They’re where buyers make the decision to buy or leave.
2) Pop-ups aren’t as effective as you think. Focus on valuable content, not annoying interruptions.
3) Content is king (but don’t be afraid to go long!). Detailed product descriptions are more effective than short ones.
4) Build trust above the fold. Establish your company’s credibility before highlighting product features.
5) Understand the filters buyers use. Address common concerns like “too good to be true” to build confidence.
Want to learn more? Check out Rishi’s website, frictionlesscommerce.com!

#ecommerce #productpages #conversionrateoptimization #marketing #experimentation

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Black Friday is coming, and the pressure is on. But don’t worry, Ilan Hurwitz has some expert advice to help you navigate the chaos.

Here are five key takeaways from his chat with Tracy Laranjo:

1) Start now! Don’t wait until the last minute to get buy-in and start testing.
2) Speak the language of profit. Showcase the ROI of CRO to your stakeholders, focusing on increasing conversions and profits.
3) Don’t be afraid to experiment. Test offers, product placements, and even email campaigns outside of standard website A/B testing.
4) Focus on the customer journey. Don’t just look at your website. Analyze customer service tickets, reviews, and LTV to identify areas for improvement.
5) Think beyond discounting. Explore creative offers like gift with purchase, cross-sells, and subscription incentives to boost sales.

Get ready to conquer Black Friday!

#BlackFriday #CRO #ecommerce #marketing #experimentation

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com