How I Tested That

Summary

In this episode I’m joined by Bobby Moesta, founder of The Re-Wired Group and the co-creator of Jobs to Be Done.

I’ve been a fan of Bobby’s work for years and instead of deep diving on yet another jobs to be done conversation with him, we explore a topic he doesn’t get to chat about very often, which is how systems thinking changes the way we experiment, and why understanding trade offs is essential to making better decisions. We chat about prototyping to learn and prototyping to verify, and it draws on lessons from his mentors, including luminaries such as Deming, Taguchi, and Christensen.

We discuss what happened to the R in R&D, and we wrap up with a fascinating potato chip experiment that reveals the gap between what customers say and what customers do.

If you are a fan of system thinking and experimentation, then you’ll love this episode.

Takeaways

1. Prototype to learn, not just to verify. Prototyping should help uncover what you don't know, rather than simply confirm what you already believe.

2. Measure outcomes, not just outputs. Start with the progress customers are trying to make, then work backward to design the systems and measurements that matter.

3. Great decisions come from understanding trade-offs. There is rarely one optimal solution. Testing helps reveal what you're willing to give up to achieve what matters most across desirability, viability, and feasibility.

4. Tackle the biggest unknowns before building. Teams often rush into execution because it's easier to work on what they know. Addressing critical unknowns early prevents costly rework later.

5. Observe what people do, not just what they say. Surveys and stated preferences can be misleading. Bobby's potato chip experiment showed how actual consumption and choices can provide stronger evidence than asking people what they like.

Guest Links

The Re-Wired Group: https://therewiredgroup.com/
LinkedIn: https://www.linkedin.com/in/bobmoesta/

Summary

In this episode, I’m joined by Anna Burch from SmartMouth, a family owned oral care company founded by her grandfather, Dr. Marvin Cohen.

What started with a dentist questioning the root cause of bad breath eventually became a nationally distributed consumer product, but having a scientifically proven solution doesn't necessarily mean customers will understand it, buy it, or even know how to use it.

Anna and I explore the challenges of testing your product design with customers, expanding into new markets, and knowing when to say no to opportunities that could undermine what makes your value proposition unique.

If you want to learn more about solving root problems instead of symptoms, and balancing testing with what makes your product special, you'll love this episode.

Takeaways

1. Solve the root cause, not just the symptoms. Dr. Marvin Cohen challenged conventional assumptions about bad breath, researching the underlying science instead of simply masking odors.

2. Innovation doesn't mean abandoning what makes you different. SmartMouth has maintained its core activated zinc technology for decades, even resisting opportunities to create cheaper products that would compromise its differentiation.

3. A great product can still have usability problems. SmartMouth's dual-solution technology requires an unconventional bottle design. Customer feedback revealed that even loyal customers struggled with the experience, leading to improvements in packaging.

4. Understand the emotional and social jobs customers are hiring your product to do. Fresh breath is the functional benefit, but confidence, self-esteem, and social connection can be just as important to customers.

5. Your go-to-market strategy must evolve with the market. SmartMouth grew from dental offices to QVC and mass retail, eventually embracing e-commerce and creator-led content to educate customers about a product that requires more explanation than traditional advertising allows.

Guest Links

SmartMouth: https://smartmouth.com/

Summary

In this episode, I’m joined by Melissa Perri. You may know her best as the author of Escaping the Build Trap and I believe she is one of the brightest minds in modern product management. 

I’ve been privileged to know Melissa since the early days of the Lean Startup movement and I was thrilled to chat with her about what has changed in product and testing over the past 15 years. 

We talk about the dangers of PM’s outsourcing their critical thinking to AI, and why experimentation with good data is maybe now more important than ever.

Melissa details out how AI fundamentally transforms product operations and how companies can test these new found capabilities and actually build things customers need.

If you want to learn how to make better product decisions, accelerate learning and avoid this latest AI version of the build trap, then you'll love this episode.

Takeaways

1. AI is accelerating the build trap. Companies can build software faster and cheaper than ever, but without clear strategy and customer understanding, they're simply building the wrong things at unprecedented speed.

2. Don't outsource judgment to AI. AI can generate specifications, analyze data, and accelerate execution, but experienced product leaders are still needed to challenge assumptions, interpret evidence, and determine what should actually be built.

3. Better data doesn't automatically mean better decisions. Organizations struggle with fragmented systems, conflicting metrics, and unreliable information. AI creates enormous opportunities to improve decision-making, but only when teams can access and trust the underlying data.

4. Product operations could be supercharged by AI. From synthesizing customer research to interrogating business performance, AI can dramatically reduce the time required to generate insights. Product Ops can help connect these capabilities to strategy, governance, and measurable business outcomes.

5. AI won't fix broken organizational systems. Functional silos, disconnected strategies, rigid funding cycles, and competing departmental goals remain major obstacles. Without addressing these systemic issues, organizations risk using AI to make dysfunctional processes run faster.

Guest Links

LinkedIn: https://www.linkedin.com/in/melissajeanperri/

Product Institute: https://productinstitute.com/

Summary

In this episode, I’m joined by Moses Adedoyin. Moses is leading Venture Design & Innovation over at GuideWell, where he’s testing new healthcare solutions to help patients.

We discuss what testing looks like when you work in one of the most heavily regulated environments imaginable. Moses shares how his background in engineering and simulation helped shape the way he approaches testing and how his team built a lab where they could test new ideas safely before introducing them into a live healthcare environment.

We deep dive into the real world practicality of using synthetic data to increase the fidelity of experiments and how that can dramatically reduce the time and cost required to evaluate new solutions. Moses and I explore digital twins, AI-assisted simulation, and how technology enables healthcare teams to spend less time on administrative work and more time actually helping patients in need.

If you want to learn more about testing in highly regulated healthcare, using AI and synthetic data, then you’ll love this episode.

Takeaways

1. Testing in regulated environments requires a different path. In healthcare, teams can’t always “just talk to customers” or run a quick experiment because PHI, consent, security, procurement, and regulation all shape what is possible.
2. Synthetic data can help teams build evidence earlier. Moses’ team uses synthetic data to test concepts, technology, and early value before involving real patient data, which helps reduce both cost and cycle time.
3. Increase data fidelity as the evidence gets stronger. Their progression moves from little or no data, to synthetic data, to de-identified data, and eventually to real PHI only when the experiment justifies it. 
4. The goal is faster, higher-quality decisions, not just faster experiments. Synthetic data can help eliminate weak technologies or partners before teams spend months navigating legal, security, and procurement requirements.
5. Synthetic data does not replace real patients. It can work well for feasibility, proof of technology, and early proof of value, but real people are still essential when testing behavior, trust, experience, or clinical outcomes.
6. A safe sandbox makes experimentation possible. Creating an environment outside the core production system allowed Moses’ team to fail faster, learn safely, and generate evidence before asking the organization to make larger investments.
7. AI should free people to do more of the work only humans can do. In healthcare, Moses sees the opportunity in automating administrative work so clinicians and care managers can spend more time directly helping patients.

Guest Links

LinkedIn: https://www.linkedin.com/in/mosesadedoyin/

Summary

In this episode, I’m joined by Martin Eriksson, founder of Mind the Product and author of The Decision Stack, to talk about how organizations can make better decisions from strategy all the way down to everyday execution.

We talk about the difference between speed and velocity, why empowered teams still need strategic context, and how assumptions exist at every layer of an organization, not just inside individual products.

We also explore what AI changes and, maybe even more importantly, what it doesn’t. Building has become drastically faster and cheaper, but that doesn’t eliminate the need to understand the customer, the business, and whether or not you should be building something in the first place.

If you want to learn more about how strategic choices connect to everyday decisions and how assumptions can be tested throughout your organization you’ll love this episode.

Takeaways

1. Speed without direction is not enough. Teams can build, ship, and test faster than ever, but without strategic alignment they may simply move in different directions more quickly. Martin frames this as the difference between speed, velocity, and organizational momentum.

2. Empowered teams still need strategic context. Autonomy works best when teams understand where the business is going, how it plans to get there, what matters now, and how their work connects to those choices.

3. Assumptions exist at every layer of the organization. They are not limited to products or experiments. Vision, strategy, objectives, opportunities, and execution all rest on assumptions that can become dangerous when they remain unspoken.

4. Strategy is a series of connected choices. The value of a decision stack is not just having vision, strategy, objectives, opportunities, and principles—it is making sure those decisions connect from the top down and can be traced back from everyday work.

5. Use testing and discovery on strategic decisions, not only product decisions. Teams have become good at de-risking what is directly in front of them, but the same tools can be applied to bigger bets around strategy, markets, and the future of the business.

6. Not every assumption can be eliminated, but it should be named. At the strategy and vision level, some choices will always remain bets on the future. Making those assumptions explicit allows leaders and boards to monitor when conditions change and decide whether the risk is still acceptable.

7. AI changes the economics of building, not the fundamentals of product. Faster and cheaper delivery reduces feasibility risk, but teams still need to understand customer value, business value, usability, distribution, pricing, and whether something should be built at all.

Guest Links

Martin’s Website: https://martineriksson.com/

Martin’s LinkedIn: https://www.linkedin.com/in/martineriksson/

Coherent & Wrong: https://www.thedecisionstack.com/coherent-and-wrong/

Summary

In this episode, I’m joined by Teresa Torres. Teresa is an author, speaker and a product discovery coach who I’ve been a fan of for quite a long time. 

We chat about why assumptions look different depending on the level you’re working at, why showing your work is so important for alignment, and how using AI inside your courses can give students more opportunities to practice and receive feedback.

We also dig into AI evals - how Teresa uses them to measure the quality of her AI coaching tools, identify failure modes, and systematically improve their performance instead of just trusting that an LLM will get it right on the first try.

If you want to learn more about testing assumptions, designing better learning loops, and using AI without giving up your critical thinking skills, this episode is for you.

Takeaways

1. AI can create a safer space for learning. Teresa found that people are often willing to ask an AI questions they might feel embarrassed asking another person.

2. An AI coach works best when it is grounded in a clear teaching model. Teresa’s interview coach was effective because it was built on years of refined curriculum, rubrics, and explicit feedback criteria.

3. AI can dramatically increase opportunities for deliberate practice. Instead of waiting for an instructor, students can practice repeatedly and receive immediate, personalized feedback.

4. Building AI tools can improve the underlying curriculum. When an agent struggles with ambiguous instructions, it exposes gaps in how the material itself is taught.

5. Evals are simply a way to measure AI quality. Teresa uses evals to identify specific failure modes, track how often they occur, and test whether changes actually improve the agent.

6. Domain expertise still matters. Recognizing that an AI coach has given subtly bad advice often requires deep knowledge of the subject, not just technical skill.

7. Product teams should define what “good” looks like for AI. Generic quality metrics are not enough; teams need acceptance criteria and evals tied to the unique value their product is supposed to create.

8. AI changes the role of the teacher, not just the tools. Teresa is moving away from being the expert who simply delivers answers and toward creating environments where people can practice, get feedback, and learn for themselves.

Guest Links

ProductTalk Website: https://www.producttalk.org/

Teresa’s LinkedIn: https://www.linkedin.com/in/teresatorres/

Summary

In this episode, I’m joined by Barry O’Reilly,  he’s an entrepreneur, advisor, author and a friend of mine who works with executives to redesign how their organizations perform. He’s also the co-founder of Nobody Studios, an AI venture studio building and launching over 100 companies.

Barry and I chat about what he’s learned from years of helping corporations experiment, redesign systems, and make better decisions under extreme uncertainty.

We talk about why so many AI transformations start in the wrong place, why leaders should focus on the work before the tools, and how capturing conversations can create a kind of organizational memory that gets smarter over time.

Barry also shares his lessons from building Nobody Studios and how that work influenced his new book Artificial Organizations.

If you want to learn more about testing with AI to redesign how work gets done while making better decisions, you’ll love this episode.

Takeaways

1. Start with the work, not the AI tools. The biggest gains come from redesigning how work and decisions happen, then choosing technology that supports that flow.
2. Human judgment becomes more important, not less. AI can capture, synthesize, and model information at scale, but leaders still need to decide what matters and what action to take.
3. Treat conversations and decisions as data assets. Capturing meetings, transcripts, decisions, and outcomes creates organizational memory that can be reused instead of constantly recreating context.
4. Better systems can outperform experience alone. Deep domain expertise still matters, but rigorous decision-making systems combined with machine intelligence can challenge gut instinct as the default.
5. AI transformations fail when they are treated as tool rollouts. Buying Copilot or another platform does not change how an organization works unless behaviors, processes, and operating systems change with it.
6. Leaders can accelerate adoption by role-modeling experimentation. Admitting “I don’t know,” trying tools in real work, and openly sharing what works and what does not creates permission for others to learn.
7. Start with one decision or workflow. Rather than attempting a company-wide AI transformation, pick a recurring decision or process, test a new way of working, and learn from the result.

Guest Links

Barry’s Website: https://barryoreilly.com/

Barry’s LinkedIn: https://www.linkedin.com/in/barryoreilly/

Artificial Organization: https://artificialorganizations.com/

Nobody Studios: https://nobodystudios.com/

Summary

In this episode, I’m joined by Mark Graban, he’s an expert in Lean Leadership, Psychological Safety, and Continuous Improvement. 

Mark and I discuss the idea of the “smallest test of change,” a principle rooted in lean thinking that can be applied far beyond manufacturing. Speaking of manufacturing, we sort of geek out on what we’ve learned from being around Toyota and seeing how they work in practice, not just in theory.

We also dive into why organizations reward people for sounding certain, how leaders unintentionally punish experimentation, and why psychological safety is essential if you expect people to admit what they don’t know.

Mark shares how he’s applying these ideas to his own work by testing a new book through iterative publishing.

If you want to learn more about creating a culture where people feel safe to speak up and use smaller tests to drive meaningful change, this episode is for you.

Takeaways

1. Start with the smallest test of change - something quick, inexpensive, and focused on learning.
2. Separate knowledge from assumptions by asking, “How do we know this is true?” before acting with certainty.
3. Psychological safety is essential for experimentation because people must feel safe admitting mistakes, asking for help, and challenging ideas.
4. Leaders shape culture through their reactions. Saying failure is acceptable means little if people are punished when a test does not work.
5. Even experienced practitioners can overbuild. Mark’s AI coaching experiment reinforced the need to test demand and usefulness before adding more features.

Guest Links

Website: https://www.markgraban.com/

LinkedIn: https://www.linkedin.com/in/mgraban/

Summary

In this episode, I’m joined by Mike Yarbrough, he’s the founder of Rustic & Main, a custom jewelry company that’s known for crafting unique, story-driven wedding bands and engagement rings. They specialize in combining traditional heavy metals with unconventional, nostalgic inlays.

Mike shares how his handmade wooden wedding band accidentally started it all by generating unexpected customer demand, while still working his day job. We discuss how the business grew from a garage operation into a team producing handcrafted rings with deeply personal stories.

We also get into the product failure that led Mike and his team to create its durability-testing process (appropriately called “the ringer”) and how he uses prototypes and limited launches to test out new revenue opportunities.

This is a story about following market signals, learning from what breaks, and building a company without losing the craftsmanship and meaning that made customers care in the first place.

Takeaways

1. Customer pull can be one of the strongest early signals that an idea is worth pursuing.
2. Starting small gave Mike time to test demand, refine the product, and understand the business before leaving software development.
3. Product failures led Rustic & Main to create a more rigorous durability-testing process for every new design.
4. Customer requests helped the company expand from wooden rings into titanium, gold, custom inlays, and new product categories.
5. Testing a new business model, such as retail, requires patience because not every opportunity grows as quickly as the original business.

Guest Links

Rustic and Main: https://rusticandmain.com/

Summary

In this episode, I’m joined by Maggie Baker, she’s the Founder of Threadeco, an eco-friendly clothing company rethinking how people discover, buy, and keep fashion.

Maggie talks about how she started with a high-end fanny pack business that landed some serious celebrity traction, but still failed when the economy shifted in 2008. That experience taught her hard lessons about pricing, channel risk and adaptability.

Years later, she turned those insights into Threadeco, when she was inspired by imperfect produce boxes.

Maggie shares how she tested her early MVP with a Google Form, friends and family, hand-packed boxes, and no real payment system. Fast forward to today, business is thriving and we discuss how she is now blending retail, styling and subscriptions together.

It is a great conversation about learning the hard way, starting scrappy, listening to customers, and building a business model that can actually adapt.

Takeaways

1. A past failure became the foundation for a better business model. Maggie’s high-end fanny pack business got retail traction and celebrity attention, but the 2008–2009 downturn exposed pricing, channel, and adaptability risks she had not fully tested.

2. Testing does not have to be sophisticated to be useful. Threadeco started with a Google Form, friends-and-family beta testers, hand-curated clothing boxes, and manual feedback loops.

3. The first version of the model revealed the wrong customer. Early testing showed that customers who only wanted to try on many items and return most of them were not a good fit for Threadico’s economics.

4. Threadeco changed the model based on learning, not imitation. Instead of copying Stitch Fix, Maggie shifted to a prepaid five-piece box with limited exchanges, creating a model that better fit her inventory, cash flow, and customer behavior.

5. The brick-and-mortar store became more than a retail channel. The store functions as a warehouse, styling space, discovery engine, and subscription acquisition channel.

6. Location testing mattered. Moving within Old Sacramento changed the business dramatically, showing how even a two-minute difference in storefront location can create a completely different retail outcome.

7. The next big test is integrating physical retail, subscriptions, styling, and tech. Maggie is exploring an app that connects customer style profiles, boxes, in-store experiences, and stylist recommendations into one ecosystem.

Guest Links

Maggie’s LinkedIn: https://www.linkedin.com/in/getajobla/
Threadeco: https://www.threadeco.com/