Product Impact Podcast | Secrets to unlocking the value of AI

Search sends fewer people to your website every month, and the answer engines keep most of the ones it does send. Kevin Wong, VP of Design at Webflow, doesn't think that means the website is dead — he thinks it means the metric you have been judging it by is.

Kevin leads product design, brand studio, content and user research at Webflow, where millions of businesses build their sites, which gives him a read on what is happening across the market rather than inside one company's analytics. Brittany and Arpy get into what that view shows: why the people who still click convert far better, how a brand earns its way into being the answer, why building one website for everyone is now the wrong goal, and — in a stretch this show has never covered — how Webflow turned design itself into something agents can execute against.

What you'll learn in this episode:

➜ Why traffic stopped being the metric that matters, and what Webflow watches instead.

➜ Answer engine optimization: how brands get cited as the answer, and why it isn't SEO with a new name.

➜ The museum is over: designing many versions of a site and serving the right one, not one design for everyone.

➜ Why Kevin won't drop users into a blank prompt box, and where he draws the line on probabilistic software.

➜ Webflow's design system rebuilt as instructions — skills that lint and review designs, recipes for each view type.

➜ Evals before you build: define what good looks like upfront, or find out too late that it never was.

..................

Kevin Wong is VP of Design at Webflow, where he leads the Product Design, Brand Studio, Content, and User Research teams for a platform used by millions of designers, marketers, and developers to build enterprise-ready web experiences. His work focuses on shaping Webflow into an AI-native website experience platform that balances ease of use with professional power. Previously he led design for Meta's Remote Presence team across Facebook, Instagram and Messenger, and before that led Airbnb's global CX Platform Design organization, building the foundational service design capabilities and tools that supported guests, hosts, and agents.

➜ Kevin on LinkedIn: https://www.linkedin.com/in/kvwong

➜ Webflow: https://webflow.com

➜ Google and LLMs Are Destroying Your Website & Traffic — Arpy's analysis (Product Impact):

https://productimpactpod.com/news/google-and-llms-are-destroying-your-website-and-traffic

➜ SparkToro's research on zero-click search: https://sparktoro.com/blog/in-2026-less-than-one-third-of-google-searches-still-send-a-click/

..................

If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.

We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.

Hosted by:

➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/

➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Join our Slack community to get honest and strategic advice for how to navigate the AI era: https://join.slack.com/t/aiproductimpact/shared_invite/zt-49ccrauua-CUJGWYPRKI89TUu6SRrPJg

This episode was brought to you by:

➜ PH1 (https://ph1.ca) — We solve the first and last mile of AI: use cases, KPIs, evals, and scaling impact.

Most companies handed their people a powerful new machine and said "go do something with AI." Melissa M. Reeve, author of Hyperadaptive: Rewiring the Enterprise to Become AI-Native, argues that's the whole reason the returns haven't shown up — and the reason so many of us feel like we're sprinting to stay in place.

Melissa has spent three decades inside organizational transformation, and her case is that AI is an organizational design problem rather than a tools problem. Brittany and Arpy get into what that means for the people living it: where the jobs actually go when work gets automated, why the career ladder is dissolving into something else, why she's genuinely bullish on UX, and what to do when it doesn't feel safe to raise your hand and ask to slow down.

In this episode we cover:

➜ The "hand wave": how "go do something with AI" cascades down a company and produces activity without impact.

➜ Melissa's AI learning flywheel — activation hubs and AI leads, so not everyone learns every new model alone.

➜ When work gets automated, jobs shift to building, monitoring, and maintaining the thing that now does it.

➜ Roles are breaking down into skills. The ladder is going away, and portfolio careers are replacing it.

➜ Why Melissa is bullish on UX: the pie expands, and high-craft work gets more valuable, not less.

➜ The rocks-and-sand problem: protect the time AI frees up, or noise collapses straight back into it.

..................

Melissa M. Reeve is the author of Hyperadaptive: Rewiring the Enterprise to Become AI-Native (IT Revolution, with Ryan Martens). She has spent more than three decades at the intersection of executive leadership, organizational transformation, and process excellence, with a Lean foundation that started on a factory floor in Japan in the mid-nineties. She went on to run the Agile Marketing Alliance, bringing agile software practices into marketing, and now advises leaders on becoming AI-native. In the book she lays out a five-stage maturity path for rewiring how an organization actually operates — governance, champions, agentic AI, and reorganizing around customer value streams.

➜ Hyperadaptive (the book): https://www.melissareeve.com/book

➜ Melissa's site: https://www.melissareeve.com/

➜ Her course, Running Hyperadaptive Organizations: https://hyperadaptive.solutions

..................

If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.

We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.

Hosted by:

➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/

➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Join our Slack community to get honest and strategic advice for how to navigate the AI era: https://join.slack.com/t/aiproductimpact/shared_invite/zt-49ccrauua-CUJGWYPRKI89TUu6SRrPJg

This episode was brought to you by:

➜ PH1 (https://ph1.ca) — We solve the first and last mile of AI: use cases, KPIs, evals, and scaling impact.

Most companies are burning millions chasing an edge with AI and getting almost nothing back. Rebecca Hinds runs the Work AI Institute at Glean, and her research maps exactly what the small group of teams actually winning with AI do differently — and why everyone else stalls. In Glean's 2026 Work AI Index, 87% of workers said AI improves their work, but only 13% of their companies say they're actually performing better because of it. The winners closed that gap on purpose, and this episode is about how.

We get brutally honest with Rebecca about our own experience — feeling like the AI's assistant, drowning in unreliable output, riding waves of excitement and dread — and she ties each one back to the data: what the leading teams do to turn individual speed into real organizational value, the 6.4 hours a week most workers lose to "botsitting," why context (not more tokens or MCP connections) is the real unlock, why domain expertise decides who actually benefits, and how the best organizations measure value without chasing vanity metrics.

In this episode we cover:

➜ What the teams winning with AI actually do

➜ How top teams turn AI into real value, fast

➜ Why context beats bigger prompts and MCP

➜ Measure value with your old KPIs, not new ones

➜ The domain expertise that decides who wins

➜ Your team wastes a day a week botsitting AI

About Rebecca Hinds:

Rebecca Hinds, PhD, is a leading expert on organizational behavior and the future of work. For more than 15 years, she has helped organizations rethink how work gets done, challenging outdated management playbooks and offering leaders a more ambitious—and more human—vision for the future.

Rebecca has founded and led two first-of-their-kind research institutes dedicated to reinventing work. In 2022, she launched the Work Innovation Lab at Asana, building it into an influential source of practical workplace research. In 2025, she founded the Work AI Institute at Glean to help organizations navigate the next transformation of work.

Called a "professional meeting hacker" by Adam Grant, Rebecca is the author of Your Best Meeting Ever, an instant national bestseller named a must-read by the Next Big Idea Club. Her award-winning research has been featured in Harvard Business Review, The New York Times, The Wall Street Journal, TIME, Forbes, and Inc. She is a co-instructor of the CNBC Make It course How to Use AI to Be More Successful at Work and has served as a columnist for Inc. and Reworked. She also serves on the advisory boards of the Remote Work Institute and the Doodle Time Institute.

Rebecca holds a B.S., M.S., and Ph.D. from Stanford University, where her doctoral research examined how emerging technologies reshape the way people work together.

 https://www.linkedin.com/in/rebecca-hinds/

https://www.rebeccahinds.com/

https://www.glean.com/work-ai-institute

..................

Research featured in this episode:

➜ The Work AI Index (2026)  ⁠https://www.glean.com/work-ai-institute/reports/work-ai-index⁠

➜ The AI Transformation 100 (December 2025, with Bob Sutton) —  ⁠https://www.glean.com/work-ai-institute/reports/ai-transformation-100?page=1⁠

➜ The ETH Zurich vibe-coding study  ⁠https://ethz.ch/en/news-and-events/eth-news/news/2026/04/what-skills-do-people-need-to-successfully-program-with-ai.html⁠

..................

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

Join our Slack community to get honest and strategic advice for how to navigate the AI era: https://join.slack.com/t/aiproductimpact/shared_invite/zt-49ccrauua-CUJGWYPRKI89TUu6SRrPJg

This episode was brought to you by:

➜ PH1 (https://ph1.ca) — We solve the first and last mile of AI: use cases, KPIs, evals, and scaling impact.

Hosted by:

➜ Arpy Dragffy Guerrero — ⁠https://www.linkedin.com/in/adragffy/⁠

➜ Brittany Hobbs — ⁠https://www.linkedin.com/in/brittanyhobbs/⁠

Christian Crumlish watched Elon Musk's DOGE team take a buzzsaw to the federal government's own product organization — the teams he'd spent years helping build — and he has a specific, ungentle theory about what they got wrong and the surprising thing they may have gotten right.

But the deeper thread of this conversation is what happens to the rest of us as agents absorb more of the work and the cost of shipping falls toward zero. Christian walks us through the real math of replacing people with agents — and the "reverse centaur" trap waiting on the other side of it — why some AI salaries have stopped tracking value entirely, and his own mid-career decision to trade a bigger paycheck for work he could still defend to himself.

His argument is oddly hopeful. Your work still has meaning; it just might not live inside the industry or the company you built your career around. This is a conversation about reinventing your career in the age of agentic AI, from someone who has done it more than once.

In this episode you'll learn:

➜ Christian Crumlish on what Elon Musk's DOGE team got wrong tearing through civic tech — and what it may have improved.

➜ The reverse unicorn: Cory Doctorow's term for when your AI agent keeps the fun work and hands you the boring parts.

➜ AI engineers and PMs pulling $1M+ comp — Christian argues that points to something more than value-based outcomes.

➜ Why the meaning in your work may not live inside the industry or business you built your career around anymore.

➜ Building a "Product OS" — skillifying the tacit product knowledge locked in one person's head.

..................

Featured quotes:

"DOGE was never about efficiency. It was about rearranging the relationship between the government and Elon Musk's business interests." — Christian Crumlish

"The money doesn't come for no reason. It comes with expectations and strings attached." — Christian Crumlish

About Christian Crumlish:

Christian Crumlish has led product and UX teams for more than two decades — Director of Messaging Products at AOL, curator of Yahoo's design pattern library, Director of Product at CloudOn, and Director of Product at 18F and California's Office of Data and Innovation. He's the author of The Internet for Busy People, The Power of Many, Designing Social Interfaces, and Product Management for UX People, and he's currently building his own AI product management assistant, Piper Morgan.

Find Christian on LinkedIn: https://www.linkedin.com/in/mediajunkie/

His newsletter, What Now?: https://designinproduct.com

Christian's profile on Rosenfeld Media: https://rosenverse.rosenfeldmedia.com/people/christian-crumlish

..................

If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.

We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.

Hosted by:

➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/

➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

Join our Slack community to get honest and strategic advice for how to navigate the AI era: https://join.slack.com/t/aiproductimpact/shared_invite/zt-49ccrauua-CUJGWYPRKI89TUu6SRrPJg

Brought to you by:

➜ PH1 (https://ph1.ca) — We solve the first and last mile of AI: use cases, KPIs, evals, and scaling impact.

➜ AI Value Acceleration (https://aivalueacceleration.com)

Two years ago, Josh Clark and Veronika Kindred previewed a book about a coming era of sentient digital experiences on this show. That book, Sentient Design, is out now — and the future it predicted arrived faster than most companies' data, governance, or restraint could keep up with.

We talk with them about why most "agentic" features are still just a chat sidebar bolted onto an old product, what a gym-booking AI agent's very bad week reveals about designing for agents instead of people, and why Google running the biggest sentient design experiment alive — its own AI Overviews — is already rewriting how businesses get found.

In this episode we cover:

➜ Don't wait for perfect data — legacy data, not ambition, is blocking three live Fortune 50 sentient design projects.

➜ Personalization is quietly becoming individualization — reading the moment, not building a permanent profile.

➜ Defensive design: teaching AI systems humility and users healthy skepticism, so neither is taken at face value.

➜ Agents need their own front door — if you don't design one, somebody finds the back door instead.

➜ AI is a reckoning, not a feature: the real opportunity is removing interface entirely, not adding more of it.

➜ Google's AI Overviews are the biggest live sentient design experiment, and businesses never agreed to it.

..................

"Putting a chat sidebar on an existing thing, it's like bolting a spoiler onto a crummy Honda." — Josh Clark

"With LLMs, it's a really siloing technology." — Veronika Kindred

"We're not gonna have a single interface that swallows them all — although clearly OpenAI would like to see that happen." — Josh Clark

About Josh Clark & Veronika Kindred:

Josh Clark is principal of Big Medium, a design studio working with organizations on next-generation interfaces, and coined the term "Sentient Design" in 2024. Veronika Kindred is a designer and researcher at Big Medium who leads Sentient Design workshops for startups and Fortune 100 companies. Together they're the co-authors of Sentient Design: Crafting Intelligent Interfaces with AI (Rosenfeld Media, 2026) — and returning guests, having first previewed the book on this show in October 2024.

Find Josh on LinkedIn: https://www.linkedin.com/in/joshclark/

Find Veronika on LinkedIn: https://www.linkedin.com/in/vkind/

Get the book: https://rosenfeldmedia.com/books/sentient-design/

Josh & Veronika set up a coupon code for Product Impact listeners: 20% off Sentient Design at Rosenfeld Media.

➜ Use coupon code: SENTIENT-PI at https://rosenfeldmedia.com/books/sentient-design/

➜ Expires September 12

..................

If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.

We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.

Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

Hosted by:

➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/

➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:

➜ PH1 (https://ph1.ca) — a strategy & research consultancy specialized in delivering evidence about the highest value use cases and customer profiles.

➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

Metalab has spent the last 20 years designing products for Windsurf, Midjourney, Suno, The Atlantic, and Uber. AI has made feature parity between products fast and cheap — anyone can ship the same baseline in a fraction of the time it used to take. Design Director Myles Palmer's answer to that isn't "taste." He thinks that word is a hollow way to dodge a much harder question.

Myles discusses what actually separates Metalab's AI-native design process from how most teams are building right now, why a vibe-coded prototype can create a dangerous false sense of confidence, and what design looks like once a product moves past the chat box.

In this episode we cover:

➜ Why Metalab's 20-year discipline, not any single tool, is what actually differentiates its AI-native design process.

➜ Myles Palmer on why "taste" is a hollow answer, and what actually differentiates once AI makes feature parity cheap.

➜ Brittany's own vibe-coded travel app cost thousands in API calls and still lost to Google — the false confidence trap.

➜ What founders actually pay for later when they skip design early, and why it defends against a "sea of average."

➜ What AI design looks like past the chat box — ambient transit, try-on shopping, and rethinking university sites.

➜ Digital twins, the 23andMe cautionary tale, and why Apple's health-data model is where consent design is headed.

.

.................

"I hate the idea of taste. The idea that taste is this differentiator drives me crazy because it's like, well, everybody's taste is different anyway." — Myles Palmer

"Investing early in design defends against that sea of average." — Myles Palmer

.................

If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.

We built ⁠https://productimpactpod.com⁠ to be your AI product insights and strategic playbook hub. Check it out.

Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

.................

Myles Palmer is a Design Director at Metalab, where he leads and shapes meaningful products and experiences for both startups and global brands. Before joining Metalab, he founded and led Companion, a design studio built on strategic creativity and sustainable team culture. Myles also founded Pair Up, a platform connecting thousands of creatives for mentorship and peer learning. His work blends strategic thinking with human-centered design and thoughtful leadership that elevates product experiences.

LinkedIn: https://www.linkedin.com/in/myles-palmer-b1b70519/

Metalab Website: www.metalab.com

.

.................

Hosted by:

➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/

➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:

➜ PH1 (https://ph1.ca) — Elevate your AI-Powered Product & Customer Experience. PH1 is the leading AI UX strategy consultant, leveraging behavioural science and model evaluations to gain you a competitive advantage. We've worked on innovative AI product discovery & corporate AI CX transformations.

➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

Tech workers are so pessimistic about their own careers that most wouldn't recommend the field to a friend starting out. Eighty-two percent say AI made them more productive — but the bar resets every quarter, so the exhaustion never ends. Burnout hit 55.7%, up from 44.7% a year ago.

Noam Segal and Lenny Rachitsky's 2026 survey put a number on that pessimism: the industry's overall willingness to recommend a tech career is negative 39. Even directors land at 33% negative; founders, the most optimistic group surveyed, still score negative 5. How AI has changed your sense of yourself as a professional — Amplified, Redefined, Destabilized, Diminished, Unchanged — predicts that outlook better than role or seniority.

Glean's Work AI Index names the hidden cost behind the productivity number: "botsitting," the 6.4 hours a week the average worker spends supervising and correcting AI instead of it saving them time. One respondent put it bluntly: "My brain is rotting. My work feels worse." Another: "I just follow Claude. I don't understand what I merge."

Join  the new AI mentorship Slack community here: tinyurl.com/productimpactslack.

In this episode we cover:

➜ 82% of tech workers say AI makes them more productive; 55.7% say they're more burned out than a year ago.

➜ A 5-way AI identity split — Amplified to Unchanged — predicts career outlook better than role or seniority.

➜ The paradox of getting good at AI: the more confident you get, the more replaceable your output feels.

➜ Optimism climbs sharply from manager to director to VP to founder — revealing who AI adoption actually serves.

➜ Three postures — AI-Pilled, AI-Learning, AI-Governing — decide where research and adjacent roles go next.

➜ Why AI that's "impressive in a demo" keeps failing multi-turn users — a pattern this show keeps finding.

..................

If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.

We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.

Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

Hosted by:

➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/

➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:

➜ PH1 (https://ph1.ca) — a strategy & research consultancy specialized in delivering evidence about the highest value use cases and customer profiles.

➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

...........

Sources referenced in this episode:

Lenny's 2026 Tech Survey: AI Burnout Is Surging, Layoff Fear Is High — https://productimpactpod.com/news/lennys-2026-tech-survey-ai-burnout-layoff-fear

How tech workers are feeling in 2026 (original survey) — https://www.lennysnewsletter.com/p/how-tech-workers-are-feeling-in-2026

Writer's 2026 enterprise AI adoption survey — https://writer.com/blog/enterprise-ai-adoption-2026/

New Report Says You're Wasting More Time Botsitting Than Getting Value from AI — https://productimpactpod.com/news/botsitting-work-ai-index-2026/

The Future of UX Researchers — https://productimpactpod.com/news/future-of-ux-researchers

s02e16 — Moritz Sudhof on AI's invisible failures — https://open.spotify.com/episode/13d4ua1scY81zmQNQEkOXy

s02e17 — Microsoft Copilot's UXR team on UX Evals and loss patterns — https://open.spotify.com/episode/3V08ELNqgGyMpypYeSgfXP

In our last episode, Stanford NLP researcher Dr. Moritz Sudhof showed that 79% of AI product failures are invisible — they don't fire alerts, don't surface in telemetry, and don't get flagged by users, and they quietly erode trust and accelerate churn. This episode is the operational follow-up: the team that built a method for catching exactly that class of failure.

Microsoft's Copilot UX research team spent a year running evaluations on real conversations — every archetype, every industry, every use case — and found that more than half of their quality failures weren't in any eval they were running. At the world's most widely deployed enterprise AI product, with sophisticated engineering and testing infrastructure, standard evals were still missing the majority of what users actually experienced as failure.

That finding isn't limited to Copilot's scale. It's a structural gap in how the industry evaluates AI quality — and if you're running automated evals and calling that sufficient, the gap in your own product is almost certainly larger than you know.

In this episode we cover:

• Token usage and adoption tell you if your AI is being used — not whether it's actually working for anyone.
• Users bring real prompts, test one model fully, then compare — that's what produces honest signal at scale.
• More than half of Copilot's loss patterns — user-driven gaps in model behavior — weren't in any existing eval.
• LLM judges get you to baseline quality. Users catch what automated testing structurally cannot.
• The flywheel: UXR evals → loss pattern taxonomy → log inspection → prompt changes → retention gains.
• This team started with 10 users and one comparative question. Signal strong enough to scale to an entire org.

"More than half of the loss patterns that we've detected were not things that we were measuring in our evals." — Wendy Wang

About the team: This work was developed by Christopher Monnier, Wendy Wang, and Chuck Kwong, UX researchers on the Microsoft Copilot team. Together, they built and continue to refine an interactive evaluation method that brings real user tasks, side-by-side product comparisons, quantitative results, and qualitative feedback into one process. The team uses this work to identify where Copilot succeeds and where people run into issues, understand the reasons behind user preferences, and turn the findings into clear opportunities for product and prompt teams to improve the experience.

• Christopher Monnier on LinkedIn: https://www.linkedin.com/in/christophermonnier/
• Chuck Kwong on LinkedIn: https://www.linkedin.com/in/charleskwong/
• Wendy Wang on LinkedIn: https://www.linkedin.com/in/wendy-wang-mertensmeyer-8386b018/
• Microsoft Copilot: https://www.microsoft.com/en-us/microsoft-copilot

..................

If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.

We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.

Hosted by:

➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/

➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:

➜ PH1 (https://ph1.ca) — a strategy & research consultancy specialized in pinpointing how to best leverage AI and improve the impact of your AI product

➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

KPMG pulled a report this year after its team accepted hallucinated information without question. Latham & Watkins submitted a court filing built on fabricated legal citations an AI invented, delivered with total confidence and perfect formatting. McDonald's AI ordering chatbot became a public failure for the same underlying reason: the system looked like it was working. Dr. Moritz Sudhof, CEO of Bigspin AI, analyzed 100,000 real conversations between users and live AI systems with Stanford NLP Group's Chris Potts and found why: 79% of AI failures are invisible to standard monitoring because they are behavioral failures, not technical ones.

Sudhof built this research on hard-won experience. As VP of AI at BetterUp, he shipped an AI coach that beat ChatGPT on every expert coaching benchmark and still lost users — until his team changed nothing but how the AI introduced itself, and outcomes doubled. That gap between what an eval measures and what actually happens in the room with a user became his research question, and then his company.

In this episode:

• The Confidence Trap: AI states something false, dressed in precise numbers and total confidence.
• Silent Mismatch: when the AI can't finish a task, it quietly answers a different question instead.
• The Drift and the Death Spiral: how a long conversation loses the plot and burns the user's patience.
• Models are post-trained to answer, not clarify — a default that gets worse as models get more capable.
• The Paradox of AI Fluency: the users who push back hardest also hit the most failures, and succeed most.
• Evals catch what you already know to test for. Reading real transcripts catches what you don't.

"The behavioral layer is where most of the damage is actually happening." — Moritz Sudhof

"The more specific and helpful it often gets, the more fake it often is." — Moritz Sudhof, on the Confidence Trap

About Moritz: Shipped conversational AI to hundreds of organizations as VP of AI at BetterUp. Lived the problem of knowing something's wrong but not what to fix.

Guest resources:

• LinkedIn: https://www.linkedin.com/in/sudhof/
• X: https://x.com/mmooritz
• Personal site: https://msudhof.com/
• Bigspin website: https://bigspin.ai

Key research:

• Invisible Failures in Human–AI Interactions: https://arxiv.org/abs/2603.15423
• A paradox of AI fluency: https://arxiv.org/abs/2604.25905

We built productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.

Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

Hosted by:

• Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
• Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:

PH1 (https://ph1.ca) — a strategy & research consultancy specialized in delivering evidence about the highest value use cases and customer profiles. AI Value Acceleration

(https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

Four data reports from 2026 tell a consistent story, and none of it matches the adoption narrative. Writer surveyed 2,400 global workers and C-suite leaders: 97% of executives deployed AI agents in the past twelve months, 29% reported significant ROI. Glean's Work AI Index found a name for what most knowledge workers are actually experiencing: botsitting — spending more time supervising and correcting AI than gaining anything back. Section's biannual proficiency survey: 67% of workers use AI weekly, 5.5% are proficient enough to generate consistent value, and 79% of managers haven't demonstrated their own AI use to their team in the past month. Token consumption per organization grew roughly 320 times in twelve months while the share of organizations reporting significant ROI stayed at 29%.

Brittany Hobbs and Arpy Dragffy work through what's causing the gap, why the teams trying hardest to close it keep hitting structural walls, and what it takes to move from measuring adoption to generating defensible value — for individual contributors, for teams, and for the organizations responsible for this investment.

What you'll learn:

• OpenAI, Writer, Glean, Section: four reports, one consistent signal — the adoption story hides the value failure.
• Glean 2026: botsitting is the dominant AI experience. More knowledge workers are losing time to AI than gaining it.
• 67% of workers use AI weekly. Only 5.5% are proficient. The problem isn't more training days. It's the model of change.
• Four years of measuring seats over outcomes has left AI leaders unable to defend their budgets. The window is closing.
• Salesforce agreed to acquire Fin for $3.6B. What they built before that exit is the lesson most orgs are ignoring.
• Boris Cherny no longer prompts — he builds loops. What that means for every team not yet running autonomous evaluation.

Articles referenced in this episode:

• 97% of Executives Deployed AI Agents. Only 29% See ROI. — Brittany's breakdown of the Writer 2026 survey and the 68-point deployment-to-value gap
• The 10% Problem: AI's Value Gap Is Wider Than Anyone Is Admitting — Why AI value is concentrating at the top and what it means for the rest of the organization
• WTF is an AI-native org anyways? Let's compare Airbnb & Meta's opposing plans. — The competing models for AI-native organization design
• OpenAI & Anthropic are charging us way more than we need — Arpy on token economics, model selection, and the cost side of AI value creation

We built productimpactpod.com to be your AI product strategy and AI product news hub. Check it out.

Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

Hosted by:

• Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
• Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by: PH1 (https://ph1.ca) — an AI strategy consultancy specialized in improving the measurable success of AI products. AI Value Acceleration

(https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.