Share Debugging AI Products: From Data Leakage to Evals with Hamel Husain

Copy link

October 02, 2025

Debugging AI Products: From Data Leakage to Evals with Hamel Husain

1 hour 26 minutes

Guest: Hamel Husain

AI products and problems discussed:

GitHub Copilot

Forecasting AirBnB Guest Growth

- NurtureBoss

Resources & Links

Hamel’s blog on AI evals

AI Evals for Engineers and PMs course on Maven (Get 35% off with this affiliate link)

Chapters:

00:00 Introduction to Hamel Hussein

00:34 Challenges in AI Consulting

02:00 Machine Learning Fundamentals

04:47 Debugging Machine Learning Models

05:00 Case Study: Airbnb's Guest Growth

08:51 Understanding Machine Learning Models

18:35 Introduction to Nurture Boss

25:40 Building AI Products with Synthetic Data

41:20 Connecting Machine Learning to Error Analysis

42:28 Real-World Example: Text Message Errors

44:15 Prioritizing and Documenting Errors

45:59 Continuous Improvement and Iteration

58:08 Using Synthetic Data for Evaluation

01:08:42 Avoiding Overfitting in Evaluations

01:19:28 Practical Tips for Error Analysis

01:25:10 Final Thoughts and Resources

...more

View all episodes

By Teresa Torres