Sign up to save your podcasts
Or
Share Episode 50: A Field Guide to Rapidly Improving AI Products -- With Hamel Husain
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Episode 50: A Field Guide to Rapidly Improving AI Products -- With Hamel Husain
If we want AI systems that actually work, we need to get much better at evaluating them, not just building more pipelines, agents, and frameworks.
In this episode, Hugo talks with Hamel Hussain (ex-Airbnb, GitHub, DataRobot) about how teams can improve AI products by focusing on error analysis, data inspection, and systematic iteration. The conversation is based on Hamel’s blog post A Field Guide to Rapidly Improving AI Products, which he joined Hugo’s class to discuss.
They cover:
🔍 Why most teams struggle to measure whether their systems are actually improving
📊 How error analysis helps you prioritize what to fix (and when to write evals)
🧮 Why evaluation isn’t just a metric — but a full development process
⚠️ Common mistakes when debugging LLM and agent systems
🛠️ How to think about the tradeoffs in adding more evals vs. fixing obvious issues
👥 Why enabling domain experts — not just engineers — can accelerate iteration
If you’ve ever built an AI system and found yourself unsure how to make it better, this conversation is for you.
LINKS
🎓 Learn more:
📺 Watch the video version on YouTube: YouTube link
View all episodes
By Hugo Bowne-Anderson
5
1111 ratings
If we want AI systems that actually work, we need to get much better at evaluating them, not just building more pipelines, agents, and frameworks.
In this episode, Hugo talks with Hamel Hussain (ex-Airbnb, GitHub, DataRobot) about how teams can improve AI products by focusing on error analysis, data inspection, and systematic iteration. The conversation is based on Hamel’s blog post A Field Guide to Rapidly Improving AI Products, which he joined Hugo’s class to discuss.
They cover:
🔍 Why most teams struggle to measure whether their systems are actually improving
📊 How error analysis helps you prioritize what to fix (and when to write evals)
🧮 Why evaluation isn’t just a metric — but a full development process
⚠️ Common mistakes when debugging LLM and agent systems
🛠️ How to think about the tradeoffs in adding more evals vs. fixing obvious issues
👥 Why enabling domain experts — not just engineers — can accelerate iteration
If you’ve ever built an AI system and found yourself unsure how to make it better, this conversation is for you.
LINKS
🎓 Learn more:
📺 Watch the video version on YouTube: YouTube link
More shows like Vanishing Gradients
View all
a16z Podcast
1,032 Listeners
Data Skeptic
480 Listeners
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
441 Listeners
Super Data Science: ML & AI Podcast with Jon Krohn
298 Listeners
NVIDIA AI Podcast
322 Listeners
DataFramed
267 Listeners
Practical AI
192 Listeners
Google DeepMind: The Podcast
198 Listeners
Machine Learning Street Talk (MLST)
88 Listeners
Dwarkesh Podcast
408 Listeners
No Priors: Artificial Intelligence | Technology | Startups
121 Listeners
Latent Space: The AI Engineer Podcast
75 Listeners
AI + a16z
31 Listeners
High Signal: Data Science | Career | AI
4 Listeners
OpenAI Podcast
28 Listeners