AI Evals and Analytics Podcast

Why do most AI teams only ask "is this actually working for the business?" after it's too late? When should you start connecting evals to business impact and how do you actually do it?

Using the same medical insurance chatbot from the last episode, we show how to bridge the gap between model metrics and the outcomes your leadership actually cares about. We introduce the Eval-to-Impact Stack: a three-layer framework that connects eval metrics, product metrics, and business metrics. 

• More details are available in our Substack post: From AI Evals to Business Impact

• Interested in AI Evals and Analytics Playbook course? Here is an exclusive discount for our listeners

00:00 – Introduction & Recap of Episode 2
00:53 – Why Teams Ask the Business Impact Question Too Late
01:38 – The Stat: 95% of Enterprise AI Pilots Fail
01:58 – The Translation Problem: Model Metrics vs. Business Metrics
02:38 – Why Evals Get Labeled as Overhead (And How to Fix It)
03:16 – The Eval-to-Impact Stack: Three Layers Explained
05:00 – Applying the Framework: Insurance Chatbot Walkthrough
07:13 – Work Backwards from Business Goals, Not Forward from Metrics
08:05 – The Cross-Functional Superpower: Speaking Both Languages
08:25 – Closing: "Build the Product Right" vs. "Build the Right Product"

Stella Liu: https://www.linkedin.com/in/wenxingl/
Amy Chen: https://www.linkedin.com/in/amy17519/

More about AI Evals and Analytics -- https://ai-evals.org/

We (Stella & Amy) created the AI Evaluation & Analytics Playbook, a practical framework that helps teams ship production-ready, trustworthy AI systems.

What is Evaluation-driven development? When should you start building evals for your product? How to build it from scrach?

Using a real-world example of a customer chatbot for a medical insurance company, we walk through the process of setting up evals from scratch: translating product requirements into quantifiable metrics, curating quality test datasets (hint: you need fewer examples than you think), and making go/no-go decisions based on eval scores.

You'll learn why accuracy and safety require different approaches, how to avoid the trap of AI-generated test data, and why 94% vs 95% accuracy matters less than you'd expect—but safety guardrails are non-negotiable. This is the practical blueprint for anyone building AI products who wants to catch problems before users do.

00:00 – Introduction: Why We Need to Talk About Evals Now
00:39 – When to Start AI Evals?
03:20 – Example Setup: Medical Insurance Customer Chatbot
04:30 – Defining Evals in Product Requirements
07:19 – What Is Evaluation-Driven Development?
08:27 – Breaking Down "Accuracy": What Does It Really Mean?
09:42 – Dataset Curation: Quality Over Quantity
11:24 – How Big Should Your Test Set Be?
12:25 – Safety Guardrails: Knowledge Boundary and PII Leakage
15:29 – Making Release Decisions with Eval Metrics
17:33 – Start with What's Critical to Your Use Case

Stella Liu: https://www.linkedin.com/in/wenxingl/
Amy Chen: https://www.linkedin.com/in/amy17519/

More about AI Evals and Analytics -- https://ai-evals.org/

We (Stella & Amy) created the AI Evaluation & Analytics Playbook, a practical framework that helps teams ship production-ready, trustworthy AI systems.

What are the skills required for AI evals? Why data scientists have a natural advantage in AI evals? 

Evaluating AI isn’t just about "vibe coding" with an AI assistant. It actually requires a solid foundation in statistics for picking sample sizes and coding to build your own testing frameworks. Data scientists have a huge head start here because they are already pros at designing metrics and communicating risks. 

In the augural episode, we also explain why Evals (pre-launch testing) and Analytics (post-launch user feedback) are two sides of the same coin: one makes sure the AI works, and the other makes sure people actually love using it.

00:00 – Introduction to AI Evals & Analytics 
01:31 – Why Data Scientists Have a Natural Advantage
01:59 – Technical Pillar: Statistics 
02:48 – Technical Pillar: Coding & Prompt Engineering 
05:03 – Technical Pillar: Dataset Generation 
08:35 – Soft Skills & Stakeholder Collaboration 
11:17 – Domain Expertise in Regulated Industries 
15:50 – New Skills for the GenAI Era 
19:25 – Why Evals and Analytics Must Come Together 

Stella Liu: https://www.linkedin.com/in/wenxingl/
Amy Chen: https://www.linkedin.com/in/amy17519/

More about AI Evals and Analytics -- https://ai-evals.org/

We (Stella & Amy) created the AI Evaluation & Analytics Playbook, a practical framework that helps teams ship production-ready, trustworthy AI systems.