What are the skills required for AI evals? Why data scientists have a natural advantage in AI evals? 

Evaluating AI isn’t just about "vibe coding" with an AI assistant. It actually requires a solid foundation in statistics for picking sample sizes and coding to build your own testing frameworks. Data scientists have a huge head start here because they are already pros at designing metrics and communicating risks. 

In the augural episode, we also explain why Evals (pre-launch testing) and Analytics (post-launch user feedback) are two sides of the same coin: one makes sure the AI works, and the other makes sure people actually love using it.

00:00 – Introduction to AI Evals & Analytics 
01:31 – Why Data Scientists Have a Natural Advantage
01:59 – Technical Pillar: Statistics 
02:48 – Technical Pillar: Coding & Prompt Engineering 
05:03 – Technical Pillar: Dataset Generation 
08:35 – Soft Skills & Stakeholder Collaboration 
11:17 – Domain Expertise in Regulated Industries 
15:50 – New Skills for the GenAI Era 
19:25 – Why Evals and Analytics Must Come Together 

Stella Liu: https://www.linkedin.com/in/wenxingl/
Amy Chen: https://www.linkedin.com/in/amy17519/

More about AI Evals and Analytics -- https://ai-evals.org/

We (Stella & Amy) created the AI Evaluation & Analytics Playbook, a practical framework that helps teams ship production-ready, trustworthy AI systems.

Powered by Firstory Hosting