Navigating the AI Frontier: The Power of Synthetic Data and Agent Evaluations in LLM Development // MLOps podcast #241 with Boris Selitser, Co-Founder and CTO/CPO of Okareo.

A big thank you to LatticeFlow for sponsoring this episode! LatticeFlow - https://latticeflow.ai/

// Abstract

Explore the evolving landscape of building LLM applications, focusing on the critical roles of synthetic data and agent evaluations. Discover how synthetic data enhances model behavior description, prototyping, testing, and fine-tuning, driving robustness in LLM applications. Learn about the latest methods for evaluating complex agent-based systems, including RAG-based evaluations, dialog-level assessments, simulated user interactions, and adversarial models. This talk delves into the specific challenges developers face and the tradeoffs involved in each evaluation approach, providing practical insights for effective AI development.

// Bio

Boris is the Co-Founder and CTO/CPO at Okareo. Okareo is a full-cycle platform for developers to evaluate and customize AI/LLM applications. Before Okareo, Boris was Director of Product at Meta/Facebook, leading teams building internal platforms and ML products. Examples include a copyright classification system across the Facebook apps and an engagement platform for over 200K developers, 500K+ creators, and 12M+ Oculus users. Boris has a bachelor’s in Computer Science from UC Berkeley.

// MLOps Jobs board

https://mlops.pallet.xyz/jobs

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

https://docs.okareo.com/blog/data_loop

https://docs.okareo.com/blog/agent_eval

The Real E2E RAG Stack // Sam Bean // MLOps Podcast #217 - https://youtu.be/8uZst7pgOw0

--------------- ✌️Connect With Us ✌️ -------------

Join our slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Boris on LinkedIn: https://www.linkedin.com/in/selitser/

Timestamps:

[00:00] Boris' preferred coffee

[00:37] Takeaways

[02:32] Please like, share, leave a review, and subscribe to our MLOps channels!

[02:48] Software Engineering and Data Science

[06:01] AI Transformative Potential Explained

[10:31] Prompt Injection Protection Strategies

[17:03] Agent's metrics for Jira

[24:11] Data and Metrics Evolution

[27:54] Evaluation Focus Enhances Systems

[31:22 - 32:52] LatticeFlow AD

[32:55] Custom Evaluation and Synthetic Data

[36:23] Synthetic data for expansion, evaluation, and map

[41:06] Diverse agents' personalities for readiness

[44:25] Agent functions

[46:17] Optimizing Routing Agents

[50:04] Adapting to tool output for decision-making

[52:56] Agent framework evolution

[55:41] Agent framework for delivering value

[57:03] Wrap up