Sign up to save your podcasts
Or
Share Episode 53: Human-Seeded Evals & Self-Tuning Agents: Samuel Colvin on Shipping Reliable LLMs
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Episode 53: Human-Seeded Evals & Self-Tuning Agents: Samuel Colvin on Shipping Reliable LLMs
Demos are easy; durability is hard. Samuel Colvin has spent a decade building guardrails in Python (first with Pydantic, now with Logfire), and he’s convinced most LLM failures have nothing to do with the model itself. They appear where the data is fuzzy, the prompts drift, or no one bothered to measure real-world behavior. Samuel joins me to show how a sprinkle of engineering discipline keeps those failures from ever reaching users.
We talk through:
• Tiny labels, big leverage: how five thumbs-ups/thumbs-downs are enough for Logfire to build a rubric that scores every call in real time
• Drift alarms, not dashboards: catching the moment your prompt or data shifts instead of reading charts after the fact
• Prompt self-repair: a prototype agent that rewrites its own system prompt—and tells you when it still doesn’t have what it needs
• The hidden cost curve: why the last 15 percent of reliability costs far more than the flashy 85 percent demo
• Business-first metrics: shipping features that meet real goals instead of chasing another decimal point of “accuracy”
If you’re past the proof-of-concept stage and staring down the “now it has to work” cliff, this episode is your climbing guide.
LINKS
🎓 Learn more:
📺 Watch the video version on YouTube: YouTube link
View all episodes
By Hugo Bowne-Anderson
5
1111 ratings
Demos are easy; durability is hard. Samuel Colvin has spent a decade building guardrails in Python (first with Pydantic, now with Logfire), and he’s convinced most LLM failures have nothing to do with the model itself. They appear where the data is fuzzy, the prompts drift, or no one bothered to measure real-world behavior. Samuel joins me to show how a sprinkle of engineering discipline keeps those failures from ever reaching users.
We talk through:
• Tiny labels, big leverage: how five thumbs-ups/thumbs-downs are enough for Logfire to build a rubric that scores every call in real time
• Drift alarms, not dashboards: catching the moment your prompt or data shifts instead of reading charts after the fact
• Prompt self-repair: a prototype agent that rewrites its own system prompt—and tells you when it still doesn’t have what it needs
• The hidden cost curve: why the last 15 percent of reliability costs far more than the flashy 85 percent demo
• Business-first metrics: shipping features that meet real goals instead of chasing another decimal point of “accuracy”
If you’re past the proof-of-concept stage and staring down the “now it has to work” cliff, this episode is your climbing guide.
LINKS
🎓 Learn more:
📺 Watch the video version on YouTube: YouTube link
More shows like Vanishing Gradients
View all
a16z Podcast
1,033 Listeners
Data Skeptic
480 Listeners
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
441 Listeners
Super Data Science: ML & AI Podcast with Jon Krohn
298 Listeners
NVIDIA AI Podcast
331 Listeners
DataFramed
267 Listeners
Practical AI
192 Listeners
Google DeepMind: The Podcast
198 Listeners
Machine Learning Street Talk (MLST)
88 Listeners
Dwarkesh Podcast
409 Listeners
No Priors: Artificial Intelligence | Technology | Startups
121 Listeners
Latent Space: The AI Engineer Podcast
75 Listeners
AI + a16z
31 Listeners
High Signal: Data Science | Career | AI
4 Listeners
OpenAI Podcast
28 Listeners