Sign up to save your podcasts
Or
Share Episode 53: Human-Seeded Evals & Self-Tuning Agents: Samuel Colvin on Shipping Reliable LLMs
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Episode 53: Human-Seeded Evals & Self-Tuning Agents: Samuel Colvin on Shipping Reliable LLMs
Demos are easy; durability is hard. Samuel Colvin has spent a decade building guardrails in Python (first with Pydantic, now with Logfire), and he’s convinced most LLM failures have nothing to do with the model itself. They appear where the data is fuzzy, the prompts drift, or no one bothered to measure real-world behavior. Samuel joins me to show how a sprinkle of engineering discipline keeps those failures from ever reaching users.
We talk through:
• Tiny labels, big leverage: how five thumbs-ups/thumbs-downs are enough for Logfire to build a rubric that scores every call in real time
• Drift alarms, not dashboards: catching the moment your prompt or data shifts instead of reading charts after the fact
• Prompt self-repair: a prototype agent that rewrites its own system prompt—and tells you when it still doesn’t have what it needs
• The hidden cost curve: why the last 15 percent of reliability costs far more than the flashy 85 percent demo
• Business-first metrics: shipping features that meet real goals instead of chasing another decimal point of “accuracy”
If you’re past the proof-of-concept stage and staring down the “now it has to work” cliff, this episode is your climbing guide.
LINKS
🎓 Learn more:
📺 Watch the video version on YouTube: YouTube link
View all episodes
By Hugo Bowne-Anderson
5
1111 ratings
Demos are easy; durability is hard. Samuel Colvin has spent a decade building guardrails in Python (first with Pydantic, now with Logfire), and he’s convinced most LLM failures have nothing to do with the model itself. They appear where the data is fuzzy, the prompts drift, or no one bothered to measure real-world behavior. Samuel joins me to show how a sprinkle of engineering discipline keeps those failures from ever reaching users.
We talk through:
• Tiny labels, big leverage: how five thumbs-ups/thumbs-downs are enough for Logfire to build a rubric that scores every call in real time
• Drift alarms, not dashboards: catching the moment your prompt or data shifts instead of reading charts after the fact
• Prompt self-repair: a prototype agent that rewrites its own system prompt—and tells you when it still doesn’t have what it needs
• The hidden cost curve: why the last 15 percent of reliability costs far more than the flashy 85 percent demo
• Business-first metrics: shipping features that meet real goals instead of chasing another decimal point of “accuracy”
If you’re past the proof-of-concept stage and staring down the “now it has to work” cliff, this episode is your climbing guide.
LINKS
🎓 Learn more:
📺 Watch the video version on YouTube: YouTube link
More shows like Vanishing Gradients
View all
Data Skeptic
476 Listeners
a16z Podcast
1,083 Listeners
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
434 Listeners
Super Data Science: ML & AI Podcast with Jon Krohn
302 Listeners
NVIDIA AI Podcast
340 Listeners
DataFramed
268 Listeners
Practical AI
210 Listeners
Google DeepMind: The Podcast
194 Listeners
Machine Learning Street Talk (MLST)
89 Listeners
Dwarkesh Podcast
489 Listeners
No Priors: Artificial Intelligence | Technology | Startups
133 Listeners
Latent Space: The AI Engineer Podcast
97 Listeners
AI + a16z
33 Listeners
High Signal: Data Science | Career | AI
18 Listeners
OpenAI Podcast
52 Listeners