Sign up to save your podcasts
Or
Share Episode 51: Why We Built an MCP Server and What Broke First
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Episode 51: Why We Built an MCP Server and What Broke First
What does it take to actually ship LLM-powered features, and what breaks when you connect them to real production data?
In this episode, we hear from Philip Carter — then a Principal PM at Honeycomb and now a Product Management Director at Salesforce. In early 2023, he helped build one of the first LLM-powered SaaS features to ship to real users. More recently, he and his team built a production-ready MCP server.
We cover:
• How to evaluate LLM systems using human-aligned judges
• The spreadsheet-driven process behind shipping Honeycomb’s first LLM feature
• The challenges of tool usage, prompt templates, and flaky model behavior
• Where MCP shows promise, and where it breaks in the real world
If you’re working on LLMs in production, this one’s for you!
LINKS
🎓 Learn more:
📺 Watch the video version on YouTube: YouTube link
View all episodes
By Hugo Bowne-Anderson
5
1111 ratings
What does it take to actually ship LLM-powered features, and what breaks when you connect them to real production data?
In this episode, we hear from Philip Carter — then a Principal PM at Honeycomb and now a Product Management Director at Salesforce. In early 2023, he helped build one of the first LLM-powered SaaS features to ship to real users. More recently, he and his team built a production-ready MCP server.
We cover:
• How to evaluate LLM systems using human-aligned judges
• The spreadsheet-driven process behind shipping Honeycomb’s first LLM feature
• The challenges of tool usage, prompt templates, and flaky model behavior
• Where MCP shows promise, and where it breaks in the real world
If you’re working on LLMs in production, this one’s for you!
LINKS
🎓 Learn more:
📺 Watch the video version on YouTube: YouTube link
More shows like Vanishing Gradients
View all
a16z Podcast
1,032 Listeners
Data Skeptic
480 Listeners
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
441 Listeners
Super Data Science: ML & AI Podcast with Jon Krohn
298 Listeners
NVIDIA AI Podcast
322 Listeners
DataFramed
267 Listeners
Practical AI
192 Listeners
Google DeepMind: The Podcast
198 Listeners
Machine Learning Street Talk (MLST)
88 Listeners
Dwarkesh Podcast
408 Listeners
No Priors: Artificial Intelligence | Technology | Startups
121 Listeners
Latent Space: The AI Engineer Podcast
75 Listeners
AI + a16z
31 Listeners
High Signal: Data Science | Career | AI
4 Listeners
OpenAI Podcast
28 Listeners