Coffee Sessions #46 with Pablo Estevez, What We Learned from 150 Successful ML-enabled Products at Booking.com.

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠

// Abstract
While most of the Machine Learning literature focuses on the algorithmic or mathematical aspects of the field, not much has been published about how Machine Learning can deliver meaningful impact in an industrial environment where commercial gains are paramount. We conducted an analysis on about 150 successful customer-facing applications of Machine Learning, developed by dozens of teams in Booking.com, exposed to hundreds of millions of users worldwide, and validated through rigorous Randomized Controlled Trials. Our main conclusion is that an iterative, hypothesis-driven process, integrated with other disciplines, was fundamental to building 150 successful products enabled by Machine Learning.

// Bio
Pablo Estevez is the Principal Data Scientist at Booking.com. He has worked on recommendations, personalization, and experimentation across the Booking.com website, as well as being a manager on several machine learning, data science, and product development teams.

// Other Links
Talk on the topic: https://www.youtube.com/watch?v=ljhtfrtuNqw&t=4h24m30s
The paper: https://blog.kevinhu.me/2021/04/25/25-Paper-Reading-Booking.com-Experiences/bernardi2019.pdf

--------------- ✌️Connect With Us ✌️ -------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/
Connect with Pablo on LinkedIn: https://www.linkedin.com/in/estevezpablo/

[00:00] Introduction to Pablo Estevez
[02:02] Pablo’s Background in Tech
[03:43] Machine Learning at Booking.com
[08:09] 150 Models: Six Key Lessons
[10:20] Reflecting on Past ML Work
[10:38] Pablo’s Role in Team
[12:49] Broader Applications, Bigger Impact
[12:55] Driving Through Business Impact
[14:40] Beyond Precision: Focus on Goals
[16:24] Diversity Enables Better Exploration
[17:43] Three-Step Problem-Solving Framework
[18:42] Framework of Problem Design
[19:12] Focus on Experimentation Culture
[20:46] Scaling Tooling for Experimentation
[22:58] Cheap Experiments, Better Insights
[28:39] Real-World Interactions and Analysis
[30:15] Connecting Hypotheses to Business Value
[31:04] Defining Experiments as Code
[31:37] Airbnb’s Workflow Example
[34:53] Decision-Making Through Experimentation Results
[35:48] Building an Experimentation Platform
[36:39] Investing in Better Infrastructure
[36:50] Experimentation Justifies Infrastructure Investment
[38:45] Monitoring Metrics for Business Value
[39:40] Connecting Models to Business Value
[41:35] Deployment at Booking.com
[45:13] Supporting More Use Cases
[46:10] Latency Challenges Business Performance
[48:43] Open-Sourcing at Booking.com
[49:30] Responsible Open-Source Maintenance Standards
[49:45] ML Open-Source Standards
[52:00] Lessons Learned Since Publication
[53:30] Structuring the Exploration Phase
[54:02] Maintainability Within Diversity