// Abstract

Finding an ML model that solves a business problem can feel like winning the lottery, but it can also be a curse. Once a model is embedded at the core of an application and used by real users, the real work begins. That's when you need to make sure that it works for everyone, that it keeps working every day, and that it can improve as time goes on. Just like building a model is all about data work, keeping a model alive and healthy is all about developing operational excellence.

First, you need to monitor your model and its predictions and detect when it is not performing as expected for some types of users. Then, you'll have to devise ways to detect drift, and how quickly your models get stale. Once you know how your model is doing and can detect when it isn't performing, you have to find ways to fix the specific issues you identify. Last but definitely not least, you will now be faced with the task of deploying a new model to replace the old one, without disrupting the day of all the users that depend on it.

A lot of the topics covered are active areas of work around the industry and haven't been formalized yet, but they are crucial to making sure your ML work actually delivers value. While there aren't any textbook answers, there is no shortage of lessons to learn.

// Bio

Emmanuel Ameisen has worked for years as a Data Scientist and ML Engineer. He is currently an ML Engineer at Stripe, where he worked on helping improve model iteration velocity. Previously, he led Insight Data Science's AI program where he oversaw more than a hundred machine learning projects. Before that, he implemented and deployed predictive analytics and machine learning solutions for Local Motion and Zipcar. Emmanuel holds graduate degrees in artificial intelligence, computer engineering, and management from three of France’s top schools.

// Related Links

https://www.amazon.com/Building-Machine-Learning-Powered-Applications/dp/149204511X https://

www.oreilly.com/library/view/building-machine-learning/9781492045106/

--------------- ✌️Connect With Us ✌️ -------------

Join our slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletter and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Adam on LinkedIn: https://www.linkedin.com/in/aesroka/

Connect with Emmanuel on LinkedIn: https://www.linkedin.com/in/ameisen/

Timestamps:

[00:00] Introduction to Emmanuel Ameisen

[03:38] Building Machine Learning Powered Applications book inspiration

[05:19] The writing process

[07:04] Over-engineering NLP

[09:13] CV driven development: intentional or natural

[11:09] Attribute to machine learning team

[14:44] Shortening iteration cycle

[16:41] Advice on how to tackle iteration

[20:00] Failure modes

[21:02] Infrastructure Iteration at Stripe

[27:06] Deployment Steps tests challenges

[29:34] "You develop operational excellence by exercising it." - Emmanuel Ameisen

[33:22] Death of a thousand cuts: Balance of work vs productionization piece balance

[36:15] Reproducibility headaches [40:04] Pipelines as software product

[41:25] Get the book Building Machine Learning Powered Applications: Going from Idea to Product book by Emmanuel Ameisen!

[42:04] Takeaways and wrap up