// Abstract:

Building Data Science and Machine Learning platforms at a scale-up. Having the main difficulty in finding correct processes and basically being a toddler who learns how to walk on a steep staircase. The transition from homegrown platform to open source solutions, supporting old solutions and maturing them with making data scientists happy.  

// Bio:

Igor is a software engineer with more than 10 years of experience. With a background in bioinformatics, he even started PhD but didn't finish it.

As a data engineer, Igor has been working for the last 6 or 7 years, or maybe more - because he was doing almost the same data engineering stuff but his position was named differently.

Igor has been doing a lot of MLOps in 4-5 years now. He doesn't know what he was doing more then - Data Engineering or MLOps. And that’s how this topic came about.  

----------- Connect With Us ✌️-------------   

Join our Slack community:  https://go.mlops.community/slack

Follow us on Twitter:  @mlopscommunity

Sign up for the next meetup:  https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Igor on LinkedIn: https://www.linkedin.com/in/igor-lushchyk/

[00:00] Introduction to Igor Lushchyk

[02:05] Igor's background in tech

[07:42] Tips you can pass on

[11:05] How these tools work and how they play together and what is underneath?

[13:18] Dedicated MLOps team

[13:55] Central Data Infrastructure Section

[16:57] Transfer over to open-source

[20:24] If you don't plan for production from the beginning, then it's going to be painful trying to go from POC to production.

[22:08] Ho do you handle data lineage?

[25:09] You chose that back in the day but you're regretting it.

[26:34] "Try to use tools which solve 80% of your use cases and maybe 20% you'll have the suffering but at least it's not 100% suffering."

[27:27] Friction points

[28:53] Interaction with Data Scientists

[29:21] "We have alignment sessions. We have different levels of representations. We share our progress."

[32:42] Build verse by decisions

[34:04] When to build or grab an open-source tool

[35:51] Build your own or buy open-source?

[37:11] Certain maturity and a certain number of engineers

[38:11] Startup to go with open-source

[40:14] Correct transition process

[40:56] "There are no other ways but to communicate with data scientists. Your team needs to have a close loop for future priorities, what to take with you and what to leave behind."

[44:51] What to use in monitoring piece

[45:36] Prometheus and Grafana

[48:07] Do you automatic retriggering monitoring of Models set up?

[51:55] Hardware for on Prim model training

[52:38] "Machine Learning model prediction is a spear bomb."

[53:55] War or horror stories

[54:15] "Guys, don't do context switching!"

[55:54] "I won't say that Adyen is a company that allows you to make mistakes but you can make mistakes."