Agentic Conversations (formally mlops.community)

Coffee Sessions #33 with Sarah Catanzaro of Amplify Partners, MLOps Investments.

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter

//Bio
Sarah Catanzaro is a Partner at Amplify Partners, where she focuses on investing in and advising high-potential startups in machine intelligence, data management, and distributed systems. Her investments at Amplify include startups like RunwayML, Maze Design, OctoML, and Metaphor Data, among others. Sarah also has several years of experience defining data strategy and leading data science teams at startups and in the defense/intelligence sector, including through roles at Mattermark, Palantir, Cyveillance, and the Center for Advanced Defense Studies.

//We had a wide-ranging discussion with Sarah, three takeaways stood out:

1. The relationship between unstructured data and structured data is due for change. In most settings, you have some form of structured data (i.e., a metadata table) and unstructured data (i.e., images, text, etc.) Managing the relationship between these forms of data can constitute the bulk of MLOps. Because of this difficulty, Sarah forecasted new tooling arising to make data management easier.
2. Academic benchmarks suffer from a lack of transparency on production/industry use cases. In conversation with Andrew Ng, Sarah shared her lesson that despite all the blame industry professionals place on academics for narrowly optimizing to benchmarks with little practical meaning, they also share the blame for making it difficult to create meaningful benchmarks. Companies are loath to share realistic data and the true context in which ML has to operate.
3. MLOps is due for consolidation, especially as companies adopt platform-driven strategies. As many of you all know, there are tons and tons of MLOps tools out there. As more companies address these challenges, Sarah predicted that many of the point solutions would start to be consolidated into larger platforms.

// Related Links
https://amplifypartners.com/team/sarah/
https://projectstoknow.amplifypartners.com/ml-and-data
https://twitter.com/sarahcat21/status/1360105479620284419

--------------- ✌️Connect With Us ✌️ -------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/
Connect with Sarah on LinkedIn: https://www.linkedin.com/in/sarah-catanzaro-9770b98/

Timestamps:
[00:00] Introduction to Sarah Catanzaro
[02:07] Sarah's background in tech
[06:00] Staying engineer-oriented despite being an investment firm
[08:50] Tools you wished you had earlier in your career
[12:36] 2 Motives of ML Engineers and ML Platform Team
[16:36] Open-sourcing
[21:29] Startup focuses on resources
[23:57] Playout of open-source project
[27:32] Consolidation
[33:18] Finding solutions
[36:18] Evolution of the MLOps industry in the coming years
[42:36] Frameworks  
[43:14] Structure data sets available to researchers. Meaningful advances in deep learning have been applied to structure data as well.

MLOps community meetup #58! Last Wednesday, we talked to Ben Wilson, Practice Lead Resident Solutions Architect, Databricks.

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠

Model Monitoring Deep Dive with the author of Machine Learning Engineering in Action. It was a pleasure getting to talk to Ben about difficulties in monitoring in machine learning. His expertise obviously comes from experience, and as he said a few times in the meetup, I learned the hard way over 10 years as a data scientist, so you don't have to!

Ben was also kind enough to give us a 35% off promo code for his book! Use the link: http://mng.bz/n2P5

//Abstract
A great deal of time is spent building out the most effectively tuned model, production-hardened code, and elegant implementation for a business problem. Shipping our precious and clever gems to production is not the end of the solution lifecycle, though, and many abandoned projects can attest to this. In this talk, we will discuss how to think about model attribution, monitoring of results, and how (and when) to report those results to the business to ensure a long-lived and healthy solution that actually solves the problem you set out to solve.

//Bio
Ben Wilson has worked as a professional data scientist for more than ten years. He currently works as a resident solutions architect at Databricks, where he focuses on machine learning production architecture with companies ranging from 5-person startups to global Fortune 100. Ben is the creator and lead developer of the Databricks Labs AutoML project, a Scala-and Python-based toolkit that simplifies machine learning feature engineering, model tuning, and pipeline-enabled modeling. He's the author of Machine Learning Engineering in Action, a primer on building, maintaining, and extending production ML projects.

//Takeaways
Understanding why attribution and performance monitoring are critical for long-term project success

Borrowing hypothesis testing, stratification for latent confounding variable minimization, and statistical significance estimation from other fields can help to explain the value of your project to a business

Unlike in street racing, drifting is not cool in ML, but it will happen. Being prepared to know when to intervene will help keep your project running.

----------- Connect With Us ✌️-------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Ben on LinkedIn: www.linkedin.com/in/benjamin-wilson-arch/

Timestamps:
[00:00] Introduction to Ben Wilson
[00:11] Ben's background in tech
[03:40] Human aspect of Machine Learning in MLOps
[05:51] MLOps is an organizational problem
[09:27] Fragile Models
[12:36] Fraud Cases
[15:21] Data Monitoring
[18:37] Importance of knowing what to monitor for
[22:00] Monitoring for outliers
[24:16] Staying out of Alert Hell
[29:40] Ground Truth
[31:25] Model vs Data Drift on Ground Truth Unavailability
[34:25] Benefit to monitor system or business-level metrics
[38:20] Experiment in the beginning, not at the end
[40:30] Adaptive windowing
[42:22] Bridge the gap
[46:42] What scarred you really bad?

MLOps community meetup #57! Last Wednesday, we talked to Josh Tobin, Founder, Stealth-Stage Startup.

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠

// Abstract:
Machine learning is quickly becoming a product engineering discipline. Although several new categories of infrastructure and tools have emerged to help teams turn their models into production systems, doing so is still extremely challenging for most companies. In this talk, we survey the tooling landscape and point out several parts of the machine learning lifecycle that are still underserved. We propose a new category of tool that could help alleviate these challenges and connect the fragmented production ML tooling ecosystem. We conclude by discussing similarities and differences between our proposed system and those of a few top companies.

// Bio:
Josh Tobin is the founder and CEO of a stealth machine learning startup. Previously, Josh worked as a deep learning & robotics researcher at OpenAI and as a management consultant at McKinsey. He is also the creator of Full Stack Deep Learning (fullstackdeeplearning.com), the first course focused on the emerging engineering discipline of production machine learning. Josh did his PhD in Computer Science at UC Berkeley, advised by Pieter Abbeel.

// Related Links
https://josh-tobin.com
course.fullstackdeeplearning.com

----------- Connect With Us ✌️-------------   
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Josh on LinkedIn: https://www.linkedin.com/in/josh-tobin-4b3b10a9/

Timestamps:
[00:00] Introduction to Josh Tobin
[01:18] Background of Josh in tech
[08:27] We're you guys behind the Rubik's Cube?
[09:26] Rubik's Cube Project
[09:51] "Research is meant to show you what's possible to solve."
[11:07] "That's one of the things that's started to change, and I think the MLOps world is maybe a part of that. What I'm excited about this is that people are focusing on the impact of their models."
[13:18] Insights on Testing
[17:11] Evaluation Store
[18:33] "Production Machine Learning is data-driven products that have predictions in the loop."
[23:40] Analyzing and moving forward
[24:02] "My medium-term mindset on how machine learning is created is that there's still gonna be humans involved, but humans will be more efficient with tools."
[25:50] Is there a market for this?
[27:40] "The long tale of machine learning use cases is becoming part of every product and service, more or less, the companies create, but it's the same way the software part of the products and services the companies create these days. It's going to create an enormous amount of value."
[30:09] Talents
[32:52] Organizational by-ends and knowledge
[35:16] Tools used for Evaluation Store
39:59] Difference from Monitoring Tool
[42:10] Who is the right person to interact with in the Evaluation Store?
[50:05] Technical challenges of Apple and Tesla
[53:30] "As Machine Learning use cases are getting more and more complicated, higher and higher dimensional data, bigger and bigger models, larger training sets, many companies would need in order to continually improve their systems over time."

Coffee Sessions #32 with D. Sculley of Google, The Godfather Of MLOps.

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠

//Bio
D is currently a director in Google Brain, leading research teams working on robust, responsible, reliable, and efficient ML and AI. In his time at Google, D worked on nearly every aspect of machine learning and has led both product and research teams, including those on some of the most challenging business problems.

// Links to D. Sculley's Papers
ML Test Score: https://research.google/pubs/pub46555/
Machine Learning: The high-interest credit card of technical debt
https://research.google/pubs/pub43146/
Google Scholar:
https://scholar.google.com/citations?user=l_O64B8AAAAJ&hl=en

--------------- ✌️Connect With Us ✌️ -------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/
Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/
Connect with D. Sculley on LinkedIn: https://www.linkedin.com/in/d-sculley-90467310/

Timestamps:
[00:00] Introduction to D. Sculley
[00:40] The Biggest Papers were written by D for Machine Learning
[02:08] What's changed since you wrote those papers?
[02:56] "No 1, there is an MLOps community."
[04:38] Old best practices
[05:12] "The fact that there are jobs titled MLOps, this is different than it was 5 or 6 years ago."
[06:30] Machine Learning Systems then and now
[07:08] "There wasn't the level of general infrastructure that was looking to offer the large-scale integrated solutions."   
[07:57] ML Test Score
[11:09] "The Test Score was really written for situations where you don't care about one prediction. You care about millions or billions of predictions per day."
[12:27] "In the end, it's not about the score. It's about the process of asking the questions, making sure that each of the important questions that you're asking yourself, you have a good answer to."  
[13:04] What else is needed in the Test Score?
[14:36] Stratified testing  
[17:05] Counterfactual testing
[18:34] Boundaries
[19:15] Dark ages
[20:27] How do you try in Triage?
[21:10] "Reliability is important. There are no small mistakes. If there are errors, they're going to get spotted and publicized. They're going to impact users' lives. The bar is really high, and it's worth the effort to ensure strong reliability."
[23:11] How do you build that interest stress test?
[24:39] "I believe that stress test is going to look like a useful way to encode expert knowledge about domain areas."
[25:37] How do I bring robustness?
[27:22] Underspecification Paper
[30:58] "It's important to be evaluating models on this auto domain stress test and make sure that we understand the implications of what we're thinking about while we are in deployment land."
[32:27] Principal challenges in productionizing Machine Learning
[34:57] "As we expose our models to more specifics, this means there are more potential places our models might be exhibiting unexpected or undesirable behaviour."
[42:37] Splintering of ML Engineering
[46:00] Communities shaping the MLOps sphere
[46:42] "It's much better to have one large community than three smaller communities because of those edufacts."
[47:47] Concept of technical debt in machine learning.
[49:28] "The good idea is to tend to make their way into the community if they are in a form that people can digest and share."

MLOps community meetup #56! Last Wednesday, we talked to  Daniel Stahl, Head of Data and Analytics Platforms, Regions Bank.

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter

// Abstract:
The Data Science practice has evolved significantly at Regions, with a corresponding need to scale and operationalize machine learning models. Additionally, highly regulated industries such as finance require a heightened focus on reproducibility, documentation, and model controls.  In this session with Daniel Stahl, we will discuss how the Regions team designed and scaled their data science platform using DevOps and MLOps practices.  This has allowed Regions to meet the increased demand for machine learning while embedding controls throughout the model lifecycle.  In the 2 years since the data science platform has been onboarded, 100% of data products have been successfully operationalized.

// Bio:
Daniel Stahl leads the ML platform team at Regions Bank and is responsible for tooling, data engineering, and process development to make operationalizing models easy, safe, and compliant for Data Scientists.  
Daniel has spent his career in financial services and has developed novel methods for computing tail risk in both credit risk and operational risk, resulting in peer-reviewed publications in the Journal of Credit Risk and the Journal of Operational Risk. Daniel has a Master's in Mathematical Finance from the University of North Carolina at Charlotte.    

 
Daniel lives in Birmingham, Alabama, with his wife and two daughters.

----------- Connect With Us ✌️-------------   
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Dan on LinkedIn: https://www.linkedin.com/in/daniel-stahl-6685a52a/

Timestamps:
[00:00] Introduction to Ben Wilson
[00:11] Ben's background in tech
[01:17] "How do you do what I have always done pretty well, which is being as lazy as possible in order to automate things that I hate doing. So I learned about Regression Problems."
[03:40] Human aspect of Machine Learning in MLOps
[05:51] MLOps is an organizational problem
[09:27] Fragile Models
[12:36] Fraud Cases
[15:21] Data Monitoring
[18:37] Importance of knowing what to monitor for
[22:00] Monitoring for outliers
[24:16] Staying out of Alert Hell
[29:40] Ground Truth
[31:25] Model vs Data Drift on Ground Truth Unavailability
[34:25] Benefit to monitor system or business-level metrics
[38:20] Experiment in the beginning, not at the end
[40:30] Adaptive windowing
[42:22] Bridge the gap
[46:42] What scarred you really bad?

MLOps community meetup #55! Last Wednesday, we talked to Igor Lushchyk, Data Engineer, Adyen.  

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter

// Abstract:
Building Data Science and Machine Learning platforms at a scale-up. Having the main difficulty in finding the correct processes, and basically being a toddler who learns how to walk on a steep staircase. The transition from homegrown platforms to open source solutions, supporting old solutions and maturing them, makes data scientists happy.  

// Bio:
Igor is a software engineer with more than 10 years of experience. With a background in bioinformatics, he even started a PhD but didn't finish it.

As a data engineer, Igor has been working for the last 6 or 7 years, or maybe more, because he was doing almost the same data engineering stuff, but his position was named differently.

Igor has been doing a lot of MLOps in 4-5 years now. He doesn't know what he was doing more than - Data Engineering or MLOps. And that’s how this topic came about.  

----------- Connect With Us ✌️-------------   
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Igor on LinkedIn: https://www.linkedin.com/in/igor-lushchyk/

Timestamps:
[00:00] Introduction to Igor Lushchyk
[02:05] Igor's background in tech
[07:42] Tips you can pass on
[11:05] How these tools work, and how they play together, and what is underneath?
[13:18] Dedicated MLOps team
[13:55] Central Data Infrastructure Section
[16:57] Transfer over to open-source
[20:24] If you don't plan for production from the beginning, then it's going to be painful trying to go from POC to production.
[22:08] How do you handle data lineage?
[25:09] You chose that back in the day, but you're regretting it.
[26:34] "Try to use tools which solve 80% of your use cases, and maybe 20% you'll have the suffering, but at least it's not 100% suffering."
[27:27] Friction points
[28:53] Interaction with Data Scientists
[29:21] "We have alignment sessions. We have different levels of representation. We share our progress."
[32:42] Build verse by decisions
[34:04] When to build or grab an open-source tool
[35:51] Build your own or buy open-source?
[37:11] Certain maturity and a certain number of engineers
[38:11] Startup to go with open-source
[40:14] Correct transition process
[40:56] "There are no other ways but to communicate with data scientists. Your team needs to have a close loop for future priorities, what to take with you, and what to leave behind."
[44:51] What to use in the monitoring piece
[45:36] Prometheus and Grafana
[48:07] Do you have automatic retriggering monitoring of Models set up?
[51:55] Hardware for on-prim model training
[52:38] "Machine Learning model prediction is a spear bomb."
[53:55] War or horror stories
[54:15] "Guys, don't do context switching!"
[55:54] "I won't say that Adyen is a company that allows you to make mistakes, but you can make mistakes."

MLOps community meetup #54! Last Wednesday, we talked to Laszlo Sragner, Founder, Hypergolic.

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter

// Abstract:
How my experience in quant finance and software engineering influenced how we ran ML at a London Fintech Startup. How to solve business problems with incremental ML? What's the difference between academic and industrial ML?

// Bio:
Laszlo worked as a quant researcher at multiple investment managers and as a DS at the world's largest mobile gaming company. As Head of Data Science at Arkera, he drove the company's data strategy, delivering solutions to Tier 1 investment banks and hedge funds. He currently runs Hypergolic (hypergolic.co.uk), an ML Consulting company helping startups and enterprises bring the maximum out of their data and ML operations.

// Takeaways
Continuous evaluation and monitoring are indistinguishable in a well-set-up product team. Separation of concerns (SE, ML, DevOps, MLOps) is very important for smooth operation, and low-friction team coordination/communication is key.
To be able to iterate business features into models, you need a modeling framework that can express these, which is usually a DL package.
DS-es are well motivated to go more technical because they see the rewards of it. All well-run (from the DS perspective) startups in my experience do the same.

// Related Links
Free eBook about MLPM: https://machinelearningproductmanual.com/
Lightweight MLOps Python package: https://hypergol.ml/
Blog: laszlo.substack.com

----------- Connect With Us ✌️-------------   
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register  

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Laszlo on LinkedIn: https://www.linkedin.com/in/laszlosragner/

Timestamps:
[00:00] Introduction to Laszlo Sranger
[02:15] Laszlo's Background
[09:18] Being a Quant, then influenced what you were doing with the Investment Banks?
[12:24] Do you think this can be applied in different use cases or specific to what you are doing?
[14:41] Do you have any thoughts of a potentially highly opinionated person?
[16:54] Product management in Machine Learning
[24:59] You have to be at a large company, or you have to have a large team? [26:38] What are your thoughts on MLOps products helping with product management for ML? Is it an overreach or scope creep?
[32:00] In the messy world of startups, due to the high cost of an MVP for NLP, is RegEx, which means to incorporate user feedback, it's incorporated by tweaking RegEx?
[33:04] Do the ensemble recent models more than older models? If so, what is the decay rate of weights for older models?
[35:40] Since the iterative management model is generic enough for most ML projects, which component of it can be easily generalized, and what tools are built for version control?
[36:38] Topic Extraction: What type of model do you train for that task?
[52:55] Thoughts on Notebooks
[53:34] "I don't hate notebooks. Let's be clear about that. I put it this way: notebooks are whiteboards. You don't want your whiteboards to be your output because they're a sketch of your solution. You want the purest solution."

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter

This is a deep dive into the most recent MLOps Engineering Labs from the point of view of Team 3.  

// Diagram Link:  
https://github.com/dmangonakis/mlops-lab-example-yelp  

--------------- ✌️Connect With Us ✌️ -------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register  

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Laszlo on LinkedIn https://www.linkedin.com/in/laszlosragner/
Connect with Artem on LinkedIn: https://www.linkedin.com/in/artem-yushkovsky/
Connect with Paulo on LinkedIn: https://www.linkedin.com/in/paulo-maia-410874119/
Connect with Dimi on LinkedIn:

Timestamps:

[00:00] Engineering Labs Recap Team Three
[01:12] Laszlo Sranger Background
[02:05] Artem Background
[04:45] Dimi Background
[06:31] Paulo Background
[08:51] Initial Product Ideas Overview
[09:12] Decent Product Using Yelp Dataset
[10:32] Backend Facade Streamlit Overview
[13:52] Questioning Bad Practices
[14:11] Demo Works But Limited
[15:12] Walking Through Streamlit Code
[15:16] Decoupled Frontend Backend Architecture
[16:54] Managerial Considerations
[19:00] Working Outside Comfort Zones
[20:36] Key Takeaways From Lab
[20:42] MLflow Architecture Insights
[22:21] Additional Considerations
[22:31] MLflow End-to-End Monitoring
[24:50] Explainability Tools and Complexity
[26:29] Real-World Issues
[26:36] Avoid Unnecessary Bells and Whistles
[28:33] Difficulties in Process
[30:25] Engineering Mistakes Reflection
[31:17] Artifact Logging Challenges
[32:00] Identifying Non-Ideal Aspects
[33:21] PyTorch Limitations
[34:52] Managing Dependencies
[35:08] Avoid Using Notebooks
[36:27] Consistent Scripts And Environments
[37:08] Replicable Docker Processes
[37:42] Future MLflow Use
[38:23] MLflow Improvement Over Time
[40:34] Kubernetes Knowledge Requirements
[41:25] Kubernetes Provides Great Output
[46:03] Current Status Limitations
[46:53] Limited Production Control
[47:40] Kubernetes Knowledge For Data Scientists
[48:14] Machine Learning Cultural Movement
[50:55] Jack Of All Trades
[51:32] Productized ML Requires Engineering
[56:27] Final Lab Reflections
[57:11] Cloud Credits For Next Lab

MLOps community meetup #53! Last Wednesday, we talked to Krishna Gade, CEO & Co-Founder, Fiddler AI.

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter

// Abstract:
Training and deploying ML models have become relatively fast and cheap, but with the rise of ML use cases, more companies and practitioners face the challenge of building “Responsible AI.” One of the barriers they encounter is increasing transparency across the entire AI lifecycle to not only better understand predictions but also to find problem drivers. In this session with Krishna Gade, we will discuss how to build AI responsibly, share examples from real-world scenarios and AI leaders across industries, and show how Explainable AI is becoming critical to building Responsible AI.

// Bio:
Krishna is the co-founder and CEO of Fiddler, an Explainable AI Monitoring company that helps address problems regarding bias, fairness, and transparency in AI. Prior to founding Fiddler, Gade led the team that built Facebook’s explainability feature ‘Why am I seeing this?’. He’s an entrepreneur with a technical background, with experience creating scalable platforms and expertise in converting data into intelligence. Having held senior engineering leadership roles at Facebook, Pinterest, Twitter, and Microsoft, he’s seen the effects that bias has on AI and machine learning decision-making processes, and with Fiddler, his goal is to enable enterprises across the globe to solve this problem.

----------- Connect With Us ✌️-------------   
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Krishna on LinkedIn: https://www.linkedin.com/in/krishnagade/

Timestamps:
[00:00] Thank you, Fiddler AI!
[01:04] Introduction to Krishna Gade
[03:19] Krisha's Background
[08:33] Everything was fine when you were doing it behind the scenes. But then, when you put it out into the wild, we just lost our "baby." It's no longer under our control.
[08:53] "You want to have the assurance of how the system works. Even if it's working fine or if it's not working fine."  
[09:37] What else is Explainability? Can you break that down for us?
[13:58] "Explainability becomes the cornerstone technology to have in place for you to build Responsible AI in production."
[14:48] For those used cases that aren't as high stakes, do you feel it's important? Is it up the food chain?
[18:47] Can we dig into that used case real fast?
[22:01] If it is a human doing it, there's a lot more room for error? Bias or theories can be introduced and then they don't have a basis in reality?
[23:51] Do you need these subject matter experts or someone who is very advanced to be able to set up what the Explainability tool should be looking for at first? Is it that plug and play, and it will know it latches on to the model?
[29:36] Does Explainable AI also entail Explainable Data? I see the point where Explainability can help with getting the insights about data after the model has been trained, but should it be handled perhaps more proactively, where you unbias the data before training the model on it?
[32:16] As a data scientist, there are situations when the prediction output is expected to support a business decision taken by senior executives. In that case, when the Explainable model gives out a prediction that doesn't align with the stakeholder's expectations, how should one navigate through this tricky situation?
[43:49] How is denen gram clustering for data explainability?

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

This is a deep dive into the most recent MLOps Engineering Labs from the point of view of Team 1.

// Diagram Link: https://github.com/mlops-labs-team1/engineering.labs#workflow

--------------- ✌️Connect With Us ✌️ -------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Alexey on LinkedIn: https://www.linkedin.com/in/alexeynaiden/
Connect with John on LinkedIn: https://www.linkedin.com/in/johnsavageireland/
Connect with Michel on LinkedIn: https://www.linkedin.com/in/michel-vasconcelos-8273008/
Connect with Varuna on LinkedIn: https://www.linkedin.com/in/vpjayasiri/

Timestamps

[00:00] Introduction to Engineering Labs Participants
[00:34] What Are Engineering Labs
[01:05] Credits to Ivan Nardini
[04:24] John Savage Profile
[05:13] Prior MLflow Knowledge
[05:50] Alexey Naiden Profile
[07:26] Varuna Jayasiri Profile
[08:28] Michel Vasconcelos Profile
[10:07] Process Using PyTorch MLflow
[13:39] Implementation Structure and Coding
[17:03] Encountering Problems Along the Way
[20:26] Overview and First Problem
[23:08] Catching Up or Comfortable
[24:12] Tool John Called Out
[24:41] Homegrown Tool Confirmation
[24:51] Engineering Labs Implementation
[26:03] Pipeline and Serving Overview
[37:26] Pet Project Limitations
[38:13] Lego-Like Modular Building Block
[40:54] PyTorch or MLflow Troubles
[42:44] Torchserve Prompt Challenges
[44:27] Considering Better Approaches
[49:05] Feedback on Labs Experience
[50:20] Michel Wants Future Participation
[51:52] Varuna Values Tangible Learning
[53:00] John Anchored in MLOps
[55:52] Alexey Reaching Checkpoint
[56:01] Michel’s Terraform Reproducibility Piece