Agentic Conversations (formally mlops.community)

Jonathan Rioux is a Managing Principal of AI Consulting for EPAM Systems, where he advises clients on how to get from idea to realized AI products with the minimum of fuss and friction.

Who's MLOps for Anyway? // MLOps Podcast #261 with Jonathan Rioux, Managing Principal, AI Consulting at EPAM Systems.

// Abstract

The year is 2024, and we are all staring into the cliff towards the abyss of disillusionment for Generative AI. Every organization, developer, and AI-adjacent individual is now talking about "making AI real" and "turning an ROI on AI initiatives". MLOps and LLMOps are taking the stage as the solution; equip your AI teams with the best tools money can buy, grab tokens by the fistful, and look at value raking in.

Sounds familiar and eerily similar to the previous ML hype cycles? From solo devs to large organizations, how can we avoid the same pitfalls as last time and get out of the endless hamster wheel?

// Bio

Jonathan is a Managing Principal of AI Consulting for EPAM, where he advises clients on how to get from idea to realized AI products with the minimum of fuss and friction. He's obsessed with the mental models of ML and how to organize harmonious AI practices. Jonathan published "Data Analysis with Python and PySpark" (Manning, 2022).

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: raiks.ca

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Jonathan on LinkedIn: https://www.linkedin.com/in/jonathanrx/

Timestamps:

[00:00] Jonathan's preferred coffee

[00:25] Takeaways

[01:44] MLOps is not being sexy

[03:49] Do not conflate MLOps with ROI

[06:21] ML Certification Business Idea

[11:02] AI Adoption Missteps

[15:40] Slack AI Privacy Risks

[18:17] Decentralized AI success

[22:00] Michelangelo Hub-Spoke Model

[27:45] Engineering tools for everyone

[33:38 - 35:20] SAS Ad

[35:21] POC to ROI transition

[42:08] Repurposing project learnings

[46:24] Balancing Innovation and ROI

[55:35] Using a classification model

[1:00:24] Chatbot evolution comparison

[1:01:20] Balancing Automation and Trust

[1:06:30] Manual to AI transition

[1:09:57] Wrap up

Shiva Bhattacharjee is the Co-founder and CTO of TrueLaw, where we are building bespoke models for law firms for a wide variety of tasks.

Alignment is Real // MLOps Podcast #260 with Shiva Bhattacharjee, CTO of TrueLaw Inc.

// Abstract

If the off-the-shelf model can understand and solve a domain-specific task well enough, either your task isn't that nuanced or you have achieved AGI. We discuss when fine-tuning is necessary over prompting and how we have created a loop of sampling, collecting feedback, and fine-tuning to create models that seem to perform exceedingly well in domain-specific tasks.

// Bio

20 years of experience in distributed and data-intensive systems spanning work at Apple, Arista Networks, Databricks, and Confluent. Currently CTO at TrueLaw, where we provide a framework to fold in user feedback, such as lawyer critiques of a given task, and fold them into proprietary LLM models through fine-tuning mechanics, resulting in 7-10x improvements over the base model.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: www.truelaw.ai

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Shiva on LinkedIn: https://www.linkedin.com/in/shivabhattacharjee/

Timestamps:

[00:00] Shiva's preferred coffee

[00:58] Takeaways

[01:17] DSPy Implementation

[04:57] Evaluating DSPy risks

[08:13] Community-driven DSPy tool

[12:19] RAG implementation strategies

[17:02] Cost-effective embedding fine-tuning

[18:51] AI infrastructure decision-making

[24:13] Prompt data flow evolution

[26:32] Buy vs build decision

[30:45] Tech stack insights

[38:20] Wrap up

Vikram Rangnekar is an open-source software developer focused on simplifying LLM integration. He created LLMClient, a TypeScript library inspired by Stanford's DSP paper. With years of experience building complex LLM workflows, he previously worked as a senior software engineer at LinkedIn on Ad Serving.

Ax a New Way to Build Complex Workflows with LLMs // MLOps Podcast #259 with Vikram Rangnekar, Software Engineer at Stealth.

// Abstract

Ax is a new way to build complex workflows with LLMs. It's a TypeScript library based on research done in the Stanford DSP paper. Concepts such as prompt signatures, prompt tuning, and composable prompts help you build RAG and agent-powered ideas that have, till now, been hard to build and maintain. Ax is designed for production usage.

// Bio

Vikram builds open-source software. Currently working on making it easy to build with LLMs. Created Ax, a TypeScript library that abstracts over all the complexity of LLMs, which is based on the research done in the Stanford DSP paper. Worked extensively with LLMs over the last few years to build complex workflows. Previously worked as a senior software engineer with LinkedIn on Ad Serving.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

The unofficial DSPy framework. Build LLM-powered Agents and "Agentic workflows" based on the Stanford DSP paper: https://axllm.dev

All the Hard Stuff with LLMs in Product Development // Phillip Carter // MLOps Podcast #170: https://youtu.be/DZgXln3v85s

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Vikram on LinkedIn: https://www.linkedin.com/in/vikramr

Timestamps:

[00:00] Vikram preferred coffee

[00:41] Takeaways

[01:05] Data Engineering for AI/ML Conference Ad

[01:41] Vikram's work these days

[04:54] Fine-tuned Model insights

[06:22] JavaScript tool evolution

[16:14] DSP knowledge distillation

[17:34] DSP vs Manual examples

[22:53] Optimizing task context

[27:58] API type validation explained

[30:25] LLM value and innovation

[34:22] Navigating complex systems

[37:30] DSP code generators explained

[40:56] Exploring LLM personas

[42:45] Optimizing small agents

[43:32] Complex task assistance

[49:53] Wrap up

MLOps Coffee Sessions #177 with Mohamed Abusaid and Mara Pometti, Building in Production Human-centred GenAI Solutions sponsored by QuantumBlack, AI by McKinsey.

Markus Stoll is the Co-Founder of Renumics and the developer behind the open-source interactive ML dataset exploration tool, Spotlight. He shares insights on:

AI in Engineering and Manufacturing
Interactive ML Data Visualization
ML Data Exploration

Follow Markus for hands-on articles about leveraging ML while keeping a strong focus on data.

Visualize - Bringing Structure to Unstructured Data // MLOps Podcast #258 with Markus Stoll, CTO of Renumics.

A huge thank you to SAS for their generous support!

// Abstract

This talk is about how data visualization and embeddings can support you in understanding your machine-learning data. We explore methods to structure and visualize unstructured data like text, images, and audio for applications ranging from classification and detection to Retrieval-Augmented Generation. By using tools and techniques like UMAP to reduce data dimensions and visualization tools like Renumics Spotlight, we aim to make data analysis for ML easier. Whether you're dealing with interpretable features, metadata, or embeddings, we'll show you how to use them all together to uncover hidden patterns in multimodal data, evaluate the model performance for data subgroups, and find failure modes of your ML models.

// Bio

Markus Stoll began his career in the industry at Siemens Healthineers, developing software for the Heavy Ion Therapy Center in Heidelberg. He learned about software quality while developing a treatment machine weighing over 600 tons. He earned a Ph.D., focusing on combining biomechanical models with statistical models, through which he learned how challenging it is to bridge the gap between research and practical application in the healthcare domain. Since co-founding Renumics, he has been active in the field of AI for Engineering, e.g., AI for Computer Aided Engineering (CAE), implementing projects, contributing to their open-source library for data exploration for ML datasets (Renumics Spotlight), and writing articles about data visualization.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: https://renumics.com/

MLSecOps Community: https://community.mlsecops.com/

Blogs: https://towardsdatascience.com/visualize-your-rag-data-evaluate-your-retrieval-augmented-generation-system-with-ragas-fc2486308557 : https://medium.com/itnext/how-to-explore-and-visualize-ml-data-for-object-detection-in-images-88e074f46361

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Markus on LinkedIn: https://www.linkedin.com/in/markus-stoll-b39a42138/

Timestamps:

[00:00] Markus' preferred coffee

[00:15] Takeaways

[01:41] Please like, share, leave a review, and subscribe to our MLOps channels!

[01:50] Register for the Data Engineering for AI/ML Conference now!

[02:27] Current focus and updates

[04:43] 3D Embeddings Visualization Explained

[07:07] Question Embeddings vs Retrieval

[08:24] Using heat maps effectively

[10:30] User insights visualization RAG

[16:59] 3D Crash Simulation Analysis

[20:33] Simulation purpose clarification

[22:34] Evaluating test data use cases

[24:22] Real-world car testing

[29:48] Identifying data issues early

[33:33] Multimodal data integration

[37:42] Custom vs Fine-tuned models

[39:45] Data processing challenges

[45:58] Use case-driven MVP

[48:26 - 50:08] SAS Ad

[50:09] Wrap up

MLOps for GenAI Applications // Special MLOps Podcast episode with Demetrios Brinkmann, Chief Happiness Engineer at MLOps Community.

// Abstract

Demetrios explores common themes in ML model testing with insights from Erica Greene (Yahoo News), Matar Haller (ActiveFence), Mohamed Elgendy (Kolena), and Catherine Nelson (Freelance Data Scientist). They discuss tiered test cases, functional testing for hate speech, differences between AI and traditional software testing, and the complexities of evaluating LLMs. Demetrios wraps up by inviting feedback and promoting an upcoming virtual conference on data engineering for AI and ML.

// Bio

At the moment, Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly MLOps Community Podcasts. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Balancing Speed and Safety // Panel // AIQCON - https://youtu.be/c81puRgu3Kw

AI For Good - Detecting Harmful Content at Scale // Matar Haller // MLOps Podcast #246 - https://youtu.be/wLKlZ6yHg1k

What is AI Quality? // Mohamed Elgendy // MLOps Podcast #229 - https://youtu.be/-Jdmq4DiOew

All Data Scientists Should Learn Software Engineering Principles // Catherine Nelson // Podcast #245 - https://youtu.be/yP6Eyny7p20

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Timestamps:

[00:00] Exploring common themes in MLOps community

[00:49] Common patterns about model output and testing

[01:34] Tiered test case strategy

[03:05] Functional testing for models

[05:24] Testing coverage and quality

[07:47] Evaluating LLMs challenges

[08:35] Please like, share, leave a review, and subscribe to our MLOps channels!

Sean Morgan is an active open-source contributor and maintainer and is the special interest group lead for TensorFlow Addons. Learn more about the platform for end-to-end AI Security at https://protectai.com/.

MLSecOps is Fundamental to Robust AI Security Posture Management (AISPM) // MLOps Podcast #257 with Sean Morgan, Chief Architect at Protect AI.

// Abstract

MLSecOps, which is the practice of integrating security practices into the AIML lifecycle (think infusing MLOps with DevSecOps practices), is a critical part of any team’s AI Security Posture Management. In this talk, we’ll discuss how to threat model realistic AIML security risks, how you can measure your organization’s AI Security Posture, and most importantly, how you can improve that security posture through the use of MLSecOps.

// Bio

Sean Morgan is the Chief Architect at Protect AI. In prior roles, he's led production AIML deployments in the semiconductor industry, evaluated adversarial machine learning defenses for DARPA research programs, and most recently scaled customers on interactive machine learning solutions at AWS. In his free time, Sean is an active open-source contributor and maintainer and is the special interest group lead for TensorFlow Addons.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Sean's GitHub: https://github.com/seanpmorgan

MLSecOps Community: https://community.mlsecops.com/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Sean on LinkedIn: https://www.linkedin.com/in/seanmorgan/

Timestamps:

[00:00] Sean's preferred coffee

[00:10] Takeaways

[01:39] Register for the Data Engineering for AI/ML Conference now!

[02:21] KubeCon Paris: Emphasis on security and AI

[05:00] Concern about malicious data during the training process

[09:29] Model builders, security, pulling foundational models, nuances

[12:13] Hugging Face research on security issues

[15:00] Inference servers exposed; potential for attack

[19:45] Balancing ML and security processes for ease

[23:23] Model artifact security in enterprise machine learning

[25:04] Scanning models and datasets for vulnerabilities

[29:23] Ray's user interface vulnerabilities lead to attacks

[32:07] ML Flow vulnerabilities present significant server risks

[36:04] Data ops essential for machine learning security

[37:32] Prioritized security in model and data deployment

[40:46] Automated scanning tool for improved antivirus protection

[42:00] Wrap up

Harcharan Kabbay is a Data Scientist & AI/ML Engineer with Expertise in MLOps, Kubernetes, and DevOps, Driving End-to-End Automation and Transforming Data into Actionable Insights.

MLOps for GenAI Applications // MLOps Podcast #256 with Harcharan Kabbay, Lead Machine Learning Engineer at World Wide Technology.

// Abstract

The discussion begins with a brief overview of the Retrieval-Augmented Generation (RAG) framework, highlighting its significance in enhancing AI capabilities by combining retrieval mechanisms with generative models. The podcast further explores the integration of MLOps, focusing on best practices for embedding the RAG framework into a CI/CD pipeline. This includes ensuring robust monitoring, effective version control, and automated deployment processes that maintain the agility and efficiency of AI applications. A significant portion of the conversation is dedicated to the importance of automation in platform provisioning, emphasizing tools like Terraform. The discussion extends to application design, covering essential elements such as key vaults, configurations, and strategies for seamless promotion across different environments (development, testing, and production). We'll also address how to enhance the security posture of applications through network firewalls, key rotation, and other measures. Let's talk about the power of Kubernetes and related tools to aid a good application design. The podcast highlights the principles of good application design, including proper observability and eliminating single points of failure. I would share strategies to reduce development time by creating templates for GitHub repositories by application types to be reused, also templates for pull requests, thereby minimizing human errors and streamlining the development process.

// Bio

Harcharan is an AI and machine learning expert with a robust background in Kubernetes, DevOps, and automation. He specializes in MLOps, facilitating the adoption of industry best practices and platform provisioning automation. With extensive experience in developing and optimizing ML and data engineering pipelines, Harcharan excels at integrating RAG-based applications into production environments. His expertise in building scalable, automated AI systems has empowered the organization to enhance decision-making and problem-solving capabilities through advanced machine-learning techniques.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Harcharan's Medium - https://medium.com/@harcharan-kabbay

Data Engineering for AI/ML Conference: https://home.mlops.community/home/events/dataengforai

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Harcharan on LinkedIn: https://www.linkedin.com/in/harcharankabbay/locale=en_US

Timestamps:

[00:00] Harcharan's preferred coffee

[00:21] Takeaways

[01:03] Against local LLMs

[02:11] Creating bad habits

[02:42] Operationalizing RAG from the CI/CD perspective

[09:39] Kubernetes vs LLM Deployment

[12:12] Tool preferences in ML

[14:39] DevOps perspective of deployment

[17:44] Terraform Licensing Controversy

[22:47] PR Review Template Guidance

[27:32] People process tech order

[29:22] Register for the Data Engineering for AI/ML Conference now!

[30:00] ML monitoring strategies explained

[39:39] Serverless vs Overprovisioning

[44:43] Model SLA's and Monitoring

[51:04] LLM to App transition

[52:42] Ensuring Robust Architecture

[58:53] Chaos engineering in ML

[1:04:43] Wrap up

Nicolas Mauti is an MLOps Engineer from Lyon (France), Working at Malt.

BigQuery Feature Store // MLOps Podcast #255 with Nicolas Mauti, Lead MLOps at Malt.

// Abstract

Need a feature store for your AI/ML applications but overwhelmed by the multitude of options? Think again. In this talk, Nicolas shares how they solved this issue at Malt by leveraging the tools they already had in place. From ingestion to training, Nicolas provides insights on how to transform BigQuery into an effective feature management system.

We cover how Nicolas' team designed their feature tables and addressed challenges such as monitoring, alerting, data quality, point-in-time lookups, and backfilling. If you’re looking for a simpler way to manage your features without the overhead of additional software, this talk is for you. Discover how BigQuery can handle it all!

// Bio

Nicolas Mauti is the go-to guy for all things related to MLOps at Malt. With a knack for turning complex problems into streamlined solutions and over a decade of experience in code, data, and ops, he is a driving force in developing and deploying machine learning models that actually work in production. When he's not busy optimizing AI workflows, you can find him sharing his knowledge at the university. Whether it's cracking a tough data challenge or cracking a joke, Nicolas knows how to keep things interesting.

// MLOps Jobs board jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related LinksNicolas' Medium - https://medium.com/@nmauti

Data Engineering for AI/ML Conference: https://home.mlops.community/home/events/dataengforai

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Nicolas on LinkedIn: https://www.linkedin.com/in/nicolasmauti/?locale=en_US

Timestamps:

[00:00] Nicolas' preferred beverage

[00:35] Takeaways

[02:25] Please like, share, leave a review, and subscribe to our MLOps channels!

[02:57] BigQuery end goal

[05:00] BigQuery pain points

[10:14] BigQuery vs Feature Stores

[12:54] Freelancing Rate Matching issues

[16:43] Post-implementation pain points

[19:39] Feature Request Process

[20:45] Feature Naming Consistency

[23:42] Feature Usage Analysis

[26:59] Anomaly detection in data

[28:25] Continuous Model Retraining Process

[30:26] Model misbehavior detection

[33:01] Handling model latency issues

[36:28] Accuracy vs The Business

[38:59] BigQuery cist-benefit analysis

[42:06] Feature stores cost savings

[44:09] When not to use BigQuery

[46:20] Real-time vs Batch Processing

[49:11] Register for the Data Engineering for AI/ML Conference now!

[50:14] Wrap up

Design and Development Principles for LLMOps // MLOps Podcast #254 with Andy McMahon, Director - Principal AI Engineer at Barclays Bank.

A huge thank you to SAS for their generous support!

// Abstract

As we move from MLOps to LLMOps, we need to double down on some fundamental software engineering practices, as well as augment and add to these with some new techniques. In this case, let's talk about this!

// Bio

Andy is a Principal AI Engineer, working in the new AI Center of Excellence at Barclays Bank. Previously, he was Head of MLOps for NatWest Group, where he led their MLOps Center of Excellence and helped build out their MLOps platform and processes across the bank. Andy is also the author of Machine Learning Engineering with Python, a hands-on technical book published by Packt.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related LinksAndy's book - https://packt.link/w3JKL

Andy's Medium - https://medium.com/@andrewpmcmahon629

SAS: https://www.sas.com/en_us/home.html

SAS® Decision Builder: https://www.sas.com/en_us/offers/23q4/microsoft-fabric.html

Data Engineering for AI/ML Conference: https://home.mlops.community/home/events/dataengforai

Harnessing MLOps in Finance // Michelle Marie Conway // MLOps Podcast Coffee #174: https://youtu.be/nIEld_Q6L-0

The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses, by Eric Ries: https://www.amazon.co.jp/-/en/Eric-Ries/dp/0307887898

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Andy on LinkedIn: https://www.linkedin.com/in/andrew-p-mcmahon/

Timestamps:

[00:00] Andy's preferred coffee

[00:09] Takeaways

[02:04] Andy's book is an Oxford curriculum

[06:13] Register for the Data Engineering for AI/ML Conference now!

[07:04] The Life Cycle of AI Executives Course

[09:55] MLOps as a term

[11:53] Tooling vs Process Culture

[15:01] Open source benefits

[17:15] End goal flexibility

[20:06] Hybrid Cloud Strategy Overview

[21:11] ROI for tool upgrades

[25:41] Long-term projects comparison

[29:02 - 30:48] SAS Ad

[30:49] AI and ML Integration

[35:40] Hybrid AI Integration Insights

[42:18] Tech trends vs Practicality

[44:39] Gen AI Tooling Debate

[51:57] Vanity metrics overview

[55:22] Tech business alignment strategy

[58:45] Aligning teams for ROI

[1:01:35] Communication mission effectively

[1:03:45] Enablement metrics

[1:06:38] Prioritizing use cases

[1:09:47] Wrap up