Agentic Conversations (formally mlops.community)

// Abstract

Yuri Plotkin is a Biomedical Engineer and Machine Learning Scientist, and the author of The Variational Book.

The Variational Book // MLOps Podcast #253 with Yuri Plotkin, an ML Scientist.

// Abstract

Curiosity has been the underlying thread in Yuri's life and interests. With the explosion of Generative AI, Yuri was fascinated by the topic and decided he needed to learn more. Yuri pursued learning by reading, deriving, and understanding seminal papers within the last generation. The endeavors culminated in the writing of a book on the topic, The Variational Book, which Yuri expects to release shortly in the coming months. A bit of detail about the topics he covers can be found here: www.thevariationalbook.com.

// Bio

Evolved from biomedical engineer to wet-lab scientist, and more recently transitioned Yuri's career to computer science with the last 10+ years developing projects at the intersection of medicine, life sciences, and machine learning. Yuri's educational background is in Biomedical Engineering, at Columbia University (M.S.) and the University of California, San Diego (B.S.). Current interests include generative AI, diffusion models, and LLMs.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: https://plotkiny.github.io/

The Variational Book: www.thevariationalbook.com

SAS: https://www.sas.com/en_us/home.html

SAS® Decision Builder: https://www.sas.com/en_us/offers/23q4/microsoft-fabric.html

Data Engineering for AI/ML Conference: https://home.mlops.community/home/events/dataengforai

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Yuri on LinkedIn: http://www.linkedin.com/in/yuri-plotkin/

Timestamps:

[00:00] Yuri's preferred coffee

[00:23] Takeaways

[01:56] Register for the Data Engineering for AIML Conference now!

[02:47] Yuri's background

[06:13] The Variational Book

[10:25] Not including LLMs in the book

[12:14] Diffusion models

[16:37] Evolution within diffusion models

[20:55] Diffusion models for video

[25:43] Evolution and optimization of algorithms

[28:53] Markovian

[33:06 - 34:52] SAS Ad

[34:53] Understanding Markovian vs Non-Markovian models

[40:28] Visualizing model evolution

[43:46] Models through time

[44:53] The Variational Book inspiration

[47:53] Influencing LLM latent space

[51:07] Understanding ML Architectures

[52:56] Balancing AI complexity

[55:05] Wrap up

// Abstract

Ron Heichmn is an AI researcher specializing in generative AI, AI alignment, and prompt engineering. At SentinelOne, Ron actively monitors emerging research to identify and address potential vulnerabilities in our AI systems, focusing on unsupervised and scalable evaluations to ensure robustness and reliability.

Harnessing AI APIs for Safer, Accurate, & Reliable Applications // MLOps Podcast #252 with Ron Heichman, Machine Learning Engineer at SentinelOne.

// Abstract

Integrating AI APIs effectively is pivotal for building applications that leverage LLMs, especially given the inherent issues with accuracy, reliability, and safety that LLMs often exhibit. I aim to share practical strategies and experiences for using AI APIs in production settings, detailing how to adapt these APIs to specific use cases, mitigate potential risks, and enhance performance. The focus will be on testing, measuring, and improving quality for RAG or knowledge workers utilizing AI APIs.

// Bio

Ron Heichman is an AI researcher and engineer dedicated to advancing the field through his work on prompt injection at Preamble, where he helped uncover critical vulnerabilities in AI systems. Currently at SentinelOne, he specializes in generative AI, AI alignment, and the benchmarking and measurement of AI system performance, focusing on Retrieval-Augmented Generation (RAG) and AI guardrails.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: https://www.sentinelone.com/

All the Hard Stuff with LLMs in Product Development // Phillip Carter // MLOps Podcast #170: https://www.youtube.com/watch?v=DZgXln3v85s&ab_channel=MLOps.community

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Ron on LinkedIn: https://www.linkedin.com/in/heichmanron/

Timestamps:

[00:00] Ron's preferred coffee

[00:20] Takeaways

[01:08] Register now for the Data Engineering for AIML Conference!

[01:59] AI vs ML Solutions

[05:42] AI Application challenges

[09:38] AI Model evolution

[19:22] AI tools accessibility challenge

[20:53] AI tools accessibility gap

[24:00] Optimizing LLM Performance

[30:31] Red teaming taxonomy

[36:11] Securing custom LLMs

[44:32] Diverse data in LLMs

[46:29] Automated data diversity feedback

[50:42] Model stress-testing process

[55:49] Early issue detection benefits

[57:41] Prompt injection patterns

[1:02:11] Best jailbreaks seen by Ron

[1:04:53] Data poisoning vulnerabilities

[1:07:48] Wrap up

This is a panel taken from the recent AI quality Conference presented by the MLOps Community and Kolena

// Abstract

Chinar Movsisyan is the co-founder and CEO of Feedback Intelligence (formerly Manot), an MLOps startup based in San Francisco. She has been in the AI field for more than 7 years, from research labs to venture-backed startups.

Reliable LLM Products, Fueled by Feedback // MLOps Podcast #250 with Chinar Movsisyan, CEO of Feedback Intelligence.

// Abstract

We live in a world driven by large language models (LLMs) and generative AI, but ensuring they are ready for real-world deployment is crucial. Despite the availability of numerous evaluation tools, many LLM products still struggle to make it to production. We propose a new perspective on how LLM products should be measured, evaluated, and improved. A product is only as good as the user's experience and expectations, and we aim to enhance LLM products to meet these standards reliably.

Our approach creates a new category that automates the need for separate evaluation, observability, monitoring, and experimentation tools. By starting with the user experience and working backward to the model, we provide a comprehensive view of how the product is actually used, rather than how it is intended to be used. This user-centric, aka feedback-centric, approach is the key to every successful product.

// Bio

Chinar Movsisyan is the founder and CEO of Feedback Intelligence, an MLOps company based in San Francisco that enables enterprises to make sure that LLM-based products are reliable and that the output is aligned with end-user expectations. With over eight years of experience in deep learning, spanning from research labs to venture-backed startups, Chinar has led AI projects in mission-critical applications such as healthcare, drones, and satellites. Her primary research interests include artificial intelligence, generative AI, machine learning, deep learning, and computer vision. At Feedback Intelligence, Chinar and her team address a crucial challenge in LLM development by automatically converting user feedback into actionable insights, enabling AI teams to analyze root causes, prioritize issues, and accelerate product optimization. This approach is particularly valuable in highly regulated industries, helping enterprises to reduce time-to-market and time-to-resolution while ensuring robust LLM products. Feedback Intelligence, which participated in the Berkeley SkyDeck accelerator program, is currently expanding its business across various verticals.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: https://www.manot.ai/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Chinar on LinkedIn: https://www.linkedin.com/in/nik-suresh/

Timestamps:

[00:00] Chinar's preferred coffee

[00:20] Takeaways

[02:25] Please like, share, leave a review, and subscribe to our MLOps channels!

[03:23] Object Detection on Drones

[06:10] Street Surveillance Detection Use Case

[08:00] Optimizing Vision Models

[09:50] Data Engineering for AI/ML Conference Ad

[10:42] Plastic surgery project

[12:33] Diffusion models are getting popular

[13:57] AI challenges in highly regulated industries

[17:48] Product metrics evaluation insights

[20:55] Chatbot effectiveness metrics

[23:15] Interpreting user signals

[24:45] Metadata tracking in LLM

[27:41] Agentic workflow

[28:53] Effective data analysis strategies

[30:41] Identifying key metrics

[33:59] AI metrics role shift

[37:20] Tooling for non-engineers

[42:12] Balancing engineering and evaluation

[44:39] Bridging SME engineering gap

[46:41] Expand expertise potential

[47:40] What's with flamingos

[48:04] Wrap up

This is a Panel taken from the recent AI Quality Conference presented by the MLOps COmmunity and Kolena

// Abstract

Timestamps:

00:00 Panelists discuss vision and strategy in AI

05:18 Steven Eliuk, IBM expertise in data services

07:30 AI as means to improve business metrics

11:10 Key metrics in production systems: efficiency and revenue

13:50 Consistency in data standards aids data integration

17:47 Generative AI presents new data classification risks

22:47 Evaluating implications, monitoring, and validating use cases

26:41 Evaluating natural language answers for efficient production

29:10 Monitoring AI models for performance and ethics

31:14 AI metrics and user responsibility for future models

34:56 Access to data is improving, promising progress

Nik Suresh wrote an evisceration of the current AI hype boom called "I Will F**king Piledrive You If You Mention AI Again."

AI Operations Without Fundamental Engineering Discipline // MLOps Podcast #250 with Nikhil Suresh, Director @ Hermit Tech.

// Abstract

Nik is on the podcast because of an anti-AI hype piece, so a reasonable thing to discuss is going to be what most companies are getting wrong when non-technical management wants to immediately roll out ML initiatives, but are unwilling to bring technical naysayers on board who will set them up for success.

// Bio

Nik is the author of ludic.mataroa.blog, who wrote "I Will [REDACTED] Piledriver You If You Mention AI Again", and mostly works in the data engineering and data science spaces. Nik's writing and company both focus on bringing more care to work, pushing back against the industry's worst excesses both technically and spiritually, and getting fundamentals right.

Nik also has a reasonably strong background in psychology. His data science training was of the pre-LLM variety, circa. 2018, when there was a lot of hype, but it wasn't this ridiculous.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: https://ludic.mataroa.blog/

Nik's blog: https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you-if-you-mention-ai-again/

Harnessing MLOps in Finance // Michelle Marie Conway // MLOps Podcast Coffee #174: https://youtu.be/nIEld_Q6L-0

Fundamentals of Data Engineering: Plan and Build Robust Data Systems Audiobook: Joe Reis, Matt Housley: https://audiobookstore.com/audiobooks/fundamentals-of-data-engineering.aspx

Bullshit Jobs: A Theory Hardcover by David Graeber: https://www.amazon.co.jp/-/en/David-Graeber/dp/0241263883

Does a Frog have Scorpion Nature podcast: https://open.spotify.com/show/57i8sYVqxG4i3NvBniLfhv

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Nik on LinkedIn: https://www.linkedin.com/in/nik-suresh/

Timestamps:

[00:00] Nik's preferred coffee

[00:30] Takeaways

[01:40] Please like, share, leave a review, and subscribe to our MLOps channels!

[01:56] AI hype and humor

[07:21] Defining project success

[08:57] Effective data utilization

[12:18] AI Hype vs Data Engineering

[14:44] AI implementation challenges

[17:44 - 18:35] Data Engineering for AI and ML Virtual Conference Ad

[18:35] Managing AI Expectations

[22:08] AI expectations vs reality

[26:00] Balancing Engineering and AI

[31:54] Highlighting engineer success

[35:25] The real challenges

[36:30] Embracing work challenges

[37:21] Dealing with podcast disappointments

[40:50] Creating content for visibility

[43:02] Exploring niche interests

[44:14] Relationship building

[47:15] Strategic approach to success

[48:36] Wrap up

Eric Landry is a seasoned AI and Machine Learning leader with extensive expertise in software engineering and practical applications in NLP, document classification, and conversational AI. With technical proficiency in Java, Python, and key ML tools, he leads the Expedia Machine Learning Engineering Guild and has spoken at major conferences like Applied Intelligence 2023 and KDD 2020.

AI in Healthcare // MLOps Podcast #249 with Eric Landry, CTO/CAIO @ Zeteo Health.

// Abstract

Eric Landry discusses the integration of AI in healthcare, highlighting use cases like patient engagement through chatbots and managing medical data. He addresses benchmarking and limiting hallucinations in LLMs, emphasizing privacy concerns and data localization. Landry maintains a hands-on approach to developing AI solutions and navigating the complexities of healthcare innovation. Despite necessary constraints, he underscores the potential for AI to proactively engage patients and improve health outcomes.

// Bio

Eric Landry is a technology veteran with 25+ years of experience in the healthcare, travel, and computer industries, specializing in machine learning engineering and AI-based solutions. Holding a Master's in SWE (NLP thesis topic) from the University of Texas at Austin, 2005. He has showcased his expertise and leadership in the field with three US patents, published articles on machine learning engineering, and speaking engagements at the 2023 Applied Intelligence Live, 2020 KDD conference, Data Science Salon 2024, and former leader of Expedia’s MLE guild. Formerly, Eric was the director of AI Engineering and Conversation Platform at Babylon Health and Expedia. Currently CTO/CAIO at Zeteo Health.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: https://www.zeteo.health/

Building Threat Detection Systems: An MLE's Perspective // Jeremy Jordan // MLOps Podcast #134: https://youtu.be/13nOmMJuiAo

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Eric on LinkedIn: https://www.linkedin.com/in/jeric-landry/

Timestamps:

[00:00] Eric's preferred coffee

[00:16] Takeaways

[01:16] Please like, share, leave a review, and subscribe to our MLOps channels!

[01:32] ML and AI in 2005

[04:43] Last job at Babylon Health

[10:57] Data access solutions

[14:35] Prioritize AI ML Team Success

[16:39] Eric's current work

[20:36] Engage in holistic help

[22:13] High-stakes chatbots

[27:30] Navigating Communication Across Diverse Communities

[31:49] When Bots Go Wrong

[34:15] Health care challenges ahead

[36:05] Behavioral health tech challenges

[39:45] Stress from Apps Notifications

[41:11] Combining different guardrails tools

[47:16] Navigating Privacy AI

[50:12] Wrap up

Aniket Kumar Singh is a Vision Systems Engineer at Ultium Cells, skilled in Machine Learning and Deep Learning. I'm also engaged in AI research, focusing on Large Language Models (LLMs).

Evaluating the Effectiveness of Large Language Models: Challenges and Insights // MLOps Podcast #248 with Aniket Kumar Singh, CTO @ MyEvaluationPal | ML Engineer @ Ultium Cells.

// Abstract

Dive into the world of Large Language Models (LLMs) like GPT-4. Why is it crucial to evaluate these models, how we measure their performance, and the common hurdles we face? Drawing from Aniket's research, he shares insights on the importance of prompt engineering and model selection. Aniket also discusses real-world applications in healthcare, economics, and education, and highlights future directions for improving LLMs.

// Bio

Aniket is a Vision Systems Engineer at Ultium Cells, skilled in Machine Learning and Deep Learning. I'm also engaged in AI research, focusing on Large Language Models (LLMs).

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: www.aniketsingh.me

Aniket's AI Research for Good blog that I plan to utilize to share any new research that would focus on the good: www.airesearchforgood.org

Aniket's papers: https://scholar.google.com/citations?user=XHxdWUMAAAAJ&hl=en

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Aniket on LinkedIn: https://www.linkedin.com/in/singh-k-aniket/

Timestamps:

[00:00] Aniket's preferred coffee

[00:14] Takeaways

[01:29] Aniket's job and hobby

[03:06] Evaluating LLMs: Systems-Level Perspective

[05:55] Rule-based system

[08:32] Evaluation Focus: Model Capabilities

[13:04] LLM Confidence

[13:56] Problems with LLM Ratings

[17:17] Understanding AI Confidence Trends

[18:28] Aniket's papers

[20:40] Testing AI Awareness

[24:36] Agent Architectures Overview

[27:05] Leveraging LLMs for tasks

[29:53] Closed systems in Decision-Making

[31:28] Navigating model Agnosticism

[33:47] Robust Pipeline vs Robust Prompt

[34:40] Wrap up