Agentic Conversations (formally mlops.community)

Richard Cloete is a computer scientist and a Laukien-Oumuamua Postdoctoral Research Fellow at the Center for Astrophysics, Harvard University. He is a member of the Galileo Project working under the supervision of Professor Avi, having recently held a postdoctoral position at the University of Cambridge, UK.

AI & Aliens: New Eyes on Ancient Questions // MLOps Podcast #288 with Richard Cloete, Laukien-Oumuamua Postdoctoral Research Fellow at Harvard University.

// Abstract

Demetrios speaks with Dr. Richard Cloete, a Harvard computer scientist and founder of SEAQR Robotics, about his AI-driven work in tracking Unidentified Aerial Phenomena (UAPs) through the Galileo Project. Dr. Cloete explains their advanced sensor setup and the challenges of training AI in this niche field, leading to the creation of AeroSynth, a synthetic data tool.

He also discusses his collaboration with the Minor Planet Center on using AI to classify interstellar objects and upcoming telescope data. Additionally, he introduces Seeker Robotics, applying similar AI techniques to oceanic research with unmanned vehicles for marine monitoring. The conversation explores AI’s role in advancing our understanding of space and the ocean.

// Bio

Richard is a computer scientist and Laukien-Oumuamua Postdoctoral Research Fellow at the Center for Astrophysics, Harvard University. As a member of the Galileo Project under Professor Avi Loeb's supervision, he develops AI models for detecting and tracking aerial objects, specializing in Unidentified Anomalous Phenomena (UAP). Beyond UAP research, he collaborates with astronomers at the Minor Planet Center to create AI models for identifying potential interstellar objects using the upcoming Vera C. Rubin Observatory. Richard is also the CEO and co-founder of SEAQR Robotics, a startup developing advanced unmanned surface vehicles to accelerate the discovery of novel life and phenomena in Earth's oceans and atmosphere. Before joining Harvard, he completed a postdoctoral fellowship at the University of Cambridge, UK, where his research explored the intersection of emerging technologies and law.

Grew up in Cape Town, South Africa, where I used to build Tesla Coils, plasma globes, radio stethoscopes, microwave guns, AM radios, and bombs...

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

Website: www.seaqr.nethttps://itc.cfa.harvard.edu/people/richard-cloete

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Richard on LinkedIn: https://www.linkedin.com/in/richard-cloete/

Timestamps:

[00:00] Richard's preferred coffee

[00:16] Takeaways

[01:42] AI for Human-Centered Design

[09:10] AI Storm Tracking Use Case

[13:34] Edge Computing Workflow

[17:18] UAP AI Space Exploration

[25:24] Seeker Robotics Exploration

[36:25] Data Demand and Use Cases

[43:57] Unexpected Movement Patterns

[46:15] Wrap up

A software engineer based in Delft, Alex Strick van Linschoten, recently built Ekko, an open-source framework for adding real-time infrastructure and in-transit message processing to web applications. With years of experience in Ruby, JavaScript, Go, PostgreSQL, AWS, and Docker, I bring a versatile skill set to the table. I hold a PhD in History, have authored books on Afghanistan, and currently work as an ML Engineer at ZenML.

Real LLM Success Stories: How They Actually Work // MLOps Podcast #287 with Alex Strick van Linschoten, ML Engineer at ZenML.

// Abstract

Alex Strick van Linschoten, a machine learning engineer at ZenML, joins the MLOps Community podcast to discuss his comprehensive database of real-world LLM use cases. Drawing inspiration from Evidently AI, Alex created the database to organize fragmented information on LLM usage, covering everything from common chatbot implementations to innovative applications across sectors. They discuss the technical challenges and successes in deploying LLMs, emphasizing the importance of foundational MLOps practices. The episode concludes with a call for community contributions to further enrich the database and collective knowledge of LLM applications.

// Bio

Alex is a Software Engineer based in the Netherlands, working as a Machine Learning Engineer at ZenML. He was previously awarded a PhD in History (specialism: War Studies) from King's College London and has authored several critically acclaimed books based on his research work in Afghanistan.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

Website: https://mlops.systems

https://www.zenml.io/llmops-databasehttps://www.zenml.io/llmops-database

https://www.zenml.io/blog/llmops-in-production-457-case-studies-of-what-actually-works

https://www.zenml.io/blog/llmops-lessons-learned-navigating-the-wild-west-of-production-llms

https://www.zenml.io/blog/demystifying-llmops-a-practical-database-of-real-world-generative-ai-implementations

https://huggingface.co/datasets/zenml/llmops-database

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Alex on LinkedIn: https://www.linkedin.com/in/strickvl

Timestamps:

[00:00] Alex's preferred tea

[00:15] Takeaways

[00:55] LLM Database Creation Insights

[03:26] Hidden Gems and LLMs

[07:04] Chatbot Governance Challenges

[15:16] AI Agents and IPOs

[19:51] AI Interface Evolution

[23:56] LLMs as Product Guides

[26:57] RAG with User Context

[30:29] User Experience Friction Points

[36:20] ROI and Engineering Insights

[41:09] Agent Debugging and Flows

[45:28] Data Viz Ideas LLM

[47:41] Wrap up

In his 13 years of software engineering, Ilya Reznik has specialized in commercializing machine learning solutions and building robust ML platforms. He's held technical lead and staff engineering roles at premier firms like Adobe, Twitter, and Meta. Currently, Ilya channels his expertise into his travel startup, Jaunt, while consulting and advising emerging startups.

Navigating Machine Learning Careers: Insights from Meta to Consulting // MLOps Podcast #286 with Ilya Reznik, ML Engineering Thought Leader at Instructed Machines, LLC.

// Abstract

Ilya Reznik's insights into machine learning and career development within the field. With over 13 years of experience at leading tech companies such as Meta, Adobe, and Twitter, Ilya emphasizes the limitations of traditional model fine-tuning methods. He advocates for alternatives like prompt engineering and knowledge retrieval, highlighting their potential to enhance AI performance without the drawbacks associated with fine-tuning.

Ilya's recent discussions at the NeurIPS conference reflect a shift towards practical applications of Transformer models and innovative strategies like curriculum learning. Additionally, he shares valuable perspectives on navigating career progression in tech, offering guidance for aspiring ML engineers aiming for senior roles. His narrative serves as a blend of technical expertise and practical career advice, making it a significant resource for professionals in the AI domain.

// Bio

Ilya has navigated a diverse career path since 2011, transitioning from physicist to software engineer, data scientist, ML engineer, and now content creator. He is passionate about helping ML engineers advance their careers and making AI more impactful and beneficial for society.

Previously, Ilya was a technical lead at Meta, where he contributed to 12% of the company’s revenue and managed approximately 30 production ML models. He also worked at Twitter, overseeing offline model evaluation, and at Adobe, where his team was responsible for all intelligent services within Adobe Analytics.

Based in Salt Lake City, Ilya enjoys the outdoors, tinkering with Arduino electronics, and, most importantly, spending time with his family.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

Website: mlepath.com

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Ilya on LinkedIn: https://www.linkedin.com/in/ibreznik/

Timestamps:

[00:00] Ilya's preferred coffee

[00:12] Takeaways

[01:10] Fine-tuning: Pros and Cons

[02:26] Fine-tuning Gamble

[07:50] LLM Architecture Tradeoffs

[11:37] NeurIPS Takeaways and Insights

[22:42] AI Benchmarking and Data Leaks

[29:30] Staff ML Engineer Path

[40:08] Staff Engineer Titles Explained

[49:00] E6 to E7 Expectations

[51:43] Tech Jobs vs Startups

[55:33] Senior to Staff Journey

[58:49] Wrap up

Tomaž Levak is the Co-founder and CEO of Trace Labs – OriginTrail core developers. OriginTrail is a web3 infrastructure project combining a decentralized knowledge graph (DKG) and blockchain technologies to create a neutral, inclusive ecosystem.

Collective Memory for AI on Decentralized Knowledge Graph // MLOps Podcast #285 with Tomaz Levak, Founder of Trace Labs, Core Developers of OriginTrail.

// Abstract

The talk focuses on how OriginTrail Decentralized Knowledge Graph serves as a collective memory for AI and enables neuro-symbolic AI. We cover the basics of OriginTrail’s symbolic AI fundamentals (i.e., knowledge graphs) and go over the details of how decentralization improves data integrity, provenance, and user control. We’ll cover the DKG role in AI agentic frameworks and how it helps with verifying and accessing diverse data sources, while maintaining compatibility with existing standards. We’ll explore practical use cases from the enterprise sector as well as the latest integrations into frameworks like ElizaOS. We conclude by outlining the future potential of decentralized AI, AI becoming the interface to “eat” SaaS, and the general convergence of AI, Internet, and Crypto.

// Bio

Tomaz Levak, founder of OriginTrail, is active at the intersection of Cryptocurrency, the Internet, and Artificial Intelligence (AI). At the core of OriginTrail is a pursuit of Verifiable Internet for AI, an inclusive framework addressing critical challenges of the world in an AI era. To achieve the goal of Verifiable Internet for AI, OriginTrail's trusted knowledge foundation ensures the provenance and verifiability of information while incentivizing the creation of high-quality knowledge. These advancements are pivotal to unlock the full potential of AI as they minimize the technology’s shortfalls, such as hallucinations, bias, issues of data ownership, and model collapse.

Tomaz's contributions to OriginTrail span over a decade and across multiple fields. He is involved in strategic technical innovations for OriginTrail Decentralized Knowledge Graph (DKG) and NeuroWeb blockchain and was among the authors of all three foundational White Paper documents that defined how OriginTrail technology addresses global challenges. Tomaz contributed to the design of OriginTrail token economies and is driving adoption with global brands such as British Standards Institution, Swiss Federal Railways, and World Federation of Haemophilia, among others.

Committed to the ongoing expansion of the OriginTrail ecosystem, Tomaz is a regular speaker at key industry events. In his appearances, he highlights the significant value that the OriginTrail DKG brings to diverse sectors, including supply chains, life sciences, healthcare, and scientific research. In a rapidly evolving digital landscape, Tomaz and the OriginTrail ecosystem as a whole are playing an important role in ensuring a more inclusive, transparent, and decentralized AI.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

Website: https://origintrail.io

Song recommendation: https://open.spotify.com/track/5GGHmGNZYnVSdRERLUSB4w?si=ae744c3ad528424b

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Tomaz on LinkedIn: https://www.linkedin.com/in/tomazlevak/

Timestamps:

[00:00] Tomaz's preferred coffee

[00:20] Takeaways

[01:08] Knowledge Graphs and V8

[02:47] Knowledge Graph Neighborhoods

[07:08] DKG Enterprise Value Prop

[17:33] DKG Multi-Chain Agent Use

[23:40] DKG Use Cases

[26:57] Wearables Data Integration Ideas

[33:19] Knowledge Graph Value Models

[36:14] Replacing Bad Information

[44:13] Node Hosting Responsibilities

[50:47] Wrap up

Krishna Sridhar is an experienced engineering leader passionate about building wonderful products powered by machine learning.

Efficient Deployment of Models at the Edge // MLOps Podcast #284 with Krishna Sridhar, Vice President of Qualcomm.

Big shout-out to Qualcomm for sponsoring this episode!

// Abstract

Qualcomm® AI Hub helps to optimize, validate, and deploy machine learning models on-device for vision, audio, and speech use cases. With Qualcomm® AI Hub, you can: Convert trained models from frameworks like PyTorch and ONNX for optimized on-device performance on Qualcomm® devices.

Profile models on-device to obtain detailed metrics, including runtime, load time, and compute unit utilization. Verify numerical correctness by performing on-device inference. Easily deploy models using Qualcomm® AI Engine Direct, TensorFlow Lite, or ONNX Runtime.

The Qualcomm® AI Hub Models repository contains a collection of example models that use Qualcomm® AI Hub to optimize, validate, and deploy models on Qualcomm® devices. Qualcomm® AI Hub automatically handles model translation from source framework to device runtime, applying hardware-aware optimizations, and performs physical performance/numerical validation. The system automatically provisions devices in the cloud for on-device profiling and inference. The following image shows the steps taken to analyze a model using Qualcomm® AI Hub.

// Bio

Krishna Sridhar leads engineering for Qualcomm™ AI Hub, a system used by more than 10,000 AI developers spanning 1,000 companies to run more than 100,000 models on Qualcomm platforms. Prior to joining Qualcomm, he was Co-founder and CEO of Tetra AI, which made it easy to efficiently deploy ML models on mobile/edge hardware. Prior to Tetra AI, Krishna helped design Apple's CoreML, which was a software system mission-critical to running several experiences at Apple, including Camera, Photos, Siri, FaceTime, Watch, and many more across all major Apple device operating systems and all hardware and IP blocks. He has a Ph.D. in computer science from the University of Wisconsin-Madison and a bachelor’s degree in computer science from Birla Institute of Technology and Science, Pilani, India.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

Website: https://www.linkedin.com/in/srikris/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Krishna on LinkedIn: https://www.linkedin.com/in/srikris/

Timestamps:

[00:00] Krishna's preferred coffee

[00:12] Takeaways

[01:27] Please like, share, leave a review, and subscribe to our MLOps channels!

[01:56] AI Entrepreneurship Journey

[04:25] Core ML and Edge AI

[08:44] AI Stack & Workflow Strategy

[11:42] On-device AI Foundations[17:15] Hardware vs Software Optimization

[21:32] On-device AI Challenges

[26:19] Small LLM Orchestration

[28:03] Memory Constraints and Shared Pools

[30:05] Qualcomm AI Hub Edge

[32:53] AI in Unexpected Places

[41:53] Deploying AI on Edge

[45:58] 4X Battery Optimization Tips

[51:00] Wrap up

Machine Learning, AI Agents, and Autonomy // MLOps Podcast #283 with Zach Wallace, Staff Software Engineer at Nearpod Inc.

// Abstract

Demetrios chats with Zach Wallace, engineering manager at Nearpod, about integrating AI agents in e-commerce and edtech. They discuss using agents for personalized user targeting, adapting AI models with real-time data, and ensuring efficiency through clear task definitions. Zach shares how Nearpod streamlined data integration with tools like Redshift and DBT, enabling real-time updates. The conversation covers challenges like maintaining AI in production, handling high-quality data, and meeting regulatory standards. Zach also highlights the cost-efficiency framework for deploying and decommissioning agents and the transformative potential of LLMs in education.

// Bio

Software Engineer with 10 years of experience. Started my career as an Application Engineer, but I have transformed into a Platform Engineer. As a Platform Engineer, I have handled the problems described below: - Localization across 6-7 different languages - Building a custom local environment tool for our engineers - Building a Data Platform - Building standards and interfaces for Agentic AI within ed-tech.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

https://medium.com/renaissance-learning-r-d/data-platform-transform-a-data-monolith-9d5290a552ef

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: /dpbrinkm/

Connect with Zach on LinkedIn: https: /zachary-wallace/

Timestamps:

[00:00] Zach's preferred coffee

[00:24] Takeaways

[01:25] Data platform pivot

[04:06] Data integration with DBT

[06:50] Data mesh partial adoption

[08:55] Data product

[10:11] Agent Architectures and Deployment

[15:35] Agent vs LLM

[20:28] AI Agent Analytics

[22:18] Agent Design and Scope

[26:52] DAG Agent Workflow Design

[30:25] Cost Considerations in AI

[35:00] Agent Deployment and Costing

[42:25] AI Evaluation Use Cases

[45:25] Agent vs ML for Contracts

[46:55] Wrap up

For three years, Egor has been bringing the power of AI to bear at Wise, across domains as varied as trading algorithms for Treasury, fraud detection, experiment analysis, causal inference, and, recently, the numerous applications unlocked by large language models. Open-source projects initiated and guided by Egor include wise-pizza, causaltune, and neural-lifetimes, with more on the way.

Machine Learning, AI Agents, and Autonomy // MLOps Podcast #282 with Egor Kraev, Head of AI at Wise Plc.

// Abstract

Demetrios chats with Egor Kraev, principal AI scientist at Wise, about integrating large language models (LLMs) to enhance ML pipelines and humanize data interactions. Egor discusses his open-source MotleyCrew framework, career journey, and insights into AI's role in fintech, highlighting its potential to streamline operations and transform organizations.

// Bio

Egor first learned mathematics in the Russian tradition, then continued his studies at ETH Zurich and the University of Maryland. Egor has been doing data science since the last century, including economic and human development data analysis for nonprofits in the US, the UK, and Ghana, and 10 years as a quant, solutions architect, and occasional trader at UBS, then Deutsche Bank. Following last decade's explosion in AI techniques, Egor became Head of AI at Mosaic Smart Data Ltd, and for the last four years has been bringing the power of AI to bear at Wise, in a variety of domains, from fraud detection to trading algorithms and causal inference for A/B testing and marketing. Egor has multiple side projects, such as RL for molecular optimization, GenAI for generating and solving high school math problems, and others.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

https://github.com/transferwise/wise-pizzahttps://github.com/py-why/causaltunehttps://www.linkedin.com/posts/egorkraev_a-talk-on-experimentation-best-practices-activity-7092158531247755265-q0kt?utm_source=share&utm_medium=member_desktop

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Egor on LinkedIn: https://www.linkedin.com/in/egorkraev/

Timestamps:

[00:00] Egor's preferred coffee

[00:40] Takeaways

[01:56] Swiss pirate party

[04:08] Ghana Work and Wise

[07:21] AI Bridge Unstructured Structured

[12:40] AI in Wise Fintech

[16:26] AI as Dag Node

[19:38] Causal Inference with ML

[25:35] Insight validation process

[31:00] Agent UX and challenges

[37:11] Monthly crew vision

[41:00] Montley Crew Framework Abstraction

[42:57] Causal tune

[47:38] Wise email campaigns

[50:29] Organizational structures in AI

[53:19] AI uncertainty and hallucinations

[55:08] Decentralized org structures

[1:01:34] Permaculture and team synergy

[1:03:29] Wrap up

Re-Platforming Your Tech Stack // MLOps Podcast #281 with Michelle Marie Conway, Lead Data Scientist at Lloyds Banking Group and Andrew Baker, Data Science Delivery Lead at Lloyds Banking Group.

// Abstract

Lloyds Banking Group is on a mission to embrace the power of cloud and unlock the opportunities that it provides. Andrew, Michelle, and their MLOps team have been on a journey over the last 12 months to take their portfolio of circa 10 Machine Learning models in production and migrate them from an on-prem solution to a cloud-based environment. During the podcast, Michelle and Andrew share their reflections as well as some dos (and don’ts!) of managing the migration of an established portfolio.

// Bio

Michelle Marie ConwayMichelle is a Lead Data Scientist in the high-performance data science team at Lloyds Banking Group. With deep expertise in managing production-level Python code and machine learning models, she has worked alongside fellow senior manager Andrew to drive the bank's transition to the Google Cloud Platform. Together, they have played a pivotal role in modernising the ML portfolio in collaboration with a remarkable ML Ops team.

Originally from Ireland and now based in London, Michelle blends her technical expertise with a love for the arts. Andrew Baker graduated from the University of Birmingham with a first-class honours degree in Mathematics and Music with a Year in Computer Science and joined Lloyds Banking Group on their Retail graduate scheme in 2015.

Since 2021, Andrew has worked in the world of data, firstly in shaping the Retail data strategy and most recently as a Data Science Delivery Lead, growing and managing a team of Data Scientists and Machine Learning Engineers. He has built a high-performing team responsible for building and maintaining ML models in production for the Consumer Lending division of the bank.

Andrew is motivated by the role that data science and ML can play in transforming the business and its processes, and is focused on balancing the power of ML with the need for simplicity and explainability that enables business users to engage with the opportunities that exist in this space and the demands of a highly regulated environment.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related LinksWebsite: https://www.michelleconway.co.uk/https://www.linkedin.com/pulse/artificial-intelligence-just-when-data-science-answer-andrew-baker-hfdge/https://www.linkedin.com/pulse/artificial-intelligence-conundrum-generative-ai-andrew-baker-qla7e/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Michelle on LinkedIn: https://www.linkedin.com/in/michelle--conway/

Connect with Andrew on LinkedIn: https://www.linkedin.com/in/andrew-baker-90952289

Timestamps:

[00:00] Michelle and Andrew's preferred coffee

[00:25] Takeaways

[01:03] Please like, share, leave a review, and subscribe to our MLOps channels!

[01:26] 5B Cloud Investment!

[04:54] MLOps end-to-end process

[06:21] ML model handoff evolution

[09:14] ML project finalization questions

[14:15] Cloud migration journey

[17:48] Cloud Flexibility vs Rigidity

[19:18] On-prem to Cloud Transition

[23:02] Vertex AI vs Pointed Solutions

[27:47] Standardizing Model Documentation

[30:17] Cloud Optimization and Efficiency

[35:47] Tech debt challenges overcome

[42:11] Chaos engineering insights

[46:11] MLOps platform team collaboration

[51:02] Wrap up

Jineet Doshi is an award-winning Scientist, Machine Learning Engineer, and Leader at Intuit with over 7 years of experience. He has a proven track record of leading successful AI projects and building machine-learning models from design to production across various domains, which have impacted 100 million customers and significantly improved business metrics, leading to millions of dollars of impact.

Holistic Evaluation of Generative AI Systems // MLOps Podcast #280 with Jineet Doshi, Staff AI Scientist or AI Lead at Intuit.

// Abstract

Evaluating LLMs is essential in establishing trust before deploying them to production. Even post-deployment, evaluation is essential to ensure LLM outputs meet expectations, making it a foundational part of LLMOps. However, evaluating LLMs remains an open problem. Unlike traditional machine learning models, LLMs can perform a wide variety of tasks, such as writing poems, Q&A, summarization, etc. This leads to the question of how to evaluate a system with such broad intelligence capabilities. This talk covers the various approaches for evaluating LLMs, such as classic NLP techniques, red teaming, and newer ones like using LLMs as a judge, along with the pros and cons of each. The talk includes an evaluation of complex GenAI systems like RAG and Agents. It also covers evaluating LLMs for safety and security, and the need to have a holistic approach for evaluating these very capable models.

// Bio

Jineet Doshi is an award-winning AI Lead and Engineer with over 7 years of experience. He has a proven track record of leading successful AI projects and building machine learning models from design to production across various domains, which have impacted millions of customers and have significantly improved business metrics, leading to millions of dollars of impact. He is currently an AI Lead at Intuit, where he is one of the architects and developers of their Generative AI platform, which is serving Generative AI experiences for more than 100 million customers around the world. Jineet is also a guest lecturer at Stanford University as part of their Building LLM Applications class. He is on the Advisory Board of the University of San Francisco’s AI Program. He holds multiple patents in the field, is on the steering committee of MLOps World Conference, and has also co-chaired workshops at top AI conferences like KDD. He holds a Master's degree from Carnegie Mellon University.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

Website: https://www.intuit.com/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Jineet on LinkedIn: https://www.linkedin.com/in/jineetdoshi/

Timestamps:

[00:00] Jineet's preferred coffee

[00:20] Takeaways

[01:24] Please like, share, leave a review, and subscribe to our MLOps channels!

[01:36] LLM evaluation at scale

[03:13] Challenges in GenAI evaluation

[08:09] Eval products vs platforms

[09:28] Evaluation methods for models

[14:03] NLP evaluation techniques

[25:06] LLM as a judge/jury

[31:56] LLMs and pizza brainstorming

[34:07] Cost per answer breakdown

[38:29] Evaluating RAG systems

[44:00] Testing with LLMs and humans

[49:23] Evaluating AI use cases

[54:19] AI workflow stress testing

[55:40] Wrap up

Robert Caulk is responsible for directing software development, enabling research, coordinating company projects, quality control, proposing external collaborations, and securing funding. He believes firmly in open-source, having spent 12 years accruing over 1000 academic citations building open-source software in domains such as machine learning, image analysis, and coupled physical processes. He received his Ph.D. from Université Grenoble Alpes, France, in computational mechanics.

Unleashing Unconstrained News Knowledge Graphs to Combat Misinformation // MLOps Podcast #279 with Robert Caulk, Founder of Emergent Methods.

// Abstract

Indexing hundreds of thousands of news articles per day into a knowledge graph (KG) was previously impossible due to the strict requirement that high-level reasoning, general world knowledge, and full-text context *must* be present for proper KG construction.

The latest tools now enable such general world knowledge and reasoning to be applied cost-effectively to high volumes of news articles. Beyond the low cost of processing these news articles, these tools are also opening up a new, controversial approach to KG building - unconstrained KGs. We discuss the construction and exploration of the largest news-knowledge-graph on the planet - hosted on an endpoint at AskNews.app. During the talk, we aim to highlight some of the sacrifices and benefits that go hand-in-hand with using the infamous unconstrained KG approach.

We conclude the talk by explaining how knowledge graphs like these help to mitigate misinformation. We provide some examples of how our clients are using this graph, such as generating sports forecasts, generating better social media posts, generating regional security alerts, and combating human trafficking.

// Bio

Robert is the founder of Emergent Methods, where he directs research and software development for large-scale applications. He is currently overseeing the structuring of hundreds of thousands of news articles per day in order to build the best news retrieval API in the world: https://asknews.app.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

Website: https://emergentmethods.ai

News Retrieval API: https://asknews.app

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Rob on LinkedIn: https://www.linkedin.com/in/rcaulk/

Timestamps:

[00:00] Rob's preferred coffee

[00:05] Takeaways

[00:55] Please like, share, leave a review, and subscribe to our MLOps channels!

[01:00] Join our Local Organizer Carousel!

[02:15] Knowledge Graphs and ontology

[07:43] Ontology vs Noun Approach

[12:46] Ephemeral tools for efficiency

[17:26] Oracle to PostgreSQL migration

[22:20] MEM Graph life cycle

[29:14] Knowledge Graph Investigation Insights

[33:37] Fine-tuning and distillation of LLMs

[39:28] DAG workflow and quality control

[46:23] Crawling nodes with Phi 3 Llama

[50:05] AI pricing risks and strategies

[56:14] Data labeling and poisoning

[58:34] API costs vs News latency

[1:02:10] Product focus and value

[1:04:52] Ensuring reliable information

[1:11:01] Podcast transcripts as News

[1:13:08] Ontology trade-offs explained

[1:15:00] Wrap up