Agentic Conversations (formally mlops.community)

MLOps Coffee Sessions #176 with MLOps vs. LLMOps Panel, Willem Pienaar, Chris Van Pelt, Aparna Dhinakaran, and Alex Ratner hosted by Richa Sachdev.

MLOps Coffee Sessions #173 with Beyang Liu, Building Cody, an Open Source AI Coding Assistant.

We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O

// Abstract

Root about the development of Cody, an open-source AI coding assistant. Cody empowers developers to query and comprehend code within codebases through the integration of robust language model capabilities. Sourcegraph tackles the intricacies of understanding intricate codebases by creating comprehensive code maps and employing AI for advanced search functionalities. Cody harnesses the potential of AI to offer features such as code exploration, natural language queries, and AI-powered code generation, augmenting developer productivity and code comprehension.

// Bio

Beyang Liu is the CTO and Co-founder of Sourcegraph. Prior to Sourcegraph, Beyang was an engineer at Palantir Technologies, building large-scale data analysis tools for Fortune 500 companies with large, complex codebases. Beyang studied computer science at Stanford, where he discovered his love for compilers and published some machine learning research as a member of the Stanford AI Lab.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related

Links⁠Website: https://beyang.com

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Beyang on LinkedIn: https://www.linkedin.com/in/beyang-liu/

Timestamps:

[00:00] Beyang's preferred coffee

[00:19] Takeaways

[01:25] Please like, share, and subscribe to our MLOps channels!

[01:48] Beyang background before Sourcegraph

[03:10] War stories

[04:30] Technological tool solution

[06:41] Landscape change in the past 10 years

[09:32] Code search engine evolution

[16:28] Vector databases

[17:40] Actual tech breakdown

[19:52] Incorporating AI into products amid organizational challenges

[25:39] Breaking down Cody

[28:04] Context fetching

[30:44] AI replicating human code understanding?

[36:22] Key for software creation

[40:26] Speak the language

[42:20] Leveraging LLMs

[44:18] Low code, no code movement

[47:54] Reliability issues amongst agents

[53:12] LLMs used in code and chat generation

[56:12] Dealing with rate limits and followers or failovers

[57:33] Unnecessary comparison

[1:00:26] Wrap up

MLOps Coffee Sessions #174 with Evaluation Panel, Amrutha Gujjar, Josh Tobin, and Sohini Roy hosted by Abi Aryan.

Connect with Amrutha on LinkedIn to learn more about her experience and discuss exciting opportunities in software development and leadership.

Josh Tobin

Founder @ GantryJosh Tobin is the founder and CEO of Gantry. Previously, Josh worked as a deep learning & robotics researcher at OpenAI and as a management consultant at McKinsey. He is also the creator of Full Stack Deep Learning (fullstackdeeplearning.com), the first course focused on the emerging engineering discipline of production machine learning. Josh did his PhD in Computer Science at UC Berkeley advised by Pieter Abbeel.

Sohini Roy

Senior Developer Relations Manager @ NVIDIASohini Bianka Roy is a senior developer relations manager at NVIDIA, working within the Enterprise Product group. With a passion for the intersection of machine learning and operations, Sohini specializes in the domains of MLOps and LLMOps. With her extensive experience in the field, she plays a crucial role in bridging the gap between developers and enterprise customers, ensuring smooth integration and deployment of NVIDIA's cutting-edge technologies.

MLOps Coffee Sessions #172 with Lingjiao Chen, FrugalGPT: Better Quality and Lower Cost for LLM Applications.

This episode is sponsored by QuantumBlack.

We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O

// Abstract

There is a rapidly growing number of large language models (LLMs) that users can query for a fee. We review the cost associated with querying popular LLM APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude. In particular, using LLMs on large collections of queries and text can be expensive. Motivated by this, we outline and discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade that learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g., GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently.

// Bio

Lingjiao Chen is a Ph.D. candidate in the Computer Science department at Stanford University. He is broadly interested in machine learning, data management, and optimization. Working with Matei Zaharia and James Zou, he is currently exploring the fast-growing marketplaces of artificial intelligence and data. His work has been published at premier conferences and journals such as ICML, NeurIPS, SIGMOD, and PVLDB, and partially supported by a Google fellowship.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links⁠Website: https://lchen001.github.io/

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance paper: https://arxiv.org/abs/2305.05176

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Lingjiao on LinkedIn:

Timestamps:

[00:00] Lingjiao's preferred coffee

[00:35] Takeaways

[02:41] Sponsor Ad: Nayur Khan of QuantumBlack

[05:27] Lingjiao's research at Stanford

[07:51] Day-to-day research overview

[10:11] Inventing data management inspired abstractions research

[13:58] Agnostic Approach to Data Management

[15:56] Frugal GPT

[18:59] Just another data provider

[19:51] Frugal GPT breakdown

[26:33] First step of optimizing the prompts

[28:04] Prompt overlap

[29:06] Query Concatenation

[32:30] Money saving

[35:04] Economizing the prompts

[38:52] Questions to accommodate

[41:33] LLM Cascade

[47:25] Frugal GPT saves cost and improves performance

[51:37] End-user implementation

[52:31] Completion Cache

[56:33] Using a vector store

[1:00:51] Wrap up

MLOps Coffee Sessions #172 with LLMs in Production Conference part 2 Building LLM Products Panel, George Mathew, Asmitha Rathis, Natalia Burina, and Sahar Mor Using hosted by TWIML's Sam Charrington.

MLOps Coffee Sessions #171 with Thibaut Labarre, Using Large Language Models at AngelList, co-hosted by Ryan Russon.

We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O

// Abstract

Thibaut innovatively addressed previous system constraints, achieving scalability and cost efficiency. Leveraging AngelList investing and natural language processing expertise, they refined news article classification for investor dashboards. Central is their groundbreaking platform, AngelList Relay, automating parsing and offering vital insights to investors. Amid challenges like Azure OpenAI collaboration and rate limit solutions, Thibaut reflects candidly. The narrative highlights prompt engineering's strategic importance and empowering domain experts for ongoing advancement.

// Bio

Thibaut LaBarre is an engineering lead with a background in Natural Language Processing (NLP). Currently, Thibaut focuses on unlocking the potential of Large Language Model (LLM) technology at AngelList, enabling everyone within the organization to become prompt engineers on a quest to streamline and automate the infrastructure for Venture Capital. Prior to that, Thibaut began his journey at Amazon as an intern, where he built Heartbeat, a state-of-the-art NLP tool that consolidates millions of data points from various feedback sources, such as product reviews, customer contacts, and social media, to provide valuable insights to global product teams. Over the span of seven years, he expanded his internship project into an organization of 20 engineers. He received a M.S. in Computational Linguistics from the University of Washington.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links⁠

Website: https://www.angellist.com/venture/relay

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Ryan on LinkedIn: https://www.linkedin.com/in/ryanrusson/

Connect with Thibaut on LinkedIn: https://www.linkedin.com/in/thibautlabarre/

Timestamps:

[00:00] Thibaut's preferred beverage

[00:50] Takeaways

[04:05] Please like, share, and subscribe to our MLOps channels!

[04:44] A huge fan of Isaac Asimov

[07:20] Thibaut Labarre background

[09:13] AngelList as an organization

[10:50] AI sense of building

[12:29] System trade-offs

[15:20] OpenAI's limitation

[16:31] Human in the loop

[17:22] Classifying relevance

[18:09] Fight for value

[19:37] Added value

[22:10] Exploring efficient ways to automate tasks.

[24:20] Investing in off-the-shelf models

[27:56] AngelList Relay

[30:49] News article and investment document classification technology

[32:39] Back-end tech

[34:09] Prompt layer

[35:28] Prompt layer as a living

[37:04] Foreseeing no human intervention

[39:00] Blocking hallucinations

[40:33] Challenges

[43:49] Investments in other models besides OpenAI

[45:20] Integration with other models

[46:28] Ethical concerns when

[48:37] OpenAI breaking Prompts

[50:46] Wrap up

MLOps Coffee Sessions #170 with Phillip Carter, All the Hard Stuff with LLMs in Product Development.

We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O

// Abstract

Delve into challenges in implementing LLMs, such as security concerns and collaborative measures against attacks. Emphasize the role of ML engineers and product managers in successful implementation. Explore identifying leading indicators and measuring ROI for impactful AI initiatives.

// Bio

Phillip is on the product team at Honeycomb, where he works on a bunch of different developer tooling things. He's an OpenTelemetry maintainer -- chances are, if you've read the docs to learn how to use OTel, you've read his words. He's also Honeycomb's (accidental) prompt engineering expert by virtue of building and shipping products that use LLMs. In a past life, he worked on developer tools at Microsoft, helping bring the first cross-platform version of .NET into the world and grow to 5 million active developers. When not doing computer stuff, you'll find Phillip in the mountains riding a snowboard or backpacking in the Cascades.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links⁠

Website: https://phillipcarter.dev/

https://www.honeycomb.io/blog/improving-llms-production-observability

https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llmhttps://phillipcarter.dev/posts/how-to-make-an-fsharp-code-fixer/

The "hard stuff" post: https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm

Our follow-up on iterating on LLMs in prod: https://www.honeycomb.io/blog/improving-llms-production-observability

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Phillip on LinkedIn: https://www.linkedin.com/in/phillip-carter-4714a135/

Timestamps:

[00:00] Phillip's preferred coffee

[00:33] Takeaways

[01:53] Please like, share, and subscribe to our MLOps channels!

[02:45] Phillip's background

[07:15] Querying Natural Language

[11:25] Function calls

[14:29] Pasting errors or traces

[16:30] Error patterns

[20:22] Honeycomb's Improvement cycle

[23:20] Prompt boxes rationale

[28:06] Prompt injection cycles

[32:11] Injection Attempt

[33:30] UI undervalued, charging the AI feature

[35:11] ROI cost

[44:26] Bridging ML and Product Perspective

[52:53] AI Model Trade-offs

[56:33] Query assistant

[59:07] Honeycomb is hiring!

[1:00:08] Wrap up

MLOps Coffee Sessions #169 with Barak Turovsky, MLOps at the Age of Generative AI.

Thanks to Weights & Biases for sponsoring this episode. Check out their new course on evaluating and fine-tuning LLMs at wandb.me/genai-mlops.course

// Abstract

The talk focuses on MLOps aspects of developing, training, and serving Generative AI/Large Language models

// Bio

Barak is an Executive in Residence at Scale Venture Partners, a leading Enterprise venture capital firm. Barak spent 10 years as Head of Product and User Experience for Languages AI and Google Translate teams within the Google AI org, focusing on applying cutting-edge Artificial Intelligence and Machine Learning technologies to deliver magical experiences across Google Search, Assistant, Cloud, Chrome, Ads, and other products. Previously, Barak spent 2 years as a product leader within the Google Commerce team. Most recently, Barak served as Chief Product Officer, responsible for product management and engineering at Trax, a leading provider of Computer Vision AI solutions for the Retail and Commerce industries. Prior to joining Google in 2011, Barak was Director of Products in Microsoft’s Mobile Advertising, Head of Mobile Commerce at PayPal, and Chief Technical Officer at an Israeli start-up. He lived more than 10 years in 3 different countries (Russia, Israel, and the US) and speaks three languages. Barak earned a Bachelor of Laws degree from Tel Aviv University, Israel, and a Master’s of Business Administration from the University of California, Berkeley.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Bio and links about Barak's work: https://docs.google.com/document/d/1E4Yrmt_Y57oTEYHQQDvt71XzSJ8Ew5WvscAQbHV4K3U/edit

Framework for evaluating Generative AI use cases: https://www.linkedin.com/pulse/framework-evaluating-generative-ai-use-cases-barak-turovsky/?trackingId=%2BMRxEZ9WTPCNH2JscILTeg%3D%3D

The Great A.I. Awakening: https://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Barak on LinkedIn: https://www.linkedin.com/in/baraktur/

Timestamps:

[00:00] Barak's preferred coffee

[00:23] Barak Turovsky's background

[03:10] Please like, share, and subscribe to our MLOps channels!

[04:09] Getting into tech

[08:39] First wave of AI

[12:39] Building a product at scale and the challenges

[15:59] Framework for evaluating Generative AI use cases

[24:33] Machine trust adoption

[29:13] Wandb's new course

[31:10] Focus on achievable use cases for LLMs

[36:36] User feedback

[38:23] Disruption of entertainment and customer interactions

[46:14] Get new tools or work with your own distribution?

[47:57] Importance of data engineers

[53:28] ML Engineers Collaborate with Product[

56:13] Wrap up

MLOps Coffee Sessions #168 with Piotr Niedźwiedź, Experiment Tracking in the Age of LLMs, co-hosted by Vishnu Rachakonda.

// Abstract

Piotr shares his journey as an entrepreneur and the importance of focusing on core values to achieve success. He highlights the mission of Neptune to support ML teams by providing them with control and confidence in their models. The conversation delves into the role of experiment tracking in understanding and debugging models, comparing experiments, and versioning models. Piotr introduces the concept of prompt engineering as a different approach to building models, emphasizing the need for prompt validation and testing methods.

// Bio

Piotr is the CEO of neptune.ai. Day to day, apart from running the company, he focuses on the product side of things. Strategy, planning, ideation, getting deep into user needs and use cases. He really likes it.
Piotr's path to ML started with software engineering. Always liked math and started programming when he was 7. In high school, Piotr got into algorithmics and programming competitions and loved competing with the best. That got him into the best CS and Maths program in Poland which funny enough today specializes in machine learning.
Piotr did his internships at Facebook and Google and was offered to stay in the Valley. But something about being a FAANG engineer didn’t feel right. He had this spark to do more, build something himself. So with a few of his friends from the algo days, they started Codilime, a software consultancy, and later a sister company Deepsense.ai machine learning consultancy, where he was a CTO.
When he came to the ML space from software engineering, he was surprised by the messy experimentation practices, lack of control over model building, and a missing ecosystem of tools to help people deliver models confidently.
It was a stark contrast to the software development ecosystem, where you have mature tools for DevOps, observability, or orchestration to execute efficiently in production. And then, one day, some ML engineers from Deepsense.ai came to him and showed him this tool for tracking experiments they built during a Kaggle competition (which they won btw), and he knew this could be big.
He asked around, and everyone was struggling with managing experiments. He decided to spin it off as a VC-funded product company, and the rest is history.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

https://neptune.ai/blog/author/piotr-niedzwiedz

https://www.youtube.com/playlist?list=PLKePQLVx9tOfKFbg9GY2Anl41Be4T1-m5 
https://thesequence.substack.com/p/-piotr-niedzwiedz-neptunes-ceo-on

https://open.spotify.com/episode/2KEqTMAHODbPKdUEtlrhm7?si=ed862b2ac7534e39
https://www.linkedin.com/in/piotrniedzwiedz/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Piotr on LinkedIn: https://www.linkedin.com/in/piotrniedzwiedz/

Timestamps:

[00:00] Introduction to Piotr Niedźwiedź

[01:35] Please like, share, and subscribe to our MLOps channels!

[01:58] Wojciech Zaremba

[05:20] The Olympiad

[06:31] Building own company

[12:16] Talking outside Poland with the same passion

[13:45] Adapting with Neptune

[19:35] Core values focus

[22:02] Product Vision changes with advances

[29:36] Control and confidence

[30:05] Experiment tracking existing use cases

[37:25] Control pane

[38:59] Piotr's prediction

[43:20] WiFi issues around the world

[44:09] Wrap up

MLOps Coffee Sessions #167 with Maxime Beauchemin, Treating Prompt Engineering More Like Code.

// Abstract

Promptimize is an innovative tool designed to scientifically evaluate the effectiveness of prompts. Discover the advantages of open-sourcing the tool and its relevance, drawing parallels with test suites in software engineering. Uncover the increasing interest in this domain and the necessity for transparent interactions with language models. Delve into the world of prompt optimization, deterministic evaluation, and the unique challenges in AI prompt engineering.

// Bio

Maxime Beauchemin is the founder and CEO of Preset, a Series B startup supporting and commercializing the Apache Superset project. Max was the original creator of Apache Airflow and Apache Superset when he was at Airbnb. Max has over a decade of experience in data engineering at companies like Lyft, Airbnb, Facebook, and Ubisoft.

// MLOps Jobs board

jobs.mlops.community

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Max's first MLOps Podcast episode: https://go.mlops.community/KBnOgN

Test-Driven Prompt Engineering for LLMs with Promptimize blog: https://maximebeauchemin.medium.com/mastering-ai-powered-product-development-introducing-promptimize-for-test-driven-prompt-bffbbca91535

https://maximebeauchemin.medium.com/mastering-ai-powered-product-development-Test-Driven Prompt Engineering for LLMs with Promptimize podcast: https://talkpython.fm/episodes/show/417/test-driven-prompt-engineering-for-llms-with-promptimize

Taming AI Product Development Through Test-driven Prompt Engineering // Maxime Beauchemin // LLMs in Production Conference lightning talk: https://home.mlops.community/home/videos/taming-ai-product-development-through-test-driven-prompt-engineering

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Max on LinkedIn: https://www.linkedin.com/in/maximebeauchemin/

Timestamps:

[00:00] Max introduces the Apache Superset project at Preset

[01:04] Max's preferred coffee

[01:16] Airflow creator

[01:45] Takeaways

[03:53] Please like, share, and subscribe to our MLOps channels!

[04:31] Check Max's first MLOps Podcast episode

[05:20] Promptimize

[06:10] Interaction with API

[08:27] Deterministic evaluation of SQL queries and AI

[12:40] Figuring out the right edge cases

[14:17] Reaction with Vector Database

[15:55] Promptomize Test Suite

[18:48] Promptimize vision

[20:47] The open-source blood

[23:04] Impact of open source

[23:18] Dangers of open source

[25:25] AI-Language Models Revolution

[27:36] Test-driven design

[29:46] Prompt tracking

[33:41] Building Test Suites as Assets

[36:49] Adding new prompt cases to new capabilities

[39:32] Monitoring speed and cost

[44:07] Creating own benchmarks

[46:19] AI feature adding more value to the end users

[49:39] Perceived value of the feature

[50:53] LLMs costs

[52:15] Specialized model versus Generalized model

[56:58] Fine-tuning LLMs use cases

[1:02:30] Classic Engineer's Dilemma

[1:03:46] Build exciting tech that's available

[1:05:02] Catastrophic forgetting

[1:10:28] Prompt-driven development

[1:13:23] Wrap up