Hugo speaks with Shreya Shankar, a researcher at UC Berkeley focusing on data management systems with a human-centered approach. Shreya's work is at the cutting edge of human-computer interaction (HCI) and AI, particularly in the realm of large language models (LLMs). Her impressive background includes being the first ML engineer at Viaduct, doing research engineering at Google Brain, and software engineering at Facebook.
In this episode, we dive deep into the world of LLMs and the critical challenges of building reliable AI pipelines. We'll explore:
The fascinating journey from classic machine learning to the current LLM revolutionWhy Shreya believes most ML problems are actually data management issuesThe concept of "data flywheels" for LLM applications and how to implement themThe intriguing world of evaluating AI systems - who validates the validators?Shreya's work on SPADE and EvalGen, innovative tools for synthesizing data quality assertions and aligning LLM evaluations with human preferencesThe importance of human-in-the-loop processes in AI developmentThe future of low-code and no-code tools in the AI landscapeWe'll also touch on the potential pitfalls of over-relying on LLMs, the concept of "Habsburg AI," and how to avoid disappearing up our own proverbial arseholes in the world of recursive AI processes.
Whether you're a seasoned AI practitioner, a curious data scientist, or someone interested in the human side of AI development, this conversation offers valuable insights into building more robust, reliable, and human-centered AI systems.
The livestream on YouTubeShreya's websiteShreya on TwitterData Flywheels for LLM ApplicationsSPADE: Synthesizing Data Quality Assertions for Large Language Model PipelinesWhat We’ve Learned From A Year of Building with LLMsWho Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human PreferencesOperationalizing Machine Learning: An Interview StudyVanishing Gradients on TwitterHugo on TwitterIn the podcast, Hugo also mentioned that this was the 5th time he and Shreya chatted publicly. which is wild!
If you want to dive deep into Shreya's work and related topics through their chats, you can check them all out here:
Outerbounds' Fireside Chat: Operationalizing ML -- Patterns and Pain Points from MLOps PractitionersThe Past, Present, and Future of Generative AILLMs, OpenAI Dev Day, and the Existential Crisis for Machine Learning EngineeringLessons from a Year of Building with LLMsCheck out and subcribe to our lu.ma calendar for upcoming livestreams!