Vanishing Gradients

Episode 32: Building Reliable and Robust ML/AI Pipelines


Listen Later

Hugo speaks with Shreya Shankar, a researcher at UC Berkeley focusing on data management systems with a human-centered approach. Shreya's work is at the cutting edge of human-computer interaction (HCI) and AI, particularly in the realm of large language models (LLMs). Her impressive background includes being the first ML engineer at Viaduct, doing research engineering at Google Brain, and software engineering at Facebook.

In this episode, we dive deep into the world of LLMs and the critical challenges of building reliable AI pipelines. We'll explore:

  • The fascinating journey from classic machine learning to the current LLM revolution
  • Why Shreya believes most ML problems are actually data management issues
  • The concept of "data flywheels" for LLM applications and how to implement them
  • The intriguing world of evaluating AI systems - who validates the validators?
  • Shreya's work on SPADE and EvalGen, innovative tools for synthesizing data quality assertions and aligning LLM evaluations with human preferences
  • The importance of human-in-the-loop processes in AI development
  • The future of low-code and no-code tools in the AI landscape
  • We'll also touch on the potential pitfalls of over-relying on LLMs, the concept of "Habsburg AI," and how to avoid disappearing up our own proverbial arseholes in the world of recursive AI processes.

    Whether you're a seasoned AI practitioner, a curious data scientist, or someone interested in the human side of AI development, this conversation offers valuable insights into building more robust, reliable, and human-centered AI systems.

    LINKS

    • The livestream on YouTube
    • Shreya's website
    • Shreya on Twitter
    • Data Flywheels for LLM Applications
    • SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines
    • What We’ve Learned From A Year of Building with LLMs
    • Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
    • Operationalizing Machine Learning: An Interview Study
    • Vanishing Gradients on Twitter
    • Hugo on Twitter
    • In the podcast, Hugo also mentioned that this was the 5th time he and Shreya chatted publicly. which is wild!

      If you want to dive deep into Shreya's work and related topics through their chats, you can check them all out here:

      1. Outerbounds' Fireside Chat: Operationalizing ML -- Patterns and Pain Points from MLOps Practitioners
      2. The Past, Present, and Future of Generative AI
      3. LLMs, OpenAI Dev Day, and the Existential Crisis for Machine Learning Engineering
      4. Lessons from a Year of Building with LLMs
      5. Check out and subcribe to our lu.ma calendar for upcoming livestreams!

        ...more
        View all episodesView all episodes
        Download on the App Store

        Vanishing GradientsBy Hugo Bowne-Anderson

        • 5
        • 5
        • 5
        • 5
        • 5

        5

        11 ratings


        More shows like Vanishing Gradients

        View all
        a16z Podcast by Andreessen Horowitz

        a16z Podcast

        1,001 Listeners

        Data Skeptic by Kyle Polich

        Data Skeptic

        470 Listeners

        Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

        Super Data Science: ML & AI Podcast with Jon Krohn

        296 Listeners

        DataFramed by DataCamp

        DataFramed

        269 Listeners

        Practical AI by Practical AI LLC

        Practical AI

        190 Listeners

        Last Week in AI by Skynet Today

        Last Week in AI

        281 Listeners

        Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

        Machine Learning Street Talk (MLST)

        88 Listeners

        Dwarkesh Podcast by Dwarkesh Patel

        Dwarkesh Podcast

        354 Listeners

        No Priors: Artificial Intelligence | Technology | Startups by Conviction

        No Priors: Artificial Intelligence | Technology | Startups

        125 Listeners

        This Day in AI Podcast by Michael Sharkey, Chris Sharkey

        This Day in AI Podcast

        190 Listeners

        Latent Space: The AI Engineer Podcast by swyx + Alessio

        Latent Space: The AI Engineer Podcast

        63 Listeners

        The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis by Nathaniel Whittemore

        The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis

        424 Listeners

        The Next Wave - AI and The Future of Technology by Hubspot Media

        The Next Wave - AI and The Future of Technology

        57 Listeners

        Training Data by Sequoia Capital

        Training Data

        36 Listeners

        High Signal: Data Science | Career | AI by Delphina

        High Signal: Data Science | Career | AI

        4 Listeners