Just Now Possible

Building Alyx: How Arize AI Dogfooded Its Way to an Agentic Future

Listen Later

Guests:

  • SallyAnn DeLucia, Director of Product, Arize
  • Jack Zhou, Staff Engineer, Arize

    • In this episode, we cover:

    • What tracing, observability, and evals really mean in GenAI applications
    • How Arize used its own platform to build Alyx, its AI agent
    • The role of customer success engineers in surfacing repeatable workflows
    • Why early prototyping looked like messy notebooks and hacked-together local apps
    • How dogfooding shaped Alyx’s evolution and built confidence for launch
    • Why evals start messy, and how Arize layered evals across tool calls, sessions, and system-level decisions
    • The importance of cross-functional, boundary-spanning teams in building AI products
    • What’s next for Alyx: moving from “on rails” workflows to more autonomous, agentic planning loops

      • Resources & Links

      • Arize AI — Sign up for a free account and try Alex
      • Arize Blog — Lessons learned from building AI products
      • Maven AI Evals Course — The course Teresa took to learn about evals (Get 35% off with Teresa’s affiliate link)
      • Cursor — The AI-powered code editor used by the Arize engineering team
      • DataDog — For understanding application traces
      • OpenAI GPT Models — GPT-3.5, GPT-4, and newer models used in early and current versions of Alex
      • Jupyter Notebooks — A tool for combining code, data, and notes, used in Arise’s prototyping
      • Axial Coding Method by Hamel Husain — A framework for analyzing data and designing evals

        • Chapters:

        00:00 Introduction to Sally Ann and Jack
        01:08 Overview of Arize.ai and Its Core Components
        01:44 Deep Dive into Tracing, Observability, and Evals
        03:56 Introduction to Alyx: Arize's AI Agent
        04:15 The Genesis and Evolution of Alyx
        08:51 Challenges and Solutions in Building Alyx
        24:33 Prototyping and Early Development of Alyx
        26:22 Exploring the Power of Coding Notebooks
        26:51 Early Experiments with Alyx
        27:59 Challenges with Real Data
        29:20 Internal Testing and Dogfooding
        31:55 The Importance of Evals
        35:16 Developing Custom Evals
        43:09 Future Plans for Alyx
        47:59 How to Get Started with Alyx

        ...more
        View all episodesView all episodes
        Download on the App Store
        Just Now PossibleBy Teresa Torres