Data Engineering Podcast

Building the Context Flywheel for AI Data Agents


Listen Later

Summary 
In this episode Prukalpa Sankar, co-founder of Atlan, talks about what it takes to build a “context flywheel” for AI agents in data-intensive organizations. She explained why model intelligence alone isn’t enough to make AI useful in production, and how real performance depends on contextual intelligence: institutional knowledge, semantic meaning, procedural know-how, and access to the right tools. She also dug into how metadata catalogs are evolving into broader context layers that serve both humans and agents, and why agentic systems are changing the economics of metadata and governance work. Prakulpa shared Atlan’s perspective on bootstrapping context from existing systems such as warehouses, BI tools, query logs, and SaaS applications, then using simulation, traces, and human governance loops to improve agent accuracy over time. 

Announcements 
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • Your host is Tobias Macey and today I'm interviewing Prukalpa Sankar about strategies for building a context flywheel for your data agents
Interview
  • Introduction
  • How did you get involved in the area of data management?
  • You have spent several years working in the metadata catalog space with Atlan. What are the notable changes in scope, adoption, and application that you have seen since we last spoke (June 2022)?
  • The recurring theme since the start of 2026 has been agentic augmentation of all engineering workflows, including data. How do you differentiate between data catalogs, semantic layers, agent memory, context layers, etc. when architecting an AI-powered data-oriented system?
  • One of the perennial problems with data catalogs, business glossaries, master data management, etc. is the up-front investment required to get a real-world impact. How can agents help reduce the activation energy needed to get to that return on effort?
  • One of the perennial problems in data engineering is fragmentation and siloing of data. This is exacerbated by AI systems due to the introduction of vector data as a new specialization. What are the forces that you are seeing play into the current set of tensions and the architectural primitives that we need to bring to bear to keep things maintainable?
  • Since the introduction of transformer-based generative models we have been combating hallucinations. While we have made progress, it is still critical to ensure accuracy and trustworthiness when working with business data. What are the policy elements of governance and technical controls to ensure a high degree of confidence in agent-generated context and business semantics?
  • What are the most interesting, innovative, or unexpected ways that you have seen teams build context layers for their agentic data workloads?
  • What are the most interesting, unexpected, or challenging lessons that you have learned while working on business context engineering?
  • When is agent-managed context the wrong choice?
  • What are your predictions for the next set of architectural shifts that will be driven by the pressures of AI-powered systems?

Contact Info
  • LinkedIn
Parting Question
  • From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
  • Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
  • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
  • If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
 
  • Atlan
  • Atlan Context Lakehouse
  • Iceberg
  • Business Glossary
  • Master Data Management
  • Semantic Layer
  • Cube.dev
  • MCP == Model Context Protocol
  • A2A == Agent to Agent Protocol
  • Decision Traces
  • Apache Doris
  • StarRocks

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
...more
View all episodesView all episodes
Download on the App Store

Data Engineering PodcastBy Tobias Macey

  • 4.5
  • 4.5
  • 4.5
  • 4.5
  • 4.5

4.5

142 ratings


More shows like Data Engineering Podcast

View all
This Week in Startups by Jason Calacanis

This Week in Startups

1,290 Listeners

The Changelog: Software Development, Open Source by Changelog Media

The Changelog: Software Development, Open Source

289 Listeners

The a16z Show by Andreessen Horowitz

The a16z Show

1,093 Listeners

Software Engineering Daily by Software Engineering Daily

Software Engineering Daily

626 Listeners

Risky Business by Risky Business Media

Risky Business

375 Listeners

Talk Python To Me by Michael Kennedy

Talk Python To Me

583 Listeners

Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

Super Data Science: ML & AI Podcast with Jon Krohn

301 Listeners

NVIDIA AI Podcast by NVIDIA

NVIDIA AI Podcast

345 Listeners

Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

Syntax - Tasty Web Development Treats

982 Listeners

Practical AI by Daniel Whitenack and Chris Benson

Practical AI

208 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

576 Listeners

The Data Engineering Show by The Firebolt Data Bros

The Data Engineering Show

8 Listeners

Latent Space: The AI Engineer Podcast by Latent.Space

Latent Space: The AI Engineer Podcast

101 Listeners

This Day in AI Podcast by Michael Sharkey, Chris Sharkey

This Day in AI Podcast

226 Listeners

The AI Daily Brief: Artificial Intelligence News and Analysis by Nathaniel Whittemore

The AI Daily Brief: Artificial Intelligence News and Analysis

682 Listeners