Summary 
In this episode Tim Sehn, founder and CEO of DoltHub, talks about Dolt - the world’s first version‑controlled SQL database - and why Git‑style semantics belong at the heart of data systems and AI workflows. Tim explains how Dolt combines a MySQL/Postgres‑compatible interface with a novel storage engine built on a “Prollytree” to enable fast, row‑level branching, merging, and diffs of both schema and data. He digs into real production use cases: powering applications that expose version control to end users, reproducible ML feature stores, managing massive configuration for games, and enabling safe agentic writes via branch‑based review flows. He compares Dolt’s approach to LakeFS, Neon, and PlanetScale, and explores developer workflows unlocked by decentralized clones, full audit logs, and PR‑style data reviews. 

Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
• Your host is Tobias Macey and today I'm interviewing Tim Sehn about Dolt, a version controlled database engine and its applications for agentic workflows

Interview
 

• Introduction
• How did you get involved in the area of data management?
• Can you describe what Dolt is and the story behind it?
• What are the key use cases that you are focused on solving by adding version control to the database layer?
• There are numerous projects related to different aspects of versioning in different data contexts (e.g. LakeFS, Datomic, etc.). What are the versioning semantics that you are focused on?
• You position Dolt as "the database for AI". How does data versioning relate to AI use cases?
• What types of AI systems are able to make best use of Dolt's versioning capabilities?
• Can you describe how Dolt and Doltgres are implemented?
• How have the design and scope of the project changed since you first started working on it?
• What are some of the architecture and integration patterns around relational databases that change when you introduce version control semantics as a core primitive?
• What are some anti-patterns that you have seen teams develop around Dolt's versioning functionality?
• What are the most interesting, innovative, or unexpected ways that you have seen Dolt used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on Dolt?
• When is Dolt the wrong choice?
• What do you have planned for the future of Dolt?

Contact Info
 

• LinkedIn

Parting Question
 

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements
 

• Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.

Links
 

• Dolt
• DoltHub
• Stockmarket Data
• LakeFS
• Datomic
• Git
• MySQL
• Prolly Tree
• Neon
• Django
• Feature Store
• MCP Server
• Nessie
• Iceberg
• PlanetScale
• O(NlogN) Big O Complexity
• B-Tree
• Git Merge
• Git Rebase
• AST == Abstract Syntax Tree
• Supabase
• CockroachDB
• Document Database
• MongoDB
• Gastown
• Beads

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Summary 
In this episode of the Data Engineering Podcast Jamie Knowles, Product Director for ER/Studio, talks about data architecture and its importance in driving business meaning. He discusses how data architecture should start with business meaning, not just physical schemas, and explores the pitfalls of jumping straight to physical designs. Jamie shares his practical definition of data architecture centered on shared semantic models that anchor transactional, analytical, and event-driven systems. The conversation covers strategies for evolving an architecture in tandem with delivery, including defining core concepts, aligning teams through governance, and treating the model as a living product. He also examines how generative AI can both help and harm data architecture, accelerating first drafts but amplifying risk without a human-approved ontology. Jamie emphasizes the importance of doing the hard work upfront to make meaning explicit, keeping models simple and business-aligned, and using tools and patterns to reuse that meaning everywhere. 

Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
• You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
• Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
• Your host is Tobias Macey and today I'm interviewing Jamie Knowles about the impact that a well-developed data architecture (or lack thereof) has on data engineering work

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you start by giving your definition of "data architecture" and what it encompasses?
• How does the nuance change depending on the type of system you are designing? (e.g. data warehouse vs. transactional application database vs. event-driven streaming service)
• In application teams that are large enough there is typically a software architect, but that work often ends up happening organically through trial and error. Who is the responsible party for designing and enforcing a proper data architecture?
• There have been several generational shifts in approach to data warehouse projects in particular. What are some of the anti-patterns that crop up when there is no-one forming a strong opinion on the design/architecture of the warehouse?
• The current stage is largely defined by the ELT pattern. What are some of the ways that workflow can encourage shortcuts?
• Often the need for a proper architecture isn't felt until an organic architecture has developed. What are some of the ways that teams can short-circuit that pain and iterate toward a more sustainable design?
• The common theme in all of the data architecture conversations that I've had is the need for business involvement. There is also a strong push for the business to just want the engineers to deliver data. What are some of the ways that AI utilities can help to accelerate delivery while also capturing business context?
• For teams that are already neck deep in a messy architecture, what are the strategies and tactics that they need to start working toward today to get to a better data architecture?
• What are the most interesting, innovative, or unexpected ways that you have seen teams approach the creation and implementation of their data architecture?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working in data architecture?
• How do you see the introduction of AI at each stage of the data lifecycle changing the ways that teams think about their architectural needs?

Contact Info

• LinkedIn

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

• Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.

Links

• Idera
• ER Studio
• ELT
• RDF == Resource Description Framework
• ORM == Object-Relational Mapping

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Summary 
In this episode Jacob Leverich, cofounder and CTO of Observe, talks about applying lakehouse architectures to observability workloads. Jacob discusses Observe’s decision to leverage cloud-native warehousing and open table formats for scale and cost efficiency. He digs into the core pain points teams face with fragmented tools, soaring costs, and data silos, and how a lakehouse approach - paired with streaming ingest via OpenTelemetry, Kafka-backed durability, curated/columnarized tables, and query orchestration - can deliver low-latency, interactive troubleshooting across logs, metrics, and traces at petabyte scale. He also explore the practicalities of loading and organizing telemetry by use case to reduce read amplification, the role of Iceberg (including v3’s JSON shredding) and Snowflake’s implementation, and why open table formats enable “your data in your lake” strategies. 
Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
• You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
• Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
• Your host is Tobias Macey and today I'm interviewing Jacob Leverich about how data lakehouse technologies can be applied to observability for unlimited scale and orders of magnitude improvement on economics

Interview
 

• Introduction
• How did you get involved in the area of data management?
• Can you start by giving an overview of what the major pain points have been in the observability space? (e.g. limited scale/retention, costs, integration fragmentation)
• What are the elements of the ecosystem and tech stacks that led to that state of the world?
• What are you building at Observe that circumvents those pain points?
• What are the major ecosystem evolutions that make this a feasible architecture? (e.g. columnar storage, distributed compute, protocol consolidation)
• Can you describe the architecture of the Observe platform?
• How have the design of the platform evolved/changed direction since you first started working on it?
• What was your process for determining which core technologies to build on top of?
• What were the missing pieces that you had to engineer around to get a cohesive and performant platform?
• The perennial problem with observability systems and data lakes is their tendency to succumb to entropy. What are the guardrails that you are relying on to help customers maintain a well-structured and usable repository of information?
• Data lakehouses are excellent for flexibility and scaling to massive data volumes, but they're not known for being fast. What are the areas of investment in the ecosystem that is changing that narrative?
• As organizations overcome the constraints of limited retention periods and anxiety over cost, what new use cases does that unlock for their observability data?
• How do AI applications/agents change the requirements around observability data? (collection, scale, complexity, applications, etc.)
• What are the most interesting, innovative, or unexpected ways that you have seen Observe/lakehouse technologies used for observability?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on Observe?
• When is Observe/lakehouse technologies the wrong choice?
• What do you have planned for the future of Observe?

Contact Info
 

• LinkedIn

Parting Question
 

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements
 

• Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.

Links
 

• Observe Inc.
• Lakehouse Architecture
• Splunk
• Observability
• RSyslog
• GlusterFS
• Dremel
• Drill
• BigQuery
• Snowflake SIGMOD Paper
• Prometheus
• Datadog
• NewRelic
• AppDynamics
• DynaTrace
• Loki
• Cortex
• Mimir
• Tempo
• Cardinality
• FluentBit
• FluentD
• OpenTelemetry
• OTLP == OpenTelemetry Line Protocol
• Kafka
• VPC Flow Logs
• Read Amplification
• Lance
• Iceberg
• Hudi
• PromQL

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Summary 
In this episode Kostas Pardalis talks about Fenic - an open-source, PySpark-inspired dataframe engine designed to bring LLM-powered semantics into reliable data engineering workflows. Kostas shares why today’s data infrastructure assumptions (BI-first, expert-operated, CPU-bound) fall short for AI-era tasks that are increasingly inference- and IO-bound. He explores how Fenic introduces semantic operators (e.g., semantic filter, extract, join) as first-class citizens in the logical plan so the optimizer can reason about inference, costs, and constraints. This enables developers to turn unstructured data into explicit schemas, compose transformations lazily, and offload LLM work safely and efficiently. He digs into Fenic’s architecture (lazy dataframe API, logical/physical plans, Polars execution, DuckDB/Arrow SQL path), how it exposes tools via MCP for agent integration, and where it fits in context engineering as a companion for memory/state management in agentic systems. 


Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
• Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
• If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
• Your host is Tobias Macey and today I'm interviewing Kostas Pardalis about Fenic, an opinionated, PySpark-inspired DataFrame framework for building AI and agentic applications

Interview
 

• Introduction
• How did you get involved in the area of data management?
• Can you describe what Fenic is and the story behind it?
• What are the core problems that you are trying to address with Fenic?
• Dataframes have become a popular interface for doing chained transformations on structured data. What are the benefits of using that paradigm for LLM use-cases?
• Can you describe the architecture and implementation of Fenic?
• How have the design and scope of the project changed since you first started working on it?
• You position Fenic as a means of bringing reliability to LLM-powered transformations. What are some of the anti-patterns that teams should be aware of when getting started with Fenic?
• What are some of the most common first steps that teams take when integrating Fenic into their pipelines or applications?
• What are some of the ways that teams should be thinking about using Fenic and semantic operations for data pipelines and transformations?
• How does Fenic help with context engineering for agentic use cases?
• What are some examples of toolchains/workflows that could be replaced with Fenic?
• How does Fenic integrate with the broader ecosystem of data and AI frameworks? (e.g. Polars, Arrow, Qdrant, LangChan/Pydantic AI)
• What are the most interesting, innovative, or unexpected ways that you have seen Fenic used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on Fenic?
• When is Fenic the wrong choice?
• What do you have planned for the future of Fenic?

Contact Info
 

• LinkedIn

Parting Question
 

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements
 

• Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.

Links
 

• Fenic
• RudderStack
• Podcast Episode
• Trino
• Starburst
• Trino Project Tardigrade
• Typedef AI
• dbt
• PySpark
• UDF == User-Defined Function
• LOTUS
• Pandas
• Polars
• Relational Algebra
• Arrow
• DuckDB
• Markdown
• Pydantic AI
• AI Engineering Podcast Episode
• LangChain
• Ray
• Dask

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Summary 
In this episode Goutham Budati about his Data–Perspective–Action framework and how it empowers data teams to become true business partners. Gautham traces his path from automating Excel reports to leading high‑impact data organizations, then breaks down why technical excellence alone isn’t enough: teams must pair reliable data systems with deliberate storytelling, clear problem framing, and concrete action plans. He digs into tactics for moving from reactive ticket-taking to proactive influence — weekly one‑page narratives, design-first discovery, sampling stakeholders for real pain points, and treating dashboards as living roadmaps. He also explores how to right-size technical scope, preserve trust in core metrics, organize teams as “build” and “storytelling” duos, and translate business macros and micros into resilient system designs. 

Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
• You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
• Your host is Tobias Macey and today I'm interviewing Goutham Budati about his data-perspective-action framework for empowering data teams to be more influential in the business

Interview
 

• Introduction
• How did you get involved in the area of data management?
• Can you describe what the Data-Perspective-Action framework is and the story behind it?
• What does it look like when someone operates at each of those three levels?
  • How does that change the day-to-day work of an individual contributor?
• Why does technically excellent data work sometimes fail to drive decisions?
  • How do you identify whether a data system or pipeline is actually creating value versus just existing?
• What's the moment when you realized that building reliable systems wasn't the same as enabling better decisions?
  • Better decisions still need to be powered by reliable systems. How do you manage the tension of focusing on up-time against focusing on impact?
• What does it mean to add "Perspective" to data? How is that different from analysis or insights?
• How do you know when you're overwhelming stakeholders versus giving them what they need?
• What changes when you start designing systems to surface signal rather than just providing comprehensive data?
• How do you learn what business context matters for turning data into something actionable?
• What does it mean to design for Action from day one? How does that change what you build?
  • How do you get stakeholders to actually act on data instead of just consuming it?
• Walk us through how you structure collaboration with business partners when you're trying to drive decisions, not just inform them.
• What's the relationship between iteration and trust when you're building data products?
• What does the transition from order-taker to strategic partner actually look like? What has to change?
  • How do you position data work as driving the business rather than supporting it?
• Why does storytelling matter for data professionals? What role does it play that technical communication doesn't cover?
• What organizational structures or team setups help data people gain influence?
• Tell us about a time when you built something technically sound that failed to create impact. What did you learn?
• What are the common patterns in dysfunctional data organizations? What causes the breakdown?
• How do you rebuild credibility when you inherit a data function that's lost trust with the business?
  • What's the relationship between technical excellence and stakeholder trust? Can you have one without the other?
• When is this framework the wrong lens? What situations call for a different approach?
• How do you balance the demand for technical depth with the need to develop business and communication skills?
• How should data professionals position themselves as AI and ML tools become more accessible?
• What shifts do you see coming in how businesses think about data work?
  • How is your thinking about data impact evolving?
• For someone who recognizes they're focused purely on the technical work and wants to expand their impact—where should they start?

Contact Info
 

• LinkedIn

Parting Question
 

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements
 

• Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
 

Summary 
In this episode PhD researcher Xinyu Zeng talks about F3, the “future-proof file format” designed to address today’s hardware realities and evolving workloads. He digs into the limitations of Parquet and ORC - especially CPU-bound decoding, metadata overhead for wide-table projections, and poor random-access behavior for ML training and serving - and how F3 rethinks layout and encodings to be efficient, interoperable, and extensible. Xinyu explains F3’s two major ideas: a decoupled, flexible layout that separates IO units, dictionary scope, and encoding choices; and self-decoding files that embed WebAssembly kernels so new encodings can be adopted without waiting on every engine to upgrade. He discusses how table formats and file formats should increasingly be decoupled, potential synergies between F3 and table layers (including centralizing and verifying WASM kernels), and future directions such as extending WASM beyond encodings to indexing or filtering. 

Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
• Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
• Your host is Tobias Macey and today I'm interviewing Xinyu Zeng about the future-proof file format

Interview
 

• Introduction
• How did you get involved in the area of data management?
• Can you describe what the F3 project is and the story behind it?
• We have several widely adopted file formats (Parquet, ORC, Avro, etc.). Why do we keep creating new ones?
• Parquet is the format with perhaps the broadest adoption. What are the challenges that such wide use poses when trying to modify or extend the specification?
• The recent focus on vector data is perhaps the most visible change in storage requirements. What are some of the other custom types of data that might need to be supported in the file storage layer?
• Can you describe the key design principles of the F3 format?
• What are the engineering challenges that you faced while developing your implementation of the F3 proof-of-concept?
• The key challenge of introducing a new format is that of adoption. What are the provisions in F3 that might simplify the adoption of the format in the broader ecosystem? (e.g. integration with compute frameworks)
• What are some examples of features in data lake use cases that could be enabled by F3?
• What are some of the other ideas/hypotheses that you developed and discarded in the process of your reasearch?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on F3?
• What do you have planned for the future of F3?

Contact Info
 

• Personal Website

Parting Question
 

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links
 

• F3 Paper
• Formats Evaluation Paper
• F3 Github
• SAL Paper
• RisingWave
• Tencent Cloud
• Parquet
• Arrow
• Andy Pavlo
• Wes McKinney
• CMU Public Seminar
• VLDB
• ORC
• Protocol Buffers
• Lance
• PAX == Partition Attributes Across
• WASM == Web Assembly
• DataFusion
• DuckDB
• DuckLake
• Velox
• Vortex File Format

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
 

Summary 
In this episode Suresh Srinivas and Sriharsha Chintalapani explore how metadata platforms are evolving from human-centric catalogs into the foundational context layer for AI and agentic systems. They discuss the origins and growth of OpenMetadata and Collate, why “context” is necessary but “semantics” is critical for precise AI outcomes, and how a schema-first, API-first, unified platform enables discovery, observability, and governance in one workflow. They share how AI agents can now automate documentation, classification, data quality testing, and enforcement of policies, and why aligning governance with user identity and intent is essential as agentic access scales. They also dig into scalability strategies, MCP-based agent workflows, AI governance (including model/agent tracking), and the emerging convergence of big data with ontologies to deliver machine-understandable meaning. 

Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Data teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.
• Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
• Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
• You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
• Your host is Tobias Macey and today I'm interviewing Suresh Srinivas and Sriharsha Chintalapani about how metadata catalogs provide the context clues necessary to give meaning to your data for AI systems

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you start by giving an overview of the roles that metadata catalogs are playing in the current state of the ecosystem?
• How has the OpenMetadata platform evolved over the past 4 years?
  • How has the focus on LLMs/generative AI changed the trajectory of services like OpenMetadata?
• The initial set of use cases for data catalogs was to facilitate discovery and documentation of data assets for human consumption. What are the structural elements of that effort that have paid dividends for an AI audience?
  • How does the AI audience change the requirements around the cataloging and presentation of metadata?
• One of the constant challenges in data infrastructure now is the tension of making data accessible to AI systems (agentic or otherwise) and incorporating AI into the inner loop of the service. What are the opportunities for bringing AI inside the boundaries of a system like OpenMetadata vs. as a client or consumer of the platform?
• The key phrase of the past ~2 years is "context engineering". What role does the metadata catalog play in that undertaking?
  • What are the capabilities that the catalog needs to be able to effectively populate and curate that context?
  • How much awareness does the LLM or agent need to have to be able to use the catalog effectively?
• What does a typical workflow/agent loop look like when it is using something like OpenMetadata in pursuit of knowledge that it needs to achieve an objective?
• How do agentic use cases strain the existing set of governance frameworks?
  • What new considerations (procedural or technical) need to be factored into governance practices to balance velocity with security?
• What are the most interesting, innovative, or unexpected ways that you have seen OpenMetadata/Collate used in AI/agentic contexts?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on OpenMetadata/Collate?
• When is OpenMetadata/Collate the wrong choice?
• What do you have planned for the future of OpenMetadata?

Contact Info

• Suresh
  • LinkedIn
• Sriharsha
  • LinkedIn

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

• OpenMetadata 
  • Podcast Episode
• Hadoop
• Hortonworks
• Context Engineering
• MCP == Model Context Protocol
• JSON Schema
• dbt
• LangSmith
• OpenMetadata MCP Server
• API Gateway

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
 

Summary 
In this solo episode of the Data Engineering Podcast, host Tobias Macey reflects on how AI has transformed the practice and pace of data engineering over time. Starting from its origins in the Hadoop and cloud warehouse era, he explores the discipline's evolution through ML engineering and MLOps to today's blended boundaries between data, ML, and AI engineering. The conversation covers how unstructured data is becoming more prominent, vectors and knowledge graphs are emerging as key components, and reliability expectations are changing due to interactive user-facing AI. The host also delves into process changes, including tighter collaboration, faster dataset onboarding, new governance and access controls, and the importance of treating experimentation and evaluation as fundamental testing practices. 

Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Data teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.
• Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
• Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details. 
• You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
• Your host is Tobias Macey and today I'm interviewing reflecting about the increasingly blurry boundaries between data engineering and AI engineering

Interview

• Introduction
• I started this podcast in 2017, right when the term "Data Engineer" was becoming widely used for a specific job title with a reasonably well-understood set of responsibilities. This was in response to the massive hype around "data science" and consequent hiring sprees that characterized the mid-2000s to mid-2010s. The introduction of generative AI and AI Engineering to the technical ecosystem is changing the scope of responsibilities for data engineers and other data practitioners. Of note is the fact that:
• AI models can be used to process unstructured data sources into structured data assets
• AI applications require new types of data assets
• The SLAs for data assets related to AI serving are different from BI/warehouse use cases
• The technology stacks for AI applications aren't necessarily the same as for analytical data pipelines
• Because everything is so new there is not a lot of prior art, and the prior art that does exist isn't necessarily easy to find because of differences in terminology
• Experimentation has moved from being just an MLOps capability into being a core need for organizations

Contact Info

• Email

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

• AI Engineering Podcast

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
 

Summary 
In this episode Michael Toy, co-creator of Malloy, talks about rethinking how we work with data beyond SQL. Michael shares the origins of Malloy from his and Lloyd Tabb’s experience at Looker, why SQL’s mental model often fights human problem solving, and how Malloy aims to be a composable, maintainable language that treats SQL as the assembly layer rather than something humans should write. He explores Malloy’s core ideas — semantic modeling tightly coupled with a query language, hierarchical data as the default mental model, and preserving context so analysis stays interactive and open-ended. He also digs into the developer experience and ecosystem: Malloy’s TypeScript implementation, VS Code integration, CLI, emerging notebook support, and how Malloy can sit alongside or replace parts of existing transformation workflows. Michael discusses practical trade-offs in language design, the surprising fit for LLM-generated queries, and near-term roadmap areas like dimensional filtering, better aggregation strategies across levels, and closing gaps that still require escaping to SQL. He closes with an invitation to contribute to the open-source project and help shape its evolution. 

Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Data teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.
• Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details. 
• Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
• You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
• Your host is Tobias Macey and today I'm interviewing Michael Toy about Malloy, a modern language for building composable and maintainable analytics and data models on relational engines

Interview
 

• Introduction
• How did you get involved in the area of data management?
• Can you describe what Malloy is and the story behind it?
  • What is the core problem that you are trying to solve with Malloy?
• There are countless projects that aim to reimagine/reinvent/replace SQL. What are the factors that make Malloy stand out in your mind?
• Who are the target personas for the Malloy language?
• One of the key success factors for any language is the ecosystem around it and the integrations available to it. How does Malloy fit in the toolchains and workflows for data engineers and analysts?
• Can you describe the key design and syntax elements of Malloy?
  • How have the scope and focus of the language evolved since you first started working on it?
• How do the structure and semantics of Malloy change the ways that teams think about their data models?
• SQL-focused tools have gained prominence as the means of building the tranfromation stage of data pipelines. How would you characterize the capabilities of Malloy as a tool for building translation pipelines?
• What are the most interesting, innovative, or unexpected ways that you have seen Malloy used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on Malloy?
• When is Malloy the wrong choice?
• What do you have planned for the future of Malloy?

Contact Info

• Website

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

• Malloy
• Lloyd Tabb
• SQL
• Looker
• LookML
• dbt
• Relational Algebra
• Typescript
• Ruby
• [Truffle](
• Malloy VSCode Plugin
• Malloy CLI
• Malloy Pick Statement

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
 

Summary
In this crossover episode, Max Beauchemin explores how multiplayer, multi‑agent engineering is transforming the way individuals and teams build data and AI systems. He digs into the shifting boundary between data and AI engineering, the rise of “context as code,” and how just‑in‑time retrieval via MCP and CLIs lets agents gather what they need without bloating context windows. Max shares hard‑won practices from going “AI‑first” for most tasks, where humans focus on orchestration and taste, and the new bottlenecks that appear — code review, QA, async coordination — when execution accelerates 2–10x. He also dives deep into Agor, his open‑source agent orchestration platform: a spatial, multiplayer workspace that manages Git worktrees and live dev environments, templatizes prompts by workflow zones, supports session forking and sub‑sessions, and exposes an internal MCP so agents can schedule, monitor, and even coordinate other agents.

Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Data teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.
• Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
• Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
• Your host is Tobias Macey and today I'm interviewing Maxime Beauchemin about the impact of multi-player multi-agent engineering on individual and team velocity for building better data systems

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you start by giving an overview of the types of work that you are relying on AI development agents for?
• As you bring agents into the mix for software engineering, what are the bottlenecks that start to show up?
• In my own experience there are a finite number of agents that I can manage in parallel. How does Agor help to increase that limit?
• How does making multi-agent management a multi-player experience change the dynamics of how you apply agentic engineering workflows?

Contact Info

• LinkedIn

Links

• Agor
• Apache Airflow
• Apache Superset
• Preset
• Claude Code
• Codex
• Playwright MCP
• Tmux
• Git Worktrees
• Opencode.ai
• GitHub Codespaces
• Ona

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA