For forty years, the rule held that transactional and analytical databases had to be separate systems, connected by fragile pipelines that move data from one to the other. That assumption is now being questioned. As AI agents start generating the majority of database activity, the old architecture is being redesigned around speed, scale, and a single copy of governed data. For anyone who works with data day to day, this raises practical questions. Do you still need separate systems for live and historical data? What happens to the pipelines you maintain? And how does your stack change when agents, not people, write most of the queries?

Reynold Xin is co-founder and Chief Architect of Databricks. He is one of the original creators of Apache Spark, where he led the design of GraphX, Project Tungsten, and Structured Streaming, co-designed DataFrames, and served as release manager for Spark 2.0. He holds a PhD in Computer Science from UC Berkeley's AMPLab and a degree in Engineering Science from the University of Toronto.

In the episode, Richie and Reynold explore self-service analytics with Genie, the ontology layer that grounds AI in enterprise data, handling hallucinations, governance and permissions for AI agents, merging transactional and analytical databases with Lakebase and LTAP, real-time analytics, controlling cost through autoscaling, the future of Spark and classic machine learning, and much more.

Links Mentioned in the Show:

• Connect with Reynold: https://www.linkedin.com/in/rxin

• Genie (Databricks data agent): https://www.databricks.com/product/genie

• Genie Ontology / Genie One: https://www.databricks.com/blog/introducing-genie-one-genie-ontology-and-genie-agents

• LTAP (Lake Transactional/Analytical Processing): https://www.databricks.com/company/newsroom/press-releases/databricks-launches-ltap-first-lake-transactionalanalytical

• Lakehouse//RT: https://www.databricks.com/blog/introducing-lakehousert-real-time-performance-unified-lakehouse

• Lakebase: https://www.databricks.com/product/lakebase

• Apache Spark: https://spark.apache.org

• AI-Native Course: Intro to AI for Work - https://www.datacamp.com/courses/introduction-to-ai-for-work

• Related Episode: AI's Impact on Databases - https://www.datacamp.com/podcast/ais-impact-on-databases

New to DataCamp? Learn on the go using the DataCamp mobile app - https://www.datacamp.com/mobile

Empower your business with world-class data and AI skills with DataCamp for business - https://www.datacamp.com/business