Data for AI

Nike had 450 different reds. University Red, Jordan Red, and 448 more. Query for "red shoes" and you either hit a graph database of colors first or you scan every string value in the column. Scott Haines spent years inside that problem, and it is why he cares about catalogs, tags, and semantics more than most people who talk about them.

Part 2 picks up where Part 1 left off. Scott is in developer relations at Databricks, working on Delta Lake, Iceberg, and Unity Catalog, and nine months into Rust. He is the author of Modern Data Engineering with Apache Spark and a co-author of Delta Lake: The Definitive Guide.

We talk about what a catalog actually is (where data lives and who owns it, which matters more once everybody's agents are querying too), where semantics go when "revenue" means three things in one company, metric views landing in Spark 4.2 and Unity Catalog 0.6, and which layers of the stack have to stay open: catalogs, table formats, and increasingly governance itself. Then the part Scott is most blunt about: the notebook that ends in DROP TABLE, the HDFS cluster at Twilio with no undo, dry-run-by-default as a design rule, the 60-page governance doc nobody read at Nike, and why he reached for the Cedar policy language there and is working with it again now. It ends on the open question: when an agent is a proxy for a lot of processes, whose permissions does it inherit?

Chapters

00:00 What Scott is working on now: Spark, Iceberg, Delta, DataFusion, Rust

00:54 Nine months into Rust, and hiring "a good engineer"

02:21 Software primitives are finally commodities

03:34 What a catalog is: tribal knowledge before, discovery and ownership after

05:08 Everybody's agents: certified data and the source of truth

06:23 From infrastructure abstractions to semantics

07:00 450 reds at Nike: ontologies, a graph of colors, tags vs scanning columns

09:00 Metric views in Spark 4.2, and SQL snippets passed around in Slack

10:37 Unity Catalog 0.6 metric views and governance dictionaries

12:56 Trust but verify, minus the verify

13:10 What has to be open: catalogs, Iceberg REST, table formats, compute

14:32 Apache Gravitino and the lock-in problem

14:44 Open governance: exchanging policies, tags, and column masks

16:16 Why governance: the notebook that drops a table in prod

17:04 HDFS with no undo, and dry run as the default

19:00 Service principals, Vault, and config-based production at Nike

20:00 Sixty pages nobody reads, and the Cedar policy language

21:00 Agents as proxies: whose permissions do they inherit?

Part 1: https://youtu.be/4nWeC_NEYmw

Scott on LinkedIn: https://www.linkedin.com/in/scotthaines/

Delta Lake: The Definitive Guide: https://www.oreilly.com/library/view/delta-lake-the/9781098151935/

Modern Data Engineering with Apache Spark: https://link.springer.com/book/10.1007/978-1-4842-7452-1

Data for AI is produced by Datastrato, the original creator of Apache Gravitino. Sponsored by Neo4j.

Join the community: https://datastrato.ai/discord

Next meetup: https://lu.ma/dataforai

Watch on YouTube: https://www.youtube.com/@data-for-ai

Scott Haines spent 25 years building real-time systems at Yahoo, Twilio, and Nike. Seven months ago he stopped, and started teaching instead. This is a conversation about what that trade costs.

Scott is now in developer relations at Databricks, working on Delta Lake, Iceberg, and Unity Catalog. He is the author of Modern Data Engineering with Apache Spark and a co-author of Delta Lake: The Definitive Guide. Before that he was the engineer on the pager: Apache Storm at Yahoo handling 46 million events a second in 2014, then the first streaming team at Twilio on Spark 1.6.

We talk about the parts of the transition nobody puts in the job description: measuring your own work when nothing merges, imposter syndrome in a front-facing role, and what it means to build community when the community is increasingly prompting its way through problems. Scott is blunt about the AI part. He spent last winter prompting his way through everything, ended up with what he calls a mountain of dead projects, and found he had to relearn how to start an application from scratch. His answer is to spend deliberate time with no AI at all, and to keep writing books and code by hand, not because he has to, but because the flow state is where the understanding lives.

Then the origin story: Hacker Dojo in Mountain View, teaching PHP and Rails as a sole proprietor, pizza at the Hadoop users group at Yahoo, why Storm felt like caffeine, and why Spark at Twilio "was like crack."

Chapters

00:00 Welcome

00:32 Seven months into DevRel: how do you know if you're doing it well?

01:30 Graphic design, Hacker Dojo, and teaching before the pager

03:27 Building a reputation twice

05:30 Imposter syndrome when you're no longer on call

06:56 Advice for anyone whose job AI is doing now

09:01 Time without AI, and the mountain of dead projects

11:18 Who are you writing for? Books, blogs, and the AI slop sniff test

15:26 Flow state as the place deep work happens

17:08 Delta Lake: The Definitive Guide, and making Parquet not suck at Twilio

19:45 Yahoo Games, the Hadoop users group, and discovering analytics

21:25 Apache Storm: 46 million events a second

22:00 Twilio, Spark 1.6, and three months to build a streaming team

23:37 Why streaming was addictive: Redis, sessionization, and replaying real time

Scott on LinkedIn: https://www.linkedin.com/in/scotthaines/

Delta Lake: The Definitive Guide (O'Reilly)

Modern Data Engineering with Apache Spark (Apress)

Data for AI is produced by Datastrato, the original creator of Apache Gravitino. Sponsored by Neo4j.

Join the community: https://datastrato.ai/discord

Next meetup: https://lu.ma/dataforai

Watch on YouTube: https://youtube.com/@data-for-ai