Dan Delorey helped to build the core technologies of Google’s cloud data services for many years before embarking on his latest adventure as the VP of Data at SoFi. From being an early engineer on the Dremel project, to helping launch and manage BigQuery, on to helping enterprises adopt Google’s data products he learned all of the critical details of how to run services used by data platform teams. Now he is the consumer of many of the tools that his work inspired. In this episode he takes a trip down memory lane to weave an interesting and informative narrative about the broader themes throughout his work and their echoes in the modern data ecosystem.

Introduction

How did you get involved in the area of data management?

Can you start by sharing what your current relationship to the data ecosystem is and the cliffs-notes version of how you ended up there?

Dremel was a ground-breaking technology at the time. What do you see as its lasting impression on the landscape of data both in and outside of Google?

You were instrumental in crafting the vision behind "querying data in place," (what they called, federated data) at Dremel and BigQuery. What do you mean by this? How has this approach evolved? What are some challenges with this approach?

Following your work on Drill you were involved with the development and growth of BigQuery and the broader suite of Google Cloud’s data platform. What do you see as the influence that those tools had on the evolution of the broader data ecosystem?

How have your experiences at Google influenced your approach to platform and organizational design at SoFi?

What’s in SoFi’s data stack? How do you decide what technologies to buy vs. build in-house?

How does your team solve for data quality and governance?

When you’re not building industry-defining data tooling or leading data strategy, you spend time thinking about the ethics of data. Can you elaborate a bit about your research and interest there?

You also have some ideas about data marketplaces, which is a hot topic these days with companies like Snowflake and Databricks breaking into this economy. What’s your take on the evolution of this space?

What are the most interesting, innovative, or unexpected data systems that you have encountered?

What are the most interesting, unexpected, or challenging lessons that you have learned while working on building and supporting data systems?

What are the areas that you are paying the most attention to?

What interesting predictions do you have for the future of data systems and their applications?

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Support Data Engineering Podcast