August 05, 2019

Solving Data Discovery At Lyft

51 minutes

Summary

Data is only valuable if you use it for something, and the first step is knowing that it is available. As organizations grow and data sources proliferate it becomes difficult to keep track of everything, particularly for analysts and data scientists who are not involved with the collection and management of that information. Lyft has build the Amundsen platform to address the problem of data discovery and in this episode Tao Feng and Mark Grover explain how it works, why they built it, and how it has impacted the workflow of data professionals in their organization. If you are struggling to realize the value of your information because you don’t know what you have or where it is then give this a listen and then try out Amundsen for yourself.

Announcements

Welcome to the Data Engineering Podcast, the show about modern data management

When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!

Finding the data that you need is tricky, and Amundsen will help you solve that problem. And as your data grows in volume and complexity, there are foundational principles that you can follow to keep data workflows streamlined. Mode – the advanced analytics platform that Lyft trusts – has compiled 3 reasons to rethink data discovery. Read them at dataengineeringpodcast.com/mode-lyft.

You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, the Open Data Science Conference, and Corinium Intelligence. Upcoming events include the O’Reilly AI Conference, the Strata Data Conference, and the combined events of the Data Architecture Summit and Graphorum. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.

Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.

To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat

Your host is Tobias Macey and today I’m interviewing Mark Grover and Tao Feng about Amundsen, the data discovery platform and metadata engine that powers self service data access at Lyft

Interview

Introduction

How did you get involved in the area of data management?

Can you start by explaining what Amundsen is and the problems that it was designed to address?

What was lacking in the existing projects at the time that led you to building a new platform from the ground up?

How does Amundsen fit in the larger ecosystem of data tools?

How does it compare to what WeWork is building with Marquez?

Can you describe the overall architecture of Amundsen and how it has evolved since you began working on it?

What were the main assumptions that you had going into this project and how have they been challenged or updated in the process of building and using it?

What has been the impact of Amundsen on the workflows of data teams at Lyft?

Can you talk through an example workflow for someone using Amundsen?

Once a dataset has been located, how does Amundsen simplify the process of accessing that data for analysis or further processing?

How does the information in Amundsen get populated and what is the process for keeping it up to date?

What was your motivation for releasing it as open source and how much effort was involved in cleaning up the code for the public?

What are some of the capabilities that you have intentionally decided not to implement yet?

For someone who wants to run their own instance of Amundsen what is involved in getting it deployed and integrated?

What have you found to be the most challenging aspects of building, using and maintaining Amundsen?

What do you have planned for the future of Amundsen?

Contact Info

Tao

feng-tao on GitHub

Mark

Website

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

Amundsen

Data Council Presentation

Strata Presentation

Blog Post

Lyft

Airflow

Podcast.__init__ Episode

Slack

Marquez

Hive

Presto

Podcast Episode

Spark

PostgreSQL

Google BigQuery

Neo4J

Apache Atlas

Tableau

Superset

Alation

Cloudera Navigator

DynamoDB

MongoDB

Druid

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Support Data Engineering Podcast

...more

View all episodes

By Tobias Macey

4.6

135135 ratings