November 22, 2017

Data Serialization Formats with Doug Cutting and Julien Le Dem - Episode 8

51 minutes

Summary

With the wealth of formats for sending and storing data it can be difficult to determine which one to use. In this episode Doug Cutting, creator of Avro, and Julien Le Dem, creator of Parquet, dig into the different classes of serialization formats, what their strengths are, and how to choose one for your workload. They also discuss the role of Arrow as a mechanism for in-memory data sharing and how hardware evolution will influence the state of the art for data formats.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure

When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at dataengineeringpodcast.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.

Continuous delivery lets you get new features in front of your users as fast as possible without introducing bugs or breaking production and GoCD is the open source platform made by the people at Thoughtworks who wrote the book about it. Go to dataengineeringpodcast.com/gocd to download and launch it today. Enterprise add-ons and professional support are available for added peace of mind.

Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.

You can help support the show by checking out the Patreon page which is linked from the site.

To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers

This is your host Tobias Macey and today I’m interviewing Julien Le Dem and Doug Cutting about data serialization formats and how to pick the right one for your systems.

Interview

Introduction

How did you first get involved in the area of data management?

What are the main serialization formats used for data storage and analysis?

What are the tradeoffs that are offered by the different formats?

How have the different storage and analysis tools influenced the types of storage formats that are available?

You’ve each developed a new on-disk data format, Avro and Parquet respectively. What were your motivations for investing that time and effort?

Why is it important for data engineers to carefully consider the format in which they transfer their data between systems?

What are the switching costs involved in moving from one format to another after you have started using it in a production system?

What are some of the new or upcoming formats that you are each excited about?

How do you anticipate the evolving hardware, patterns, and tools for processing data to influence the types of storage formats that maintain or grow their popularity?

Contact Information

Doug:

cutting on GitHub

Blog

@cutting on Twitter

Julien

@J_ on Twitter

Blog

julienledem on GitHub

Links

Apache Avro

Apache Parquet

Apache Arrow

Hadoop

Apache Pig

Xerox Parc

Excite

Nutch

Vertica

Dremel White Paper

Twitter Blog on Release of Parquet

CSV

XML

Hive

Impala

Presto

Spark SQL

Brotli

ZStandard

Apache Drill

Trevni

Apache Calcite

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Support Data Engineering Podcast

...more

View all episodes

By Tobias Macey

4.5

142142 ratings