Data Engineering Podcast

By Tobias MaceyTechnology Education

Favorites
137
Followers
Typical duration
56 min
per episode

Based on Podcast App listening data

Data Engineering Podcast episodes

November 14, 2017Buzzfeed Data Infrastructure with Walter Menendez - Episode 7
Summary
Buzzfeed needs to be able to understand how its users are interacting with the myriad articles, videos, etc. that they are posting. This lets them produce new content that will continue to be well-received. To surface the insights that they need to grow their business they need a robust data infrastructure to reliably capture all of those interactions. Walter Menendez is a data engineer on their infrastructure team and in this episode he describes how they manage data ingestion from a wide array of sources and create an interface for their data scientists to produce valuable conclusions.
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at dataengineeringpodcast.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.
Continuous delivery lets you get new features in front of your users as fast as possible without introducing bugs or breaking production and GoCD is the open source platform made by the people at Thoughtworks who wrote the book about it. Go to dataengineeringpodcast.com/gocd to download and launch it today. Enterprise add-ons and professional support are available for added peace of mind.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
You can help support the show by checking out the Patreon page which is linked from the site.
To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
Your host is Tobias Macey and today I’m interviewing Walter Menendez about the data engineering platform at Buzzfeed
Interview
Introduction
How did you get involved in the area of data management?
How is the data engineering team at Buzzfeed structured and what kinds of projects are you responsible for?
What are some of the types of data inputs and outputs that you work with at Buzzfeed?
Is the core of your system using a real-time streaming approach or is it primarily batch-oriented and what are the business needs that drive that decision?
What does the architecture of your data platform look like and what are some of the most significant areas of technical debt?
Which platforms and languages are most widely leveraged in your team and what are some of the outliers?
What are some of the most significant challenges that you face, both technically and organizationally?
What are some of the dead ends that you have run into or failed projects that you have tried?
What has been the most successful project that you have completed and how do you measure that success?
Contact Info
@hackwalter on Twitter
walterm on GitHub
Links
Data Literacy
MIT Media Lab
Tumblr
Data Capital
Data Infrastructure
Google Analytics
Datadog
Python
Numpy
SciPy
NLTK
Go Language
NSQ
Tornado
PySpark
AWS EMR
Redshift
Tracking Pixel
Google Cloud
Don’t try to be google
Stop Hiring DevOps Engineers and Start Growing Them
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
44 min
August 06, 2017Astronomer with Ry Walker - Episode 6
Summary
Building a data pipeline that is reliable and flexible is a difficult task, especially when you have a small team. Astronomer is a platform that lets you skip straight to processing your valuable business data. Ry Walker, the CEO of Astronomer, explains how the company got started, how the platform works, and their commitment to open source.
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at www.dataengineeringpodcast.com/linode?utm_source=rss&utm_medium=rss and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
You can help support the show by checking out the Patreon page which is linked from the site.
To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
This is your host Tobias Macey and today I’m interviewing Ry Walker, CEO of Astronomer, the platform for data engineering.
Interview
Introduction
How did you first get involved in the area of data management?
What is Astronomer and how did it get started?
Regulatory challenges of processing other people’s data
What does your data pipelining architecture look like?
What are the most challenging aspects of building a general purpose data management environment?
What are some of the most significant sources of technical debt in your platform?
Can you share some of the failures that you have encountered while architecting or building your platform and company and how you overcame them?
There are certain areas of the overall data engineering workflow that are well defined and have numerous tools to choose from. What are some of the unsolved problems in data management?
What are some of the most interesting or unexpected uses of your platform that you are aware of?
Contact Information
Email
@rywalker on Twitter
Links
Astronomer
Kiss Metrics
Segment
Marketing tools chart
Clickstream
HIPAA
FERPA
PCI
Mesos
Mesos DC/OS
Airflow
SSIS
Marathon
Prometheus
Grafana
Terraform
Kafka
Spark
ELK Stack
React
GraphQL
PostGreSQL
MongoDB
Ceph
Druid
Aries
Vault
Adapter Pattern
Docker
Kinesis
API Gateway
Kong
AWS Lambda
Flink
Redshift
NOAA
Informatica
SnapLogic
Meteor
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
43 min
June 18, 2017Rebuilding Yelp's Data Pipeline with Justin Cunningham - Episode 5
Summary
Yelp needs to be able to consume and process all of the user interactions that happen in their platform in as close to real-time as possible. To achieve that goal they embarked on a journey to refactor their monolithic architecture to be more modular and modern, and then they open sourced it! In this episode Justin Cunningham joins me to discuss the decisions they made and the lessons they learned in the process, including what worked, what didn’t, and what he would do differently if he was starting over today.
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at www.dataengineeringpodcast.com/linode?utm_source=rss&utm_medium=rss and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
You can help support the show by checking out the Patreon page which is linked from the site.
To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
Your host is Tobias Macey and today I’m interviewing Justin Cunningham about Yelp’s data pipeline
Interview with Justin Cunningham
Introduction
How did you get involved in the area of data engineering?
Can you start by giving an overview of your pipeline and the type of workload that you are optimizing for?
What are some of the dead ends that you experienced while designing and implementing your pipeline?
As you were picking the components for your pipeline, how did you prioritize the build vs buy decisions and what are the pieces that you ended up building in-house?
What are some of the failure modes that you have experienced in the various parts of your pipeline and how have you engineered around them?
What are you using to automate deployment and maintenance of your various components and how do you monitor them for availability and accuracy?
While you were re-architecting your monolithic application into a service oriented architecture and defining the flows of data, how were you able to make the switch while verifying that you were not introducing unintended mutations into the data being produced?
Did you plan to open-source the work that you were doing from the start, or was that decision made after the project was completed? What were some of the challenges associated with making sure that it was properly structured to be amenable to making it public?
What advice would you give to anyone who is starting a brand new project and how would that advice differ for someone who is trying to retrofit a data management architecture onto an existing project?
Keep in touch
Yelp Engineering Blog
Email
Links
Kafka
Redshift
ETL
Business Intelligence
Change Data Capture
LinkedIn Data Bus
Apache Storm
Apache Flink
Confluent
Apache Avro
Game Days
Chaos Monkey
Simian Army
PaaSta
Apache Mesos
Marathon
SignalFX
Sensu
Thrift
Protocol Buffers
JSON Schema
Debezium
Kafka Connect
Apache Beam
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
43 min
March 18, 2017ScyllaDB with Eyal Gutkind - Episode 4
Summary
If you like the features of Cassandra DB but wish it ran faster with fewer resources then ScyllaDB is the answer you have been looking for. In this episode Eyal Gutkind explains how Scylla was created and how it differentiates itself in the crowded database market.
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
You can help support the show by checking out the Patreon page which is linked from the site.
To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
Your host is Tobias Macey and today I’m interviewing Eyal Gutkind about ScyllaDB
Interview
Introduction
How did you get involved in the area of data management?
What is ScyllaDB and why would someone choose to use it?
How do you ensure sufficient reliability and accuracy of the database engine?
The large draw of Scylla is that it is a drop in replacement of Cassandra with faster performance and no requirement to manage th JVM. What are some of the technical and architectural design choices that have enabled you to do that?
Deployment and tuning
What challenges are inroduced as a result of needing to maintain API compatibility with a diferent product?
Do you have visibility or advance knowledge of what new interfaces are being added to the Apache Cassandra project, or are you forced to play a game of keep up?
Are there any issues with compatibility of plugins for CassandraDB running on Scylla?
For someone who wants to deploy and tune Scylla, what are the steps involved?
Is it possible to join a Scylla cluster to an existing Cassandra cluster for live data migration and zero downtime swap?
What prompted the decision to form a company around the database?
What are some other uses of Seastar?
Keep in touch
Eyal
LinkedIn

ScyllaDB
Website
@ScyllaDB on Twitter
GitHub
Mailing List
Slack

Links
Seastar Project
DataStax
XFS
TitanDB
OpenTSDB
KairosDB
CQL
Pedis
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
36 min
March 05, 2017Defining Data Engineering with Maxime Beauchemin - Episode 3
Summary
What exactly is data engineering? How has it evolved in recent years and where is it going? How do you get started in the field? In this episode, Maxime Beauchemin joins me to discuss these questions and more.
Transcript provided by CastSource
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
You can help support the show by checking out the Patreon page which is linked from the site.
To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
Your host is Tobias Macey and today I’m interviewing Maxime Beauchemin
Questions
Introduction
How did you get involved in the field of data engineering?
How do you define data engineering and how has that changed in recent years?
Do you think that the DevOps movement over the past few years has had any impact on the discipline of data engineering? If so, what kinds of cross-over have you seen?
For someone who wants to get started in the field of data engineering what are some of the necessary skills?
What do you see as the biggest challenges facing data engineers currently?
At what scale does it become necessary to differentiate between someone who does data engineering vs data infrastructure and what are the differences in terms of skill set and problem domain?
How much analytical knowledge is necessary for a typical data engineer?
What are some of the most important considerations when establishing new data sources to ensure that the resulting information is of sufficient quality?
You have commented on the fact that data engineering borrows a number of elements from software engineering. Where does the concept of unit testing fit in data management and what are some of the most effective patterns for implementing that practice?
How has the work done by data engineers and managers of data infrastructure bled back into mainstream software and systems engineering in terms of tools and best practices?
How do you see the role of data engineers evolving in the next few years?
Keep In Touch
@mistercrunch on Twitter
mistercrunch on GitHub
Medium
Links
Datadog
Airflow
The Rise of the Data Engineer
Druid.io
Luigi
Apache Beam
Samza
Hive
Data Modeling
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
46 min
January 22, 2017Dask with Matthew Rocklin - Episode 2
Summary
There is a vast constellation of tools and platforms for processing and analyzing your data. In this episode Matthew Rocklin talks about how Dask fills the gap between a task oriented workflow tool and an in memory processing framework, and how it brings the power of Python to bear on the problem of big data.
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
You can help support the show by checking out the Patreon page which is linked from the site.
To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
Your host is Tobias Macey and today I’m interviewing Matthew Rocklin about Dask and the Blaze ecosystem.
Interview with Matthew Rocklin
Introduction
How did you get involved in the area of data engineering?
Dask began its life as part of the Blaze project. Can you start by describing what Dask is and how it originated?
There are a vast number of tools in the field of data analytics. What are some of the specific use cases that Dask was built for that weren’t able to be solved by the existing options?
One of the compelling features of Dask is the fact that it is a Python library that allows for distributed computation at a scale that has largely been the exclusive domain of tools in the Hadoop ecosystem. Why do you think that the JVM has been the reigning platform in the data analytics space for so long?
Do you consider Dask, along with the larger Blaze ecosystem, to be a competitor to the Hadoop ecosystem, either now or in the future?
Are you seeing many Hadoop or Spark solutions being migrated to Dask? If so, what are the common reasons?
There is a strong focus for using Dask as a tool for interactive exploration of data. How does it compare to something like Apache Drill?
For anyone looking to integrate Dask into an existing code base that is already using NumPy or Pandas, what does that process look like?
How do the task graph capabilities compare to something like Airflow or Luigi?
Looking through the documentation for the graph specification in Dask, it appears that there is the potential to introduce cycles or other bugs into a large or complex task chain. Is there any built-in tooling to check for that before submitting the graph for execution?
What are some of the most interesting or unexpected projects that you have seen Dask used for?
What do you perceive as being the most relevant aspects of Dask for data engineering/data infrastructure practitioners, as compared to the end users of the systems that they support?
What are some of the most significant problems that you have been faced with, and which still need to be overcome in the Dask project?
I know that the work on Dask is largely performed under the umbrella of PyData and sponsored by Continuum Analytics. What are your thoughts on the financial landscape for open source data analytics and distributed computation frameworks as compared to the broader world of open source projects?
Keep in touch
@mrocklin on Twitter
mrocklin on GitHub
Links
http://matthewrocklin.com/blog/work/2016/09/22/cluster-deployments?utm_source=rss&utm_medium=rss
https://opendatascience.com/blog/dask-for-institutions/?utm_source=rss&utm_medium=rss
Continuum Analytics
2sigma
X-Array
Tornado
Website
Podcast Interview

Airflow

Luigi

Mesos

Kubernetes

Spark

Dryad

Yarn

Read The Docs

XData

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
47 min
January 14, 2017Pachyderm with Daniel Whitenack - Episode 1
Summary
Do you wish that you could track the changes in your data the same way that you track the changes in your code? Pachyderm is a platform for building a data lake with a versioned file system. It also lets you use whatever languages you want to run your analysis with its container based task graph. This week Daniel Whitenack shares the story of how the project got started, how it works under the covers, and how you can get started using it today!
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
You can help support the show by checking out the Patreon page which is linked from the site.
To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
Your host is Tobias Macey and today I’m interviewing Daniel Whitenack about Pachyderm, a modern container based system for building and analyzing a versioned data lake.
Interview with Daniel Whitenack
Introduction
How did you get started in the data engineering space?
What is pachyderm and what problem were you trying to solve when the project was started?
Where does the name come from?
What are some of the competing projects in the space and what features does Pachyderm offer that would convince someone to choose it over the other options?
Because of the fact that the analysis code and the data that it acts on are all versioned together it allows for tracking the provenance of the end result. Why is this such an important capability in the context of data engineering and analytics?
What does Pachyderm use for the distribution and scaling mechanism of the file system?
Given that you can version your data and track all of the modifications made to it in a manner that allows for traversal of those changesets, how much additional storage is necessary over and above the original capacity needed for the raw data?
For a typical use of Pachyderm would someone keep all of the revisions in perpetuity or are the changesets primarily just useful in the context of an analysis workflow?
Given that the state of the data is calculated by applying the diffs in sequence what impact does that have on processing speed and what are some of the ways of mitigating that?
Another compelling feature of Pachyderm is the fact that it natively supports the use of any language for interacting with your data. Why is this such an important capability and why is it more difficult with alternative solutions?
How did you implement this feature so that it would be maintainable and easy to implement for end users?

Given that the intent of using containers is for encapsulating the analysis code from experimentation through to production, it seems that there is the potential for the implementations to run into problems as they scale. What are some things that users should be aware of to help mitigate this?

The data pipeline and dependency graph tooling is a useful addition to the combination of file system and processing interface. Does that preclude any requirement for external tools such as Luigi or Airflow?

I see that the docs mention using the map reduce pattern for analyzing the data in Pachyderm. Does it support other approaches such as streaming or tools like Apache Drill?

What are some of the most interesting deployments and uses of Pachyderm that you have seen?

What are some of the areas that you are looking for help from the community and are there any particular issues that the listeners can check out to get started with the project?

Keep in touch
Daniel
Twitter – @dwhitena

Pachyderm
Website

Free Weekend Project
GopherNotes
Links
AirBnB
RethinkDB
Flocker
Infinite Project
Git LFS
Luigi
Airflow
Kafka
Kubernetes
Rkt
SciKit Learn
Docker
Minikube
General Fusion
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
45 min
January 08, 2017Introducing The Show
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
You can help support the show by checking out the Patreon page which is linked from the site.
To help other people find the show you can leave a review on iTunes, or Google Play Music, share it on social media, and tell your friends and co-workers.
I’m your host, Tobias Macey, and today I’m speaking with Maxime Beauchemin about what it means to be a data engineer.
Interview
Who am I
Systems administrator and software engineer, now DevOps, focus on automation
Host of Podcast.__init__
How did I get involved in data management
Why am I starting a podcast about Data Engineering
Interesting area with a lot of activity
Not currently any shows focused on data engineering
What kinds of topics do I want to cover
Data stores
Pipelines
Tooling
Automation
Monitoring
Testing
Best practices
Common challenges
Defining the role/job hunting
Relationship with data engineers/data analysts
Get in touch and subscribe
Website
Newsletter
Twitter
Email
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
5 min

About Data Engineering Podcast

From the publisher's feed

This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some…