July 29, 2019

Simplifying Data Integration Through Eventual Connectivity

53 minutes

Summary

The ETL pattern that has become commonplace for integrating data from multiple sources has proven useful, but complex to maintain. For a small number of sources it is a tractable problem, but as the overall complexity of the data ecosystem continues to expand it may be time to identify new ways to tame the deluge of information. In this episode Tim Ward, CEO of CluedIn, explains the idea of eventual connectivity as a new paradigm for data integration. Rather than manually defining all of the mappings ahead of time, we can rely on the power of graph databases and some strategic metadata to allow connections to occur as the data becomes available. If you are struggling to maintain a tangle of data pipelines then you might find some new ideas for reducing your workload.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!

To connect with the startups that are shaping the future and take advantage of the opportunities that they provide, check out Angel List where you can invest in innovative business, find a job, or post a position of your own. Sign up today at dataengineeringpodcast.com/angel and help support this show.

You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Upcoming events include the O’Reilly AI Conference, the Strata Data Conference, and the combined events of the Data Architecture Summit and Graphorum. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.

Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.

To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat

Your host is Tobias Macey and today I’m interviewing Tim Ward about his thoughts on eventual connectivity as a new pattern to replace traditional ETL

Interview

Introduction

How did you get involved in the area of data management?

Can you start by discussing the challenges and shortcomings that you perceive in the existing practices of ETL?

What is eventual connectivity and how does it address the problems with ETL in the current data landscape?

In your white paper you mention the benefits of graph technology and how it solves the problem of data integration. Can you talk through an example use case?

How do different implementations of graph databases impact their viability for this use case?

Can you talk through the overall system architecture and data flow for an example implementation of eventual connectivity?

How much up-front modeling is necessary to make this a viable approach to data integration?

How do the volume and format of the source data impact the technology and architecture decisions that you would make?

What are the limitations or edge cases that you have found when using this pattern?

In modern ETL architectures there has been a lot of time and work put into workflow management systems for orchestrating data flows. Is there still a place for those tools when using the eventual connectivity pattern?

What resources do you recommend for someone who wants to learn more about this approach and start using it in their organization?

Contact Info

@jerrong on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

Eventual Connectivity White Paper

CluedIn

Podcast Episode

Copenhagen

Ewok

Multivariate Testing

CRM

ERP

ETL

ELT

DAG

Graph Database

Apache NiFi

Podcast Episode

Apache Airflow

Podcast.init Episode

BigQuery

RedShift

CosmosDB

SAP HANA

IOT == Internet of Things

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Support Data Engineering Podcast

...more

View all episodes

By Tobias Macey

4.6

135135 ratings