Data Engineering Podcast

Digging Into Data Replication At Fivetran

Listen Later
Summary

The extract and load pattern of data replication is the most commonly needed process in data engineering workflows. Because of the myriad sources and destinations that are available, it is also among the most difficult tasks that we encounter. Fivetran is a platform that does the hard work for you and replicates information from your source systems into whichever data warehouse you use. In this episode CEO and co-founder George Fraser explains how it is built, how it got started, and the challenges that creep in at the edges when dealing with so many disparate systems that need to be made to work together. This is a great conversation to listen to for a better understanding of the challenges inherent in synchronizing your data.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and Corinium Global Intelligence. Upcoming events include the O’Reilly AI Conference, the Strata Data Conference, and the combined events of the Data Architecture Summit and Graphorum. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing George Fraser about FiveTran, a hosted platform for replicating your data from source to destination
    • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you start by describing the problem that Fivetran solves and the story of how it got started?
    • Integration of multiple data sources (e.g. entity resolution)
    • How is Fivetran architected and how has the overall system design changed since you first began working on it?
    • monitoring and alerting
    • Automated schema normalization. How does it work for customized data sources?
    • Managing schema drift while avoiding data loss
    • Change data capture
    • What have you found to be the most complex or challenging data sources to work with reliably?
    • Workflow for users getting started with Fivetran
    • When is Fivetran the wrong choice for collecting and analyzing your data?
    • What have you found to be the most challenging aspects of working in the space of data integrations?}}
    • What have been the most interesting/unexpected/useful lessons that you have learned while building and growing Fivetran?
    • What do you have planned for the future of Fivetran?
      • Contact Info
      • LinkedIn
      • @frasergeorgew on Twitter
        • Parting Question
        • From your perspective, what is the biggest gap in the tooling or technology for data management today?
          • Links
          • Fivetran
          • Ralph Kimball
          • DBT (Data Build Tool)
            • Podcast Interview
            • Looker
              • Podcast Interview
              • Cron
              • Kubernetes
              • Postgres
                • Podcast Episode
                • Oracle DB
                • Salesforce
                • Netsuite
                • Marketo
                • Jira
                • Asana
                • Cloudwatch
                • Stackdriver

                  • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                  Support Data Engineering Podcast

                  ...more
                  View all episodesView all episodes
                  Download on the App Store
                  Data Engineering PodcastBy Tobias Macey
                  • 4.6
                  • 4.6
                  • 4.6
                  • 4.6
                  • 4.6

                  4.6

                  135 ratings

                  More shows like Data Engineering Podcast

                  View all
                  Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

                  Software Engineering Radio - the podcast for professional software developers

                  272 Listeners

                  The Changelog: Software Development, Open Source by Changelog Media

                  The Changelog: Software Development, Open Source

                  283 Listeners

                  The Cloudcast by Massive Studios

                  The Cloudcast

                  152 Listeners

                  Thoughtworks Technology Podcast by Thoughtworks

                  Thoughtworks Technology Podcast

                  41 Listeners

                  Data Skeptic by Kyle Polich

                  Data Skeptic

                  482 Listeners

                  Talk Python To Me by Michael Kennedy

                  Talk Python To Me

                  592 Listeners

                  Software Engineering Daily by Software Engineering Daily

                  Software Engineering Daily

                  624 Listeners

                  The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by Sam Charrington

                  The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

                  443 Listeners

                  Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                  Super Data Science: ML & AI Podcast with Jon Krohn

                  298 Listeners

                  Python Bytes by Michael Kennedy and Brian Okken

                  Python Bytes

                  213 Listeners

                  DataFramed by DataCamp

                  DataFramed

                  266 Listeners

                  Practical AI by Practical AI LLC

                  Practical AI

                  189 Listeners

                  The Stack Overflow Podcast by The Stack Overflow Podcast

                  The Stack Overflow Podcast

                  64 Listeners

                  The Real Python Podcast by Real Python

                  The Real Python Podcast

                  140 Listeners

                  Latent Space: The AI Engineer Podcast by swyx + Alessio

                  Latent Space: The AI Engineer Podcast

                  77 Listeners