Data Engineering Podcast

Defining Data Engineering with Maxime Beauchemin - Episode 3

Listen Later
Summary

What exactly is data engineering? How has it evolved in recent years and where is it going? How do you get started in the field? In this episode, Maxime Beauchemin joins me to discuss these questions and more.

Transcript provided by CastSource

Preamble
  • Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
  • You can help support the show by checking out the Patreon page which is linked from the site.
  • To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
  • Your host is Tobias Macey and today I’m interviewing Maxime Beauchemin
    • Questions
    • Introduction
    • How did you get involved in the field of data engineering?
    • How do you define data engineering and how has that changed in recent years?
    • Do you think that the DevOps movement over the past few years has had any impact on the discipline of data engineering? If so, what kinds of cross-over have you seen?
    • For someone who wants to get started in the field of data engineering what are some of the necessary skills?
    • What do you see as the biggest challenges facing data engineers currently?
    • At what scale does it become necessary to differentiate between someone who does data engineering vs data infrastructure and what are the differences in terms of skill set and problem domain?
    • How much analytical knowledge is necessary for a typical data engineer?
    • What are some of the most important considerations when establishing new data sources to ensure that the resulting information is of sufficient quality?
    • You have commented on the fact that data engineering borrows a number of elements from software engineering. Where does the concept of unit testing fit in data management and what are some of the most effective patterns for implementing that practice?
    • How has the work done by data engineers and managers of data infrastructure bled back into mainstream software and systems engineering in terms of tools and best practices?
    • How do you see the role of data engineers evolving in the next few years?
      • Keep In Touch
      • @mistercrunch on Twitter
      • mistercrunch on GitHub
      • Medium
        • Links
        • Datadog
        • Airflow
        • The Rise of the Data Engineer
        • Druid.io
        • Luigi
        • Apache Beam
        • Samza
        • Hive
        • Data Modeling

          • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

          Support Data Engineering Podcast

          ...more
          View all episodesView all episodes
          Download on the App Store
          Data Engineering PodcastBy Tobias Macey
          • 4.5
          • 4.5
          • 4.5
          • 4.5
          • 4.5

          4.5

          142 ratings

          More shows like Data Engineering Podcast

          View all
          This Week in Startups by Jason Calacanis

          This Week in Startups

          1,299 Listeners

          The Changelog: Software Development, Open Source by Changelog Media

          The Changelog: Software Development, Open Source

          288 Listeners

          The a16z Show by Andreessen Horowitz

          The a16z Show

          1,106 Listeners

          Software Engineering Daily by Software Engineering Daily

          Software Engineering Daily

          630 Listeners

          Risky Business by Risky Business Media

          Risky Business

          372 Listeners

          Talk Python To Me by Michael Kennedy

          Talk Python To Me

          583 Listeners

          Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

          Super Data Science: ML & AI Podcast with Jon Krohn

          309 Listeners

          NVIDIA AI Podcast by NVIDIA

          NVIDIA AI Podcast

          346 Listeners

          Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

          Syntax - Tasty Web Development Treats

          987 Listeners

          Practical AI by Practical AI LLC

          Practical AI

          210 Listeners

          Dwarkesh Podcast by Dwarkesh Patel

          Dwarkesh Podcast

          550 Listeners

          The Data Engineering Show by The Firebolt Data Bros

          The Data Engineering Show

          10 Listeners

          Latent Space: The AI Engineer Podcast by Latent.Space

          Latent Space: The AI Engineer Podcast

          104 Listeners

          This Day in AI Podcast by Michael Sharkey, Chris Sharkey

          This Day in AI Podcast

          227 Listeners

          The AI Daily Brief: Artificial Intelligence News and Analysis by Nathaniel Whittemore

          The AI Daily Brief: Artificial Intelligence News and Analysis

          680 Listeners