Data Engineering Podcast

Reflecting On The Past 6 Years Of Data Engineering

Listen Later
Summary

This podcast started almost exactly six years ago, and the technology landscape was much different than it is now. In that time there have been a number of generational shifts in how data engineering is done. In this episode I reflect on some of the major themes and take a brief look forward at some of the upcoming changes.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • Your host is Tobias Macey and today I'm reflecting on the major trends in data engineering over the past 6 years
    • Interview
    • Introduction
    • 6 years of running the Data Engineering Podcast
    • Around the first time that data engineering was discussed as a role
      • Followed on from hype about "data science"
      • Hadoop era
      • Streaming
      • Lambda and Kappa architectures
        • Not really referenced anymore
        • "Big Data" era of capture everything has shifted to focusing on data that presents value
          • Regulatory environment increases risk, better tools introduce more capability to understand what data is useful
          • Data catalogs
            • Amundsen and Alation
            • Orchestration engine
              • Oozie, etc. -> Airflow and Luigi -> Dagster, Prefect, Lyft, etc.
              • Orchestration is now a part of most vertical tools
              • Cloud data warehouses
              • Data lakes
              • DataOps and MLOps
              • Data quality to data observability
              • Metadata for everything
                • Data catalog -> data discovery -> active metadata
                • Business intelligence
                  • Read only reports to metric/semantic layers
                  • Embedded analytics and data APIs
                  • Rise of ELT
                    • dbt
                    • Corresponding introduction of reverse ETL
                    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on running the podcast?
                    • What do you have planned for the future of the podcast?
                      • Parting Question
                      • From your perspective, what is the biggest gap in the tooling or technology for data management today?
                        • Closing Announcements
                        • Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
                        • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
                        • If you've learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
                        • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

                          • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                          Sponsored By:

                          • Materialize: ![Materialize](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/NuMEahiy.png)
                          Looking for the simplest way to get the freshest data possible to your teams? Because let's face it: if real-time were easy, everyone would be using it. Look no further than Materialize, the streaming database you already know how to use.
                          Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Delivered as a single platform with the separation of storage and compute, strict-serializability, active replication, horizontal scalability and workload isolation — Materialize is now the fastest way to build products with streaming data, drastically reducing the time, expertise, cost and maintenance traditionally associated with implementation of real-time features.
                          Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.
                          Go to [materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access)

                          Support Data Engineering Podcast

                          ...more
                          View all episodesView all episodes
                          Download on the App Store
                          Data Engineering PodcastBy Tobias Macey
                          • 4.5
                          • 4.5
                          • 4.5
                          • 4.5
                          • 4.5

                          4.5

                          142 ratings

                          More shows like Data Engineering Podcast

                          View all
                          This Week in Startups by Jason Calacanis

                          This Week in Startups

                          1,301 Listeners

                          The Changelog: Software Development, Open Source by Changelog Media

                          The Changelog: Software Development, Open Source

                          288 Listeners

                          The a16z Show by Andreessen Horowitz

                          The a16z Show

                          1,107 Listeners

                          Software Engineering Daily by Software Engineering Daily

                          Software Engineering Daily

                          630 Listeners

                          Risky Business by Risky Business Media

                          Risky Business

                          373 Listeners

                          Talk Python To Me by Michael Kennedy

                          Talk Python To Me

                          583 Listeners

                          Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                          Super Data Science: ML & AI Podcast with Jon Krohn

                          308 Listeners

                          NVIDIA AI Podcast by NVIDIA

                          NVIDIA AI Podcast

                          347 Listeners

                          Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

                          Syntax - Tasty Web Development Treats

                          988 Listeners

                          Practical AI by Practical AI LLC

                          Practical AI

                          211 Listeners

                          Dwarkesh Podcast by Dwarkesh Patel

                          Dwarkesh Podcast

                          549 Listeners

                          The Data Engineering Show by The Firebolt Data Bros

                          The Data Engineering Show

                          9 Listeners

                          Latent Space: The AI Engineer Podcast by Latent.Space

                          Latent Space: The AI Engineer Podcast

                          104 Listeners

                          This Day in AI Podcast by Michael Sharkey, Chris Sharkey

                          This Day in AI Podcast

                          227 Listeners

                          The AI Daily Brief: Artificial Intelligence News and Analysis by Nathaniel Whittemore

                          The AI Daily Brief: Artificial Intelligence News and Analysis

                          683 Listeners