Data Engineering Podcast

The Importance Of Data Contracts As The Interface For Data Integration With Abhi Sivasailam


Listen Later

Summary

Data platforms are exemplified by a complex set of connections that are subject to a set of constantly evolving requirements. In order to make this a tractable problem it is necessary to define boundaries for communication between concerns, which brings with it the need to establish interface contracts for communicating across those boundaries. The recent move toward the data mesh as a formalized architecture that builds on this design provides the language that data teams need to make this a more organized effort. In this episode Abhi Sivasailam shares his experience designing and implementing a data mesh solution with his team at Flexport, and the importance of defining and enforcing data contracts that are implemented at those domain boundaries.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
  • Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
  • StreamSets DataOps Platform is the world’s first single platform for building smart data pipelines across hybrid and multi-cloud architectures. Build, run, monitor and manage data pipelines confidently with an end-to-end data integration platform that’s built for constant change. Amp up your productivity with an easy-to-navigate interface and 100s of pre-built connectors. And, get pipelines and new hires up and running quickly with powerful, reusable components that work across batch and streaming. Once you’re up and running, your smart data pipelines are resilient to data drift. Those ongoing and unexpected changes in schema, semantics, and infrastructure. Finally, one single pane of glass for operating and monitoring all your data pipelines. The full transparency and control you desire for your data operations. Get started building pipelines in minutes for free at dataengineeringpodcast.com/streamsets. The first 10 listeners of the podcast that subscribe to StreamSets’ Professional Tier, receive 2 months free after their first month.
  • Your host is Tobias Macey and today I’m interviewing Abhi Sivasailam about the different social and technical interfaces available for defining and enforcing data contracts
  • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you start by explaining what your working definition of a "data contract" is?
      • What are the goals and purpose of these contracts?
      • What are the locations and methods of defining a data contract?
        • What kind of information needs to be encoded in a contract definition?
        • How do you manage enforcement of contracts?
        • manifestations of contracts in data mesh implementation
        • ergonomics (technical and social) of data contracts and how to prevent them from prohibiting productivity
        • What are the most interesting, innovative, or unexpected approaches to data contracts that you have seen?
        • What are the most interesting, unexpected, or challenging lessons that you have learned while working on data contract implementation?
        • When are data contracts the wrong choice?
        • Contact Info
          • LinkedIn
          • @_abhisivasailam on Twitter
          • Website
          • Parting Question
            • From your perspective, what is the biggest gap in the tooling or technology for data management today?
            • Closing Announcements
              • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
              • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
              • If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
              • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
              • Links
                • Flexport
                • Debezium
                  • Podcast Episode
                  • Data Mesh At Flexport Presentation
                  • Data Mesh
                    • Podcast Episode
                    • Column Names As Contracts podcast episode with Emily Riederer
                    • dbtplyr
                    • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                      Support Data Engineering Podcast

                      ...more
                      View all episodesView all episodes
                      Download on the App Store

                      Data Engineering PodcastBy Tobias Macey

                      • 4.6
                      • 4.6
                      • 4.6
                      • 4.6
                      • 4.6

                      4.6

                      135 ratings


                      More shows like Data Engineering Podcast

                      View all
                      Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

                      Software Engineering Radio - the podcast for professional software developers

                      272 Listeners

                      The Changelog: Software Development, Open Source by Changelog Media

                      The Changelog: Software Development, Open Source

                      282 Listeners

                      The Cloudcast by Massive Studios

                      The Cloudcast

                      152 Listeners

                      Thoughtworks Technology Podcast by Thoughtworks

                      Thoughtworks Technology Podcast

                      42 Listeners

                      Data Skeptic by Kyle Polich

                      Data Skeptic

                      481 Listeners

                      Talk Python To Me by Michael Kennedy

                      Talk Python To Me

                      590 Listeners

                      Software Engineering Daily by Software Engineering Daily

                      Software Engineering Daily

                      626 Listeners

                      The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by Sam Charrington

                      The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

                      441 Listeners

                      Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                      Super Data Science: ML & AI Podcast with Jon Krohn

                      299 Listeners

                      Python Bytes by Michael Kennedy and Brian Okken

                      Python Bytes

                      213 Listeners

                      DataFramed by DataCamp

                      DataFramed

                      265 Listeners

                      Practical AI by Practical AI LLC

                      Practical AI

                      189 Listeners

                      The Stack Overflow Podcast by The Stack Overflow Podcast

                      The Stack Overflow Podcast

                      64 Listeners

                      The Real Python Podcast by Real Python

                      The Real Python Podcast

                      140 Listeners

                      Latent Space: The AI Engineer Podcast by swyx + Alessio

                      Latent Space: The AI Engineer Podcast

                      76 Listeners