Data Engineering Podcast

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

Listen Later
Summary

Despite the best efforts of data engineers, data is as messy as the real world. Entity resolution and fuzzy matching are powerful utilities for cleaning up data from disconnected sources, but it has typically required custom development and training machine learning models. Sonal Goyal created and open-sourced Zingg as a generalized tool for data mastering and entity resolution to reduce the effort involved in adopting those practices. In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudder
  • Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. By the time errors have made their way into production, it’s often too late and damage is done. Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. No more shipping and praying, you can now know exactly what will change in your database! Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.
  • Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
  • Your host is Tobias Macey and today I’m interviewing Sonal Goyal about Zingg, an open source entity resolution framework for data engineers
    • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Zingg is and the story behind it?
    • Who is the target audience for Zingg?
      • How has that informed your efforts in the development and release of the project?
      • What are the use cases where entity resolution is helpful or necessary in a data engineering context?
      • What are the range of options that are available for teams to implement entity/identity resolution in their data?
        • What was your motivation for creating an open source solution for this use case?
        • Why do you think there has not been a compelling open source and generalized solution previously?
        • Can you describe how Zingg is implemented?
          • How have the design and goals shifted since you started working on the project?
          • What does the installation and integration process look like for Zingg?
          • Once you have Zingg configured, what is the workflow for a data engineer or analyst?
          • What are the extension/customization options for someone using Zingg in their environment?
          • What are the most interesting, innovative, or unexpected ways that you have seen Zingg used?
          • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Zingg?
          • When is Zingg the wrong choice?
          • What do you have planned for the future of Zingg?
            • Contact Info
            • LinkedIn
            • @sonalgoyal on Twitter
            • sonalgoyal on GitHub
              • Parting Question
              • From your perspective, what is the biggest gap in the tooling or technology for data management today?
                • Closing Announcements
                • Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
                • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
                • If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
                • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
                  • Links
                  • Zingg
                  • Entity Resolution
                  • MDM == Master Data Management
                    • Podcast Episode
                    • Snowflake
                      • Podcast Episode
                      • Snowpark
                      • Spark
                      • Milvus
                        • Podcast Episode
                        • Pinecone
                          • Podcast Episode
                          • DuckDB
                            • Podcast Episode

                              • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                              Support Data Engineering Podcast

                              ...more
                              View all episodesView all episodes
                              Download on the App Store
                              Data Engineering PodcastBy Tobias Macey
                              • 4.6
                              • 4.6
                              • 4.6
                              • 4.6
                              • 4.6

                              4.6

                              134 ratings

                              More shows like Data Engineering Podcast

                              View all
                              Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

                              Software Engineering Radio - the podcast for professional software developers

                              262 Listeners

                              The Changelog: Software Development, Open Source by Changelog Media

                              The Changelog: Software Development, Open Source

                              285 Listeners

                              The Cloudcast by Massive Studios

                              The Cloudcast

                              154 Listeners

                              Thoughtworks Technology Podcast by Thoughtworks

                              Thoughtworks Technology Podcast

                              42 Listeners

                              Data Skeptic by Kyle Polich

                              Data Skeptic

                              474 Listeners

                              Talk Python To Me by Michael Kennedy

                              Talk Python To Me

                              583 Listeners

                              Software Engineering Daily by Software Engineering Daily

                              Software Engineering Daily

                              629 Listeners

                              AWS Podcast by Amazon Web Services

                              AWS Podcast

                              200 Listeners

                              Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                              Super Data Science: ML & AI Podcast with Jon Krohn

                              294 Listeners

                              Python Bytes by Michael Kennedy and Brian Okken

                              Python Bytes

                              212 Listeners

                              DataFramed by DataCamp

                              DataFramed

                              269 Listeners

                              Practical AI by Practical AI LLC

                              Practical AI

                              196 Listeners

                              The Stack Overflow Podcast by The Stack Overflow Podcast

                              The Stack Overflow Podcast

                              63 Listeners

                              The Real Python Podcast by Real Python

                              The Real Python Podcast

                              137 Listeners

                              Latent Space: The AI Engineer Podcast by swyx + Alessio

                              Latent Space: The AI Engineer Podcast

                              64 Listeners