Data Engineering Podcast

Let The Whole Team Participate In Data With The Quilt Versioned Data Hub


Listen Later

Summary

Data is a team sport, but it's often difficult for everyone on the team to participate. For a long time the mantra of data tools has been "by developers, for developers", which automatically excludes a large portion of the business members who play a crucial role in the success of any data project. Quilt Data was created as an answer to make it easier for everyone to contribute to the data being used by an organization and collaborate on its application. In this episode Aneesh Karve shares the journey that Quilt has taken to provide an approachable interface for working with versioned data in S3 that empowers everyone to collaborate.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • Truly leveraging and benefiting from streaming data is hard - the data stack is costly, difficult to use and still has limitations. Materialize breaks down those barriers with a true cloud-native streaming database - not simply a database that connects to streaming systems. With a PostgreSQL-compatible interface, you can now work with real-time data using ANSI SQL including the ability to perform multi-way complex joins, which support stream-to-stream, stream-to-table, table-to-table, and more, all in standard SQL. Go to dataengineeringpodcast.com/materialize today and sign up for early access to get started. If you like what you see and want to help make it better, they're hiring across all functions!
  • Your host is Tobias Macey and today I'm interviewing Aneesh Karve about how Quilt Data helps you bring order to your chaotic data in S3 with transactional versioning and data discovery built in
  • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Quilt is and the story behind it?
      • How have the goals and features of the Quilt platform changed since I spoke with Kevin in June of 2018?
      • What are the main problems that users are trying to solve when they find Quilt?
        • What are some of the alternative approaches/products that they are coming from?
        • How does Quilt compare with options such as LakeFS, Unstruk, Pachyderm, etc.?
        • Can you describe how Quilt is implemented?
        • What are the types of tools and systems that Quilt gets integrated with?
          • How do you manage the tension between supporting the lowest common denominator, while providing options for more advanced capabilities?
          • What is a typical workflow for a team that is using Quilt to manage their data?
          • What are the most interesting, innovative, or unexpected ways that you have seen Quilt used?
          • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Quilt?
          • When is Quilt the wrong choice?
          • What do you have planned for the future of Quilt?
          • Contact Info
            • LinkedIn
            • @akarve on Twitter
            • Parting Question
              • From your perspective, what is the biggest gap in the tooling or technology for data management today?
              • Closing Announcements
                • Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
                • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
                • If you've learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
                • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
                • Links
                  • Quilt Data
                    • Podcast Episode
                    • UW Madison
                    • Docker Swarm
                    • Kaggle
                    • open.quiltdata.com
                    • FinOS Perspective
                    • LakeFS
                      • Podcast Episode
                      • Pachyderm
                        • Podcast Episode
                        • Unstruk
                          • Podcast Episode
                          • Parquet
                          • Avro
                          • ORC
                          • Cloudformation
                          • Troposphere
                          • CDK == Cloud Development Kit
                          • Shadow IT
                            • Podcast Episode
                            • Delta Lake
                              • Podcast Episode
                              • Apache Iceberg
                                • Podcast Episode
                                • Datasette
                                • Frictionless
                                • DVC
                                  • Podcast.__init__ Episode
                                  • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                                    Sponsored By:

                                    • Materialize: ![Materialize](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/NuMEahiy.png)
                                    Looking for the simplest way to get the freshest data possible to your teams? Because let's face it: if real-time were easy, everyone would be using it. Look no further than Materialize, the streaming database you already know how to use.
                                    Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Delivered as a single platform with the separation of storage and compute, strict-serializability, active replication, horizontal scalability and workload isolation — Materialize is now the fastest way to build products with streaming data, drastically reducing the time, expertise, cost and maintenance traditionally associated with implementation of real-time features.
                                    Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.
                                    Go to [materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access)

                                    Support Data Engineering Podcast

                                    ...more
                                    View all episodesView all episodes
                                    Download on the App Store

                                    Data Engineering PodcastBy Tobias Macey

                                    • 4.6
                                    • 4.6
                                    • 4.6
                                    • 4.6
                                    • 4.6

                                    4.6

                                    134 ratings


                                    More shows like Data Engineering Podcast

                                    View all
                                    Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

                                    Software Engineering Radio - the podcast for professional software developers

                                    262 Listeners

                                    The Changelog: Software Development, Open Source by Changelog Media

                                    The Changelog: Software Development, Open Source

                                    285 Listeners

                                    The Cloudcast by Massive Studios

                                    The Cloudcast

                                    154 Listeners

                                    Thoughtworks Technology Podcast by Thoughtworks

                                    Thoughtworks Technology Podcast

                                    43 Listeners

                                    Data Skeptic by Kyle Polich

                                    Data Skeptic

                                    474 Listeners

                                    Talk Python To Me by Michael Kennedy

                                    Talk Python To Me

                                    584 Listeners

                                    Software Engineering Daily by Software Engineering Daily

                                    Software Engineering Daily

                                    630 Listeners

                                    AWS Podcast by Amazon Web Services

                                    AWS Podcast

                                    200 Listeners

                                    Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                                    Super Data Science: ML & AI Podcast with Jon Krohn

                                    295 Listeners

                                    Python Bytes by Michael Kennedy and Brian Okken

                                    Python Bytes

                                    212 Listeners

                                    DataFramed by DataCamp

                                    DataFramed

                                    267 Listeners

                                    Practical AI by Practical AI LLC

                                    Practical AI

                                    196 Listeners

                                    The Stack Overflow Podcast by The Stack Overflow Podcast

                                    The Stack Overflow Podcast

                                    63 Listeners

                                    The Real Python Podcast by Real Python

                                    The Real Python Podcast

                                    137 Listeners

                                    Latent Space: The AI Engineer Podcast by swyx + Alessio

                                    Latent Space: The AI Engineer Podcast

                                    64 Listeners