Data Engineering Podcast

Building An Internal Database As A Service Platform At Cloudflare

Listen Later
Summary

Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? In this episode Vignesh Ravichandran explains how his team at Cloudflare provides PostgreSQL as a service to their developers for low latency and high uptime services at global scale. This is an interesting and insightful look at pragmatic engineering for reliability and scale.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack
  • This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold
  • You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free!
  • Your host is Tobias Macey and today I'm interviewing Vignesh Ravichandran about building an internal database as a service platform at Cloudflare
    • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you start by describing the different database workloads that you have at Cloudflare?
      • What are the different methods that you have used for managing database instances?
      • What are the requirements and constraints that you had to account for in designing your current system?
      • Why Postgres?
      • optimizations for Postgres
        • simplification from not supporting multiple engines
        • limitations in postgres that make multi-tenancy challenging
        • scale of operation (data volume, request rate
        • What are the most interesting, innovative, or unexpected ways that you have seen your DBaaS used?
        • What are the most interesting, unexpected, or challenging lessons that you have learned while working on your internal database platform?
        • When is an internal database as a service the wrong choice?
        • What do you have planned for the future of Postgres hosting at Cloudflare?
          • Contact Info
          • LinkedIn
          • Website
            • Parting Question
            • From your perspective, what is the biggest gap in the tooling or technology for data management today?
              • Closing Announcements
              • Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
              • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
              • If you've learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
              • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
                • Links
                • Cloudflare
                • PostgreSQL
                  • Podcast Episode
                  • IP Address Data Type in Postgres
                  • CockroachDB
                    • Podcast Episode
                    • Citus
                      • Podcast Episode
                      • Yugabyte
                        • Podcast Episode
                        • Stolon
                        • pg_rewind
                        • PGBouncer
                        • HAProxy Presentation
                        • Etcd
                        • Patroni
                        • pg_upgrade
                        • Edge Computing

                          • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                          Sponsored By:

                          • Rudderstack: ![Rudderstack](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/CKNV8HZ6.png)
                          Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)
                        • Materialize: ![Materialize](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/NuMEahiy.png)
                          • You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date.
                          That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI. Built on Timely Dataflow and Differential Dataflow, open source frameworks created by cofounder Frank McSherry at Microsoft Research, Materialize is trusted by data and engineering teams at Ramp, Pluralsight, Onward and more to build real-time data products without the cost, complexity, and development time of stream processing.
                          Go to [materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access) today and get 2 weeks free!
                        • Datafold: ![Datafold](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/zm6x2tFu.png)
                          • This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!

                          Support Data Engineering Podcast

                          ...more
                          View all episodesView all episodes
                          Download on the App Store
                          Data Engineering PodcastBy Tobias Macey
                          • 4.5
                          • 4.5
                          • 4.5
                          • 4.5
                          • 4.5

                          4.5

                          142 ratings

                          More shows like Data Engineering Podcast

                          View all
                          This Week in Startups by Jason Calacanis

                          This Week in Startups

                          1,301 Listeners

                          The Changelog: Software Development, Open Source by Changelog Media

                          The Changelog: Software Development, Open Source

                          288 Listeners

                          The a16z Show by Andreessen Horowitz

                          The a16z Show

                          1,109 Listeners

                          Software Engineering Daily by Software Engineering Daily

                          Software Engineering Daily

                          631 Listeners

                          Risky Business by Risky Business Media

                          Risky Business

                          373 Listeners

                          Talk Python To Me by Michael Kennedy

                          Talk Python To Me

                          583 Listeners

                          Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                          Super Data Science: ML & AI Podcast with Jon Krohn

                          308 Listeners

                          NVIDIA AI Podcast by NVIDIA

                          NVIDIA AI Podcast

                          347 Listeners

                          Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

                          Syntax - Tasty Web Development Treats

                          990 Listeners

                          Practical AI by Practical AI LLC

                          Practical AI

                          211 Listeners

                          Dwarkesh Podcast by Dwarkesh Patel

                          Dwarkesh Podcast

                          549 Listeners

                          The Data Engineering Show by The Firebolt Data Bros

                          The Data Engineering Show

                          9 Listeners

                          Latent Space: The AI Engineer Podcast by Latent.Space

                          Latent Space: The AI Engineer Podcast

                          105 Listeners

                          This Day in AI Podcast by Michael Sharkey, Chris Sharkey

                          This Day in AI Podcast

                          227 Listeners

                          The AI Daily Brief: Artificial Intelligence News and Analysis by Nathaniel Whittemore

                          The AI Daily Brief: Artificial Intelligence News and Analysis

                          681 Listeners