Data Engineering Podcast

Ceph: A Reliable And Scalable Distributed Filesystem with Sage Weil - Episode 40

Listen Later
Summary

When working with large volumes of data that you need to access in parallel across multiple instances you need a distributed filesystem that will scale with your workload. Even better is when that same system provides multiple paradigms for interacting with the underlying storage. Ceph is a highly available, highly scalable, and performant system that has support for object storage, block storage, and native filesystem access. In this episode Sage Weil, the creator and lead maintainer of the project, discusses how it got started, how it works, and how you can start using it on your infrastructure today. He also explains where it fits in the current landscape of distributed storage and the plans for future improvements.

Preamble
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute.
  • Are you struggling to keep up with customer request and letting errors slip into production? Want to try some of the innovative ideas in this podcast but don’t have time? DataKitchen’s DataOps software allows your team to quickly iterate and deploy pipelines of code, models, and data sets while improving quality. Unlike a patchwork of manual operations, DataKitchen makes your team shine by providing an end to end DataOps solution with minimal programming that uses the tools you love. Join the DataOps movement and sign up for the newsletter at datakitchen.io/de today. After that learn more about why you should be doing DataOps by listening to the Head Chef in the Data Kitchen at dataengineeringpodcast.com/datakitchen
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Sage Weil about Ceph, an open source distributed file system that supports block storage, object storage, and a file system interface.
    • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you start with an overview of what Ceph is?
      • What was the motivation for starting the project?
      • What are some of the most common use cases for Ceph?


      • There are a large variety of distributed file systems. How would you characterize Ceph as it compares to other options (e.g. HDFS, GlusterFS, LionFS, SeaweedFS, etc.)?

      • Given that there is no single point of failure, what mechanisms do you use to mitigate the impact of network partitions?

        • What mechanisms are available to ensure data integrity across the cluster?


        • How is Ceph implemented and how has the design evolved over time?

        • What is required to deploy and manage a Ceph cluster?

          • What are the scaling factors for a cluster?
          • What are the limitations?


          • How does Ceph handle mixed write workloads with either a high volume of small files or a smaller volume of larger files?

          • In services such as S3 the data is segregated from block storage options like EBS or EFS. Since Ceph provides all of those interfaces in one project is it possible to use each of those interfaces to the same data objects in a Ceph cluster?

          • In what situations would you advise someone against using Ceph?

          • What are some of the most interested, unexpected, or challenging aspects of working with Ceph and the community?

          • What are some of the plans that you have for the future of Ceph?

            • Contact Info
            • Email
            • @liewegas on Twitter
            • liewegas on GitHub
              • Parting Question
              • From your perspective, what is the biggest gap in the tooling or technology for data management today?
                • Links
                • Ceph
                • Red Hat
                • DreamHost
                • UC Santa Cruz
                • Los Alamos National Labs
                • Dream Objects
                • OpenStack
                • Proxmox
                • POSIX
                • GlusterFS
                • Hadoop
                • Ceph Architecture
                • Paxos
                • relatime
                • Prometheus
                • Zabbix
                • Kubernetes
                • NVMe
                • DNS-SD
                • Consul
                • EtcD
                • DNS SRV Record
                • Zeroconf
                • Bluestore
                • XFS
                • Erasure Coding
                • NFS
                • Seastar
                • Rook

                  • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                  Support Data Engineering Podcast

                  ...more
                  View all episodesView all episodes
                  Download on the App Store
                  Data Engineering PodcastBy Tobias Macey
                  • 4.5
                  • 4.5
                  • 4.5
                  • 4.5
                  • 4.5

                  4.5

                  142 ratings

                  More shows like Data Engineering Podcast

                  View all
                  This Week in Startups by Jason Calacanis

                  This Week in Startups

                  1,299 Listeners

                  The Changelog: Software Development, Open Source by Changelog Media

                  The Changelog: Software Development, Open Source

                  288 Listeners

                  The a16z Show by Andreessen Horowitz

                  The a16z Show

                  1,106 Listeners

                  Software Engineering Daily by Software Engineering Daily

                  Software Engineering Daily

                  630 Listeners

                  Risky Business by Risky Business Media

                  Risky Business

                  372 Listeners

                  Talk Python To Me by Michael Kennedy

                  Talk Python To Me

                  583 Listeners

                  Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                  Super Data Science: ML & AI Podcast with Jon Krohn

                  309 Listeners

                  NVIDIA AI Podcast by NVIDIA

                  NVIDIA AI Podcast

                  346 Listeners

                  Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

                  Syntax - Tasty Web Development Treats

                  987 Listeners

                  Practical AI by Practical AI LLC

                  Practical AI

                  210 Listeners

                  Dwarkesh Podcast by Dwarkesh Patel

                  Dwarkesh Podcast

                  550 Listeners

                  The Data Engineering Show by The Firebolt Data Bros

                  The Data Engineering Show

                  10 Listeners

                  Latent Space: The AI Engineer Podcast by Latent.Space

                  Latent Space: The AI Engineer Podcast

                  104 Listeners

                  This Day in AI Podcast by Michael Sharkey, Chris Sharkey

                  This Day in AI Podcast

                  227 Listeners

                  The AI Daily Brief: Artificial Intelligence News and Analysis by Nathaniel Whittemore

                  The AI Daily Brief: Artificial Intelligence News and Analysis

                  680 Listeners