Data Engineering Podcast

Set Up Your Own Data-as-a-Service Platform On Dremio with Tomer Shiran - Episode 58

Listen Later
Summary

When your data lives in multiple locations, belonging to at least as many applications, it is exceedingly difficult to ask complex questions of it. The default way to manage this situation is by crafting pipelines that will extract the data from source systems and load it into a data lake or data warehouse. In order to make this situation more manageable and allow everyone in the business to gain value from the data the folks at Dremio built a self service data platform. In this episode Tomer Shiran, CEO and co-founder of Dremio, explains how it fits into the modern data landscape, how it works under the hood, and how you can start using it today to make your life easier.

Preamble
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Tomer Shiran about Dremio, the open source data as a service platform
    • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you start by explaining what Dremio is and how the project and business got started?
      • What was the motivation for keeping your primary product open source?
      • What is the governance model for the project?


      • How does Dremio fit in the current landscape of data tools?

        • What are some use cases that Dremio is uniquely equipped to support?
        • Do you think that Dremio obviates the need for a data warehouse or large scale data lake?


        • How is Dremio architected internally?

          • How has that architecture evolved from when it was first built?


          • There are a large array of components (e.g. governance, lineage, catalog) built into Dremio that are often found in dedicated products. What are some of the strategies that you have as a business and development team to manage and integrate the complexity of the product?

            • What are the benefits of integrating all of those capabilities into a single system?
            • What are the drawbacks?


            • One of the useful features of Dremio is the granular access controls. Can you discuss how those are implemented and controlled?

            • For someone who is interested in deploying Dremio to their environment what is involved in getting it installed?

              • What are the scaling factors?


              • What are some of the most exciting features that have been added in recent releases?

              • When is Dremio the wrong choice?

              • What have been some of the most challenging aspects of building, maintaining, and growing the technical and business platform of Dremio?

              • What do you have planned for the future of Dremio?

                • Contact Info
                • Tomer
                  • @tshiran on Twitter
                  • LinkedIn


                  • Dremio

                    • Website
                    • @dremio on Twitter
                    • dremio on GitHub


                      • Parting Question
                      • From your perspective, what is the biggest gap in the tooling or technology for data management today?
                        • Links
                        • Dremio
                        • MapR
                        • Presto
                        • Business Intelligence
                        • Arrow
                        • Tableau
                        • Power BI
                        • Jupyter
                        • OLAP Cube
                        • Apache Foundation
                        • Hadoop
                        • Nikon DSLR
                        • Spark
                        • ETL (Extract, Transform, Load)
                        • Parquet
                        • Avro
                        • K8s
                        • Helm
                        • Yarn
                        • Gandiva Initiative for Apache Arrow
                        • LLVM
                        • TLS

                          • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                          Support Data Engineering Podcast

                          ...more
                          View all episodesView all episodes
                          Download on the App Store
                          Data Engineering PodcastBy Tobias Macey
                          • 4.5
                          • 4.5
                          • 4.5
                          • 4.5
                          • 4.5

                          4.5

                          142 ratings

                          More shows like Data Engineering Podcast

                          View all
                          Software Engineering Radio by se-radio@computer.org

                          Software Engineering Radio

                          271 Listeners

                          The Changelog: Software Development, Open Source by Changelog Media

                          The Changelog: Software Development, Open Source

                          289 Listeners

                          Data Skeptic by Kyle Polich

                          Data Skeptic

                          479 Listeners

                          Software Engineering Daily by Software Engineering Daily

                          Software Engineering Daily

                          626 Listeners

                          Talk Python To Me by Michael Kennedy

                          Talk Python To Me

                          585 Listeners

                          Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                          Super Data Science: ML & AI Podcast with Jon Krohn

                          302 Listeners

                          CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer

                          CoRecursive: Coding Stories

                          190 Listeners

                          DataFramed by DataCamp

                          DataFramed

                          269 Listeners

                          Practical AI by Practical AI LLC

                          Practical AI

                          211 Listeners

                          AWS Podcast by Amazon Web Services

                          AWS Podcast

                          202 Listeners

                          The Stack Overflow Podcast by The Stack Overflow Podcast

                          The Stack Overflow Podcast

                          64 Listeners

                          The Real Python Podcast by Real Python

                          The Real Python Podcast

                          142 Listeners

                          Last Week in AI by Skynet Today

                          Last Week in AI

                          305 Listeners

                          This Day in AI Podcast by Michael Sharkey, Chris Sharkey

                          This Day in AI Podcast

                          227 Listeners

                          The Pragmatic Engineer by Gergely Orosz

                          The Pragmatic Engineer

                          64 Listeners