Data Engineering Podcast

Presto Powered Cloud Data Lakes At Speed Made Easy With Ahana

Listen Later
Summary

The Presto project has become the de facto option for building scalable open source analytics in SQL for the data lake. In recent months the community has focused their efforts on making it the fastest possible option for running your analytics in the cloud. In this episode Dipti Borkar discusses the work that she and her team are doing at Ahana to simplify the work of running your own PrestoDB environment in the cloud. She explains how they are optimizin the runtime to reduce latency and increase query throughput, the ways that they are contributing back to the open source community, and the exciting improvements that are in the works to make Presto an even more powerful option for all of your analytics.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
  • Schema changes, missing data, and volume anomalies caused by your data sources can happen without any advanced notice if you lack visibility into your data-in-motion. That leaves DataOps reactive to data quality issues and can make your consumers lose confidence in your data. By connecting to your pipeline orchestrator like Apache Airflow and centralizing your end-to-end metadata, Databand.ai lets you identify data quality issues and their root causes from a single dashboard. With Databand.ai, you’ll know whether the data moving from your sources to your warehouse will be available, accurate, and usable when it arrives. Go to dataengineeringpodcast.com/databand to sign up for a free 30-day trial of Databand.ai and take control of your data quality today.
  • Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
  • Your host is Tobias Macey and today I’m interviewing Dipti Borkar, cofounder Ahana about Presto and Ahana, SaaS managed service for Presto
    • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Ahana is and the story behind it?
    • There has been a lot of recent activity in the Presto community. Can you give an overview of the options that are available for someone wanting to use its SQL engine for querying their data?
      • What is Ahana’s role in the community/ecosystem?
      • (happy to skip this question if it’s too contentious) What are some of the notable differences that have emerged over the past couple of years between the Trino (formerly PrestoSQL) and PrestoDB projects?
      • Another area that has been seeing a lot of activity is data lakes and projects to make them more manageable and feature complete (e.g. Hudi, Delta Lake, Iceberg, Nessie, LakeFS, etc.). How has that influenced your product focus and capabilities?
        • How does this activity change the calculus for organizations who are deciding on a lake or warehouse for their data architecture?
        • Can you describe how the Ahana Cloud platform is architected?
          • What are the additional systems that you have built to manage deployment, scaling, and multi-tenancy?
          • Beyond the storage and processing, what are the other notable tools and projects that have become part of the overall stack for supporting open analytics?
          • What are some areas of ongoing activity that you are keeping an eye on as you build out the Ahana offerings?
          • What are the most interesting, innovative, or unexpected ways that you have seen Ahana/Presto used?
          • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Ahana?
          • When is Ahana the wrong choice?
          • What do you have planned for the future of Ahana?
            • Contact Info
            • LinkedIn
            • @dborkar on Twitter
              • Parting Question
              • From your perspective, what is the biggest gap in the tooling or technology for data management today?
                • Links
                • Ahana
                • Alluxio
                  • Podcast Episode
                  • Couchbase
                  • Kinetica
                  • Tensorflow
                  • PyTorch
                    • Podcast.__init__ Episode
                    • AWS Athena
                    • AWS Glue
                    • Hive Metastore
                    • Clickhouse
                    • Dremio
                      • Podcast Episode
                      • Apache Drill
                      • Teradata
                      • Snowflake
                        • Podcast Episode
                        • BigQuery
                        • RaptorX
                        • Aria Optimizations for Presto
                        • Apache Ranger
                          • Presto Plugin
                          • Trino
                            • Podcast Episode
                            • Starburst
                              • Podcast Episode
                              • Hive
                              • Iceberg
                                • Podcast Episode
                                • Hudi
                                  • Podcast Episode
                                  • Delta Lake
                                    • Podcast Episode
                                    • Superset
                                      • Podcast.__init__ Episode
                                      • Data Engineering Podcast Episode
                                      • Nessie
                                      • LakeFS
                                      • Amundsen
                                        • Podcast Episode
                                        • DataHub
                                          • Podcast Episode
                                          • OtterTune
                                            • Podcast Episode

                                              • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                                              Support Data Engineering Podcast

                                              ...more
                                              View all episodesView all episodes
                                              Download on the App Store
                                              Data Engineering PodcastBy Tobias Macey
                                              • 4.5
                                              • 4.5
                                              • 4.5
                                              • 4.5
                                              • 4.5

                                              4.5

                                              142 ratings

                                              More shows like Data Engineering Podcast

                                              View all
                                              The Changelog: Software Development, Open Source by Changelog Media

                                              The Changelog: Software Development, Open Source

                                              289 Listeners

                                              Software Engineering Daily by Software Engineering Daily

                                              Software Engineering Daily

                                              623 Listeners

                                              Talk Python To Me by Michael Kennedy

                                              Talk Python To Me

                                              583 Listeners

                                              Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                                              Super Data Science: ML & AI Podcast with Jon Krohn

                                              302 Listeners

                                              NVIDIA AI Podcast by NVIDIA

                                              NVIDIA AI Podcast

                                              334 Listeners

                                              Practical AI by Practical AI LLC

                                              Practical AI

                                              203 Listeners

                                              AWS Podcast by Amazon Web Services

                                              AWS Podcast

                                              205 Listeners

                                              Last Week in AI by Skynet Today

                                              Last Week in AI

                                              305 Listeners

                                              Dwarkesh Podcast by Dwarkesh Patel

                                              Dwarkesh Podcast

                                              517 Listeners

                                              The Data Engineering Show by The Firebolt Data Bros

                                              The Data Engineering Show

                                              8 Listeners

                                              No Priors: Artificial Intelligence | Technology | Startups by Conviction

                                              No Priors: Artificial Intelligence | Technology | Startups

                                              130 Listeners

                                              Latent Space: The AI Engineer Podcast by swyx + Alessio

                                              Latent Space: The AI Engineer Podcast

                                              92 Listeners

                                              This Day in AI Podcast by Michael Sharkey, Chris Sharkey

                                              This Day in AI Podcast

                                              228 Listeners

                                              The AI Daily Brief: Artificial Intelligence News and Analysis by Nathaniel Whittemore

                                              The AI Daily Brief: Artificial Intelligence News and Analysis

                                              631 Listeners

                                              AI + a16z by a16z

                                              AI + a16z

                                              36 Listeners