Data Engineering Podcast

Analytics Engineering Without The Friction Of Complex Pipeline Development With Optimus and dbt

Listen Later
Summary

One of the most impactful technologies for data analytics in recent years has been dbt. It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts. To help simplify the adoption and management of dbt projects Nandam Karthik helped create Optimus. In this episode he shares his experiences working with organizations to adopt analytics engineering patterns and the ways that Optimus and dbt were combined to let data analysts deliver insights without the roadblocks of complex pipeline management.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
  • Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. By the time errors have made their way into production, it’s often too late and damage is done. Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. No more shipping and praying, you can now know exactly what will change in your database! Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudder
  • Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
  • Your host is Tobias Macey and today I’m interviewing Nandam Karthik about his experiences building analytics projects with dbt and Optimus for his clients at Sigmoid.
    • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Sigmoid is and the types of projects that you are involved in?
      • What are some of the core challenges that your clients are facing when they start working with you?
      • An ELT workflow with dbt as the transformation utility has become a popular pattern for building analytics systems. Can you share some examples of projects that you have built with this approach?
        • What are some of the ways that this pattern becomes bespoke as you start exploring a project more deeply?
        • What are the sharp edges/white spaces that you encountered across those projects?
        • Can you describe what Optimus is?
          • How does Optimus improve the user experience of teams working in dbt?
          • What are some of the tactical/organizational practices that you have found most helpful when building with dbt and Optimus?
          • What are the most interesting, innovative, or unexpected ways that you have seen Optimus/dbt used?
          • What are the most interesting, unexpected, or challenging lessons that you have learned while working on dbt/Optimus projects?
          • When is Optimus/dbt the wrong choice?
          • What are your predictions for how "best practices" for analytics projects will change/evolve in the near/medium term?
            • Contact Info
            • LinkedIn
              • Parting Question
              • From your perspective, what is the biggest gap in the tooling or technology for data management today?
                • Closing Announcements
                • Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
                • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
                • If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
                • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
                  • Links
                  • Sigmoid
                  • Optimus
                  • dbt
                    • Podcast Episode
                    • Airflow
                    • AWS Glue
                    • BigQuery

                      • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                      Sponsored By:

                      • Datafold: ![Datafold](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/zm6x2tFu.png)
                      Datafold helps you deal with data quality in your pull request. It provides automated regression testing throughout your schema and pipelines so you can address quality issues before they affect production. No more shipping and praying, you can now know exactly what will change in your database ahead of time.
                      Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI, so in a few minutes you can get from 0 to automated testing of your analytical code. Visit our site at [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold)
                      today to book a demo with Datafold.

                      Support Data Engineering Podcast

                      ...more
                      View all episodesView all episodes
                      Download on the App Store
                      Data Engineering PodcastBy Tobias Macey
                      • 4.6
                      • 4.6
                      • 4.6
                      • 4.6
                      • 4.6

                      4.6

                      134 ratings

                      More shows like Data Engineering Podcast

                      View all
                      Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

                      Software Engineering Radio - the podcast for professional software developers

                      262 Listeners

                      The Changelog: Software Development, Open Source by Changelog Media

                      The Changelog: Software Development, Open Source

                      285 Listeners

                      The Cloudcast by Massive Studios

                      The Cloudcast

                      154 Listeners

                      Thoughtworks Technology Podcast by Thoughtworks

                      Thoughtworks Technology Podcast

                      42 Listeners

                      Data Skeptic by Kyle Polich

                      Data Skeptic

                      474 Listeners

                      Talk Python To Me by Michael Kennedy

                      Talk Python To Me

                      583 Listeners

                      Software Engineering Daily by Software Engineering Daily

                      Software Engineering Daily

                      629 Listeners

                      AWS Podcast by Amazon Web Services

                      AWS Podcast

                      200 Listeners

                      Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                      Super Data Science: ML & AI Podcast with Jon Krohn

                      294 Listeners

                      Python Bytes by Michael Kennedy and Brian Okken

                      Python Bytes

                      212 Listeners

                      DataFramed by DataCamp

                      DataFramed

                      269 Listeners

                      Practical AI by Practical AI LLC

                      Practical AI

                      196 Listeners

                      The Stack Overflow Podcast by The Stack Overflow Podcast

                      The Stack Overflow Podcast

                      63 Listeners

                      The Real Python Podcast by Real Python

                      The Real Python Podcast

                      137 Listeners

                      Latent Space: The AI Engineer Podcast by swyx + Alessio

                      Latent Space: The AI Engineer Podcast

                      64 Listeners