Data Engineering Podcast

Accelerate Development Of Enterprise Analytics With The Coalesce Visual Workflow Builder


Listen Later

Summary

The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. Coalesce is a platform designed to reduce repetitive work for common workflows by adopting a visual pipeline builder to support your data warehouse transformations. In this episode Satish Jayanthi explains how he is building a framework to allow enterprises to move quickly while maintaining guardrails for data workflows. This allows everyone in the business to participate in data analysis in a sustainable manner.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
  • Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
  • Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. By the time errors have made their way into production, it’s often too late and damage is done. Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. No more shipping and praying, you can now know exactly what will change in your database! Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.
  • Are you looking for a structured and battle-tested approach for learning data engineering? Would you like to know how you can build proper data infrastructures that are built to last? Would you like to have a seasoned industry expert guide you and answer all your questions? Join Pipeline Academy, the worlds first data engineering bootcamp. Learn in small groups with likeminded professionals for 9 weeks part-time to level up in your career. The course covers the most relevant and essential data and software engineering topics that enable you to start your journey as a professional data engineer or analytics engineer. Plus we have AMAs with world-class guest speakers every week! The next cohort starts in April 2022. Visit dataengineeringpodcast.com/academy and apply now!
  • Your host is Tobias Macey and today I’m interviewing Satish Jayanthi about how organizations can use data architectural patterns to stay competitive in today’s data-rich environment
  • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what you are building at Coalesce and the story behind it?
    • What are the core problems that you are focused on solving with Coalesce?
    • The platform appears to be fairly opinionated in the workflow. What are the design principles and philosophies that you have embedded into the user experience?
    • Can you describe how Coalesce is implemented?
    • What are the pitfalls in data architecture patterns that you commonly see organizations fall prey to?
      • How do the pre-built transformation templates in Coalesce help to guide users in a more maintainable direction?
      • The platform is currently tied to Snowflake as the underlying engine. How much effort will it be to expand your integrations and the scope of Coalesece?
      • What are the most interesting, innovative, or unexpected ways that you have seen Coalesce used?
      • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Coalesce?
      • When is Coalesce the wrong choice?
      • What do you have planned for the future of Coalesce?
      • Contact Info
        • LinkedIn
        • Parting Question
          • From your perspective, what is the biggest gap in the tooling or technology for data management today?
          • Closing Announcements
            • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
            • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
            • If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
            • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
            • Links
              • Coalesce
              • Data Warehouse Toolkit
              • Wherescape
              • dbt
                • Podcast Episode
                • Type 2 Dimensions
                • Firebase
                • Kubernetes
                • Star Schema
                • Data Vault
                  • Podcast Episode
                  • Data Mesh
                    • Podcast Episode
                    • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                      Support Data Engineering Podcast

                      ...more
                      View all episodesView all episodes
                      Download on the App Store

                      Data Engineering PodcastBy Tobias Macey

                      • 4.6
                      • 4.6
                      • 4.6
                      • 4.6
                      • 4.6

                      4.6

                      135 ratings


                      More shows like Data Engineering Podcast

                      View all
                      Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

                      Software Engineering Radio - the podcast for professional software developers

                      272 Listeners

                      The Changelog: Software Development, Open Source by Changelog Media

                      The Changelog: Software Development, Open Source

                      282 Listeners

                      The Cloudcast by Massive Studios

                      The Cloudcast

                      152 Listeners

                      Thoughtworks Technology Podcast by Thoughtworks

                      Thoughtworks Technology Podcast

                      42 Listeners

                      Data Skeptic by Kyle Polich

                      Data Skeptic

                      481 Listeners

                      Talk Python To Me by Michael Kennedy

                      Talk Python To Me

                      591 Listeners

                      Software Engineering Daily by Software Engineering Daily

                      Software Engineering Daily

                      627 Listeners

                      The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by Sam Charrington

                      The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

                      440 Listeners

                      Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                      Super Data Science: ML & AI Podcast with Jon Krohn

                      298 Listeners

                      Python Bytes by Michael Kennedy and Brian Okken

                      Python Bytes

                      213 Listeners

                      DataFramed by DataCamp

                      DataFramed

                      265 Listeners

                      Practical AI by Practical AI LLC

                      Practical AI

                      189 Listeners

                      The Stack Overflow Podcast by The Stack Overflow Podcast

                      The Stack Overflow Podcast

                      64 Listeners

                      The Real Python Podcast by Real Python

                      The Real Python Podcast

                      140 Listeners

                      Latent Space: The AI Engineer Podcast by swyx + Alessio

                      Latent Space: The AI Engineer Podcast

                      76 Listeners