Data Engineering Podcast

Astronomer with Ry Walker - Episode 6

Listen Later
Summary

Building a data pipeline that is reliable and flexible is a difficult task, especially when you have a small team. Astronomer is a platform that lets you skip straight to processing your valuable business data. Ry Walker, the CEO of Astronomer, explains how the company got started, how the platform works, and their commitment to open source.

Preamble
  • Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
  • When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at www.dataengineeringpodcast.com/linode?utm_source=rss&utm_medium=rss and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
  • You can help support the show by checking out the Patreon page which is linked from the site.
  • To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
  • This is your host Tobias Macey and today I’m interviewing Ry Walker, CEO of Astronomer, the platform for data engineering.
    • Interview
    • Introduction
    • How did you first get involved in the area of data management?
    • What is Astronomer and how did it get started?
    • Regulatory challenges of processing other people’s data
    • What does your data pipelining architecture look like?
    • What are the most challenging aspects of building a general purpose data management environment?
    • What are some of the most significant sources of technical debt in your platform?
    • Can you share some of the failures that you have encountered while architecting or building your platform and company and how you overcame them?
    • There are certain areas of the overall data engineering workflow that are well defined and have numerous tools to choose from. What are some of the unsolved problems in data management?
    • What are some of the most interesting or unexpected uses of your platform that you are aware of?
      • Contact Information
      • Email
      • @rywalker on Twitter
        • Links
        • Astronomer
        • Kiss Metrics
        • Segment
        • Marketing tools chart
        • Clickstream
        • HIPAA
        • FERPA
        • PCI
        • Mesos
        • Mesos DC/OS
        • Airflow
        • SSIS
        • Marathon
        • Prometheus
        • Grafana
        • Terraform
        • Kafka
        • Spark
        • ELK Stack
        • React
        • GraphQL
        • PostGreSQL
        • MongoDB
        • Ceph
        • Druid
        • Aries
        • Vault
        • Adapter Pattern
        • Docker
        • Kinesis
        • API Gateway
        • Kong
        • AWS Lambda
        • Flink
        • Redshift
        • NOAA
        • Informatica
        • SnapLogic
        • Meteor

          • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

          Support Data Engineering Podcast

          ...more
          View all episodesView all episodes
          Download on the App Store
          Data Engineering PodcastBy Tobias Macey
          • 4.5
          • 4.5
          • 4.5
          • 4.5
          • 4.5

          4.5

          142 ratings

          More shows like Data Engineering Podcast

          View all
          The Changelog: Software Development, Open Source by Changelog Media

          The Changelog: Software Development, Open Source

          289 Listeners

          Software Engineering Daily by Software Engineering Daily

          Software Engineering Daily

          624 Listeners

          Talk Python To Me by Michael Kennedy

          Talk Python To Me

          583 Listeners

          Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

          Super Data Science: ML & AI Podcast with Jon Krohn

          302 Listeners

          NVIDIA AI Podcast by NVIDIA

          NVIDIA AI Podcast

          343 Listeners

          Practical AI by Practical AI LLC

          Practical AI

          204 Listeners

          AWS Podcast by Amazon Web Services

          AWS Podcast

          205 Listeners

          Last Week in AI by Skynet Today

          Last Week in AI

          305 Listeners

          Dwarkesh Podcast by Dwarkesh Patel

          Dwarkesh Podcast

          523 Listeners

          The Data Engineering Show by The Firebolt Data Bros

          The Data Engineering Show

          8 Listeners

          No Priors: Artificial Intelligence | Technology | Startups by Conviction

          No Priors: Artificial Intelligence | Technology | Startups

          129 Listeners

          Latent Space: The AI Engineer Podcast by swyx + Alessio

          Latent Space: The AI Engineer Podcast

          92 Listeners

          This Day in AI Podcast by Michael Sharkey, Chris Sharkey

          This Day in AI Podcast

          227 Listeners

          The AI Daily Brief: Artificial Intelligence News and Analysis by Nathaniel Whittemore

          The AI Daily Brief: Artificial Intelligence News and Analysis

          633 Listeners

          AI + a16z by a16z

          AI + a16z

          36 Listeners