Data Engineering Podcast

Astronomer with Ry Walker - Episode 6

Listen Later
Summary

Building a data pipeline that is reliable and flexible is a difficult task, especially when you have a small team. Astronomer is a platform that lets you skip straight to processing your valuable business data. Ry Walker, the CEO of Astronomer, explains how the company got started, how the platform works, and their commitment to open source.

Preamble
  • Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
  • When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at www.dataengineeringpodcast.com/linode?utm_source=rss&utm_medium=rss and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
  • You can help support the show by checking out the Patreon page which is linked from the site.
  • To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
  • This is your host Tobias Macey and today I’m interviewing Ry Walker, CEO of Astronomer, the platform for data engineering.
    • Interview
    • Introduction
    • How did you first get involved in the area of data management?
    • What is Astronomer and how did it get started?
    • Regulatory challenges of processing other people’s data
    • What does your data pipelining architecture look like?
    • What are the most challenging aspects of building a general purpose data management environment?
    • What are some of the most significant sources of technical debt in your platform?
    • Can you share some of the failures that you have encountered while architecting or building your platform and company and how you overcame them?
    • There are certain areas of the overall data engineering workflow that are well defined and have numerous tools to choose from. What are some of the unsolved problems in data management?
    • What are some of the most interesting or unexpected uses of your platform that you are aware of?
      • Contact Information
      • Email
      • @rywalker on Twitter
        • Links
        • Astronomer
        • Kiss Metrics
        • Segment
        • Marketing tools chart
        • Clickstream
        • HIPAA
        • FERPA
        • PCI
        • Mesos
        • Mesos DC/OS
        • Airflow
        • SSIS
        • Marathon
        • Prometheus
        • Grafana
        • Terraform
        • Kafka
        • Spark
        • ELK Stack
        • React
        • GraphQL
        • PostGreSQL
        • MongoDB
        • Ceph
        • Druid
        • Aries
        • Vault
        • Adapter Pattern
        • Docker
        • Kinesis
        • API Gateway
        • Kong
        • AWS Lambda
        • Flink
        • Redshift
        • NOAA
        • Informatica
        • SnapLogic
        • Meteor

          • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

          Support Data Engineering Podcast

          ...more
          View all episodesView all episodes
          Download on the App Store
          Data Engineering PodcastBy Tobias Macey
          • 4.6
          • 4.6
          • 4.6
          • 4.6
          • 4.6

          4.6

          135 ratings

          More shows like Data Engineering Podcast

          View all
          Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

          Software Engineering Radio - the podcast for professional software developers

          272 Listeners

          The Changelog: Software Development, Open Source by Changelog Media

          The Changelog: Software Development, Open Source

          283 Listeners

          The Cloudcast by Massive Studios

          The Cloudcast

          153 Listeners

          Thoughtworks Technology Podcast by Thoughtworks

          Thoughtworks Technology Podcast

          41 Listeners

          Data Skeptic by Kyle Polich

          Data Skeptic

          483 Listeners

          Talk Python To Me by Michael Kennedy

          Talk Python To Me

          592 Listeners

          Software Engineering Daily by Software Engineering Daily

          Software Engineering Daily

          624 Listeners

          The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by Sam Charrington

          The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

          444 Listeners

          Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

          Super Data Science: ML & AI Podcast with Jon Krohn

          298 Listeners

          Python Bytes by Michael Kennedy and Brian Okken

          Python Bytes

          213 Listeners

          DataFramed by DataCamp

          DataFramed

          266 Listeners

          Practical AI by Practical AI LLC

          Practical AI

          190 Listeners

          The Stack Overflow Podcast by The Stack Overflow Podcast

          The Stack Overflow Podcast

          64 Listeners

          The Real Python Podcast by Real Python

          The Real Python Podcast

          140 Listeners

          Latent Space: The AI Engineer Podcast by swyx + Alessio

          Latent Space: The AI Engineer Podcast

          77 Listeners