DataTalks.Club

Data Observability - Barr Moses

Listen Later

We covered:

  • Barr’s background
  • Market gaps in data reliability
  • Observability in engineering
  • Data downtime
  • Data quality problems and the five pillars of data observability
  • Example: job failing because of a schema change
  • Three pillars of observability (good pipelines and bad data)
  • Observability vs monitoring
  • Finding the root cause
  • Who is accountable for data quality? (the RACI framework)
  • Service level agreements
  • Inferring the SLAs from the historical data
  • Implementing data observability
  • Data downtime maturity curve
  • Monte carlo: data observability solution
  • Open source tools
  • Test-driven development for data
  • Is data observability cloud agnostic?
  • Centralizing data observability
  • Detecting downstream and upstream data usage
  • Getting bad data vs getting unusual data


    • Links:

    • Learn more about Monte Carlo: https://www.montecarlodata.com/
    • The Data Engineer's Guide to Root Cause Analysis: https://www.montecarlodata.com/the-data-engineers-guide-to-root-cause-analysis/
    • Why You Need to Set SLAs for Your Data Pipelines: https://www.montecarlodata.com/how-to-make-your-data-pipelines-more-reliable-with-slas/
    • Data Observability: The Next Frontier of Data Engineering: https://www.montecarlodata.com/data-observability-the-next-frontier-of-data-engineering/
    • To get in touch with Barr, ping her in the DataTalks.Club group or use [email protected]


      • Join DataTalks.Club: https://datatalks.club/slack.html

      ...more
      View all episodesView all episodes
      Download on the App Store
      DataTalks.ClubBy DataTalks.Club
      • 5
      • 5
      • 5
      • 5
      • 5

      5

      7 ratings

      More shows like DataTalks.Club

      View all
      Radiolab by WNYC Studios

      Radiolab

      43,981 Listeners

      Hidden Brain by Hidden Brain, Shankar Vedantam

      Hidden Brain

      43,720 Listeners

      The Knowledge Project by Shane Parrish

      The Knowledge Project

      2,672 Listeners

      Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

      Super Data Science: ML & AI Podcast with Jon Krohn

      304 Listeners

      Data Engineering Podcast by Tobias Macey

      Data Engineering Podcast

      146 Listeners

      The Real Python Podcast by Real Python

      The Real Python Podcast

      141 Listeners

      Huberman Lab by Scicomm Media

      Huberman Lab

      29,222 Listeners

      The Ezra Klein Show by New York Times Opinion

      The Ezra Klein Show

      16,042 Listeners

      ReThinking by TED

      ReThinking

      618 Listeners

      Data Career Podcast: Helping You Land a Data Analyst Job FAST by Avery Smith - Data Career Coach

      Data Career Podcast: Helping You Land a Data Analyst Job FAST

      161 Listeners

      The Analytics Engineering Podcast by dbt Labs, Inc.

      The Analytics Engineering Podcast

      28 Listeners

      The Tucker Carlson Show by Tucker Carlson Network

      The Tucker Carlson Show

      16,880 Listeners