
Sign up to save your podcasts
Or
Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.
Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems.
Key Takeaways:
.
(03:14) Supporting data privacy and engineering efficiency within data systems.
(10:41) Validating outputs with reconciliation checks to catch transformation issues.
(16:06) Applying standardized patterns for auditing, validating and publishing data.
(19:28) Capturing historical check results to monitor system health and improvements.
(21:29) Treating data quality and availability as separate monitoring concerns.
(26:26) Using containerization strategies to streamline pipeline executions.
(29:47) Leveraging orchestration platforms for better visibility and retry capability.
(31:59) Managing business pressure without sacrificing data quality practices.
(35:46) Starting simple with quality checks and evolving toward more complex frameworks.
Resources Mentioned:
Joseph Machado
https://www.linkedin.com/in/josephmachado1991/
Netflix | LinkedIn
https://www.linkedin.com/company/netflix/
Netflix | Website
https://www.netflix.com/browse
Start Data Engineering
https://www.startdataengineering.com/
Apache Airflow
https://airflow.apache.org/
dbt Labs
https://www.getdbt.com/
Great Expectations
https://greatexpectations.io/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/
https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
5
2020 ratings
Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.
Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems.
Key Takeaways:
.
(03:14) Supporting data privacy and engineering efficiency within data systems.
(10:41) Validating outputs with reconciliation checks to catch transformation issues.
(16:06) Applying standardized patterns for auditing, validating and publishing data.
(19:28) Capturing historical check results to monitor system health and improvements.
(21:29) Treating data quality and availability as separate monitoring concerns.
(26:26) Using containerization strategies to streamline pipeline executions.
(29:47) Leveraging orchestration platforms for better visibility and retry capability.
(31:59) Managing business pressure without sacrificing data quality practices.
(35:46) Starting simple with quality checks and evolving toward more complex frameworks.
Resources Mentioned:
Joseph Machado
https://www.linkedin.com/in/josephmachado1991/
Netflix | LinkedIn
https://www.linkedin.com/company/netflix/
Netflix | Website
https://www.netflix.com/browse
Start Data Engineering
https://www.startdataengineering.com/
Apache Airflow
https://airflow.apache.org/
dbt Labs
https://www.getdbt.com/
Great Expectations
https://greatexpectations.io/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/
https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
481 Listeners
38 Listeners
142 Listeners
265 Listeners
140 Listeners
289 Listeners
8,909 Listeners
2,146 Listeners
12 Listeners
8 Listeners
8 Listeners
15 Listeners
450 Listeners