Data failures are inevitable but how you manage them can define the success of your operations. In this episode, we dive deep into the challenges of data engineering and AI with Brendan Frick, Senior Engineering Manager, Data at GumGum. Brendan shares his unique approach to managing task failures and DAG issues in a high-stakes ad-tech environment.

Brendan discusses how GumGum leverages Apache Airflow to streamline data processes, ensuring efficient data movement and orchestration while minimizing disruptions in their operations.

Key Takeaways:

(02:02) Brendan’s role at GumGum and its approach to ad tech.

(04:27) How GumGum uses Airflow for daily data orchestration, moving data from S3 to warehouses.

(07:02) Handling task failures in Airflow using Jira for actionable, developer-friendly responses.

(09:13) Transitioning from email alerts to a more structured system with Jira and PagerDuty.

(11:40) Monitoring task retry rates as a key metric to identify potential issues early.

(14:15) Utilizing Looker dashboards to track and analyze task performance and retry rates.

(16:39) Transitioning from Kubernetes operator to a more reliable system for data processing.

(19:25) The importance of automating stakeholder communication with data lineage tools like Atlan.

(20:48) Implementing data contracts to ensure SLAs are met across all data processes.

(22:01) The role of scalable SLAs in Airflow to ensure data reliability and meet business needs.

Resources Mentioned:

Brendan Frick -

https://www.linkedin.com/in/brendan-frick-399345107/

GumGum -

https://www.linkedin.com/company/gumgum/

Apache Airflow -

https://airflow.apache.org/

Jira -

https://www.atlassian.com/software/jira

Atlan -

https://atlan.com/

Kubernetes -

https://kubernetes.io/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning