Legacy architecture and AI workloads pose unique challenges at scale, especially in a global enterprise with complex data systems. In this episode, we explore strategies to proactively monitor and optimize pipelines while minimizing downstream failures.

Adonis Castillo Cordero, Senior Automation Manager at Procter & Gamble, joins us to share actionable best practices for dependency mapping, anomaly detection and architecture simplification using Apache Airflow.

Key Takeaways:

(03:13) Integrating legacy data systems into modern architecture.

(05:51) Designing workflows for real-time data processing.

(07:57) Mapping dependencies early to avoid pipeline failures.

(09:02) Building automated monitoring into orchestration frameworks.

(12:09) Detecting anomalies to prevent performance bottlenecks.

(15:24) Monitoring data quality to catch silent failures.

(17:02) Prioritizing responses based on impact severity.

(18:55) Simplifying dashboards to highlight critical metrics.

Resources Mentioned:

Adonis Castillo Cordero

https://www.linkedin.com/in/adoniscc/

Procter & Gamble | LinkedIn

https://www.linkedin.com/company/procter-and-gamble/

Procter & Gamble | Website

http://www.pg.com

Apache Airflow

https://airflow.apache.org/

OpenLineage

https://openlineage.io/

Azure Monitor

https://azure.microsoft.com/en-us/products/monitor/

AWS Lookout for Metrics

https://aws.amazon.com/lookout-for-metrics/

Monte Carlo

https://www.montecarlodata.com/

Great Expectations

https://greatexpectations.io/

https://www.astronomer.io/events/roadshow/london/

https://www.astronomer.io/events/roadshow/new-york/

https://www.astronomer.io/events/roadshow/sydney/ 

https://www.astronomer.io/events/roadshow/san-francisco/ 

https://www.astronomer.io/events/roadshow/chicago/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning