In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate decisions. William Orgertrice III, Data Engineer at Cargill, joins us to share the DAG design and monitoring practices he presented at Airflow Summit 2025 and how his team is rolling out Airflow across 60+ internal teams as part of Cargill's new Minerva data platform.

Key Takeaways:

00:00 Introduction.

01:45 Cargill is one of the largest privately owned companies in the US, operating across 70 countries and serving 125+ markets.

03:45 William's team on the Cargill Data Platform supports 60+ internal teams, providing data products that drive decisions across finance, inventory and operations.

05:10 Cargill chose Airflow as a core component of its new Minerva data platform to replace older ETL tooling with a more supportable, observable stack.

06:26 Native SLA sensors and dependency management were specific features that made Airflow the right fit for Cargill's batch ingestion pipelines.

09:00 Cargill is running Airflow through Astronomer as their managed solution, with some teams already in production.

13:22 Every task in a DAG should have a single, documented purpose — one task doing everything makes troubleshooting significantly harder.

14:40 A DAG that never enters a failed state but keeps running indefinitely will spend compute budget without alerting anyone.

15:25 In shared Airflow environments, embedding contact information and owner tags in DAGs ensures the right team is reached when something breaks upstream.

21:00 William flags connection testing as a friction point in pipeline development — verifying a connection string before building the full job would reduce iteration time.

Resources Mentioned:

Cargill | Website

https://www.cargill.com/food-beverage

Airflow Community on Slack 

https://airflow.apache.org/community/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow