In this episode, we explore how teams scale Apache Airflow in complex environments and what it takes to make orchestration work across many stakeholders. We look at real-world challenges around visibility, ownership and predictability as data platforms grow.

Egor Tarasenko, Data and AI Engineer at Ponder Labs, joins us to share how Ponder Labs customizes Airflow for education organizations using plugins, event-driven architectures and AI-powered tooling. He explains how his team supports large charter school networks and why structure, consistency and extensibility become critical at scale.

Key Takeaways:

00:00 Introduction.

01:21 Ponder Labs helps education organizations bring data from many systems together so it becomes useful for teachers, school leaders and administrators.

03:10 Airflow serves as the backbone for orchestrating ingestion, transformation and reverse ETL across client data platforms.

05:43 Everything is triggered from Airflow to maintain dependency, visibility and a single operational picture.

09:05 Managing hundreds of DAGs requires a focus on structure, visibility and consistency across teams.

09:51 Treating DAGs like APIs helps teams scale without needing deep knowledge of upstream logic.

12:00 Custom plugins like schedule insights help predict DAG run times across layered dependencies.

15:00 AI-powered Airflow chat enables non-technical stakeholders to understand DAG ownership dependencies and cluster activity.

22:06 Migrating plugins to Airflow 3 improves developer experience through cleaner APIs and faster extensibility.

Resources Mentioned:

Egor Tarasenko

https://www.linkedin.com/in/egorseno/

Apache Airflow

https://airflow.apache.org

dbt

https://www.getdbt.com

Astronomer Astro Platform

https://www.astronomer.io

Egor Tarasenko on Substack 

https://egortarasenko.substack.com

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow