Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge.

In this episode, we’re joined by Ethan Shalev, Data Engineer at Wix, to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development.

Key Takeaways:

00:00 Introduction.

02:13 Wix structures data engineering across multiple product-focused organizations.

03:40 Migrating nearly 8,000 DAGs to Airflow 3 requires careful planning.

04:31 Migration creates an opportunity to remove long-standing legacy Airflow code.

05:32 Internal playbooks and Cursor rules standardize and speed up DAG migrations.

07:39 Airflow 3 introduces backfills, DAG versioning and asset-aware scheduling.

09:16 Deferrable operators reduce scheduler congestion in large Airflow environments.

12:54 AI-generated code still requires review and strong testing practices.

14:52 Moving to managed Airflow reduces operational burden on internal platform teams.

15:57 Improving multi-tenancy and UI personalization remains a key Airflow need.

Resources Mentioned:

Ethan Shalev

https://www.linkedin.com/in/eshalev/

Wix | LinkedIn

https://www.linkedin.com/company/wix-com/

Wix | Website

https://www.wix.com/

Apache Airflow

https://airflow.apache.org/

Astronomer

https://www.astronomer.io/

Trino

https://trino.io/

Apache Iceberg

https://iceberg.apache.org/

Cursor

https://cursor.sh/

Airflow Summit

https://airflowsummit.org/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow