Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off.

Key Takeaways:

• 00:00 Introduction.
• 01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry.
• 03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run.
• 05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action.
• 06:58 Why XComs don't solve this problem: they get reinitialized on every retry.
• 09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403.
• 11:42 Subclassing the retry rule for more complex logic, including dynamic retry counts that used to require hacking the metadatabase.
• 15:52 Updates to asset partitions in 3.3: segment-based partitioning with fan-out and roll-up mappers for downstream DAGs.
• 21:42 Running tasks in Java and Go, moving Airflow toward a multi-language orchestrator.
• 23:33 DAG versioning improvement: choose whether a manual rerun uses the most recent DAG version or the original version from that run.
• 25:03 Advice for teams still on Airflow 2: use the upgrade ebook and Astro's AI migration tooling to handle the undifferentiated heavy lifting.

Resources Mentioned:

• Astronomer
• Airflow 3.3 Release Notes
• The Task State Store
• Updates to the asset partitions feature
• Retry policies
• Multi-language support
• 3.3 Webinar
• Upgrading from Airflow 2 to 3 ebook

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow