Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution.

Key Takeaways:

• 00:00 Introduction.
• 01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement.
• 02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience.
• 02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle.
• 04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC.
• 05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode.
• 08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly.
• 09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster.
• 11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way.
• 12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production.
• 14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording.
• 15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution.

Resources Mentioned:

• [Ontario Teachers' Pension Plan](otpp.com)
• [Apache Airflow](airflow.apache.org)
• [Astronomer](astronomer.io)
• [Cosmos](astronomer.io/cosmos)

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow