The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Using Airflow to orchestrate geospatial data pipelines unlocks powerful efficiencies for data teams. The combination of scalable processing and visual observability streamlines workflows, reduces costs and improves iteration speed.

In this episode, Alex Iannicelli, Staff Software Engineer at Overture Maps Foundation, and Daniel Smith, Senior Solutions Architect at Wherobots, join us to discuss leveraging Apache Airflow and Apache Sedona to process massive geospatial datasets, build reproducible pipelines and orchestrate complex workflows across platforms.

Key Takeaways:

00:00 Introduction.

03:22 How merging multiple data sources supports comprehensive datasets.

04:20 The value of flexible configurations for running pipelines on different platforms.

06:35 Why orchestration tools are essential for handling continuous data streams.

09:45 The importance of observability for monitoring progress and troubleshooting issues.

11:30 Strategies for processing large, complex datasets efficiently.

13:27 Expanding orchestration beyond core pipelines to automate frequent tasks.

17:02 Advantages of using open-source operators to simplify integration and deployment.

20:32 Desired improvements in orchestration tools for usability and workflow management.

Resources Mentioned:

Alex Iannicelli

https://www.linkedin.com/in/atiannicelli/

Overture Maps Foundation | LinkedIn

https://www.linkedin.com/company/overture-maps-foundation/

Overture Maps Foundation | Website

https://overturemaps.org

Daniel Smith

https://www.linkedin.com/in/daniel-smith-analyst/

Wherobots | LinkedIn

https://www.linkedin.com/company/wherobots

Wherobots | Website

https://www.wherobots.com

Apache Airflow

https://airflow.apache.org/

Apache Sedona

https://sedona.apache.org/

Github repo

https://github.com/wherobots/airflow-providers-wherobots

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

PepsiCo’s data platform drives insights across finance, marketing and data science. Delivering stability, scalability and developer delight is central to its success, and engineering leadership plays a key role in making this possible.

In this episode, Kunal Bhattacharya, Senior Manager of Data Platform Engineering at PepsiCo, shares how his team manages Airflow at scale while ensuring security, performance and cost efficiency.

Key Takeaways:

00:00 Introduction.

02:31 Enabling developer delight by extending platform capabilities.

03:56 Role of Snowflake, dbt and Airflow in PepsiCo’s data stack.

06:10 Local developer environments built using official Airflow Helm charts.

07:13 Pre-staging and PR environments as testing playgrounds.

08:08 Automating labeling and resource allocation via DAG factories.

12:16 Cost optimization through pod labeling and Datadog insights.

14:01 Isolating dbt engines to improve performance across teams.

16:12 Wishlist for Airflow 3: Improved role-based grants and database modeling.

Resources Mentioned:

Kunal Bhattacharya

https://www.linkedin.com/in/kunaljubce/

PepsiCo | LinkedIn

https://www.linkedin.com/company/pepsico/

PepsiCo | Website

https://www.pepsico.com

Apache Airflow

https://airflow.apache.org/

Snowflake

https://www.snowflake.com

dbt

https://www.getdbt.com

Kubernetes

https://kubernetes.io

Great Expectations

https://greatexpectations.io

Monte Carlo

https://www.montecarlodata.com

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.

In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.

Key Takeaways:

00:00 Introduction.

02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.

04:27 How Airflow became the central scheduler for batch jobs.

08:57 Credential management challenges that led to decoupling scheduling and orchestration.

12:21 Heimdall simplifies multi-application access through a unified interface.

13:15 Standardized operators in Airflow using Heimdall integration.

17:13 Open-source contributions and early adoption of Heimdall within Pattern.

21:01 Community support for Airflow and satisfaction with scheduling flexibility.

Resources Mentioned:

William Graham

https://www.linkedin.com/in/willgraham2/

Pattern | LinkedIn

https://www.linkedin.com/company/pattern-hq/

Pattern | Website

https://pattern.com

Apache Airflow

https://airflow.apache.org

Heimdall on GitHub

https://github.com/patterninc/heimdall

Netflix Genie

https://netflix.github.io/genie/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

The evolution of Airflow continues to shape data orchestration and monitoring strategies. Leveraging it beyond traditional ETL use cases opens powerful new possibilities for proactive support and internal operations.

In this episode, we are joined by Collin McNulty, Sr. Director of Global Support at Astronomer, who shares insights from his journey into data engineering and the lessons learned from leading Astronomer’s Customer Reliability Engineering (CRE) team.

Key Takeaways:

00:00 Introduction.

03:07 Lessons learned in adapting to major platform transitions.

05:18 How proactive monitoring improves reliability and customer experience.

08:10 Using automation to enhance internal support processes.

12:09 Why keeping systems current helps avoid unnecessary issues.

15:14 Approaches that strengthen system reliability and efficiency.

18:46 Best practices for simplifying complex orchestration dependencies.

23:24 Anticipated innovations that expand orchestration capabilities.

Resources Mentioned:

Collin McNulty

https://www.linkedin.com/in/collin-mcnulty/

Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/

Astronomer | Website

https://www.astronomer.io

Apache Airflow

https://airflow.apache.org/

Prometheus

https://prometheus.io/

Splunk

https://www.splunk.com/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

The shift to a unified data platform is reshaping how pharmaceutical companies manage and orchestrate data. Establishing standards across regions and teams ensures scalability and efficiency in handling large-scale analytics.

In this episode, Evgenii Prusov, Senior Data Platform Engineer of Daiichi Sankyo Europe GmbH, joins us to discuss building and scaling a centralized data platform with Airflow and Astronomer.

Key Takeaways:

00:00 Introduction.

02:49 Building a centralized data platform for 15 European countries.

05:19 Adopting SaaS to manage Airflow from day one.

07:01 Leveraging Airflow for data orchestration across products.

08:16 Teaching non-Python users how to work with Airflow is challenging.

12:25 Creating a global data community across Europe, the US and Japan.

14:04 Monthly calls help share knowledge and align regional teams.

15:47 Contributing to the open-source Airflow project as a way to deepen expertise.

16:32 Desire for more guidelines, debugging tutorials and testing best practices in Airflow.

Resources Mentioned: 

Evgenii Prusov

https://www.linkedin.com/in/prusov/

Daiichi Sankyo Europe GmbH | LinkedIn

https://www.linkedin.com/company/daiichi-sankyo-europe-gmbh/

Daiichi Sankyo Europe GmbH | Website

https://www.daiichi-sankyo.eu

Apache Airflow

https://airflow.apache.org/

Astronomer

https://www.astronomer.io/

Snowflake

https://www.snowflake.com/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

StyleSeat is revolutionizing how beauty and wellness professionals grow their businesses through data-driven tools. From streamlining scheduling to optimizing marketing, their platform empowers professionals to focus on their craft while expanding their client base.

In this episode, Paschal Onuorah, Senior Data Engineer at StyleSeat, shares how the company leverages Airflow, dbt, and Cosmos to drive marketplace intelligence, improve client connections and deliver measurable growth for professionals.

Key Takeaways:

00:00 Introduction.

05:44 The role of the data engineering team in driving business success.

08:52 Leveraging technology for real-time business intelligence.

10:52 Data-driven strategies for improving marketing outcomes.

13:05 How adopting the right tools can increase revenue growth.

14:25 Advantages of simplifying and integrating technical workflows.

18:45 Benefits of multi-environment configurations for development and production.

20:17 Foundational skills and best practices for learning Airflow effectively.

22:33 Opportunities for deeper tool integration and improved data visualization.

Resources Mentioned:

Paschal Onuorah

https://www.linkedin.com/in/onuorah-paschal/

StyleSeat | LinkedIn

https://www.linkedin.com/company/styleseat/

StyleSeat | Website

https://www.styleseat.com

Apache Airflow

https://airflow.apache.org/

dbt

https://www.getdbt.com/

Astronomer Cosmos

https://www.astronomer.io/cosmos/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

The evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to enabling remote execution, these advancements reshape how organizations manage data pipelines.

In this episode, we’re joined by Ian Buss, Principal Software Engineer at Astronomer, and Piotr Chomiak, Principal Product Manager at Astronomer, who share insights into the Astro Executor and remote execution.

Key Takeaways:

00:00 Introduction.

04:13 How product leadership drives scalability for enterprise needs.

08:23 Architectural changes that improve reliability and remove bottlenecks.

10:15 Metrics that enhance visibility into system performance.

12:54 The role of remote execution in addressing security requirements.

15:56 Differences between open-source solutions and managed offerings.

19:04 Broad industry adoption and applicability of remote execution.

20:39 Future advancements in language support and multi-tenancy.

Resources Mentioned:

Ian Buss

https://www.linkedin.com/in/ian-buss/

Piotr Chomiak

https://www.linkedin.com/in/piotr-chomiak-b1955624/

Astronomer | Website

https://www.astronomer.io

Apache Airflow

https://airflow.apache.org/

Airflow Slack Community

https://airflow.apache.org/community/

Beyond Analytics conference

https://astronomer.io/beyond/dataflowcast

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Scaling 2,000+ data pipelines isn’t easy. But with the right tools and a self-hosted mindset, it becomes achievable.

In this episode, Sébastien Crocquevieille, Data Engineer at Numberly, unpacks how the team scaled their on-prem Airflow setup using open-source tooling and Kubernetes. We explore orchestration strategies, UI-driven stakeholder access and Airflow’s evolving features.

Key Takeaways:

00:00 Introduction.

02:13 Overview of the company’s operations and global presence.

04:00 The tech stack and structure of the data engineering team.

04:24 Running nearly 2,000 DAGs in production using Airflow.

05:42 How Airflow’s UI empowers stakeholders to self-serve and troubleshoot.

07:05 Details on the Kubernetes-based Airflow setup using Helm charts.

09:31 Transition from GitSync to NFS for DAG syncing due to performance issues.

14:11 Making every team member Airflow-literate through local installation.

17:56 Using custom libraries and plugins to extend Airflow functionality.

Resources Mentioned:

Sébastien Crocquevieille

https://www.linkedin.com/in/scroc/

Numberly | LinkedIn

https://www.linkedin.com/company/numberly/

Numberly | Website

https://numberly.com/

Apache Airflow

https://airflow.apache.org/

Grafana

https://grafana.com/

Apache Kafka

https://kafka.apache.org/

Helm Chart for Apache Airflow

https://airflow.apache.org/docs/helm-chart/stable/index.html

Kubernetes

https://kubernetes.io/

GitLab

https://about.gitlab.com/

KubernetesPodOperator – Airflow

https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html

Beyond Analytics Conference

https://astronomer.io/beyond/dataflowcast

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Managing financial data at scale requires precise orchestration and proactive monitoring to maintain operational efficiency.

In this episode, we are joined by Adeolu Adegboye, Data Engineer at Moniepoint Group, who shares how his team uses data pipelines and workflow automation to manage high volumes of transactions, ensure timely alerts and support diverse stakeholders across the business.

Key Takeaways:

(00:00) Introduction. 

(02:48) The role of data engineering in supporting all business operations.

(04:17) Leveraging workflow orchestration to manage daily processes.

(05:20) Proactively monitoring for anomalies to prevent potential issues.

(08:12) Simplifying complex insights for non-technical teams.

(13:01) Improving efficiency through dynamic and parallel workflows.

(14:19) Optimizing system performance to handle large-scale operations.

(17:19) Exploring creative and innovative uses for workflow automation.

Resources Mentioned:

Adeolu Adegboye

https://www.linkedin.com/in/adeolu-adegboye/

Moniepoint Group | LinkedIn

https://www.linkedin.com/company/moniepoint-inc/

Moniepoint Group | Website

https://www.moniepoint.com

Apache Airflow

https://airflow.apache.org/

ClickHouse

https://clickhouse.com/

Grafana

https://grafana.com/

Beyond Analytics Conference

https://astronomer.io/beyond/dataflowcast

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

The evolution of Airflow has reached a milestone with the introduction of remote execution in Airflow 3, enabling flexible orchestration across distributed environments.

In this episode, Jens Scheffler, Test Execution Cluster Technical Architect at Bosch, shares insights on how his team’s need for large-scale, cross-environment testing influenced the development of the Edge Executor and shaped this major release.

Key Takeaways:

(02:39) The role of remote execution in supporting large-scale testing needs.

(04:44) How community support contributed to the Edge Executor’s development.

(08:41) Navigating network and infrastructure limitations within secure environments.

(13:25) Transitioning from database-heavy processes to an API-driven model.

(14:16) How the new task SDK in Airflow 3 improves distributed task execution.

(16:54) What is required to set up and configure the Edge Executor.

(19:36) Managing multiple queues to optimize tasks across different environments.

(23:30) Examples of extreme distance use cases for edge execution.

Resources Mentioned:

Jens Scheffler

https://www.linkedin.com/in/jens-scheffler/

Bosch | LinkedIn

https://www.linkedin.com/company/bosch/

Bosch | Website

https://www.bosch.com/

Apache Airflow

https://airflow.apache.org/

Edge Executor (Edge3 Provider Package)

https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html

Astronomer’s Astro Executor

https://www.astronomer.io/docs/astro/astro-executor/

Beyond Analytics Conference

https://astronomer.io/beyond/dataflowcast

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning