The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

A 13% reduction in failure rates — this is how two data scientists at Astronomer revolutionized their data pipelines using Apache Airflow.

In this episode, we enter the world of data orchestration and AI with Maggie Stark and Marion Azoulai, both Senior Data Scientists at Astronomer. Maggie and Marion discuss how their team re-architected their use of Airflow to improve scalability, reliability and efficiency in data processing. They share insights on overcoming challenges with sensors and how moving to datasets transformed their workflows.

Key Takeaways:

(02:23) The data team’s role as a centralized hub within Astronomer.

(05:11) Airflow is the backbone of all data processes, running 60,000 tasks daily.

(07:13) Custom task groups enable efficient code reuse and adherence to best practices.

(11:33) Sensor-heavy architectures can lead to cascading failures and resource issues.

(12:09) Switching to datasets has improved reliability and scalability.

(14:19) Building a control DAG provides end-to-end visibility of pipelines.

(16:42) Breaking down DAGs into smaller units minimizes failures and improves management.

(19:02) Failure rates improved from 16% to 3% with the new architecture.

Resources Mentioned:

Maggie Stark -

https://www.linkedin.com/in/margaretstark/

Marion Azoulai -

https://www.linkedin.com/in/marionazoulai/

Astronomer | LinkedIn -

https://www.linkedin.com/company/astronomer/

Apache Airflow -

https://airflow.apache.org/

Astronomer | Website -

https://www.astronomer.io/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Mastering the flow of data is essential for driving innovation and efficiency in today’s competitive landscape. In this episode, we explore the evolution of data orchestration and the pivotal role of Apache Airflow in modern data workflows.

Ben Tallman, Chief Technology Officer at M Science, joins us and shares his extensive experience with Airflow, detailing its early adoption, evolution and the profound impact it has had on data engineering practices. His insights reveal how leveraging Airflow can streamline complex data processes, enhance observability and ultimately drive business success.

Key Takeaways:

(02:31) Benjamin’s journey with Airflow and its early adoption.

(05:36) The transition from legacy schedulers to Airflow at Apigee and later Google.

(08:52) The challenges and benefits of running production-grade Airflow instances.

(10:46) How Airflow facilitates the management of large-scale data at M Science.

(11:56) The importance of reducing time to value for customers using data products.

(13:32) Airflow’s role in ensuring observability and reliability in data workflows.

(17:00) Managing petabytes of data and billions of records efficiently.

(19:08) Integration of various data sources and ensuring data product quality.

(20:04) Leveraging Airflow for data observability and reducing time to value.

(22:04) Benjamin’s vision for the future development of Airflow, including audit trails for variables.

Resources Mentioned:

Ben Tallman -

https://www.linkedin.com/in/btallman/

M Science -

https://www.linkedin.com/company/m-science-llc/

Apache Airflow -

https://airflow.apache.org/

Astronomer -

https://www.astronomer.io/

Databricks -

https://databricks.com/

Snowflake -

https://www.snowflake.com/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Welcome to The Data Flowcast: Mastering Airflow for Data Engineering & AI — the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward.

Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems.

#AI #Automation #Airflow #MachineLearning

Data orchestration is revolutionizing the way companies manage and process data. In this episode, we explore the critical role of data orchestration in modern data workflows and how Apache Airflow is used to enhance data processing and AI model deployment.

Hannan Kravitz, Data Engineering Team Leader at Artlist, joins us to share his insights on leveraging Airflow for data engineering and its impact on their business operations.

Key Takeaways:

(01:00) Hannan introduces Artlist and its mission to empower content creators.

(04:27) The importance of collecting and modeling data to support business insights.

(06:40) Using Airflow to connect multiple data sources and create dashboards.

(09:40) Implementing a monitoring DAG for proactive alerts within Airflow​​.

(12:31) Customizing Airflow for business metric KPI monitoring and setting thresholds​​.

(15:00) Addressing decreases in purchases due to technical issues with proactive alerts​​.

(17:45) Customizing data quality checks with dynamic task mapping in Airflow​​.

(20:00) Desired improvements in Airflow UI and logging capabilities​​.

(21:00) Enabling business stakeholders to change thresholds using Streamlit​​.

(22:26) Future improvements desired in the Airflow project​.

Resources Mentioned:

Hannan Kravitz -

https://www.linkedin.com/in/hannan-kravitz-60563112/

Artlist -

https://www.linkedin.com/company/art-list/

Apache Airflow -

https://airflow.apache.org/

Snowflake -

https://www.snowflake.com/

Streamlit -

https://streamlit.io/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Data engineering is constantly evolving and staying ahead means mastering tools like Apache Airflow. In this episode, we explore the world of data engineering with Alexandre Magno Lima Martins, Senior Data Engineer at Teya. Alexandre talks about optimizing data workflows and the smart solutions they've created at Teya to make data processing easier and more efficient.

Key Takeaways:

(02:01) Alexandre explains his role at Teya and the responsibilities of a data platform engineer.

(02:40) The primary use cases of Airflow at Teya, especially with dbt and machine learning projects.

(04:14) How Teya creates self-service DAGs for dbt models.

(05:58) Automating DAG creation with CI/CD pipelines.

(09:04) Switching to a multi-file method for better Airflow performance.

(12:48) Challenges faced with Kubernetes Executor vs. Celery Executor.

(16:13) Using Celery Executor to handle fast tasks efficiently.

(17:02) Implementing KEDA autoscaler for better scaling of Celery workers.

(19:05) Reasons for not using Cosmos for DAG generation and cross-DAG dependencies.

(21:16) Alexandre's wish list for future Airflow features, focusing on multi-tenancy.

Resources Mentioned:

Alexandre Magno Lima Martins -

https://www.linkedin.com/in/alex-magno/

Teya -

https://www.linkedin.com/company/teya-global/

Apache Airflow -

https://airflow.apache.org/

dbt -

https://www.getdbt.com/

Kubernetes -

https://kubernetes.io/

KEDA -

https://keda.sh/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Managing data workflows well can change the game for any company. In this episode, we talk about how Airflow makes this possible. Larry Komenda, Chief Technology Officer at Campbell, shares how Airflow supports their operations and improves efficiency.

Larry discusses his role at Campbell, their switch to Airflow, and its impact. We look at their strategies for testing and maintaining reliable workflows and how these help their business.

Key Takeaways:

(02:26) Strong technology and data systems are crucial for Campbell’s investment process.

(05:03) Airflow manages data pipelines efficiently in the market data team.

(07:39) Airflow supports various departments, including trading and operations.

(09:22) Machine learning models run on dedicated Airflow instances.

(11:12) Reliable workflows are ensured through thorough testing and development.

(13:45) Business tasks are organized separately from Airflow for easier testing.

(15:30) Non-technical teams have access to Airflow for better efficiency.

(17:20) Thorough testing before deploying to Airflow is essential.

(19:10) Non-technical users can interact with Airflow DAGs to solve their issues.

(21:55) Airflow improves efficiency and reliability in trading and operations.

(24:40) Enhancing the Airflow UI for non-technical users is important for accessibility.

Resources Mentioned:

Larry Komenda -

https://www.linkedin.com/in/larrykomenda/

Campbell -

https://www.linkedin.com/company/campbell-and-company/

30% off Airflow Summit Ticket -

https://ti.to/airflowsummit/2024/discount/30DISC_ASTRONOMER

Apache Airflow -

https://airflow.apache.org/

NumPy -

https://numpy.org/

Python -

https://www.python.org/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

The world of timekeeping for knowledge workers is transforming through the use of AI and machine learning. Understanding how to leverage these technologies is crucial for improving efficiency and productivity.

In this episode, we’re joined by Vincent La, Principal Data Scientist at Laurel, and Jim Howard, Principal Machine Learning Engineer at Laurel, to explore the implementation of AI in automating timekeeping and its impact on legal and accounting firms.

Key Takeaways:

(01:54) Laurel's mission in time automation.

(03:39) Solving clustering, prediction and summarization with AI.

(06:30) Daily batch jobs for user time generation.

(08:22) Knowledge workers touch 300 items daily.

(09:01) Mapping 300 activities to seven billable items.

(11:38) Retraining models for better performance.

(14:00) Using Airflow for retraining and backfills.

(17:06) RAG-based summarization for user-specific tone.

(18:58) Testing Airflow DAGs for cost-effective summarization.

(22:00) Enhancing Airflow for long-running DAGs.

Resources Mentioned:

Vincent La -

https://www.linkedin.com/in/vincentla/

Jim Howard -

https://www.linkedin.com/in/jameswhowardml/

Laurel -

https://www.linkedin.com/company/laurel-ai/

Apache Airflow -

https://airflow.apache.org/

Ernst & Young -

https://www.ey.com/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

The world of timekeeping for knowledge workers is transforming through the use of AI and machine learning. Understanding how to leverage these technologies is crucial for improving efficiency and productivity.

In this episode, we’re joined by Vincent La, Principal Data Scientist at Laurel, and Jim Howard, Principal Machine Learning Engineer at Laurel, to explore the implementation of AI in automating timekeeping and its impact on legal and accounting firms.

Key Takeaways:

(01:54) Laurel's mission in time automation.

(03:39) Solving clustering, prediction and summarization with AI.

(06:30) Daily batch jobs for user time generation.

(08:22) Knowledge workers touch 300 items daily.

(09:01) Mapping 300 activities to seven billable items.

(11:38) Retraining models for better performance.

(14:00) Using Airflow for retraining and backfills.

(17:06) RAG-based summarization for user-specific tone.

(18:58) Testing Airflow DAGs for cost-effective summarization.

(22:00) Enhancing Airflow for long-running DAGs.

Resources Mentioned:

Vincent La -

https://www.linkedin.com/in/vincentla/

Laurel | LinkedIn -

https://www.linkedin.com/company/laurel-ai/

Jim Howard -

https://www.linkedin.com/in/jameswhowardml/

Apache Airflow -

https://airflow.apache.org/

Ernst & Young -

https://www.ey.com/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Discover the cutting-edge methods Vibrant Planet uses to revolutionize geospatial data processing and resource management.

In this episode, we delve into the intricacies of scaling geospatial data processing and resource allocation with experts from Vibrant Planet. Joining us are Cyrus Dukart, Engineering Lead, and David Sacerdote, Staff Software Engineer, who share their innovative approaches to handling large datasets and optimizing resource use in Airflow.

Key Takeaways:

(00:00) Inefficiencies in resource allocation.

(03:00) Scientific validity of sharded results.

(05:53) Tech-based solutions for resource management.

(06:11) Retry callback process for resource allocation.

(08:00) Running database queries for resource needs.

(10:05) Importance of remembering resource usage.

(13:51) Generating resource predictions.

(14:44) Custom task decorator for resource management.

(20:28) Massive resource usage gap in sharded data.

(21:14) Fail-fast model for long-running tasks.

Resources Mentioned:

Cyrus Dukart -

https://www.linkedin.com/in/cyrus-dukart-6561482/

David Sacerdote -

https://www.linkedin.com/in/davidsacerdote/

Vibrant Planet -

https://www.linkedin.com/company/vibrant-planet/

Apache Airflow -

https://airflow.apache.org/

Kubernetes -

https://kubernetes.io/

Vibrant Planet -

https://vibrantplanet.net/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

The world of data orchestration and machine learning is rapidly evolving, and tools like Apache Airflow are at the forefront of these changes. Understanding how to effectively utilize these tools can significantly enhance data processing and AI model deployment.

This episode features Julian LaNeve, CTO at Astronomer, and David Xue, Machine Learning Engineer at Astronomer. They delve into the intricacies of data orchestration, generative AI and the practical applications of these technologies in modern data workflows.

Key Takeaways:

(01:51) The pressure to engage in the generative AI space.

(02:02) Generative AI can elevate data utilization to the next level.

(02:43) The transparency issues with commercial AI models.

(04:27) High-quality data in model performance is crucial.

(06:40) Running new models on smaller devices, like phones.

(12:19) Fine-tuning LLMs to handle millions of task failures.

(16:54) Teaching AI to understand specific logs, not general passages, is a goal.

(21:56) Using Airflow as a general-purpose orchestration tool.

(22:00) Airflow is adaptable for various use cases, including ETL and ML systems.

Resources Mentioned:

Julian LaNeve - https://www.linkedin.com/in/julianlaneve/

Atronomer - https://www.linkedin.com/company/astronomer/

David Xue - https://www.linkedin.com/in/david-xue-uva/

Apache Airflow - https://airflow.apache.org/

Meta’s Open Source Llama 3 model: https://ai.meta.com/blog/meta-llama-3/https://ai.meta.com/blog/meta-llama-3/

Microsoft’s Phi-3 model: https://www.microsoft.com/en-us/research/publication/phi-3-technical-report-a-highly-capable-language-model-locally-on-your-phone/

GPT-4 - https://www.openai.com/research/gpt-4

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#ai #automation #airflow #machinelearning