Data orchestration is revolutionizing the way companies manage and process data. In this episode, we explore the critical role of data orchestration in modern data workflows and how Apache Airflow is used to enhance data processing and AI model deployment.

Hannan Kravitz, Data Engineering Team Leader at Artlist, joins us to share his insights on leveraging Airflow for data engineering and its impact on their business operations.

Key Takeaways:

(01:00) Hannan introduces Artlist and its mission to empower content creators.

(04:27) The importance of collecting and modeling data to support business insights.

(06:40) Using Airflow to connect multiple data sources and create dashboards.

(09:40) Implementing a monitoring DAG for proactive alerts within Airflow​​.

(12:31) Customizing Airflow for business metric KPI monitoring and setting thresholds​​.

(15:00) Addressing decreases in purchases due to technical issues with proactive alerts​​.

(17:45) Customizing data quality checks with dynamic task mapping in Airflow​​.

(20:00) Desired improvements in Airflow UI and logging capabilities​​.

(21:00) Enabling business stakeholders to change thresholds using Streamlit​​.

(22:26) Future improvements desired in the Airflow project​.

Resources Mentioned:

Hannan Kravitz -

https://www.linkedin.com/in/hannan-kravitz-60563112/

Artlist -

https://www.linkedin.com/company/art-list/

Apache Airflow -

https://airflow.apache.org/

Snowflake -

https://www.snowflake.com/

Streamlit -

https://streamlit.io/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning