Managing data workflows well can change the game for any company. In this episode, we talk about how Airflow makes this possible. Larry Komenda, Chief Technology Officer at Campbell, shares how Airflow supports their operations and improves efficiency.

Larry discusses his role at Campbell, their switch to Airflow, and its impact. We look at their strategies for testing and maintaining reliable workflows and how these help their business.

Key Takeaways:

(02:26) Strong technology and data systems are crucial for Campbell’s investment process.

(05:03) Airflow manages data pipelines efficiently in the market data team.

(07:39) Airflow supports various departments, including trading and operations.

(09:22) Machine learning models run on dedicated Airflow instances.

(11:12) Reliable workflows are ensured through thorough testing and development.

(13:45) Business tasks are organized separately from Airflow for easier testing.

(15:30) Non-technical teams have access to Airflow for better efficiency.

(17:20) Thorough testing before deploying to Airflow is essential.

(19:10) Non-technical users can interact with Airflow DAGs to solve their issues.

(21:55) Airflow improves efficiency and reliability in trading and operations.

(24:40) Enhancing the Airflow UI for non-technical users is important for accessibility.

Resources Mentioned:

Larry Komenda -

https://www.linkedin.com/in/larrykomenda/

Campbell -

https://www.linkedin.com/company/campbell-and-company/

30% off Airflow Summit Ticket -

https://ti.to/airflowsummit/2024/discount/30DISC_ASTRONOMER

Apache Airflow -

https://airflow.apache.org/

NumPy -

https://numpy.org/

Python -

https://www.python.org/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning