
Sign up to save your podcasts
Or
The evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas.
In this episode, we’re joined by Anant Agarwal, Software Engineer at Instacart, who shares insights into Instacart's Airflow journey, from its early adoption in 2019 to the present-day centralized cluster approach. Anant discusses the challenges of managing disparate clusters, the implementation of remote executors, and the strategic standardization of infrastructure and DAG patterns to streamline workflows.
Key Takeaways:
(03:49) The impact of external events on business growth and technological evolution.
(04:31) Challenges of managing decentralized systems across multiple teams.
(06:14) The importance of standardizing infrastructure and processes for scalability.
(09:51) Strategies for implementing efficient and repeatable deployment practices.
(12:17) Addressing diverse user personas with tailored solutions.
(14:47) Leveraging remote execution to enhance flexibility and scalability.
(18:36) Benefits of transitioning to a centralized system for organization-wide use.
(20:57) Maintaining an upgrade cadence to stay aligned with the latest advancements.
(23:35) Anticipation for new features and improvements in upcoming software versions.
Resources Mentioned:
Anant Agarwal
https://www.linkedin.com/in/anantag/
Instacart | LinkedIn
https://www.linkedin.com/company/instacart/
Instacart | Website
https://www.instacart.com
Apache Airflow
https://airflow.apache.org/
AWS Amazon
https://aws.amazon.com/ecs/
Terraform
https://www.terraform.io/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
5
2020 ratings
The evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas.
In this episode, we’re joined by Anant Agarwal, Software Engineer at Instacart, who shares insights into Instacart's Airflow journey, from its early adoption in 2019 to the present-day centralized cluster approach. Anant discusses the challenges of managing disparate clusters, the implementation of remote executors, and the strategic standardization of infrastructure and DAG patterns to streamline workflows.
Key Takeaways:
(03:49) The impact of external events on business growth and technological evolution.
(04:31) Challenges of managing decentralized systems across multiple teams.
(06:14) The importance of standardizing infrastructure and processes for scalability.
(09:51) Strategies for implementing efficient and repeatable deployment practices.
(12:17) Addressing diverse user personas with tailored solutions.
(14:47) Leveraging remote execution to enhance flexibility and scalability.
(18:36) Benefits of transitioning to a centralized system for organization-wide use.
(20:57) Maintaining an upgrade cadence to stay aligned with the latest advancements.
(23:35) Anticipation for new features and improvements in upcoming software versions.
Resources Mentioned:
Anant Agarwal
https://www.linkedin.com/in/anantag/
Instacart | LinkedIn
https://www.linkedin.com/company/instacart/
Instacart | Website
https://www.instacart.com
Apache Airflow
https://airflow.apache.org/
AWS Amazon
https://aws.amazon.com/ecs/
Terraform
https://www.terraform.io/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
481 Listeners
38 Listeners
142 Listeners
265 Listeners
140 Listeners
289 Listeners
8,909 Listeners
2,146 Listeners
12 Listeners
8 Listeners
8 Listeners
15 Listeners
450 Listeners