Harnessing data at scale is the key to driving innovation in autonomous vehicle technology. In this episode, we uncover how advanced orchestration tools are transforming machine learning operations in the automotive industry. Serjesh Sharma, Supervisor ADAS Machine Learning Operations (MLOps) at Ford Motor Company, joins us to discuss the challenges and innovations his team faces working to enhance vehicle safety and automation. Serjesh shares insights into the intricate data processes that support Ford’s Advanced Driver Assistance Systems (ADAS) and how his team leverages Apache Airflow to manage massive data loads efficiently.

Key Takeaways:

(01:44) ADAS involves advanced features like pre-collision assist and self-driving capabilities.

(04:47) Ensuring sensor accuracy and vehicle safety requires extensive data processing.

(05:08) The combination of on-prem and cloud infrastructure optimizes data handling.

(09:27) Ford processes around one petabyte of data per week, using both CPUs and GPUs.

(10:33) Implementing software engineering best practices to improve scalability and reliability.

(15:18) GitHub Issues streamline onboarding and infrastructure provisioning.

(17:00) Airflow's modular design allows Ford to manage complex data pipelines.

(19:00) Kubernetes pod operators help optimize resource usage for CPU-intensive tasks.

(20:35) Ford's scale challenges led to customized Airflow configurations for high concurrency.

(21:02) Advanced orchestration tools are pivotal in managing vast data landscapes in automotive innovation.

Resources Mentioned:

Serjesh Sharma - www.linkedin.com/in/serjeshsharma/

Ford Motor Company - www.linkedin.com/company/ford-motor-company/

Apache Airflow - airflow.apache.org/

Kubernetes - kubernetes.io/

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning