// Abstract

A few years ago Uber set out to create an ads platform for the Uber Eats app that relied heavily on three pillars; Speed, Reliability, and Accuracy. Some of the technical challenges they were faced with included exactly-once semantics in real-time. To accomplish this goal, they created the architecture diagram above with lots of love from Flink, Kafka, Hive, and Pinot. You can dig into the whole paper (https://go.mlops.community/k8gzZd) to see all the reasoning for their design decisions.

// Bio

Jacob Tsafatinos is a Staff Software Engineer at Elemy. He led the efforts of the Ad Events Processing system at Uber and has previously worked on a range of problems including data ingestion for search and machine learning recommendation pipelines. In his spare time, he can be found playing lead guitar in his band Good Kid.

https://mlops.pallet.xyz/jobs

// Related Links

Uber blog

https://eng.uber.com/author/jacob-tsafatinos/

https://eng.uber.com/real-time-exactly-once-ad-event-processing/

--------------- ✌️Connect With Us ✌️ -------------

Join our slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Mihail on LinkedIn: https://www.linkedin.com/in/mihaileric/

Connect with Jacob on LinkedIn: https://www.linkedin.com/in/jacobtsaf/

Timestamps:

[00:00] Introduction to Jacob Tsafatinos

[00:40] Takeaways

[04:25] Jacob's band

[05:29] Lyrics about software engineers or artistic stuff

[06:20] Connection of hobby and real-time system

[08:43] How to game Spotify Algorithm?

[10:00] Data stack for analytics

[13:28] Uber blog

[16:28] Video mess up

[17:04] Considerations and importance of the Uber System

[21:22] Challenges encountered through the Uber System journey

[26:06] Crucial to building the system

[28:13] Not exactly real-time

[30:22] Design decisions main questions

[34:23] Testament to OSS  

[36:58] Real-time processing systems for analytical use cases vs Real-time processing systems for predictive use cases

[38:46] Real-time systems necessity

[41:04] Potential that opens up new doors

[41:40] Runaway or learn it?

[46:09] Real-time use case target

[49:31] Resource constrained

[50:48] ML Oops stories

[52:45] Wrap up