This episode will give you a good understanding of why we need a new database for storing "Derived Data" in a low latency, high performance manner, which is very important for Machine Learning workloads.

Chapters:

00:00 Introduction

01:42 The Evolution of LinkedIn's Databases

03:15 Challenges with Voldemort and the Birth of VeniceDB

08:42 Understanding Derived Data

13:33 Planet-Scale Applications and Multi-Region Support

17:40 Writing Data into VeniceDB

22:53 Merging Data in VeniceDB

40:31 Understanding the Architecture

40:47 Components of the Write Path

41:56 Leader and Follower Architecture

43:58 Partitioning and DaVinci Client

47:57 Read Patterns and Client Options

54:25 Fault Tolerance and Recommender Systems

01:01:19 Kafka Integration and Deployment

01:06:56 Roadmap and Future Improvements

Important links:

VeniceDB blog: https://www.linkedin.com/blog/engineering/open-source/open-sourcing-venice-linkedin-s-derived-data-platform

VeniceDB docs: https://venicedb.org/

Qcon: https://youtu.be/pJeg4V3JgYo?si=vblGUxp5fNdKPHoC

Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator

If you like this episode, please hit the like button and share it with your network.

Also please subscribe if you haven't yet.

Database internals series: https://youtu.be/yV_Zp0Mi3xs

Popular playlists:

Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-

Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17

Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d

Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN

Stay Curios! Keep Learning!

#kafka #linkedin #venicedb #Rocksdb