Philip brings a ton of experience in the world of data processing and has shared some of the best practices in implementing these systems. We dive deep into the world of data processing, covering batch and streaming systems, their challenges, tradeoffs and use cases.

Chapters:

00:00 Batch vs Realtime Stream Processing

03:25 What is Batch and Reatlime processing?

18:29 How does Batch and Realtime compare in terms of Latency and Throughput?

27:24 Where is the cost saving coming from? Compute?Storage? or Network?

31:38 Moving from Batch to Stream processing

37:50 How is Idempotency implemented in Streaming systems?

48:50 How do we approach Schema evolution in Batch and Streaming systems?

57:16 Summary - key points to keep in mind

Do checkout Estuary if you deal with a ton of data, and don't want to deal with the painful operations, infrastructure management, schema migrations etc and only want to focus on building highly scalable and resilient applications.

References:

Estuary: https://estuary.dev/

Flow documentation: https://docs.estuary.dev

If you like this video please hit the like button, share it with your network (whoever works with a ton of data) and subscribe to the channel.

Feel free to watch related episodes in the playlist:    • Distributed Syste...  

Modern Databases:    • Modern Databases  

Software Engineering:    • Software Engineering  

Distributed Systems:    • Distributed Systems  

Cheers,

The GeekNarrator