Sign up to save your podcasts
Or
Share Diving into Kafka Internals with David Jacot
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Diving into Kafka Internals with David Jacot
In this video I talk to David Jacot who works as a Staff Software Engineer at @Confluent and has been a long time Kafka user, committer and PMC member. We covered how Kafka works internally in great depth.
We use Kafka for various use cases and it works great, but going one level below the abstraction and truly understanding the protocols, techniques and algorithms used is a fun ride.
Chapters:
00:00 Kafka Internals with David Jacot
03:33 Defining Kafka
05:16 Kafka Architecture(s)
11:39 Write Path - Producer sending data
18:35 How does replication work?
25:47 How do we track replication progress?
30:42 Failure Modes: Leader fails
38:18 Consumers: Push vs Pull
40:54 Consumers: How does fetch works?
49:03 Consuming number of bytes vs records
50:50 Optimising consumption
01:00:21 Offset management and choosing partitions
01:09:10 Ending notes
I hope you like this episode and more importantly you learnt some amazing techniques Kafka uses to ensure durability, low latency, simplicity and scalability in its architecture.
Do give this episode a like and share it with your network. Also please subscribe to the channel for content like this.
Other playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
Other episodes:
KsqlDB: https://youtu.be/2yE86P6uD_0
Exactly once semantics: https://youtu.be/twgbAL_EaQw
David's Linkedin: https://www.linkedin.com/in/davidjacot/
our website: www.geeknarrator.com
Cheers,
The GeekNarrator
View all episodes
By Kaivalya Apte
5
33 ratings
In this video I talk to David Jacot who works as a Staff Software Engineer at @Confluent and has been a long time Kafka user, committer and PMC member. We covered how Kafka works internally in great depth.
We use Kafka for various use cases and it works great, but going one level below the abstraction and truly understanding the protocols, techniques and algorithms used is a fun ride.
Chapters:
00:00 Kafka Internals with David Jacot
03:33 Defining Kafka
05:16 Kafka Architecture(s)
11:39 Write Path - Producer sending data
18:35 How does replication work?
25:47 How do we track replication progress?
30:42 Failure Modes: Leader fails
38:18 Consumers: Push vs Pull
40:54 Consumers: How does fetch works?
49:03 Consuming number of bytes vs records
50:50 Optimising consumption
01:00:21 Offset management and choosing partitions
01:09:10 Ending notes
I hope you like this episode and more importantly you learnt some amazing techniques Kafka uses to ensure durability, low latency, simplicity and scalability in its architecture.
Do give this episode a like and share it with your network. Also please subscribe to the channel for content like this.
Other playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
Other episodes:
KsqlDB: https://youtu.be/2yE86P6uD_0
Exactly once semantics: https://youtu.be/twgbAL_EaQw
David's Linkedin: https://www.linkedin.com/in/davidjacot/
our website: www.geeknarrator.com
Cheers,
The GeekNarrator
More shows like The GeekNarrator
View all
Software Engineering Radio - the podcast for professional software developers
272 Listeners
The Changelog: Software Development, Open Source
284 Listeners
Thoughtworks Technology Podcast
40 Listeners
Talk Python To Me
590 Listeners
Software Engineering Daily
621 Listeners
Odd Lots
1,784 Listeners
Data Engineering Podcast
140 Listeners
Practical AI
192 Listeners
The Stack Overflow Podcast
62 Listeners
The Real Python Podcast
139 Listeners
Dwarkesh Podcast
408 Listeners
Oxide and Friends
47 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
461 Listeners
Money Stuff: The Podcast
371 Listeners
The Pragmatic Engineer
63 Listeners