An interview with Greg Harris and Ivan Yurchenko, part of the authors of the new KIP-1150 that introduces diskless topics to Apache Kafka.
Diskless topics are a special kind of topic that write directly to an object store. This outsources the data replication and durability guarantees to the store (e.g., S3) and avoids the costly inter-zone data transfer fees that clouds charge you for. The result is a 80%+ reduced Kafka bill and a simpler architecture where brokers are stateless (or have less state) – making them significantly easier to maintain operationally.
In this podcast, we dive deep into the technical details and tradeoffs in this complex KIP.
⏱️ TIMELINE
- 04:00 – Why Diskless?
- 09:17 – Architecture Walkthrough: How Does It Work
- 18:50 – The Batch Coordinator
- 22:00 – The Different Batch Coordinator Implementations
- 28:00 – The Inkless Fork (And Why)
- 39:40 – Topic-Based Batch Coordinator
- 45:40 – Bottlenecks in the Batch Coordinator?
- 47:48 – The Read Path
- 54:40 – Caching
- 01:03:00 – Shared Log Segment Merging (AKA Object Compaction)
- 01:19:06 – Latency
- 01:26:30 – Practical Real-World Latency Requirements
- 01:39:00 – The Power of Open Source
- 01:47:30 – Cost Breakdown
- 01:58:00 – Ops and Engineering Time Savings
- 02:06:30 – Client Bootstrapping, Leaderless Partitions, One Broker Connection per Client
- 02:11:20 – Multi-Tenant Kafka Future
- 02:14:05 – Iceberg, Parquet, and First-Class Schema Support
- 02:23:10 – S3 Express
If you found this episode interesting, please give it a like to signal to the algorithm that it's good. It takes 2 seconds to do, and it took us 5 hours to produce.
💬 HOW TO GET INVOLVED
We recommend participating in the Apache Kafka mailing list.
Beyond that, you can find Greg and Ivan here:
- Ivan Yurchenko – X: @ivan0yu
- Greg Harris – X: @gharris1727
⚠️ DISCLAIMERS
All views expressed in this podcast are personal opinions of the participants. They do not represent the views or positions of Aiven, the employer of Ivan and Greg, or any other entity.
Apache®, Apache Kafka®, Kafka, and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.
