
Sign up to save your podcasts
Or
How do you retrofit a clustered data-processing system to use cheap commodity storage? That’s the big question in this episode as we look at one of the many attempts to build a version of Kafka that uses object storage services like S3 as its main disk, sacrificing a little latency for cheap, infinitely-scalable disks.
There are several companies trying to walk down that road, and it’s clearly big business - one of them recently got bought out for a rumoured $250m. But one of them is actively trying to get those changes back into the community, as are pushing to make Apache Kafka speak object storage natively.
Joining me to explain why and how are Josep Prat and Filip Yonov of Aiven. We break down what it takes to make Kafka’s storage layer optional on a per-topic basis, how they’re making sure it’s not a breaking change, and how they plan to get such a foundational feature merged.
–
Announcement Post: https://aiven.io/blog/guide-diskless-apache-kafka-kip-1150
Aiven’s (Temporary) Fork, Project Inkless: https://github.com/aiven/inkless/blob/main/docs/inkless/README.md
Kafka Improvement Process (KIP) Articles:
Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices
Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join
Filip on LinkedIn: https://www.linkedin.com/in/filipyonov
Josep on LinkedIn: https://www.linkedin.com/in/jlprat/
Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social
Kris on Mastodon: http://mastodon.social/@krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
5
2828 ratings
How do you retrofit a clustered data-processing system to use cheap commodity storage? That’s the big question in this episode as we look at one of the many attempts to build a version of Kafka that uses object storage services like S3 as its main disk, sacrificing a little latency for cheap, infinitely-scalable disks.
There are several companies trying to walk down that road, and it’s clearly big business - one of them recently got bought out for a rumoured $250m. But one of them is actively trying to get those changes back into the community, as are pushing to make Apache Kafka speak object storage natively.
Joining me to explain why and how are Josep Prat and Filip Yonov of Aiven. We break down what it takes to make Kafka’s storage layer optional on a per-topic basis, how they’re making sure it’s not a breaking change, and how they plan to get such a foundational feature merged.
–
Announcement Post: https://aiven.io/blog/guide-diskless-apache-kafka-kip-1150
Aiven’s (Temporary) Fork, Project Inkless: https://github.com/aiven/inkless/blob/main/docs/inkless/README.md
Kafka Improvement Process (KIP) Articles:
Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices
Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join
Filip on LinkedIn: https://www.linkedin.com/in/filipyonov
Josep on LinkedIn: https://www.linkedin.com/in/jlprat/
Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social
Kris on Mastodon: http://mastodon.social/@krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
272 Listeners
284 Listeners
590 Listeners
621 Listeners
269 Listeners
140 Listeners
987 Listeners
189 Listeners
62 Listeners
139 Listeners
72 Listeners
47 Listeners
75 Listeners
19 Listeners
53 Listeners