Developer Voices

Making Apache Kafka Diskless (with Filip Yonov & Josep Prat)


Listen Later

How do you retrofit a clustered data-processing system to use cheap commodity storage? That’s the big question in this episode as we look at one of the many attempts to build a version of Kafka that uses object storage services like S3 as its main disk, sacrificing a little latency for cheap, infinitely-scalable disks.

There are several companies trying to walk down that road, and it’s clearly big business - one of them recently got bought out for a rumoured $250m. But one of them is actively trying to get those changes back into the community, as are pushing to make Apache Kafka speak object storage natively.

Joining me to explain why and how are Josep Prat and Filip Yonov of Aiven. We break down what it takes to make Kafka’s storage layer optional on a per-topic basis, how they’re making sure it’s not a breaking change, and how they plan to get such a foundational feature merged.

Announcement Post: https://aiven.io/blog/guide-diskless-apache-kafka-kip-1150

Aiven’s (Temporary) Fork, Project Inkless: https://github.com/aiven/inkless/blob/main/docs/inkless/README.md

Kafka Improvement Process (KIP) Articles:

  • KIP-1150: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
  • KIP-1163: Diskless Core: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1163%3A+Diskless+Core
  • KIP-1164: Topic Based Batch Coordinator: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1164%3A+Topic+Based+Batch+Coordinator
  • KIP-1165: Object Compaction for Diskless: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1165%3A+Object+Compaction+for+Diskless

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Filip on LinkedIn: https://www.linkedin.com/in/filipyonov

Josep on LinkedIn: https://www.linkedin.com/in/jlprat/

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

...more
View all episodesView all episodes
Download on the App Store

Developer VoicesBy Kris Jenkins

  • 5
  • 5
  • 5
  • 5
  • 5

5

28 ratings


More shows like Developer Voices

View all
Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

Software Engineering Radio - the podcast for professional software developers

272 Listeners

The Changelog: Software Development, Open Source by Changelog Media

The Changelog: Software Development, Open Source

284 Listeners

Talk Python To Me by Michael Kennedy

Talk Python To Me

590 Listeners

Software Engineering Daily by Software Engineering Daily

Software Engineering Daily

621 Listeners

Soft Skills Engineering by Jamison Dance and Dave Smith

Soft Skills Engineering

269 Listeners

Data Engineering Podcast by Tobias Macey

Data Engineering Podcast

140 Listeners

Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

Syntax - Tasty Web Development Treats

987 Listeners

CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer

CoRecursive: Coding Stories

189 Listeners

The Stack Overflow Podcast by The Stack Overflow Podcast

The Stack Overflow Podcast

62 Listeners

The Real Python Podcast by Real Python

The Real Python Podcast

139 Listeners

Signals and Threads by Jane Street

Signals and Threads

72 Listeners

Oxide and Friends by Oxide Computer Company

Oxide and Friends

47 Listeners

Latent Space: The AI Engineer Podcast by swyx + Alessio

Latent Space: The AI Engineer Podcast

75 Listeners

Rust in Production by Matthias Endler

Rust in Production

19 Listeners

The Pragmatic Engineer by Gergely Orosz

The Pragmatic Engineer

53 Listeners