How do you retrofit a clustered data-processing system to use cheap commodity storage? That’s the big question in this episode as we look at one of the many attempts to build a version of Kafka that uses object storage services like S3 as its main disk, sacrificing a little latency for cheap, infinitely-scalable disks.

There are several companies trying to walk down that road, and it’s clearly big business - one of them recently got bought out for a rumoured $250m. But one of them is actively trying to get those changes back into the community, as are pushing to make Apache Kafka speak object storage natively.

Joining me to explain why and how are Josep Prat and Filip Yonov of Aiven. We break down what it takes to make Kafka’s storage layer optional on a per-topic basis, how they’re making sure it’s not a breaking change, and how they plan to get such a foundational feature merged.

–

Announcement Post: https://aiven.io/blog/guide-diskless-apache-kafka-kip-1150

Aiven’s (Temporary) Fork, Project Inkless: https://github.com/aiven/inkless/blob/main/docs/inkless/README.md

Kafka Improvement Process (KIP) Articles:

• KIP-1150: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
• KIP-1163: Diskless Core: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1163%3A+Diskless+Core
• KIP-1164: Topic Based Batch Coordinator: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1164%3A+Topic+Based+Batch+Coordinator
• KIP-1165: Object Compaction for Diskless: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1165%3A+Object+Compaction+for+Diskless

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Filip on LinkedIn: https://www.linkedin.com/in/filipyonov

Josep on LinkedIn: https://www.linkedin.com/in/jlprat/

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Java’s has been evolving faster than any 30 year old language has a right to do, and there’s probably no-one more pleased about it than my guest this week - Josh Long. He’s a Java & Kotlin programming, a JVM enthusiast in general, and an advocate for Spring, and he has chapters full of news about what’s been happening in Javaland over the past few years. Everything from new threading models to C interop changes, custom primitives to high performance computing and all the ways in which Java is modernising for age of AI workloads.

If you’re out of touch with the latest in the JVM, or don’t know how much its changed, Josh’s brain is full of all the news you need to catch up.

–

Project Valhalla (Value Objects): https://openjdk.org/projects/valhalla/

Project Panama (JVM’s new native code support): https://openjdk.org/projects/panama/

Jextract: https://github.com/openjdk/jextract

Spring Initializer: http://start.spring.io/

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

I’m joined this week by one of the authors of Apache Kafka In Action, to take a look at the state of Kafka, event systems & stream-processing technology. It’s an approach (and a whole market) that’s had at least a decade to mature, so how has it done? What does Kafka offer to developers and businesses, and which parts do they actually care about? What have streaming data systems promised and what have they actually delivered? What’s still left to build?

–

Apache Kafka in Action: https://www.manning.com/books/apache-kafka-in-action

Pat Helland, Data on the Inside vs Data on the Outside: https://queue.acm.org/detail.cfm?id=3415014

Out of the Tar Pit: https://curtclifton.net/papers/MoseleyMarks06a.pdf

Martin Kleppmann, Turning the Database Inside-Out: https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html

Data Mesh by Zhamak Dehghani: https://www.amazon.co.uk/Data-Mesh-Delivering-Data-Driven-Value/dp/1492092398

Quix Streams: https://github.com/quixio/quix-streams

XTDB: https://xtdb.com/

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Anatoly’s Website: https://zelenin.de/

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

Building a database is a serious undertaking. There are just so many parts that you have to implement before you even get to a decent prototype, and so many hours of work before you could begin working on the ideas that would make your database unique. Apache DataFusion is a project that hopes to change all that, but building an extensible, composable toolkit of database pieces, which could let you build a viable database extremely quickly, and then innovate from that starting point. And even if you’re not building a database, it’s a fascinating project to explain how databases are built.

Joining me to explain it all is Andrew Lamb, one of DataFusion’s core contributors, and he’s going to take us through the whole stack, how it’s built and how you could use it. Along the way we cover everything from who’s building interesting new databases and how you manage a large, open-source Rust project.

–

DataFusion Homepage: https://datafusion.apache.org/

DataFusion on Github: https://github.com/apache/datafusion

DataFusion Architecture (with diagrams!): https://youtu.be/NVKujPxwSBA?si=tw9ACxlbdpBuVsnv&t=1045

Datalog: https://docs.racket-lang.org/datalog/

Tokio: https://tokio.rs/

Andrew’s Homepage: http://andrew.nerdnetworks.org/

Andrew’s Blog Post about Tokio: https://thenewstack.io/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/

Velox: https://velox-lib.io/

Arroyo: https://www.arroyo.dev/

Synnada: https://www.synnada.ai/

LanceDB: https://lancedb.com/

SDF+DBT: https://docs.sdf.com/integrations/dbt/integrating

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Jupyter’s become an incredibly popular programming and data science tool, but how does it actually work? How have they built an interactive language execution engine? And if we understand the architecture, what else could it be used for?

Joining me to look inside the Jupyter toolbox are Afshin Darian and Sylvain Corlay, two of Jupyters long-standing contributors and project-steerers. They’ve going to take us on a journey that starts with today’s userbase, goes through the execution protocol and ends with a look at what Jupyter will be in the future - an ambitious framework for interactive, collaborative applications and more.

–

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Jupyter Homepage: https://jupyter.org/

Jupyter Xeus: https://github.com/jupyter-xeus/xeus

Jupyter AI: https://github.com/jupyterlab/jupyter-ai

Jupyter CAD: https://github.com/jupytercad/JupyterCAD

Jupyter GIS: https://github.com/geojupyter/jupytergis/

Jupyter GIS Announcement: https://blog.jupyter.org/real-time-collaboration-and-collaborative-editing-for-gis-workflows-with-jupyter-and-qgis-d25dbe2832a6

QGIS: https://qgis.org/

ZeroMQ: https://zeromq.org/

Sylvain on LinkedIn: https://www.linkedin.com/in/sylvaincorlay

Darian on LinkedIn: https://www.linkedin.com/in/afshindarian

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Ever since we invented makefiles, the programming world has been wrestling with the problem of building software stacks reliably. This week we’re going to look at one of the most ambitious solutions available - Nix. Nix tries to do everything from invoking your compiler to installing your language, and even providing your operating system. But how does it work in theory, and how well does it work in practice?

Joining me to discuss is Julian Arni, a Nix-enthusiast and creator of a build/test/deploy service called Garnix.

Nix has been one of my go-to tools for years - I hope it’ll find its way into your stack.

–

Nix Overview: https://nixos.org/explore/

Nix Tutorial: https://nix.dev/tutorials/first-steps/

Nix Flakes: https://nixos.wiki/wiki/Flakes

The Nix Package List: https://search.nixos.org/packages

Garnix.IO: https://garnix.io/

Julian’s NixCon Talk, Call by Hash: https://www.youtube.com/watch?v=fU9ogB9hZZA

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Graphite is a new image editor with an interesting architecture - it’s a classic UI-driven app, an image-manipulation language, and a library of programmable graphics primitives that any Rust coder could use, extend or add to. The result is something that you can use like Photoshop or Inkscape, or make use of in batch pipelines, a bit like ImageMagick.

Joining me to discuss it are Keavon Chambers & Dennis Kobert, who are hammering away on building a project that’s potentially as demanding as Photoshop, but with a more ambitious architecture. How can they hope to compete? Perhaps in the short term by doing what regular image And is the future of image editing modular?

–

Graphite Homepage: https://graphite.rs/

Graphite Web Version: https://editor.graphite.rs/

Graphite on Github: https://github.com/GraphiteEditor/Graphite

Signed Distance Fields: https://jasmcole.com/2019/10/03/signed-distance-fields/

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

ReScript is a strongly-typed programming language that compiles to JavaScript, and that puts it squarely in competition with TypeScript. So why would a JavaScript developer choose to learn it next? What does it offer that makes it a tempting proposition? And how are the ReScript developers making life easier for anyone who wants to make the switch?

To answer all these questions and more, I’m joined this week by Gabriel Nordeborn, one of ReScript’s compiler contributors. 

--

ReScript: https://rescript-lang.org/

ReScript & React: https://rescript-lang.org/docs/react/latest/introduction

Reanalyze: https://github.com/rescript-lang/reanalyze

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Gabriel on BlueSky: https://bsky.app/profile/did:plc:c55mqp6e6r24rrrypkmx7ker

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Trustfall is a library based on a simple question - what happens if we can query absolutely anything? If you could join REST APIs and databases with filesystems and dockerfiles? It’s possible in theory because those are all just datasources. Predrag Gruevski is trying to make it easy by building a universal query engine, with pluggable datasources, all in Rust.

This week we dive into Trustfall to figure out how it works. How do you model nearly anything as a datasource? How do you make it easy to extend? And what does it take to optimize a query that’s going to be spread out over multiple systems and potentially multiple servers? Questions, questions, questions - all about the act of asking our systems questions. 😃

–

Trustfall Source: https://github.com/obi1kenobi/trustfall

Cargo Semver Checks: https://crates.io/crates/cargo-semver-checks

How to Query Almost Everything: https://predr.ag/querying/

SemVer In Rust (talk): https://predr.ag/blog/semver-in-rust-tooling-breakage-and-edge-cases/#cargo-semver-checks-lints-are-database-queries-in-disguise

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

Dimitris Kyriakoudis is a researcher, programmer and musician who's combining all three talents to build dedicated music hardware. Specifically a device called the µseq, which reads Lisp programs and uses them to drive synthesizers to make music. In this episode we go through the full platform that he's building, from soldering resistors to an RPi chip, up through writing a Lisp interpreter, to the design ideas that make Lisp a good choice for composing both software and music.

–

uSeq Homepage: https://www.emutelabinstruments.co.uk/useq/

Emute Lab’s Homepage: https://www.emutelab.org/

Buy a uSeq: https://www.signalsounds.com/emute-lab-instruments-useq-live-coding-voltage-generator-eurorack-module/

Build a uSeq (DIY Kit): https://www.thonk.co.uk/shop/emute-lab-useq/

SICP (book): https://en.wikipedia.org/wiki/Structure_and_Interpretation_of_Computer_Programs

Machina Bristronica (expo): https://machinabristronica.uk/

Sonic Pi: https://sonic-pi.net/

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

0:00 Intro

2:20 What is µseq?

5:40 Live Coding As Another Instrument

17:42 Why Choose Lisp?

25:03 Different Dialects For Different Musical Tasks?

32:34 Live Coding As Academic Research

44:11 How Do You Fabricate Production Hardware?

49:00 The Triple-E Triangle

1:09:53 How Well Has This Theory Worked Out?

1:20:01 What's This Like To Play Live?

1:25:17 Comparisons With Sonic Pi

1:33:06 Outro