Developer Voices

I’m joined this week by one of the authors of Apache Kafka In Action, to take a look at the state of Kafka, event systems & stream-processing technology. It’s an approach (and a whole market) that’s had at least a decade to mature, so how has it done? What does Kafka offer to developers and businesses, and which parts do they actually care about? What have streaming data systems promised and what have they actually delivered? What’s still left to build?

–

Apache Kafka in Action: https://www.manning.com/books/apache-kafka-in-action

Pat Helland, Data on the Inside vs Data on the Outside: https://queue.acm.org/detail.cfm?id=3415014

Out of the Tar Pit: https://curtclifton.net/papers/MoseleyMarks06a.pdf

Martin Kleppmann, Turning the Database Inside-Out: https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html

Data Mesh by Zhamak Dehghani: https://www.amazon.co.uk/Data-Mesh-Delivering-Data-Driven-Value/dp/1492092398

Quix Streams: https://github.com/quixio/quix-streams

XTDB: https://xtdb.com/

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Anatoly’s Website: https://zelenin.de/

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

Building a database is a serious undertaking. There are just so many parts that you have to implement before you even get to a decent prototype, and so many hours of work before you could begin working on the ideas that would make your database unique. Apache DataFusion is a project that hopes to change all that, but building an extensible, composable toolkit of database pieces, which could let you build a viable database extremely quickly, and then innovate from that starting point. And even if you’re not building a database, it’s a fascinating project to explain how databases are built.

Joining me to explain it all is Andrew Lamb, one of DataFusion’s core contributors, and he’s going to take us through the whole stack, how it’s built and how you could use it. Along the way we cover everything from who’s building interesting new databases and how you manage a large, open-source Rust project.

–

DataFusion Homepage: https://datafusion.apache.org/

DataFusion on Github: https://github.com/apache/datafusion

DataFusion Architecture (with diagrams!): https://youtu.be/NVKujPxwSBA?si=tw9ACxlbdpBuVsnv&t=1045

Datalog: https://docs.racket-lang.org/datalog/

Tokio: https://tokio.rs/

Andrew’s Homepage: http://andrew.nerdnetworks.org/

Andrew’s Blog Post about Tokio: https://thenewstack.io/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/

Velox: https://velox-lib.io/

Arroyo: https://www.arroyo.dev/

Synnada: https://www.synnada.ai/

LanceDB: https://lancedb.com/

SDF+DBT: https://docs.sdf.com/integrations/dbt/integrating

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Jupyter’s become an incredibly popular programming and data science tool, but how does it actually work? How have they built an interactive language execution engine? And if we understand the architecture, what else could it be used for?

Joining me to look inside the Jupyter toolbox are Afshin Darian and Sylvain Corlay, two of Jupyters long-standing contributors and project-steerers. They’ve going to take us on a journey that starts with today’s userbase, goes through the execution protocol and ends with a look at what Jupyter will be in the future - an ambitious framework for interactive, collaborative applications and more.

–

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Jupyter Homepage: https://jupyter.org/

Jupyter Xeus: https://github.com/jupyter-xeus/xeus

Jupyter AI: https://github.com/jupyterlab/jupyter-ai

Jupyter CAD: https://github.com/jupytercad/JupyterCAD

Jupyter GIS: https://github.com/geojupyter/jupytergis/

Jupyter GIS Announcement: https://blog.jupyter.org/real-time-collaboration-and-collaborative-editing-for-gis-workflows-with-jupyter-and-qgis-d25dbe2832a6

QGIS: https://qgis.org/

ZeroMQ: https://zeromq.org/

Sylvain on LinkedIn: https://www.linkedin.com/in/sylvaincorlay

Darian on LinkedIn: https://www.linkedin.com/in/afshindarian

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Ever since we invented makefiles, the programming world has been wrestling with the problem of building software stacks reliably. This week we’re going to look at one of the most ambitious solutions available - Nix. Nix tries to do everything from invoking your compiler to installing your language, and even providing your operating system. But how does it work in theory, and how well does it work in practice?

Joining me to discuss is Julian Arni, a Nix-enthusiast and creator of a build/test/deploy service called Garnix.

Nix has been one of my go-to tools for years - I hope it’ll find its way into your stack.

–

Nix Overview: https://nixos.org/explore/

Nix Tutorial: https://nix.dev/tutorials/first-steps/

Nix Flakes: https://nixos.wiki/wiki/Flakes

The Nix Package List: https://search.nixos.org/packages

Garnix.IO: https://garnix.io/

Julian’s NixCon Talk, Call by Hash: https://www.youtube.com/watch?v=fU9ogB9hZZA

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Graphite is a new image editor with an interesting architecture - it’s a classic UI-driven app, an image-manipulation language, and a library of programmable graphics primitives that any Rust coder could use, extend or add to. The result is something that you can use like Photoshop or Inkscape, or make use of in batch pipelines, a bit like ImageMagick.

Joining me to discuss it are Keavon Chambers & Dennis Kobert, who are hammering away on building a project that’s potentially as demanding as Photoshop, but with a more ambitious architecture. How can they hope to compete? Perhaps in the short term by doing what regular image And is the future of image editing modular?

–

Graphite Homepage: https://graphite.rs/

Graphite Web Version: https://editor.graphite.rs/

Graphite on Github: https://github.com/GraphiteEditor/Graphite

Signed Distance Fields: https://jasmcole.com/2019/10/03/signed-distance-fields/

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

ReScript is a strongly-typed programming language that compiles to JavaScript, and that puts it squarely in competition with TypeScript. So why would a JavaScript developer choose to learn it next? What does it offer that makes it a tempting proposition? And how are the ReScript developers making life easier for anyone who wants to make the switch?

To answer all these questions and more, I’m joined this week by Gabriel Nordeborn, one of ReScript’s compiler contributors. 

--

ReScript: https://rescript-lang.org/

ReScript & React: https://rescript-lang.org/docs/react/latest/introduction

Reanalyze: https://github.com/rescript-lang/reanalyze

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Gabriel on BlueSky: https://bsky.app/profile/did:plc:c55mqp6e6r24rrrypkmx7ker

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Trustfall is a library based on a simple question - what happens if we can query absolutely anything? If you could join REST APIs and databases with filesystems and dockerfiles? It’s possible in theory because those are all just datasources. Predrag Gruevski is trying to make it easy by building a universal query engine, with pluggable datasources, all in Rust.

This week we dive into Trustfall to figure out how it works. How do you model nearly anything as a datasource? How do you make it easy to extend? And what does it take to optimize a query that’s going to be spread out over multiple systems and potentially multiple servers? Questions, questions, questions - all about the act of asking our systems questions. 😃

–

Trustfall Source: https://github.com/obi1kenobi/trustfall

Cargo Semver Checks: https://crates.io/crates/cargo-semver-checks

How to Query Almost Everything: https://predr.ag/querying/

SemVer In Rust (talk): https://predr.ag/blog/semver-in-rust-tooling-breakage-and-edge-cases/#cargo-semver-checks-lints-are-database-queries-in-disguise

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

Dimitris Kyriakoudis is a researcher, programmer and musician who's combining all three talents to build dedicated music hardware. Specifically a device called the µseq, which reads Lisp programs and uses them to drive synthesizers to make music. In this episode we go through the full platform that he's building, from soldering resistors to an RPi chip, up through writing a Lisp interpreter, to the design ideas that make Lisp a good choice for composing both software and music.

–

uSeq Homepage: https://www.emutelabinstruments.co.uk/useq/

Emute Lab’s Homepage: https://www.emutelab.org/

Buy a uSeq: https://www.signalsounds.com/emute-lab-instruments-useq-live-coding-voltage-generator-eurorack-module/

Build a uSeq (DIY Kit): https://www.thonk.co.uk/shop/emute-lab-useq/

SICP (book): https://en.wikipedia.org/wiki/Structure_and_Interpretation_of_Computer_Programs

Machina Bristronica (expo): https://machinabristronica.uk/

Sonic Pi: https://sonic-pi.net/

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

0:00 Intro

2:20 What is µseq?

5:40 Live Coding As Another Instrument

17:42 Why Choose Lisp?

25:03 Different Dialects For Different Musical Tasks?

32:34 Live Coding As Academic Research

44:11 How Do You Fabricate Production Hardware?

49:00 The Triple-E Triangle

1:09:53 How Well Has This Theory Worked Out?

1:20:01 What's This Like To Play Live?

1:25:17 Comparisons With Sonic Pi

1:33:06 Outro

If you want to build really large software systems well, you have to stop thinking of them as just software systems. Beyond a certain size, everything your software touches becomes part of the wider system. You’re part of the system, your users are part of the system, and every other employee & department & priority eventually forms part of that system. And that can make it incredibly difficult to make changes, or even to understand which changes will actually matter.

That might be overwhelming, but there's hope. Understanding how systems work and how to improve them is something that can be learnt, and improved at. So in the pursuit of better systems understanding, I’m joined this week by Diana Montalion, coder, architect, and author of Learning Systems Thinking. She takes us through how she sees systems, and how we can get better at discovering and understanding our own systems, as we both chew through some difficult systems we’ve worked on in the hope of figuring out how to do it better next time…

–

Learning Systems Thinking (book): https://www.amazon.co.uk/Learning-Systems-Thinking-Essential-Professionals-ebook/dp/B0D9KWZRT2

Diana’s Website: https://montalion.com/

Scientific Management (Tailorism): https://en.wikipedia.org/wiki/Scientific_management

Jay Wright Forrester: https://en.wikipedia.org/wiki/Jay_Wright_Forrester

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Diana on LinkedIn: https://www.linkedin.com/in/dianamontalion/

Diana on YouTube: https://www.youtube.com/@dianamontalion

Kris on BlueSky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

To kick off 2025 we’re looking at Fyrox a game engine built in Rust, largely by one person - Dmitry Stepanov. For an individual project, it’s covered an incredible amount of ground, covering the rendering and animation features you’d expect from a game engine, with some features that might surprise you - like Rust scripting support with hot-reloading.

As we dive into Fyrox, Dmitry explains what it takes to build a game engine, why he chose Rust (and why he’s happy with the choice), and how one person can hope to build a project of that size.

–

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Fyrox Homepage: https://fyrox.rs/

The Fyrox Book: https://fyrox-book.github.io/

Rapier Physics Engine: https://rapier.rs/

The Mine (on Steam): https://store.steampowered.com/app/898980/The_Mine/

Dmitry’s Engine: https://github.com/mrDIMAS/DmitrysEngine

GJK Collision Detection Algorithm: https://en.wikipedia.org/wiki/Gilbert%E2%80%93Johnson%E2%80%93Keerthi_distance_algorithm

WPF: https://en.wikipedia.org/wiki/Windows_Presentation_Foundation

PICO-8: https://www.lexaloffle.com/pico-8.php

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins