Developer Voices

SQLite could do with a little competition, so when I invited the co-creator of DuckDB in to talk, I thought we'd be discussing the perils of trying to build a new in-process database engine. I quickly realised things went much deeper than just a tech refresh.

Hannes Mühleisen joins me this week to blend his academic credentials as a database researcher with his vehement need to make that research practical. And so we dive into what modern database literature has to say on making queries faster, more parallelizable, and closer to the metal, and how it all comes together in a user-friendly package that’s found its way into my day-to-day workload, and might well help out yours.

If you’re curious about the gory details of database queries, how they can take advantage of modern hardware, or how all that research actually turns into a useful tool, Hannes has some great answers.

--

DuckDB: https://duckdb.org/

Database Systems Book: http://infolab.stanford.edu/~ullman/dscb.html

Kris’ first computer: https://en.wikipedia.org/wiki/File:ZX_Spectrum_Plus2_(retouched).jpg

Volcano Query Evaluation System [pdf]: https://paperhub.s3.amazonaws.com/dace52a42c07f7f8348b08dc2b186061.pdf

Morsel Query Engine [pdf]: https://cs.brown.edu/~kayhan/papers/morsel_cp.pdf

Unnesting Arbitrary Queries [pdf]: https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf

Papers Hannes' team have published: https://duckdb.org/why_duckdb#peer-reviewed-papers-and-thesis-works

DuckDB on Mastodon: https://mastodon.social/@duckdb

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Mastodon: https://mastodon.social/@krisajenkins

--

#softwaredevelopment #podcast #programming #database #duckdb #sql #sqlite

This week we talk to Simon Peyton Jones, a veteran language designer and researcher, and key figure in the development of Haskell. Haskell. Simon has made countless contributions to advancement of functional programming, and computer programming in general, and is currently working at Epic Games, working on the foundations of their new programming language, Verse.

We discuss how programming languages are made, focussing on a big design idea from both Haskell and Verse: building a large language from a small, tightly designed core. Then we move into Simon's current work exploring Functional Logic Programming, the big new idea that underpins Verse. It's an idea that blends the fundamentals FP with the core ideas of logic languages like Prolog in an entirely new way. Not even Simon knows exactly where the idea will lead, but it's a fascinating idea that could potentially bring constraint-solving and deduction right into the heart of modern software.

Additionally, Simon discusses his involvement in reshaping the way we teach computing in England. He's been working hard to give computing education the same importance as the teaching of mathematics and sciences - something we should all have a fundamental understanding of.

Simon's one of the smartest, nicest people in programming. Come as hear his brilliant brain at work. :-D

–

Verse: https://github.com/UnrealVerseGuru/VerseProgrammingLanguage

The Verse Language Reference: https://dev.epicgames.com/documentation/en-us/uefn/verse-language-reference

The Verse Calculus [pdf]: https://simon.peytonjones.org/assets/pdfs/verse-March23.pdf

https://en.wikipedia.org/wiki/Simon_Peyton_Jones

The LogicT monad: https://hackage.haskell.org/package/logict

Can programming be liberated from the von Neumann style?: https://dl.acm.org/doi/10.1145/359576.359579

CAS - Computing At School: https://www.computingatschool.org.uk/

Computer Science Teachers Association: https://csteachers.org/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Benthos wants to be part of your Data Engineering toolkit - it’s there as a quick and easy way to set up data pipelines and start streaming data out of A and into B. In contrast to a lot of the tools we’ve talked about on Developer Voices, Benthos seems focussed on cutting development time down to a minimum, so you can quickly configure a new pipeline and test it out, without making a whole sprint of the task. As quick as a quick-and-dirty shell script, without the dirt. 😉

So this week we’re talking to the creator of Benthos, Ashley Jeffs, to hear why he created Benthos, what it can do for you, and what its strengths and weaknesses are. And Jeff’s refreshingly candid about when you should and shouldn’t use it. If you ever need to get data from an HTTP connection into S3, or S3 into Kafka, or Kafka into a flat file, Benthos might just save you a few hours of development.

–

Benthos: https://www.benthos.dev/

A list of supported inputs, processors & outputs: https://www.benthos.dev/docs/about#components

All their cute blobfish logos: https://www.benthos.dev/blobfish/

IDML: https://idml.io/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

–

#software #podcast #dataengineering #datascience

The world of game programming might seem a million miles away from 'regular' programming. But they still have to deal with the same kinds of data, scale and concurrency problems that we’re all familiar with in the software world. And that makes the gaming world an interesting place for new ideas - under the hood they’re solving those same problems we face, but often with some novel ideas about the solutions.  So this week we’re off to the massive open world that is game development, to see what we can learn that might make lives easier in the non-gaming space. Joining us for that is Tyler Cloutier, the founder of Clockwork Labs. They’re building SpaceTimeDB, a curiously-distributed database built to be the underlying platform for their new MMORPG, BitCraft. Digging down into the architecture of SpaceTimeDB, we pick Tyler’s brain for nuggets of information on event sourcing, request/response vs. subscriptions, transactions, security and much more. All in an effort to make our programmers and data scientists’ lives easier.

--

SpaceTimeDB: https://spacetimedb.com/

BitCraft: https://bitcraftonline.com/

“4X games” defined: https://en.wikipedia.org/wiki/4X

Plan 9 O.S.: https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs

Tyler on LinkedIn: https://www.linkedin.com/in/tylercloutier/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Mastodon: https://mastodon.social/@krisajenkins

--

#programming #podcast #softwaredevelopment #software #gamedev #gamedevelopment

Odin’s creator, Bill Hall, makes some bold claims about the language, including that it’s “programming done right”. Before that starts a war on the internet, we’d best ask him to explain what that means, and how Odin tries to achieve it. And while we get deep into the details, overall his answer seems to be, “By gathering masses of feedback and then refining C until it feels joyous again.

Of all the C-like languages we’ve looked at on Developer Voices, Odin seems to be the most at-ease with its progenitor. It’s not trying to be a revolutionary new way of thinking about systems programming; it’s just trying to rethink C for modern conventions. If Bill’s hit his goals, it might be the most comfortable way to get a language that’s C, but C done better…

–

Odin: https://odin-lang.org/

Odin Packages: https://pkg.odin-lang.org/

Newsqueak [pdf]: https://swtch.com/~rsc/thread/newsqueak.pdf

EmberGen: https://jangafx.com/software/embergen/

Raylib: https://www.raylib.com/

RayLib bindings for Odin: https://github.com/odin-lang/Odin/tree/master/vendor/raylib

Verse language: https://dev.epicgames.com/documentation/en-us/uefn/verse-language-reference

Algorithms + Data Structures = Programs: https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures_%3D_Programs

Bill on Twitter: https://twitter.com/TheGingerBill

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

--

#podcast #software #softwareprogramming #programming #odin #odinlang

This week’s guest describes Event Sourcing as, “all I’m going to use for the rest of my career.” But what is Event Sourcing? How should we think about it, and how does it encourage us to think about writing software?

In this episode we take a close look at systems designed around the idea of Events, with guest Bobby Calderwood. Bobby’s been designing (and helping others design) event based architectures for many years, and enthusiastically recommends it not only as a system-design technique, but as a way of solving business problems faster and more reliably.

During this discussion we look at the various ways of defining event systems, what tools we need to implement them, and the advantages of thinking about software from an event-based perspective. Along the way we discuss everything from Clojure, Bitemporality & Datomic to Kafka and more traditional databases - all in the service of capturing real-world events and building simple systems around them.

–

EventStoreDB: https://developers.eventstore.com/

The CloudEvents standard: https://cloudevents.io/

Datomic: https://www.datomic.com/

Adam Dymitruk’s Event Modelling Explanation: https://eventmodeling.org/

Bobby’s Event Modelling course: https://developer.confluent.io/courses/event-modeling/intro/

Bobby on Twitter: https://twitter.com/bobbycalderwood

Boddy on LinkedIn: https://www.linkedin.com/in/bobbycalderwood/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

–

#software #softwarepodcast #programming #eventsourcing #eventdrivenarchitecture #kafka

One of our oldest languages meets one of our newest sciences in this episode, as we talk with Professor Christian Schafmeister, an award-winning nanotech researcher who's been developing a language and a design suite to help research the future molecular machines.

In this episode Christian gives us a quick chemistry lesson to explain what his research is trying to achieve, then we get into the software that's doing it: A new flavour of Common Lisp. But why Lisp? What advantages does a 60 year old language design offer? How does he strike a balance between high-level language features and the need for exceptional performance and parallelism?  And what tricks does his development environment have that modern IDEs could still learn a thing or two from?

--

Clasp (the Lisp): https://github.com/clasp-developers/clasp

Cando (the design language): https://github.com/cando-developers/cando

The Feynman Prize: https://en.wikipedia.org/wiki/Feynman_Prize_in_Nanotechnology

Alphafold: https://alphafold.ebi.ac.uk/

More on LEaP: https://ambermd.org/tutorials/pengfei/

Interactive Development of Crash Bandicoot: https://all-things-andy-gavin.com/2011/03/12/making-crash-bandicoot-gool-part-9/ 

Christian's Research Group: https://www.schafmeistergroup.com/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

--

#programming #software #lisplang #commonlisp #nanotech

Sometimes, what a programming language makes harder is just as important as what it makes easier. For a simple example, think of GOTO. We’ve been wisely avoiding it for decades because it makes confusing control flow desperately easy. Types and tests are other examples - they’re as much about specifying what shouldn’t work as what should. And perspective is what makes this week’s topic particularly interesting: Roc is a language that’s functional, fast, friendly, and extremely interested in making your life easier by enabling some possibilities and restricting others.

So this week we’re joined by Richard Feldman, the creator of Roc. He’s been an advocate of the Elm programming language for years, for its tight focus on taking the best bits of Functional Programming to the browser. And in recent years he’s been inspired to build his own language, taking that philosophy to other places and platforms.

But which bits are “the best bits”? And how do they change when the domain you’re coding for changes? How is Roc built and how would we build systems in it? Let’s find out…

--
Roc’s homepage: https://www.roc-lang.org/

Richard’s GOTO Copenhagen 2021 talk: https://www.youtube.com/watch?v=3n17wHe5wEw

Richard on Twitter: https://twitter.com/rtfeldman

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Mastodon: https://mastodon.social/@krisajenkins

Kris on Twitter: https://twitter.com/krisajenkins

One of the recurring themes in the big data & data streaming worlds at the moment is developer experience. It seems like every major tool is trying to answer this question: how do we make large-scale data processing feel trivial?

In some places the answer is any library you like as long as it’s Python. In other realms, a mixture of Java and SQL shows promise. But as this week’s guest—Luca Pette—would say, the Unix design metaphor has plenty to give and keep on giving.

So in this episode of Developer Voices we look at TypeStream - his Kotlin project that provides a shell-like interface to data pipelines, and is gradually expanding to make integration pipelines as simple as `cat /dev/kafka | tee /dev/postgres`.

--

Luca on Twitter: https://twitter.com/lucapette

Luca on LinkedIn: https://www.linkedin.com/in/lucapette/

Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

TypeStream homepage: https://www.typestream.io/

TypeStream installation guide: https://docs.typestream.io/tutorial/installation

Crafting interpreters: https://craftinginterpreters.com/

…by Bob Nystrom: https://twitter.com/munificentbob

NuShell: https://github.com/nushell/nushell

#podcast #apachekafka #bigdata

This week we're back on systems programming with Hare. A C-like language for the ages. We talk to its creator, Drew DeVault, about what he thinks we can learn from the past 50 years of programming, and how we can build that hindsight into a new language that will last for the next 100. 

In among all that long-term ambition we talk cover everything from error handling, typed unions and linear types, to metaprogramming and Drew's microkernel operating system. It's called Ares, and it is, of course, built in Hare.

--

Drew's Homepage: https://drewdevault.com/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ 

A summary of Hare’s features: https://harelang.org/tutorials/introduction/

Hare Community Resources: https://harelang.org/community/

SXMO Mobile: https://sxmo.org/

QBE Compiler Backend: https://c9x.me/compile/users.html

Ares OS Source Code: https://sr.ht/~sircmpwn/helios/

OSDev Wiki: https://wiki.osdev.org/Expanded_Main_Page

The Ares System [pdf]: https://mirror.drewdevault.com/ares.pdf

#programming #podcast #harelang #qbe #microkernel