One of the recurring themes in the big data & data streaming worlds at the moment is developer experience. It seems like every major tool is trying to answer this question: how do we make large-scale data processing feel trivial?

In some places the answer is any library you like as long as it’s Python. In other realms, a mixture of Java and SQL shows promise. But as this week’s guest—Luca Pette—would say, the Unix design metaphor has plenty to give and keep on giving.

So in this episode of Developer Voices we look at TypeStream - his Kotlin project that provides a shell-like interface to data pipelines, and is gradually expanding to make integration pipelines as simple as `cat /dev/kafka | tee /dev/postgres`.

--

Luca on Twitter: https://twitter.com/lucapette

Luca on LinkedIn: https://www.linkedin.com/in/lucapette/

Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

TypeStream homepage: https://www.typestream.io/

TypeStream installation guide: https://docs.typestream.io/tutorial/installation

Crafting interpreters: https://craftinginterpreters.com/

…by Bob Nystrom: https://twitter.com/munificentbob

NuShell: https://github.com/nushell/nushell

#podcast #apachekafka #bigdata

This week we're back on systems programming with Hare. A C-like language for the ages. We talk to its creator, Drew DeVault, about what he thinks we can learn from the past 50 years of programming, and how we can build that hindsight into a new language that will last for the next 100. 

In among all that long-term ambition we talk cover everything from error handling, typed unions and linear types, to metaprogramming and Drew's microkernel operating system. It's called Ares, and it is, of course, built in Hare.

--

Drew's Homepage: https://drewdevault.com/

Kris on Twitter: https://twitter.com/krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ 

A summary of Hare’s features: https://harelang.org/tutorials/introduction/

Hare Community Resources: https://harelang.org/community/

SXMO Mobile: https://sxmo.org/

QBE Compiler Backend: https://c9x.me/compile/users.html

Ares OS Source Code: https://sr.ht/~sircmpwn/helios/

OSDev Wiki: https://wiki.osdev.org/Expanded_Main_Page

The Ares System [pdf]: https://mirror.drewdevault.com/ares.pdf

#programming #podcast #harelang #qbe #microkernel

A few months ago, Michael Drogalis quit his job and decided launch 4 viable startup business ideas in 4 months, publically documenting every step of the journey. Over here at Developer Voices it seemed fun, inspired, and just crazy enough to work.

We had him on the podcast a few months back just as that journey was beginning, and since he launched his first startup things have changed,. The reception has been better than he expected and the plan has been updated to go all-in on idea number one. But why? What's changed? What happened between brainstorming 4 ideas and launching #1 into the world? How is he figuring out what problems to solve, and how is he coping with the workload of being a solopreneur with a business idea and only himself to rely on?

It's definitely time for an update, and to see what we can learn from a fellow geek who wants to start a business, but most of all wants to build technology that people find useful and valuable. Let's hope he succeeds...

--

ShadowTraffic: https://shadowtraffic.io/
Michael’s Previous Appearance: https://youtu.be/jqS2TbxssQE
Follow Michael’s journey: https://michaeldrogalis.substack.com/
Michael on LinkedIn: https://www.linkedin.com/in/michael-drogalis-01029924/
Michael on Twitter: https://twitter.com/MichaelDrogalis
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

--

#podcast #softwareprogramming #programming #startup #technology #kafka #postgres #shadowtraffic #entrepreneur

Integration is probably the last, hardest, and least well thought-out part of any large software project. So anything that makes the data-streaming job easier is worth knowing about. So this week we turn our attention to Apache Flink, a flexible system for grabbing, transforming and shipping data between systems using Java, Python or good ol’ SQL.

So this week Robert Metzger—Apache Flink expert and PMC member—joins us to explain what problems Flink solves and how it solves them reliably. We cover the range from simple use cases to realtime aggregations & joins to its high availability strategy.

If you’re working on systems that include more than one database, then you’re definitely going to face the kinds of problems that Flink tackles.

--

Apache Flink: https://flink.apache.org/
Robert on Twitter: https://twitter.com/rmetzger_
Robert on LinkedIn: https://www.linkedin.com/in/metzgerrobert/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

–

#software #programming #podcast #flink #apacheflink #dataintegration

Zig is a programming language that’s attempting to become “the new C” - the language of choice for low-level systems programming and embedded hardware. Going into that space not only puts it in competition with C and C++, but also other newcomers like Rust and Go. So what makes Zig special?

Joining us to discuss it is Loris Cro from the Zig Foundation. We talk through Zig’s reasons to exist, its language design features, which parts of the C ecosystem it's tackling, and how the Zig Foundation is set up for the long-term health of the language.

–

Loris’ website: https://kristoff.it/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Zig homepage: https://ziglang.org/
The “learn zig” guide: https://ziglearn.org/
Learn Zig with Ziglings: https://ziglings.org/
Find the Zig community: https://github.com/ziglang/zig/wiki/Community
Rust’s cargo-zigbuild: https://github.com/rust-cross/cargo-zigbuild
Using zig as a better linker: https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html
"The Economics of Programming Languages" by Evan Czaplicki (Strange Loop 2023) - https://youtu.be/XZ3w_jec1v8

–

#programming #programminglanguages #software #zig #llvm #rust #go

Would you ever take on a rewrite of one of the largest and most popular Apache projects? And if so, what would you keep the same and what would you change?

This week we’re talking to Christina Lin, who’s part of Redpanda, a company that’s rewriting parts of the Apache Kafka ecosystem in C++, with the aim of getting performance gains that aren’t feasible in Java. It seems like a huge mountain to climb, and a fascinating journey to be on, so let’s ask why and how they’ve taken on this challenge…

Christina on Twitter: https://twitter.com/Christina_wm
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Redpanda: https://redpanda.com/
Redpanda University: https://university.redpanda.com/
Seestar framework: https://seastar.io/
Apache Flink: https://flink.apache.org/

#redpanda #kafka #apachekafka #streaming #python

This week we’re looking at Debezium - an open source project that taps into a huge number of databases and lets you stream data to other systems in real time. It’s a huge project that covers a wide range of uses: Some people use it to replicate from Oracle to MySQL, others to do smart cache invalidation, and others to build a bridge from an existing relational database to the event-sourcing world. If you’re working on a system that has more than one kind of database, it may be an essential tool. But what exactly does it do, and how does it do it?

Joining us for a deep dive is Debezium expert and former project lead, Gunnar Morling. He takes us through all things Debezium, from who uses Debezium and why; which systems you can connect to and what data you get out from it; and how a project of this scope is developed.

Gunnar on Twitter: https://twitter.com/gunnarmorling
Gunnar on LinkedIn: https://www.linkedin.com/in/gunnar-morling-2b44b7229/
Gunnar’s blog: https://www.morling.dev/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Gunnar’s talk on the new incremental snapshot mechanism: https://events.bizzabo.com/468544/agenda/session/1136877
Getting into Real Time data: https://youtu.be/ftAVFxa5AwI
Stripe’s talk at Flink Forward: https://www.slideshare.net/FlinkForward/squirreling-away-640-billion-how-stripe-leverages-flink-for-change-data-capture
Francesco’s “When JDBC Goes Wrong” talk: https://www.confluent.io/events/kafka-summit-london-2022/jdbc-source-connector-what-could-go-wrong/

#debezium #microservicesarchitecture #cdc #database #postgresql #oracle #mysql #eventsourcing #kafka #jdbc #realtime

Ever read a bad README? We all have, and most of the time, we’ve just moved right along. A programmer that can’t communicate their ideas will find no-one uses their software. And that’s true even outside of the open-source world. The best software doesn’t win - the best software _that people can understand_ wins. So how do we get better at communicating our code? What do we talk about when we talk about software?

Joining to discuss that question is a data-streaming expert and skilled communicator, Francesco Tisiot. Unusually, this episode is recorded on location, as we met up in the hallway of a recent tech conference.

Francesco on Twitter: https://twitter.com/FTisiot
Francesco on LinkedIn: https://www.linkedin.com/in/francescotisiot/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

#podcast #podcasts #devrel #opensource #software #presentations

As interesting and useful as LLMs (Large Language Models) are proving, they have a severe limitation: they only know about the information they were trained on. If you train it on a snapshot of the internet from 2023, it’ll think it’s 2023 forever. So what do you do if you want to teach it some new information, but don’t want to burn a million AWS credits to get there?

In exploring that answer, we dive deep into the world of semantic search, augmented LLMs, and exactly how vector databases bridge that gap from the old dog to the new tricks. Along the way we’ll go from an easy trick to teach ChatGPT some new information by hand, all the way down to how vector databases store documents by their meaning, and how they efficiently search through those meanings to give custom, relevant answers to your questions.

--

Zain on Twitter: https://twitter.com/zainhasan6
Zain on LinkedIn: https://www.linkedin.com/in/zainhas
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
HNSW Paper: https://arxiv.org/abs/1603.09320
ImageBind - One Embedding Space To Bind Them All (pdf): https://openaccess.thecvf.com/content/CVPR2023/papers/Girdhar_ImageBind_One_Embedding_Space_To_Bind_Them_All_CVPR_2023_paper.pdf
Weaviate: https://weaviate.io/
Source: https://github.com/weaviate/weaviate
Examples: https://github.com/weaviate/weaviate-examples
Community Links: https://forum.weaviate.io/ and https://weaviate.io/slack

--

#vectordb #vectordatabase #semanticsearch #openai #chatgpt #weaviate #knn

Real-time data is gradually becoming a standard requirement in systems design. Our customers are beginning to demand it, our colleagues in other departments are starting to expect it. Whether you’re letting people book a taxi, recommending their next binge-watch, or delivering business reports to management, faster data is just obviously better. Or is it?

Does real-time data matter everywhere, or does it just have sweet spots in some sectors and some use-cases? Is it a cost-benefit question - is the idea great in theory, but still too hard to adopt in practice? Would everyone be streaming their data live if streaming their data live was easier? If the future of data is, “now, not later,” then what’s holding that future back?

In this week’s Developer Voices we talk to Thomas Camp of Ably, and chew through the use-cases, software stacks, and education needed to speed up the way we process data. We consider everything from the front-end to the back, from user experience to business needs, and from greenfield projects to incrementally adapting existing systems.

If you’re wondering what all the batch vs. streaming fuss is about, or you want to know how you can drag the industry there sooner, we have some answers. It’ll only take an hour. 😉

--

Thomas on LinkedIn: https://www.linkedin.com/in/thomascamp333/
Ably: https://ably.com/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/