Stream Processing In Real Time And At Scale In Pure Python With Bytewax

Summary

Analysis of streaming data in real time has long been the domain of big data frameworks, predominantly written in Java. In order to take advantage of those capabilities from Python requires using client libraries that suffer from impedance mis-matches that make the work harder than necessary. Bytewax is a new open source platform for writing stream processing applications in pure Python that don’t have to be translated into foreign idioms. In this episode Bytewax founder Zander Matheson explains how the system works and how to get started with it today.

Announcements

Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.

When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!

The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. Select Star’s data discovery platform solves that out of the box, with a fully automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Just connect it to your dbt, Snowflake, Tableau, Looker, or whatever you’re using and Select Star will set everything up in just a few hours. Go to pythonpodcast.com/selectstar today to double the length of your free trial and get a swag package when you convert to a paid plan.

Need to automate your Python code in the cloud? Want to avoid the hassle of setting up and maintaining infrastructure? Shipyard is the premier orchestration platform built to help you quickly launch, monitor, and share python workflows in a matter of minutes with 0 changes to your code. Shipyard provides powerful features like webhooks, error-handling, monitoring, automatic containerization, syncing with Github, and more. Plus, it comes with over 70 open-source, low-code templates to help you quickly build solutions with the tools you already use. Go to dataengineeringpodcast.com/shipyard to get started automating with a free developer plan today!

Your host as usual is Tobias Macey and today I’m interviewing Zander Matheson about Bytewax, an open source Python framework for building highly scalable dataflows to process ANY data stream.

Interview

Introductions

How did you get introduced to Python?

Can you describe what Bytewax is and the story behind it?

Who are the target users for Bytewax?

What is the problem that you are trying to solve with Bytewax?

What are the alternative systems/architectures that you might replace with Bytewax?

Can you describe how Bytewax is implemented?

What are the benefits of Timely Dataflow as a core building block for a system like Bytewax?

How have the design and goals of the project changed/evolved since you first started working on it?

What are the axes available for scaling Bytewax execution?

How have you approached the design of the Bytewax API to make it accessible to a broader audience?

Can you describe what is involved in building a project with Bytewax?

What are some of the stream processing concepts that engineers are likely to run up against as they are experimenting and designing their code?

What is your motivation for providing the core technology of your business as an open source engine?

How are you approaching the balance of project governance and sustainability with opportunities for commercialization?

What are the most interesting, innovative, or unexpected ways that you have seen Bytewax used?

What are the most interesting, unexpected, or challenging lessons that you have learned while working on Bytewax?

When is Bytewax the wrong choice?

What do you have planned for the future of Bytewax?

Keep In Touch

Slack

Twitter

LinkedIn

Picks

Tobias

Alta Racks

Zander

Atherton Bikes

Links

Bytewax

GitHub

Flink

Data Engineering Podcast Episode

Spark Streaming

Kafka Connect

Faust

Podcast Episode

Ray

Podcast Episode

Dask

Data Engineering Podcast Episode

Timely Dataflow

PyO3

Materialize

Data Engineering Podcast Episode

HyperLogLog

Python River Library

Shannon Entropy Calculation

The blog post using incremental shannon entropy

NATS

waxctl

Prometheus

Grafana

Streamz

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA