Speed Up Your Python Data Applications By Parallelizing Them With Bodo

Summary

The speed of Python is a subject of constant debate, but there is no denying that for compute heavy work it is not the optimal tool. Rather than rewriting your data oriented applications, or having to rearchitect them, the team at Bodo wrote a compiler that will do the optimization for you. In this episode Ehsan Totoni explains how they are able to translate pure Python into massively parallel processes that are optimized for high performance compute systems.

Announcements

Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.

When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!

Your host as usual is Tobias Macey and today I’m interviewing Ehsan Totoni about Bodo, an inferential compiler for Python that automatically parallelizes your data oriented projects

Interview

Introductions

How did you get introduced to Python?

Can you describe what Bodo is and the story behind it?

What are some of the use cases that it is being applied to?

What are the motivating factors for something like Dask or Ray as compared to Bodo?

What are the software patterns that contribute to slowdowns in data processing code?

What are some of the ways that the compiler is able to optimize those operations?

Can you describe how Bodo is implemented?

How does Bodo process the Python code for compiling to the optimized form?

What are the compilation techniques for understanding the semantics of the code being processed?

How do you manage packages that rely on C extensions?

What do you use as an intermediate representation for translating into the optimized output?

What is the workflow for applying Bodo to a Python project?

What debugging utilities does it provide for identifying any errors that occur due to the added parallelism?

What kind of support does Bodo have for optimizing a machine learning project with Bodo? (e.g. using PyTorch/Tensorflow/MxNet/etc.)

When working with a workflow orchestrator such as Dagster for Airflow, what would the integration process look like for being able to take advantage of the optimized Bodo output?

What are the most interesting, innovative, or unexpected ways that you have seen Bodo used?

What are the most interesting, unexpected, or challenging lessons that you have learned while working on Bodo?

When is Bodo the wrong choice?

What do you have planned for the future of Bodo?

Keep In Touch

LinkedIn

@EhsanTn on Twitter

ehsantn on GitHub

Picks

Tobias

Paracord Crafts

Ehsan

[

Closing Announcements

Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.

Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.

If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.

To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Links

Bodo

Data Engineering Podcast Episode

University of Illinois Urbana-Champaign

HPC

MPI

Elastic Fabric Adapter

All-to-All Communication

Dask

Data Engineering Podcast Episode

Ray

Podcast Episode

Pandas Extension Arrays

Podcast Episode

GeoPandas

Numba

LLVM

scikit-learn

Horovod

Dagster

Podcast.__init__ Episode

Data Engineering Podcast Episode

Airflow

Podcast Episode

IPython Parallel

Parquet

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA