October 24, 2015

Kay Hayen on Nuitka

1 hour 34 minutes

Visit our site to listen to past episodes, support the show, and sign up for our mailing list.

Summary

Kay Hayen is a systems engineer from Germany who has dedicated his spare time to the creation of Nuitka, a library that will compile your Python project to C++. In this episode we talked to Kay about what inspired him to create the project, how it operates, and some of the challenges he has faced. It is a very interesting project and it has the potential to let you run your Python code in a whole new way!

Brief Introduction

Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.

Subscribe on iTunes, Stitcher, TuneIn or RSS

Give us feedback! Leave a review on iTunes, Tweet to us, send us an email, leave us a message on Google+, or leave a comment on our show notes

I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com

I would also like to thank Hired, a job marketplace for developers, for sponsoring this episode of Podcast.__init__. Use the link hired.com/podcastinit to double your signing bonus. Linode has also sponsored this episode and you can get a $10 credit using the link linode.com/podcastinit to try out their fast and reliable linux virtual servers.

We are recording today on October 6th, 2015 and your hosts as usual are Tobias Macey and Chris Patti

Today we are interviewing Kay Hayen about the Nuitka project

On Hired software engineers & designers can get 5+ interview requests in a week and each offer has salary and equity upfront. With full time and contract opportunities available, users can view the offers and accept or reject them before talking to any company. Work with over 2,500 companies from startups to large public companies hailing from 12 major tech hubs in North America and Europe. Hired is totally free for users and If you get a job you’ll get a $2,000 “thank you” bonus. If you use our special link to signup, then that bonus will double to $4,000 when you accept a job. If you’re not looking for a job but know someone who is, you can refer them to Hired and get a $1,337 bonus when they accept a job.

Use the promo code podcastinit10 to get a $10 credit when you sign up!

Interview with Kay Hayen

Introductions

German, family with 2 kids, one cat

Working in ATM (Air Traffic Management), tracker product

Systems Engineer

Nuitka as a hobbyist

How did you get introduced to Python?

Once was Perl “Guru”.

Python was getting a lot of positive press

Team decision to want to use readable stuff

CPAN was still more complete, but Python was making inroads

Can you describe how to pronounce the name of your project?

Wife Anna, Russian, Annuitka -> Nuitka

Can you briefly describe what Nuitka is and what your motivation was for creating it?

I was thinking a fully integrated and compatible compiler should be possible.

Why is nobody doing it?

I can do it.

I am doing it.

Take Python beyond current use cases.

Everbody currently using Python needs no compiler, or wouldn’t use it

Less need for time consuming C++/Python hybrid coding

Simple code should compile to fast code by default

Complex code should still work

On the project web site it says that Nuitka does a lot of clever things after being fed a Python project. Can you provide some details as to what some of that cleverness is?

Re-formulations of Python into simpler Python

No “class”

No “assert”

No complex assignments

SSA tracing

Attaching uses to assignments properly

Despite try/finally

Loops

Avoids checks for known defined/undefined values

Function inlining (coming)

Constant propagation

Closure variable removal

What is libpython and how is it used in both Nuitka and CPython?

Core of the Python interpreter

With Python VM and C interface

Nuitka can fall back to it

Avoiding it as often as we can, key to performance

Is there any way to provide hints to Nuitka to generate more optimized output?

Nuitka is yet to make a difference based on type information

Not yet there, but coming soonish. SSA was pre-requisite

PEP 484 will be unreliable type information, mostly useless

I want type hints that are checked at Python run time

What are some of the biggest challenges in generating statically compiled code from a language as dynamic as Python?

Python is compiled to .pyc files

Compatible Frame stack, cached

Exception handling of Python is terrible

CPython type system designed to be extensible

Extension types for functions, bound/unbound methods, generators, etc.

Many details to get right

Are there any particular Python constructs that Nuitka is unable to translate and as a corollary to that is the compilation step lossy at all or do you have some way of ensuring that the functionality of the program remains unaltered?

Big point, no price attached

Except for not having bytecode, there is nothing missing

No pdb support

Edit / run cycle is not accelerated

That said: PyQt (integrated), PySide (available, unmerged), wxPython (available, maybe merged) needed patches to take compiled function/method objects for function objects too

Are there any particular types of programs that benefit the most from Nuitka’s compilation?

Bindings with ctypes of cffi compile into zero overhead C calls (planned)

Scientific programs are the most obvious goal (float type inference)

CPU bound or low latency programs

Is it possible to feed an entire project with multiple modules into Nuitka all at once or is the standard use to perform compilation one source file or submodule at a time?

You give it the main program and it recurses imports according to “PYTHONPATH”

nuitka –recurse-all “/usr/bin/hg” supposed to work

Might have to give directories with program plug-ins

I’m curious about what led you to choose compilation to C++ for Nuitka rather than making Nuitka an LLVM back end like Numba?

When I started Nuitka, I was using C++0x and variadic templates

Wanted to make a proof of concept that compatibility and integration is feasible

From there, code generation got less high level to goto ridden C

How does Nuitka compare to projects like Numba or Cython?

Graceful degradation goal

Complete compatibility with Python whole stack

How does Nuitka compare to PyPy? – Kay

PyPy is the coolest project ever

Pure Python goals shared

How can users evaluate the performance of Nuitka – Kay

They currently cannot

Developing a tool to compare CPython and Nuitka runs

Based on vmprof from PyPy people

Identify parts of program where Nuitka is slower

Links to source code

To be done, help needed.

Nuitka is only starting to get to serious performance

Compatibility is such a high bar to take

C++ to C took a year (avoiding C++ exceptions)

SSA literally took forever

Picks

Tobias

Forbidden Island

Forbidden Desert

Otto Project

Chris

Grimm Super Symmetry

Are You Listening To?: Boston

Ripple

Kay

Learn being skeptic, Atheist Experience

MicroPython

Keep In Touch

Nuitka Homepage

Google+

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

...more

View all episodes

By Tobias Macey

4.4

100100 ratings

October 24, 2015

Kay Hayen on Nuitka

1 hour 34 minutes

Visit our site to listen to past episodes, support the show, and sign up for our mailing list.

Summary

Brief Introduction

Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.

Subscribe on iTunes, Stitcher, TuneIn or RSS

Give us feedback! Leave a review on iTunes, Tweet to us, send us an email, leave us a message on Google+, or leave a comment on our show notes

I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com

We are recording today on October 6th, 2015 and your hosts as usual are Tobias Macey and Chris Patti

Today we are interviewing Kay Hayen about the Nuitka project

Use the promo code podcastinit10 to get a $10 credit when you sign up!

Interview with Kay Hayen

Introductions

German, family with 2 kids, one cat

Working in ATM (Air Traffic Management), tracker product

Systems Engineer

Nuitka as a hobbyist

How did you get introduced to Python?

Once was Perl “Guru”.

Python was getting a lot of positive press

Team decision to want to use readable stuff

CPAN was still more complete, but Python was making inroads

Can you describe how to pronounce the name of your project?

Wife Anna, Russian, Annuitka -> Nuitka

Can you briefly describe what Nuitka is and what your motivation was for creating it?

I was thinking a fully integrated and compatible compiler should be possible.

Why is nobody doing it?

I can do it.

I am doing it.

Take Python beyond current use cases.

Everbody currently using Python needs no compiler, or wouldn’t use it

Less need for time consuming C++/Python hybrid coding

Simple code should compile to fast code by default

Complex code should still work

On the project web site it says that Nuitka does a lot of clever things after being fed a Python project. Can you provide some details as to what some of that cleverness is?

Re-formulations of Python into simpler Python

No “class”

No “assert”

No complex assignments

SSA tracing

Attaching uses to assignments properly

Despite try/finally

Loops

Avoids checks for known defined/undefined values

Function inlining (coming)

Constant propagation

Closure variable removal

What is libpython and how is it used in both Nuitka and CPython?

Core of the Python interpreter

With Python VM and C interface

Nuitka can fall back to it

Avoiding it as often as we can, key to performance

Is there any way to provide hints to Nuitka to generate more optimized output?

Nuitka is yet to make a difference based on type information

Not yet there, but coming soonish. SSA was pre-requisite

PEP 484 will be unreliable type information, mostly useless

I want type hints that are checked at Python run time

What are some of the biggest challenges in generating statically compiled code from a language as dynamic as Python?

Python is compiled to .pyc files

Compatible Frame stack, cached

Exception handling of Python is terrible

CPython type system designed to be extensible

Extension types for functions, bound/unbound methods, generators, etc.

Many details to get right

Big point, no price attached

Except for not having bytecode, there is nothing missing

No pdb support

Edit / run cycle is not accelerated

That said: PyQt (integrated), PySide (available, unmerged), wxPython (available, maybe merged) needed patches to take compiled function/method objects for function objects too

Are there any particular types of programs that benefit the most from Nuitka’s compilation?

Bindings with ctypes of cffi compile into zero overhead C calls (planned)

Scientific programs are the most obvious goal (float type inference)

CPU bound or low latency programs

Is it possible to feed an entire project with multiple modules into Nuitka all at once or is the standard use to perform compilation one source file or submodule at a time?

You give it the main program and it recurses imports according to “PYTHONPATH”

nuitka –recurse-all “/usr/bin/hg” supposed to work

Might have to give directories with program plug-ins

I’m curious about what led you to choose compilation to C++ for Nuitka rather than making Nuitka an LLVM back end like Numba?

When I started Nuitka, I was using C++0x and variadic templates

Wanted to make a proof of concept that compatibility and integration is feasible

From there, code generation got less high level to goto ridden C

How does Nuitka compare to projects like Numba or Cython?

Graceful degradation goal

Complete compatibility with Python whole stack

How does Nuitka compare to PyPy? – Kay

PyPy is the coolest project ever

Pure Python goals shared

How can users evaluate the performance of Nuitka – Kay

They currently cannot

Developing a tool to compare CPython and Nuitka runs

Based on vmprof from PyPy people

Identify parts of program where Nuitka is slower

Links to source code

To be done, help needed.

Nuitka is only starting to get to serious performance

Compatibility is such a high bar to take

C++ to C took a year (avoiding C++ exceptions)

SSA literally took forever

Picks

Tobias

Forbidden Island

Forbidden Desert

Otto Project

Chris

Grimm Super Symmetry

Are You Listening To?: Boston

Ripple

Kay

Learn being skeptic, Atheist Experience

MicroPython

Keep In Touch

Nuitka Homepage

Google+

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

...more

More shows like The Python Podcast.init

View all

Software Engineering Radio - the podcast for professional software developers

272 Listeners

The Changelog: Software Development, Open Source

284 Listeners

Data Skeptic

480 Listeners

Talk Python To Me

590 Listeners

Software Engineering Daily

621 Listeners

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

441 Listeners

Super Data Science: ML & AI Podcast with Jon Krohn

297 Listeners

Python Bytes

215 Listeners

Machine Learning Guide

763 Listeners

Home Assistant Podcast

65 Listeners

Syntax - Tasty Web Development Treats

987 Listeners

DataFramed

267 Listeners

Practical AI

192 Listeners

The Real Python Podcast

139 Listeners

中国好生意

2 Listeners

Share Kay Hayen on Nuitka

Sign up to save your podcasts

Kay Hayen on Nuitka

Kay Hayen on Nuitka

More shows like The Python Podcast.init

Software Engineering Radio - the podcast for professional software developers

The Changelog: Software Development, Open Source

Data Skeptic

Talk Python To Me

Software Engineering Daily

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

Super Data Science: ML & AI Podcast with Jon Krohn

Python Bytes

Machine Learning Guide

Home Assistant Podcast

Syntax - Tasty Web Development Treats

DataFramed

Practical AI

The Real Python Podcast

中国好生意