Would you move your data science project from a laptop to the cloud? Would you also like to have snapshots of your project saved along the way so that you can go back in time or share the state of your project with another team member? This week on the show, we have Savin Goyal from Netflix. Savin is the technical lead for machine learning infrastructure at Netflix. He joins us to talk about Metaflow, an open-source tool to simplify building, managing, and scaling data science projects.
Metaflow addresses the needs of the numerous data scientists who work at Netflix. Machine learning is key strength for the streaming service. They tried several existing tools to scale their own internal infrastructure and after this experimentation developed Metaflow.  
We talk about the history of the project and how someone could get started with the open-source version. Savin also contrasts the cost of infrastructure as compared to data scientists and the cost of their time.

Course Spotlight: Simplify Python GUI Development With PySimpleGUI
In this step-by-step course, you’ll learn how to create a cross-platform graphical user interface (GUI) using Python and PySimpleGUI. A graphical user interface is an application that has buttons, windows, and lots of other elements that the user can use to interact with your application.

Topics:

00:00:00 – Introduction
00:01:53 – What is Metaflow? 
00:04:15 – Savin’s background in data science and infrastructure
00:06:06 – Democratization of infrastructure and iteration of tools
00:10:34 – What information is saved about the infrastructure requirements for a project?
00:17:17 – How are the requirements annotated?
00:18:39 – Sponsor: Digital Ocean’s App Platform
00:19:15 – How do project snapshots work?
00:29:33 – Cost of infrastructure vs data scientists
00:32:28 – Working with data at Netflix scale
00:37:55 – Video Course Spotlight 
00:39:06 – Getting an organization to use new tools and then making open-source
00:49:51 – Documentation of Metaflow and getting started on solving infrastructure problems
00:53:57 – What made you interested in working on infrastructure tools?
00:55:13 – What is something you are excited about in the world of Python?
00:56:18 – What do you want to learn next?
00:58:14 – Thanks and goodbye

Show Links:

Metaflow: A framework for real-life data science
Metaflow: Tutorials
More Data Science, Less Engineering with Netflix’s Metaflow By Savin Goyal - YouTube
R: The R Project for Statistical Computing
Tidyverse: R packages for data science
Anything you can do, I can do (kinda). Tidyverse pipes in Pandas
reticulate: R Interface to Python
Apache Airflow: Programmatically author, schedule and monitor workflows
Directed acyclic graph (DAG) - Wikipedia article
Serializing Objects With the Python pickle Module - Real Python Course

Level up your Python skills with our expert-led courses:

Learn Text Classification With Python and Keras
Using Jupyter Notebooks
Simplify Python GUI Development With PySimpleGUI
 Support the podcast & join our community of Pythonistas

Scaling Data Science and Machine Learning Infrastructure Like Netflix

A weekly Python podcast hosted by Christopher Bailey with interviews, coding tips, and conversation with guests from the Python community.

The show covers a wide range of topics including Python programming best practices, career tips, and related software development topics.

Join us every Friday morning to hear what's new in the world of Python programming and become a more effective Pythonista.

Technology

Tech News

Podcasting

Gadgets

Software How-To

How do you effectively break a software problem into individual steps? What are signs you’re writing overly clever code? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder’s Weekly articles and projects.
We discuss an article about de-warping images of book pages. We both found the piece a good study on decomposing a complex software problem. 
Christopher discusses an article titled “Clever code is probably the worst code you could write.” Early in a programming career, it’s easier to write complex and difficult-to-read code. The real challenge is progressing towards writing clearer and readable code. 
We also share several other articles and projects from the Python community, including a news roundup, what the __pycache__ folder is for in Python, what’s new in Django 5.1, a discussion about software engineering hiring and firing, a project for setting up repeated tasks, and a simple way to create reusable template components in Django.
This episode is sponsored by Sentry.

Course Spotlight: Efficient Iterations With Python Iterators and Iterables
In this video course, you’ll learn what iterators and iterables are in Python. You’ll learn how they differ and when to use them in your code. You’ll also learn how to create your own iterators and iterables to make data processing more efficient.

Topics:

00:00:00 – Introduction
00:02:18 – PEP 667: Consistent Views of Namespaces (Accepted)
00:03:08 – PEP 649 Re-targeted to 3.14
00:03:50 – Untold Stories From 6 Years Working on Python Packaging
00:04:38 – What Is the __pycache__ Folder in Python?
00:09:57 – Sponsor: Sentry
00:11:04 – What’s New in Django 5.1
00:17:48 – Page Dewarping
00:26:55 – Video Course Spotlight
00:28:26 – Clever Code Is Probably the Worst Code You Could Write
00:33:19 – Software Engineering Hiring and Firing
00:51:22 – Metronomes: An Easy Way to Set Up Regular Tasks
00:52:26 – django-web-components: Create reusable template components in Django
00:54:03 – Thanks and goodbye

News:

PEP 667: Consistent Views of Namespaces (Accepted)
PEP 649 Re-targeted to 3.14 – Python Enhancement Proposal 649: Deferred Evaluation Of Annotations Using Descriptors has been re-targeted to the Python 3.14 release
Untold Stories From 6 Years Working on Python Packaging – Sumana gave the closing keynote address at PyCon US this year and this posting shares all the links and references from the talk.

Show Links:

What Is the __pycache__ Folder in Python? – In this tutorial, you’ll explore Python’s __pycache__ folder. You’ll learn about when and why the interpreter creates these folders, and you’ll customize their default behavior. Finally, you’ll take a look under the hood of the cached .pyc files.
What’s New in Django 5.1 – Django 5.1 has gone alpha so the list of features targeting this release has more or less solidified. This article introduces you to what is coming in Django 5.1.
Page Dewarping – This article shows the techniques behind a page flattening algorithm. It starts with images of a book’s page which are curled from the spine of the book, and creates a resulting PDF that is a flat version.
Clever Code Is Probably the Worst Code You Could Write – When you come across a clever bit of code, it is hard not to admire it, but often times, clear, readable code is the hardest code to write.

Discussion:

Software Engineering Hiring and Firing – This article is a deep dive on the hiring and firing practices in the software field, and unlike most articles focuses on senior engineering roles. It isn’t a “first job” post, but a “how the decision process works” article.

Projects:

Metronomes: An Easy Way to Set Up Regular Tasks
django-web-components: A simple way to create reusable template components in Django

Level up your Python skills with our expert-led courses:

Get Started With Django: Build a Portfolio App
Efficient Iterations With Python Iterators and Iterables
For Loops in Python (Definite Iteration)
 Support the podcast & join our community of Pythonistas

Decomposing Software Problems & Avoiding the Trap of Clever Code

How do you start adding unit tests to your Python code? Can the built-in unittest framework cover most or all of your needs? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder’s Weekly articles and projects.
We dig into a recent tutorial by Leodanis Pozo Ramos about writing unit tests using Python’s unittest. The tutorial covers organizing your tests, exploring assert methods, creating test fixtures, and debugging failing tests.
We explore a collection of Python data visualizations and tutorials from the Python Graph Gallery. The website features hundreds of charts and graphs built using popular plotting libraries. Each chart type features a foundational tutorial that introduces the structure and application.
We also share several other articles and projects from the Python community, including a news roundup, the new REPL coming in Python 3.13, a pytest daemon to 10X test iteration speed, a discussion about software friction, a Raspberry Pi document scanner, and a project for controlling time per iteration loop.

Course Spotlight: Building a Python GUI Application With Tkinter
In this video course, you’ll learn the basics of GUI programming with Tkinter, the de facto Python GUI framework. Master GUI programming concepts such as widgets, geometry managers, and event handlers. Then, put it all together by building two applications: a temperature converter and a text editor.

Topics:

00:00:00 - Introduction
00:02:08 - Python Software Foundation Board Election Dates for 2024
00:02:35 - 2023 PSF Annual Impact Report
00:03:03 - Python’s unittest: Writing Unit Tests for Your Code
00:09:41 - What’s New in Python 3.13
00:10:38 - The New REPL in Python 3.13
00:13:39 - Best Python Chart Examples
00:15:27 - Animation with text that highlights important events
00:16:39 - Sankey Diagram with Python and Plotly
00:18:55 - Video Course Spotlight
00:20:25 - Pytest Daemon: 10X Local Test Iteration Speed
00:23:58 - Software Friction
00:35:41 - A Raspberry Pi Document Scanner
00:39:00 - pacemaker: For Controlling Time Per Iteration Loop in Python
00:41:55 - Thanks and goodbye

News:

Python Software Foundation Board Election Dates for 2024
2023 PSF Annual Impact Report

Show Links:

Python’s unittest: Writing Unit Tests for Your Code – In this tutorial, you’ll learn how to use the unittest framework to create unit tests for your Python code. Along the way, you’ll also learn how to create test cases, fixtures, test suites, and more.
What’s New in Python 3.13 – Python 3.13 has gone into beta, which means the feature freeze is now in place. This is the official listing of the new features in 3.13. This release includes changes to the REPL, new typing features, experimental support for disabling the GIL, dead battery removal, and more.
The New REPL in Python 3.13 – Python 3.13 just hit feature freeze with the first beta release, and it includes a host of improvements to the REPL. Automatic indenting, block-level editing, and more make the built-in REPL more powerful and easier to use.
Best Python Chart Examples
Animation with text that highlights important events - Python Graph Gallery
Sankey Diagram with Python and Plotly - Python Graph Gallery
Pytest Daemon: 10X Local Test Iteration Speed – Discord has a large Python monolith with lots of imports, which now takes 13 seconds to start up. On the server that’s not a problem but to run a test it is. Ruby’s solution is to have a daemon that hot loads a test on a process that already has the imports completed.

Discussion:

Software Friction – Friction is everywhere in software development. Two setbacks are more than twice as bad as one setback. This article discusses the sources of software friction and what you can do about it.

Projects:

A Raspberry Pi Document Scanner
pacemaker: For Controlling Time Per Iteration Loop in Python

Additional Links:

unittest — Unit testing framework — Python 3.12.3 documentation
Testing with Python (part 1): the basics - Bite code!
Sankey Diagrams – A Sankey diagram says more than 1000 pie charts
tidytuesday: Official repo for the #tidytuesday project
tidytuesday - dataset_announcements
Chaos Monkey
OpenCV: OpenCV modules
How to Train Your Robot

Level up your Python skills with our expert-led courses:

Building a Python GUI Application With Tkinter
Testing Your Code With pytest
Creating PyQt Layouts for GUI Applications
 Support the podcast & join our community of Pythonistas

Building Python Unit Tests & Exploring a Data Visualization Gallery

What’s it like to learn Python as a visually impaired or blind developer? How can you improve the accessibility of your Python web applications and learn current guidelines? This week on the show, Real Python community member Audrey van Breederode discusses her programming journey, web accessibility, and assistive technology.
Audrey shares her background as a system administrator and instructor. While she was learning Python, she discovered Real Python. Audrey provided some feedback about the built-in video player’s accessibility. Dan reached out and worked with Audrey on some website improvements for the visually impaired.
We discuss navigating modern websites and using assistive technology. Audrey also provides resources Python developers can use to improve the accessibility of their applications.

Course Spotlight: HTML and CSS Foundations for Python Developers
There’s no way around HTML and CSS when you want to build web apps. Even if you’re not aiming to become a web developer, knowing the basics of HTML and CSS will help you understand the Web better. In this video course, you’ll get an introduction to HTML and CSS for Python programmers.

Topics:

00:00:00 – Introduction
00:03:12 – Work background
00:08:30 – Language for assistive tools and programming
00:10:30 – What led you to learning Python?
00:13:48 – Screen readers, braille display, and white space
00:17:22 – Discovering Real Python
00:22:41 – Accessibility survey and navigating websites
00:30:04 – Digging deeper into learning Python
00:35:42 – Video Course Spotlight    
00:37:03 – Navigating changes in code
00:39:53 – Working with the terminal
00:42:14 – Accessibility of Python GUI libraries
00:44:22 – Django framework
00:47:11 – Screen readers and JAWS
00:53:19 – What are you excited about in the world of assistive technology?
00:57:03 – What are you excited about in the world of Python?
00:59:11 – What do you want to learn next?
01:00:09 – Thanks and goodbye

Show Links:

JAWS® – Freedom Scientific
What is JAWS? - YouTube
Retinal detachment - Wikipedia
Focus 80 Blue 5th Gen – Freedom Scientific
WebAIM: Screen Reader User Survey #10 Results
Web Content Accessibility Guidelines (WCAG) 2.2
SecureCRT - The rock-solid Telnet and SSH client for Windows, macOS, and Linux
P1 Monitor docker container for smart meters - Marcel Claassen
Python Basics: Introduction to Python (Learning Path) – Real Python
The web framework for perfectionists with deadlines - Django
Surf’s Up! Surfing the Internet with JAWS
NV Access - Download NVDA
Picture Smart Challenges – Freedom Scientific

Level up your Python skills with our expert-led courses:

Get Started With Django: Build a Portfolio App
Python Basics: Dictionaries
HTML and CSS Foundations for Python Developers
 Support the podcast & join our community of Pythonistas

Considering Accessibility & Assistive Tech as a Python Developer

Would you like to get more practice working with APIs in Python? How about exploring the globe using the data from OpenStreetMap? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder’s Weekly articles and projects.
We share an article from the Pybites blog about building queries using the Overpass API for OpenStreetMap. The post explores the data structures, tags, query formats, and how to use Overpass in Python.
Christopher discusses a Real Python article by recent guest Stephen Gruppetta about lazy evaluation in Python. The piece covers the advantages of generator expressions or functions and the potential disadvantages of using lazy versus eager evaluation methods.
We also share several other articles and projects from the Python community, including a news roundup, handling control-c in asyncio, preventing data leakage in pandas and scikit-learn, discussing the Django developer survey results, asking developers why they aren’t shipping faster, using UV to install into isolated environments, and a couple of tools for retrying in Python.
This episode is sponsored by Sentry.

Course Spotlight: How to Set Up a Django Project
In this course, you’ll learn the necessary steps that you’ll need to take to set up a new Django project. You’ll learn the basic setup for any new Django project, which needs to happen before programming the specific functionality of your project.

Topics:

00:00:00 – Introduction
00:02:33 – PyPy v7.3.16 Release
00:02:54 – PEP 745: Python 3.14 Release Schedule
00:03:31 – The BASIC programming language turns 60
00:05:24 – Asyncio Handle Control-C (SIGINT)
00:07:37 – OpenStreetMaps, Overpass API and Python
00:11:53 – What’s Lazy Evaluation in Python?
00:16:10 – Sponsor: Sentry
00:17:17 – How to Prevent Data Leakage in pandas & scikit-learn
00:24:09 – Django Developers Survey 2023 Results
00:33:07 – Video Course Spotlight
00:34:21 – I Asked 100 Devs Why They Aren’t Shipping Faster?
00:47:03 – pipxu: Install in Isolated Environments Using UV
00:49:05 – tenacity: Retrying Library for Python
00:50:04 – stamina: Production-Grade Retries for Python
00:52:00 – Thanks and goodbye

News:

PyPy v7.3.16 Release
PEP 745: Python 3.14 Release Schedule
The BASIC programming language turns 60 - Ars Technica

Show Links:

Asyncio Handle Control-C (SIGINT) – When the user presses CTRL-C on the keyboard, the OS raises an interrupt signal to your program. When writing concurrent code this can get complicated as the signal goes to the process. This article shows you how to handle capturing CTRL-C elegantly when using asyncio.
OpenStreetMaps, Overpass API and Python – OpenStreetMaps (OSM) is an open source mapping project that allows people to browse the world map and to plan routes. Not only does it have the expected web interface, but it also has an API known as Overpass. This article shows you two ways to use Python to query Overpass.
What’s Lazy Evaluation in Python? – This tutorial explores lazy evaluation in Python and looks at the advantages and disadvantages of using lazy and eager evaluation methods. By the end of this tutorial, you’ll clearly understand which approach is best for you, depending on your needs.
How to Prevent Data Leakage in pandas & scikit-learn – How you impute missing values in machine learning data sets can affect the quality of your training. This article teaches you what data leakage is and what steps you should take to avoid it.

Discussion

Django Developers Survey 2023 Results
I Asked 100 Devs Why They Aren’t Shipping Faster? – Daksh asked 100 developers why they aren’t shipping faster and this blog post shares what he learned. Problems include dependency bugs, overly complicated code bases, waiting on requirements, and more.

Projects:

pipxu: Install in Isolated Environments Using UV
tenacity: Retrying Library for Python
stamina: Production-Grade Retries for Python

Additional Links:

NASA’s Voyager 1 spacecraft finally phones home after 5 months of no contact - Space
Python & APIs: A Winning Combo for Reading Public Data – Real Python
Loading Data from OpenStreetMap with Python and the Overpass API - Nikolai Janakiev
Leakage (machine learning) - Wikipedia
Djangonaut Space - Where Contributors launch!
Debris That Fell Off a Boeing 767 Found Outside the House of Lawyer

Level up your Python skills with our expert-led courses:

Get Started With Django: Build a Portfolio App
Python Generators 101
How to Set Up a Django Project
 Support the podcast & join our community of Pythonistas

Querying OpenStreetMaps via API & Lazy Evaluation in Python

Do you get stressed while trying to learn Python? Do you prefer to build small programs or projects as you continue your coding journey? This week on the show, Real Python author Stephen Gruppetta is here to talk about his new book, “The Python Coding Book.”
Stephen has been teaching Python to adults and children for many years. With his new book, he wants to share the relaxed and friendly learning environment he’s developed. We discuss using analogies to explain programming concepts and constructing complete programs as chapter goals.  
Over the last year, Stephen focused on writing. He started his newsletter, The Python Coding Stack, wrote more tutorials for Real Python and authored the book. 
This episode is sponsored by Mailtrap.

Course Spotlight: Python Basics: Code Your First Python Program
In this video course, you’ll write your first Python program. Along the way, you’ll learn about errors, declare variables and inspect their values, and try your hand at writing comments.

Topics:

00:00:00 – Introduction
00:02:32 – Education and programming background
00:05:50 – Developing a teaching style
00:10:36 – A friendly and relaxed programming book
00:14:31 – Making mistakes
00:18:29 – Sponsor: Mailtrap
00:19:03 – What was your curation process like?
00:21:22 – First chapter building an actual program
00:25:08 – Glossary terms and exercises
00:27:48 – Why feature an IDE?
00:34:07 – Monty and the White Room analogy
00:37:46 – What, no turtle?
00:42:21 – Video Course Spotlight
00:44:00 – Shift toward teaching
00:46:50 – Teaching adults and children
00:51:23 – Python sequences tutorial
00:53:48 – Building community and social media
00:58:12 – What are you excited about in the world of Python?
01:00:49 – What do you want to learn next?
01:03:07 – Thanks and goodbye

Show Links:

Learn Python Coding - The Python Coding Book
Rambling Reflections - Twelve Months of The Python Coding Stack
On Writing: A Memoir of the Craft - Stephen King - Wikipedia
Episode #4: Learning Python Through Errors – The Real Python Podcast
Build a Python Turtle Game: Space Invaders Clone – Real Python
Python Sequences: A Comprehensive Guide – Real Python
Django in Action
The Python Coding Place – The Place to Learn Python
Stephen Gruppetta (@s_gruppetta_ct) / X

Level up your Python skills with our expert-led courses:

Python Basics: Setting Up Python
Python Turtle for Beginners
Python Basics: Code Your First Python Program
 Support the podcast & join our community of Pythonistas

Scaling Data Science and Machine Learning Infrastructure Like Netflix

Download our free app to listen on your phone