The Real Python Podcast

Measuring Multiple Facets of Python Performance With Scalene


Listen Later

When choosing a tool for profiling Python code performance, should it focus on the CPU, GPU, memory, or individual lines of code? What if it looked at all those factors and didn’t alter code performance while measuring it? This week on the show, we talk about Scalene with Emery Berger, Professor of Computer Science at the University of Massachusetts Amherst.

Emery talks about his background in memory management and his collaboration on Hoard, a scalable memory manager system used in Mac OS X. We discuss the need for improving code performance on modern computer architecture. He highlights this idea by contrasting the familiar limitations of Moore’s law with the lesser-known rule of Dennard scaling.

Working with his students in the university lab, they developed Scalene. Scalene is a high-performance CPU, GPU, and memory profiler. It can look at code from the individual function or line-by-line level and compare time spent in Python vs C code. Emery talks about the recent Scalene feature of AI-powered optimization proposals and covers a couple of examples. He also shares a collection of additional Python code-assistant tools from their lab.

Course Spotlight: What Does if name == “main” Mean in Python?

In this video course, you’ll learn all about Python’s name-main idiom. You’ll learn what it does in Python, how it works, when to use it, when to avoid it, and how to refer to it.

Topics:

  • 00:00:00 – Introduction
  • 00:02:13 – College of Information and Computer Sciences
  • 00:03:25 – Memory management systems background
  • 00:05:15 – Dennard Scaling vs Moore’s Law
  • 00:10:12 – Starting work on Python profiling
  • 00:15:00 – Deciding on a statistical profiler
  • 00:17:05 – Wanting to trace memory
  • 00:21:21 – Finding memory issues
  • 00:23:59 – Line-by-line analysis
  • 00:25:56 – Video Course Spotlight
  • 00:27:14 – Measuring profiler performance
  • 00:30:30 – Memory leak detection
  • 00:34:31 – When should you run a profiler?
  • 00:37:27 – Considerations for measuring cloud performance
  • 00:39:12 – Working with Jupyter and Conda
  • 00:42:18 – Common issues and AI solutions
  • 00:45:50 – Using a profiler to learn a codebase
  • 00:50:48 – Examples of AI-powered optimizations
  • 00:55:50 – What are you excited about in the world of Python?
  • 00:58:30 – What do you want to learn next?
  • 01:01:48 – How can people follow your work online?
  • 01:02:56 – Thanks and goodbye
  • Show Links:

    • Emery Berger - Professor of Computer Science, UMass Amherst
    • Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals
    • Hoard
    • Moore’s law - Wikipedia
    • Dennard scaling - Wikipedia
    • Scalene: A high-performance, high-precision CPU+GPU+memory profiler for Python - PyCon 2021 - YouTube
    • Python Performance Matters - Strange Loop 2022 - YouTube
    • Triangulating Python Performance Issues with Scalene
    • ChatDBG: Puts root causes analysis into your debugger, and suggests fixes
    • Commentator: Automatically writes comments and type annotations for your code
    • Pythoness: Automatically generates Python code from natural language description
    • Slipcover: Near Zero-Overhead Python Code Coverage
    • emeryberger - GitHub
    • Level up your Python skills with our expert-led courses:

      • Debugging in Python With pdb
      • Testing Your Code With pytest
      • What Does if name == "main" Mean in Python?
      • Support the podcast & join our community of Pythonistas

        ...more
        View all episodesView all episodes
        Download on the App Store

        The Real Python PodcastBy Real Python

        • 4.7
        • 4.7
        • 4.7
        • 4.7
        • 4.7

        4.7

        134 ratings


        More shows like The Real Python Podcast

        View all
        Hanselminutes with Scott Hanselman by Scott Hanselman

        Hanselminutes with Scott Hanselman

        377 Listeners

        Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

        Software Engineering Radio - the podcast for professional software developers

        265 Listeners

        The Changelog: Software Development, Open Source by Changelog Media

        The Changelog: Software Development, Open Source

        287 Listeners

        LINUX Unplugged by Jupiter Broadcasting

        LINUX Unplugged

        262 Listeners

        Thoughtworks Technology Podcast by Thoughtworks

        Thoughtworks Technology Podcast

        41 Listeners

        Talk Python To Me by Michael Kennedy

        Talk Python To Me

        584 Listeners

        Software Engineering Daily by Software Engineering Daily

        Software Engineering Daily

        628 Listeners

        Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

        Super Data Science: ML & AI Podcast with Jon Krohn

        294 Listeners

        Python Bytes by Michael Kennedy and Brian Okken

        Python Bytes

        213 Listeners

        Data Engineering Podcast by Tobias Macey

        Data Engineering Podcast

        140 Listeners

        Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

        Syntax - Tasty Web Development Treats

        988 Listeners

        CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer

        CoRecursive: Coding Stories

        186 Listeners

        DataFramed by DataCamp

        DataFramed

        269 Listeners

        Practical AI by Practical AI LLC

        Practical AI

        190 Listeners

        The Stack Overflow Podcast by The Stack Overflow Podcast

        The Stack Overflow Podcast

        63 Listeners