The Real Python Podcast

Using a Memory Profiler in Python & What It Can Teach You


Listen Later

Have you used a memory profiler to gauge the performance of your Python application? Maybe you’re using it to troubleshoot memory issues when loading a large data science project. What could running a profiler show you about a codebase you’re learning? This week on the show, Pablo Galindo Salgado returns to talk about Memray, a powerful tracing memory profiler.

Pablo developed Memray while working at Bloomberg to track memory allocations beyond Python code into native extensions and the interpreter itself. It’s a compelling tool that provides fine-grain reports to help you understand where memory is used.

Pablo shares the reporting that Memray provides, including live mode, flame graphs, and a pytest plug-in. We also discuss how a tracing memory profiler can help you understand a new codebase.

He walks through how he developed the first prototype internally and eventually moved the project into open source. This is the first part of my conversation with Pablo. In a couple of weeks, you’ll get the second part, where we talk about Python guilds inside large companies and his work as the release manager for Python 3.10 and 3.11.

Course Spotlight: SQLite and SQLAlchemy in Python: Moving Your Data Beyond Flat Files

In this video course, you’ll learn how to store and retrieve data using Python, SQLite, and SQLAlchemy as well as with flat files. Using SQLite with Python brings with it the additional benefit of accessing data with SQL. By adding SQLAlchemy, you can work with data in terms of objects and methods.

Topics:

  • 00:00:00 – Introduction
  • 00:02:48 – When should you use a memory profiler?
  • 00:05:13 – Fine-grain reporting
  • 00:13:17 – Sampling profiler vs tracing profiler
  • 00:19:46 – Sponsor: Deepgram
  • 00:20:31 – What is a flame graph?
  • 00:30:36 – Using Rich for terminal reporters
  • 00:40:08 – Currently only Linux and macOS
  • 00:41:13 – pytest plug-in
  • 00:42:03 – Showing native allocation details
  • 00:44:20 – Video Course Spotlight
  • 00:45:52 – Using a profiler to learn a codebase
  • 00:54:39 – Moving from internal project to open source
  • 01:02:17 – Thanks and goodbye
  • Show Links:

    • memray: Memray is a memory profiler for Python - GitHub bloomberg/memray
    • memray - PyPI
    • Bloomberg publishes Memray, a new open source memory profiler for Python code - Bloomberg LP
    • What is a Flame Graph? How it Works & Use Cases - Datadog
    • Gantt chart - Wikipedia
    • py-spy - PyPI
    • scalene - PyPI
    • Reduce your Python program’s memory usage with Fil
    • fil: A Python memory profiler for data processing and scientific computing applications - GitHub pythonspeed/filprofiler
    • Episode #24: Options for Packaging Your Python Application: Wheels, Docker, and More - The Real Python Podcast
    • rich - PyPI
    • Textualize
    • Brendan Gregg
    • Linux perf Examples
    • Deepgram - Speech-to-Text for Developers & Enterprise
    • Level up your Python skills with our expert-led courses:

      • Debugging in Python With pdb
      • SQLite and SQLAlchemy in Python: Moving Your Data Beyond Flat Files
      • Testing Your Code With pytest
      • Support the podcast & join our community of Pythonistas

        ...more
        View all episodesView all episodes
        Download on the App Store

        The Real Python PodcastBy Real Python

        • 4.7
        • 4.7
        • 4.7
        • 4.7
        • 4.7

        4.7

        136 ratings


        More shows like The Real Python Podcast

        View all
        Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

        Software Engineering Radio - the podcast for professional software developers

        272 Listeners

        The Changelog: Software Development, Open Source by Changelog Media

        The Changelog: Software Development, Open Source

        283 Listeners

        Thoughtworks Technology Podcast by Thoughtworks

        Thoughtworks Technology Podcast

        41 Listeners

        Talk Python To Me by Michael Kennedy

        Talk Python To Me

        592 Listeners

        Software Engineering Daily by Software Engineering Daily

        Software Engineering Daily

        625 Listeners

        Soft Skills Engineering by Jamison Dance and Dave Smith

        Soft Skills Engineering

        269 Listeners

        Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

        Super Data Science: ML & AI Podcast with Jon Krohn

        296 Listeners

        Python Bytes by Michael Kennedy and Brian Okken

        Python Bytes

        213 Listeners

        Data Engineering Podcast by Tobias Macey

        Data Engineering Podcast

        142 Listeners

        Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

        Syntax - Tasty Web Development Treats

        983 Listeners

        DataFramed by DataCamp

        DataFramed

        266 Listeners

        Kubernetes Podcast from Google by Abdel Sghiouar, Kaslin Fields

        Kubernetes Podcast from Google

        181 Listeners

        Practical AI by Practical AI LLC

        Practical AI

        189 Listeners

        The Stack Overflow Podcast by The Stack Overflow Podcast

        The Stack Overflow Podcast

        64 Listeners

        The Pragmatic Engineer by Gergely Orosz

        The Pragmatic Engineer

        52 Listeners