The Real Python Podcast

Creating a Python Code Completer & More Abstract Syntax Tree Projects


Listen Later

How does a code completion tool work? What is an Abstract Syntax Tree, and how is it created in Python? How does an AST help you write programs and projects that inspect and modify your Python code? This week on the show, Meredydd Luff, co-founder of Anvil, shares his PyCon talk, “Building a Python Code Completer.”

Meredydd talks about his experience building a code completion engine for the Anvil platform. The conversation leads us to discuss how Python parses the code that you type. We examine tokenization, abstract syntax trees, and how parsing has changed in Python.

We cover additional projects that you can explore once you have a tool that inspects the Python code you’re writing. Join us as we dive into multiple rabbit holes of research and exploration.

Course Spotlight: Python Assignment Expressions and Using the Walrus Operator

In this course, you’ll learn about assignment expressions and the walrus operator. The biggest change in Python 3.8 was the inclusion of the := operator, which you can use to assign variables in the middle of expressions. You’ll see several examples of how to take advantage of this feature.

Topics:

  • 00:00:00 – Introduction
  • 00:02:11 – Building a Python Code Completer - Talk
  • 00:08:48 – How is recursion being used?
  • 00:10:28 – Abstract Syntax Trees, Parsers, and Tokens
  • 00:14:58 – Sponsor: CData Software
  • 00:15:39 – Is parsing into tokens a separate step?
  • 00:21:45 – What makes the new PEG parser unique?
  • 00:25:32 – Using the AST doesn’t have to be scary.
  • 00:27:33 – More on the PEG parser
  • 00:29:54 – Keeping scope and holding completions
  • 00:39:43 – Video Course Spotlight
  • 00:41:15 – Was this your first expirement in creating a code completer?
  • 00:46:46 – How can adding type hints assist a completer?
  • 00:50:53 – Projects you can build using AST
  • 00:53:02 – Linter project
  • 00:54:15 – Automatic code formatter
  • 00:54:59 – Code testing and the pytest project
  • 00:56:22 – Security tools
  • 00:57:11 – Friendlier tracebacks
  • 00:59:57 – Code completion vs GitHub Copilot
  • 01:04:14 – Code completion in a text editor vs IDE
  • 01:08:53 – What are you excited about in the world of Python?
  • 01:09:54 – What do you want to learn next?
  • 01:11:20 – How can people follow your work?
  • 01:12:31 – Thanks and goodbye
  • Show Links:

    • Talk - Meredydd Luff: Building a Python Code Completer - YouTube
    • Anvil | Build Web Apps with Nothing but Python
    • Episode #63: Create Web Applications Using Only Python With Anvil – The Real Python Podcast
    • “Writing a PEG parser for fun and profit” - Guido van Rossum (North Bay Python 2019) - YouTube
    • PEG Parsing Series Overview. My series of blog posts about PEG | by Guido van Rossum
    • Abstract syntax tree - Wikipedia
    • Parse tree - Wikipedia
    • ast — Abstract Syntax Trees — Python 3.10.6 documentation
    • Full Grammar specification — Python 3.10.6 documentation
    • Python AST Explorer
    • Examples of working with ASTs — Green Tree Snakes 1.0 documentation
    • bellybutton: Custom Python linting through AST expressions
    • black · PyPI
    • pytest: helps you write better programs — pytest documentation
    • friendly-traceback: Friendlier Python tracebacks.
    • Raspberry Pi Pico – Raspberry Pi
    • Raspberry Pi Pico W : ID 5526 : $6.00 : Adafruit Industries, Unique & fun DIY electronics and kits
    • State management - Wikipedia
    • Recoil
    • Meredydd Luff (@meredydd) / Twitter
    • Level up your Python skills with our expert-led courses:

      • Python Assignment Expressions and Using the Walrus Operator
      • Testing Your Code With pytest
      • Writing Beautiful Pythonic Code With PEP 8
      • Support the podcast & join our community of Pythonistas

        ...more
        View all episodesView all episodes
        Download on the App Store

        The Real Python PodcastBy Real Python

        • 4.7
        • 4.7
        • 4.7
        • 4.7
        • 4.7

        4.7

        139 ratings


        More shows like The Real Python Podcast

        View all
        The Changelog: Software Development, Open Source by Changelog Media

        The Changelog: Software Development, Open Source

        288 Listeners

        Software Engineering Daily by Software Engineering Daily

        Software Engineering Daily

        625 Listeners

        Talk Python To Me by Michael Kennedy

        Talk Python To Me

        579 Listeners

        Soft Skills Engineering by Jamison Dance and Dave Smith

        Soft Skills Engineering

        289 Listeners

        Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

        Super Data Science: ML & AI Podcast with Jon Krohn

        302 Listeners

        Python Bytes by Michael Kennedy and Brian Okken

        Python Bytes

        213 Listeners

        Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

        Syntax - Tasty Web Development Treats

        988 Listeners

        Darknet Diaries by Jack Rhysider

        Darknet Diaries

        8,088 Listeners

        Tech Brew Ride Home by Morning Brew

        Tech Brew Ride Home

        969 Listeners

        Practical AI by Practical AI LLC

        Practical AI

        200 Listeners

        AWS Podcast by Amazon Web Services

        AWS Podcast

        207 Listeners

        Django Chat by William Vincent and Carlton Gibson

        Django Chat

        75 Listeners

        Last Week in AI by Skynet Today

        Last Week in AI

        310 Listeners

        Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

        Machine Learning Street Talk (MLST)

        100 Listeners

        The Pragmatic Engineer by Gergely Orosz

        The Pragmatic Engineer

        70 Listeners