The Real Python Podcast

Large Language Models on the Edge of the Scaling Laws


Listen Later

What’s happening with the latest releases of large language models? Is the industry hitting the edge of the scaling laws, and do the current benchmarks provide reliable performance assessments? This week on the show, Jodie Burchell returns to discuss the current state of LLM releases.

The most recent release of GPT-5 has been a wake-up call for the LLM industry. We discuss how the current scaling of these systems is reaching a diminishing edge. Jodie also shares how many AI model assessments and benchmarks are flawed. We also take a sober look at the productivity gains from using these tools for software development within companies.

We discuss how newer developers should consider additional factors when looking at the current job market. Jodie digs into how economic changes and rising interest rates are influencing layoffs and hiring freezes. Then we share a wide collection of resources for you to continue exploring these topics.

This episode is sponsored by InfluxData.

Course Spotlight: Exploring Python Closures: Examples and Use Cases

Learn about Python closures: function-like objects with extended scope used for decorators, factories, and stateful functions.

Topics:

  • 00:00:00 – Introduction
  • 00:03:00 – Recent conferences and talks
  • 00:04:18 – What’s going on with LLMs?
  • 00:06:06 – What happened with the GPT-5 release?
  • 00:08:14 – Simon Willison - 2025 in LLMs so far
  • 00:09:00 – How did we get here?
  • 00:10:37 – OpenAI’s and scaling laws
  • 00:12:25 – Pivoting to post-training
  • 00:16:01 – Some history of AI eras
  • 00:17:54 – Issues with measuring performance and benchmarks
  • 00:22:19 – Chatbot Arena
  • 00:24:06 – Languages are finite
  • 00:26:22 – LLMs and the illusion of humanity
  • 00:30:41 – Sponsor: InfluxData
  • 00:31:34 – Types of solutions to move past these limits
  • 00:36:57 – Does AI actually boost developer productivity?
  • 00:44:19 – Agentic Al Programming with Python
  • 00:48:02 – Results of non-programmers vibe coding
  • 00:50:18 – Back to the concept of overfitting
  • 00:52:52 – The money involved in training
  • 00:56:50 – Video Course Spotlight
  • 00:58:21 – Deepseek and new methods of training
  • 01:01:02 – Quantizing and fitting on a local machine
  • 01:04:48 – The layoffs and the economic changes
  • 01:10:32 – AI implementation failures
  • 01:21:01 – Don’t doubt yourself as a developer
  • 01:24:06 – What are you excited about in the world of Python?
  • 01:25:39 – What do you want to learn next?
  • 01:26:42 – What’s the best way to follow your work online?
  • 01:27:04 – Thanks and goodbye
  • Survey:

    • Listener Survey - Help Shape the Future of the Real Python Podcast
    • Show Links:

      • EuroPython 2025 - July 14th-20th 2025 - Prague, Czech Republic & Remote
      • Episode #232: Exploring Modern Sentiment Analysis Approaches in Python
      • GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
      • GPT 5’s Rocky Launch Highlights AI Disillusionment - IEEE Spectrum
      • 2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison
      • Attention is All You Need - Google
      • Scaling laws for neural language models - OpenAI
      • What if AI Doesn’t Get Much Better Than This? - Cal Newport
      • Hiltzik: AI hype is fading fast - Los Angeles Times
      • Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford - YouTube
      • Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity - METR
      • Amazon Cloud Chief: Replacing Junior Staff With AI Is ‘Dumbest’ Idea - Business Insider
      • 20 LLM evaluation benchmarks and how they work
      • MMLU - Measuring Massive Multitask Language Understanding
      • HellaSwag: Can a Machine Really Finish Your Sentence?
      • Mechanical Turk - Wikipedia
      • Amazon Mechanical Turk
      • Chatbot Arena - LMArena
      • LLMs Can’t Reason - The Reversal Curse, The Alice In Wonderland Test, And The ARC - AGI Challenge - CustomGPT
      • Mirror, mirror: LLMs and the illusion of humanity - Jodie Burchell - NDC Oslo 2024 - YouTube
      • Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
      • Context Rot: How Increasing Input Tokens Impacts LLM Performance - YouTube
      • Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford - YouTube
      • AWS CEO says no more programmers in 2 years - Tech Industry - Blind
      • MIT report: 95% of generative AI pilots at companies are failing - Fortune
      • Agentic Al Programming with Python - Talk Python To Me Podcast
      • Vibe coding through the GPT-5 mess - The Verge
      • Overfitting - Wikipedia
      • Andrej Karpathy - Busy Person’s Intro to LLMs - YouTube
      • AI Isn’t Taking Your Job – The Economy Is - Andrew Stiefel
      • Commonwealth Bank backtracks on AI job cuts, apologizes for ‘error’ as call volumes rise - ABC News
      • Klarna CEO Reverses Course By Hiring More Humans, Not AI | Entrepreneur
      • Has Duolingo Lost Its Streak? - Matt Jones - Medium
      • McDonald’s removes AI drive-throughs after order errors
      • OpenAI Usage Plummets in the Summer, When Students Aren’t Cheating on Homework
      • What Happened When I Tried to Replace Myself with ChatGPT in My English Classroom - Literary Hub
      • Learning to code in the age of AI — Sheena O’Connell - YouTube
      • Jodie Burchell - The JetBrains Blog
      • Jodie Burchell’s Blog - Standard error
      • Jodie Burchell (@t-redactyl.bsky.social) — Bluesky
      • Jodie Burchell 🇦🇺🇩🇪 (@[email protected]) - Fosstodon
      • JetBrains: Essential tools for software developers and teams
      • Level up your Python skills with our expert-led courses:

        • Python Decorators 101
        • A History of Python Versions and Features
        • Exploring Python Closures: Examples and Use Cases
        • Support the podcast & join our community of Pythonistas

          ...more
          View all episodesView all episodes
          Download on the App Store

          The Real Python PodcastBy Real Python

          • 4.7
          • 4.7
          • 4.7
          • 4.7
          • 4.7

          4.7

          138 ratings


          More shows like The Real Python Podcast

          View all
          Software Engineering Radio by se-radio@computer.org

          Software Engineering Radio

          271 Listeners

          The Changelog: Software Development, Open Source by Changelog Media

          The Changelog: Software Development, Open Source

          291 Listeners

          Software Engineering Daily by Software Engineering Daily

          Software Engineering Daily

          624 Listeners

          Talk Python To Me by Michael Kennedy

          Talk Python To Me

          588 Listeners

          Thoughtworks Technology Podcast by Thoughtworks

          Thoughtworks Technology Podcast

          41 Listeners

          Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

          Super Data Science: ML & AI Podcast with Jon Krohn

          301 Listeners

          Python Bytes by Michael Kennedy and Brian Okken

          Python Bytes

          214 Listeners

          Data Engineering Podcast by Tobias Macey

          Data Engineering Podcast

          146 Listeners

          Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

          Syntax - Tasty Web Development Treats

          984 Listeners

          CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer

          CoRecursive: Coding Stories

          188 Listeners

          DataFramed by DataCamp

          DataFramed

          268 Listeners

          Kubernetes Podcast from Google by Abdel Sghiouar, Kaslin Fields

          Kubernetes Podcast from Google

          182 Listeners

          Practical AI by Practical AI LLC

          Practical AI

          211 Listeners

          AWS Podcast by Amazon Web Services

          AWS Podcast

          203 Listeners

          The Stack Overflow Podcast by The Stack Overflow Podcast

          The Stack Overflow Podcast

          62 Listeners