The Real Python Podcast

Constructing and Judging Modern Agentic Workflows

Listen Later

How can you improve your LLM agent systems through specification enrichment? What are the advantages of having an LLM act as a judge within an agent system? This week on the show, Senior IEEE Member and Quality Engineer Suneet Malhotra joins us to discuss building and evaluating agentic architecture.

Suneet Malhotra is an independent practitioner-researcher with 18 years of experience in Quality Engineering (QE) and test automation for consumer-scale platforms. He discusses building specification-enrichment loops, monitoring performance, and using Cohen’s kappa to measure agreement between LLM judgments.

Suneet is currently publishing multiple papers that are under peer review on these topics. He also provides links to his work and GitHub projects if you want to experiment with these concepts and methods yourself.

Quick Survey: Get more out of the podcast show notes

Video Course Spotlight: Testing MCP Servers With a Python MCP Client

Learn how to build a Python MCP client that tests MCP servers from your terminal. List their tools, prompts, and resources, then call each one.

Topics:

  • 00:00:00 – Introduction
  • 00:00:56 – Survey: RP Podcast show notes
  • 00:02:11 – How did you get into testing?
  • 00:05:04 – Has working for large public-facing corporations changed how you approach testing?
  • 00:07:06 – Writing a paper on LLM-as-Judge
  • 00:09:22 – Looking across the Software Development Lifecycle
  • 00:14:46 – Agentic AI: theater vs methodology
  • 00:17:23 – Specification enrichment
  • 00:27:52 – Video Course Spotlight
  • 00:29:18 – Saving the specifications
  • 00:31:27 – Using the LLM as a judge & Cohen’s kappa
  • 00:39:31 – How can people try out the project?
  • 00:43:35 – What are some of the failure modes you’ve seen?
  • 00:50:26 – What’s an inexpensive way to try these ideas out?
  • 00:54:04 – What are you excited about in the world of Python?
  • 00:56:14 – What do you want to learn next?
  • 00:57:14 – How can people follow your work online?
  • 00:57:32 – Thanks and goodbye

    • Show Links:

    • Suneet Malhotra - AI-Driven Quality Engineering Leader
    • SuneetMalhotra - GitHub
    • Suneet Malhotra - ORCID
    • Cross-Layer Observability for LLM-Assisted Test Automation — Reference Implementation and Evaluation Data - Zenodo
    • Specification Enrichment v28 (EISEJ submission) — Reference implementation and empirical evaluation - Zenodo
    • Visual Oracle Bench — Two-Judge Synthetic-HTML Pilot for LLM-as-Judge Visual Regression Detection with Specificity Reporting - Zenodo
    • Cohen’s kappa - Wikipedia
    • Suneet Malhotra - LinkedIn
    • Survey: Get more out of every Real Python Podcast episode

      • Level up your Python skills with our expert-led courses:

      • Improving Your Tests With the Python Mock Object Library
      • Building Type-Safe LLM Agents With Pydantic AI
      • Testing MCP Servers With a Python MCP Client

        • Support the podcast & join our community of Pythonistas

        ...more
        View all episodesView all episodes
        Download on the App Store
        The Real Python PodcastBy Real Python
        • 4.7
        • 4.7
        • 4.7
        • 4.7
        • 4.7

        4.7

        139 ratings

        More shows like The Real Python Podcast

        View all
        The Changelog: Software Development, Open Source by Changelog Media

        The Changelog: Software Development, Open Source

        289 Listeners

        Software Engineering Daily by Software Engineering Daily

        Software Engineering Daily

        626 Listeners

        Talk Python To Me by Michael Kennedy

        Talk Python To Me

        583 Listeners

        Soft Skills Engineering by Jamison Dance and Dave Smith

        Soft Skills Engineering

        288 Listeners

        Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

        Super Data Science: ML & AI Podcast with Jon Krohn

        301 Listeners

        Python Bytes by Michael Kennedy and Calvin Hendryx-Parker

        Python Bytes

        213 Listeners

        Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

        Syntax - Tasty Web Development Treats

        982 Listeners

        Darknet Diaries by Jack Rhysider

        Darknet Diaries

        8,051 Listeners

        Tech Brew Ride Home by Morning Brew

        Tech Brew Ride Home

        963 Listeners

        Practical AI by Daniel Whitenack and Chris Benson

        Practical AI

        208 Listeners

        AWS Podcast by Amazon Web Services

        AWS Podcast

        203 Listeners

        Django Chat by William Vincent and Carlton Gibson

        Django Chat

        75 Listeners

        Last Week in AI by Skynet Today

        Last Week in AI

        314 Listeners

        Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

        Machine Learning Street Talk (MLST)

        99 Listeners

        The Pragmatic Engineer by Gergely Orosz

        The Pragmatic Engineer

        74 Listeners