LessWrong (30+ Karma)

“On METR’s AI Coding RCT” by Zvi

Listen Later

METR ran a proper RCT experiment seeing how much access to Cursor (using Sonnet 3.7) would accelerate coders working on their own open source repos.

Everyone surveyed expected a substantial speedup. The developers thought they were being substantially sped up.

Instead, it turned out that using Cursor slowed them down.

That surprised everyone, raising the question of why.

Currently our best guess is this comes down to a combination of two factors:

  1. Deeply understood open source repos are close to a worst-case scenario for AI tools, because they require bespoke outputs in various ways and the coder has lots of detailed local knowledge of the codebase that the AI lacks.
  2. The coders in question mostly did not have experience with similar AI tools. The lack of a learning curve during the experiment challenges this, but the tools very clearly have a [...]

    3. ---

    Outline:

    (01:27) Epic Fail

    (02:42) The Core Result

    (07:10) Okay So That Happened

    (12:21) Beginner Mindset

    (19:43) Overall Takeaways

    ---

    First published:

    July 18th, 2025

    Source:

    https://www.lesswrong.com/posts/m2QeMwD7mGKH6vDe2/on-metr-s-ai-coding-rct

    ---

    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

    ...more
    View all episodesView all episodes
    Download on the App Store
    LessWrong (30+ Karma)By LessWrong

    More shows like LessWrong (30+ Karma)

    View all
    Making Sense with Sam Harris by Sam Harris

    Making Sense with Sam Harris

    26,311 Listeners

    Conversations with Tyler by Mercatus Center at George Mason University

    Conversations with Tyler

    2,461 Listeners

    The Peter Attia Drive by Peter Attia, MD

    The Peter Attia Drive

    8,597 Listeners

    Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas by Sean Carroll | Wondery

    Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas

    4,170 Listeners

    ManifoldOne by Steve Hsu

    ManifoldOne

    97 Listeners

    Your Undivided Attention by The Center for Humane Technology, Tristan Harris, Daniel Barcay and Aza Raskin

    Your Undivided Attention

    1,608 Listeners

    All-In with Chamath, Jason, Sacks & Friedberg by All-In Podcast, LLC

    All-In with Chamath, Jason, Sacks & Friedberg

    10,041 Listeners

    Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

    Machine Learning Street Talk (MLST)

    97 Listeners

    Dwarkesh Podcast by Dwarkesh Patel

    Dwarkesh Podcast

    528 Listeners

    Hard Fork by The New York Times

    Hard Fork

    5,529 Listeners

    The Ezra Klein Show by New York Times Opinion

    The Ezra Klein Show

    16,055 Listeners

    Moonshots with Peter Diamandis by PHD Ventures

    Moonshots with Peter Diamandis

    570 Listeners

    No Priors: Artificial Intelligence | Technology | Startups by Conviction

    No Priors: Artificial Intelligence | Technology | Startups

    138 Listeners

    Latent Space: The AI Engineer Podcast by swyx + Alessio

    Latent Space: The AI Engineer Podcast

    93 Listeners

    BG2Pod with Brad Gerstner and Bill Gurley by BG2Pod

    BG2Pod with Brad Gerstner and Bill Gurley

    473 Listeners