Awesome Agents Podcast

METR: Half of SWE-Bench Passes Fail Real Code Review


Listen Later

METR found maintainers would reject roughly half of AI PRs that pass SWE-bench automated grading, with a 24-point gap that suggests benchmark scores substantially overstate production readiness.
...more
View all episodesView all episodes
Download on the App Store

Awesome Agents PodcastBy Awesome Agents