TL;DR

• On 18 real tasks from two large open-source repositories, early-2025 AI agents often implement functionally correct code that cannot be easily used as-is, because of issues with test coverage, formatting/linting, or general code quality.
• This suggests that automatic scoring used by many benchmarks may overestimate AI agent real-world performance.

---

First published:

Source:

Linkpost URL:
https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.