Engineering Enablement by DX

Measuring AI code assistants and agents with the AI Measurement Framework


Listen Later

In this episode of Engineering Enablement, DX CTO Laura Tacho and CEO Abi Noda break down how to measure developer productivity in the age of AI using DX’s AI Measurement Framework. Drawing on research with industry leaders, vendors, and hundreds of organizations, they explain how to move beyond vendor hype and headlines to make data-driven decisions about AI adoption.


They cover why some fundamentals of productivity measurement remain constant, the pitfalls of over-relying on flawed metrics like acceptance rate, and how to track AI’s real impact across utilization, quality, and cost. The conversation also explores measuring agentic workflows, expanding the definition of “developer” to include new AI-enabled contributors, and avoiding second-order effects like technical debt and slowed PR throughput.

Whether you’re rolling out AI coding tools, experimenting with autonomous agents, or just trying to separate signal from noise, this episode offers a practical roadmap for understanding AI’s role in your organization—and ensuring it delivers sustainable, long-term gains.


Where to find Laura Tacho:

• X: https://x.com/rhein_wein

• LinkedIn: https://www.linkedin.com/in/lauratacho/

• Website: https://lauratacho.com/

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda 

• Substack: ​​https://substack.com/@abinoda 


In this episode, we cover:

(00:00) Intro

(01:26) The challenge of measuring developer productivity in the AI age

(04:17) Measuring productivity in the AI era — what stays the same and what changes

(07:25) How to use DX’s AI Measurement Framework 

(13:10) Measuring AI’s true impact from adoption rates to long-term quality and maintainability

(16:31) Why acceptance rate is flawed — and DX’s approach to tracking AI-authored code

(18:25) Three ways to gather measurement data

(21:55) How Google measures time savings and why self-reported data is misleading

(24:25) How to measure agentic workflows and a case for expanding the definition of developer

(28:50) A case for not overemphasizing AI’s role

(30:31) Measuring second-order effects 

(32:26) Audience Q&A: applying metrics in practice

(36:45) Wrap up: best practices for rollout and communication 


Referenced:

  • DX Core 4 Productivity Framework
  • Measuring AI code assistants and agents
  • AI is making Google engineers 10% more productive, says Sundar Pichai - Business Insider
...more
View all episodesView all episodes
Download on the App Store

Engineering Enablement by DXBy DX

  • 5
  • 5
  • 5
  • 5
  • 5

5

38 ratings


More shows like Engineering Enablement by DX

View all
Software Engineering Radio by se-radio@computer.org

Software Engineering Radio

273 Listeners

The Changelog: Software Development, Open Source by Changelog Media

The Changelog: Software Development, Open Source

290 Listeners

a16z Podcast by Andreessen Horowitz

a16z Podcast

1,087 Listeners

Software Engineering Daily by Software Engineering Daily

Software Engineering Daily

625 Listeners

The Cloudcast by Massive Studios

The Cloudcast

154 Listeners

Soft Skills Engineering by Jamison Dance and Dave Smith

Soft Skills Engineering

283 Listeners

Thoughtworks Technology Podcast by Thoughtworks

Thoughtworks Technology Podcast

42 Listeners

Data Engineering Podcast by Tobias Macey

Data Engineering Podcast

144 Listeners

Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

Syntax - Tasty Web Development Treats

986 Listeners

REWORK by 37signals

REWORK

210 Listeners

Practical AI by Practical AI LLC

Practical AI

211 Listeners

The Stack Overflow Podcast by The Stack Overflow Podcast

The Stack Overflow Podcast

62 Listeners

No Priors: Artificial Intelligence | Technology | Startups by Conviction

No Priors: Artificial Intelligence | Technology | Startups

129 Listeners

Latent Space: The AI Engineer Podcast by swyx + Alessio

Latent Space: The AI Engineer Podcast

94 Listeners

The Pragmatic Engineer by Gergely Orosz

The Pragmatic Engineer

64 Listeners