The Joe Reis Show

AI Agents Can't Fix Data - Josh Wills on Where AI Breaks in Data Engineering


Listen Later

Josh Wills has spent 25 years writing data pipelines, with a career spanning Cloudera, as Director of Data Engineering at Slack, on the dbt DuckDB adapter, and now training foundation models at Datology AI. He uses coding agents every day. And he keeps running into the same wall: the agents jump to conclusions, fix the wrong thing, and ship pipelines no one understands.In this conversation, we unpack why AI agents struggle with the messiest, highest-stakes parts of data work, and what it means for the engineers managing them.We get into:- Big Data is back- Why AI agents jump to conclusions on benchmarks and complex bottlenecks- The $200K vibe-coded pipeline problem nobody wants to talk about- Why there's no training data for the gnarly enterprise pipelines that actually power businesses- "We're all managers now" - managing unreliable agents like managing unreliable people- Wicked problems and the limits of intelligence- Why politics is the last human endeavor to fall to LLMs (the data is never written down)- Whether classical ML still has a place (yes)- What Josh would tell a new grad starting in data today

...more
View all episodesView all episodes
Download on the App Store

The Joe Reis ShowBy Joe Reis

  • 5
  • 5
  • 5
  • 5
  • 5

5

17 ratings


More shows like The Joe Reis Show

View all
The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch by Harry Stebbings

The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch

536 Listeners

The Knowledge Project by Shane Parrish

The Knowledge Project

2,672 Listeners

The a16z Show by Andreessen Horowitz

The a16z Show

1,105 Listeners

The Enterprise AI Show by Massive Studios

The Enterprise AI Show

154 Listeners

Talk Python To Me by Michael Kennedy

Talk Python To Me

583 Listeners

Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

Super Data Science: ML & AI Podcast with Jon Krohn

306 Listeners

Equity by TechCrunch, Rebecca Bellan, Kirsten Korosec, Anthony Ha, Sean O'Kane, Theresa Loconsolo

Equity

341 Listeners

DataFramed by DataCamp

DataFramed

266 Listeners

Practical AI by Practical AI LLC

Practical AI

212 Listeners

The Daily Stoic by Daily Stoic | Backyard Ventures

The Daily Stoic

4,942 Listeners

The Real Python Podcast by Real Python

The Real Python Podcast

140 Listeners

Deep Questions with Cal Newport by Cal Newport

Deep Questions with Cal Newport

1,348 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

551 Listeners

The MAD Podcast with Matt Turck by Matt Turck

The MAD Podcast with Matt Turck

27 Listeners

AI + a16z by a16z

AI + a16z

34 Listeners