Raw Data with Rob Collie

The Apparent Meaninglessness of AI Benchmarks, plus How to Explain AI Opportunities to Others


Listen Later

Every week brings a new AI benchmark. Higher scores. Bigger claims. Louder voices insisting this changes everything. And yet, when you put AI in front of a real business problem, none of that noise seems to help. In this episode, Rob and Justin dig into why AI benchmarks often feel strangely meaningless in practice and why that disconnect is the point. Benchmarks aren't useless. They're just answering a different question than the one most businesses are asking.

This isn't just random conjecture either. Rob walks through what he's learned building actual AI workflows and why a twenty percent improvement on a leaderboard rarely translates into anything you can feel on the job. They talk about why model choice usually isn't the bottleneck, why swapping models should be easy if you've built things the right way, and why the most successful AI work rarely shows up as a flashy demo. Most of the value is happening quietly, off-screen, inside systems that look a lot more like normal software than artificial intelligence.

Rob and Justin also talk about why explaining AI is often harder than building it. The first demo people see tends to stick, even when it's the wrong one. Consumer AI feels magical. Business AI face plants unless it's built with intent, structure, and real context. This episode gives leaders better language for that gap, without hype or panic. If you're done chasing benchmarks and just want a way to think about AI that survives contact with reality, this episode's for you.

...more
View all episodesView all episodes
Download on the App Store

Raw Data with Rob CollieBy P3 Adaptive

  • 5
  • 5
  • 5
  • 5
  • 5

5

53 ratings


More shows like Raw Data with Rob Collie

View all
Freakonomics Radio by Freakonomics Radio + Stitcher

Freakonomics Radio

32,006 Listeners

Planet Money by NPR

Planet Money

30,695 Listeners

Odd Lots by Bloomberg

Odd Lots

1,940 Listeners

Talk Python To Me by Michael Kennedy

Talk Python To Me

585 Listeners

Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

Super Data Science: ML & AI Podcast with Jon Krohn

302 Listeners

Founders by David Senra

Founders

2,169 Listeners

The Indicator from Planet Money by NPR

The Indicator from Planet Money

9,529 Listeners

DataFramed by DataCamp

DataFramed

269 Listeners

The Journal. by The Wall Street Journal & Spotify Studios

The Journal.

6,093 Listeners

All-In with Chamath, Jason, Sacks & Friedberg by All-In Podcast, LLC

All-In with Chamath, Jason, Sacks & Friedberg

9,932 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

501 Listeners

Kasper On BI by Kasper de Jonge

Kasper On BI

7 Listeners

The Ezra Klein Show by New York Times Opinion

The Ezra Klein Show

15,938 Listeners

Explicit Measures Podcast by PowerBI.Tips

Explicit Measures Podcast

34 Listeners

The AI Daily Brief: Artificial Intelligence News and Analysis by Nathaniel Whittemore

The AI Daily Brief: Artificial Intelligence News and Analysis

610 Listeners