DevOps Paradox

DOP 333: The Hidden Problems Behind Every Data Pipeline


Listen Later

#333: Pete Hunt, CEO of Dagster and early React team member, explores the evolution from Facebook's early React development through trust and safety infrastructure at Twitter, to building modern data orchestration tools. The conversation reveals how similar infrastructure problems plague every industry - whether you're launching rockets or managing porta-potties, the core challenges remain consistent: late data, quality issues, and mysterious errors that require both automated solutions and human oversight.

The discussion dives into the technical realities of scaling systems, from the microservices complexity trap to the current AI adoption wave. Hunt shares candid insights about leadership challenges, including how well-intentioned technology recommendations can backfire, and why most data projects fail despite sophisticated multi-agent orchestration. The conversation touches on career advancement pressures that drive unnecessary complexity and the importance of focusing on actual user adoption rather than technical sophistication.

This episode features Pete Hunt in conversation with hosts Darin and Viktor, covering everything from regular expression nightmares to the future of data infrastructure and the lessons learned from building products that people actually use.

Pete's contact information:

X: https://x.com/floydophone

LinkedIn: https://www.linkedin.com/in/pwhunt/

YouTube channel:

https://youtube.com/devopsparadox

Review the podcast on Apple Podcasts:

https://www.devopsparadox.com/review-podcast/

Slack:

https://www.devopsparadox.com/slack/

Connect with us at:

https://www.devopsparadox.com/contact/

...more
View all episodesView all episodes
Download on the App Store

DevOps ParadoxBy Darin Pope & Viktor Farcic

  • 5
  • 5
  • 5
  • 5
  • 5

5

25 ratings


More shows like DevOps Paradox

View all
Software Engineering Radio - the podcast for professional software developers by team@se-radio.net (SE-Radio Team)

Software Engineering Radio - the podcast for professional software developers

273 Listeners

The Changelog: Software Development, Open Source by Changelog Media

The Changelog: Software Development, Open Source

289 Listeners

Software Engineering Daily by Software Engineering Daily

Software Engineering Daily

623 Listeners

LINUX Unplugged by Jupiter Broadcasting

LINUX Unplugged

272 Listeners

The Cloudcast by Massive Studios

The Cloudcast

151 Listeners

Thoughtworks Technology Podcast by Thoughtworks

Thoughtworks Technology Podcast

44 Listeners

Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

Syntax - Tasty Web Development Treats

988 Listeners

REWORK by 37signals

REWORK

214 Listeners

CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer

CoRecursive: Coding Stories

188 Listeners

DataFramed by DataCamp

DataFramed

270 Listeners

Kubernetes Podcast from Google by Abdel Sghiouar, Kaslin Fields

Kubernetes Podcast from Google

180 Listeners

AWS Podcast by Amazon Web Services

AWS Podcast

206 Listeners

The Stack Overflow Podcast by The Stack Overflow Podcast

The Stack Overflow Podcast

64 Listeners

Latent Space: The AI Engineer Podcast by swyx + Alessio

Latent Space: The AI Engineer Podcast

98 Listeners

The Pragmatic Engineer by Gergely Orosz

The Pragmatic Engineer

72 Listeners