Programming Throwdown

185: Workflow Orchestrators


Listen Later

Intro topic: Asymmetric Returns


News/Links:

  • NanoChat by Andrej Karpathy
    • https://github.com/karpathy/nanochat
  • Pydantic AI
    • https://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/
  • 1000th Starlink this year
    • https://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/
  • ChatGPT Apps SDK
    • https://openai.com/index/introducing-apps-in-chatgpt/

Book of the Show

  • Patrick
    • The Will of the Many by James Islington
      • https://amzn.to/43IfU8Q
  • Jason
    • Interview with DHH (Founder of Ruby on Rails)
      • https://www.youtube.com/watch?v=vagyIcmIGOQ


Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h


Tool of the Show

  • Patrick
    • Factorio
      • https://www.factorio.com/ 
  • Jason
    • nip.io 


Topic: Workflow Orchestrators

  • Why
    • Batch jobs (embarrassingly parallel)
    • Long-running tasks (e.g. transcoding video)
    • Checkpointing/resuming
  • How
    • Message Queues
    • Containerization
    • Worker Pools & Autoscaling
    • History & Backfill
  • Steps to run workflows:
    • Containerize the workflow definition and send to the cloud
    • Containerize all the individual tasks
    • Submit job(s)
  • Examples
    • Airflow
      • Legacy but dominant
    • Dagster
      • Great UX for python developers
    • Temporal: https://temporal.io/
      • The new hotness
    • Ray
      • Low-level but very powerful
    • Kubeflow
      • Designed for ML workflows, integrated dashboard

★ Support this podcast on Patreon ★
...more
View all episodesView all episodes
Download on the App Store

Programming ThrowdownBy Patrick Wheeler and Jason Gauci

  • 4.6
  • 4.6
  • 4.6
  • 4.6
  • 4.6

4.6

552 ratings


More shows like Programming Throwdown

View all
Software Engineering Radio - the podcast for professional software developers by team@se-radio.net (SE-Radio Team)

Software Engineering Radio - the podcast for professional software developers

274 Listeners

Hanselminutes with Scott Hanselman by Scott Hanselman

Hanselminutes with Scott Hanselman

383 Listeners

The Changelog: Software Development, Open Source by Changelog Media

The Changelog: Software Development, Open Source

287 Listeners

Software Engineering Daily by Software Engineering Daily

Software Engineering Daily

624 Listeners

LINUX Unplugged by Jupiter Broadcasting

LINUX Unplugged

270 Listeners

Talk Python To Me by Michael Kennedy

Talk Python To Me

582 Listeners

Soft Skills Engineering by Jamison Dance and Dave Smith

Soft Skills Engineering

288 Listeners

Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

Syntax - Tasty Web Development Treats

989 Listeners

The freeCodeCamp Podcast by freeCodeCamp.org

The freeCodeCamp Podcast

486 Listeners

CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer

CoRecursive: Coding Stories

189 Listeners

.NET Rocks! by Carl Franklin and Richard Campbell

.NET Rocks!

245 Listeners

The Stack Overflow Podcast by The Stack Overflow Podcast

The Stack Overflow Podcast

62 Listeners

The Real Python Podcast by Real Python

The Real Python Podcast

141 Listeners

Oxide and Friends by Oxide Computer Company

Oxide and Friends

66 Listeners

The Pragmatic Engineer by Gergely Orosz

The Pragmatic Engineer

71 Listeners