Programming Throwdown

185: Workflow Orchestrators


Listen Later

Intro topic: Asymmetric Returns


News/Links:

  • NanoChat by Andrej Karpathy
    • https://github.com/karpathy/nanochat
  • Pydantic AI
    • https://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/
  • 1000th Starlink this year
    • https://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/
  • ChatGPT Apps SDK
    • https://openai.com/index/introducing-apps-in-chatgpt/

Book of the Show

  • Patrick
    • The Will of the Many by James Islington
      • https://amzn.to/43IfU8Q
  • Jason
    • Interview with DHH (Founder of Ruby on Rails)
      • https://www.youtube.com/watch?v=vagyIcmIGOQ


Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h


Tool of the Show

  • Patrick
    • Factorio
      • https://www.factorio.com/ 
  • Jason
    • nip.io 


Topic: Workflow Orchestrators

  • Why
    • Batch jobs (embarrassingly parallel)
    • Long-running tasks (e.g. transcoding video)
    • Checkpointing/resuming
  • How
    • Message Queues
    • Containerization
    • Worker Pools & Autoscaling
    • History & Backfill
  • Steps to run workflows:
    • Containerize the workflow definition and send to the cloud
    • Containerize all the individual tasks
    • Submit job(s)
  • Examples
    • Airflow
      • Legacy but dominant
    • Dagster
      • Great UX for python developers
    • Temporal: https://temporal.io/
      • The new hotness
    • Ray
      • Low-level but very powerful
    • Kubeflow
      • Designed for ML workflows, integrated dashboard

★ Support this podcast on Patreon ★
...more
View all episodesView all episodes
Download on the App Store

Programming ThrowdownBy Patrick Wheeler and Jason Gauci

  • 4.6
  • 4.6
  • 4.6
  • 4.6
  • 4.6

4.6

552 ratings


More shows like Programming Throwdown

View all
Software Engineering Radio by se-radio@computer.org

Software Engineering Radio

273 Listeners

Hanselminutes with Scott Hanselman by Scott Hanselman

Hanselminutes with Scott Hanselman

381 Listeners

The Changelog: Software Development, Open Source by Changelog Media

The Changelog: Software Development, Open Source

290 Listeners

Software Engineering Daily by Software Engineering Daily

Software Engineering Daily

626 Listeners

LINUX Unplugged by Jupiter Broadcasting

LINUX Unplugged

266 Listeners

Talk Python To Me by Michael Kennedy

Talk Python To Me

583 Listeners

Soft Skills Engineering by Jamison Dance and Dave Smith

Soft Skills Engineering

287 Listeners

Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

Syntax - Tasty Web Development Treats

984 Listeners

The freeCodeCamp Podcast by freeCodeCamp.org

The freeCodeCamp Podcast

484 Listeners

CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer

CoRecursive: Coding Stories

190 Listeners

.NET Rocks! by Carl Franklin and Richard Campbell

.NET Rocks!

244 Listeners

The Stack Overflow Podcast by The Stack Overflow Podcast

The Stack Overflow Podcast

63 Listeners

The Real Python Podcast by Real Python

The Real Python Podcast

143 Listeners

Oxide and Friends by Oxide Computer Company

Oxide and Friends

60 Listeners

The Pragmatic Engineer by Gergely Orosz

The Pragmatic Engineer

64 Listeners