Meta Tech Podcast

84: Trust But Canary: Configuration Safety at Scale


Listen Later

Have you ever wondered how Meta makes config rollouts safe at scale? In this episode, Pascal sits down with Ishwari and Joe to discuss Meta's approach for propagating changes across services in seconds and discuss why speed increases the need for strong safeguards. Catch the episode to discover canarying and progressive rollouts, the health checks and monitoring signals used to catch regressions early, and how incident reviews focus on improving systems rather than blaming people. We also hear how data and early AI/ML are slashing alert noise and speeding up bisecting when something goes wrong.

Got feedback? Send it to us on Threads (https://threads.net/@metatechpod), Instagram (https://instagram.com/metatechpod) and don't forget to follow our host Pascal (https://mastodon.social/@passy, https://threads.net/@passy_). Fancy working with us? Check out https://www.metacareers.com/.

Links

  • FFmpeg at Meta: Media Processing at Scale - https://engineering.fb.com/2026/03/02/video-engineering/ffmpeg-at-meta-media-processing-at-scale/

  • Reliably Changing Configuration @ Scale - https://atscaleconference.com/reliably-changing-configuration-scale/

Timestamps

  • Intro 0:06

  • Introduction and Overview of Configuration Changes 2:31

  • Understanding Configurations in Distributed Systems 4:02

  • Meta's Configuration Management Systems 6:43

  • Safeguards and Incident Prevention 9:22

  • Deployment Mechanisms: Canary and Progressive Rollouts 12:06

  • Challenges in Configuration Consumption 14:39

  • Health Checks and Incident Response 17:13

  • Mitigation Strategies for Configuration Issues 19:18

  • Balancing Developer Velocity and Configuration Safety 21:09

  • Data-Driven Improvements in Incident Management 22:12

  • Leveraging AI for Change Detection 26:05

  • Challenges in Deployment and Testing 28:21

  • Reinventing Change Safety Strategies 30:24

  • War Stories: Learning from Past Incidents 32:59

  • Outro 36:10

...more
View all episodesView all episodes
Download on the App Store

Meta Tech PodcastBy Meta

  • 4.5
  • 4.5
  • 4.5
  • 4.5
  • 4.5

4.5

43 ratings


More shows like Meta Tech Podcast

View all
Freakonomics Radio by Freakonomics Radio + Stitcher

Freakonomics Radio

32,246 Listeners

Odd Lots by Bloomberg

Odd Lots

1,993 Listeners

The Changelog: Software Development, Open Source by Changelog Media

The Changelog: Software Development, Open Source

288 Listeners

Decoder with Nilay Patel by The Verge

Decoder with Nilay Patel

3,141 Listeners

Software Engineering Daily by Software Engineering Daily

Software Engineering Daily

626 Listeners

HBR IdeaCast by Harvard Business Review

HBR IdeaCast

154 Listeners

NVIDIA AI Podcast by NVIDIA

NVIDIA AI Podcast

343 Listeners

Practical AI by Practical AI LLC

Practical AI

212 Listeners

AWS Podcast by Amazon Web Services

AWS Podcast

204 Listeners

My First Million by Hubspot Media

My First Million

2,660 Listeners

The Stack Overflow Podcast by The Stack Overflow Podcast

The Stack Overflow Podcast

63 Listeners

矽谷輕鬆談 Just Kidding Tech by 柯柯與肯吉在矽谷

矽谷輕鬆談 Just Kidding Tech

161 Listeners

All-In with Chamath, Jason, Sacks & Friedberg by All-In Podcast, LLC

All-In with Chamath, Jason, Sacks & Friedberg

10,254 Listeners

Hard Fork by The New York Times

Hard Fork

5,576 Listeners

Piers Morgan Uncensored by Piers Morgan Uncensored

Piers Morgan Uncensored

512 Listeners