Metrics & Mayhem

The launch pre-mortem: you can't promise a quiet launch, but you can give your team the bad night in advance.

Tested, green dashboards, everyone is confident. It wobbled forty minutes in, and the team fell apart, not because the wobble was big, but because nobodyhad told them it was coming.

A dad tells his ten-year-old before a cup final: " You're going to have to suffer today. The boy plays through the hardmoment because he was warned. This episode is about doing that for your team.You can't promise a quiet launch; you can give them a bad night in advance.

That's a pre-mortem.

The habit: a ten-minute pre-mortem before your next change. It's 3am, this broke, tell me the story.

The book is out. Metrics & Mayhem: ACTO's Guide to Observability That Actually Works. Kindle is live now; paperback and hardback launched on 1 June.

Get your free chapter here:https://www.masteringobservability.com/metrics-and-mayhem/free-chapter

Newsletter:https://masteringobservability.com

LinkedIn:https://www.linkedin.com/in/allanmann1/

Sev1, and someone's typing in capitals. This should never have shipped. Who approved this? The channel tightens, the debugging slows, and now everyone's managing the angry person as well as the outage.

I heard someone this week say anger almost always hides fear. In therapy, they call it not hysterical but historical. This episode puts that in the war room: the engineer shouting at 2 am isn't angry at the deploy; he's afraid, and nobody gave him the words. It's also why blameless post-mortems actually work.

The habit: before you respond to the heat, ask one silent question: what is this person afraid of right now?

The book is out. Metrics & Mayhem: A CTO's Guide to Observability That Actually Works. Kindle is live now; paperback and hardback launched on 1 June.

Get your free chapter here:https://www.masteringobservability.com/metrics-and-mayhem/free-chapter

Newsletter:https://masteringobservability.com

LinkedIn:https://www.linkedin.com/in/allanmann1/

I avoided building one dashboard for about a year. The cost-per-request board for a service I already suspected was showing a bad number. Every deferral had a reason. Every reason was a way of not looking.

Joseph Campbell: The cave you fear to enter holds the treasure you seek. Every team has a cave, the part of the system nobody instruments because they're afraid of what it'll say. This episode is about treating that avoidance as a signal rather than a priority call.

The habit: name your cave, and put one crude panel on it this week.

The book is out. Metrics & Mayhem: A CTO's Guide to Observability That Actually Works. Kindle is live now; paperback and hardback launched on 1 June.

Get your free chapter here:https://www.masteringobservability.com/metrics-and-mayhem/free-chapter

Newsletter: https://masteringobservability.com

LinkedIn: https://www.linkedin.com/in/allanmann1/

A duty engineer showed me the page that woke her at 3 a.m. It said: Error rate elevated. Three words and a graph. ThenShe lay there guessing which service, how bad, the big one or the noisy one.

That gap is the problem. A context-free alert is an open loop, and open loops fill with fear. This episode is about treating an alert as what it actually is: a message to a tired human at theworst hour of their day. What broke, how bad, what to do

The book is out. Metrics & Mayhem: A CTO's Guide to Observability That Actually Works. Kindle is live now

Get your free chapter here:https://www.masteringobservability.com/metrics-and-mayhem/free-chapter

Newsletter:https://masteringobservability.com

LinkedIn:https://www.linkedin.com/in/allanmann1/

Most observability work is reactive, not preventive.

This Signal Drop is about the most expensive habit in IT operations: treating the response to the incident as thestrategy.

It covers why positioning beats reaction, what the unbuilt position actually costs you, and the merge-time habit that lets you stop paying the heroism tax. One idea, one habit, five minutes.

Your reliability metrics don't climb in a straight line, and most of that month-to-month movement is noise wearing the costume of signal.

This episode is about why one bad month isn't a failed strategy, and how chasing every wobble quietly wrecks the thing you're trying to fix.

The habit to take away: decide what counts as real before you see the number, not after.

Returning after two months of silence. An ended contract in Abu Dhabi, a war that began a week later, and the integrity decision behind the gap.

This Signal Drop is about the line you refuse to sell, the small increments of compromise that erode an IT Ops career, and how to name yours before someone else moves it for you.

Deep Dive (28mins): Auto-remediation is replacing the 3 AM scramble. Gartner says 60% of large enterprises will adopt self-healing infrastructure by 2026. PagerDuty'searly adopters are resolving incidents 50% faster. But the DORA 2024 report found AI tooling correlates with worse delivery performance, for the second year running, and the Catchpoint SRE Report showed toil rising, not falling.

What happens to on-call when the easy pages disappear, and the remaining 20% of incidents just get harder? Four concrete actions for the next sprint.Based on the blog post: https://masteringobservability.com/p/the-midnight-pager-is-dying-what-replaces-it-is-harder

Observability, SRE, AIOps, On-Call, Auto-Remediation, IT Operations, Leadership

Most leaders don't fail because they're bad at leading. They fail because they stay in one mode too long. This Signal Drop breaks incident leadership into four practical modes: Direct, Shield, Coach, Delegate. Pick the wrong one and the room fills with noise. Pick the right one, and people can think. Includes a ten-second habit you can use before your next call.

In IT Operations, accountability isn’t blame. It’s ownership. When nobody owns the outcome, decisions wobble, incidents drag, and teams default to theatre. This Signal Drop is about making ownership explicit under pressure: one owner, one outcome, one next check. Blameless doesn’t mean ownerless. Clear accountability creates calm, faster decisions, and better reliability.