Metrics & Mayhem

It is three in the morning, and your on-call engineer is doing the one job a machine should be faster at: reading the logs, checking what changed, forming a theory. This year somebody will offer to take that off you and call it an AI-SRE. Tech Tuesday 5 shows what one actually is, in six beats, and the one rule that decides whether you let it anywhere near production.

The rule: Automate the action. Never the accountability.
Companion read: masteringobservability.com/p/ai-sre-inefficient-queries

Free chapter: masteringobservability.com/subscribe

Hosted on beehiiv. Read the full Privacy Policy at beehiiv.com/privacy.

Everyone is talking about observability 2.0 and wide events. This is the plain-English version, and the honest part: they are powerful, and they are not new.

I cover what a wide event actually is, why it answers the questions logs and metrics cannot, the canonical-logs heritage that predates the hype, where it fits with tracing and OpenTelemetry, and the honest ledger of cost and cardinality, so you can decide whether it belongs in your estate.

Read the full written version: https://www.masteringobservability.com/p/what-is-a-wide-event

Free chapter of Metrics & Mayhem: https://www.masteringobservability.com/metrics-and-mayhem/free-chapter

The demo shows you the tower, the agent that fixes the incident on its own. It never shows you the year underground that made it possible: the clean telemetry, the tags, the traceability.

This Signal Drop is about why AI ops is only ever as good as the foundation beneath it, and why putting an agent on a weak foundation does not fix your gaps; it exposes them at speed. What you'll take away: why an AI agent reasoning over messy data is confidently wrong; the three unglamorous parts of the foundation (telemetry, tags, traceability); why the groundwork pays for itself before any AI arrives; and a one-incident test of where your foundation actually stands.

Newsletter and a free chapter of Mastering Observability:

https://www.masteringobservability.com/subscribe

Metrics, logs and traces tell you the box is hot. They cannot tell you which function is burning the CPU. OpenTelemetry Profiles, the fourth signal, finally can.

In this Tech Tuesday, Allan Mann explains continuous profiling: what an OpenTelemetry profile actually is, continuous versus on-demand, the eBPF collector that runs as an OTel Collector receiver, and how profile samples correlate to trace_id and span_id so a flame graph lines up against the exact trace. Plus the honest caveats: it is alpha, the tooling is catching up, and always-on data carries a real storage and cardinality cost.

Watch on YouTube: https://youtu.be/q4Byz7OLgVY

Read the companion explainer: https://www.masteringobservability.com/p/what-is-opentelemetry-profiling

Newsletter and a free chapter of Mastering Observability: https://www.masteringobservability.com/subscribe

When can you let an AI agent act on its own in production, and when does a human have to stay on the gate?

This Signal Drop gives you the whole AI-ops governance model in one line: you decide what to automate by blast radius, not by how confident the tool is. Low blast radius, let it run. High blast radius, a human decides, every time. Because you can automate the action, but you can never automate the accountability.

What you'll take away:

- Why "how much can we automate?" is the wrong question to start with

- The blast-radius model: the open gate, the conditional gate, and the closed gate

- Why confidence is the wrong axis, and worst-case is the right one

- Two incidents on one night, and how the same tool earns two different answers

- The one habit: label your top five automated remediations and name who owns each gate.

Metrics and Mayhem, a CTO's guide to observability that actually works, is out now in Kindle, paperback and hardback. Grab a free chapter at masteringobservability.com/subscribe.

Your observability is working, nothing is on fire, and that is exactly when someone asks why you are still paying for all of it. This Signal Drop is about the prevention paradox: the better your monitoring works, the harder it becomes to fund, because the value is invisible by design.

The fix is not better tooling; it is better measurement. Put a number on the invisible side of the scale, and speak to the business in the currency it actually cares about, before the renewal lands.

What you'll take away:
- Why a quiet quarter is the worst sales pitch your monitoring will ever make
- The prevention paradox, and why being good at your job hides your value
- The one-slide habit that makes a near-miss visible before the renewal
- How to explain observability in the language the business actually cares about
- How to decouple your instrumentation from your vendor without a two-year migration

Chapters:
00:00 The Observability Prevention Paradox
05:32 Make Invisible Value Visible
07:04 Signal Check: Listener Q&A
07:38 Explaining Observability in Business Terms
08:42 Decoupling from Your Observability Vendor
10:15 Closing Thoughts & How to Connect

Metrics and Mayhem, a CTO's guide to observability that actually works, is out now in Kindle, paperback and hardback. Grab a free chapter at https://www.masteringobservability.com/metrics-and-mayhem/book

AI agent observability is the missing camera on the agents now acting in your production systems.

AI agents are moving beyond answering questions. They investigate incidents, call tools, analyse production data and, in some cases, apply changes. But when an agent makes a decision, can your team explain exactly what it did and why? Watching only the prompt and the final answer is not enough. The important part is everything in between: the tools it called, the information it received, the choices it made and the paths it decided not to follow. An unobserved agent is an unowned actor.

In this Tech Tuesday, Allan covers:

- Why is watching the prompt and the answer is not enough

- What an AI decision graph is

- The four parts every agent trace should capture: ask, decision, action, outcome

- How OpenTelemetry is helping standardise AI agent tracing

- Why does auditability not automatically make an agent safe

- The question every technology leader should ask an AI vendor: show me the trace

The principle is simple: there is no autonomy without auditability. The trace is the camera, not the gate. For any agent changing production systems, human approval should stay firmly in place.

📕 Metrics & Mayhem: A CTO's Guide to Observability That Actually Works is out now on Amazon and Kindle. Free chapter: https://www.masteringobservability.com/metrics-and-mayhem/free-chapter

Tech Tuesday from Metrics & Mayhem: one important technology topic, explained clearly and practically. New episodes every Tuesday.

#AIAgentObservability #AIAgents #Observability #OpenTelemetry #SRE #TechTuesday #MetricsAndMayhem

Good tools answer the question so fast you forget you were the one who asked it. Dashboards, runbooks and AI incident summaries are all useful, and all quietly retire the questions you used to ask yourself. Curiosity is a muscle: when your systems stop requiring it, you stop exercising it, and a genuinely novel failure then finds you out of practice.

This week: why good tooling de-skills incident investigation, a Hard Stop worth sitting with, a 20-minute habit to keep the muscle warm, and the new Signal Check segment answering two listener questions (how to find the change behind an incident, and whether AI is really improving IT operations).

None of this is anti-AI. Automation, mature tooling and plain seniority all quietly retire your questions. The point is to notice and put the reps back on purpose.

Metrics & Mayhem: A CTO's Guide to Observability That Actually Works is out now on Amazon and Kindle. Free chapter: https://www.masteringobservability.com/metrics-and-mayhem/free-chapter

MCP, the Model Context Protocol, is the open standard for connecting AI models to your tools and data. In the first Tech Tuesday, we explain what MCP is, the problem it solves, and where it fits in 2026, in plain terms. Introduced by Anthropic in late 2024 and now stewarded by the Agentic AI Foundation under the Linux Foundation, MCP is fast becoming the way agents reach real systems, which makes it an operational question, not just a lab one.

One thing, explained well. That is Tech Tuesday.

Free chapter of Metrics & Mayhem: https://www.masteringobservability.com/metrics-and-mayhem/free-chapter

Automate incident triage, and your best responders quietly lose the reps that kept them sharp, until the automation paradox meets a novel fault.I once watched one of the best incident teams I have worked with get quietly worse over about six weeks, and every change we made was, on paper, an improvement. We automated triage. The routine got handled, fewer people were woken, and the numbers looked great.Then a novel fault arrived, the kind that is in no runbook, and the team stalled. The automation had never seen it either, and the humans were rusty, because we had taken away their reps. This episode is about the hidden price ofautomating the routine, and how to keep your team's edge on purpose.The habit: run one live incident drill the automation cannot help with. A novel fault, no runbook, your best people, cold.The book is out. Metrics & Mayhem: A CTO's Guide to Observability That Actually Works. Kindle is live now; paperback and hardback launched on 1 June.Get your free chapter here: https://www.masteringobservability.com/metrics-and-mayhem/free-chapterNewsletter: https://masteringobservability.comLinkedIn: https://www.linkedin.com/in/allanmann1/