Off-Call

• Defining SRE as "Ops with Boundaries"
• WhatsUp Gold
• Has System Visibility Increased Over Time?
• Anti-Pattern: SRE as Keepers of Observability
• SRE Education at the School of Hard Knocks
• Reflecting on 15 Years of On-Call
• Holidays On-Call
• Tracking and Quantifying Toil
• Should Engineering Managers Be On-Call???

• Connect on LinkedIn
• Check out his services at Certo Modo
• Recommended Reads
  • more about quantifying toil  In Defense of Time Tracking  on the Certo Modo blog
  • The Goal  aka spiritual predecessor to the Phoenix Project
• Send out the bat signal at sev0.help
• Listen to Amin's podcast Reliability Rebels

Follow Off-Call  x Paige: 

• Mastodon
• Bluesky
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io

Guest: Hanson Ho, Android Architect at Embrace

• Mastodon
• LinkedIn
• Bluesky
• Blog
• Twitter

Recommended Resources

• DroidconSF 2023: Combating sampling bias in production
• OTel Community Day 24: OpenTelemetry for Mobile Apps
• O11ycast: Ep. #72, Mobile Observability with Hanson Ho
• Test Case Scenario: Mobile App Testing Automation
   

Check out Differential Diagnosis (DDx) yourself with this self-guided demo

Follow Off-Call  x Paige: 

• Mastodon
• Bluesky
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io

Topics covered

• What it’s really like being on-call for mobile apps
• Reasons mobile teams are siloed in eng orgs
• Whether or not mobile observability is a misnomer
• How OpenTelemetry can bridge the gap between backend and mobile
• Paige learns with mobile there is no rollback….only roll forward

Guest: Hanson Ho, Android Architect at Embrace

• Mastodon
• LinkedIn
• Bluesky
• Blog
• Twitter

Recommended Resources

• DroidconSF 2023: Combating sampling bias in production
• OTel Community Day 24: OpenTelemetry for Mobile Apps
• O11ycast: Ep. #72, Mobile Observability with Hanson Ho
• Test Case Scenario: Mobile App Testing Automation
   

Take a self-guided tour of Chronosphere Telemetry Pipelines .

Follow Off-Call  x Paige: 

• Mastodon
• Bluesky
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io

Covering the challenges of managing a self-hosted observability stack and Dan’s unique on-call experiences, like troubleshooting an incident from an airplane! Later we dive into the Production Engineering initiative at HashiCorp designed to immerse software engineers in proactively exploring observability data to gain a deeper understanding of system behavior. Of course we cover the significance of understanding roles in incident management, like incident leads and scribes, and how to identify your personal style of leading incidents. 

Check out Dan's course Leading Incidents   and catch up on his Monitorama talk "No Observability Without Theory" or read the blog version. Connect with him on Mastodon 

Follow Off-Call  x Paige: 

• Mastodon
• Bluesky
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io

• [00:30] Matthew's First Monitoring Tool was.....
• [02:08] 11 years (and counting) on-call!
• [07:40] That Time Storing A Password In Cleartext Saved The Day
• [12:29] Unwinding Off Call & Signs of Burnout
• [17:13] Responding To A Customer's Malicious Insider Incident
• [19:32] What app devs need to know about monitoring databases
• [28:32] Has database monitoring gotten better over time?
• [32:15] Operating DBs in the time of Kubernetes
• [34:44] Wrap Up 

Paige chats with Matthew Sanabria, Staff SRE at Cockroach Labs, about memorable moments from his 11 years on-call, the warning signs of burnout, and what developers and operators can do for each other to improve database observability for everyone!

Find Matthew

• Personal Site https://matthewsanabria.dev/
• Twitter https://twitter.com/sudomateo
• Mastodon https://mastodon.online/@[email protected]
• Cockroach Labs https://www.cockroachlabs.com/
• Platform Engineering New York (PENY) Meetup https://www.meetup.com/platform-engineering-new-york/

Follow Off-Call  x Paige: 

• Mastodon
• Bluesky
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io

Overview

• 00:45 | Network Observability FTW!
• 07:09  | How has networking changed in the last 5-10 years?
• 10:28   | Has Kubernetes made network engineers' lives easier?

Paige and Leon cover need to know networking concepts for developers and answer the very important questions “How much has networking really changed in the last 5-10 years?” AND “Has Kubernetes made any of this stuff easier?!”

Leon’s Links

• Adatosystems.com
• LinkedIn
• Mastodon
• Bluesky

Recommended Resources

• Alerts don't suck YOUR alerts suck! 
• Technically Religious
• Imperfect Genius - Rachel Foster
• Screaming In The Cloud - Corey Quinn
• www.paigerduty.com
• Telemetry Now
• What's New at Kentik
• Kentik Close-Up

Follow Off-Call  x Paige: 

• Mastodon
• Bluesky
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io

It's here! Part 2 (of 3) of my conversation with Leon Adato!.

We chat about how to relate monitoring to the only 3 things businesses care about, how  the medical field's "See One, Teach One, Do One" can be adapted for onboarding devs, the real life give and take with on-call, and our requirements for paging alerts!

• 03:33 On-Call Advice for Newbies
• 09:25 Leon's Off-Call Life
• 13:32 Tempering On-Call With A Dose of Reality
• 15:27 The Only Experience That Matters Is The Users'
• 18:14 Wrap Up 

Find Leon around the web:

• Alerts don't suck YOUR alerts suck! 
• Adatosystems.com
• LinkedIn
• Mastodon
• Bluesky
• Telemetry Now
• What's New at Kentik

Follow Off-Call  x Paige: 

• Mastodon
• Bluesky
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io

About Leon Adato

In my sordid career, I have been an actor, bug exterminator and wild-animal remover (nothing crazy like pumas or wildebeests. Just skunks, snakes, and raccoons.), electrician, carpenter, stage-combat instructor, ASL interpreter, and Sunday school teacher. Oh, yeah, I’ve also worked with computers.

While my first keyboard was an IBMs electric, and my first digital experience was on an Atari 400, my professional work in tech started in 1989 (when you got Windows 286 for free on twelve 5¼” when you bought Excel 1.0). Since then I’ve worked as a classroom instructor, courseware designer, helpdesk operator, desktop support staff, sysadmin, network engineer, and software distribution technician.

Then, about 25 years ago, I got involved with monitoring. I’ve worked with a wide range of tools: Tivoli, BMC, OpenView, janky perl scripts, Nagios, SolarWinds, DOS batch files, Zabbix, Grafana, New Relic, and other assorted nightmare fuel. I’ve designed solutions for companies that were modest (~10 systems), significant (5,000 systems), and ludicrous (250,000 systems). In that time, I’ve learned a lot about monitoring and observability in all its many and splendid forms.

Find Leon around the web:

• Adatosystems.com
• LinkedIn
• Mastodon
• Bluesky
• Telemetry Now
• What's New at Kentik

Follow Off-Call  x Paige: 

• Mastodon
• Bluesky
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io