This is Fine! A podcast about resilience engineering and software

If you want to come to the event to chat with Layer Aleph folks about Crisis Engineering you can sign up here: https://resilienceinsoftware.org/events/272451

A small snippet of our communication with Riverside (we’re pretty sure it’s AI, folks!)

—---------------------

Hi Colette,

Thank you for pushing back on our last reply, and I owe you a correction.

You were right about pressing record

Our logs confirm you did start a recording. The countdown you saw was real, and telling you the record button wasn't clicked was wrong. I'm sorry for that.

What the recording data shows

Here's the difficult part. That first recording started and then stopped 13 seconds later. Everything after that point wasn't being captured, even though your studio session stayed open and you carried on recording your episode.

You then started a second recording about eight minutes afterwards, which is the 8-minute 48-second take with Clint.

Those two are the only recordings that exist for that day. There's nothing else stored and nothing recoverable from the period in between, because no content was being written during it.

—----------------------

You can find Layer Aleph here: https://layeraleph.com/

And a good roundup of their recommended reading that was referenced for Crisis Engineering is here: https://crisisengineering.layeraleph.com/reading-list/

Some writing on Mikey’s Pomona College course is here: https://www.pomona.edu/news/2024/02/12-cover-time-back-campus-mikey-dickerson-01-pilots-return-pomona

Weaver referenced Karl Weick early on as an influence. Some of Karl Weick on sensemaking can be found here: https://pubsonline.informs.org/doi/abs/10.1287/orsc.1050.0133

And he also is part of the HRO (high reliability organization) group, some of that writing is here: https://journals.sagepub.com/doi/abs/10.2307/41165243

Obviously if we talk about Richard Cook we start with https://how.complexsystems.fail/

Downs’ Inside Bureaucracy seems to have limited printing options: https://a.co/d/08dulLA1

A small plug of Pandoc if you’re looking to get markdown into a .doc format: https://pandoc.org/

It has been 0 episodes since we have mentioned the Dependency comic from xkcd: https://www.explainxkcd.com/wiki/index.php/2347:_Dependency

If you want to come to the event to chat with Layer Aleph folks about Crisis Engineering you can sign up here: https://resilienceinsoftware.org/events/272451

Chad’s thesis from Lund is here: https://lup.lub.lu.se/luur/download?func=downloadFile&recordOId=9076274&fileOId=9076276

The great Barry Turner book Man-Made Disasters from the 1970s has recently seen 2nd edition released, it used to be a very rare and difficult book to find: https://www.amazon.com/dp/0750620870

Some Patterson and/or Woods on Handover Communications in medical settings: https://www.jstor.org/stable/45126978 and https://journals.sagepub.com/doi/abs/10.1177/2327857919081001 and https://qualitysafety.bmj.com/content/21/Suppl_1/i19.short and re: NASA - https://journals.sagepub.com/doi/abs/10.1177/107118139704100155

Communication coordination during handovers for railway dispatch: https://rosap.ntl.bts.gov/view/dot/9020

Chad’s talk on Handover Communications at SRECon can be found here: https://www.usenix.org/conference/srecon23americas/presentation/todd

The Law of Fluency comes from Woods’ work in Cognitive Systems Engineering, here’s a paper that mentions it: https://www.researchgate.net/profile/David-Woods-19/publication/224753269_The_Messy_Details_Insights_From_the_Study_of_Technical_Work_in_Healthcare/links/0deec51bb2fc4d4e3d000000/The-Messy-Details-Insights-From-the-Study-of-Technical-Work-in-Healthcare.pdf but there are many other books and papers that do as well.

Common Ground in Joint Activity is another theme that comes up in the Cognitive Systems Engineering world. Gary Klein has written about it lots of places, perhaps most notably here: https://ieeexplore.ieee.org/abstract/document/1363742 and here: https://books.google.com/books?hl=en&lr=&id=371wV4dI7ckC&oi=fnd&pg=PA139&dq=common+ground+joint+activity&ots=1RKpCK7p7g&sig=XkyZygexYgU36UC-QtIzM3S67Cg#v=onepage&q=common%20ground%20joint%20activity&f=false

A Tale of Two Safeties is not by Allspaw/Cook, it’s by Erik Hollnagel: https://hf-sts.com/wp-content/uploads/2018/07/3689b-a_tale_of_two_safetieshollnagel.pdf

Aerobarrier is super cool if you want to check out how it works: https://glbipro.com/aerobarrier-2/

Colette isn’t drinking alcohol, it’s a delicious herbal mocktail by Casamara Club: https://www.casamaraclub.com/products/como

(Maybe we can get them to sponsor us someday)

On the passive house that survived the Palisades fire: https://www.reddit.com/r/Damnthatsinteresting/comments/1hy22ui/house_designed_on_passive_house_principles/

Layer Aleph https://layeraleph.com/ folks have written Crisis Engineering: https://bookshop.org/p/books/crisis-engineering-time-tested-tools-for-turning-chaos-into-clarity-marina-nitze/44736d1287a7da6e and there will be an RISF event with them in September https://resilienceinsoftware.org/events/272451 , which means we’re hoping to get them on the pod in August, FYI!

 More about the Lund program here: https://www.humanfactors.lth.se/

Dr. Woods’ OSU program is here: https://ise.osu.edu/human-systems-integration

Our semi-academic, semi-practical space is the Resilience in Software Foundation slack - lots of folks from all ends of the spectrum talk there about RE concepts and practical applications for how to solve things (and pet pictures). You can get access to it by joining the Resilience in Software Foundation: resilienceinsoftware.org

The program in Australia is NOT in Melbourne, it’s in Brisbane at Griffith University, you can see Sidney Dekker’s profile there: https://experts.griffith.edu.au/19027-sidney-dekker as part of the Humanities and Social Sciences school. Drew Rae is also there, and his podcast is here: https://safetyofwork.com/

https://www.adaptivecapacitylabs.com/ and https://www.uptimelabs.io/ are very practical approaches to a lot of these things. Blackrock 3 https://www.blackrock3.com/ does training for incident command.

The new website is live! thisisfinepod.com

You can find John Allspaw at Adaptive Capacity Labs: https://www.adaptivecapacitylabs.com

Mike McGill, the skateboarder: https://en.wikipedia.org/wiki/Mike_McGill

Annie Duke’s Thinking in Bets, referenced by our question-asker is a great one: https://bookshop.org/p/books/thinking-in-bets-making-smarter-decisions-when-you-don-t-have-all-the-facts-annie-duke/31466984521c3d8a?ean=9780735216372&next=t

Naturalistic Decision Making has its own association, which has a ton of resources (and a conference!) - https://naturalisticdecisionmaking.org/

They also have a podcast! https://naturalisticdecisionmaking.org/new-podcast/

Gary Klein is the NDM guy - https://bookshop.org/p/books/seeing-what-others-don-t-the-remarkable-ways-we-gain-insights-chief-scientist-gary-klein/c4ae5e017fe005ff?ean=9781610393829&next=t

We contrast him and his style of approaching cognition and decision making with Kahneman and Tversky.

Kahneman and Tversky wrote a lot, but Judgement Under Uncertainty is probably the most famous? https://www.science.org/doi/abs/10.1126/science.185.4157.1124

And Kahneman wrote Thinking Fast and Slow: https://bookshop.org/p/books/thinking-fast-and-slow-daniel-kahneman-phd/83a544fe6f98df87?ean=9780606275644&next=t

It has been zero episodes since we’ve mentioned Lisanne Bainbridge’s Ironies of Automation: https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf

But also she has Verbal Reports as evidence of the process operator’s knowledge: https://www.sciencedirect.com/science/article/abs/pii/S1071581979603075?via%3Dihub

And the Etsy Debriefing Guide is super great: https://extfiles.etsy.com/DebriefingFacilitationGuide.pdf

Sidney Dekker and The Field Guide are foundational: https://bookshop.org/p/books/the-field-guide-to-understanding-human-error-sidney-dekker/3a4209dfc8b3a721?ean=9781472439055&next=t

From Dekker’s field guide (pg 47) there is a list referencing Gary Klein’s questions for an incident investigation:

Cues: 

What were you seeing?

What were you focusing on?

What were you expecting to happen?

Interpretation: 

If you had to describe the situation to your colleague at that point, what would you have told?

Errors: 

What mistakes (for example in interpretation) were likely at this point?

Previous experience/knowledge:

Were you reminded of any previous experience?

Did this situation fit a standard scenario?

Were you trained to deal with this situation?

Were there any rules that applied clearly here?

Did any other sources of knowledge suggest what to do?

Goals:

What were you trying to achieve?

Were there multiple goals at the same time?

Was there time pressure or other limitations on what you could do?

Taking action:

How did you judge you could influence the course of events?

Did you discuss or mentally imagine a number of options or did you know straight away what to do?

Outcome:

Did the outcome fit your expectation?

Did you have to update your assessment of the situation?

John mentioned Uptime Labs, who do staged worlds for software incidents: https://uptimelabs.io/

Facets of Complexity in Situated Work is here: https://www.researchgate.net/publication/345523195_Facets_of_Complexity_in_Situated_Work

On the Jamie Zawinski quote: https://regex.info/blog/2006-09-15/247

If you don’t know the parable of the blind men and the elephant: https://en.wikipedia.org/wiki/Blind_men_and_an_elephant

Mitchell Hashimoto’s post on leaving Github: https://mitchellh.com/writing/ghostty-leaving-github

The Reddit post on Github’s availability historically (that we find questionable): https://www.reddit.com/r/github/comments/1rnvhs9/githubs_historic_downtime_scraped_and_plotted/

A reminder, the Messy 9 are: congestion, cascade, conflict, lag, saturation, friction, tempo, surprise, tangles

We have sometimes loved his stuff, but Gergely is annoying us with these posts: https://newsletter.pragmaticengineer.com/p/the-pulse-is-github-still-best-for?r=78c7k&utm_medium=email

https://x.com/GergelyOrosz/status/2048017382036082706

You can find the RISF store with Hindsight Bias merch here: https://www.bonfire.com/store/risf/

You can find a copy of Richard Cook’s Two Years Before the Mast at Lorin’s Blog: https://surfingcomplexity.blog/wp-content/uploads/2026/03/twoyearsbeforethemast.pdf

A reminder, Richard Cook’s How Complex Systems Fail can be found at http://how.complexsystems.fail

Some writing on the 1996 Annenberg conference: https://www.researchgate.net/publication/351953417_Coming_Together_The

Folk models paper (not by Woods, by Dekker and Hollnagel), which is specifically targeting Situational Awareness as being a folk model: https://link.springer.com/article/10.1007/s10111-003-0136-9

Some stuff about SNAFU Catchers: https://www.snafucatchers.com/

And https://snafucatchers.github.io/

Eric referenced our conversation with Beth Long about Building and Revising Adaptive Capacity, which she co-wrote with Richard Cook about New Relic’s real-life example of resilience engineering: https://youtu.be/A_rU4-M61Hk  and https://www.sciencedirect.com/science/article/abs/pii/S0003687020301903?via%3Dihub for the paper

Erik Hollnagel’s RAG get’s referenced: https://erikhollnagel.com/onewebmedia/RAG%20Outline%20V2.pdf

Once again, we link you to Lorin’s Law: https://surfingcomplexity.blog/2017/06/24/a-conjecture-on-why-reliable-systems-fail/

Eric is referencing Lund, that is their Human Factors and Systems Safety program: https://www.humanfactors.lth.se/

Check out Crisis Engineering! https://crisisengineering.layeraleph.com/crisis-engineering-the-book/

The upcoming RISF event on Practice of Practice Gamelan: https://resilienceinsoftware.org/events/245030

Colette’s talk at SRECon intro: https://www.usenix.org/conference/srecon26americas/presentation/alexander

Clint’s talk at SRECon intro: https://www.usenix.org/conference/srecon26americas/presentation/byrum

Dan Slimmon is an excellent engineer (per Clint’s shoutout) and ALSO an excellent podcast creator/host: https://techblows.net/

Michelle Brush’s Keynote summary is here: https://www.usenix.org/conference/srecon26americas/presentation/brush

Jevon’s Paradox: https://en.wikipedia.org/wiki/Jevons_paradox

Dr. Nicole Forsgren’s talk summary: https://www.usenix.org/conference/srecon26americas/presentation/forsgren

DORA is always worth a dive into if you haven’t taken a look yet: https://dora.dev/

The blog post Colette mentioned comparing AI gold rush to Mao’s Revolution: https://leehanchung.github.io/blogs/2026/04/05/the-ai-great-leap-forward/

Many people have written about why MTTR is a bad metric to track, you can read a write up from Adrian Hornsby here: https://newsletter.resiliumlabs.com/p/mttr-problems-better-incident-metrics

And watch the OG, Courtney Nash, speak about it here: https://www.youtube.com/watch?v=uhCgBOHo8EY

Beth Long’s SRE Soundbath: https://www.usenix.org/conference/srecon26americas/presentation/long

Vanessa Huerta-Granda’s talk is summarized here: https://www.usenix.org/conference/srecon26americas/presentation/huerta-granda

Martin Smith and Abe Hoffman’s talk is summarized here: https://www.usenix.org/conference/srecon26americas/presentation/hoffman

Some information about Metrist: https://vault42consulting.com/about/portfolio/metrist

AI Agents Good Bad and Ugly talk: https://www.usenix.org/conference/srecon26americas/presentation/budichenko

The CAST talk: https://www.usenix.org/conference/srecon26americas/presentation/barroso

Engineering a Safer World by Nancy Leveson is worth a look: https://bookshop.org/p/books/engineering-a-safer-world-systems-thinking-applied-to-safety-nancy-g-leveson/57b01ef464f9f81b?ean=9780262533690&next=t

Erik Hollnagel wrote the book on FRAM and it has a lot of support in the safety world across industries: https://functionalresonance.com/ and https://etn-peter.eu/2021/02/11/fram-in-a-nutshell/ are good resources.

Daria Barteneva’s closing keynote on game theory and SRE was great: https://www.usenix.org/conference/srecon26americas/presentation/barteneva

Some good stuff on Above the Line/Below the Line, if you’re curious: 

https://queue.acm.org/detail.cfm?id=3380777

https://www.youtube.com/watch?v=xA5U85LSk0M

Lorin Hochstein’s closing keynote on storytelling was rad: https://www.usenix.org/conference/srecon26americas/presentation/hochstein

SRECon EMEA 2026 (in Dublin) has their CFP up: https://www.usenix.org/conference/srecon26emea/call-for-participation

As always, you can check out the Resilience in Software Foundation at resilienceinsoftware.org

You can find the 2025 DORA Report here: https://dora.dev/research/2025/dora-report/

Read more of Fred’s work/opinions here: https://ferd.ca/

If you want to know more about Lund’s Human Factors and Systems Safety program, you can read here: https://www.humanfactors.lth.se/

DORA has some good writeups of generative leadership and Westrum’s model here: https://dora.dev/capabilities/generative-organizational-culture/

We can reset the counter, it’s been 0 episodes since we mentioned Lorin’s Law: https://surfingcomplexity.blog/2017/06/24/a-conjecture-on-why-reliable-systems-fail/

Fred writes well about the Law of Stretched Systems: https://ferd.ca/the-law-of-stretched-cognitive-systems.html

We’re still trying to schedule a DORA event with our friends who make the report, but keep an eye out on https://resilienceinsoftware.org/events - it will pop up there when we do!

The Keewenaw snow gauge that Colette mentioned is a tourist attraction. If you want to see where measurements are at for the season you can find them here: https://www.pasty.com/snow/

The paper we’re talking about today can be found here: https://www.sciencedirect.com/science/article/abs/pii/S0003687020301903

If you want to know more about SNAFU Catchers, you can see their website here: https://www.snafucatchers.com/

They produced the STELLA report: https://snafucatchers.github.io/

Richard Cook’s Bone Talk is kind of famous - here’s a version from REDeploy: https://www.youtube.com/watch?v=8LbePBiOvZ4

Some writing from New Relic about NERFs: https://newrelic.com/blog/observability/best-practices-incident-commander-training

We failed to mention it in the podcast itself, but Michael Wettick did a great thesis at Lund on asking for help in software operations incidents: https://lup.lub.lu.se/luur/download?func=downloadFile&recordOId=9150096&fileOId=9150099

Speaking of Hitchhiker’s Guide, etsy has some cool merch: https://www.etsy.com/listing/1071043200/dont-panic-hitchhikers-guide-to-the

You can find David Woods’ paper on Graceful Extensibility here: https://link.springer.com/article/10.1007/s10669-018-9708-3

Our Paper Club event on this paper on March 17th can be signed up for here: https://resilienceinsoftware.org/events/164680

Colette mentioned Menlo Innovations https://menloinnovations.com/ and Atomic Object https://atomicobject.com/ who both build custom software for folks. The CEO of Menlo is Richard Sheridan who wrote Joy, Inc. - https://bookshop.org/p/books/joy-inc-how-we-built-a-workplace-people-love-richard-sheridan/7677689?ean=9781591847120&next=t

Chad Todd’s thesis on Handovers in Software Operations is worth a read: https://lup.lub.lu.se/luur/download?func=downloadFile&recordOId=9076274&fileOId=9076276

Clint refers to Zingerman’s and their servant leadership model, one of Colette’s favorite places to learn about leadership from. If you want to know more, go to https://www.zingtrain.com/ and in particular, read https://shop.zingtrain.com/products/a-lapsed-anarchists-approach-to-being-a-better-leader

Special thanks to John Allspaw, Sheeri Cabral, Martin Smith, and David Woods for joining us!

Ben Affleck’s been making the promo rounds, but the specific convo we reference is recapped here: https://www.moviemaker.com/ben-affleck-ai-explains/

The Messy 9 are:

congestion

cascade

conflict

lag

saturation

friction

tempo

surprise

tangles

Dave’s been doing a set of videos on Resilience Engineering, some of which have some crossover with the Messy 9 - you can find the first one here:

https://resiliencefoundations.github.io/video-1-introduction-pt-1-it's-all-about-viability.html

Previous TiF episode on the messy 9:

https://www.thisisfinepod.com/the-pod/complex-systems-and-the-messy-nine-wspecial-guests-dave-woods-and-john-allspaw

Richard Cook on Above the Line/Below the Line: 

Written - https://dl.acm.org/doi/pdf/10.1145/3379510

A good excerpt from a talk from John Allspaw on Above the Line/Below the Line: https://www.youtube.com/watch?v=8bxj-FLEi10&list=PLb1aZTnPf3-OMChMkrr6WsokRI6LOnuem

Colette mentioned the competence knowledge model: https://en.wikipedia.org/wiki/Four_stages_of_competence

There’s a good argument based on the conversation here that AI makes it harder for Consciously Incompetent people to graduate to Conscious Competence. And, in Martin’s case, it makes Unconsciously Competent folks need to backtrack into Conscience Competence to “teach” it how to do things they don’t always think about.

We can reset the clock to 0 episodes since we’ve mentioned the Ironies of Automation: https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf

There is a good blog on Jamie Zawinski’s saying on regular expressions here: https://regex.info/blog/2006-09-15/247

Alex Gorbachev and The Battle Against Any Guess seems to have become a paper https://www.researchgate.net/publication/251255185_Battle_Against_Any_Guess

Dave talks about Robust Yet Fragile as part of Resilience Engineering here: https://www.youtube.com/watch?v=gFotUdLL2zs

Lorin Hochstein’s blog post that Dave is referencing is https://surfingcomplexity.blog/2026/01/19/amdahl-gustafson-coding-agents-and-you/

Fred writes a good one on the Law of Stretched Systems: ​​https://ferd.ca/the-law-of-stretched-cognitive-systems.html

The 1985 paper Dave keeps mentioning could be any number of things he released that year, but I have a hunch it’s this one: https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/511 or this one: https://link.springer.com/chapter/10.1007/978-3-642-50329-0_11

Dave references a lot of things around the economic sustainability around AI, and Ed Zitron has been writing quite a bit about that for the last year and change. See: https://www.wheresyoured.at/wheres-the-money/

https://www.wheresyoured.at/big-tech-2tr/

Among others.