Arrested DevOps

Joe Lahey, Matty, Trevor, Bridget and Jessica record the 2021 year-end wrap-up in one room, which Joe admits has an agenda line reading "stuff and junk." The conversation covers YouTube and Spotify numbers, favorite and most popular episodes, high points of the year, a long stretch on pets, and what everyone is looking forward to in 2022. The cold open is Jessica: "It really is about getting your system to teach you what you need, not about dumping fucking metrics out your butt."

Matty opens with the YouTube year-in-review email: almost 6,000 views and 37,000 watch minutes, which is less than what one episode of the podcast gets listened to. That's why the show doesn't optimize for video. Matty adds that doing the show on Twitch would be less work than the podcast since there's no editing, and Bridget does not want a comment stream. Hangouts on Air history gets a mention: when Jeffrey Snover tweeted the join link instead of the watch link, a few extra people showed up on the call, and the show's highest live viewership was around 12.

The podcast had 11 episodes in 2021, which Matty notes is close to one a month and not unusual for the show, with four more ready to go. As of December 15th the most listened-to episode was All Things Docker. Bridget's theory is that recording it live on Zoom at US lunchtime, early in 2021, started a cascade of people linking to it. Second was Foundational Practices with Johan Abildskov, where Matty kept mishearing the pronunciation of the guest's name, which turned into a conversation about communication breakdown from context. Third was Doing Releases Right, the first episode of the year. Cumulative downloads are 1.6 million.

Spotify stats get their own stretch: follower count up 62 percent and listeners up 16 percent, top listening countries of Slovenia, Kenya, Serbia, Lesotho and Nigeria, and 137 people for whom the show is the most-listened podcast on Spotify. Over time the show has been listened to more than 14,000 times in Minneapolis-St. Paul, and San Francisco is the highest city. Bridget calls it "our podcast on podcasting."

Jessica picks Words Are Hard with Emily Freeman, with one complaint: Matty and Emily kept referring to a class with a "secret formula for presenting to power" and never said what it was. Matty's pick is Drawing DevOps, with the sketch-recording side of Ashton's work and what drawing taught about DevOps over the years. Joe says it would have been a good episode for Joe to be on, since Joe came to DevOpsDays as an outsider who was voluntold into running audiovisual. Bridget's picks are the Open Service Mesh multicluster episode and Brigade with Kent Rancourt, for talking to maintainers about things that aren't in the repo. Joe's favorite was the Tech Twitter episode, purely as an editing challenge with around 15 guests.

Matty recommends Descript for editing, since the transcript is built into the editor and cuts are made against the text, though it can be overzealous about filler words and "verbal tics." Separate tracks per speaker (Zencastr or Squadcast) make the transcription much better, and the Deserted Island DevOps episode was hard because the tool can't tell ten voices apart. Jessica says text-based editing is the future for audio but not for video. Matty apologizes for missing accessibility and says transcribing the whole back catalog would cost over $45,000, so the show is doing it going forward.

The pets run long: "free as in puppy" and "free as in mattress" for open source, the dog and cat gods comic, and Joe and Bridget's house name, Gazebo 7, from a Simpsons episode, which is also the trivia team name. There's also a section on media: Trevor concedes that Babylon 5 is better than DS9, Matty and Bridget discuss the Cowboy Bebop reboot, Jessica watched Schmigadoon, and Joe reports on a water glass eggs video that ended with "We have the technology. They're called grocery stores or refrigerators."

Bridget is looking forward to being in charge of less: Andy Fehner takes over DevOpsDays Minneapolis, with Bridget in an advisory role, which leaves room for different decisions and other voices. Jessica is returning to a family beach trip at Tybee Island, Georgia. Trevor plans another century, a Seattle to Portland ride, and a triathlon in Chicago. Matty, who took over as global DevOpsDays chair from Bridget, hopes for DevOpsDays Chicago back in person in May, with the CFP open until the end of January, and for KubeCon in Valencia. Matty also ties the ADO streak of episodes to Seinfeld's method, and to normalization of deviance once a streak is broken. Joe is looking forward to a rescheduled Canadian fishing trip.

Matty talks with Emily Bache, a technical agile coach and author who lives in Sweden and works at ProAgile, about technical agile coaching, testing and ensemble programming. Emily has been a professional developer for more than 20 years, started as a Java and Python programmer, got into extreme programming around 2000, wrote a book on coding dojos that came out in 2011, and published a new book in January, Technical Agile Coaching with the Samman Method. Samman is a Swedish word meaning together, which Emily chose so people could find it online. Emily says the method draws on many influences and is "not just me."

Emily's colleagues at ProAgile coach leadership, teams, processes and management. The technical coaching Emily does focuses on developers, to some extent testers, and on code and technical ways of working, which needs a different skill set and gives different results, and "to be really successful, you need both kinds of coaching." Matty says that's a gap in many Agile transformations, which stop at how to organize work, and the software engineering practices get left to the teams. Emily says DevOps is also very much about technical practices, and says the title technical agile coach, not DevOps consultant, is historical, since Agile came first.

For Emily, testing is about feedback loops: they give the feedback needed to write good code, know you're on track and know it's safe to deliver. Emily teaches unit test design and has worked as an architect on larger integration and system tests, and does a lot with approval testing. Emily explains that in a regular test you arrange, act and assert, while in an approval test you compare the system's output to a version you approved earlier, recorded from the system, using a straight diff. In essence it's "a fancy way of doing assertEquals," with a human decision on whether to approve. It works best with tools, and Emily is involved in two open source ones, Approvals and TextTest, both linked in the existing notes.

Matty raises shift left and the worry that getting developers to write tests means no testers. Emily says there's absolutely a place for people skilled at testing: they set the strategy for where automation matters, do exploratory testing and find areas lacking coverage, and can contribute to automation. Matty adds that Etsy's John Allspaw answered the question of why Etsy still had a web operations team with "I've got so much for them to do," and the same applies to testers. Matty also notes that DevOps is named after two roles but has always been about being cross-functional.

Emily says most of the coaching is with developers: take smaller steps, get better feedback, commit more often, practice continuous integration and test-driven development, and learn to split tasks. Developers should push small commits several times an hour, each with passing tests, and the tests should run in a short time, which usually means unit tests. Many developers neglect test design because they think it's not production code. Emily says continuous delivery and pipelines depend on a steady stream of small, safe changes.

Matty asks about criticism of the pull request workflow. Emily isn't a great fan: review forces you to drop what you're doing, the discussion delays merging into master, and teammates can't build on the work until they see it. Emily wants to keep code review and design discussion but not tie it to a pull request or an integration gate, and wrote a blog post on a technique called pre-tested integration. Matty says pull requests suit open source projects where you can't pair with thousands of people, but inside an organization in the same time zone you can just pair, and that PRs encourage long-lived branches. Emily's test: if you diff the code on each team member's machine, the difference should be at most a few hours of work, so everyone designs from the same state of play, and "that's where you start to really see teamwork."

Emily says pairing is a skill, and if nobody taught you, you may not be doing it well. Emily does more ensemble working, which is another word for mob programming, preferred because it sounds friendlier, and Matty adopts it. As coach, Emily can ask the ensemble to write a test now or back out a change to do it in smaller steps. About 10 sessions of 2 hours usually teaches a team to ensemble, and they keep using it for onboarding, starting a new task or a critical bug. Matty says live streaming coding while learning a new project has turned into pseudo pair programming with the audience, and Emily says remote work lowered the barrier to collaboration and remote ensembles work pretty well.

Emily learns by practicing code katas, including doing the same kata in a new language, and designing exercises for refactoring, and likes learning in a group or with a teacher. On whether practice is getting better, Emily says it's big and diverse: at one organization with a 30-year-old C codebase, Emily is helping write better unit tests, while a JavaScript and React team was so agile there was little to teach. If things aren't getting better where you are, you could think about going somewhere else, and the Accelerate report shows the variety of organizations.

Emily's one thing is "keep learning." Emily runs one-hour learning hours with teams: a new technique in about five minutes, an exercise, and a reflection, with a growing collection of lesson plans, scheduled in everyone's calendar, and notes that sometimes people cancel for a crisis. Matty says intentionality around learning matters, and learning is easier to protect when the whole team commits, though it takes teaching other people your expectations. Emily says the Samman method is basically ensemble working plus learning hours.

Approval testing tools mentioned

Matty talks with Steve Giguere, a developer advocate at BridgeCrew (recently acquired by Palo Alto) who previously worked at StackRox, Aqua Security and Synopsys, about what DevSecOps looks like in practice. BridgeCrew is a sponsor of the show, and Matty says Steve is on for the conversation, not because of that. The cold open is Steve on shift left: "It was just us saying it. We weren't actually shifting the word shift left."

Steve's cynical definition is that DevSecOps is "security self-pending an invite to the DevOps party," a term security invented in reaction to the success of DevOps. The legitimate version is embedding security by default in DevOps "so we never have to use that word again." Matty says DevOps is unfortunately named, since it was so called because Agile System Administration was too long for a conference, and recalls Julian Dunn saying security is just another aspect of quality. Steve says people misunderstand DevOps as automation when it's cultural, and that security's self-induced purgatory comes from a world of throwing work over the wall and catching it at the end, so automation feels like the best way in.

Matty adds that in most DevOps transformations, security feels bolted on, and cites a talk called The 5 Love Languages of DevOps: what makes DevOps appealing to a feature builder differs from what lands with security. Audits are theater because people lie or misremember and computers don't, and automated change control won't let a deploy through when "the lights aren't green." Steve says security has to matter to everybody, and automation has to be "drip-fed, not sledgehammered."

Matty asks how to get everyone to care about security without sliding into NoOps. Steve's definition of shift left is that it's not all the way left: a bit of security in the middle during CD for quick preflight checks, and sprinkles of security toward development and even education and threat modeling. Steve recalls static analysis tools that returned 10,000 findings that nobody acted on, versus everybody doing small things, like checking Dockerfile misconfigurations, so low-hanging fruit gets removed along the way. The aim is to find each person's easy version of security that takes no time and has a big impact later.

Matty adds that security tooling has been expensive and not democratized, so it lands on the right, using Qualys licensing as an example while inviting correction, and brings up "trust but verify": developers run checks at commit, and the pipeline verifies. Steve says the industry has made some things easier, with convergence on VS Code plugins that highlight issues in YAML, Terraform, Docker and Node.js, and GitOps, where a check can auto-generate a pull request or a logged suppression. The hard part is large rearchitectures, like monolith to microservices, which confuse many InfoSec people.

Matty says the change is not learning a tool but understanding what Kubernetes does well enough to threat model it, and that nobody can hold the whole system in their head anymore. Steve says it depends on whether you're InfoSec, raised on networking and firewalls, or AppSec, obsessed with the OWASP Top 10, and notes security has its own silo between those two, which should take a page from DevOps. Steve's advice is to understand your attack surface and low-hanging fruit. Basic misconfigurations without CVEs are still bad, probably worse, and misconfiguration gets a bad rap. Steve recalls a podcast in which Ian Coldwater said early Kubernetes had real attackers because of insecure defaults, and now it's mostly misconfiguration.

Steve says security needs to hang out with the DevOps people, drop the culture of no, and include developers in tool-buying decisions so they aren't mandating. Matty says security reviews of new tools are often non-collaborative, handled through a ServiceNow ticket with no context, and Steve says if developers suggest a security tool, "just buy it," because developers like to pull tools and it opens a conversation.

For starting now, Steve says use free tools: the CNCF security landscape, Trivy for container image scanning, which works in VS Code and on the command line, and Checkov from BridgeCrew for scanning infrastructure as code such as CloudFormation and Terraform for things like open S3 buckets. Steve notes that if developers use the open source tool and security wants visibility across 1,000 developers, then you buy the commercial version. Steve hosts a security-themed podcast (the transcript renders the name as CozyCast, and the existing links spell it Cosecast).

Matty shares a line from the Food Fight show's founder (name garbled in the transcript) that the dirty secret of tech podcasting is that it's how you get someone to spend an hour talking to you, and that you can't control your audience. For learning outside security, Steve recommends We Hack Purple, run by Tanya Janca, and certifications for a wide and shallow picture, and Steve did a software lifecycle practitioner certification years ago. For SREs and ops, Steve points to security chaos engineering and a red-blue exercise on a new Terraform deployment. Steve learns from short YouTube videos at 1.5x or 2x with a GitHub repo showing "what perfect looks like," going straight to the end and then back, then building and getting it wrong, and praises the CKA course for short sections followed immediately by hands-on work.

Bridget talks with Kent Rancourt, a senior engineer at Microsoft based in Connecticut who is also a dad, martial arts instructor, comic book nerd and Lego maniac, about Brigade and its upcoming v2. Brigade's tagline is event-driven scripting for Kubernetes, and the episode is billed as "not just Kubernetes." The cold open is Kent: "We weren't going to just be talking about Kubernetes, and yet we've said Kubernetes so many times."

Kent came from a startup that Microsoft acquired around 2017. That company held an annual offsite with a Shark Tank-style exercise, a hackathon where you only needed an idea and a low-fidelity proof of concept. Helm won the first year and Brigade won the second, and both came from comparing Kubernetes to an operating system: what features of a traditional OS don't exist yet in Kubernetes? Helm closed the gap of no package manager, and Brigade closed the gap of no scripting environment. Kent isn't aware of any alternate name for Brigade, though thinks Armada would have been more fitting given Kubernetes's nautical names.

Kent says the thing to emphasize is that Brigade is event-driven, unlike Kubernetes's declarative model of "do this and reconcile." Something happened, and now you handle that event. Kent compares it to AppleScript, where a gesture triggers a script. Kent calls it good for background work, "your minions," and says it's often mistaken for a CI/CD platform, which it isn't, though it does CI and CD well. The Brigade team uses it tied to GitHub, so opening a pull request triggers the build and tests and sends results back.

Kent says v1 was a minimal viable product and lightweight, with consequences: there was no API, and every event Brigade responded to was a Kubernetes secret, created either by a user at the command line or by a gateway, which bridges external systems like GitHub to Brigade, by talking directly to Kubernetes. A user therefore needed credentials to the cluster, and a cluster operator wouldn't hand those over unless the user was a competent Kubernetes user, a high barrier to entry. The aim of v2 was to abstract Kubernetes away from the end user, so Brigade went from "event-driven scripting platform for Kubernetes" to "event-driven scripting parentheses for Kubernetes." Kubernetes is now an implementation detail. There are no plans to support other orchestrators, but the architecture makes it possible, and Kent doesn't want to assume Kubernetes will be around forever.

Before starting v2 the team put a proposal of roughly 20 pages before the community explaining what was not optimal in v1 and couldn't be fixed without breaking changes. Kent says v2 is a complete rewrite, and the project is light on process compared with Kubernetes's KEPs or Helm's HIPs.

Kent says a fairly vibrant community going into v2 seems to have stepped back and let the maintainers handle the shift, which "gives me the sads." The project is in beta, and Kent says it's safe now for people to come back. It's much easier to build integrations, with a rich API and language bindings for Go, JavaScript and TypeScript, and a Rust SDK in the works, and Kent has new swag set aside for anyone who helps kick the tires or contributes an integration. V2 work happens in the v2 branch on GitHub, and there's a Brigade channel on the Kubernetes Slack.

There's no exact release date, but Kent says v2 is definitely coming in Q4 of that year. Feature development is pretty much done, and the team is avoiding breaking changes, as Kubernetes does once something is beta. Brigade is a CNCF sandbox project that would like to reach incubation, contingent on v2 going GA and more community building. Kent says most maintainers work at Microsoft, a few others aren't currently active, and the project wants to diversify its maintainers, including by employer, and adds that contributions needn't be code.

Kent says most repositories under the Brigade core GitHub organization have a .brigade folder with the project definitions and scripts used to build those projects with Brigade 2. The main repo is the plain Brigade one, and there are gateway repositories for GitHub, Bitbucket, Docker Hub, Azure Container Registry and CloudEvents, with Slack and Teams in progress. Most are simple, and Kent invites people to clone one and adapt it to another system. Kent asks people to star and watch the repositories.

Bridget chats with Kent Rancourt about Brigade, a tool for running scriptable, automated tasks (in Kubernetes).

Brigade art by Ronan Flynn-Curran

Matty talks with Emily Freeman, author of DevOps for Dummies, about how DevOps has changed and what comes next. Both had just started new jobs: Emily came from Microsoft's cloud advocacy team to AWS, working on DevOps strategy and messaging across the product line, and Matty left Red Hat for Pulumi, an infrastructure as code startup, with several former Chef colleagues, doing developer advocacy. Matty notes the show began at the end of 2013, so it has existed for most of the time DevOps has. The cold open is Emily on the explicit tag: "Have you fucking met me?"

Emily says the market is hungry for the next phase. Agile was about 20 years ago and DevOps 11 or 12. When Andrew Clay Shafer and Patrick Debois started it in 2009, teams were siloed with code thrown over a wall, and DevOps was a solution for bringing them together, which has mostly worked, though renaming the operations team a DevOps team "is not DevOps." Now the systems are distributed, the people are too, and the cloud adds managed services that remove pain but can cost visibility. Emily is thinking about how to capture telemetry and observability without getting into the weeds, and whether machine learning can help.

Matty says black boxes aren't inherently bad, and you need to know the promise is fulfilled and what to do if not. Matty compares it to Chef customers who asked how Chef does all 16 steps to build a server, when changing step 2 makes steps 3 through 10 unnecessary. Emily says systems are "a spaghetti of roots" like plant roots, so looking only at system A may miss the impact of B or C. Matty says you need to know things to the boundary at which you can influence change, like not knowing the power company rerouted your electricity.

Matty says DevOps over-rotated on culture because engineers will play with tools anyway, and that "not all silos are bad," as with literal grain silos, which exist for safety. The goal is cross-functionality, not NoOps. On shifting left, Matty says you push awareness and consideration left, not the work, and recalls the example of a colleague at a 1990s mail order company who kept being asked to teach everything about Photoshop in an afternoon. Emily agrees that being aware of someone's specialty, constraints and motivations is different from knowing how to do it, such as reading JavaScript without being a front-end engineer.

Emily adds that terms like "shift left" have gotten hand-wavy and people want to know how to apply them. Matty recalls 2014, when every devopsdays talk was about empathy because nobody knew what it was, and now people ask how to deal with people. Matty jokes that removing code from prod is "shipping left." Emily thinks DevOps is in late adoption, and Matty says the adoption phases overlap, noting that after a year working with state and local government many people still need the basics and others ask how to do it.

Matty has been thinking about cloud engineering as a discipline, which is what many people mean by SRE, although SRE is a specific way to implement it. Matty says a platform team is essentially an internal Heroku: a service with an interface, so feature teams don't need the nuts and bolts. Emily says it's a "service with an interface. That's it." Matty says a platform doesn't need bespoke Kubernetes, and can be cobbled together from other services. Matty also says the multi-cloud myth is the lowest common denominator unless you build a platform in front. Emily describes cloud engineering as being an infrastructure optimizer.

Emily asks how to stop valuing people by their place in the stack: front-end, back-end, platform, infrastructure. Some organizations favor devs, and Google in Emily's view over-indexes on SREs. Emily says much of spreading awareness is removing the fear of not being valuable, and that the empathy talked about for 12 years means "this shit is hard." Matty says it's a visibility problem: ops is like being a corporate lawyer, known only when something fails. Sales is the most visible part of an organization because everyone understands its value, and the further down the stack, the closer to utility, like a power company, the less visibility. Matty says this is why NoOps was a bad idea and likes the term operational excellence.

Matty recalls an early guest saying the value of DevOps is that it frees up the most important resource, the developers, which was the quiet part said out loud. Emily says massive companies saying they're obsessed with the developer is fine if inclusive, but about 90 percent of the operations community does not identify as a developer. Emily defaulted to engineer for the book and notes AWS uses builders, a term Emily hasn't heard outside that ecosystem. Matty: "Words are hard, which is that is what we're going to call this episode." Emily says you can appreciate someone's value without taking on their responsibility, like knowing how to change windshield wipers but not tires.

Bridget talks about multicluster service mesh, at a "201 level" after the earlier service mesh episode, with Phil Gibson, a PM at Microsoft focused on cloud-native security projects including Open Service Mesh, and Annie Wang, a PM intern on the team that summer, in the last week of the internship and studying computer science with a focus on cloud computing. The cold open is Bridget: "This all, I'm not gonna lie, sounds very complicated."

Phil says the most accepted description of a multicluster service mesh is being able to manage multiple Kubernetes services connected in east-west communication across clusters, and notes it means different things to different people. Annie explains that before multicluster, pods had to exit a cluster to reach services in another cluster, which is north-south traffic, and with east-west traffic, pods can reach the other cluster's ingress without having to leave. Phil adds two pieces: a distributed control plane, so you can still configure the mesh if one system goes down, and an ingress for the service mesh, separate from the traditional ingress controller that exposes services to the world, where routes are populated and synced through the control plane so service A in cluster 1 knows the route to service B in cluster 2. On security, Phil says a cluster can itself be a security boundary with its own RBAC and policy controls.

Annie lists the benefits: a mesh limited to one Kubernetes cluster is limited to that cluster's size, so multiple clusters let applications scale horizontally. Disaster recovery and failover mean you can deploy the same service to several clusters and divert traffic away from an unhealthy one, and you can get zero downtime during Kubernetes upgrades by updating one cluster at a time. Bridget asks about API deprecations across clusters on different versions. Phil says that's the hard part, that Kubernetes became popular by offloading labor-intensive configuration, and that the team wants to keep the experience simplified and shield users from the heavy lifting, which is still being worked out. Annie says the philosophy is to align with OSM: enable complex use cases while keeping things simple. Bridget notes complexity is conserved, and Phil says to expect YAML, ConfigMaps and new CRDs.

Annie says the internship was very overwhelming at first, with terms like Kubernetes, service mesh and even OSS new, since Kubernetes isn't taught in school. Annie is not a service mesh user, and talking to users about what they liked and disliked was valuable. Phil says the team gave the intern multicluster, the hardest thing they're dealing with. Annie says the most interesting part was learning how building features in open source differs from a regular product team, where you have a defined group of customers: you have to rally the community to figure out whether a problem is real, which is why Annie published a blog post to invite discussion.

Phil says the benefit of open source is fast feedback, since a project will live or die on the vine. Phil looks for gaps and pain points that customers mention repeatedly, and says "you got to have thick skin in the open source game." On the Service Mesh Interface spec, which has no code, Phil says people ask where to download it, but it's an agreed-upon specification of how APIs interact, and getting consensus from a large community is slow, but listening to the community is good hygiene.

Annie and Phil discovered that the need for multicluster is tied to maturity with Kubernetes. They assumed everyone wants it, from an enterprise mindset of a critical application, distributed service mesh, but early users had relatively simple applications. Phil says that if you've just refactored a VM application into a container and exposed it on port 80, multicluster is far away. Growth comes first by adding more apps to a cluster, and then the enterprise asks for regions and failover. Phil also explains mTLS with a shopping site: with TLS you trust the site, and with mutual TLS the site also knows who you are.

Annie says Kubernetes is new enough that school still uses VMs, and Annie will bring Kubernetes back to classmates but not multicluster service mesh. Annie says university teaches the fundamentals to learn the rest. Phil's career advice is to follow your passion, noting Phil left Microsoft for Docker after a container tutorial made a light bulb come on, and came back. Phil says contributing doesn't have to mean writing lots of code, and filing an issue asking why something doesn't work a certain way counts. Bridget points out Annie's one-word documentation pull request, a clarification that was valuable, and Phil likes the embrace of content as contribution. Phil's and Annie's advice for learning is to spin up every service mesh that supports multicluster and read their docs. Phil is on Twitter as @PhillipGibson, with a lot of barbecue brisket.

art credit: "Spiral" by roland - CC0 1.0

Matty talks with Johan Abildskov, a DevOps consultant who works with teams on how they interact as much as on pipelines and cloud, and who has written a book on Git. The episode starts from something Johan said beforehand: "If you invest enough into foundational practices, you can ignore them." The cold open is Johan: "I have read the Google SRE book so you don't have to."

Johan says that when you visit software teams, a lot of effort goes to things that should be boring: daily ceremonies, meetings, arguments over who failed a build and why the Git branching strategy is so complex. Because teams don't invest enough in core practices, those stay roadblocks to thinking at the level of abstraction they want. Johan's example is skill in an IDE, which never feels important enough to invest in, so Johan keeps paying a little and never becomes awesome at it. The idea came from the book The Art of Learning, recommended by the streamer Day9, about practicing fundamentals until they become muscle memory. Johan says we have an intuition for this in physical things but not in programming, and we lack the vocabulary for building automated responses that free the mind for creative work.

Matty says muscle memory comes from practicing with intent, and that you can practice in a bubble, like a Vim playground or Vim Adventures, but the skills must be used in real work. Matty has read about VS Code tricks and then gone back to old habits. Matty's approach, from bowling and public speaking, is to think about one thing at a time, such as making gestures big in a talk, and to do the same with a team, choosing one practice to be intentional about for a sprint, because foundational things are so broad that you feel you need to do all of it and so do none.

Johan says most software teams lack intentionality, discipline and explicitness, and that test-driven development forces doing things with intention. Matty says not all work is equal, since exploratory work like a spike is a different kind of work, and Johan says naming what mode you're in helps: with a spike, you're accountable to see what sticks.

Johan separates awareness from proficiency. Practicing a new IDE trick in a sandbox makes you aware, then applying it in context until it becomes muscle memory is "very Toyota Kata thinking." For other basics, such as fast and stable builds or a programming language so unfamiliar you can't read the screen, you need to practice outside your context so the smaller component parts stop mattering, like no longer pondering what float64 means. Matty compares it to thinking in a language instead of translating, and adds progressive disclosure: people with a new tool want to solve their specific problem right away, but first need the vocabulary, like the staging area in Git.

Johan brings up the zone of proximal development, meaning what you can do alone, with help, or not at all, and says experts forget what was difficult. Putting too much on a slide makes it hard for newcomers, who don't know what they can ignore, which is why teams argue endlessly about one repository versus many, GitFlow versus trunk-based development, or merge versus rebase, which Johan says isn't that important. To teach, you have to decompose things or hide details, even if not technically correct, so the learner gets the correct intuition. Matty suggests a simplified test in a pipeline demo, like checking 1 plus 1, to show that something tested a thing, and says it's fine if the educator corrects it soon and tells people it's a simplified view. Johan says being exhaustive does people a disservice, and taking responsibility to filter is like reading the SRE book so they don't have to.

Matty says the best conversations often go somewhere other than planned, as in the Tim Banks episode, which started as ops life and became gatekeeping, and compares it to open spaces: whatever conversation happens is the right one. Matty says you can't control your audience: early listeners of the show weren't the target audience. Matty recalls that people tell Matty things they got from talks that Matty didn't intend, and gives the example of the film Memento and the question of whether a character was ever a cop. Matty also says some meetings need structure, like a stand-up, but others should stay open.

Johan says being agile requires strong foundations, and suggests a stack of index cards as an agenda where each topic change adds a card on top and each completed topic removes one. Johan says that if important things go unaddressed they become urgent, and under pressure you fall back on what's the path of least resistance, so the right way has to be the muscle-memory one. Johan adds that people often believe unit tests will slow them down when they'd finish sooner by writing them, and there's a disconnect between how we believe we use time and how we spend it.

Matty, crediting John Allspaw, says the gap isn't only between management and practitioners: we also do it to ourselves, and closing it takes honesty and psychological safety, like logging food in MyFitnessPal without recording what you actually ate.

Johan has an antipathy for GitFlow, which has done good for the community, but in organizations it often achieves the opposite, as feature branches get huge and end in horrible merges, and people afraid of their workflow postpone. Johan says mono versus many repositories is a discussion people can feel, but the better question is which developer workflows you want to enable. Johan says you shouldn't care about Git or Kubernetes but about the platform built on top, and that if an IDE and a backend like GitHub can't handle your version control tasks, they are too complex. Johan's point is that "a developer doesn't want to do a push. A developer wants to move some code somewhere."

Matty adds commit-message formatting as another over-rotation, between the pedantic commit hook and "fix typo." Johan warns against elaborate gated workflows that don't match how work is done, and suggests documenting the process as it is first, then changing it organically, because friction kills productivity, motivation and trust.

Matty talks with Ashton Rodenhiser, who draws graphic recordings of talks at many conferences, including devopsdays events, and runs Mind's Eye Creative. Matty says that in the devopsdays Chicago budget, the line item has become simply Ashton. The cold open is what the devopsdays Toronto organizers said to a cold email: "We don't really know what you're talking about, but it sounds kind of cool."

Ashton calls the conference work graphic recording: listening as an outsider, processing what people say, and outputting it as drawings and text. Graphic facilitation uses the same skills in a facilitated session such as strategic planning or brainstorming, where the graphic captures the collective voice of the room and not a single speaker. Ashton describes being "a fly on the wall."

Ashton studied early childhood education, worked at a nonprofit family center, moved into family support work, and fell in love with facilitation, which has the thread of teaching, and "you don't actually have to know the information." A friend sent a one-day graphic facilitation course in 2013, which Ashton says was the perfect marriage of creativity and group process. Ashton applied the skills right away with a young adult group exploring pluralism, spent 2014 and 2015 playing with it on the side, and got a scholarship to a 2015 conference of the International Forum for Visual Practitioners in Austin, Texas. Ashton left inspired that others had figured out how to do it full-time, and officially started Mind's Eye Creative in August 2016, after what Ashton calls a dark launch.

Ashton says the work sells by being experienced, and got conferences by cold email. The fifth cold email went to the devopsdays Toronto organizers, who said they didn't know what it was but it sounded kind of cool. It was the first tech conference and the first job Ashton had to fly to, and Ashton watched DevOpsDays talks on the plane, worried about not understanding them. At that first event Ashton was in the green room watching a feed, so attendees didn't see it unfold, but in later years Ashton was out in front of people. Toronto was followed by Dallas and Columbus in 2017, and Chicago in 2018.

Ashton says not knowing the industry can be a strength, since it lets Ashton hear the larger picture without the baggage of the industry, and aims to draw what was said without interpreting too much. Having some context now helps with drawing things like Kubernetes and avoiding misspelled words, and Ashton challenges themself to draw time five or six different ways so as not to repeat.

Ashton likes talks with a human element and appreciates community and collaboration themes, and feels a responsibility when speakers share vulnerabilities to represent them in a way they'll feel okay about. Ashton recalls a DevSecCon talk by someone who helps people with autism find technology jobs.

Matty notes that a recording preserves a moment in time, while a facilitated meeting's output is fluid. Ashton says graphic facilitation can be used whenever a group, small or large, is having a conversation, and usually works with a separate facilitator, since it's hard to hold the visual role and make sure people feel safe and heard. Matty agrees with Ron Swanson that you shouldn't "half-ass 2 jobs, whole-ass one." Ashton says drawing in a session lets people see they're heard, telling of a group unhappy to be there that Ashton won over by hiding a little anchor in the drawing after overhearing a side joke. It's also a gentle way to tell people their point has been captured and the group can move on, and shows connections: in a session for a town's energy project, most of the conversation led back to one topic, which gave the town direction. Matty points out notes are linear and hide connections across pages, and Ashton says facilitation drawings are messier, but "we have to get through the draft."

Matty says the game Drawful has no erase button, and Ashton has practice with that. On paper, Ashton has been at it long enough to judge lettering and drawing size against a talk's time, roughly so much space per five minutes, though speakers may spend 20 minutes on the first of five points, so Ashton sends out "invisible questions" about where the speaker will go next. Ashton channels Bob Ross and happy little accidents, and carries white mailing labels in different sizes to cover mistakes, since they photograph invisibly and can be written over. If Ashton can't spell something, Ashton leaves a blank and comes back, and recalls misspelling "health" three times during a mental health event. Digital has made erasing possible over the past year.

Ashton says what stands out is how tight-knit the community is, and that Ashton hangs out on Twitter because that's where the community is. As an outsider to technology, Ashton noticed how fast things change: the first Kubernetes talk Ashton drew was in 2018 and it was a new idea, and now "if you don't use it, you're a loser." Ashton says the last year was all observability, SLOs, SLIs and SRE, and jokes that buzzword bingo would be a win.

Ashton's advice to people who say they're bad at drawing is to try, reframing drawing as "a thinking and an understanding and an idea tool," not something for a gallery, so it's fine if it looks horrible. The second tip is to turn plain white paper on its side, because our brains think spatially and not in top-down lines, and with nowhere obvious to start you choose where to put your first mark and make better connections. Ashton has a free 30-page ebook on the visual communication method, linked in the existing notes, and is on Twitter as Mind's Eye CCF, for Creative Consulting and Facilitation.

(episode art by Ashton Rodenhiser)

In this episode, Jess talks with guest Cat Swetel about her career, writings, and thoughts on DevOps.

Jess: "Cat is known internationally for her Penguin Power Stance and for standing on the edge of now!"

Jess and Cat talk about projects that fit Cat's definition of the edge of now.

Cat: "I do believe that DevOps is inherently feminist, because it puts the emphasis on that maintenance and reproductive work rather than producing working software."

Jess brings up Eric Evans' concept of software as gardening.

The panel discusses reproductive vs productive work. Cat talks about what she sees in healthy DevOps teams.

Jess and Cat talk about the challenges of working with legacy systems.

The panel discusses metacommunication and Gregory Bateson.

Cat: "In that situation, everyone is operating as specified but it's still not working! So that's when it becomes necessary to not have so much of that transactional interaction...It has to be something more generative."

Cat and Jess talk about the ethics of "care vs fair". Cat dives into how feminist theory has impacted her thinking on DevOps and systems.

Cat: "If we valued caring for the systems and caring for each other, rather than thinking fair or unfair...Let's check, are we caring for these systems, are we being mindful? Then rad, let's keep going."

Matty talks with Katy Farmer, a technical community manager at CircleCI with a background in engineering, a little development and a lot of people, about what it means to be part of a community and what tech asks of the people in one. Matty frames it from the member side: most listeners are probably members of communities, not builders of them. The cold open is Katy on ambassador programs: "I think I work here now, but for free."

Katy observes that tech companies make things that make developers' lives easier and then ask the community to build things and show what they can do. It's voluntary, but Katy wonders about it: Katy doesn't follow companies on personal Twitter, so why would someone follow the company's account, and what are they getting? Being part of a community is often "more gives than gets." Katy adds that community is an important part of the business model for many tech companies, recalling one with a big open source arm whose core functionality wouldn't work without community contributions, so the company benefits economically.

Matty mentions Mary Thengvall's rings of community: people who are aware, users who aren't customers, customers, and at the center advocates, which isn't a job title but people who do it on their own and where scale happens. Matty says the community isn't only people who contribute code: people write tutorials, do Twitch streams, help others and tell others, which is hard to measure. Katy calls community a branch of the business, where if it isn't a success it stalls marketing and then sales, and wonders what makes the unpaid work worthwhile.

Matty asks whether other industries have this, using Peloton and CrossFit as examples where networks enthusiasts build add value, but notes you can't go work on Ford's assembly line for 20 minutes. Katy says developers have the same skills the tools require, so they can build the thing the company builds, just for something else. Matty notes that what companies ask of communities is often the same thing people did all day at work, and says when someone isn't paid for it, "that's volunteer work," so call it what it is. Katy says the reward then isn't necessarily for the volunteer, like planting trees along the highway as a kid, and asks whether the reward is knowing you did something nobody else did.

Matty says there isn't one reason, since people participate for different reasons: identity, recognition (Matty is recognition-driven), the satisfaction of solving a problem, or feeling smarter than everyone else. Katy says that makes personas hard to build, and ends up with buckets like swag-driven and recognition-driven. Katy also says a community member still expects to be treated as a customer by a vendor they pay, but wouldn't expect a candy company to care about their opinion on banana flavor. Matty adds that people's personas change by community and time of year, and that the value is in being inclusive of different reasons for participating.

Matty says depending on community contributions is hard because you can't capacity plan or burn down. Matty's open source podcast theme gets spiky attention, with months of nothing followed by seven features in two weeks, and contributors who are active for a couple of months and then leave. Katy says people have seasons of community, and used to write essays on Digimon forums. On ambassador programs, Katy was once a random Hotmail ambassador at college and received money and a sweatshirt, but says programs often come with criteria and pressure, like a second job. There's a tipping point where "I think I work here now, but for free."

Matty says the first few advocates at a company get a lot of pressure, since they're asked for everything, and organizations should encourage people to continue doing what they already did and not make it a KPI, since DevRel teams tracking advocates in a CRM may panic when someone stops being a super fan for a moment. Matty's maxim: "you can't manufacture a moment," so nurture where it's happening. Katy says Docker Captains worked because the skill became valuable on a resume organically, which wasn't repeatable elsewhere. Matty adds that you can make the soil fertile, but what grows might be an artichoke and not the organic avocado you wanted, and tells of a community manager held to Ubuntu's forum activity for a far smaller paid SaaS product.

Matty says the value of a community isn't inherently the number of people in it, so month-over-month growth may eventually stop, and that's probably fine. Matty asks how to observe the value, avoiding a KPI. Katy says after a year you instinctively know how it's doing, by seeing every interaction across Twitter, the forum and Slack. For a community Katy manages, success is people asking lots of questions and getting interaction: "I measure community in curiosity." Katy is on Twitter as TheCaterTot and wants to hear from listeners.