Arrested DevOps

Matty and Marino Wijay went into this one without a plan, and the plan that emerged was the honest one: what the hell is actually going on in tech right now. Both have been around long enough to remember big shifts before (DevOps itself, the rise of Kubernetes and cloud native) but neither can remember change compounding this fast. Marino traces it to a specific hinge point: a predictable decade of innovation through the 2010s, then 2021 hits, zero interest rate money floods in, "you were limitless at that point," and a wave of startups and advocates get built on cheap capital. When rates rise, those ideas fizzle, the people promoting them lose their jobs, and the industry lurches toward the next hot thing. "Something called Loops was a huge thing two weeks ago, and now no one talks about it."

Marino draws on his data center days, back when a SAN install or rack expansion needed three distinct layers of expertise: contractors doing the physical racking and cabling, a team doing the finer cabling and bring-up, and a business layer figuring out what the infrastructure was actually for. "Those three layers, interestingly enough, have collapsed into two layers now." Physical labor gets automated away like it did in manufacturing, and what's left is a single human-plus-AI layer making the calls. The specialties that used to sit between those layers, especially networking, don't disappear so much as go invisible: "No one really talks about networking as much anymore, despite the fact that every single one of these models is running on hardware that's running on high-performance computing networking." The unsexy, boring infrastructure work is where the real money quietly lives now.

That collapse pushes everyone toward the same shortcut: "I need to just build, build, build with AI," and the result is that every AI-built product looks identical. Marino's line for it: open enough of these sites and "I'm looking at the same set of swatches," because the models are all referencing the same patterns. It's a preference problem as much as a technology one. Everyone wants the instant gratification of shipping a result, and nobody wants to be the one still doing the boring, careful thing while a competitor ships fast, so the shortcut becomes the default.

Marino's most unfiltered riff of the episode: a lot of what looks like innovation is actually VCs hedging bets across five companies solving the same non-problem, and a tax and accounting structure that rewards hiring tech people and building software regardless of whether anyone needed it. "Tech has been the largest accounting hack ever." Data centers get sited for cheap water and power with real cost to the communities around them, models get bigger because bigger is fundable, and somewhere in the OpenAI-NVIDIA-Oracle circularity, "we're all in this endless cycle of tax evasion, tax accounting, tax hacks." His read isn't purely cynical, though: underneath the noise there's real opportunity in the unglamorous infrastructure work nobody wants to do, if you're willing to look past the hype for it.

Matty pushes back with a story from his PagerDuty days, arguing with a co-founder convinced that manual, dedicated bug-fix teams were extinct: "Alex, you talk to people who know what PagerDuty is in the first place and decided they wanna talk to you." A week at a manufacturing trade show reinforced the point for both of them: plenty of shops, tech and non-tech alike, are quietly not participating in any of this, "we still are building shit, and we do it the way we do." Whatever conclusion you're drawing about AI adoption from your own timeline and network is a sample of one bubble, not the industry.

Closing out, Marino's advice for keeping up: lean on the people who've kept their credibility instead of trading it for sponsorship money, because those are the ones who'll actually tell you when something's overhyped or underhyped. He points to Keith Townsend as an example of someone pragmatic who gets his hands dirty before rendering a verdict. Matty ties it back to a conversation he had with Emily Freeman after his own layoff: the useful answer usually sits between "I will not touch a single thing that uses AI" and "the only way you should communicate is through the agent."

If you want more on how to actually keep learning in the middle of all this, Matty points back to three earlier episodes: Learning to Learn, Learning Stuff, and Managing Your Mental Stack.

Naga Sujitha Vummaneni and Sundeep Bobba co-authored CI/CD as a Control System, which takes Jez Humble and Dave Farley's Continuous Delivery, now pushing 20 years old, and reframes it through control theory. As Sujitha puts it: "The pipelines are activators, observability is the feedback signal, policy is the constraint, and deployment strategy is how you regulate the risk." Once you see a CI/CD pipeline that way, a lot of what look like tooling problems turn out to be system-behavior problems, and most pipelines today are open loop: "They measure everything and on nothing." Sujitha traces the pattern back to a moment from the Arrested DevOps episode with Hannah Foxwell and Robert Warner: enterprise clients insisting continuous delivery would never work at their company, until it was just how everyone shipped. The premise of the book is that this framing was always available, but AI agents moving at machine speed finally make it urgent.

Sundeep's answer to "should the agent handle this?" isn't a yes/no: it's a bounded operating envelope. Low-risk, reversible actions like restarting an unhealthy service or quarantining a known-bad artifact are fair game for automation. Changing security policy, touching production data, or anything with real customer blast radius pushes the threshold for human involvement way up. His framework is four questions: "How confident are we in the signal? What is the blast radius? Is the action reversible? And who owns the risk if the decision is wrong?" Sujitha adds the control-theory language underneath it: signal quality is itself a gate, because an automated rollback that fires on a noisy metric is worse than no automation at all, "you get flapping." Bounded blast radius, rate limits, and cooldowns exist so the system doesn't correct itself into a new outage.

Matty pushes on the gap between guardrails you write down and guardrails that actually run. Telling an agent to always scan for secrets before committing is no different than the developer who says "you're right, I should have run that" after skipping a check themselves, "it's the same trust-but-verify thing we've been doing forever, except now it's exacerbated." The book's answer is to stop treating this as purely a tooling problem: platform controls enforce the non-negotiables, prompts tell the agent what good behavior looks like, and runtime feedback tells you whether the controls are actually producing the outcome you expected. Sundeep frames it as fundamentally organizational: "Who owns the control, who can change it, what evidence proves it ran, and what happens when it fails, and who is allowed to accept an exception."

Sujitha's closing point reframes what a bypassed control actually means. "The bypass is a signal, not a violation. When engineers route around a control, the control was misdesigned. It might be in the wrong place, or too slow, or solving a problem they don't even have." She names the familiar shapes: the emergency-change process used for 40 percent of changes, the security scan everyone has a documented exception for, the staging environment nobody deploys to because it's never in a usable state, the approve button clicked 200 times a day without being read. Each one means leadership believes there's a control while the dashboard shows green and nobody actually knows what's happening behind it, control theory's version of losing observability of your own control layer.

For smaller, scrappier teams, both guests argue the model still applies, maybe more cleanly. Sujitha notes that small teams often close feedback loops faster because the control boundary and the team boundary line up. Sundeep's advice: "Start with one service or one delivery path and make the loop visible. Know what signals tell us the system is healthy, what decisions we're making from those signals, and what actions we can safely automate." Observability is foundational to all of it, since without actionable feedback there's nothing to close the loop with. The through-line for organizations of any size: "It's whether you have a closed loop of feedback, decisions, constraints, and action, rather than a pile of automation."

Long before "DevOps" had a name, manufacturing plants had already split into two tribes: the automation and manufacturing engineers who built up their own scrappy plant-floor IT, and the corporate IT department issuing laptops and running the domain a few buildings over. They grew up independently, with different tools and different goals, until, inevitably, they collided. IT wanted plant data flowing to corporate; OT wanted to reach out for data too. Add different incentive structures on top, and, as Matty points out, it's the same "wall of confusion" that gave rise to DevOps in the first place, just wearing safety glasses instead of a hoodie.

Doug's clearest example: a robot on the plant floor goes down because of a network issue, and the routers are owned by IT: a help desk routed through Bogotá with zero context on an explosives manufacturing line. The ticket crawls through triage while production sits at zero and the CEO starts calling. Doug's fix was to log into the router console himself and get it running again. "Everyone's like, 'Ooh, yeah, you're a hero.' Like, no, I'm not a hero. I broke the rules just to get it going." It's shadow IT, industrial-plant edition: the AWS-credit-card moment of manufacturing.

What actually helped, in Doug's experience, wasn't process; it was knowing people. "It's always gonna be personal relationships," he says, and his pitch to leadership has been simple: IT needs someone who's had OT experience, and OT needs a rep who understands IT's constraints, because the requirements on both sides have already converged. Neither side can keep pretending plant data stays on the plant floor; predictive maintenance and machine learning mean data has to leave the building, sent out to third parties for analysis, whether or not the old "peace treaty" between IT and OT accounted for it. That same shift is dragging OT into IT's security concerns for the first time, too. Doug is blunt that OT folks "just don't have a good concept of the security requirements," a gap he's only closed by talking directly to security engineers in IT.

Matty pulls in The Phoenix Project and its inspiration, The Goal, both built on the theory of constraints and value-stream mapping from actual manufacturing. The irony: software loves borrowing metaphors from manufacturing (Andon cords, the Toyota Production System) while the people doing real manufacturing haven't necessarily absorbed the lessons about visibility and bottlenecks that DevOps eventually learned from them. Doug agrees the CAMS/CALMS framework (culture, automation, measurement, learning or lean, sharing) maps cleanly onto OT/IT, especially the parts industrial teams already live: lean practices are baked into manufacturing, but the value an OT change delivers is far easier to point to (100 widgets became 110) than the value of infrastructure work that only shows up when it's absent.

The conversation lands on a favorite Matty theme: people work to the incentives you give them, not the ones you intended. A mandate for developers to write ten tests a sprint produces assert(true). A cash bonus for testers finding bugs and developers fixing them produces a black market in planted bugs. And "litigating severity" on an incident call is really about whoever's compensation is tied to P1 counts. Matty points to the STELLA Report as the same pattern in postmortems generally: after a well-handled incident, it's easy to reason "nothing bad happened, so you shouldn't have shut it down," the same Y2K cognitive dissonance in miniature. Doug's closing advice for anyone straddling the OT/IT divide is almost embarrassingly simple: find out the name of your IT person. Treat them like a teammate with real expertise, not a request queue ("not AWS," as Matty puts it) and the rest gets a lot easier.

Hannah Foxwell, who has spent over a decade in DevOps and platform engineering, draws a striking parallel to earlier transformations: "It used to be that testers didn't trust developers and ops didn't trust testers and there were all these silos. Now we're putting AI agents in the mix. Can we trust them? Should we trust them?"

This isn't just déjà vu—it's a fundamental challenge that resurfaces with every major shift in how we build software. As Robert Werner points out, management had to give up control and push trust to the edges of organizations during the agile transformation. With cloud adoption came self-service and automation. Now, with AI, we're dealing with non-deterministic black boxes that we need to trust to be "right often enough."

One of the biggest challenges isn't the technology itself—it's the lack of shared understanding. Hannah launched "AI for the Rest of Us," a community now with over 1,000 members, after realizing that AI fluency is essential for making good decisions about where and how to use these tools.

"I went to a talk at a conference thinking I'd learn about AI in one talk and become an expert by tomorrow," Hannah recalls. "It just didn't happen like that. There's a whole new domain with new vocabulary, new concepts, new techniques."

The community focuses on making AI accessible without dumbing it down—providing talks and content that explain complex concepts in simple language so more people can participate in the conversation about AI's role in software development.

The technology is evolving so rapidly that best practices barely have time to solidify before they're obsolete. Robert describes how hiring strategies at startups are changing every few weeks as new capabilities emerge. "Things that weren't feasible last week are suddenly possible," he notes.

But this speed creates a dangerous tension. Organizations are pushing hard for AI adoption while the guardrails, workflows, and cultural practices needed to use it safely are still being figured out. As Matty observes, this leads to perverse incentives—developers required to "use AI" who find ways to tick the box without actually deriving value, just like teams that once added meaningless tests to meet sprint requirements.

A critical question emerges: if AI generates the code, who owns it? Who's responsible when something goes wrong?

Hannah frames it in familiar DevOps terms: "Does anybody really want to own a service if they didn't write it and they don't understand how it works? It's the ops challenge again—AI throwing code over the wall to us."

Robert's answer is pragmatic and honest: humans will need to take responsibility for validating AI-generated code, even if it's tedious work most developers won't enjoy. His company, Leap, is building tools specifically to make that verification process as convenient and enjoyable as possible, because he believes there's simply no other way to do it safely.

There's an ironic twist in how AI agents work best: they need excellent documentation. Organizations improving their documentation to support AI-powered development are inadvertently following DevOps best practices that benefit human developers too.

But as Matty discovered building his own project, AI-generated documentation can be dangerously unreliable. The tools will confidently document features that don't exist, pulling from incomplete PRDs or speculative notes in the codebase. Great documentation trains better agents, but agents shouldn't write that documentation—creating a challenge that requires human judgment and oversight.

The parallels to earlier shifts are instructive. Hannah remembers enterprise clients who insisted continuous delivery would "never work here." Now it's common practice. The same resistance appeared with cloud adoption and agile methodologies.

What worked then still matters now:

For developers and teams trying to navigate this transformation, Hannah and Robert offer grounded guidance:

Keep your eyes open: Watch for patterns of success and failure. Who's making this work, and what do they have in common?

Build community: Find or create spaces where people can share honestly about what's working and what isn't, without the pressure to pretend everything's perfect.

Be selective about information sources: With so much noise and hype, focus on quality outlets. Ignore things for a few weeks, and if they keep coming up, that's when to invest your time.

Practice regularly: The technology evolves so fast that hands-on experience goes stale quickly. Even if it's not your main job, refresh your skills every few months.

Be specific and constrained: AI coding assistants work best with clear, narrow requests. Frustration comes from asking too much or being too vague.

We're in the Nokia phone stage of AI-assisted development, as Robert puts it—the technology will look completely different in just a few years. But unlike waiting passively for that future to arrive, developers and teams are actively creating it through the choices they make today about how to integrate these tools.

The question isn't whether AI will transform software development—it already is. The question is whether we'll learn from past transformations to build better practices, stronger safety nets, and more trustworthy systems. Or whether we'll repeat old mistakes at unprecedented speed.

As Hannah emphasizes, having more people with AI fluency means better conversations and better decisions at a pivotal moment in history. The rollercoaster is moving whether we're ready or not. The best approach is to keep your eyes open, stay connected to community, and remain thoughtfully critical about what works and what doesn't.

Learn more about Hannah's work at AI for the Rest of Us. Use code ADO20 for 20% off tickets to their London conference on October 15-16, 2025.

Matty talks with returning guest Kat Cosgrove, now head of developer advocacy at Minimus, about why security stays in the news, how to live with an endless stream of CVEs, and which tools belong in every production environment. Kat's employer builds container images with very few vulnerabilities, which comes up in the tooling section.

Kat's opening point: "Security is like a never not hot topic," and the stakes have risen as more of life and government runs on computers. The worms and Trojans of the 1990s look charming next to Meltdown, Spectre and Heartbleed, and a chain email that crashed your computer has been replaced by something that can take down much of the planet, as with the CrowdStrike outage, which Kat notes was a bug and not a vulnerability. Matty goes back to the Morris worm of the 1980s, when about 100 computers were on the internet and sysops hopped on conference calls because they were the only people affected. Today the whole thing is held together with "baling wire and good intentions."

Matty says that complexity makes it worse, since a vulnerable package may sit 17 layers below the thing you use, and brings up left-pad and the earlier Who Owns Your Availability episode. Kat uses the Equifax breach as the example: a wildly outdated version of Apache Struts, which anyone could imagine they'd never let happen, until it's three dependencies deep and nobody knows it's there. Kat has credit monitoring for life because of it. The breach is also how the industry got software bills of materials and a pile of security tooling.

Matty notes that an SBOM only helps after the fact, while something like Dependabot tells you about known vulnerabilities in what you already have, which is a bit more proactive. Both agree teams pull in far more dependencies than they need.

Kat says companies spend an outrageous amount of time and money mitigating CVEs, sometimes with whole teams, and still only handle the very high and maybe high ones. If the fix costs more than the potential exploit is worth, it stays: "Most CVEs just get left chilling there." So anyone who thinks their software is secure is probably wrong, and nothing is immune. Kat cites a Kubernetes Ingress NGINX vulnerability scored 9.7 or 9.8, which affected 38 percent of cloud environments, and says it was not fun from a maintainer's perspective.

Matty asks about the dependency of a dependency whose maintainer left five years ago. Kat's answer is to fork it and fix it: "It's open source, baby. PRs welcome," and if the fork is public you risk becoming the new person in the XKCD comic. Kat still thinks open source is the best and most secure way to build software, since flaws are found and fixed faster, though they're also exploited faster, and a community of dedicated nerds can outrun proprietary teams.

Kat's list for anyone who should pause the episode and install something starts with observability and monitoring, such as OpenTelemetry or Prometheus, because "you cannot just be like raw-dogging production": you need to notice weird traffic from a weird location hitting a weird endpoint. Next is alerting smart enough that engineers aren't firehosed with nonsense, then dependency management, meaning you know your dependencies and theirs. Kat adds infrastructure as code, instead of clicking through the AWS console, since most DevOps tooling has a security angle, and an SBOM, which Kat admits everyone is tired of hearing about after roughly three years of KubeCon talks but which still matters.

Kat's employer makes very small container images, most with zero CVEs, rebuilt automatically when an update lands, which gives an application a clean starting point, though it can't fix questionable code a team adds. Along the way Kat and Matty decide that raw-dogging alone wouldn't earn the explicit tag, and then Matty swears and it does.

Kat says security and computers are a burnout factory and begs listeners to have a hobby that doesn't involve computers: "literally go outside and touch grass," especially anyone on the "mitigating CVEs treadmill," a much less fun cousin of the loot treadmill in games like Diablo. Matty recalls a friend who took up piano and kept it off social media for the same reason, and admits that all of Matty's own hobbies used to be reframings of work. Kat floats a miniature rage room with racks at conferences, like the therapy dogs and baby goats that have shown up at events, and both complain about status LEDs that can't be turned off on printers and routers. Kat's router lives in the bedroom, under a t-shirt.

Matty talks with Kat Morgan about AI from the point of view of two people who use it daily and have mixed feelings about it: where it helps, where it's a mess, and how to work with it responsibly. Matty currently works in a marketing-adjacent role that involves some coding, at a company that makes Steampipe, and uses Cursor mostly for prototyping. The cold open is Kat, "very strongly opposed to abusing the robots."

Matty frames the question by saying people are bad at nuance and asking someone's opinion on AI is like asking what they think about computers, since some people picture generative art and others picture assistive tools or agents. Kat lists the ethical threads that deserve attention: intellectual property and whether creators can keep a roof over their heads, accessibility and whether LLMs open the digital world to people who need it, and the ecological and academic impacts. Kat compares it to security and documentation, which end up being everyone's responsibility, and expects everyone to need enough AI knowledge to tell where the tools stop and where humans have to start. Even abstaining means gaining that awareness. Matty adds that the less educated we are the more likely we are to be steamrolled, and that for some uses "the juice is not worth the squeeze," like burning resources on a cute cartoon of a friend.

Kat notes that any line drawn today can move tomorrow as the tools change, and that LLMs have changed how long a tech career seems feasible given a tendency toward carpal tunnel. Kat also points out that significant models already run on a MacBook, which Kat compares to the room-sized computer that became a pocket one.

Matty uses ChatGPT for alt text on social media, which makes doing it well faster. With Cursor, Matty's good experiences come when Matty already knows what to build. One was a private website for friends to watch old videos, with S3 and signed URLs, in a well-trodden React setup where Matty had the architecture in mind and Cursor implemented it. The bad one was asking Cursor to write a Steampipe plugin for Bluesky: "oh boy was that terrible." Matty concludes vibe coding has not made Matty worried that engineers will go away.

Kat says context is everything: about 30 to 50 percent of a context window goes to building context, including recent library versions, docs for new functions, and the project's structure and hygiene. Kat recently spent about 90 minutes on context and planning, with a task file where a markdown checkbox marks not started, in progress and done. When Kat told the agent to knock out the tasks, the work Kat expected to take four or five hours finished in a few prompts, leaving time to review the code line by line and check whether the first version was the user experience the team was after.

Both describe the rabbit hole: "just this function" becomes "just this entire feature." Matty ended up awake until 5:30 in the morning on a refactor of a podcast Hugo theme. Matty notes an upside, which is that an agent waits patiently with its context, so returning after two weeks costs nothing. Kat says burnout from layoffs and volatility has been real, and that Kat uses an LLM as an executive decision-making regulator, asking whether a tangent moves toward the milestone, which has helped with planning, estimating and sticking to a plan without difficult conversations with a manager.

Matty compares working with agents to a senior engineer working with a junior, where mistakes are expected and the process of plan, develop, test, iterate, document and commit exists to catch them. Kat adds that LLMs were trained on GitHub and respond well to work framed as GitHub issues: have the agent write an issue, edit it to the real requirements, then tell it to pull the issue and work it. That demands good code hygiene, with modular code, documented interfaces and an easy data model so a context window stays coherent for a one to two hour pairing session. Kat also uses the GitHub MCP server to have the agent comment on issues when the plan pivots and to pick up from the issue history on Monday mornings, and thinks a service account would help distinguish what a person wrote from what the agent wrote. Matty keeps issues in solo repos and has one with over a hundred comments that are all Matty talking to Matty.

Kat is uncomfortable with data centers powered by gas turbines and plans to run a local setup, and cites a DevOpsDays Chicago talk by Paul Czarkowski, which wasn't recorded, about running open models locally on something like a Mac Mini. Kat doesn't want secrets in a context window, so secrets go in a file in the home directory that git commands can't reach, and the agent runs in a dev container, never directly on the host. Matty notes that an env file in the agent's workspace is exactly what it can read. "It's important to try and make it safe to make mistakes," Kat says.

Matty's last caution is that an agent will troubleshoot things that aren't broken: it once tried to uninstall a plugin because it ran the wrong CLI command, and it will "fix" a function that was already fixed, since the context window is small. Kat says healthy skepticism is warranted because it messes up a lot and isn't replacing people.

Matty says you should always be polite to agents, partly as a joke about who they'll remember and partly because being polite to them makes us more inclined to be polite to everyone. Kat's version: neural networks are loosely aligned with how our brains work, and reinforcing demeaning behavior in a chat reinforces it in us, making it harder to tell the difference between treating computers that way and treating people that way. "We actually have to respect our own presence enough to appreciate that what we put out in the world will also change ourselves."

Matty talks with Andrew Zigler, developer advocate at Mattermost, about what it takes to run a developer community that is open first, and why it's harder than it sounds. Matty says neither of them knew the topic going in, then draws on time at PagerDuty, Chef and Pulumi. The cold open is Andrew: "being default open doesn't mean you have to sacrifice standards or sacrifice productivity or what you're aiming for as a company. It just means that you have to do the work to align people to it."

Andrew says the first hurdle in working default open is the echo chamber feeling: you post things you'd normally keep in closed channels, and either the same few people respond or nobody does, and you can't tell whether the message is landing or being read passively. DevRel also ends up as a counterbalance between what the community thinks of the project and what is happening inside the company. Building trust takes a long time, and the rule is to listen and incorporate the feedback you ask for, because otherwise you lose that trust.

Matty adds that the company behind a project always has more gravity, since the people paid to work on it attend every roadmap meeting, while volunteers and even people at another company's open source program office can only look in now and then. Andrew agrees that the loudest and most contributory voices are usually paid staff. Andrew's answer is to give other contributors gradients of engagement and a way to validate their experience and reward them, whether with swag, a platform, shared stage time, or simply being heard. People on the edges who never see contributors like themselves celebrated don't find a way in, so part of the job is teaching people to contribute where they are.

Matty compares Chef and Pulumi. At Chef, Matty says, "I work at Chef because I believe in Chef, not the other way around": employees were members of the community first, and community summits had staff and outsiders on equal footing. At Pulumi, when a community event was planned, it was hard to get employees to see that they were part of the community too and to join on a workday. Matty is careful to say it wasn't malicious, and that leadership has to reinforce it, since engineers with deliverables won't otherwise take the day. "No one plans for it to go bad," Andrew says, and adds that this is a universal experience in open source. Andrew suggests sharing the tools too, such as cloud workspaces that let outsiders build the software the way staff do, since internal build systems are where open projects quietly stop being open.

Matty raises the open roadmap problem: customers can't be named, and a feature may matter because one large customer wants it. Matty notes almost no company would give the community an equal say over the project it stewards. Andrew says "being default open doesn't mean that everything has to be open. It just means that you have to default to being open," and that it's fine to ask at the start whether something needs to be open. The key is to communicate clearly about what isn't, to keep trust. If community-built features make a product sticky in the enterprise, Andrew adds, some of what comes back should go to the community, for meetups, lightning talks and swag, "by sharing the spoils."

Matty asks how to open source advocacy itself, since people whose title is developer advocate can't do all of it. Matty describes a friend who asked how engineering blogs work, and the pattern of nothing for three months and then 12 posts in a day, since engineers write when they can. Matty's fix is to have people whose job is content work with subject matter experts. Andrew loves an engineering blog run by engineers, since the community wants to hear from the React developer about a new design library, not executive marketing, and says it needs pairing with a marketing team to tie back to strategy.

Andrew's core idea: "The biggest thing that you can do as a developer advocate is to create more developer advocates," meaning people without the title who have opportunities to engage. Open source communities are seasonal, since people contribute when they have free time, and "you can't control the waves, but you can control what the waves do when they wash up on the shore." Advocates should share strategy and plans, and write retrospectives about conferences, demos and talks, so contributors get excited and come along. Andrew has had contributors ask for feedback on their conference talks, which means trust has been earned and that advocacy created another advocate.

Andrew's example is GitLab's DevRel team, who plan events in the open on GitLab with epics and milestones and post retros with photos, talks and contributors. A contributor in Brussels can see the team is coming in three months and sign up on the issue. Planning for sponsorships, collateral and demos tends to happen invisibly on Slack, Discord or Asana, and fixing that invisibility matters.

Matty says a contributor writing program is "so much work" that contacts at other companies who ran one mostly wished they hadn't, since it multiplies content production problems. It doesn't have to be a formal program: signal boost a post about your product, or invite the author to collaborate. Matty also pleads with bloggers to put a way to reach them on their blogs, because Pulumi wanted to syndicate community posts with canonical tags and couldn't find the authors, and some authors turn out to have moved on from the tool years ago, though the post is still good. Matty's own most visited page is an old post about SharePoint 2010 search in a one-way trust, written to document a fix for a team.

Andrew ran a community writing program at Mattermost, which is rewarding for contributors, especially those in emerging tech hubs, and draining for staff, and can drift into a side project that doesn't connect back to company goals. Community members often assume the company won't notice their work, Andrew says, when in fact staff are right on top of anything that comes in and want to lift it up, and helping them get past that feeling builds confidence.

Jessica Kerr talks with Chelsea Troy, a staff data engineer on the machine learning operations team at Mozilla, about what staff engineering actually involves, how MLOps differs from DevOps, and when to use machine learning at all. The team exists to help Mozilla's other teams get models into production, and Mozilla has around 750 people. The promised topic of surfing doesn't come up in the recording. The cold open is Chelsea: "all jobs, if you do them long enough, become either management or marketing or some combination of management and marketing."

Chelsea started out planning to stay on the senior engineering track forever and write code for a whole career, and has since concluded that the skills early-career developers dismiss as soft become nearly the entire job: coordinating within and across teams, making sure everyone knows a product exists and how to use it. The system turns out to include people, regulations and corporate bureaucracy as well as the repositories. It's a lesson people seem to have to reach themselves, like advice to a friend in a bad relationship, and Jessica's version is that you don't break up with your technical skills, you open the relationship. Chelsea adds that you don't get to build things unless people want them, at least not if you want to avoid shelfware.

Chelsea says measuring engineers on productivity metrics inherited from an assembly line, where nails accumulate linearly through the day, is a disservice. Software is knowledge work: "We don't create value by doing the same thing over and over." It creates value by gathering context into an understanding of how to solve a problem now while keeping as many likely directions open, which Chelsea credits to Kent Beck's term optionality. From outside, that looks like nothing, nothing, nothing and then a big release, after months of understanding the problem, talking to users and socializing changes, since changes that torch people's context take power away from them. Writing the code comes last and is the easiest step. "We are treating lines of code like nails." Jessica adds that "our job is not what we do. It's what we know," and Chelsea says much of the day job is finding knowledge for someone and routing it to them, and that early in a career, how much code you can write depends on how good the decision makers about the repository are.

Mozilla's values include data privacy and ethical use of machine learning, which are hard to prioritize when teams spend their energy on getting any model into production through individual heroic efforts. In the fall a team began evaluating products to give data scientists and machine learning engineers a turnkey path to production. Chelsea notes the product landscape is young, with upstarts of around 15 employees, so the larger asks sometimes can't be met and documentation doesn't always match behavior.

Chelsea's advice on evaluating them is to separate optimizing metrics, where more is always better, from satisficing metrics with a good-enough threshold. Engineers tend to treat everything as an optimizing metric and end up in decision deadlock, for instance over scale, when a small startup isn't going to see a billion users at once. Jessica: "Problems you want to have." The optimizing metric that often isn't on the grid is the availability and flexibility of support. Chelsea fought to include it, and found that "the products we chose that have really, really responsive support teams are the products that we have managed to get into production at this point." A support engineer on Slack, for example at Weights and Biases, will take a custom chart problem and reproduce it in their own project. Free and open source tools the team liked ideologically didn't make the cut, since there's nobody whose job is making sure it works for the people using it, which Jessica sums up: "Wow, it's almost like the code isn't the whole thing." Jessica adds that at Honeycomb, good support gets renewals.

Chelsea came to MLOps from software engineering through data science, not through DevOps, and describes a data engineer as someone thrown into the model, the data science code or the app code as needed. Chelsea's view is that it isn't necessarily different in kind but that machine learning models add special considerations. Jessica compares deterministic program execution to the internal combustion engine, now one category among vehicle types. If the same input produces different output, that would worry a DevOps person and not necessarily an MLOps person, though there's a different decision tree, since some causes are fine and some are awful, and diagnosing a malfunctioning model is harder because the internals are automated.

Chelsea tells of someone distraught that ChatGPT claimed to have run code and reported wrong output, and of spending too long explaining the mechanism when the question was really "how could ChatGPT do this to me?" The person wanted to imagine a world where the tool had an interpreter, and Chelsea's answer was that one can imagine that world, but it's not the one we're in. A software engineering background alone isn't enough to debug these systems, and tools built to operationalize deterministic code lack needed features.

One such feature is checking that production data still matches the test data the model was evaluated on. Chelsea cites Andrew Ng's Machine Learning Yearning and a contrived example of a cat-photo model trained on high-quality images that then meets blurry phone photos. At Mozilla, Chelsea works on a system that sanitizes search data, discarding anything that might contain personal information. It drops anything with numerals unless it's on a human-made allow list, and uses spaCy's named entity recognizer to filter names. The team has to be sure the people using the feature are in the population the recognizer was trained and tested on, so it monitors an aggregate distribution of languages in search data, without storing the searches, to catch a shift before anything is stored long term.

Chelsea says MLOps and DevOps share a focus on catchability. There are three risk amplifiers: catastrophicness, likelihood, and insidiousness, "how likely is it that this thing goes uncaught if it happens." Security, DevOps and MLOps focus on insidiousness more than other roles do.

Chelsea's view is that the most effective systems are a series of steps with humans in the loop, not one generalized system doing everything. The example is who-to-follow recommendations on social media. A popularity-based model produces "the Beyoncé problem," which spirals up the already popular and doesn't connect niche audiences. A person can often tell when someone has gamed the engagement system, and human judgment is hard to automate and, Chelsea adds, hard to beat with a model, even though it's spotty at best. Better would be to break it into three simpler steps: find what topics people discuss, figure out who is knowledgeable on them, probably with a human in the loop, and recommend those people to people who want to learn. Chelsea notes that Follow Fridays and hashtags were user inventions, and that classical models on tabular data are often enough. Jessica's summary: for any problem where you'd reach for machine learning or generative AI, break it down into parts where a model helps, parts where a deterministic rule works, and low-volume, high-value parts where a real person should be asked.

Chelsea writes at chelseatroy.com, linked below.

Read more of Chelsea Troy's writing here!

Joe, Matty, Trevor, Bridget and Jessica record the tenth anniversary episode, which is also episode 200, as a decade-end wrap-up. This is a second take, which Joe opens with "Here we go, take 2. Slower, more intense," and Matty notes that "it's not a year-end wrap-up, it's a decade-end wrap-up." The hosts cover how the show started, how each host joined, conference and live-recording stories, guest and download stats, and what each is up to now. The recording ends with a supercut of the show's cold opens, which the episode's existing link also points to. The cold open is Joe's take-two line.

Matty tells the story of the show's first sponsor. PagerDuty reached out, and when Matty and Trevor didn't know what a sponsorship costs, they said $50 an episode, and Matty's one smart move was limiting it to a six-episode deal. Later Matty heard PagerDuty's ad on another podcast and learned at a conference bar that the other host had been paid $1,000 an episode for a similar audience. The renewal went to a number between the two. Years later, after Matty worked at PagerDuty, a sponsorship of the show drew lawyers who needed to be sure it wasn't double dipping. Matty adds that some sponsors come for a couple of episodes and some keep returning, and that the long back catalog helps.

Arrested DevOps began as a blog Matty never wrote, named by a friend who is a writer. Matty had learned DevOps from podcasts like the Ship Show, DevOps Cafe and Food Fight, and wanted a show for people who didn't know who John Allspaw was, the ones whose boss "read about DevOps in the in-flight magazine." Matty met Trevor at an Azure meetup in Chicago and invited Trevor to co-host, since Matty knew operations and wanted someone with a developer background. The two then got on a Hangout with Nathan Harvey and a fellow podcaster for advice, where the banana stand tagline was coined. Matty also has a lost episode 0, a five-minute recording testing Hangouts. Jessica says the "banana pants" variant came from a first episode where a script-reading panic and a lot of Dora the Explorer produced "in the banana pants."

The first episode with a cold open is 44. Joe picked them, and Joe's rule was: "If there's interesting profanity, that's the cold open." Matty also runs through the guest spiel: make peace with the explicit tag, stop and restart if you slip, and clap loudly if you blow an NDA, which has happened exactly once in ten years.

Bridget appeared first as a guest on an episode Matty remembers as being about conferences, whose art was a photo of Bridget's badges, published September 23, 2014, and Matty's checking turns up that the first episode Bridget hosted was November 18, 2014, DevOps in the Enterprise. They disagree about whether the invitation came at the DevOpsDays Chicago afterparty. Jessica had been on the show, liked it, and asked to join, after the two met at the first Redeploy conference, where Matty was a fan of Jessica's other podcast, Greater Than Code. Jeff Smith joined as an occasional host, and Joe, who started as the editor, also became a host, with a different editor before Joe. Since Joe went back to full-time work, Matty does the editing, which explains the quality, and the tail of each edit is worse because Matty got tired as an editor.

Bridget recalls curating a GOTO Chicago track as five live podcast panels in a row, with Matty, to an audience, in a breakout room with an air wall, and a guest loud enough that a proctor asked them to stop yelling. The first of them was Old Geeks Yell at Cloud with Andrew Clay Shafer and Bryan Cantrill. Bridget's takeaway: "If you're going to take on way too much, just have great guests. That'll cover a multitude of sins." Joe's tip for live episodes is to line up the AV crew ahead and pass a single mic, and says live episodes are easier to edit because fewer people talk over each other. The first live one may be eating sushi with Andrew Clay Shafer at DevOpsDays Minneapolis 2015, and Matty adds that the title was aspirational: the sushi took years to happen. Matty also notes that the DevOpsDays site doesn't handle a speaker called ADO well.

Bridget highlights Who Owns Your Availability from 2016, recorded the day of the left-pad incident, as still current for software supply chain discussions. Matty calls out 2023's three platform engineering episodes (Daniel Bryant, Pete Cheslock and Matt Kuritz) and says "I still think that Pete is right, that platform engineering really is DevOps with better marketing," with the earlier platforms episode with Kelsey Hightower and Andrew Clay Shafer always linked. Jessica picks the one-on-one conversation with an enterprise practitioner, since those stories show real companies struggling and let listeners know they're not alone.

Fan stories: a stranger at KubeCon Chicago told Bridget that a computer science course uses the show. Matty recalls an early fan letter that went unanswered on LinkedIn for years. Matty credits the show for much of a career, since "I owe a fair amount, if not all of my career, to this podcast," and notes how easy it is to be a guest on someone else's show. Matty also mentions using the same pun twice, in database episode titles in 2019 and 2023.

The most frequent guest is Andrew Clay Shafer with eight episodes, about 4 percent, then Nicole Forsgren and Sasha Rosenbaum with seven each. Matty says downloads stood at 1,911,233 plus about 35,000 on Spotify, since Spotify hosts its own copy and doesn't show in the regular stats, so two million is near. Episode 1 is most downloaded, then episode 92, CI/CD Oh My with Jez Humble. The most viewed YouTube video is Old Geeks Yell at Cloud at about 16,000 views, then Bryan Cantrill's fireside chat at about 8,000, and Matty jokes that the show should have Bryan on more often.

Matty also fixes episode data live and discovers guest entries were lost in migrations: the site started on Jekyll for about seven episodes, moved to WordPress with a custom plugin, then to Hugo. After Bridget found a Google alert for another podcast using the theme, Matty turned it into Castanet, a podcast theme that is now poorly maintained. Jessica's verdict: "our memory is terrible and our data is also crap."

Trevor is job hunting, looking at solution architecture or DevRel, and painting miniatures. Jessica is in a second year as engineering manager of developer relations at Honeycomb, is at a conference run by the Linux Foundation that added AI content to a Cassandra event, and would like the show to cover DevOps for LLM integrations. Bridget works in product at Microsoft, which means listening to other people's stories more than telling them. Joe travels for work to hotel ballrooms in Nashville and Cleveland. Matty leads developer relations and growth at Aiven, a Finnish data platform company, did not reach United's top tier this year, pays around $100 a day for dog care when traveling, and runs the team with a social contract about staying off Slack on holiday. Matty adds that because the company's field isn't Matty's expertise, leading the team works better than trying to do the work.

Bridget's closing idea, for anyone tired of telling the same stories: step back, help a colleague prep a talk, and "we can always make room to inspire someone else or to teach someone else or to go to a conference that we're not even speaking at."

Matty talks with Ben Greenberg, who had been head of DevRel at Fuel Labs for about three weeks at the time of recording and also runs a small consultancy, Yalla DevRel. The topic is what to do when you're suddenly in charge of a team, or starting any senior role. Ben had guested before, but that episode was recorded in May and shelved because its social networking content had aged, and Matty says this one is aimed at something timeless. The cold open is Matty on the classic interview question: "What's your biggest weakness, Ben? I work too hard."

Matty's warning for new senior hires: no matter how often people say they want a change agent, "here's the secret: they do not." Ben calls it the hero complex of hiring, the lowercase-m messiah, and describes doing something new in interviews: sharing a real, still-raw failure instead of a humblebrag, on the theory that if Fuel Labs still wanted Ben afterward, the fit would be real. Matty recalls the first manager job at Apartments.com, a role that was half manager and half sysadmin, where Matty arrived fired up about process and after about four weeks realized it was time to stop and listen. Matty's rule: "you have 2 ears and 1 mouth," and "every process, every way of doing things in a company is organizational scar tissue." The likelier explanation for something being done a certain way is that there's a reason, not that nobody has heard of Salesforce.

Ben contrasts the invading colonizer who uproots things with the person who sits with people, listens to their stories and helps shape direction from within, and admits that's hard when you were hired to do things.

Matty describes starting a role while knowing a new boss would arrive in three months. A coach suggested using the time for discovery so that the new boss could be handed what Matty had learned and a plan, without having made changes yet. Matty's onboarding technique: in every get-to-know-you meeting, mostly about the person and not the job, schedule a follow-up four to six weeks out, when there's enough context for a substantive conversation. Ben adds asking each person who else to meet, which widens the circle quickly, and meeting ecosystem partners and developers who build on the company's infrastructure. Matty recommends The First 90 Days, with the caveat that it reads like Harvard Business Review articles stapled together, and that it helps with figuring out what kind of organization you're in.

Matty also uses a model from an executive coach, Bill Joy, who taught engagement and skill as two axes: a new starter is high engagement and low skill, even a principal engineer or a new CEO. "You could be the new CEO. You are low skill." Activity for its own sake looks busy but doesn't win collaboration, and Ben says DevRel suffers from filling calendars without knowing what moved the needle.

Matty's standing advice is to learn how the company makes money. In DevRel interviews Matty's philosophy is that "if you don't have a way of demonstrating value, a way of demonstrating value will be assigned to you and you won't like it." Take care of the things tied to business outcomes and nobody will question the rest. Matty jokes about wanting a Nicole Forsgren for DevRel. Ben says the same applies to SDK engineers, technical writers and DevOps: do less, more completely and at a higher standard, instead of covering everything.

Ben inherited a team of three, dispersed across the Americas, each there about a year, and wants a team "that knows more than you." Ben's first career was as a director at nonprofits, and Ben's earlier stint as a young manager was marked by opinions, talking more than listening and buying into the change agent paradigm, which cost Ben. This time Ben has been listening and taking notes, then drafted a quarterly plan, shared it with Ben's own boss, then the team with an invitation to tear it apart, and scheduled an offsite to build Q1 objectives and key results together. Ben's method is to be a facilitator and consensus builder, and says "ownership is how you create a sense of culture of staying." A key goal is career paths for the team, since nobody had held the role for a while, recognizing what people did in the past year and getting them to where they should be.

Matty points to Lindsay Holmwood's post "It's Not a Promotion, It's a Career Change," and tells how a new CTO suggested moving Matty from director to infrastructure architect because technology, not managing people, seemed to be what got Matty out of bed. Matty's conditions were no pay cut and a letter making clear it wasn't a demotion, which Matty says illustrates the problem. Both believe individual contributors should be able to earn more than their managers, and Ben cites Aaron Bassett's talk that promotion shouldn't require becoming a manager. Matty says team lead and people manager are very different jobs: leading an open source project is not performance management, career coaching or time-off requests, and as Ben puts it, "you're never putting an open source contributor on a PIP."

Matty credits Apartments.com for heavy manager training, since many managers were first-timers from internal promotion, and Bill Joy's framing that the hardest transition is the first one, from individual contributor to manager, because you must stop doing what you're used to. Matty also says that leading a DevRel team in data, and not in a space like Kubernetes or infrastructure as code where Matty knows the material, has been easier to do humbly, since "I couldn't get a job on my own team," while still knowing how to do DevRel. Ben calls that humility a good topic on its own.