Arrested DevOps

Sylvain Kalache runs the AI lab at Rootly, and he starts by explaining the two kinds of memory behind the tools the industry has settled on calling AI SRE. The long-term one is a knowledge graph built from your code base, past incident reports, knowledge base, and Slack channels, the machine version of the instinct a veteran gets when something breaks and they think "I think I know where it is." The live one is whatever just happened: recent deploys, telemetry, traces. From the two, the tool forms hypotheses and starts investigating. Sylvain's claim is that the unglamorous part, collecting the data, is already "a huge time saver," since the tool can chase ten hypotheses at once and put what it finds on one dashboard.

Matty has been waiting for this for a while. Years ago he talked with Damon Edwards, Rundeck's founder, about having a robot run the same first-look checks while the responder is still getting out of bed. That never needed AI. What AI adds is the guessing: it doesn't have to be told which logs to pull for this particular shape of incident.

For toil-level incidents, Sylvain says the tools get close to a hundred percent accuracy on sev 3s and sev 2s. Past that it gets "jagged." His best example comes from an AI and reliability conference in Berlin, where an Anthropic reliability engineer described an outage in which a request carrying 22 images crashed the service. Claude didn't stop at the bug. It noticed the same request coming from 4,000 accounts, concluded this was an attack and not just an outage, and suggested contacting the safety team about abuse. A human patching the bug, Sylvain notes, would likely never have looked past it.

The same talk had the opposite case: the model grabbing a red herring, a cache hit rate graph that told a very persuasive story and turned out to have nothing to do with the problem. Matty points out that humans fall for this constantly, which is a large part of why incident commanders exist.

Matty admits his skin shivers a little when he says it, but he floats the question anyway. Everything you have to watch out for with an LLM responder is what a good incident commander already does for human ones: sitting there asking "is that what we need to do right now to get service restored?" So either side of the role might work with the right setup. What he finds dangerous is the person who says "it can do every damn thing" if you just write the right agent description.

He also brings the PagerDuty-era version of this argument. Back when it was machine learning and not LLMs, the pitch was to train on all your incident data so it knows what to do. He and a colleague there kept running into two problems. Organizations don't have enough data, and the cost of a wrong answer matters: a bad Netflix recommendation wastes an episode of your time, a bad restaurant recommendation costs you hundreds of dollars, and a model that decides something probably wasn't a sev 1 when it was has the same problem. Sylvain agrees, and says that among Rootly's customers using the AI SRE, "the human is always involved." Nobody he sees runs it blind.

The bigger risk, Sylvain argues, is what happens to the people. If the tool takes every easy incident, you lose the small ones responders use to practice, to understand how their systems work, and to build the instinct they'll need when the huge one arrives and the AI can't solve it. He points to aviation, where American Airlines pilots in the 1990s coined "children of magenta," after the magenta line on the screens they were watching. Autopilot tends to be engaged in exactly the situations it can't handle, so the industry's fix was required simulator training every year for emergencies a pilot may never see.

Matty piles on from his "Fight, Flight, or Freeze" talk: practicing in a calm moment is how procedures get into muscle memory for the 3am version. He quotes Rein Heinrichs, "If you're bad at having incidents, have you tried having more of them?" He also notes that better systems and bigger on-call rotations mean you can go months without a page, then wake up unable to remember how to log into the tool. Game days are the obvious answer, and the obvious problem has always been that someone has to build and inject a realistic failure. LLMs could do that part. Rootly has already gone one step further with Rootly Academy, built with Uptime Labs, where a human plays incident commander and directs a cast of LLM personas playing the responders.

Sylvain's prediction is that the industry will push this too far, get bitten, and pull back. Incidents will get shorter on average, but he expects bigger major incidents that run for hours or days, because the engineers have lost touch with their systems. He spent half a decade running a software engineering school on the bet that people learn by doing instead of listening, and its students got hired at Facebook, LinkedIn, Meta, Google, and Nvidia. Applied to incidents: "just watching it is not the same as doing it." There are two ways this goes, he says. Either the models get so good that the lost knowledge doesn't matter, or there's a reckoning. "I don't have an answer to this."

Matty adds two things. Product owners make excellent incident commanders partly because the experience makes them care about reliability, and that visceral exposure goes missing when the entire story is that Claude fixed it overnight. And if your coding agents don't get the context of the incidents that happened, you're skipping the same feedback loop you'd expect human developers to get from postmortems, which were already write-only before LLMs showed up.

Sylvain brings data. Looking at the average number of incidents per Rootly customer, 2026 is running at three times 2023. "So far it's not becoming easier, it's becoming harder." Responders are also getting less backup: the old move of tapping the developer who wrote the thing on the shoulder now gets you "Hey, like, sorry, I didn't write it. I just prompted my agent." His summary of the on-call job today is more incidents with less help, and the trend could go either way.

Matty ties it to the humane ops work from his PagerDuty days, when burnout and responder health were a bigger part of the conversation. One CTO or CIO he talked to answered concerns about on-call load with "that's why I pay him so much money." Sylvain's response, in full: "It's dumb."

Rootly's AI lab built a methodology to catch overworked responders, and it combines two kinds of data. The observed kind is how many incidents someone handled, whether they worked nights or off hours, how severe the incidents were, and, newly added, how many tokens they're using, since "some people are token maxing and actually getting addicted." The self-reported kind borrows from medicine: just ask "how do you feel?", the way an Apple Watch does, and track the trend. It isn't a diagnostic tool, Sylvain stresses, but it gives you a signal. They released it as an open source project called On-Call Health, and he says large organizations are already using it.

Matty goes digging mid-conversation and finds the ancestor: Etsy's Ops Weekly, from the days when Etsy was a monitoring company that also sold tea cozies. Its GitHub repo was archived three years ago and the last commit is nine or ten years old, but it came with a sleep tracker and a metric called MTTS, mean time to sleep. Matty says he can't describe how good it makes him feel to see the idea carried forward, and he lobbies Sylvain on the spot to come back for an entire episode on caring for the humans on call. Sylvain: "I'm down."

Matty also points back to a few earlier episodes on the same ground: Incident Retrospectives, Incidents and Accidents, Cognitive Neuroscience, and Let's Be Careful Out There.

Matty and Jacquie open with the caricature below, drawn at a vendor's caricature station at the first post-pandemic KubeCon. It's supposed to show the two of them fighting, since she was at Hashi and he was at Pulumi at the time. Neither of them works at those places now, so Matty figures they can be better friends.

Jacquie's definition of meaningful control has three parts: you understand the dependencies you've created and what they're tied to, you can verify that they meet your needs consistently, and you have a realistic way to change course. "How easily can we pivot if we need to, and how clear are we on what it does?" Her own first magic moment was configuration management, back when she was an intern and three people were SSHing into a few hundred VMs to run four to six commands on each, except sometimes it was seven. "I can just do this thing one place, and it happens everywhere."

Matty has the same origin story with a Windows vulnerability (maybe SQL Slammer, he doesn't remember): multiple days across three shifts in the data center, physically logged into servers over KVM, with a different login for each of something like 250 Windows NT domains. Then patch management tools showed up and you could just group machines and tell them to go. The trouble is what comes after the magic. The tool becomes load-bearing, then an albatross, because its opinions are now baked into how you work. It was great because the competition was Do Nothing Incorporated.

The new twist is speed. Matty describes starting the year by building a plugin and skills for his team, then watching people adopt it at wildly different rates, some still working up to opening ChatGPT. When the way of working is defined by the tool, and the tool changes every two weeks, "it all becomes yak shaves." He wonders how anyone preps a conference talk three months out when it will be stale by the time they give it. Jacquie says DevRel has the same problem: every release means being expected to be an expert on it, with demos, within 48 hours.

Her description of the process is a pipeline from discovery to something dependable: try it, break it, break it again, then ask whether it works three times in a row or once out of ten, and where the guardrails go. She also names the social side. Everyone's posting highlight reels, so "you're comparing your own friction points with everyone else's highlights," and there are too many launches to try them all the day they land. Python to Go to Rust felt fast at the time, she says, but "it objectively wasn't."

Matty's favorite counterexample is a Chef meetup in Austin, where a presenter explained how migrating from Chef to Ansible proved Ansible was better. Matty, who worked at Chef, sat there thinking: "No, you're just smarter now." He gets the same feeling when people insist a different model should do the code review so it isn't reviewing its own work. The analogy to humans breaks down, since Opus is not a person.

Matty points out this isn't purely an AI problem, since every layer of abstraction removes some understanding of what's underneath it. He remembers a Gartner session from the early days of cloud predicting that the sysadmin's job would become vendor manager. You don't know Apache's internals the way you used to either, because you run the module that encodes best practice. Now Claude Code can write a whole website in a JavaScript framework he doesn't understand, and he suspects the people who build reliable systems this way are the ones who already had first principles.

The counterargument is the photographers who never learned in a darkroom. Photoshop's metaphors came from that world, dodge and burn included, and a whole generation learned it another way anyway. The difference, Matty says, is that photography didn't move as fast as this.

Jacquie remembers a more vibrant meetup and conference scene a decade ago, when she could go to a DevOps one, a DevOps Enterprise one, and a Polyhack one, her favorite and now gone. They were places to scratch beneath the surface and hear what actually came out of a project, not a highlight reel. She misses that kind of under-the-hood writing too, citing the Netflix chaos engineering blog.

Matty thinks meetups drifted into mini conferences and mini webinars, and that funding now tends to mean vendors who want to talk about their product. His version of the original: the first Chef meetup in Chicago, seven or eight people at a bar, "nobody had slides, nobody had a mic." The Chef Community Summit was two to three days of open spaces and no presentations, and he doesn't think you could fund one today.

They also talk about the cost of being visible. Matty did a livestream at Pulumi where he did something dumb with Docker, someone politely pointed it out, and he spent the rest of the day feeling like he should have known. Jacquie has done upwards of 40 livestreams of learning in public and still runs into the same pull, because after 11 years people expect her to have answers. Her reasoning for doing it anyway: "If I can't do it, how do I expect anyone else to?" Matty adds the economy as a factor, since being on the record as imperfect feels riskier when jobs are scarce. Her line for the rest of it: "No one cares as much as you think they will."

When Matty asks when meaningful control clicked for her, Jacquie goes to a healthcare job in Canada. Data residency rules meant they used AWS but not S3, which she says is a global service that could replicate to any other region. Control meant answering where the data goes, who can access it, how it's retained or used, and "how do we know what we're using is what we think we're using?" Those questions led her to a lot of Kubernetes, to learning the CAP theorem, and eventually to building their own managed database services in Kubernetes. Later, some of the industry tools grew data residency features of their own, and the question became how to make it easy to pivot out of what they'd built.

The other half of control is who supports the tool. Jacquie points to the ingress-nginx retirement as the kind of thing happening more often: tools that were stable for years no longer being stable. She ties it to the economy, since open source has less funding and everyone has less time to give away, and the floodgates of PRs make communities harder to get involved in. She mentions Linus Torvalds gatekeeping which AI commits are allowed into Linux, and asks the same list of questions about every new tool: the maintainers, the documentation writers, the community knowledge base, the sponsors, the investors. "I don't have any answers here so much as it's just things that we look at."

Matty says the thing that makes all of this hard is that the boring parts, the plumbing, are what make the magic dependable, and there's no time to bake those practices in while the sand is shifting. AI-generated code is quick to make, but who maintains it? His internal marketing tool is the most heavily operationalized AI-built tool in his company, with ridiculous tests and CI and CD pipelines, only because he's been in ops for so long. The agent may not think of any of that, and neither will someone who doesn't live in that world.

Jacquie says Terraform is the apt comparison. Plans can run to 70,000 lines of changes, so her teams flagged every delete for close human eyes, and she thinks the open question is what the deletes are in the agent world. Matty's skills and Agent.md files are built around what could go wrong, which he thinks isn't the normal instinct. He also worries that the guardrails he writes are all Claude-specific, so they go away if the tool does. He wrote up the practical version in a post about a designer on his team who can ship to production without knowing what CD means.

His answer comes from the book Switch: make the right way the easy way. The story he always tells from it is the machine with a blade that kept cutting people's hands. Instead of signs and safety training, the plant redesigned it so you needed both hands to turn it on. If you have to actively opt into the safety, "it doesn't happen."

Jacquie thinks the questions are familiar from ten years ago: multi-cloud, disaster recovery, vendor lock-in, being able to take a backup and pivot. With proprietary models that can change pricing, push you out, or change how everything works, she expects a move toward open models, open weights you can adjust yourself, and small models trained to do one thing at a time.

Matty pushes back on the friction. People use the central services because they're easier, and he'd happily buy an appliance for this, in the spirit of buying a TiVo because "I just wanna record the Cubs game." Jacquie hadn't planned to bring it up (it's literally not in her notes), but she works at Nebius, where one of the products is Token Factory, a set of open models the company has done its own accuracy and training work on. You switch between them by changing the endpoint. She runs them through Hermes and OpenCode for things like video editing and making video games for fun, and her sign-off is the Nebius builder program, with credits, and an online hackathon with up to $50,000 in prizes that runs through the end of October. Matty promises to ship the episode before then.

The conversation ends with a long tangent on World of Warcraft. Jacquie was a competitive player who paid her rent doing it, and started in Wrath of the Lich King, when veterans complained about "all these Wrath babies." Matty uses it to make a point about DevOps: when the movement was three years old, being there for one year wasn't much of it, but when it's twenty years old and you missed only the first two, that's almost all of it. Jacquie jokes about a follow-up called "What playing video games competitively taught me about working in tech," and Matty suggests a joint episode with Kat Cosgrove, who didn't play competitively but sure plays a lot of damn video games.

Matty also points back to his recent episode with Marino Wijay, WTF Is Going On, for more on keeping up with all of this.

Matty and Marino Wijay went into this one without a plan, and the plan that emerged was the honest one: what the hell is actually going on in tech right now. Both have been around long enough to remember big shifts before (DevOps itself, the rise of Kubernetes and cloud native) but neither can remember change compounding this fast. Marino traces it to a specific hinge point: a predictable decade of innovation through the 2010s, then 2021 hits, zero interest rate money floods in, "you were limitless at that point," and a wave of startups and advocates get built on cheap capital. When rates rise, those ideas fizzle, the people promoting them lose their jobs, and the industry lurches toward the next hot thing. "Something called Loops was a huge thing two weeks ago, and now no one talks about it."

Marino draws on his data center days, back when a SAN install or rack expansion needed three distinct layers of expertise: contractors doing the physical racking and cabling, a team doing the finer cabling and bring-up, and a business layer figuring out what the infrastructure was actually for. "Those three layers, interestingly enough, have collapsed into two layers now." Physical labor gets automated away like it did in manufacturing, and what's left is a single human-plus-AI layer making the calls. The specialties that used to sit between those layers, especially networking, don't disappear so much as go invisible: "No one really talks about networking as much anymore, despite the fact that every single one of these models is running on hardware that's running on high-performance computing networking." The unsexy, boring infrastructure work is where the real money quietly lives now.

That collapse pushes everyone toward the same shortcut: "I need to just build, build, build with AI," and the result is that every AI-built product looks identical. Marino's line for it: open enough of these sites and "I'm looking at the same set of swatches," because the models are all referencing the same patterns. It's a preference problem as much as a technology one. Everyone wants the instant gratification of shipping a result, and nobody wants to be the one still doing the boring, careful thing while a competitor ships fast, so the shortcut becomes the default.

Marino's most unfiltered riff of the episode: a lot of what looks like innovation is actually VCs hedging bets across five companies solving the same non-problem, and a tax and accounting structure that rewards hiring tech people and building software regardless of whether anyone needed it. "Tech has been the largest accounting hack ever." Data centers get sited for cheap water and power with real cost to the communities around them, models get bigger because bigger is fundable, and somewhere in the OpenAI-NVIDIA-Oracle circularity, "we're all in this endless cycle of tax evasion, tax accounting, tax hacks." His read isn't purely cynical, though: underneath the noise there's real opportunity in the unglamorous infrastructure work nobody wants to do, if you're willing to look past the hype for it.

Matty pushes back with a story from his PagerDuty days, arguing with a co-founder convinced that manual, dedicated bug-fix teams were extinct: "Alex, you talk to people who know what PagerDuty is in the first place and decided they wanna talk to you." A week at a manufacturing trade show reinforced the point for both of them: plenty of shops, tech and non-tech alike, are quietly not participating in any of this, "we still are building shit, and we do it the way we do." Whatever conclusion you're drawing about AI adoption from your own timeline and network is a sample of one bubble, not the industry.

Closing out, Marino's advice for keeping up: lean on the people who've kept their credibility instead of trading it for sponsorship money, because those are the ones who'll actually tell you when something's overhyped or underhyped. He points to Keith Townsend as an example of someone pragmatic who gets his hands dirty before rendering a verdict. Matty ties it back to a conversation he had with Emily Freeman after his own layoff: the useful answer usually sits between "I will not touch a single thing that uses AI" and "the only way you should communicate is through the agent."

If you want more on how to actually keep learning in the middle of all this, Matty points back to three earlier episodes: Learning to Learn, Learning Stuff, and Managing Your Mental Stack.

Naga Sujitha Vummaneni and Sundeep Bobba co-authored CI/CD as a Control System, which takes Jez Humble and Dave Farley's Continuous Delivery, now pushing 20 years old, and reframes it through control theory. As Sujitha puts it: "The pipelines are activators, observability is the feedback signal, policy is the constraint, and deployment strategy is how you regulate the risk." Once you see a CI/CD pipeline that way, a lot of what look like tooling problems turn out to be system-behavior problems, and most pipelines today are open loop: "They measure everything and on nothing." Sujitha traces the pattern back to a moment from the Arrested DevOps episode with Hannah Foxwell and Robert Warner: enterprise clients insisting continuous delivery would never work at their company, until it was just how everyone shipped. The premise of the book is that this framing was always available, but AI agents moving at machine speed finally make it urgent.

Sundeep's answer to "should the agent handle this?" isn't a yes/no: it's a bounded operating envelope. Low-risk, reversible actions like restarting an unhealthy service or quarantining a known-bad artifact are fair game for automation. Changing security policy, touching production data, or anything with real customer blast radius pushes the threshold for human involvement way up. His framework is four questions: "How confident are we in the signal? What is the blast radius? Is the action reversible? And who owns the risk if the decision is wrong?" Sujitha adds the control-theory language underneath it: signal quality is itself a gate, because an automated rollback that fires on a noisy metric is worse than no automation at all, "you get flapping." Bounded blast radius, rate limits, and cooldowns exist so the system doesn't correct itself into a new outage.

Matty pushes on the gap between guardrails you write down and guardrails that actually run. Telling an agent to always scan for secrets before committing is no different than the developer who says "you're right, I should have run that" after skipping a check themselves, "it's the same trust-but-verify thing we've been doing forever, except now it's exacerbated." The book's answer is to stop treating this as purely a tooling problem: platform controls enforce the non-negotiables, prompts tell the agent what good behavior looks like, and runtime feedback tells you whether the controls are actually producing the outcome you expected. Sundeep frames it as fundamentally organizational: "Who owns the control, who can change it, what evidence proves it ran, and what happens when it fails, and who is allowed to accept an exception."

Sujitha's closing point reframes what a bypassed control actually means. "The bypass is a signal, not a violation. When engineers route around a control, the control was misdesigned. It might be in the wrong place, or too slow, or solving a problem they don't even have." She names the familiar shapes: the emergency-change process used for 40 percent of changes, the security scan everyone has a documented exception for, the staging environment nobody deploys to because it's never in a usable state, the approve button clicked 200 times a day without being read. Each one means leadership believes there's a control while the dashboard shows green and nobody actually knows what's happening behind it, control theory's version of losing observability of your own control layer.

For smaller, scrappier teams, both guests argue the model still applies, maybe more cleanly. Sujitha notes that small teams often close feedback loops faster because the control boundary and the team boundary line up. Sundeep's advice: "Start with one service or one delivery path and make the loop visible. Know what signals tell us the system is healthy, what decisions we're making from those signals, and what actions we can safely automate." Observability is foundational to all of it, since without actionable feedback there's nothing to close the loop with. The through-line for organizations of any size: "It's whether you have a closed loop of feedback, decisions, constraints, and action, rather than a pile of automation."

Long before "DevOps" had a name, manufacturing plants had already split into two tribes: the automation and manufacturing engineers who built up their own scrappy plant-floor IT, and the corporate IT department issuing laptops and running the domain a few buildings over. They grew up independently, with different tools and different goals, until, inevitably, they collided. IT wanted plant data flowing to corporate; OT wanted to reach out for data too. Add different incentive structures on top, and, as Matty points out, it's the same "wall of confusion" that gave rise to DevOps in the first place, just wearing safety glasses instead of a hoodie.

Doug's clearest example: a robot on the plant floor goes down because of a network issue, and the routers are owned by IT: a help desk routed through Bogotá with zero context on an explosives manufacturing line. The ticket crawls through triage while production sits at zero and the CEO starts calling. Doug's fix was to log into the router console himself and get it running again. "Everyone's like, 'Ooh, yeah, you're a hero.' Like, no, I'm not a hero. I broke the rules just to get it going." It's shadow IT, industrial-plant edition: the AWS-credit-card moment of manufacturing.

What actually helped, in Doug's experience, wasn't process; it was knowing people. "It's always gonna be personal relationships," he says, and his pitch to leadership has been simple: IT needs someone who's had OT experience, and OT needs a rep who understands IT's constraints, because the requirements on both sides have already converged. Neither side can keep pretending plant data stays on the plant floor; predictive maintenance and machine learning mean data has to leave the building, sent out to third parties for analysis, whether or not the old "peace treaty" between IT and OT accounted for it. That same shift is dragging OT into IT's security concerns for the first time, too. Doug is blunt that OT folks "just don't have a good concept of the security requirements," a gap he's only closed by talking directly to security engineers in IT.

Matty pulls in The Phoenix Project and its inspiration, The Goal, both built on the theory of constraints and value-stream mapping from actual manufacturing. The irony: software loves borrowing metaphors from manufacturing (Andon cords, the Toyota Production System) while the people doing real manufacturing haven't necessarily absorbed the lessons about visibility and bottlenecks that DevOps eventually learned from them. Doug agrees the CAMS/CALMS framework (culture, automation, measurement, learning or lean, sharing) maps cleanly onto OT/IT, especially the parts industrial teams already live: lean practices are baked into manufacturing, but the value an OT change delivers is far easier to point to (100 widgets became 110) than the value of infrastructure work that only shows up when it's absent.

The conversation lands on a favorite Matty theme: people work to the incentives you give them, not the ones you intended. A mandate for developers to write ten tests a sprint produces assert(true). A cash bonus for testers finding bugs and developers fixing them produces a black market in planted bugs. And "litigating severity" on an incident call is really about whoever's compensation is tied to P1 counts. Matty points to the STELLA Report as the same pattern in postmortems generally: after a well-handled incident, it's easy to reason "nothing bad happened, so you shouldn't have shut it down," the same Y2K cognitive dissonance in miniature. Doug's closing advice for anyone straddling the OT/IT divide is almost embarrassingly simple: find out the name of your IT person. Treat them like a teammate with real expertise, not a request queue ("not AWS," as Matty puts it) and the rest gets a lot easier.

Hannah Foxwell, who has spent over a decade in DevOps and platform engineering, draws a striking parallel to earlier transformations: "It used to be that testers didn't trust developers and ops didn't trust testers and there were all these silos. Now we're putting AI agents in the mix. Can we trust them? Should we trust them?"

This isn't just déjà vu—it's a fundamental challenge that resurfaces with every major shift in how we build software. As Robert Werner points out, management had to give up control and push trust to the edges of organizations during the agile transformation. With cloud adoption came self-service and automation. Now, with AI, we're dealing with non-deterministic black boxes that we need to trust to be "right often enough."

One of the biggest challenges isn't the technology itself—it's the lack of shared understanding. Hannah launched "AI for the Rest of Us," a community now with over 1,000 members, after realizing that AI fluency is essential for making good decisions about where and how to use these tools.

"I went to a talk at a conference thinking I'd learn about AI in one talk and become an expert by tomorrow," Hannah recalls. "It just didn't happen like that. There's a whole new domain with new vocabulary, new concepts, new techniques."

The community focuses on making AI accessible without dumbing it down—providing talks and content that explain complex concepts in simple language so more people can participate in the conversation about AI's role in software development.

The technology is evolving so rapidly that best practices barely have time to solidify before they're obsolete. Robert describes how hiring strategies at startups are changing every few weeks as new capabilities emerge. "Things that weren't feasible last week are suddenly possible," he notes.

But this speed creates a dangerous tension. Organizations are pushing hard for AI adoption while the guardrails, workflows, and cultural practices needed to use it safely are still being figured out. As Matty observes, this leads to perverse incentives—developers required to "use AI" who find ways to tick the box without actually deriving value, just like teams that once added meaningless tests to meet sprint requirements.

A critical question emerges: if AI generates the code, who owns it? Who's responsible when something goes wrong?

Hannah frames it in familiar DevOps terms: "Does anybody really want to own a service if they didn't write it and they don't understand how it works? It's the ops challenge again—AI throwing code over the wall to us."

Robert's answer is pragmatic and honest: humans will need to take responsibility for validating AI-generated code, even if it's tedious work most developers won't enjoy. His company, Leap, is building tools specifically to make that verification process as convenient and enjoyable as possible, because he believes there's simply no other way to do it safely.

There's an ironic twist in how AI agents work best: they need excellent documentation. Organizations improving their documentation to support AI-powered development are inadvertently following DevOps best practices that benefit human developers too.

But as Matty discovered building his own project, AI-generated documentation can be dangerously unreliable. The tools will confidently document features that don't exist, pulling from incomplete PRDs or speculative notes in the codebase. Great documentation trains better agents, but agents shouldn't write that documentation—creating a challenge that requires human judgment and oversight.

The parallels to earlier shifts are instructive. Hannah remembers enterprise clients who insisted continuous delivery would "never work here." Now it's common practice. The same resistance appeared with cloud adoption and agile methodologies.

What worked then still matters now:

For developers and teams trying to navigate this transformation, Hannah and Robert offer grounded guidance:

Keep your eyes open: Watch for patterns of success and failure. Who's making this work, and what do they have in common?

Build community: Find or create spaces where people can share honestly about what's working and what isn't, without the pressure to pretend everything's perfect.

Be selective about information sources: With so much noise and hype, focus on quality outlets. Ignore things for a few weeks, and if they keep coming up, that's when to invest your time.

Practice regularly: The technology evolves so fast that hands-on experience goes stale quickly. Even if it's not your main job, refresh your skills every few months.

Be specific and constrained: AI coding assistants work best with clear, narrow requests. Frustration comes from asking too much or being too vague.

We're in the Nokia phone stage of AI-assisted development, as Robert puts it—the technology will look completely different in just a few years. But unlike waiting passively for that future to arrive, developers and teams are actively creating it through the choices they make today about how to integrate these tools.

The question isn't whether AI will transform software development—it already is. The question is whether we'll learn from past transformations to build better practices, stronger safety nets, and more trustworthy systems. Or whether we'll repeat old mistakes at unprecedented speed.

As Hannah emphasizes, having more people with AI fluency means better conversations and better decisions at a pivotal moment in history. The rollercoaster is moving whether we're ready or not. The best approach is to keep your eyes open, stay connected to community, and remain thoughtfully critical about what works and what doesn't.

Learn more about Hannah's work at AI for the Rest of Us. Use code ADO20 for 20% off tickets to their London conference on October 15-16, 2025.

Matty talks with returning guest Kat Cosgrove, now head of developer advocacy at Minimus, about why security stays in the news, how to live with an endless stream of CVEs, and which tools belong in every production environment. Kat's employer builds container images with very few vulnerabilities, which comes up in the tooling section.

Kat's opening point: "Security is like a never not hot topic," and the stakes have risen as more of life and government runs on computers. The worms and Trojans of the 1990s look charming next to Meltdown, Spectre and Heartbleed, and a chain email that crashed your computer has been replaced by something that can take down much of the planet, as with the CrowdStrike outage, which Kat notes was a bug and not a vulnerability. Matty goes back to the Morris worm of the 1980s, when about 100 computers were on the internet and sysops hopped on conference calls because they were the only people affected. Today the whole thing is held together with "baling wire and good intentions."

Matty says that complexity makes it worse, since a vulnerable package may sit 17 layers below the thing you use, and brings up left-pad and the earlier Who Owns Your Availability episode. Kat uses the Equifax breach as the example: a wildly outdated version of Apache Struts, which anyone could imagine they'd never let happen, until it's three dependencies deep and nobody knows it's there. Kat has credit monitoring for life because of it. The breach is also how the industry got software bills of materials and a pile of security tooling.

Matty notes that an SBOM only helps after the fact, while something like Dependabot tells you about known vulnerabilities in what you already have, which is a bit more proactive. Both agree teams pull in far more dependencies than they need.

Kat says companies spend an outrageous amount of time and money mitigating CVEs, sometimes with whole teams, and still only handle the very high and maybe high ones. If the fix costs more than the potential exploit is worth, it stays: "Most CVEs just get left chilling there." So anyone who thinks their software is secure is probably wrong, and nothing is immune. Kat cites a Kubernetes Ingress NGINX vulnerability scored 9.7 or 9.8, which affected 38 percent of cloud environments, and says it was not fun from a maintainer's perspective.

Matty asks about the dependency of a dependency whose maintainer left five years ago. Kat's answer is to fork it and fix it: "It's open source, baby. PRs welcome," and if the fork is public you risk becoming the new person in the XKCD comic. Kat still thinks open source is the best and most secure way to build software, since flaws are found and fixed faster, though they're also exploited faster, and a community of dedicated nerds can outrun proprietary teams.

Kat's list for anyone who should pause the episode and install something starts with observability and monitoring, such as OpenTelemetry or Prometheus, because "you cannot just be like raw-dogging production": you need to notice weird traffic from a weird location hitting a weird endpoint. Next is alerting smart enough that engineers aren't firehosed with nonsense, then dependency management, meaning you know your dependencies and theirs. Kat adds infrastructure as code, instead of clicking through the AWS console, since most DevOps tooling has a security angle, and an SBOM, which Kat admits everyone is tired of hearing about after roughly three years of KubeCon talks but which still matters.

Kat's employer makes very small container images, most with zero CVEs, rebuilt automatically when an update lands, which gives an application a clean starting point, though it can't fix questionable code a team adds. Along the way Kat and Matty decide that raw-dogging alone wouldn't earn the explicit tag, and then Matty swears and it does.

Kat says security and computers are a burnout factory and begs listeners to have a hobby that doesn't involve computers: "literally go outside and touch grass," especially anyone on the "mitigating CVEs treadmill," a much less fun cousin of the loot treadmill in games like Diablo. Matty recalls a friend who took up piano and kept it off social media for the same reason, and admits that all of Matty's own hobbies used to be reframings of work. Kat floats a miniature rage room with racks at conferences, like the therapy dogs and baby goats that have shown up at events, and both complain about status LEDs that can't be turned off on printers and routers. Kat's router lives in the bedroom, under a t-shirt.

Matty talks with Kat Morgan about AI from the point of view of two people who use it daily and have mixed feelings about it: where it helps, where it's a mess, and how to work with it responsibly. Matty currently works in a marketing-adjacent role that involves some coding, at a company that makes Steampipe, and uses Cursor mostly for prototyping. The cold open is Kat, "very strongly opposed to abusing the robots."

Matty frames the question by saying people are bad at nuance and asking someone's opinion on AI is like asking what they think about computers, since some people picture generative art and others picture assistive tools or agents. Kat lists the ethical threads that deserve attention: intellectual property and whether creators can keep a roof over their heads, accessibility and whether LLMs open the digital world to people who need it, and the ecological and academic impacts. Kat compares it to security and documentation, which end up being everyone's responsibility, and expects everyone to need enough AI knowledge to tell where the tools stop and where humans have to start. Even abstaining means gaining that awareness. Matty adds that the less educated we are the more likely we are to be steamrolled, and that for some uses "the juice is not worth the squeeze," like burning resources on a cute cartoon of a friend.

Kat notes that any line drawn today can move tomorrow as the tools change, and that LLMs have changed how long a tech career seems feasible given a tendency toward carpal tunnel. Kat also points out that significant models already run on a MacBook, which Kat compares to the room-sized computer that became a pocket one.

Matty uses ChatGPT for alt text on social media, which makes doing it well faster. With Cursor, Matty's good experiences come when Matty already knows what to build. One was a private website for friends to watch old videos, with S3 and signed URLs, in a well-trodden React setup where Matty had the architecture in mind and Cursor implemented it. The bad one was asking Cursor to write a Steampipe plugin for Bluesky: "oh boy was that terrible." Matty concludes vibe coding has not made Matty worried that engineers will go away.

Kat says context is everything: about 30 to 50 percent of a context window goes to building context, including recent library versions, docs for new functions, and the project's structure and hygiene. Kat recently spent about 90 minutes on context and planning, with a task file where a markdown checkbox marks not started, in progress and done. When Kat told the agent to knock out the tasks, the work Kat expected to take four or five hours finished in a few prompts, leaving time to review the code line by line and check whether the first version was the user experience the team was after.

Both describe the rabbit hole: "just this function" becomes "just this entire feature." Matty ended up awake until 5:30 in the morning on a refactor of a podcast Hugo theme. Matty notes an upside, which is that an agent waits patiently with its context, so returning after two weeks costs nothing. Kat says burnout from layoffs and volatility has been real, and that Kat uses an LLM as an executive decision-making regulator, asking whether a tangent moves toward the milestone, which has helped with planning, estimating and sticking to a plan without difficult conversations with a manager.

Matty compares working with agents to a senior engineer working with a junior, where mistakes are expected and the process of plan, develop, test, iterate, document and commit exists to catch them. Kat adds that LLMs were trained on GitHub and respond well to work framed as GitHub issues: have the agent write an issue, edit it to the real requirements, then tell it to pull the issue and work it. That demands good code hygiene, with modular code, documented interfaces and an easy data model so a context window stays coherent for a one to two hour pairing session. Kat also uses the GitHub MCP server to have the agent comment on issues when the plan pivots and to pick up from the issue history on Monday mornings, and thinks a service account would help distinguish what a person wrote from what the agent wrote. Matty keeps issues in solo repos and has one with over a hundred comments that are all Matty talking to Matty.

Kat is uncomfortable with data centers powered by gas turbines and plans to run a local setup, and cites a DevOpsDays Chicago talk by Paul Czarkowski, which wasn't recorded, about running open models locally on something like a Mac Mini. Kat doesn't want secrets in a context window, so secrets go in a file in the home directory that git commands can't reach, and the agent runs in a dev container, never directly on the host. Matty notes that an env file in the agent's workspace is exactly what it can read. "It's important to try and make it safe to make mistakes," Kat says.

Matty's last caution is that an agent will troubleshoot things that aren't broken: it once tried to uninstall a plugin because it ran the wrong CLI command, and it will "fix" a function that was already fixed, since the context window is small. Kat says healthy skepticism is warranted because it messes up a lot and isn't replacing people.

Matty says you should always be polite to agents, partly as a joke about who they'll remember and partly because being polite to them makes us more inclined to be polite to everyone. Kat's version: neural networks are loosely aligned with how our brains work, and reinforcing demeaning behavior in a chat reinforces it in us, making it harder to tell the difference between treating computers that way and treating people that way. "We actually have to respect our own presence enough to appreciate that what we put out in the world will also change ourselves."

Matty talks with Andrew Zigler, developer advocate at Mattermost, about what it takes to run a developer community that is open first, and why it's harder than it sounds. Matty says neither of them knew the topic going in, then draws on time at PagerDuty, Chef and Pulumi. The cold open is Andrew: "being default open doesn't mean you have to sacrifice standards or sacrifice productivity or what you're aiming for as a company. It just means that you have to do the work to align people to it."

Andrew says the first hurdle in working default open is the echo chamber feeling: you post things you'd normally keep in closed channels, and either the same few people respond or nobody does, and you can't tell whether the message is landing or being read passively. DevRel also ends up as a counterbalance between what the community thinks of the project and what is happening inside the company. Building trust takes a long time, and the rule is to listen and incorporate the feedback you ask for, because otherwise you lose that trust.

Matty adds that the company behind a project always has more gravity, since the people paid to work on it attend every roadmap meeting, while volunteers and even people at another company's open source program office can only look in now and then. Andrew agrees that the loudest and most contributory voices are usually paid staff. Andrew's answer is to give other contributors gradients of engagement and a way to validate their experience and reward them, whether with swag, a platform, shared stage time, or simply being heard. People on the edges who never see contributors like themselves celebrated don't find a way in, so part of the job is teaching people to contribute where they are.

Matty compares Chef and Pulumi. At Chef, Matty says, "I work at Chef because I believe in Chef, not the other way around": employees were members of the community first, and community summits had staff and outsiders on equal footing. At Pulumi, when a community event was planned, it was hard to get employees to see that they were part of the community too and to join on a workday. Matty is careful to say it wasn't malicious, and that leadership has to reinforce it, since engineers with deliverables won't otherwise take the day. "No one plans for it to go bad," Andrew says, and adds that this is a universal experience in open source. Andrew suggests sharing the tools too, such as cloud workspaces that let outsiders build the software the way staff do, since internal build systems are where open projects quietly stop being open.

Matty raises the open roadmap problem: customers can't be named, and a feature may matter because one large customer wants it. Matty notes almost no company would give the community an equal say over the project it stewards. Andrew says "being default open doesn't mean that everything has to be open. It just means that you have to default to being open," and that it's fine to ask at the start whether something needs to be open. The key is to communicate clearly about what isn't, to keep trust. If community-built features make a product sticky in the enterprise, Andrew adds, some of what comes back should go to the community, for meetups, lightning talks and swag, "by sharing the spoils."

Matty asks how to open source advocacy itself, since people whose title is developer advocate can't do all of it. Matty describes a friend who asked how engineering blogs work, and the pattern of nothing for three months and then 12 posts in a day, since engineers write when they can. Matty's fix is to have people whose job is content work with subject matter experts. Andrew loves an engineering blog run by engineers, since the community wants to hear from the React developer about a new design library, not executive marketing, and says it needs pairing with a marketing team to tie back to strategy.

Andrew's core idea: "The biggest thing that you can do as a developer advocate is to create more developer advocates," meaning people without the title who have opportunities to engage. Open source communities are seasonal, since people contribute when they have free time, and "you can't control the waves, but you can control what the waves do when they wash up on the shore." Advocates should share strategy and plans, and write retrospectives about conferences, demos and talks, so contributors get excited and come along. Andrew has had contributors ask for feedback on their conference talks, which means trust has been earned and that advocacy created another advocate.

Andrew's example is GitLab's DevRel team, who plan events in the open on GitLab with epics and milestones and post retros with photos, talks and contributors. A contributor in Brussels can see the team is coming in three months and sign up on the issue. Planning for sponsorships, collateral and demos tends to happen invisibly on Slack, Discord or Asana, and fixing that invisibility matters.

Matty says a contributor writing program is "so much work" that contacts at other companies who ran one mostly wished they hadn't, since it multiplies content production problems. It doesn't have to be a formal program: signal boost a post about your product, or invite the author to collaborate. Matty also pleads with bloggers to put a way to reach them on their blogs, because Pulumi wanted to syndicate community posts with canonical tags and couldn't find the authors, and some authors turn out to have moved on from the tool years ago, though the post is still good. Matty's own most visited page is an old post about SharePoint 2010 search in a one-way trust, written to document a fix for a team.

Andrew ran a community writing program at Mattermost, which is rewarding for contributors, especially those in emerging tech hubs, and draining for staff, and can drift into a side project that doesn't connect back to company goals. Community members often assume the company won't notice their work, Andrew says, when in fact staff are right on top of anything that comes in and want to lift it up, and helping them get past that feeling builds confidence.

Jessica Kerr talks with Chelsea Troy, a staff data engineer on the machine learning operations team at Mozilla, about what staff engineering actually involves, how MLOps differs from DevOps, and when to use machine learning at all. The team exists to help Mozilla's other teams get models into production, and Mozilla has around 750 people. The promised topic of surfing doesn't come up in the recording. The cold open is Chelsea: "all jobs, if you do them long enough, become either management or marketing or some combination of management and marketing."

Chelsea started out planning to stay on the senior engineering track forever and write code for a whole career, and has since concluded that the skills early-career developers dismiss as soft become nearly the entire job: coordinating within and across teams, making sure everyone knows a product exists and how to use it. The system turns out to include people, regulations and corporate bureaucracy as well as the repositories. It's a lesson people seem to have to reach themselves, like advice to a friend in a bad relationship, and Jessica's version is that you don't break up with your technical skills, you open the relationship. Chelsea adds that you don't get to build things unless people want them, at least not if you want to avoid shelfware.

Chelsea says measuring engineers on productivity metrics inherited from an assembly line, where nails accumulate linearly through the day, is a disservice. Software is knowledge work: "We don't create value by doing the same thing over and over." It creates value by gathering context into an understanding of how to solve a problem now while keeping as many likely directions open, which Chelsea credits to Kent Beck's term optionality. From outside, that looks like nothing, nothing, nothing and then a big release, after months of understanding the problem, talking to users and socializing changes, since changes that torch people's context take power away from them. Writing the code comes last and is the easiest step. "We are treating lines of code like nails." Jessica adds that "our job is not what we do. It's what we know," and Chelsea says much of the day job is finding knowledge for someone and routing it to them, and that early in a career, how much code you can write depends on how good the decision makers about the repository are.

Mozilla's values include data privacy and ethical use of machine learning, which are hard to prioritize when teams spend their energy on getting any model into production through individual heroic efforts. In the fall a team began evaluating products to give data scientists and machine learning engineers a turnkey path to production. Chelsea notes the product landscape is young, with upstarts of around 15 employees, so the larger asks sometimes can't be met and documentation doesn't always match behavior.

Chelsea's advice on evaluating them is to separate optimizing metrics, where more is always better, from satisficing metrics with a good-enough threshold. Engineers tend to treat everything as an optimizing metric and end up in decision deadlock, for instance over scale, when a small startup isn't going to see a billion users at once. Jessica: "Problems you want to have." The optimizing metric that often isn't on the grid is the availability and flexibility of support. Chelsea fought to include it, and found that "the products we chose that have really, really responsive support teams are the products that we have managed to get into production at this point." A support engineer on Slack, for example at Weights and Biases, will take a custom chart problem and reproduce it in their own project. Free and open source tools the team liked ideologically didn't make the cut, since there's nobody whose job is making sure it works for the people using it, which Jessica sums up: "Wow, it's almost like the code isn't the whole thing." Jessica adds that at Honeycomb, good support gets renewals.

Chelsea came to MLOps from software engineering through data science, not through DevOps, and describes a data engineer as someone thrown into the model, the data science code or the app code as needed. Chelsea's view is that it isn't necessarily different in kind but that machine learning models add special considerations. Jessica compares deterministic program execution to the internal combustion engine, now one category among vehicle types. If the same input produces different output, that would worry a DevOps person and not necessarily an MLOps person, though there's a different decision tree, since some causes are fine and some are awful, and diagnosing a malfunctioning model is harder because the internals are automated.

Chelsea tells of someone distraught that ChatGPT claimed to have run code and reported wrong output, and of spending too long explaining the mechanism when the question was really "how could ChatGPT do this to me?" The person wanted to imagine a world where the tool had an interpreter, and Chelsea's answer was that one can imagine that world, but it's not the one we're in. A software engineering background alone isn't enough to debug these systems, and tools built to operationalize deterministic code lack needed features.

One such feature is checking that production data still matches the test data the model was evaluated on. Chelsea cites Andrew Ng's Machine Learning Yearning and a contrived example of a cat-photo model trained on high-quality images that then meets blurry phone photos. At Mozilla, Chelsea works on a system that sanitizes search data, discarding anything that might contain personal information. It drops anything with numerals unless it's on a human-made allow list, and uses spaCy's named entity recognizer to filter names. The team has to be sure the people using the feature are in the population the recognizer was trained and tested on, so it monitors an aggregate distribution of languages in search data, without storing the searches, to catch a shift before anything is stored long term.

Chelsea says MLOps and DevOps share a focus on catchability. There are three risk amplifiers: catastrophicness, likelihood, and insidiousness, "how likely is it that this thing goes uncaught if it happens." Security, DevOps and MLOps focus on insidiousness more than other roles do.

Chelsea's view is that the most effective systems are a series of steps with humans in the loop, not one generalized system doing everything. The example is who-to-follow recommendations on social media. A popularity-based model produces "the Beyoncé problem," which spirals up the already popular and doesn't connect niche audiences. A person can often tell when someone has gamed the engagement system, and human judgment is hard to automate and, Chelsea adds, hard to beat with a model, even though it's spotty at best. Better would be to break it into three simpler steps: find what topics people discuss, figure out who is knowledgeable on them, probably with a human in the loop, and recommend those people to people who want to learn. Chelsea notes that Follow Fridays and hashtags were user inventions, and that classical models on tabular data are often enough. Jessica's summary: for any problem where you'd reach for machine learning or generative AI, break it down into parts where a model helps, parts where a deterministic rule works, and low-volume, high-value parts where a real person should be asked.

Chelsea writes at chelseatroy.com, linked below.

Read more of Chelsea Troy's writing here!