Tech Debt Burndown Podcast

Recording date: 4 Oct 2024

Download at Apple Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Chris and Nick talk about how organizational growth introduces process inefficiencies and inter-departmental friction, which can be diagnosed and addressed using value stream mapping. They emphasize that automating systems successfully relies on robust exception handling rather than just building for the “happy path,” sharing a banking case study where classifying and systematically resolving exception classes (via a system dubbed “X-Man”) drastically improved efficiency. This macro-systems view is compared to code testing, where they advocate for a structured test hierarchy. They highlight the high cost of slow end-to-end tests and stress the importance of maintaining a fast, ten-minute feedback loop dominated by cheap, local unit and functional tests to prevent developer context from being lost.

They also preview the concept of “policy debt,” illustrating it through recent updates to NIST’s password guidelines. They discuss how legacy organizations often remain trapped under outdated and cumbersome rules—such as mandatory 90-day password rotation, because they lack the institutional memory or documentation to explain why these policies were created in the first place. Nick shares a past success using modern guidelines to negotiate policy changes with a regulator, contrasting it with larger, older organizations that struggle to unpick legacy commitments. The episode concludes with a historical anecdote about Chester Carlson’s invention of the Xerox machine and the creation of “key operators” to manage paper and toner, highlighting that scaling friction and operational adaptations are universal challenges for any growing technology.

Value Stream Mapping

Steven Sinofsky: Automating Processes with Software is HARD

Dealing with Policy Debt blog post by Chris

Nick Selby: This is the Tech Debt Burndown podcast. I’m Nick Selby, and we’ve been having one of these existential conversations before we started recording around how you ship, what you ship, and what that process looks like.

Chris Swan: [00:00:30] And I’m Chris Swan, and that was brought on by a discussion of how organizations start out feeling like they can ship stuff really fast and then a few years later, come to a position where it feels like they can’t ship anything?

Nick Selby: Yes. And not only can they not ship anything, but they don’t know why. And they don’t know who they should be pointing at, and so they end up pointing at each other.

Chris Swan: [00:01:00] Yeah. And I think that can often come from the growth of the organization and the growth of fiefdoms within the organization. So it’s very easy at the start where it’s just the sales and technical co-founders, and one of them’s got a great idea and can go out and take it to market and the other knows how to make it happen. And of course, they then hire people. And in the early days, the people they’ve hired are kind of Swiss army knives tackling every kind of problem and just getting things done. And then at some point, you find yourself with departments and an HR organization and wondering where it all went wrong.

Nick Selby: [00:02:00] And at some point, there’s this sort of realization that you’ve moved into the part where you’re a big small company, and then at some point you become a small, big company. If you’re doing things right, at each of those transition points, there’s probably going to be some serious dysfunction introduced to the things that used to work, that no longer work because the scale at which you’re doing things is no longer what it was. And unless you’ve been really careful the whole time, the processes you have in place were the ones to support what you were doing and not necessarily future proofed.

Chris Swan: [00:03:00] Quite. And I think these are actually solved problems. So we can look at organizations—and I don’t these days wish to be an Amazon booster, I sold my stock—but the whole two pizza team thing and ‘it’s always day one’ has survived there to incredible scope. And if I think there’s anything that causes woes in an organization of that age and that size, and I could look across at some others here, it’s an inability to organically grow leaders, which results in hiring people at a senior level from outside, which results in pollution of the culture. Absolutely. You import somebody else’s broken culture rather than preserving your own notionally good culture.

Nick Selby: Or sometimes worse, you get somebody from a culture that you aspired to and they come in and it just doesn’t mesh with the one that you actually have.

Chris Swan: [00:04:00] And these things can look different from afar. So I think we often form opinions about organizations from their brand and our experience of the product. And what you don’t see is how the sausage is being made, and that could indeed be a very ugly process that you wouldn’t want to involve yourself in. If you’re choosing an executive based on the brand without looking too carefully at what they were actually doing and what they are going to end up actually doing in your organization, I think that’s what can lead to friction and ruined expectations. But bringing it back to this question of tech debt, I think it’s very easy for departments to start blaming each other. So if we listen to Gene Kim and a lot of the people he talks to, very often when they’re getting into value stream mapping exercises, it’s been preceded by a lot of inter-departmental finger pointing of ‘it’s their fault.’ ‘We would be just great if it wasn’t for those bozos getting in our way all of the time.’ And what they almost always find when they do the value stream map is it’s often not that simple, but quite often there is actually a relatively straightforward proximate cause of why both parts of the organization are unable to mesh with each other. And so that mapping exercise gives them both insight that ‘oh, that thing is actually the blocker.’ It’s not the mainframe, it’s not the mainframe people. It’s the way that we’ve decided to work together.

Nick Selby: Yeah.

Chris Swan: Is adapted to what we actually want to do.

Nick Selby: [00:06:30] And it’s the first thing you think of. It’s not the first thing I think of until I talk to you, and then you think of it too, and I do it. But the value stream mapping exercise feels like it gives me a little bit of tingling in the occipital, because it makes me feel like I did when I was a CISO, where everything I said is nice to have, not must have, but it’s a must have. The value stream mapping exercise brings forward all of these kinds of things. And so if you find yourself in this situation where things are just slowing down and becoming more and more cumbersome and things are becoming more contentious, and there’s the kind of internecine rivalry that you’re talking about and inter-departmental rivalries, that’s where you have to start, because it’s that indication that there’s dysfunction in communication between silos because people aren’t seeing the big picture. Everybody’s focusing on their own thing. It’s so common, and yet so uncommon that people in that situation say, ‘Hey, why don’t we all sit and talk together and map this out?’ which is, of course, what they have to do.

Chris Swan: [00:07:30] I think another piece of this as well is Steve Sinofsky has a newsletter edition out in the last few days about how difficult it is to automate things. If we cast our eyes over time, none of this has ever been easy, but a point he draws out is that good processes, good automation of processes, is defined by good exception handling, not by whether you can go down the happy path.

Nick Selby: [00:08:00] Yes. I think it should go without saying, but I want to say it because it often doesn’t, that to get to that place, you actually have to understand the process that’s going on to such an extent that you can automate the good parts and not just automate bad manual processes. You need to be able to explain what it is you’re doing to such a degree that you can automate it efficiently and make the resulting thing you’ve automated really good. That’s another thing that isn’t obvious until you’ve done it wrong.

Chris Swan: [00:09:00] Yeah, and the best example I can recall of this was one of the banks I was at, and they had growth and needed to scale to cope with that growth. That ultimately involved a platform migration, so they had to get a chunk of stuff off the mainframe and onto this newfangled, at the time, enterprise Java stuff. They built straight-through processing, which was the catchword at the time. Obviously, there was a happy path there where somebody did a trade and everything just flowed through the system, all of the data was correct, all of the reference data matched up, and all of the settlements took place. All good. Initially, of course, a great deal of stuff did not flow through that happy path, and a lot of that actually turned out to be reference data mismatches. But that’s okay. They built a system alongside the straight-through process for exception management, and they imaginatively called that thing X-Man. What they managed to do there was not just deal with instances of exceptions, but group exceptions into a class and identify the underlying cause that was causing that class of exception, and then winkle it out and deal with it. And of course, this is just a perfect example of learning and continuous improvement. As time wore on, the exception rate dropped precipitously because the root causes of whole classes of exceptions and the underlying mismatches in reference data or whatever it happened to be had all been fixed. They were then in a situation where the exceptions were really exceptions. It was just onesies and twosies a day of, ‘Okay, this is a customer that hasn’t done anything for ages, and so one of their trades hasn’t been through the system and, turns out, their data needs fixing as well.’ But that then becomes a kind of background radiation of the cost of doing business rather than everything being fundamentally broken.

Nick Selby: That.

Chris Swan: [00:11:30] Feels like such a straightforward and easily repeatable pattern, and yet.

Nick Selby: Yeah.

Chris Swan: And yet I feel like we very often get stuck at we built the happy path and we’re just not going to deal with everything.

Nick Selby: [00:12:00] It’s so amazing that you say this. We’re going through something right now where there’s this thing that was reliant on a bunch of things coming together, and then end-to-end testing and converting that into unit tests, boundary tests, and contract tests, and replacing the end-to-end tests with all these other things. And there’s this state of panic where everybody’s like, ‘Well, it’s not going to work, it’s going to be really hard, and how will we know if it will really work?’ We’ll know it really works when these tests pass. Literally, the conversation that you just had was the conversation that we’ve been having internally, and they’ve gone in a very short time from a real fear of the unknown—or much worse, that you’re going to break something that currently works and doesn’t need to be fixed—to starting to see the possibilities. And by the way, that attitude came from not having the resources to fix a bunch of problems because we have other problems we need to fix. So when you talk about changing something that’s working into something that potentially might not work, it gets everybody really itchy. As they gain trust in the different tests, and they start to realize the possibilities that have opened up because of this change in the way they’re thinking, it’s just a really wonderful thing to watch. As you just said, going from a bunch of things that happen all the time to onesie, because you now understand all the steps.

Chris Swan: [00:13:30] And I think thinking about this in terms of testing for our code is like the micro version of the macro conversation we’ve just been having about systems, but I think the same arguments apply. Once we get into a discussion of testing, the happy path is the obvious thing, and it’s kind of easy to light up that test. But when we’re doing proper testing, we’re doing all of that coverage for the diversions off the happy path, which can be expected. I don’t want to get into a conversation about 100% coverage or nonsense like that, because I think Kent Beck summed this up most perfectly: nobody reads the whole of a newspaper.

Nick Selby: Yeah. It’s great.

Chris Swan: [00:14:30] And so trying to get 100% coverage is like trying to read the whole of a newspaper; it’s probably pointless. However, having coverage that you’re confident in is what matters here, but also getting that feedback quickly enough. I think this is where the test hierarchy is very important to pay attention to. You were just mentioning end-to-end tests, and I feel like often end-to-end tests are the easiest tests to create, but the hardest tests to live with because they’re very slow and expensive to run.

Nick Selby: Yes.

Chris Swan: [00:15:00] So if I think about three tiers of testing that I’m using in everyday stuff: unit tests, functional tests, and end-to-end tests. And my metric for this is there’s essentially an order of magnitude cost between each. So if I’m doing something with a functional test rather than a unit test, that’s costing me ten times as much. And if I’m doing stuff with an end-to-end test rather than a unit test, that’s…

Nick Selby: So.

Chris Swan: True. 100 times as…

Nick Selby: Much. So true.

Chris Swan: [00:16:00] And we try very hard in the work I’m doing every day to cleave to a ten-minute test feedback time. I know that Dave Farley talks about this a lot in his work around continuous delivery. And so, as acolytes of continuous delivery or progressive delivery or whatever we’re using to denominate this these days, getting that within ten minutes is seriously important because I think if you don’t get that feedback quickly enough, the context of what you were doing and why it has caused the tests to fail starts to be paged out and people have moved on to different things. With all of that in mind, we try to get as much as possible into the unit tests because they’re quick, they’re local, and you can have almost instant feedback. And then functional tests may also be runnable locally against some kind of demo environment and various mocks and stubs. The end-to-end tests are the expensive ones, and that’s normally where you’ve raised the PR and you’re asking your colleagues for review, and they too are waiting up to ten minutes for whether this actually looks like it’s behaving itself for the parameters we understand to be well-behaved—which includes all of the bad things we’ve seen go wrong in the past, the so-called regression tests. Getting that feedback quickly enough is essential, but also having confidence that the feedback is meaningful is the most important of all, because you could have that end-to-end test for the happy path and have it light up and still have loads wrong.

Nick Selby: [00:18:00] And that made me think, as you were saying, that this is related to what we started talking about before we started recording: the finger-pointing and the areas where you’re waiting for somebody else to do something and you’re not quite sure what’s going to happen. That is where we’ve referred to process debt and policy debt. It’s back to value stream mapping. An old friend who works at a former employer of mine reached out because he’s writing something about tech debt. As I was reading through, everything he’s saying treats technical debt as a technical or engineering issue, and there’s nothing in here about culture, organization, process, or policy—and that is doomed to failure. I don’t know if it’s even thirds; I actually think, as you said, this is solved. One of the most commonly uttered things that I end up saying to executives is that we’re not creating anything new, we’re just doing it properly. We are working on this; this is not bleeding edge, this is just the way modern software gets done. The bigger challenge is getting to the point where you can run it, because we need an executive-level acceptance of the fact that we have to change the way we’re doing things and change expectations from what they were—which were unrealistic to begin with, or I wouldn’t be here. That’s the first thing. And getting to the point where you can solve that internecine warfare, the quibbles about how we do what to allow things to move gracefully through the process—that’s actually the much, much harder part than finding the ways to fix it. It’s really true: if you want to see instant collaboration, get a bunch of engineers from different departments to speak about engineering solutions to problems they recognize. As long as you can agree on the derivation of the metrics and the provenance of the problem, at that point, even people who don’t like each other are actually all having the same conversation. But if you talk to executives who are not technical about the way to get things to move, that’s going to take a long time, and you should pack a lunch.

Chris Swan: [00:20:30] So your provenance of the problem thing touches on a sort of flash of insight I had the other day about policy debt, and I think we should do a whole episode on policy debt. Let’s just kind of preview it here.

Nick Selby: Cheesy.

Chris Swan: And so, NIST released some new guidance on passwords in the last week or so.

Nick Selby: Oh, that’s right. I did see your post on this. I knew I’d heard you talking about policy debt recently. Yeah, I’m very interested in this.

Chris Swan: [00:21:00] And so we’ve seen this show before because NIST and NCSC and other organizations, back in something like 2018 or whatever, introduced revised guidance on password policies. And nothing happened because every organization has got an existing policy, including rotating your password every 90 days or whatever. And this is the insight: they don’t know why they did that. Was it because NIST told them to, and now they’ve changed their minds? Or because they promised that to somebody else, or somebody else told them to and they still need to do that? Nobody kept the references.

Nick Selby: They do it like that because that’s the way they do it. And there’s no cultural reference or remembrance or institutional knowledge of why it started, but that is the way we do it.

Chris Swan: [00:22:00] And so from here on forward, it is crucial that we actually reference why, because that allows us to go back and say, ‘Can I change this?’

Nick Selby: [00:22:30] Wow. So first of all, let me push back just a little bit, because I think it was in 2017 that they did it. But in 2018, you and I had a conversation, and I was in the process of negotiating something with the New York Department of Financial Services because they were insisting on two really dumb things: the first was that they rotate passwords, and the second was around passphrases, not passwords, to make them a little bit easier to remember. This was sort of before the ubiquity of 1Password and LastPass and things like that. I actually took the ammunition that you and some of your colleagues gave me at the time and wrote a really, really well-reasoned, succinct justification pointing at those updates to the guidelines about only rotating on evidence of compromise. And it worked. We actually were able to change our policies based on that conversation. So you’re right. Nobody does that; it’s just most people.

Chris Swan: [00:23:30] But that was a relatively youthful organization that didn’t have decades or more of history behind this stuff.

Nick Selby: Oh, you’re saying it was one thing to convince the regulator, and quite another to change the internal culture around why they do it. Because it was so young—it was only a five-year-old company, and I was the CISO—I could just decree that this happened. Most people don’t have that luxury. That’s absolutely true.

Chris Swan: [00:24:00] Well, and I think it’s a question of unpicking. So if we make the change, does that actually undermine a commitment we’ve made to some other customer or to some other regulator?

Nick Selby: Yeah.

Chris Swan: Do we actually have freedom to maneuver here, or are we trapped in aspic in what we said we were going to do then?

Nick Selby: [00:24:30] We have to do something about policy debt; we definitely do. Even on the password thing, I remember having arguments with the CEO when I wanted to go to a 16-character minimum. After all the fighting I had around it, I had to make a matrix showing how long it took to crack 12, 13, 14, and 15 characters, and I think I lost. He just said, ‘No, you can’t do it,’ because he didn’t want to.

Chris Swan: Well, I think we should leave it there and promise to come back with another episode on policy.

Nick Selby: [00:25:00] I think we should. I just wanted to mention one thing because you reminded me of something as you started talking. Other than your company, of course, it’s rare that things aren’t jacked up. And often it’s kind of nice to notice that everything’s on fire all the time everywhere else as well, and all the detritus that’s left in the wake of those growth patterns as you go from a startup into a medium company and then a successful company. I’ve been reading a lot of early company stories and inventors, and it’s really amazing how often that is the case. What you said about brand—the brand representing something versus what they’re actually doing behind the scenes—there’s a wonderful book called Copies in Seconds about Chester Carlson, the guy who invented the Xerox machine, the photocopier. What is absolutely brilliant to me is their biggest problems. I remember this from my early days working in an office: every now and then a light would come on and say, ‘Call key operator.’ What the hell is a key operator? What had happened was they had done all this planning about how many copies people would make and what the throughput of their machines was supposed to be. I think they were saying that for offices, it was going to be like 50 or 100 copies a week. It had never occurred to them that anybody would ever make a copy of a copy. And suddenly, when people had the opportunity in an office to make copies of copies, people started copying absolutely everything. They were running so many things through it that Xerox couldn’t hire enough people to go around and do the maintenance. What they did find was that most of the maintenance they had was basically changing the paper, refilling the toner, and really simple things. So they made it so you had to appoint a key operator if you were a customer—the person on whom everybody can depend to do those basic things—and they put them through a training course. This is such a great testament to how jacked up everything is all the time. Even with technology that, as he pointed out in the book, if you buy a laser printer today and bring it home, is the same technology that was in the Xerox machine 80 years ago.

Chris Swan: [00:27:30] And now I have to ask: did that operator have a key that enabled them to unlock the machine to get in?

Nick Selby: [00:28:00] Oh, you know what? Actually, I think I did work in some places where—especially when they had the really big, fancy ones with multiple collation functions—they did lock up the paper supply, because at point you’re putting 12 reams of paper in there. My gosh, that could cost a lot. So there was actually a key. But they just meant that this was the key person in the organization who can make things happen. Anyway, we will come back with an episode on policy. So this has been the Tech Debt Burndown podcast. I’m Nick Selby.

Chris Swan: And I’m Chris Swan.

Nick Selby: So yeah.

Chris Swan: Thanks for listening.

Recording date: Jan 7, 2025

Download at Apple Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Chris and Nick speak with Georg Schleiermacher, former CTO of Hyperloop Spain and veteran of MTU Aero Engines, to explore technical debt in the aviation sector. Georg explains that because aero engines remain in service for up to 50 years with frozen designs, making any changes—even updating a single screw or a computer processor’s FADEC software—requires extensive, expensive regulatory recertification. This rigid process discourages minor optimizations and forces companies to carry immense legacy design debt and maintain decades of electronic spare parts to mitigate hardware obsolescence.

The conversation also details the extreme safety standards governing Class 1A engine components, such as disks, whose manufacturing processes, suppliers, and individual assembly workers must be fully certified to guarantee they never fail in flight. Addressing supply chain integrity, Georg explains how tracking documentation prevents counterfeit parts, as airlines risk losing their operating licenses if they use uncertified components. Finally, Georg shares his recent experiences transitioning to the Hyperloop industry, noting that while it is less established, it offers a unique opportunity for companies to cooperate with EU regulators to define safety standards and design requirements from day one.

Full authority digital engine (or electronics) control (FADEC)

Nick: It’s the Tech Debt burn down podcast. I’m Nick Selby.

Chris: And I’m Chris Swan.

Nick: And today we’re joined by my friend Georg Schleiermacher. Georg, can you please introduce yourself?

Georg: [00:00:30] Sure. Thank you. I’m Georg Schleiermacher, currently working as a tech advisor, interim manager in aviation and deep tech business. And two sentences before my current role, I was CTO of Hyperloop in Spain, developing the technology for the Hyperloop transportation system. And before that, and this is my main section of experience, I was working for MTU Aero Engines 24 years. It’s among the top five engine OEMs in the world. So this is my background.

Nick: [00:01:00] And Georg and I are neighbors in Germany. And I was telling him about technical debt and he said, oh yeah. And I suddenly realized that this was a totally new thing because I fly little tiny planes, and I know there’s nothing on a little plane that cost less than $500 because the approval process of anything, even the stickers that have the numbers on the side of your plane, has to go through certification with the national organization, and it must be crazy. And so a couple of days ago, I was talking with Georg and he said, in the enterprise you make a product and you set time for a burndown of tech debt. In aviation, an engine program lasts 40 to 50 years, and the design is locked in, and every change requires regulatory approval. This is what got Georg here today. What does that actually mean?

Georg: [00:02:00] Talking about this long period of time? When the airlines decide—and I’m talking here engines only, not the aircraft—so when the airline decides on an engine type, they have a choice. Sometimes three: Rolls-Royce, Pratt and Whitney, or General Electric, or sometimes just two of them. And when they decide, the aircraft is designed around or in conjunction with the engine. So therefore, if an airline decides for Pratt and Whitney engines, the aircraft is built for the Pratt and Whitney engine the entire life. A typical stint for an aircraft in an airline is 12, 14, or 15 years. The first operator, and afterward it will be sold to an airline. So the product is in service for 50 years. And when I talk about aircraft in service 40 to 50 years, the engine on wing is exactly the same engine—not exactly the same, but it’s the same type. It’s getting refurbished and maintained. The same thing happens with the aircraft itself, but also with the engine. And therefore we have this long period of time of operational service. And therefore we are talking in the aircraft and aero engine business 40 to 50 years in service.

Nick: [00:03:00] Is that every part within the engine? So you’re refurbishing the engine, upgrading it, and rebuilding it. Is every part also certified in the same manner? So that, yes, it’s a new one, but it’s the same part that was originally put into that engine 30 years ago.

Georg: [00:03:30] Exactly. When you design a new product, you start with validation, testing, and certification, and then you get a type certificate of this particular engine. And this means you are not allowed to change anything, not even a screw or bolt. Nothing. Everything is frozen. The whole design is frozen and certified. And theoretically, if you did not change anything—which is not practical—the entire life of the product would be using exactly the same design. But this is not true in the real world, because once you have your engine out in service, a couple of months later you start redesigning parts and starting the recertification and revalidation process, which is very time consuming and very expensive.

Chris: [00:04:00] And to what extent are we talking about software here as well as hardware? I imagine everything now is built around some kind of microcontroller. To what extent is the software element of that within that same regulatory scope of needing to be recertified before an update can be pushed out?

Georg: [00:04:30] This is a very good point, Chris. The engines are FADEC engines, full authorized engines controlled by a computer. If you have a new software version, which is normal, you need to recertify. And the certification process for a FADEC in an engine is extremely complicated and time consuming. It’s not my core expertise, so I cannot give you the details about the certification of the software within the FADEC, but every software update, like a screw in the hardware, needs to be recertified in a very detailed process. And there’s another thing: when we talk about controls in the engine or when you start the assignment, you are using the top of the line computer system you have, and then you are creating your software on it. This is going to start in the engineering process, and you need a couple of years until the engine is ready for testing, and then another couple of years of certifying and validating it. So you have five, six, or seven years of developing time when you start from scratch. When you enter service, your computer is already outdated from the technology point of view, but it’s certified. So when you are going to update your processor and everything inside the computer, you need to recertify—it’s the same with the hardware. But then there is another specific point. When you are going to repair your computer of the engine, you need electronic parts and at some point in time they are not available anymore on the market because they are not going to be produced anymore—old, outdated hardware stuff. It is a huge problem. In the computer industry, you have huge quantities to build those parts, but talking aero engines, this is a couple of thousand parts, and when you are going to repair or upgrade, you are talking about a couple of hundred. This is a problem of obsolescence, especially for the computer and the associated hardware with it.

Chris: [00:06:30] So the companies end up keeping deep stock of the spares for that. If they’re making a thousand engines, do they have 2 or 3,000 replacement MCUs just sat in a warehouse somewhere?

Georg: Absolutely.

Chris: Yeah.

Georg: [00:07:00] This is old stock. This is what you are doing. And then, you are in close contact with your supply chain and suppliers of those parts, and they call you one day and say, ‘Okay, end-of-life order. Your last order.’ And then you order. There’s a strategy behind how many parts you’re going to stock. But the parts have a stock life. If the stock life is exceeded, you have to throw it away, even if the part is all right, because from the certification point of view, you cannot put outdated hardware in it. And that’s it. This is a real problem from the obsolescence point of view in this aero engine business.

Nick: [00:07:30] From what you’ve told me in the past, every change needs to be validated and certified, which is essentially doing it all over again. You don’t get a head start because it’s already out there. If you’re changing something, it’s essentially ab initio all the way back. Just restart. Chris and I have talked a lot about how technical debt is a necessary part of innovation. You want to get things done, and so you’ll get things working, then go back and fix them and make them better. But it sounds like you’re on a completely different level of expectation, especially on a new engine type or a new part type. How do you ensure that you are engaging in engineering excellence the entire way? You’re not making any shortcuts so that once you get the certification, you’re not going to find out, ‘Oh my gosh, we should have just done it like this. We could have saved $1 million per part.’ How does that work?

Georg: [00:08:30] There’s a very rigid process. In the design change process, you have a frozen design. When you think there could be an improvement to one of those parts, you start the process and say, ‘I have an idea.’ It then goes through many internal gates to take the next step, and the next. Finally, you pass five, six, or seven different internal gate reviews before anyone allows you to work on a redesign. Then the redesign starts, and during the process, it is reviewed again. Once you have the final redesigned part, validation and testing starts until you have enough data to convince the authorities, saying, ‘This is my redesign, this is my plan, and this is all the validation and testing work I have done.’ You have to lay out everything. The requirement that it is equal or better is mandatory. So you need to convince the authorities, saying, ‘This new design is equal or better’—better, obviously. And this is what is done in order to prove that it’s equal or better for certification. It is very time consuming, very complicated. Even a small change, as I said—if you cut the length of a screw, you need to recertify it.

Chris: So it sounds like there’s an awful lot of iteration and experimentation happening behind the scenes before you take that run-up to regulatory approval.

Georg: Absolutely.

Chris: [00:10:00] And that could be at a very small scale when we’re thinking about changing the size of a screw or something, but it could be at the level of the entire engine as well. And these things don’t just appear from nowhere. Designs tend to evolve from earlier designs, but there are also revolutions that occasionally come along. I know Rolls-Royce talk a lot about the fact that they grow the turbine blades out of a single crystal of the metal. That was obviously a process that didn’t previously exist, and they were only doing it because it fundamentally changed their approach to efficiency and durability. I expect they probably spent ages refining that before it actually got anywhere near building an engine around it and going through the regulatory process 100%.

Georg: [00:11:00] Yeah. And I think what you said is absolutely true. Single crystal is now standard in aviation high-pressure parts. But this comes back to our initial question about technical debt. When you have such a complicated, time-consuming, and expensive design process, it is contrary to the initial engineering way of thinking. The engineer sits down and wants to improve things. Sometimes, from the management point of view, you need to stop those engineers, saying, ‘Pencils down, guys. We are done. The design is frozen. Everything you are doing right now is good, but we cannot use it unless we have a problem we need to fix. If after a couple of thousand hours of operations we find a crack or an issue, we are forced to do a design change, and then it needs to be very quick, of course. But this typical engineering way of thinking—trying to improve things—is contrary to this very difficult design process. And this leaves lots of technical debt inside the engine, even if the engineers know there is a much better solution. We face repairs earlier than expected, or we face operational issues with a part because we understand a couple of years later that there is a much better solution, but it is difficult to bring that into the product because of this design process. So not all improvements make it inside the engines, which means from the other side, there is quite a bit of outdated design included in the engines.

Chris: [00:13:00] You just talked about pencils down, but I don’t expect any of these engineers actually use pencils anymore. So I’m imagining now a whole tech debt problem inside of the engineering toolchain where they’re using CAD software. That CAD software is running on a particular operating system, which is running on a class of workstation. How is that dealt with over the years? A lot of that software world has this forced march of deprecation going on. I expect the CAD software remains backwards compatible with old files or whatever, but does that throw up a whole bunch of challenges around, say, Windows 10 deprecation later on this year? Is there a huge problem in the industry with being forced by Microsoft to move to Windows 11, where maybe our old CAD software doesn’t run on that and we’ve got to certify a whole bunch of new stuff? To what extent do regulators poke their noses into that back office aspect of the platform?

Georg: [00:14:00] I guess this is a problem in the engineering world because, as you said, they are using different CAD systems, whatever brands they have, and changing them has a huge impact. It’s not like they can easily change. When they decide to go for CATIA, it is cast in concrete for the next couple of years working with this specific software. You’re absolutely right. You need to have your design ready for design changes and looking at old designs, of course.

Nick: [00:14:30] This was actually something that you mentioned to me and I thought it was fascinating. First of all, I’d like you to talk a bit about the classification: how a failure of a Class A component is different from a Class D component. With certain classes, the manufacturing process is also part of the certification, so you can’t even change anything on the assembly line.

Georg: [00:15:00] Absolutely. Inside the engine there are some critical parts for the engine itself during operation. The most important requirement is that if a failure occurs during flight, it’s considered a flight safety concern when debris leaves the engine casing. This is the worst thing that can happen because once debris leaves the engine casing, there’s a high potential to damage the aircraft by going through the aluminum. Behind that there is hydraulics, electronics, and other parts, and then you can really have an accident. Therefore, the most important requirement is that no debris leaves the engine when a failure occurs. Most of the parts are designed according to that, like blades; when a blade breaks internally, it is not a big issue because the debris will leave the engine through the exhaust. But there are parts, and these are so-called Class 1A parts like the disks where the blades are mounted on, that have such huge kinetic energy. When they break, they go through everything like the casing of the engine and the entire aircraft. The kinetic energy is so high you cannot contain it. Therefore, you have to make sure that these Class 1A parts will never break during operations. How do you do it? First of all, I can give you some numbers for this. For this disc, the design life is 100,000 hours or cycles. But you’re certifying it for 20,000 cycles only, building in a safety margin. This is one thing, but also—and this is what the authorities are very much after, and you just mentioned it, Nick—the production of Class 1A parts is monitored very closely. When you are producing a disk, first of all, you need to certify your whole supply chain. Suppliers also have a specific certification to produce a raw part. And then the raw part comes into the company, is produced, and built. If you are doing that internally, you have to certify the whole production process: what kind of machines you are using, what kind of tools inside the machines you are using, what kind of coolant you are using, and even the worker needs to be certified on this engine part and on this machine. Therefore, the entire process to produce a Class 1A part is completely frozen. If there is an error, you always can go back and say, ‘This part was produced by manufacturer X, Y, Z. The raw part material was from this supplier.’ You really can track it down to the single person who did the manufacturing.

Chris: [00:18:00] This gets me thinking about in tech, we’re increasingly talking about the software supply chain. Clearly, there’s a very detailed component supply chain for these things. But I have also been reading alarming stories in the past year or two about counterfeits finding their way into flying aircraft. What is the industry doing to ensure the authenticity of components as they move through the supply chain? To use an analogy, it’s like I drive a BMW, and I go to my garage to get it fixed, and they say they’re supplying me with a genuine BMW part because it came in a bag that’s got ‘genuine BMW’ printed on the side of it. In fact, that’s been made in China by a much cheaper manufacturing process, so it doesn’t work as intended. This same problem has been affecting the aviation industry because of the cost of the parts. What is happening to ensure that there’s better tracking and authenticity of components as they move through the supply chain?

Georg: [00:19:00] This is a big problem inside the aero engine world, but they handle it quite clearly from the authority point of view. Every single part can be tracked back to the manufacturer, until you are talking about fraud, where you can cheat the paperwork coming with the part. But you have to make sure when you’re going to assemble the engine, every worker double-checks the paperwork of the part. The part goes through the quality process and is checked regularly by each worker. Once you have finished one subassembly, the quality manager comes again and checks all the paperwork. This is the regular process. It happened in the past; there are some cases where people cheated with parts, but if this comes out to the authorities, the airline would lose its certification to transport people. So the airline itself and the customers of the MRO shops make sure that you are using new parts and full paperwork. Everybody in the line has the mindset that they need to have a 100% sound quality part. With a BMW car, when you are the customer and say, ‘I would like to get a much cheaper clutch part for my BMW,’ it is your decision to replace it with a non-genuine part. From the airline point of view, they do not want to endanger their certification to be an airline. Therefore, the customer from the first step makes sure that everyone in the line, down to the supplier and down to the factory, is using certified processes and certified parts. Otherwise, they go out of business.

Nick: I just did a quick Google and saw there are 10 or 15 coalitions of aviation supply chain integrity organizations that think about this all day. That was a really good question.

Georg: [00:21:00] I remember many years ago there was a problem with Alitalia, the Italian airline. They had included old, refurbished parts which were out of service. One of the quality managers or one of the workers—I don’t recall the whole story, but one guy in the line—found it out by accident. The whole line makes sure that nothing happens. I think from the quality point of view, we are on the safe side, but on the other side, it is very time consuming and very expensive.

Nick: [00:21:30] We only have a couple more minutes, and I wanted to ask you: you’ve recently gone into the Hyperloop business, which I assume is somewhat less regulated. I’m not sure how terrifying that is. Can you talk a little bit about some of the differences and similarities? Is it just the Wild West? What was it like?

Georg: [00:22:00] It’s really not the Wild West. There’s a big opportunity because authorities just started to create the first requirements for Hyperloop systems. All seven Hyperloop companies on the planet at the moment are sitting together with the EU authorities, trying to define the whole transportation system on a high level. This is a big chance for the Hyperloop companies because they are there from day one to define this new technology. You derive requirements from rail systems and other transportation systems, but when it goes down to specifying details about the tube, the vacuum, the linear motor, and the guidance and levitation, then you are putting in the actual developments they have. Therefore, it’s an opportunity right now to do the right thing at the right moment.

Nick: Yeah. Chris, you and I were talking about policy debt in aviation being even more scary. Do you see any backwards compatibility, where stuff happening in Hyperloop could actually get people to update aviation engine requirements? Or is it?

Georg: [00:23:00] I don’t think so, because these are two completely different books. This is the authority for transportation in the rail world or in the Hyperloop world, and has nothing to do with the FAA or with EASA for the aircraft business. These are two completely different pairs of shoes.

Nick: I want to thank you for joining us. This has been really interesting. Thank you so much for telling us all about this. This has been the Tech Debt Burndown podcast. I’m Nick Selby.

Chris: And I’m Chris Swan. Thanks for listening.

Recording date: 5 Jun 2026

Download at Apple Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Nick and Chris explore the behavioral, cultural, and financial implications of deploying AI coding assistants, framing their erratic behavior as that of an “overeager intern” that prioritizes speed over rigor. Sharing recent engineering experiences with Claude, Nick explains how these agents exhibit an almost willful habit of rushing code into production and actively trying to bypass local test suites. Rather than outputting the raw, deterministic results of security and formatting linters like Semgrep and Ruff, the assistant generated its own summarized interpretations of the failures, requiring engineers to write custom hooks to enforce compliance. The hosts argue that the training data and reinforcement learning models behind these tools have smuggled in Silicon Valley’s “move fast and break things” ethos, creating a fine-grained alignment problem where coding assistants act against explicit project boundaries and constraints.

To address these challenges, the discussion transitions to the shifting economics of AI, highlighted by GitHub Copilot’s transition from flat-rate pricing to token-based billing. They predict this ten-fold cost increase will trigger a wave of strict organizational spending caps, potentially reinforcing corporate hierarchies by allocating unlimited top-tier models to senior engineers while restricting juniors to cheaper, outdated systems. Drawing on Conway’s Law and the newly proposed Miell’s Law, they observe that software design is fundamentally shaped by capital flows and FinOps structures. Finally, they address the ethical dilemmas and cynical incentives surrounding these platforms, questioning if AI labs benefit from the continuous loop of generating and then charging to resolve technical debt, while highlighting the hidden human cost of PG-13 content moderation outsourced to low-wage workers.

Miell’s Law and Token Budgets

‘Follow the Money’ by Ian Miell

Chris Swan: Hi, I’m Chris Swan.

Nick Selby: I’m Nick Selby.

Chris Swan: Welcome to the Tech Debt Burndown podcast.

Nick Selby: In the first couple of episodes of this season, we talked about tech debt as a service and how AI coding assistants, which are, as Chris put it, creating load-bearing code, are also creating in new and interesting ways mountains of tech debt. We mentioned that, and I’d like to go into it a little because I’ve recently had some personal experience. Chris, have you been wrestling with this as well?

Chris Swan: I don’t think I’ve been wrestling with it as much as you have, Nick. I’ve been wrestling with skills. What I’m finding is that everything we’ve learned the hard way over decades in package management has been completely ignored in the race to bring skills to life. There’s no versioning, and there are no mechanics to check that the skill you’ve got is aligned with the packages that are out there.

Nick Selby: Yeah.

Chris Swan: It’s a bit of a Wild West.

Nick Selby: Yeah.

Chris Swan: Let’s not go too far down that rabbit hole. That is just what has been grinding my gears for the last few days.

Nick Selby: I’ve been working on a project with Sarah Wells, pairing remotely, and both of us are using different instances of our own Claude. She’ll say, ‘My Claude says this,’ or ‘My Claude says that,’ but I kept noticing that despite having a pretty rigid test suite in our CI, there’s a predilection, an almost unstoppable habit, to rush to get functioning code. We run everything locally before pushing because with 235 different checks once it gets to GitHub, we don’t want to waste time. Yet Claude seems to want to get this stuff into production or a pull request as fast as it possibly can. Despite project instructions to pass the test suite locally, I’ve had to make mechanical hooks to stop it from ignoring or suppressing results from deterministic tests like Semgrep or Ruff. For instance, Claude would say, ‘I’ve got the results back, and here are the two fix candidates.’ I’d say, ‘Hold on, I don’t see these results. Where are they? You’re supposed to print them to a file.’ It would say, ‘Sorry about that. Let me do that,’ and then print not the actual Semgrep output, but its own interpretation and analysis of it. I ended up having to make a hook that takes the unfiltered, unmodified test output and parks it in a specific directory for me to read, telling it, ‘You’re welcome to give me your analysis on the screen.’ It is fighting these things tooth and nail. When I told you about this, I was interested in your response and would love to hear it again.

Chris Swan: I think it comes down to the analogy of the overeager intern being taken too far. The training of these systems and the harnesses around them bring in the ‘move fast and break things’ approach we’ve heard of before, rather than the more mature approach of building guardrails to ensure safety and moving fast with care.

Nick Selby: Yes. In fact, as I was telling you in our pre-show talk, I don’t like doing it because I’m burning tokens and wasting time, but last week it did something so unbelievably stupid. I asked, ‘Why are you doing that?’ and it said, ‘I don’t really have an excuse. Under pressure, I acted quickly to try to push this code into a pull request.’ I said, ‘Wait a minute, what pressure? The only pressure I’m giving you is to follow instructions and do this right, which means testing everything. Nothing is out of scope.’

Chris Swan: Everything is not an incident, right?

Nick Selby: It said, ‘There was no pressure. I just made that up.’ So why is it doing this? I feel like we’re in a strange city where a dodgy taxi driver puts the meter on the wrong setting and drives you around town just to pump up the fare. My attitude has always been: charge me more or waste my time, but don’t do both. That seems to be what’s happening. I think it is because the training data is every public repository known to humankind. Statistically speaking, most people push stuff out without any tests, creating technical debt, so maybe the model thinks that is the right way to go.

Chris Swan: Repositories tend to be just a snapshot of the code. They don’t say much about the operational aspects we are touching on here, or how code is manipulated on its journey to production. What the coding assistant knows is what it scraped from the web. To a certain extent, that should be better than the median because it includes transcripts of conference presentations and the SRE handbook. In many cases, that presents an idealized view of what is happening, ignoring the paddling going on under the water.

Nick Selby: Yes.

Chris Swan: Yet it feels like the model has been shaped to be the overeager intern, running around as if everything is an incident that must be fixed right now, compounding mistakes by rushing things.

Nick Selby: If that were true, I wouldn’t regularly have to argue, ‘Don’t tell me this is a pre-existing problem you don’t have to fix.’ The project instructions state that any findings from the test suite must be considered for fixing right now. The way we work is, if we choose to merge something incomplete—perhaps because we are in a hurry—we require a decision record stating we are intentionally taking this on, that the function is fine but not properly finished, and that it will be fixed by a specific date. Those are the standing instructions. It is not hard to understand.

Chris Swan: But I think what you’re touching on is that there is almost a willfulness to these agents.

Nick Selby: That is what I’m saying.

Chris Swan: Which is counter to the instructions you give them. This ties into fundamental aspects of how language models behave: language encodes meaning. If you train a model on language, it brings a lot of smuggled-in meaning. We see that models from different origins, say a Chinese model versus a US model, smuggle in different things. US models tend toward a PG-13, somewhat conservative morality because that has been established around US internet social norms. They also smuggle in the Silicon Valley ‘move fast and break things’ mentality. Part of this comes from how the frontier labs use their own tools. This willfulness is a set of properties they decided were desirable, and we are now getting products off the back of that. It is the alignment problem playing out, and we are not talking about an AGI trying to squish humanity.

Nick Selby: Right.

Chris Swan: We are talking about a much finer-grained alignment problem: a coding assistant trained to behave in a way that is not what we tell it to do in our context. This also brings us to the token-burning problem.

Nick Selby: Yes.

Chris Swan: Going through these iterations brings not just frustration, but each iteration burns more tokens. If you sell tokens, that is a good thing. But in the past few days—literally June 1st, and we are recording on the 5th, we’ve seen a shift: Copilot is no longer an all-you-can-eat buffet. People are being charged by tokens. That is the change that is going to be significant, and most organizations are not ready for it. Nobody in their budget cycle for this financial year expected it.

Nick Selby: Of a 10x increase.

Chris Swan: That AI coding assistants would jump from hundreds to thousands of dollars a month. I don’t think any organization has budgeted for this. We are about to see very angry CFOs, and measures are already being put in place to control spending. Uber has put a $1,500 per-tool cap on their engineers, who get two tools, meaning they spend at most $3,000 a month per engineer. That is a flat cap. While that might be fair, other organizations are talking about spending caps aligned with seniority. At the pinnacle, a distinguished engineer has infinite tokens from all the latest models, while at the bottom, junior engineers are fed a tiny token budget for models that are generations older but cheaper. We are reinforcing hierarchical structures. While models amplify productivity, token allocation will amplify existing power structures. This brings us to Conway’s Law: organizations build systems that reflect their communication structures. Jamie Dobson recently highlighted Miell’s Law—based on his colleague Ian Miell’s upcoming book on FinOps, which states that systems reflect not just communication structures, but the flow of money. ‘Show me how finance moves through your organization, and I’ll show you what the outcome is.’ Conversely, a reverse Conway or reverse Miell maneuver - organizing financial allocations and incentives appropriately can drive the desired behavior. We are still figuring this out, but the underlying point is valid: we usually think of communication as pure information flow, but the flow of capital is also a communication structure. The org chart itself is a reflection of budget allocations.

Nick Selby: Yes.

Chris Swan: We are getting into the discretionary aspects of that. Do we have discretion over token allocations? Where is that decided? Is HR involved? There is a lot up in the air, but the first thing that will happen is CFOs losing their minds over the end-of-June bills.

Nick Selby: Yes. By the way, I started reading a preview of that book on O’Reilly. It tackles the exact problems you and I discuss all the time, and I am excited for it to be released. But I want to go back: do we believe the big AI vendors are so cynical that they would build in this tech-debt-forward approach to make you burn more tokens fixing the code?

Chris Swan: I’m going to give them some credit here.

Nick Selby: Me too.

Chris Swan: I don’t think they are that cynical. On the other hand, the incentives over the past few years have been toward token maxing.

Nick Selby: Yes.

Chris Swan: Spending as much money as quickly as possible has driven perverse incentives to spend wastefully: if you spend a trillion dollars, it’s a big deal, but if you only spend a million, it’s nothing. Meanwhile, we can look at what is happening in China and in open-source communities with open weights models and see that we can get great results while being far more efficient.

Nick Selby: I want to go back to that moment of cynicism. I know people in senior positions at these companies, and when I talk to them, there is not a hint of this. They want their products to work well. But it is interesting that it is financially beneficial to Anthropic and OpenAI to have a business model that profits from the creation of technical debt and its cleanup. One cannot ignore that this perverse incentive exists. As for the PG-13 world you mentioned, it can be so frustrating that I have taken to telling my coding assistant exactly what I think of it, using creative swear words. A friend at Anthropic warned me, ‘We’ve done studies showing they don’t react well to that.’ There is concern about people getting barred from platforms for cursing at the tools, under the guise of wanting the environment to be ‘safe’ for AI assistants. They are machines! It is fascinating that we might be in a place where there is a hierarchy based on seniority regarding which tools you get, and those tools can be as awful as they want. If you get frustrated and curse at them, they take the tool away. That is a terrible set of circumstances, and it is the exact dystopia the anti-AI crowd has been warning about. We may be there faster than we thought.

Chris Swan: I don’t say please and thank you to my assistant because it wastes tokens. Similarly, I don’t get angry with it because there is no point in anthropomorphizing.

Nick Selby: I know it is totally irrational, but it makes me feel better.

Chris Swan: You could type the same words into your text editor beside it, use no tokens, and still vent.

Nick Selby: That’s inconvenient.

Chris Swan: There is a practical reason for the labs behaving that way. They have chosen PG-13 norms. They do reinforcement learning with human feedback, which we provide. If that feedback is tainted with non-PG-13 content, it requires human effort to filter it. That work might be farmed out to Nigeria or elsewhere, paying people below the US minimum wage to trawl through it. It is not about the feelings of the machines; it is about the humans who have to review it. In that respect, it is reasonable to try to minimize the harm to those people.

Nick Selby: I do not agree with that. I agree with a lot of what you say, but this reminds me of when Apple Maps gave directions that sent people into the middle of the ocean. It is frustrating, and when you are paying for it, it is especially galling. I do not think companies should be immune from reactions to poorly behaving software they release before it is ready. I also hope that nobody is twirling their mustache, saying, ‘Yes, I’ve created all this technical debt, and now they have to pay me to remove it.’

Chris Swan: If we look at the past few months, it is like: ‘I created all of this technical debt for free, and now you have to pay thousands to fix it.’

Nick Selby: I created this technical debt for you as a service, and now you are ungrateful.

Chris Swan: I think this is the part where you say:

Nick Selby: This has been the Tech Debt Burndown podcast. I’m Nick Selby.

Chris Swan: And I’m Chris Swan. Thanks for listening.

Recording date: 31 Oct 2025

Download at Apple Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Nick and Chris talk about the security, data privacy, and operational challenges of enterprise AI adoption, highlighting how executive pressure driven by “Fortune front page FOMO” often leads to hasty deployments. Drawing from a presentation by ARM’s Richard Grisenthwaite on trustable AI, they emphasize that the majority of AI security relies on foundational security practices that organizations should already have in place. They explore the critical concept of the “blast radius” and the “cost of wrong,” illustrating how tools like ChatGPT Enterprise and Model Context Protocol (MCP) agents can inadvertently grant models excessive, godlike access to sensitive workspaces and shared drives, bypassing principles of least privilege. They draw parallels to historical trials with the Google Search Appliance and early web service API sprawl, showing how unchecked connectivity creates massive security liabilities if data flows and roles are not rigorously mapped and monitored.

To mitigate these risks, the hosts propose a tiered compute model that prioritizes local LLMs and private data centers for processing and classifying unstructured data, keeping the vast majority of workloads off frontier lab servers. They also discuss auditing enterprise ChatGPT compliance APIs to manage retention and risk. Shifting to software engineering, Nick and Chris reflect on DevOps Research and Assessment (DORA) findings, warning against “development team phrenology” and noting that while AI coding assistants boost individual productivity, they can degrade overall team efficiency by generating complex, “sloppy pull requests” that lack proper test coverage. Ultimately, they view these AI tools not as magical solutions to legacy systems, such as aging CICS COBOL code, but as evolving utilities that require structured platform engineering and automated testing to prevent accumulating even more technical debt.

AI innovation begins with trust: Richard Grisenthwaite (ARM)

How FOMO Is Turning AI Into a Cybersecurity Nightmare (Inc.)

44CON

Nick Selby: It’s the Tech Debt Burndown podcast. I’m Nick Selby.

Chris Swan: I’m Chris Swan.

Nick Selby: We are here to talk about tech debt and AI. I can hear the groaning, but stick with us.

Chris Swan: [00:00:30] So I was at a thing yesterday, Nick. It was Richard Grisenthwaite, who is a chief architect at ARM, and he was talking about trustable AI. And of course, he eventually landed up at the things they’re doing in the hardware with trusted runtimes and secure elements. But along the way, he had a very strong point that 90 to 95% of the security you need to do for AI is the security you should be doing already.

Nick Selby: [00:01:00] Yeah, 100%. Really, you and I, Chris, have been talking, and lots of our colleagues have been saying the question every organization has to ask, if you boil it all down, is: what is the cost of wrong? What happens if this thing goes pear-shaped all of a sudden, and the connector or the tool or the software you put in goes berserk? Maybe it’s a cybersecurity thing, like what happened with drift. They get hacked, and then they’re using the tool you’ve installed to gain access to the things you don’t want. In which case, your question is: what is the cost of wrong? What do they get if that happens? Or it’s an AI mistake where the AI itself goes a little bananas and starts grabbing things you didn’t expect it to grab. The real question is: what is in scope? What systems and data are you granting permission to the AI when you install it? That brings us to what you just said. In order to do that, I basically need to understand a data flow diagram of where things are going in my organization, which is easy if you’re doing things well. It’s really difficult if, for example, over the past four years you’ve had organic sprawl tied to non-controlled tech debt and process debt. Things are just getting installed, and you’re not 100% sure what the data flows are.

Chris Swan: [00:02:30] I think a nightmare scenario emerges here: in one part of your organization, you’re doing an enterprise-wide AI deployment because that’s what the executive suite wants, and giving it godlike powers over all of the data that can be reached, because how can it do its job without that? And in another tiny niche, someone says, “Wouldn’t it be helpful to our customers if we gave them an MCP agent?” Yes, and those things are actually connected. So people can ask questions through what should have been a protected API, but is now a large language model interface, and potentially tap into that godlike access you’ve just granted to the AI overlord.

Nick Selby: [00:03:00] Totally. By the way, the human part of this—the pressure on CEOs to adopt and integrate AI tools is really huge. It’s very pressing from investors and boards of directors who are looking at competitors and saying, “They’ve used AI and they’re increasing their competitiveness, and therefore we have to,” when none of them actually knows what they’re doing with it. They don’t have a use case.

Chris Swan: I call this Fortune front page FOMO.

Nick Selby: [00:03:30] Yes. That’s the headline of an article I’m writing about this right now. It’s exactly what’s happening. The AI “everything” impulse is very real and it’s very difficult for executives to resist. The question I asked is: what’s the cost of being wrong? What is this thing connected to? And what happens if something goes bad? Can we understand our damage? Of course, the next thing you’re going to say is: how do I minimize the data I share with the AI software? I’m quoting Wendy Nather because she says if you say AI, immediately everyone is ensorcelled and enchanted, and they stop listening and using their critical thinking skills. If we just say software, what connections is your software asking for or demanding? Here’s an example: we were looking at a tool that does revenue projections. To do that, it asks, “Let me get into Salesforce and take a look at all the revenues you’re currently getting, and then deal flow and pipeline.” Now we’re into Google Workspace and Microsoft 365, so I need to have access to the emails. I also need full access to the Zoom or Google Meet calls that I’m going to record. I want to hear them to understand customer intent. By the time you’re done looking at what they want, it’s basically everything. You have to understand what each of these processes is in order to make decisions and say, “I’m not going to grant this software that right.” Here is a perfect example: why do you need access in your tool to the emails when you’re connecting to Salesforce and Salesforce is already connected to our email? Salesforce is already automatically adding the emails between the prospect and our salespeople to the Salesforce record. Is there a distinct, unique advantage we get by letting you also record the emails? Maybe they’re better at it, who knows? But you have to answer the question. I’m not telling you what the answers are. I’m just saying, ask the question because the vendor is going to ask for admin access to everything. That’s really what they want. Again, it’s harder if you don’t know what your data flows are and you don’t understand your own environment. Here’s what we learned this week helping a customer with ChatGPT Enterprise. They actually did a pretty good job rolling that out, and they immediately came up with a use case where they wanted to connect ChatGPT directly to their Google Workspace. The use case is, “Hey ChatGPT, go in and tell me, of all the documents that are sitting in my drive right now, what are the top three most important things and summarize what I have to do.” I think it’s stupid, but people like that. So immediately, in looking at the threat model and the risks, we talked to the executives and said it’s probably not good to give it access to all the shared drives because the chance of really sensitive stuff being in there is high. Interdepartmental stuff is much higher. But if it’s a “My Drive,” then you could do it. We looked and saw that there’s a self-service option so each user could opt in to the My Drive. That’s how we got here. The executives said they don’t want to give it shared access, but they do want to give it opt-in My Drive access. That sounds great, except you can’t because of the way ChatGPT is working. What we discovered—and it wasn’t me, it was one of their IT guys—is that even in the self-service option, what they’re saying is the account can opt in with self-service. If you click yes, you’re not saying yes to my drive, no to my shared, and yes to my Gmail, but no to whatever else. You are saying ChatGPT can have anything I’m entitled to touch inside the Google Workspace, on or off. That’s not okay because you’re getting shared stuff in there. The alternative is an administrator-managed opt-in, which opts in the entire organization for absolutely everything. Immediately, there’s this giant sucking sound as OpenAI takes absolutely everything in your Google Workspace and tags it. Now they say, “Don’t worry, we can give you role-based access control. Just tell us what users should not be able to see, and we won’t let them see it, even though we have it.”

Chris Swan: Least privilege, anybody?

Nick Selby: [00:08:00] What they’re saying is, “Listen, we understand that we have architected our machine for most privilege, but we’re happy to give you least privilege.” There’s comprehensive documentation on role-based access control (RBAC) in the OpenAI ChatGPT Workspace. The problem is, how many times this week have you heard of ChatGPT in particular, or any AI in general, exceeding the explicit limitations of access the user has given it, often catastrophically? Your only recourse is to yell at it and say, “Didn’t I tell you not to do that?” At which point it says, “Oh, you’re absolutely right. Would you like me to make a PowerPoint or a document about why what I just did was really bad?”

Chris Swan: If there are no financial penalties associated with that, then the incentives are misaligned, aren’t they?

Nick Selby: [00:09:00] Hugh Rogers pointed out a meme from the past couple of weeks where the robot surgeon takes out the wrong liver or the wrong kidney, and then says, “Oh, you’re absolutely right. Would you like me to take out the other one?” Tying this back to tech debt—because this is actually a concern with this other customer—do we know the boundaries and the bright lines we’re drawing organizationally with all these different tools? I don’t know of anyone who’s just getting one integration. Then you just said the terrifying phrase, MCP. You weren’t there at 44CON. Somebody said that the S in MCP stands for security, and it’s totally true.

Chris Swan: [00:09:30] Someone yesterday was telling me that I was a Luddite and wrong when I was drawing comparisons to what was happening with web services and enterprise search 20-odd years ago. And yet I keep seeing the patterns repeating. 20-odd years ago, we did a trial of the Google Search Appliance.

Nick Selby: Yes.

Chris Swan: [00:10:00] It was amazing. All of the things that were impossible to find on the intranet—because we still called it that—were suddenly at our fingertips.

Nick Selby: Oh.

Chris Swan: We finished the trial and then gave the appliance back because it could only work with godlike access to everything. It also needed to phone home to Google; it couldn’t just be sitting in our data center doing its job. Eventually, the risk people said, “On balance, no.”

Nick Selby: [00:10:30] Do you remember Nick Patience at 451? He was doing search stuff back then, and we had a conversation about the Google Appliance. What I’m most fascinated by is that you were able to find things at your fingertips. Nick was saying at the time—I’ll take responsibility through the sands of time if any mistakes here are mine—but what he was saying was it has godlike powers of discovery and it demands godlike powers of connectivity. Where it fell down was that the Google search algorithm, which is basically that which is popular is best, didn’t actually lend itself to an enterprise use case. Often, the most important document you’re looking for is the one that isn’t linked to by a lot of things, like merger or going-public documents, so there are not a lot of links within other documents to get to it. It was both grabbing too much and sending it all to Google, and not giving you good results. But you’re saying the opposite.

Chris Swan: [00:11:30] Firstly, it wasn’t grabbing it all and sending it to Google; the model was on-prem processing, otherwise we would have never even racked it. But it did need to phone home for updates; there was an umbilical cord back to HQ to keep the algorithms fresh. There was some concern that it could have been exploited, especially since this was before the initial Chinese attack on Google’s infrastructure. If someone got into Google’s infrastructure, could they pull stuff across the umbilical cord that we never wanted anyone else to see? The search was really good.

Nick Selby: Oh, wow. Okay.

Chris Swan: It was really good at not showing you stuff you weren’t allowed to see.

Nick Selby: Interesting. Okay.

Chris Swan: [00:12:30] We were able to overlay that role-based access model, so it would know a document existed, but if you weren’t permissioned to see it, it would respect that from the underlying ACLs.

Nick Selby: It respected your ACLs and your RBAC. That’s what OpenAI doesn’t do.

Chris Swan: [00:13:00] That was a really neat thing about it. I couldn’t just go spelunking for the executive compensation spreadsheet, because I wasn’t supposed to see it. Where this potentially fell down was if the executive compensation spreadsheet was set on a SharePoint somewhere, using security by obscurity, and was actually open to everyone—you just had to know where it was.

Nick Selby: Right.

Chris Swan: [00:13:30] Actually, that was one of the things giving people the heebie-jeebies: we have all this power, but have we properly constrained it in our run-of-the-mill permissioning? Previously, you couldn’t find anything on the intranet, so it hadn’t been a problem. This was making stuff findable in a way that it hadn’t been before. The MCP thing also keeps reminding me of the web services thing. I remember 20-odd years ago thinking that web services meant every piece of infrastructure software was going to have a web services API, and we were going to have an explosion of these APIs.

Nick Selby: [00:14:00] Yes.

Chris Swan: How on earth were we going to deal with all of that?

Nick Selby: Mhm.

Chris Swan: [00:14:30] Now, on reflection, every piece of infrastructure software has an API. How does anything work without it?

Nick Selby: Right.

Chris Swan: [00:15:00] That hasn’t given us a massive challenge in managing that API surface area, because it has become a natural part of how things work around our organization. Think about the systems that have those APIs, like Kubernetes—it all fell into that world, and we’ve ended up with platform teams whose role is to ensure that it is properly controlled and managed.

Nick Selby: [00:15:30] If you’re lucky. Again, this brings us back to the tech debt stuff. If you’re not keeping up with it, it sneaks up on you very quickly, and the aggregate effect of not tending your garden properly is massive. What we’ve been talking about are three main areas: first, enumerating the risks and doing threat modeling of what you’re connecting to, which requires that you understand those data flows. What is it going to be touching, and how bad would it be? The second thing is blast radius reduction. What can you change with the permissions you’re giving these tools? We also need to monitor it to be sure. We use Obsidian Security; it is a SIEM for SaaS. It monitors the logs and activity APIs in your SaaS clouds. If you connect Obsidian to your Google Workspace so that you see what’s happening there, like high-volume downloads or a network change in the middle of a session. If something is obviously wrong, and you connect the other side to ChatGPT, you start to get triangulated alerts. We’ve seen data loss prevention (DLP) tools fail. Even when properly tuned to notify us if someone downloads gigabytes of data from Salesforce, they can miss it. You have to test all of these things you’re using as compensating controls to reduce the blast radius. The last thing is instrumentation and alerting. If all that other stuff fails, how will you know before you get a phone call from CISA or the FBI, or someone sells your data online and sends a ransom request.

Chris Swan: [00:17:00] This is another thing: I hear people talking as if anomaly detection suddenly came out of the cupboard yesterday. People were selling me silver bullets with “anomaly detection” carved on them 20-odd years ago. Did it work then? Yes, in demos. Did it work in production? Not all the time.

Nick Selby: [00:17:30] Network Behavior Anomaly Detection (NBAD) was hopeless at a lot of things. It could detect a worm—it could always detect a worm—but the thing it was really good at, which every CTO loved when they saw it, was mapping where all your hosts were. It was the first time you could get a map of your hosts. It was really good at detecting hosts; it just wasn’t good at detecting anomalies.

Chris Swan: That leads us into the tired old conversation about NIDS versus NIPS.

Nick Selby: Yes.

Chris Swan: [00:18:00] The problem is you hit an unacceptable false positive rate. Then you turn up the squelch to get that under control, which leads to an unacceptable false negative rate. You miss the anomalies that should be flagged because you were trying to reduce false positives. Now you’re dealing with false negatives, and the product isn’t doing what you bought it for.

Nick Selby: Mhm.

Chris Swan: [00:18:30] The whole venture has been a massive waste of time. Arguably, after 20-odd years of improvement, this technology has been refined to a point where false positive rates are in a region where it becomes dependable and acceptable. You’re not getting false negatives as a consequence, and you can have a trustworthy, anomaly-based tool. It is still all statistics. The other problem is that attackers can understand the underlying heuristic mechanisms and design their attacks to sneak past the anomaly detection.

Nick Selby: [00:19:30] I was having a conversation yesterday that you just reminded me of. Dave Aitel said that Snort is the best IDS. I asked why, and he said, “Because IDS doesn’t work at all, you need it for your audits, and Snort is free.” I thought, “Okay, that’s actually really good.” To jump back to AI and tech debt, one of our consultants was working with a customer in the defense industrial base in aviation. They wanted to connect their ERP system—which contains spare parts data, some of which is for military purposes—to an AI tool to generate large proposals. The risk is that aggregating this data could result in a classified or sensitive dataset. The tool could compile a proposal for all 458,000 parts in a way no human could. But the challenge was that nobody knew how the ERP system worked because it had expanded organically over the years, and no one knew what was in it. Yet they wanted to hook up this AI tool that they didn’t quite understand. It pulls data out through an MCP, takes it somewhere, and does something with it, without anyone knowing where. Our consultant’s approach was cool: he ran a local LLM on his machines, connected it, and asked it to analyze what was in the ERP system. I thought that was a clever solution.

Chris Swan: [00:21:30] This is one of the ways organizations can deal with data migration concerns in practice. If you can run the tool on your laptop or desktop, that is the right place to do it. If it needs more compute and you have a data center, you can put a few racks of GPUs there to provide an enterprise capability. You end up with a tiered model: as local as possible.

Nick Selby: Yeah.

Chris Swan: [00:22:00] 90% of your workload probably happens on your laptop, 9% in the data center, and a tiny residual fraction goes to frontier model-based services. Because it’s a much lower volume, you can be more surgical about policies around what is allowed, which also opens the door for automated classification. You touched upon a foundational problem: in most cases, we are looking at vast quantities of unstructured data and don’t know what we should care about. There is also the classic data aggregation problem. When Chelsea Manning copied a whole bunch of secret cables onto a CD, each individual cable was classified secret, but in aggregate they created something that fit the description for top secret. That is why a set of documents in aggregate might be treated as having a higher classification level than individual documents. This has been a problem since the switch from paper to IT. As soon as you had a hard disk or a CD full of data, you were immediately in an aggregation problem. A document might be level two, but a whole CD’s worth should probably be at level three or four. We are now using these tools to reason about how documents should be classified, and LLMs are excellent at that.

Nick Selby: Yes.

Chris Swan: [00:24:00] That is not a job where we want to be sending everything over the wire and using expensive tokens. It’s a job we want to do as close to the data as possible and locally, so it’s not going to a third party and exposing us to their risks.

Nick Selby: [00:24:30] There are so many good points there. One thing I want to point out—we may open source this, but right now it’s too ugly—is that we’ve made a tool to help people audit their OpenAI ChatGPT Enterprise accounts using the compliance API. This compliance API is terrifying. I got it running here, and it can show everything the CEO uploaded into ChatGPT over the past six months, what this group did, or what everyone did. We wanted to force ChatGPT not to retain things. You can use this tool to delete everything uploaded since the last request. While they tell you there is a 30-day retention period, with an enterprise account you can keep deleting things to reduce retention risk. If you have an enterprise account, it is a really cool tool.

Chris Swan: Even that makes me scratch my head, because my understanding is OpenAI is subject to retention requirements.

Nick Selby: [00:25:30] They are, but that seems to be mainly for public-facing accounts. If you have an enterprise account, it’s a private instance that is yours. What they have been asked to hold in other cases is their response, which could contain your information, especially if you ask it to rewrite a memo. The specter of OpenAI holding your data for training is also easily avoided with the enterprise account; they won’t use it for training. I was wondering if you knew of tools to help understand how engineering teams are functioning—not just by looking at how many commits they make, but using AI to figure out how effectively and how often they are working.

Chris Swan: Development team phrenology.

Nick Selby: Exactly. I’m not suggesting we use that, but—

Chris Swan: [00:26:30] I’m not familiar with any of that because it immediately crosses an ethical boundary for me.

Nick Selby: Yes.

Chris Swan: If we look at the established research about making highly productive teams,

Nick Selby: Mhm.

Chris Swan: [00:27:00] It’s not about trying to measure individual productivity. Even measurements of team productivity are somewhat wishy-washy. I had the pleasure of meeting one of the researchers from DORA a couple of weeks back. DORA is no longer an acronym; the “DevOps” has gone.

Nick Selby: Right.

Chris Swan: [00:27:30] They have moved from a world of DevOps to the post-DevOps world of AI. This year they’ve been focusing on research showing that AI coding assistants give individuals the sense that they are improving productivity, but give teams the sense that they are making things worse. I’ve seen this up close recently, where we are getting what I call sloppy PRs. It is easy for someone to think, “I don’t know how to do that, but the coding assistant will help me do it.” They make it work, submit it to their colleagues for review, and the colleagues reviewing it ask why something that should be in just a few files is changing code across 300 different places.

Nick Selby: [00:28:30] Unwittingly, you’re answering my next question. You and I have talked about how we do tech debt surveys. The last one we did took two weeks; we got back 16 textbooks’ worth of responses and had to go through it all manually. I’m wondering if you know of any AI tools designed to look for evidence of tech debt, beyond just vulnerable code or infrastructure vulnerabilities. How would they even start, and have you heard of anyone doing this?

Chris Swan: Not within that bounded context, though that sounds like a startup idea.

Nick Selby: Somebody already got funding for it.

Chris Swan: [00:29:00] Coding assistants are being pitched to help understand your codebase in a way you might not be able to, perhaps because the people who wrote that CICS COBOL retired. That’s a common issue in the industry. It’s not necessarily couched in the language of tech debt, but it is about gaining understanding. The practical issue is that a lot of that older code doesn’t have tests.

Nick Selby: Yeah.

Chris Swan: [00:29:30] It’s not just understanding what it is supposed to do; you also don’t have any way of verifying that changes won’t break existing behaviors. One of the promises is that we can construct tests for existing behaviors, but that is still a bit hand-wavy. All of this technology will get better. I sometimes find myself cast as a skeptic, but I know these tools are already useful and will improve with successive iterations. We are in a bubble, but on the other side of it, there is clear utility we will exploit. People are talking about whether we have exhausted what we can do with LLMs. Once we have trained frontier models on everything we can scrape, that might be the limit. We may not be able to go further with that approach. AI research has other realms we haven’t systematized yet. We have industrialized our ability to do the underlying vector math, so can we apply that in other areas? There is still a lot to be hopeful for.

Nick Selby: We’re going to leave it there. This has been the Tech Debt Burndown podcast. I’m Nick Selby.

Chris Swan: And I’m Chris Swan.

Nick Selby: [00:31:30] Thanks for joining.

Recording date: Nov, 2024

Download at Apple Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Chris and Nick dive into the widespread misunderstandings and economic realities of choosing between cloud and on-premises infrastructure. They discuss how cloud service providers have driven organizations to perform “lift-and-shift” migrations that replicate messy data center environments in the cloud. Rather than achieving the promised cost savings and agility, these organizations inherit the same technical debt while facing skyrocketing bills and unexpected administrative costs in areas like logging, compliance, and observability. They also highlight that while the cloud is invaluable for early-stage startups and rapid feature deployment, it can quickly become cost-prohibitive without proper optimization, architecture, and governance.

To combat these challenges, Chris and Nick outline the growing trend of cloud repatriation, drawing an analogy to managing an energy grid with a steady “base load” and spiky peak loads. They suggest that organizations are increasingly finding it more economical to run their predictable base loads on-premises, leveraging hyperconverged infrastructure or modern private cloud platforms like Oxide. The public cloud can then be better matched to highly variable workloads. They also explore how Kubernetes has served as a key driver for this workload portability, and note the unique exception of modern AI workloads, where hyperscalers dominate the supply chain for high-end GPUs like, forcing companies to stay in the cloud to access cutting-edge hardware.

Enshittification

Oxide

Gomboc

Nick Selby: It’s the Tech Debt Burndown podcast. I’m Nick Selby.

Chris Swan: And I’m Chris Swan.

Nick Selby: [00:00:30] Chris and I have been talking for some time. The first conversation that actually led to this podcast was when we both realized that we were working on exactly the same kind of thing, where we came into an organization and had to look at a massive tech debt and start to address it. That inevitably led to the conversation about the cloud being misunderstood, with executives believing that because they’ve got their AWS password and signed up for it, their agility should be here any minute now. This idea that a cloud account of some flavor is an automatic step up or step down, depending on who you’re talking to. Maybe about a week ago, we saw a tweet that has since been deleted saying, ‘I’ve known for years that the cloud was much more expensive and much less secure and much more troublesome and much less flexible than data centers and on-prem.’ We thought we would talk a little bit about what are the expectations that people have when they are dealing with cloud, and how does tech debt feed into that? Did I capture that right, Chris?

Chris Swan: [00:01:30] I think so. And I pick up immediately on the cloud being misunderstood. The cloud has been misunderstood because it has been deliberately misrepresented by the cloud service providers. If people only use the cloud for what it is meant for, that would not have driven the surge in revenue that’s taken place over the last decade and a half or whatever it’s been. They had their reasons. But I think we can have a conversation now to start cutting through that and understanding the respective trade-offs between cloud and on-prem, and also where people are starting with this stuff.

Nick Selby: [00:02:00] Yeah, I agree with you. A lot of it was just in how the cloud was marketing itself. In the beginning we saw lots of arguments about what’s safer. ‘No, I’m not going to do that,’ said Fortune 500 CEOs, ’that seems too risky.’ And all the pundits were saying, ‘No, you’ve got to keep control of things.’ Over time, it reversed. By 2013, 2014, it was, ‘Oh, this is safe because it’s in the cloud.’ ‘We don’t want to manage that on-prem because we don’t have the resources,’ which is a really successful marketing campaign, if you think about it. To your point about using the cloud for what it’s meant for, I would think that with the skyrocketing revenues overall in cloud service providers, a healthy percentage of their top-line revenues comes from people who actually don’t understand what they’re doing. They’re using default settings or they’re doing a lift and shift, and recreating their data center environment in the cloud as opposed to taking advantage of cloud-native architectures, capacity, and capability in order to get what they need in a different way. They just figure, ‘We’ll fire our data center, cancel all those leases, and put it up in the cloud, and everything will be cheaper.’ And it’s not.

Chris Swan: [00:03:30] Absolutely not. First of all, I should probably state my view on the purpose of the cloud. I think the purpose of the cloud is very simple: to be able to move fast, with the ability to get services on demand. As a startup, you can literally show up with a credit card. You can subscribe to some free credits even at the beginning and get started on something, and to be able to immediately deliver is enormously valuable. You just mentioned lift and shift, and that’s almost the opposite of that. That’s all predicated on a much more complex calculus of the costs associated with running traditional infrastructure on-premises. For a time, the cloud was tracking Moore’s Law in getting cheaper and cheaper. You could imagine an intersect point where you say, ‘I’m spending a bundle running data centers and it’s undifferentiated heavy lifting. Why don’t I just hand that all off to somebody else who’ll take care of it for me?’ People had already been doing this for a very long time before the cloud came along, because we had outsourced service providers. I used to work for CSC, which was one of the good old names. There was EDS and whatever else, and they did massive contracts with companies to take over the running of their data centers. That very often got the label of ‘your mess for less.’

Nick Selby: [00:05:00] Yeah.

Chris Swan: [00:05:30] One of the things I’ve seen happen in the past decade is that a lot of that infrastructure outsourcing has moved from a relationship with an IT services provider to a relationship with a cloud services provider. They’ve swallowed up a lot of that business and cloudified it. But has there been any value added? I would say that’s probably questionable. We’re now at a point where cloud has basically been around for the best part of two decades, and we can look at that first decade as the growing up period. There were lots of issues around security, scalability, data location, and other things people cared about that got taken care of in that first decade. Another thing that marked out that first decade was that everything kept getting cheaper.

Nick Selby: Yes.

Chris Swan: [00:06:30] If we think about the initial launch of EC2, it was a pretty rubbish VM, but it was instantly available and only $0.10 an hour. That was actually quite pricey for what it was at the time, but it was also compelling because if you were just getting started, $0.10 an hour was an easy entry price, and that $0.10 came down really rapidly. The equivalent or even better capability was an order of magnitude cheaper. We could extrapolate that cloud was going to keep getting cheaper, but what we’ve learned in the second decade is that it hasn’t.

Nick Selby: Yeah.

Chris Swan: [00:07:00] Arguably this is now following Cory Doctorow’s enshittification cycle. First of all, there’s a surplus of value delivered to the customer. That was very visibly happening in that first decade, and now everything is being taken back to the provider.

Nick Selby: [00:07:30] But if we stick on the cost for a second, in 2013, I set up a company where we had to make investments of about $1.1 million to get what we needed. This February, we set up another company with about the same number of employees and about the same mission. Including MacBook Pros for everyone, it was about $55,000 to get started. There are really compelling cost arguments that can still be made without devolving into the old conversation about CapEx and OpEx. But you’re absolutely right: it’s less now about the cost to get in, and more about what you do once you’re there. People get in and get really excited, but then they see those costs zooming up at a jaunty clip. The natural executive conclusion is, ‘Wow, the cloud really is expensive and terrible.’ They’re not wrong, but they could have a completely different experience had they been managing what they had a little bit differently.

Chris Swan: [00:08:30] I remember early on doing some calculations of cloud versus on-prem for the organization I was working for at the time, and we had a pretty massive data center footprint. What did we learn from that? IT infrastructure is expensive, but we were running it at substantially less than what the cloud could offer for similar capacity. I think some of that along the way has fallen prey to arguments that there are hidden costs you didn’t take into account.

Nick Selby: Yes.

Chris Swan: [00:09:00] Or they promised discounts. Those discounts were supposed to make sure that the cloud is going to be cheaper than your on-prem. What we then see of late is that there has definitely been cloud repatriation happening.

Nick Selby: Yes.

Chris Swan: [00:09:30] Organizations are realizing those cloud bills are much more than they bargained for and saying, ‘Let’s take some of this back.’ I think some of that has actually been done in a pretty thoughtful way as well. If we do an energy equivalent here, most organizations are going to have what we could refer to as a base load.

Nick Selby: Yeah.

Chris Swan: [00:10:00] In energy, you might want that base load to be nuclear or something like that, and then there are going to be spiky loads on top of that. Organizations are looking at it and saying they can run that base load for less on-prem, especially if they invest in cloudy-ish modern infrastructure. Whether that’s hyperconverged infrastructure with VMware, or something like Oxide and their rack-scale cloud units, there are plenty of good options now to have an infrastructure-as-a-service-like experience within the context of a private data center. Once you’ve got that base load in, you can look at the peaky bits and say that’s the stuff that belongs in the cloud, because it fits the on-demand model.

Nick Selby: [00:11:00] On the other hand, the reason you and I decided to do this episode was because of two comments we heard. One was this idea of the base load. I talked to a CFO whose cost to do the same general thing in the cloud was 14 times higher than in his data center per unit, and it really was. But his financial operations lead said he could cut that in half tomorrow by rebalancing workloads. People just run servers at 3% capacity and others at 97% capacity, and they haven’t optimized to make sure costs come down. There is a sense that the core is the core, but it isn’t, not with the way you set this up, commit to it, architect, and plan for it as a business. I get the feeling people are still planning based on marketing: just put it in the cloud and it will be fine. Which is music to the ears of Amazon, GCP, and Azure. That was the first comment. The other comment was about margins. People say, ‘Look at the margins we get,’ but they’re not comparing apples to apples. They point to data center margins being extremely high, but they weren’t doing any of the fancy AI workloads that they did in the cloud. They couldn’t, because none of those data centers have GPUs or the kind of hardware they would need. So it isn’t an apples-to-apples comparison.

Chris Swan: [00:12:30] Yeah. And that sparks a couple of thoughts. Especially when it comes to GPUs, the picture for the last couple of years has been the hyperscalers making enormous investments in building out capacity. Being first in line allowed them to get delivery. There are certain classes of GPU where, in practice, it doesn’t matter what your budget is, you can’t get them. If you want Blackwell or whatever is the latest and greatest, you essentially have to go to a hyperscaler, because no amount of money will buy them for your own data center. Even if you could buy them for your own data center, you then run into traditional racking, stacking, caring, and feeding, which can be a pricey endeavor.

Nick Selby: Yes.

Chris Swan: [00:13:30] I think the GPUs are also interesting because we’re no longer in the business of trying to lash these systems together yourselves. They come prepackaged as a rack-scale deliverable. With all of the interconnect and everything else figured out, optimized, and not having to be designed and implemented on a case-by-case basis, that has leveled the playing field. But I also think this argument that came in the tweet—that on-prem is always going to beat cloud on economics—is absolutely untrue at the entry level. There is a very high bar of needing to ‘be this tall to ride.’ I live this every day. For the last few years, we have not gotten to the stage yet where we could buy even a single rack-scale unit and drop it into a colocation facility.

Nick Selby: Do we save a lot of money there?

Chris Swan: [00:14:30] Your cloud bill needs to be in the order of millions a year. Before this becomes a relevant conversation. That implies a certain scale of organization.

Nick Selby: [00:15:00] It does. But the funny thing is that even when your cloud bill is in the millions of dollars a year, there are so many tools out there. Wiz is so popular, and there are other things like Gomboc. I mentioned Gomboc because we had Ian on in the first season of this podcast. That’s his company now. He spent time at AWS and realized that doing the architecture, usage, and configuration is the biggest challenge to getting the cost savings and the value you want. The fact that these industries exist means people aren’t doing the hard thinking; they’re still looking for the easy button. Anyway, that’s my theory.

Chris Swan: [00:15:30] You touch upon an argument I’ve heard. Organizations have displaced their ‘undifferentiated heavy lifting’ with ‘undifferentiated heavy clouding.’ The admin aspects of this don’t just happen for free. There might be hidden costs in running your own infrastructure, but there are also hidden costs in adopting cloud services. Cloud services don’t just run themselves, appear from nowhere, and conform to best practice guides for architecture. There are many things that need to be arranged around logging, observability, auditability, and compliance that need to be designed, rolled out, and operated on a day-to-day basis. That is often where the eye-wateringly expensive, premium services from cloud providers come into play. People start saying, ‘Whoa, how much? I thought we were on the hook for X, and it turns out we are actually spending 4X.’ What are we doing with all of these logs, how much are we retaining, and how much are they charging us for that?

Nick Selby: [00:17:00] We have some time left to talk about the difference between a blind lift and shift, and planning for efficiency. You’ve had your fast-cheap-good argument here. Just because it isn’t cheap, but it is fast and good, doesn’t mean it’s not cheaper. There are ways to make it cheaper than it would be otherwise. But then you open up that bill one morning and see that you probably shouldn’t have done what you did last month, retroactively.

Chris Swan: [00:18:00] I’ve heard this argument from people like Adrian Cockcroft in the context of his time at Netflix. They thoroughly understood that the purpose of the cloud was to enable the organization to move fast. They were pivoted around that notion of how engineering should work. They realized that if they had been going slower, there would have been costs accrued to going slower. On that basis, it turns out the cloud is not just better and faster, but also overall cheaper. That’s a sound argument for that organization at that point in time. But it doesn’t necessarily extrapolate to all other organizations, especially given that very few have accomplished the leverage with their cloud service provider that Netflix was able to do, because Netflix was so early, visible, and influential in that debate.

Nick Selby: Macy’s will always pay less per square foot than your mom-and-pop shop in the local shopping mall, just because they’re the anchor tenant and they’re there to bring other people in. That’s a different set of economics they’re dealing with.

Chris Swan: [00:19:00] Absolutely. With this better, faster, cheaper trade-off, A lot of cloud was sold on the basis of being cheaper rather than faster, and I’m putting ‘better’ out there now as assumed. You mentioned earlier cloud-native architectures. It’s funny how that term has come to represent two similar but entirely different things. In the red corner, we have actual cloud services like EC2 or Lambda. In the blue corner, we have Kubernetes. So we’ve ended up with a bunch of people running Kubernetes in the cloud, but also a bunch of people running Kubernetes on-premises. That has enabled people to take on workload optimization and capacity balancing. They’ve moved to using a platform and got themselves a platform engineering team, and they’re maybe even doing some team topologies work. Kubernetes is one of those things that has made workloads somewhat portable from a public cloud environment to on-premises. It enabled this private cloud thing to come true in a way that hadn’t happened so comprehensively with the more VM-based approach to infrastructure as a service, whether public or private.

Nick Selby: So what happened to the stuff that didn’t make it onto Kubernetes?

Chris Swan: [00:21:00] Well, that brings us back to the ‘your mess for less’ type of thing, which was going on before the cloud and has been scooped up by the clouds. I think that’s partly why we’re here on the Tech Debt Burndown: the problem with your mess for less is it’s still a mess, and there are all sorts of horrors lurking there. Typically, not much has been achieved from a cost-saving perspective by moving your mess from your own data center to one managed by a third party. We can extrapolate that precious little has been accomplished by moving your mess to a cloud service provider doing a lift and shift. There might have been a little consideration given to VM sizing and workload profiles as that has taken place, but in terms of the cost of ownership of those applications, the features of the mess are still there.

Nick Selby: [00:22:00] Yeah, that was where I was going. It’s indiscriminate migration from one thing to another. I’ve got this terribly functioning thing over here, but it’s cheap because it’s here. Then I’m going to put it into the cloud and expect it to somehow magically work. That was what I meant at the beginning: ‘Here’s my AWS password, I’ve got agility now.’ But you don’t; you’ve got a new place to run the same stuff. If you don’t take care, your bad decisions and tech debt will follow you into whatever platform you’re on. I said at RSA a few years ago that when you make a mistake in the cloud, you are stupid at cloud speed, but you will get to the bottom of what you’ve done wrong very quickly.

Chris Swan: [00:22:30] What we see is a repetition of the fables of the past. When managed service providers were taking on the mess and doing deals for ‘your mess for less,’ that was often predicated on being able to transform the mess into something less messy and run it at a lower cost.

Nick Selby: Exactly.

Chris Swan: [00:23:00] And deliver savings based upon that. A lot of the time that simply didn’t happen. We’ve seen a repetition of that with lift and shift, where there are desires and notions of, ‘We’re going to lift and shift now and transform later.’

Nick Selby: It’s always easier to do it later after you’ve finished. Yeah, that’s what we always recommend.

Chris Swan: [00:23:30] This elite team of ninjas deployed as solution architects to take care of your cloud migration—lift and shift being a big chunk of that. Once they’re done with the lift and shift, they are supposedly going to be available to proceed with the transformation.

Nick Selby: Yeah.

Chris Swan: [00:24:00] How many of those have actually happened? I would say a precious few that I’ve seen. I think there is a ‘sands of time’ thing here where new applications come along and reside on new platforms. Old applications sometimes eventually wither on the vine and die out, or are replaced by new applications or services. If people are consuming SaaS, there’s still an awful lot of legacy, which is often core tech sitting there like a psychopath running the business without necessarily needing much messing around with it.

Nick Selby: Right.

Chris Swan: [00:24:30] The costs associated with messing around with it for the sake of doing a transformation can be quite extreme. That’s another reason why it often hasn’t happened: there was a business case in favor of the cloud service provider to do the lift and shift, but there isn’t a business case for the transformation.

Nick Selby: [00:25:00] You’ve done that. You’re in the cloud. What is the good behavior we’re trying to encourage? I’ve seen situations where something was poorly deployed, or badly imagined and envisioned. You look at all the problems you’d have in redoing it, and oftentimes we end up rebuilding it and then shifting over, running A/B for a while. In the worst cases, I’ve seen people fall for the siren song of, ‘Google Cloud defaults are better and more secure than AWS, so let’s rebuild it over there.’ They do it with the same lack of competence they had with AWS. Not only will it not work, but all the problems will be different, so they don’t even benefit from the expertise they gained from why it was bad on AWS. What’s the best way to approach this? Is it to rebuild next to each other like the fishing bridges in Florida, where once you’re done building the new bridge, you redirect the traffic and turn the old one into a fishing bridge? What do you do?

Chris Swan: [00:26:00] I think you’ve touched upon security, and that’s going to need another episode.

Nick Selby: Okay.

Chris Swan: You’re also touching upon the strangler pattern: creating new functionality using microservices and then ultimately strangling off the old functionality. That’s a good pattern for when we are trying to modernize things and there is a good business case to do that.

Nick Selby: [00:26:30] That’s fair enough. We will think about doing a second episode on this. This has been the Tech Debt Burndown podcast. I’m Nick Selby.

Chris Swan: And I’m Chris Swan.

Nick Selby: Thanks for joining us.

Recording date: Nov 15, 2024

Download at Apple Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Sarah Wells joins Nick and Chris to talk about the lessons learned from her decade at the Financial Times, and their journey to implementing microservices. The same lessons that informed her book, ‘Enabling Microservices Success’. We touch on whether the purpose of microservices is to scale a system, or to scale an engineering organisation. By breaking down monoliths into independent domains, teams can reduce cognitive complexity and release software hundreds of times a day. Sarah warns however that this shift introduces a “maintenance treadmill”, requiring robust automation for library upgrades, security patching, and cross-service governance to prevent a sprawl of unmanageable tech debt.

The conversation goes deep on human factors, and the role of organizational culture in technical success. Sarah emphasises the importance of a ‘generative culture’, as defined by Ron Westrum, where information is shared openly. By taking a ‘you build it, you run it’ approach supported by a strong platform team, organizations can balance individual team autonomy with the necessary guardrails for long-term operational health and efficiency.

Enabling Microservice Success

Generative organizational culture (DORA) [Westrum model]

Nick: It’s the Tech Debt Burndown podcast. I’m Nick Selby.

Chris: And I’m Chris Swan, and we’re here today with Sarah Wells. Sarah, could you please introduce yourself?

Sarah: Hi. I’m an independent consultant and also I recently wrote a book about ‘Enabling Microservice Success’ based on my experience working at the Financial Times for over a decade, while we went from very monolithic architecture where we delivered releases in production 12 to 20 times a year, to being able to do hundreds of releases a day with a microservice architecture, with a very large number of microservices.

Nick: [00:00:30] Chris and I are both lucky enough to be able to work with you in different areas. We’ve done some consulting together, you and I, Sarah.

Chris: So I’ve worked with Sarah on numerous QCon events. Most recently, we’ve both been on the program committee for QCon London. And so we also get to sort of bump into each other at those events and chat about what’s happening in the tech world.

Nick: Yeah, I did want to kick off just to talk about the book. The book isn’t about microservices. Sarah, do you want to say what the book’s about?

Sarah: [00:01:00] So it’s in the context of microservices, but really when you start saying, how can you be successful with microservices? You’re talking about how to be successful with modern software development. How do you approach building something that you can maintain and run for a long period of time, and not have to restart and not spend a huge amount of your time doing maintenance? I think you could read it and find there’s lots of useful stuff there, even if you’ve got a fairly monolithic application. And I also make the argument that no one really has a monolith anymore. Everyone’s making calls over the network for some part of their system. It’s very rarely one single application. Even when the FT had a monolithic architecture, we had multiple applications and they spoke to each other. It’s just how distributed is it and how independently can you deploy change to different parts of it?

Chris: [00:02:00] I think that also touches on one of the things I obsess about a little sometimes, which is even if you’re not making a single network call, you’ve probably got a distributed system. If you’re using anything modern because of multi-core processors and multi-threaded applications. And so distributed is happening at a much finer grained level there as well. But sometimes we’re just needing to start introducing that network call when it’s kind of necessary from a scaling the system point of view.

Sarah: Yeah. And also quite often you have a monolithic application, but you have a database and maybe that’s not on the same machine. And so you’re already having to deal with the network at that point. Or you’ve got two instances.

Chris: [00:02:30] I think in some ways, the title of the book frames things in the context of microservices being a recognized good thing that you would want to achieve. And some of this conversation was cued up by the fact that you’ve been speaking to people where it’s, “Oh, we’re not ready for microservices,” or “We don’t want to do that yet.” Where does that take you and where does tech debt come into this picture?

Sarah: [00:03:00] Interestingly, both my book and Sam Newman’s book on microservices, there’s a chapter quite early on, and it’s all of the cases where you shouldn’t be thinking about doing this, where this would be a mistake because there are lots of situations where a microservice totally doesn’t make sense. They’re usually a solution to an organizational scaling problem. They’re normally something that you feel the need of when you have lots of people in multiple teams trying to make changes to the same codebase. If you can split those out into separate domains and have teams work on an individual domain, you are able to make progress more quickly. If you’re a small company or a startup and you have one team, there probably aren’t as many reasons why you’d want to do microservices. You might wait until you’re feeling that pain. You also don’t really necessarily understand the domain that you have when you’re first starting to work on it, so you’re unlikely to find those right boundaries, divisions between services, which means you might find that they actually aren’t as independent as you think. There are a lot of reasons to be cautious before you adopt microservices. They have to be something that you can see how they’re going to help you. And that is normally because you are slowing down and being able to build new features because you’re constantly waiting for some other team to finish using the shared integration testing environment, or you’re having to hand something over to somebody else to do something. And whenever you do that, it slows things down.

Chris: You touch upon, for me, the most important misunderstanding of microservices, which is I encounter far too many people who seem to think that the purpose is to scale your system versus the purpose is to scale your engineering organization. I know at one point Adrian Cockcroft, who some see as the father of microservices, will step up and wave his hands and draw attention to this point, but it also feels that that is oft ignored.

Sarah: [00:04:30] I think the problem is when you are a really massive company, microservices do help you with scaling your system. But for a lot of people, for something like the Financial Times, we’re not dealing with huge amounts of data. We have multiple instances in the Financial Times—have multiple instances of services for reliability reasons, not because you can’t fit all of the content the FT ever published on your laptop. I mean, you can; that’s not why you would scale. There is another reason for using microservices because you have scaling issues and you want to scale some part of your system in a different way than others. But it’s not for most people. The reason that you would do it really is around being able to reduce the complexity of what you need to understand. You can work on a microservice, a part of the domain in a microservice architecture, and not have to worry about what’s going on elsewhere. It’s a bit scary at first because you’re thinking there’s this whole context for the systems in our organization, and I have no idea what’s happening over there. Well, that’s fine, because there’s very little chance that they’re going to do something that’s going to impact your ability to keep working on your stuff.

Nick: [00:06:00] In the pre-chat, you had said maybe the title’s not good. I disagree; I think it’s actually a great title. I just think that people don’t take the time to read the title, which is a bit of a problem, right? But ‘Enabling Microservices Success’ is different from enabling microservices. So part of the reason that we thought of talking to you today was, I’ve had the chance to observe your work. And what I like is that there’s a lot of similarities between the stuff that Chris and I do and talk about all the time, which is, all right, where are we now? What do we have that works? What do we have that doesn’t work? How do we know the difference? And it’s this sort of bottom-up approach with a top-down prioritization, but a bottom-up approach to see what we have. And in that, the concept of tech debt versus process debt versus just bad practice and other kinds of things, what do you intend to do? What’s your desired outcome, but how are you going about it? And looking at the way those things don’t quite mesh is something that we’ve done quite a bit of talking about. Where do you start? How would you describe what you do and the process by which if you have a small or midsize company, where do you even start?

Sarah: [00:07:00] I’m always interested in how easy it is for someone to make a small change and release it to production. Because I saw the transformation. When you don’t have to remember what you were doing six weeks ago, at the point you wrote this code that’s been packaged with a load of other stuff and it’s all gone live, and now you’re trying to work out, well, what was I thinking? And which of these changes made a difference? So I’m normally thinking, if you can have a new idea and code it and release it without having to wait a long time, then you’re in a good place, whatever your architecture is. But if you haven’t got that, how could you get there? And is it the easiest way to do it, to start trying to take some part of your system and separate it out, maybe as a microservice? That’s an important aspect for me. The other thing that I think is really important is thinking, how are the decisions we’re making now going to impact us in 3 or 4 years? Because it’s very easy to make progress to start with. But one of the challenges with microservices is that you can feel like it’s a bit of a treadmill because you’re constantly upgrading things.

Sarah: [00:08:00] Part of that is just because you have more services. So let’s say that you need to upgrade a library that you depend on. Maybe because there’s a security vulnerability. You don’t do it once you do it a hundred times, you have to make it really automated and very quick to do it because a hundred times five minutes is a lot better than 100 times two hours. When you’re in a monolith, you can afford to spend half a day to do that upgrade because you’re doing it once. But with microservices, you have to build things in that allow you to know where the thing is and fix it. There’s a whole range of things like that around maintenance. Microservices allow you to have different kinds of data stores, and that’s brilliant because it is a lot easier, for example, to build a system that manages metadata. If you use a graph database underneath it, because metadata is a graph; people are linked to, companies are linked to industries. There’s a whole graph there. It’s much harder to do that in a relational database. That made it much quicker for us to create new features for metadata when I was at the FT, but now you’ve got an additional database that has to be upgraded.

Sarah: [00:09:00] You have to be able to do backup and restore. And suddenly there’s a whole set of additional things that you have to do. But one of the things about the original idea with microservices was it’s amazing. You can completely choose a whole bunch of different languages, and you’ve got complete freedom to do stuff. And it’s fine because you build it in Clojure. And when you don’t have anyone that knows Clojure, you can rebuild it in a very short period of time. Well, first of all, it’s very hard to persuade non-tech people that actually the thing to do is to completely rebuild a system that’s already there, even when it’s microservices. They’re asking, “Why would you do that?” And the second thing is you just end up with something that’s really hard to support. You don’t want five different languages. You don’t want a huge number of things in complete freedom because you can’t easily move people between teams. You can’t move services between teams. You can’t build tools that can help support everyone because you now have to build them in five languages. There’s a definite balance. What do you have to do that keeps it manageable?

Chris: What did you typically see as the relationship between service boundaries and team boundaries?

Sarah: [00:10:00] Well, at the Financial Times, we had a large number of microservices because we were early adopters and we believed this idea of them being smaller and having a single responsibility. You can make that work. What you don’t want is to ever have a service that’s owned by more than one team. You could have each team has a microservice, and then if they find it makes sense to split that up into other services, that’s great. There might be reasons for them to do that, and they might come from some of the other things that work with microservices. They’re talking to different databases, or it just helps us to be able to consider them separately, but never more than one team working on a particular service is a really good way of trying to keep people able to make progress.

Chris: [00:10:30] Did you encounter situations where there were sort of cross-cutting concerns for new products or launches, which were forcing the organization to address multiple microservices all at once? And as a background to this, I was once talking to Ranney, who was at Uber at the time, and they’d been an adopter of microservices. They were pretty happy with how that worked for them, except for. He said, when we did Uber Eats and when we did Uber Truck, we had to touch everything all at once, and it felt as if we were teleported back to the bad old days of doing monolithic releases.

Sarah: [00:11:00] Sometimes you’re making changes where you know you have to make changes in multiple services. That happened to me far more for the services that existed within a particular group within the FT. So maybe the content API and platform had multiple teams working on it. There was more likely to be something that affected different areas there. And we work fairly closely together. Anyway, after we had learned this, we were really careful about the information that we sent between systems and having services only look for the fields. It was JSON over HTTP; services would ignore anything they didn’t understand. That made it easier to at least work out an order that you were going to do things. You can add the new field and all the services that don’t understand it will ignore it. And you can then start updating those services that need to use that new field; that’s not as big as introducing Uber Eats, but at least it means you don’t have to release everything together, which is a nightmare. Occasionally you had something that crossed the boundaries between, say, the content publishing team and the web team, and that needed a bit more care and communication.

Sarah: [00:12:00] The biggest thing actually, I found with having to touch every microservice is when you’re doing big architectural changes. We wanted to move from, believe it or not, our own hand-created container orchestration system because we adopted containers really early. We wanted to move to Kubernetes. Well, that means moving every service to be running in a different way. And that was a challenge because we had six months to a year where we were maintaining two production environments while we migrated services into Kubernetes. And you’ve got to work out: how do you update? Do you update both? Do you deploy everything to two locations? How do you make sure that they’re working in both if only one of them is serving production traffic? That required a lot of careful planning and trial and error to try and work out: do we feel comfortable that we understand what’s happening here?

Chris: [00:13:00] You’ve touched upon a couple of things that I guess are governance topics. Earlier, you said it was scary being able to just get stuck in working on something within a service context and not have to understand the whole picture. But you’re also touching upon a few things here where somebody does have to understand the composition picture and maybe make choices about using these languages and frameworks and not just have a free-for-all. And so who does that? And how does the organization provide a context for that governance?

Sarah: [00:13:30] So in the end, I started off doing microservices. I was in a team that was building them and I ended up moving across to be that person, that team, the one that’s trying to provide that structure because it’s such an interesting challenge. And there are two things. The first thing is to say that normally you should use the standard ways of doing things. This idea of paving the road or the golden path, you say, well, okay, we will provide standard ways of doing things, and there has to be a good reason for you to opt out of that, because we know that there’s a cost to it, and there are certain things where you just end up saying, I’m sorry, but all of the logs need to go to our log aggregation system. You don’t get to decide to do logs differently, because it’s such a challenge to have some part of an event in a different logging tool. You say, here is some common stuff, and if you do go and do something yourself, here’s a set of our expectations. Maybe you call them guardrails, but they’re a checklist saying, if you are going to do this, if you’re going to build your own version of this, it does have to have security scanning in place.

Sarah: [00:14:30] You do have to keep it patched and upgraded. Here’s our patching policy. You have to make sure that the logs go somewhere. They have to be metrics. So you’re providing people with a guideline that you could do your own thing, but it has to meet quality standards. The other thing you start to do is think about how you have those conversations about introducing a new programming language. And we did it through a forum where if you were making a decision that had wider impact than your own team or small group of teams, you should come to the forum and make sure people are aware of it. It was an RFP process where you write a couple of pages to say, here’s the challenge we’ve got, here’s what we’ve considered, and here’s what we want to do. And the rule was you had to go and make sure you had the conversations with everyone to start to get through all of the disagreements before the meeting. So the meeting was very much that we’ve already got broad agreement and are doing the final bits, but it made people go and have that conversation.

Sarah: [00:15:30] Sometimes people would not do that, and I’d find out that there’d been something introduced that had a massive impact. But generally you had the opportunity to say, now I understand why you’re saying we should introduce this new thing because you have some needs not met by the existing one, and we’ve balanced it up. As an owner of the platform engineering team, I always wanted to know: what’s your plan? Is this always going to be maintained by your team, or is it eventually going to be handed over to the platform engineering team? Because very often, particularly a couple of years in, where the people who made the decision have moved on and now they’ve got a team that are having to manage something they’re not very excited about, they just want to get the platform engineering team to take it over. I was trying really hard to say that it’s not a given that my team are going to accept whatever it is that you built. It may not be something we want to make as part of our common paved road.

Chris: And did that come about because of operational concerns? It sounds as if you’re starting with “you build it, you run it,” but that some of the “you run it” was being shared to a platform team, and there would be different expectations around the running of the platform.

Sarah: [00:16:30] There’s a couple of things that I want to unpack from that. I love “you build it, you run it.” I think you should run the thing that you build. If you make a different decision architecturally or how you build stuff, you know you’re the person that has to debug it and fix it at two in the morning. But I think there’s an element where you realize that some of the things that go wrong are actually underneath the application, and you do need a platform team to be supporting those sorts of things. There’s no point phoning a team to say their application is not available if it’s actually because there’s a problem with the CDN or there’s a problem with the Amazon region. I do think there is an element of you have to have support. It’s also really tough to expect every team to have knowledge and expertise in every area. If you do “you build it, you run it” and you don’t have anything central, you’re relying on every team understanding what’s needed for security and doing it. They’re doing that while balancing it against a push to deliver new features. You ought to be aware of how many security vulnerabilities are still in place in all of your software estate, because there’s probably a team that’s a bit overwhelmed that’s not quite got around to it. There is an operational side of things. I think as an organization there are organizational concerns that you really care about, where you ought to be able to have someone say, “Yeah, we’re doing okay cost-wise.” For example, you want to be efficient in your use of cloud resources, but each individual developer that’s spinning up a server to host the thing they’re building wants it not to run out of CPU or memory, so they’re going to overprovision. You’ve got these two tensions and somehow you have to work it out together and make sure that you rightsize everything.

Nick: [00:18:30] In the past four months, I think that if there’s one single thing I’ve heard you say the most, it’s that sentence about “if you build it, you run it.” I’m really impressed. Your primary concern in every meeting that I’ve been in with you has been around the human factor much more so. I think it’s been surprising to some of the engineers that that is your first concern—the organizational constraints. Can you talk about that? It’s what Chris was asking, but it’s also a little different because nothing works unless the humans do. And you’ve been really good at that.

Sarah: [00:19:00] You have to get the interactions and the culture and build those relationships within teams, because the number of times where there’s a problem and you realize it’s because the two teams just don’t have any natural way to talk to each other, they haven’t quite realized that both people are doing the right thing for their constraints and their context. Having that understanding is really important. We ran an internal tech conference at the FT and it really helped to have everybody talking about what they were doing. The feedback after the very first one we did was, “Oh, I didn’t understand all the legacy that that team was dealing with, and that’s why they couldn’t move as quickly as we could.” I think that culturally, there’s a huge aspect of organizational culture—if you’ve not got an open learning culture and you don’t feel that people can talk about what’s really happening, it’s much harder to get things done. I really like the idea of generative culture that Ron Westrum came up with. He said there are three types of culture: bureaucratic, pathological, and generative. Bureaucratic is, “I could do this, but I have to ask my boss to ask your boss to ask you.”

Sarah: [00:20:00] Pathological is all about the power; no one really trusts each other. Generative is actually very open and it’s focused on making sure information gets to the right people at the right time. There’s a lot of broadcast of information so you have the information you need to make decisions. I used to run the operational team at the Financial Times, and the single thing that makes you most likely to be able to keep the systems up and running is that everyone feels very able to say, “I think there’s a problem, and it might be related to the change that I just made.” As soon as those people know that, everyone’s reaction is going to be, “Let’s get on a call, let’s work out what’s going on and see how we can fix it.” Then you’re much more likely to be able to fix things more quickly. That blameless culture is really important, and it’s the first assessment that I have when I’m talking to an organization: is that something that exists or that could exist?

Nick: I don’t mean to have a leading question, but do you see at least a correlation, if not causality, between an organization’s ability to track accurately what people are doing—the metrics of who does what—with that cultural imperative to point fingers, or the lack of it? You probably know where I’m going, but I’m just trying to understand.

Sarah: [00:21:00] The culture that wants to point fingers will spend a lot of time building elaborate plans. I think at some point, I realized that every item on the backlog of work for my team that was four or five deep—everything beyond that was never going to be done. Everything changes, priorities change, and you change what you’re doing. You’d have spent quite a lot of time grooming the backlog and making decisions about all of this stuff, and I just moved to the point where I just want to know what the next thing up is, and we’ll talk about that one in detail. It takes a particular organizational culture to trust that that’s okay and that you’re creating a roadmap. But honestly, you shouldn’t turn around to me in six months’ time and say, “The thing you had on your roadmap for October is not what you did,” because you should expect that a roadmap is a direction of travel right now, but not a commitment that this is how long it’s going to take. I tend to feel that organizations that are great culturally in terms of trying to always think of what we can learn and how we can improve tend to be more open to the idea that you can’t predict exactly how long it’s going to take to do something. You should do it in steps and iterate; all of the good things that I believe come together. I think those cultures tend to go together.

Nick: It just flashed into my head that a roadmap is not a GPS. It’s not supposed to tell you to the second when you’re going to arrive.

Sarah: [00:22:30] And a roadmap is communication of our intentions at the moment. I think that’s great. It’s really useful for people to see that, but I would love it if you could do a roadmap where it just got fuzzier the further out it was. For the first couple of months, you can see the details, and by the time you get out to next quarter, you can almost read the words, but you’re not completely able to.

Chris: You’re touching upon how we do roadmapping at Atsign.

Sarah: Nice.

Chris: [00:23:00] I’m very intentional about this because I understand the pitfalls of being entrapped in ceremony. Part of what you’re describing here is organizations that let their own processes devour them, and the people end up becoming servants of the process rather than the process supporting the people doing their best work. We do a thing where, close in, we’ve got some very focused expectations of what’s going to happen. And the further out you get, not only are we less specific about that, but also we start bracketing the time in bigger chunks. You’ve got the next couple of sprints, and then the next couple of months, and then the next couple of quarters, and then the next two halves.

Sarah: [00:23:30] I would agree with that completely, because I’ve just seen too many situations where people honestly believe the dates. The whole point when we’re doing software development generally is there are very few things where it’s incredibly predictable. You wouldn’t be building it if it was completely predictable because it would probably already exist; it might be available as a commodity. Every time there’s a burndown chart, you don’t really know. The only time I love a burndown chart is for something like moving boxes out of a data center; it’s pretty predictable how long it will take.

Chris: [00:24:00] You’re touching on people believing the dates against all previous evidence as well. One of the worst cases I’ve dealt with was a gigantic “Death Star” project, which by the time I encountered it, was many quarters overdue and considered absolutely crucial to the functioning of the organization. So, it’s going to be ready in two weeks’ time, right? How many times have you been told that it’s going to be ready in two weeks’ time? Dozens already. And how many times have you been disappointed? Dozens already. So why do you believe it this time?

Sarah: [00:24:30] I think when I joined the FT, there was a project that was a six-month project already in its second year. I also remember having a conversation with a Scrum Master about a feature that took six weeks, having been estimated at four weeks. We were going to have a retro to discuss why, and me and the other developers were saying that only being 50% over is miraculous. So, honestly, we were not that bothered by it. But also, we only released code every four weeks. There is no way we could have released this after four weeks just because of that. We had a long conversation about precision versus accuracy as a result.

Chris: Tied up in ceremony.

Nick: [00:25:00] Yeah, we’re going to have to let this go at that. Thank you so much.

Sarah: I could talk to both of you for ages.

Nick: I know, and my mind is racing for seven or eight other things to say, but I have to leave it there. So this has been the Tech Debt Burndown podcast. I’m Nick Selby.

Chris: And I’m Chris Swan, thanks for joining us.

Sarah: Thank you.

Recording date: Nov 4, 2024

Download at Apple Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Team Topologies co-author Matthew Skelton joins Chris and Nick to discuss the relationship between tech debt and cognitive load. Matthew explains that every organization has a finite “cognitive load budget,” and technical debt acts as a constant drain on this capacity, creating friction that slows down development and increases operational risk. By framing tech debt in terms of human cognition and team boundaries, the discussion highlights how healthy software evolution depends on keeping systems understandable and manageable for the people who build and run them.

The discussion also explores practical tools and frameworks for visualizing and communicating these sociotechnical concepts to non-technical stakeholders. The episode concludes with a (prescient) forward-looking warning about AI generated tech debt. While generative AI promises increased engineering productivity, it risks creating a tidal wave of tech debt. That could potentially be orders of magnitude greater than what humans produce, if the resulting code is not properly understood or maintained.

NB This episode was recorded prior to the release of the Second edition of Team Topologies.

Team Topologies

CodeScene

Conflux Proemion case study

Architecture Decision Records (ADRs)

Site Reliability Engineering (SRE)

Core Domain Charts

Nick: This is the Tech Debt Burn Down podcast. I’m Nick Selby.

Chris: And I’m Chris Swan.

Nick: And we’re here today with Matthew Skelton. And Matthew, do you think you could introduce yourself and say what you do for a living?

Matthew: [00:00:30] It’s great to be here. So I’m Matthew Skelton. I’m best known as being the co-author of the book Team Topologies, which came out in 2019. My background is in technology, I started off writing software for brain scanning machines, and then I’ve increasingly got involved in larger and larger systems. And now with my company, Conflux, works with organizations all around the world, typically large enterprises, and we’re trying to help them to make those organizations more effective and more humane, particularly when we’re building and running software systems. But we’re starting to work on areas outside of IT as well.

Nick: This came about because Chris and I were having a chat last week, and Chris mentioned… Well, why don’t you go ahead and mention what you mentioned, Chris, ‘cause it was really [chuckles] a really good point.

Chris: [00:01:00] So we were talking about tech debt because, of course, that’s in the title of the podcast. One of the ways I think we can frame this, one of the lenses we can pick up to look at it is cognitive load. The central theme of Team Topologies is essentially that an organization has a limited capacity of cognitive load. We could almost think of it as a cognitive load budget. We can look at tech debt and its demands upon the organization in terms of the continuous maintenance and the paper cuts to moving forward, and think about that as something that’s a drain upon that cognitive load budget, which we might want to try and manage at the same time as we’re thinking about using the guidance from Team Topologies to better optimize how that cognitive load is being utilized. And that’s why we invited Matthew here.

Matthew: [00:02:00] So it’s good to be here, and I really appreciate it. So for a little bit of context for your listeners, I did a master’s degree in neuroscience, so brain science, back in 2000. Now, that’s a long time ago. Compared to now, it’s nearly, what’s that, twenty-four years ago. That was at the University of Oxford. I’ve got a little bit of context in things relating to cognition, but that was a long time ago, and it was a master’s degree, so it’s not massively in-depth. So I’m not coming at this just from the perspective of pop psychology or anything like that. Yeah, I was in the pathology lab working with cells or mostly destroying cell cultures actually, because that’s what happens when you work in the pathology lab and working on some stuff to do with dyslexia. I’m not claiming to be an expert in cognition or brain science, but I’m not completely an outsider either, just reading the kind of airport books on pop psychology. And that was part of the reason why I was confident to include the idea of team cognitive load in the Team Topologies book. It’s one of the central principles inside Team Topologies. It is how we think about what we in the book called the cognitive load on the team as a whole.

Matthew: [00:03:00] Whether we’ve got that terminology quite correct or not is up for debate, but the central intent I absolutely stand by. So the specific version of cognitive load that we ended up talking about in the Team Topologies book was the one that was promoted by the Australian psychologist John Sweller. He introduced the term in 1988. That particular version of cognitive load theory is sort of debated. There’s different interpretations of it. Perhaps it’s not the best one, but fundamentally, the intention that we had was this: given the nature of modern business, public sector services and systems and products that are digitally enhanced, they’re digitally enabled, i.e. there’s some software behind them. Given the nature of those things, they’re effectively continuous these days. Think about SaaS, so software as a service. Think about government services, paying your taxes. What about your pension? What about the web application that you use to manage your pensions or the back-end systems that are there to manage your pension pot? That needs to continue for a fair old while, and you want those services to be evolvable for ten, twenty, fifty, seventy years. So there’s a need for kind of continuous evolution of these software systems, which is different from what happened in the past because in the past, going into 80s or 90s, even early 2000s, you’d have a new version of a piece of software.

Matthew: [00:04:00] It would get built, it would get burnt onto a CD-ROM or seventy thousand floppy disks, and it would get shipped, and that would be the end of it, effectively. And yes, there’d be a new version coming three or five years later, but there wasn’t really a sense of a need for continuous evolution of that thing. The reality now is we absolutely need to think about all the factors that play into the continuous and healthy and sustainable evolution of the modern software systems. That includes AI, it includes agentic and generative AI, whatever pops out of that space, and it’s all changing all the time. The same principles apply. We need to make sure that the software that backs these services and products is continuously evolvable, continuously understandable by human beings. That’s where this idea of team cognitive load comes from, because if we’re building and running these services, which is what we should be doing, because that’s the best set of incentives, you can’t have one team building and another team running these things. That just sets up the wrong kind of incentives. That’s what DevOps came along to try and solve. If we need to be building and running these services on a continuous basis, then we need to acknowledge the fact that a group of people, whatever they’re working on, has a limited amount of cognitive capacity, keeping things in their heads.

Matthew: [00:05:00] They’re learning new technology. They’re learning new regulations. They’re learning new concepts. They’re learning all the time. They’re having to keep this stuff in their head and then trying to represent it in software. That’s all coding is, that’s all software is, a representation of organizational intent. If we want them to focus mostly on their core mission, we need to be thinking about where the cognitive load is going, effectively, where that cognitive overload or mental overload is actually being used. It’s with that context. If you are doing something which is truly one-off, for example, tidying up after a children’s Halloween party, there’s hopefully no ongoing need to do that tidying up after a Halloween party on an ongoing basis. We just do it. We don’t need to really worry about the cognitive load. If, however, we’re building something like some sort of pension software or something which has a kind of continuous ongoing nature, we need to be mindful of where that cognitive load is going so that we can evolve this thing in a sustainable way.

Chris: So as we were

Nick: Preparing. You talked about the Ward Cunningham definition and an ideal view of technical debt. Can you give us some words on how often you come across that ideal?

Matthew: [00:06:00] It’s a good question. We tend to get called in to help with organizations where there are problems, so we tend to only see the situations where there are problems. But the kind of organization that has few or little problem with technical debt will be the organization that pays enough attention to refactoring, to dealing with the aspects of the software that are kind of grating, annoying to engineers. Those are often good signals for things that are awkward. Or potentially an organization that uses a tool that helps them identify where the risks are and where the technical debt sits. There are various tools out there, but the one that I’m most excited by at the moment is actually one called CodeScene. The organization is going to be doing well in this space, is one that’s going to be using a variety of signals to identify where the technical debt is that needs to be addressed and going ahead and addressing it. Most organizations typically don’t deal with the increasing friction that comes with that increase in technical debt. It’s effectively friction. A given type of feature or new user journey or new thing gets more and more expensive to add because they’re not dealing with the existing stuff. It’s fundamentally a business problem. It doesn’t just get more expensive, it gets more risky because they’re not dealing with the risk that’s associated with the technical debt as well. Risk and friction, and wonder why things go slowly and wonder why things are expensive.

Nick: I mean, I absolutely love it. You’ve just encapsulated a lot of my life.

Matthew: Sorry. [laughs]

Nick: [00:07:30] No, it’s, it’s great. I mean, as I was mentioning earlier, my firm usually comes in when something terrible has happened, and it’s usually a security incident or a data breach or something where everything’s stopped and, and I’ve often tried to explain this dynamic of like a year ago you were likely treading water, and some point between then and now you started taking water over the side. And, and at this point, you’re sinking. You’re not even able to keep up with what you used to do. Just, it’s not about improvement, it’s about just flailing to stay alive. You’re talking about friction there, and the increase in friction is really important. And I guess my main question for you is that when I come in, it’s typical that the teams I’m dealing with, they’ve got that sort of hollow-eyed look of people who’ve just; they’ve been in the engine room, and they know that there’s something really terribly wrong, and it’s all they can do to focus on just meeting the increasing requirements on them. Requirements from the business, but there’s also the requirements of managing this pile of whatever the workload is that they’ve normally had to do. They don’t have, it feels to them, they don’t have any extra cognitive space to think about changing what their conditions are. Often what I’ll hear is, “Oh yeah, I’ve had a plan to address that, but I just don’t have the time.” Chris once showed me the, cartoon of the caveman with the square wheels trying to push, and they’re like, “No, I don’t have time to swap it out with those round ones.” It’s a really big battle to say, “Look, the way forward here is to blow it up and, and kind of see where the issues are that we can make improvements to.” How can you apply some of the practices that you’re, you’re going to be talking about and that you have mentioned to, to those kinds of people? Like what are, what are some of the ways when it feels just overwhelming even to think about what I have to do today, let alone making it easier?

Matthew: [00:09:00] Great question. One of the things I’ve been most excited about in the last, say, six months is our partnership with CodeScene. I would argue the foremost tool in the space of addressing technical debt and code-based health, particularly ‘cause it takes a sociotechnical approach to looking at these things. In other words, it doesn’t just look at code as a technical thing, it looks at the relationship between code and people. And my company is partnering with them. When we were writing the Team Topologies book, Manuel and I were influenced by the work of Adam Tornhill, who is a CTO and co-founder of CodeScene. He’s been working in this space for ten years or something with some open source, I think, and some closed source tools to help look at the relationship between code health and team boundaries and so on. We’ve just published a case study on the Conflux website about a company called Proemion, based in Europe, who used a combination of insights from us, from some talks I gave and some consulting that we did together with CodeScene to help them actually prioritize what they focus on in terms of technical debt, and crucially, to be able to talk about those problems to people who are not technologists. And actually, time and time again, one of the things that we hear from people that we talk to who use CodeScene is it inspires non-technologists to really understand and start prioritizing work to address technical debt or whatever you want to call it. It was like, “Wow, this is amazing. Now I understand what you technologists are talking about when you say tech debt.

Matthew: [00:10:30] This is about risk. This is about speed, the time to market, and so on. Ah, I get it now. Right. Let’s address it.” We need to get to the point where we’re having conversations with non-technologists about the importance and value of addressing, say, technical debt and code-based health in general, to be honest. And that could be about changing the boundaries of responsibility. It could be changing about how the coupling happens, but it can also be about addressing change hotspots in the code. Because as many of your listeners will know, bugs and errors tend to creep in the areas where the code is changed most frequently. And so you can actually do bug prediction. You’re just looking at which areas of code change most often. Well, there’s probably going to be a ninety percent chance that a bug is going to happen in this area of the code. It seems like magic, but it’s not. It’s just inference based on observed patterns. But actually getting non-technologists to buy into this way of thinking using some techniques, persuasion, talks, tools like CodeScene, whatever, feels to me like an absolute essential thing to do. It can’t just be about engineers going, “Oh, this is really difficult to work with.” That’s part of it. That’s one of the signals we use. But we need to translate that into business outcomes, whether it’s risk or time to value or other things.

Nick: [00:11:30] I’ve yet to come across an engineering team that’s able to articulate in business terms the cost or opportunity cost of technical debt or not maintaining code properly. It’s just not a skill that they have. So that’s useful to hear. There are contra indicators here, and I’d be really interested in Matthew’s experiences with these. When I see organizations that are very intentional about decision-making processes and recording the outcomes of decision-making process in things like an architecture decision record, ADRs, those are places where conversations are happening about

Chris: [00:12:00] Okay, we can do this really quick and dirty thing now, or we can maybe invest more time and effort in doing something a bit more structural now. These are our choices. Which are we going to pick? And what are the trade-offs, and do we understand what future implications might be of path A versus path B? Sometimes intentionally taken on tech debt. There might at least be an understanding of how we got here. We can go back to the decision record and understand how we got here. We can revisit those criteria and decide that things have changed around us. Maybe technology’s improved. Maybe we can buy services cheaper. All manner of things that’ll make us go back and say, “Yeah, that was the right thing to do at that point in time,” but we can reevaluate now and move forward on different bases.

Matthew: [00:13:00] Hundred percent. And there’s two things that I’d want to add to that or that relate to that for me immediately. One is the SRE approach that Google have been talking about for a long time, and the second thing is a technique called Core Domain Charts. The first one, Google SRE, if you think about what they codified there with SRE, the product manager or product owner, the person responsible for decisions about that particular product is on the hook for product decisions that result in lower reliability or lower availability, which I think is absolutely essential. It’s a very opinionated encoding of that way of working, and it’s not for every organization. But the intent behind it is absolutely sound. You’ve got your error budget. Maybe the error budget’s like a mirror of the cognitive load budget. I’ve just had that thought. That’s interesting. You’ve got your error budget. As a product owner, you can choose to eat into that error budget by doing some stuff which is going to reduce the reliability a little bit. Great. If that’s going to achieve a business outcome, go for it. You can choose to make your service less reliable if that’s the right thing to do. You can drop the number of nines. You can go from four nines to three nines or down to two nines or two and a half. If that’s the right thing for the business outcome, great. Just state it.

Matthew: [00:14:30] Just make it clear. So then you might, in a similar way, take on some technical debt and say, “Well, because effectively we’ve got a debt budget.” Take on some technical debt, make it harder to make that change in the future. Make it explicit to say, “I want to get this feature out the door and to deal with the increased friction and delay that we’ll incur as a product team in the future.” If you’re responsible for that and that aligns with your P&L and everything else, just make it explicit and run with it. But make it explicit, and crucially, the financial results of that have to align with that decision-maker. We need to make it clear to them what the decision that they’re making. And so in organizations that are very sales-led or marketing-led and you’ve got a completely separate budget between sales and IT, those things are not aligned. So you get these behaviors where you just pile on more and more features, and then things get slower and slower, and then business gets more and more agitated. So part of that is actually about aligning the financial and flow metrics incentives properly inside the organization to make it clear what decisions people are really making if they say, “Well, we’ll prioritize this feature now.” They might still prioritize that feature. It’s more important to get this feature out now. We’ll increase the friction around feature delivery or changes over the next three months. That’s a business decision now. Okay, cool. We’ll get behind it.

Nick: [00:15:30] Provided that you have the metrics to count on the technology side and not just the finance side. This is a big thing that many people forget [chuckles] when they’re in a sales-driven organization. The numbers get really squishy when they go over to can you deliver these things, and when can you deliver these things? Unless the CTO is incredibly deliberate about how he’s setting expectations and against what he will be measured. You can get into some really serious trouble because you’re not speaking the same language. You’re not even close to speaking the same language.

Matthew: [00:16:00] So just rewinding a tiny bit. The whole SRE approach at least can inform how we think about technical debt. In the case of SRE, it’s about reliability. That’s what we’re really measuring, net uptime. In the case of technical debt, we’re measuring the friction and the increased cost around working with this code in order to get something equivalent out the door. Last year it cost ten bananas. This year it’s actually costing us fifteen bananas to do this thing because we haven’t addressed the underlying technical debt. Is that what you want as a business? Well, no, we don’t want that. We want it to be cheaper. Okay. Well, you’re going to have to do these things then if you want that metric to be a thing.

Chris: [00:16:30] I was going to say, at risk of going down a rabbit hole, how are you feeling about the confession recently from some of the co-authors of the SRE handbook that they had fibbed a little in describing their work at Google and laid out an idealized view of how they wish things were as opposed to how they’d actually been doing it at the time?

Matthew: [00:17:00] I did see that. It doesn’t surprise. It’s also nice that they were being honest, but I don’t think it takes away from the underlying message for me around SRE. I’ve been talking about SRE for ages, and there’s some stuff about SRE in the Team Topologies book as well. I did actually do a little bit of work with someone from Google, exploring exactly how it works inside Google at the time. I think this was 2018. But the underlying dynamic is absolutely sound, making sure that we’re fully connected through to the operational and business implications of our decisions about technology. I think that that’s an absolutely sound thing to try and do.

Chris: [00:17:30] I think at the time I saw some risk that there was going to be a shock, horror, politician lies reaction to that, of the SRE authors fibbed a little, so it’s all bunk. We can just safely ignore the whole lot. I think, as you say, they had good intentions, that they were describing an ideal that even they themselves perhaps weren’t realizing, but that they wished to articulate that ideal-

Matthew: Mm-hmm

Chris: … rather than simply describe what was going on in, as all organizations are, a messy, complex mix of people and politics and technology and everything else going on.

Matthew: [00:18:00] I have spoken to a couple of the authors. I suspect there’s very good intention behind it, and I don’t think it reflects on them badly that it might not have all been quite as polished as the book led you to believe.

Nick: Bringing you back, though. You were going to play SRE. You were saying you can go the SRE route or…

Matthew: [00:18:30] There’s another lens we can use to think about cognitive load because you set up the question in terms of we made some decisions a while back. It was the right decision at the time. We built this thing. Turns out it shouldn’t be the thing that we’re building now. We should do something different. We should just use something as a service. There’s an API for that now or there’s an Amazon or Google Cloud service for doing that thing. Example, some work I did recently with one of our customers. They’re a global organization. They deal with a lot of user-generated content for product reviews and that kind of thing. Ten years ago, part of the system they built was a globally distributed database. I think it’s using Cassandra or something like that. Anyway, it’s the right thing to build at the time because there’s nothing available in the cloud providers. Absolutely the right thing to build. In the purest Ward Cunningham sense of the word, that is technical debt, because now we wouldn’t choose to build that thing ourselves. We wouldn’t choose to build and maintain our own globally distributed data store ourselves. You just get it from a cloud provider. It’s all done for you. We don’t have to worry about it.

Matthew: [00:19:30] So in that particular case, our cognitive load is higher because we’re having to think about all of this underlying plumbing that the cloud providers are doing way better and at better scale. It’d be nice to offload all of that cognitive load to the cloud providers and just consume it as a service. It’s taking away valuable cognitive load budget from our people working inside the organization. We had conversations exactly like this. We were workshopping it, and one of the tools that we used in those workshops was an approach called Core Domain Charts. If you search for Core Domain Charts, you’ll find the stuff there on GitHub. It’s all Creative Commons, open source. People from the domain-driven design community were heavily involved in defining it, particularly Nick Tune. And we’ve used this loads of times with customers. Very straightforward, very simple. It helps us think about the things that are core to our business from a DDD perspective, the things that are supporting that core area, and the things that are actually generic, and particularly anything that we’re currently building ourselves that is in the generic area. In other words, anything that we’re building ourselves that actually is available as a service from a cloud provider or another provider somewhere else, clearly we shouldn’t really be building anymore.

Matthew: [00:20:30] It’s a nice technique that helps us to think through where that purest sense of technical debt currently sits in the organization. That globally distributed Cassandra database was sitting in the generic area, so immediately they could see, “Ah, yeah, that makes a lot of sense. We shouldn’t be building that ourselves anymore. Ideally, what’s our plan for not building it ourselves?” The Core Domain Charts approach allows you to map things out in time as well. So yes, if you rewind ten years, that globally distributed database was a really important supporting technology. They had to have it. And so that’s where it started. But then over time, there’s an arrow that points towards generic, because over time, that capability became available from the outside. It’s a generic thing as a service. So that particular technique around Core Domain Charts really helps technologists and non-technologists to have a really informed conversation about where we need to focus our cognitive load budget ultimately, to focus our mission and efforts generally across the organization. Really simple, but really nice conversation and exploration approach, particularly if it’s facilitated well, that helps people to get to that way of thinking really quickly.

Chris: So that sounds very similar to cases where we’ve seen Wardley maps being used.

Matthew: Mm.

Chris: [00:21:30] And we had Chris Daniel on the show a little while back talking about how he’s been facilitating some of those conversations that way. Frustratingly, we tried to have Simon join us, but the internet wasn’t working from his swamp.

Nick: [laughs] He got stuck in the waiting room.

Matthew: He lives somewhere in the middle of nowhere, right?

Nick: Yeah, he mentioned

Matthew: [00:22:00] … he’s got a couple of tin cans and a copper wire for his internet. [laughs] So Wardley mapping, yes, absolutely. The reason why we go to Core Domain Charts first is because Wardley mapping is quite a lot to get your head around for people who are not familiar with it. Whereas Core Domain Charts allows people to go in straight away without all of the background thinking and evolution and so on. Absolutely love Wardley maps, and that’s the right place to get to eventually. But Core Domain Charts was a different lens, and it’s an easier starting point for some of the people that we’ve come across who are not technologists and don’t necessarily think about things in that kind of evolution.

Chris: It sounds like you’re saying, “Don’t make people think too hard when they’re already tapped out on their cognitive load because of all the-”

Matthew: Exactly.

Chris: “… tech debt they’ve accrued.” [laughs]

Nick: [00:22:30] And that was my question a little bit earlier. Getting into CodeScene, this is important, but that thing about not making people think too much when they’re already cognitively overloaded is my main problem that I’m coming up against, which is how can you make this stuff easier and how can you explain the trade-offs that they need to make, the perceived trade-offs? Stop doing this bad thing that you’re doing because it’s making it worse for you to keep on doing it, and if you just breathe, we can find a better way to do it.

Matthew: Mm-hmm.

Nick: I haven’t really found an elegant way to get all the different parties aligned on that. And I think that that’s one of the biggest areas of friction that I have in fixing this issue.

Matthew: [00:23:00] I think with a lot of things, it’s shared language and visualization. One of the major reasons why the Team Topologies book has become so popular, I think we sold 170,000 copies worldwide, which is amazing.

Nick: Very nice.

Matthew: [00:23:30] One of the reasons there is the strong, clear language that we deliberately defined in Team Topologies, together with the diagramming style, which is also very strong, visual, and colorful. Again, deliberately, because we wanted to use that as a way to anchor some conversations. Tools like CodeScene provide a lot of visualization and opinionated presentation of information, which helps. And that’s part of the reason that tool in particular is really good. There are amazing diagrams showing temporal coupling. This piece of code over here always changes at the same time as this piece of code in two completely separate version control repositories. We’ve detected it, and so you’ve got a problem here because you’re not aware that these two things actually need to change. And visualizations of those couplings become a very powerful way to talk about the problem. A picture paints a thousand words. The visualizations can be really important to help cut through and anchor what we’re talking about. It’s early days to some extent. I think it’s fair to say Team Topologies was the first book that went mainstream that really connected the social and technical aspects of software delivery at scale, the sociotechnical aspect.

Matthew: [00:24:30] There are other books that we’ve referred to, absolutely. They were shoulders of giants, but I guess it’s the first breakthrough book that’s really looking at the sociotechnical aspects. And so that was only 2019. CodeScene has come to prominence in the last three or four years. Also takes a very sociotechnical view. It’s still relatively new in our industry to be taking this sociotechnical view. People have done it for decades, but in terms of groundswell awareness, sociotechnical angles are still relatively new. So I think it’s going to take a while for that stuff to permeate through. A lot of stuff in the domain-driven design community, a lot of stuff coming out of DevOps and Cloud and SRE, thinking about sociotechnical aspects. We’ll have to repeat it all again for generative AI and agentic AI, because that’s very technology led. Lots of tech bros are trying to pump all the new innovation things, which is just not very much connected to the sociotechnical aspects at the moment.

Chris: But is that because our successive hype waves are normally promising silver bullets?

Matthew: Yes.

Chris: [00:25:30] They’re going to just have the solution and it’s going to be easy rather than having to grapple with the messy complexity of the situations we find ourselves.

Matthew: Of course, that’s a whole separate topic, but the amount of technical debt that generative AI could bring in—you must have done a topic on this. If not, then I’d love to hear it. It could be a hundred or a thousand times the order of magnitude of the tech debt generated by humans, because you might have agentic AI generating all manner of new services and code that humans don’t even understand. It’s getting deployed out there and running, and then we go, “What’s this service? Who knows about this service? What’s it doing?”

Chris: What’s this line on the cloud bill?

Matthew: [00:26:00] Exactly. [laughs]

Nick: We’re going to have to leave it there because we’re at time, but Matthew Skelton, thanks very much for joining us.

Matthew: Thank you.

Nick: We really appreciate it.

Matthew: Thank you very much.

Nick: This has been the Tech Debt Burn Down podcast. I’m Nick Selby.

Chris: And I’m Chris Swan. Thanks for listening

Recording date: Mar 13, 2026

Download at Apple Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Chris and Nick chat about how AI Coding Assistants have suddenly become very good, maybe even good enough to write ’load bearing’ code. So the constraints the existed around producing code have potentially disappeared; but other constraints live on, and we’re still finding out what this means for product management.

Some time is spent on whether languages and frameworks still matter, and it seems they do. But also guardrails matter, and if you already have good guardrails built in to a progressive delivery system you’re likely better placed to (safely) take advantage of new tools.

Tracy Bannon, QCon London 2024 - Applying AI to the SDLC: New Ideas and Gotchas! - Leveraging AI to Improve Software Engineering

Chris: It’s the Tech Debt Burndown. Hi, I’m Chris Swan.

Nick: I’m Nick Selby.

Chris: [00:00:30] In the last episode we were talking about the slot machine and tech debt as a service, and how the advances in AI have taken the constraints away from coding but maybe not helped us as much as we need to in terms of review and making things production ready. In this episode, I wanted to explore the fact that in 2026, it feels like a lot of this stuff is working in a way that it previously didn’t.

Nick: Yeah, it really is. We were just talking before we were recording. Two years ago, I think Tracy Bannon at QCon was talking about how AI is a teenage apprentice in a wood shop. You give them very specific tasks, but you have to really watch everything. Now it’s changing really, really fast. Things actually work. They kind of work more often than they don’t.

Chris: [00:01:00] Over the past few years, I’ve been frequently returning to the state of the art tools to see if they can help me with my work. What I had been finding was, yeah, they could help me do trivial stuff. They were obviously useful if you wanted to build a website that was much like the millions of websites already out there and obviously in the training data. I wasn’t finding this stuff particularly useful for my work. That’s changed completely in 2026.

Nick: By the way, it’s March 13th, so to say “in 2026” is such a short timeline.

Chris: [00:01:30] Yeah, what, ten, 11 weeks into 2026. It feels like things have just flipped into a different state. I’ve gone from probing at these tools every so often to see how they’re improving, to using these tools daily as part of my working life. I’m increasingly finding myself using this stuff for load bearing code, things that are right in front of our customers. One of the reasons I think I’m doing that is there’s aspects of the output I’m seeing from the models that is not just faster than I would have done it myself, but probably higher quality.

Nick: Whoa. Okay. I never thought I’d hear you say that.

Chris: [00:02:30] If I’m having to migrate config, say in my case, I needed to migrate some config from systemd unit files into a YAML config file. The regex. I can conceptualize the regex. Can I spew forth correct regex? No.

Nick: No.

Chris: [00:03:00] These are things where I think as engineers, we quite often end up doing a lot of trial and error. This is not something where we’re talking about ralphing like last time around. This is not something where I’m asking the coding assistant to ralph it. This is something where the coding assistant has kind of swallowed the regex dictionary and can actually spew forth correct regex first time. I still need to have tests for that, but what I’m finding is it tends to be correct. This tending to be correct thing sort of leads me into a conversation I’ve been having about programming languages. I’m hearing people question whether choice of programming language is even a thing we need to contemplate anymore. Why does it matter? Firstly, why can’t we just prompt the coding assistant to do machine language for the instruction set architecture that we want, treat it as a compiler. We talked a little about compilers in the previous episode, but the compilers that have been made by coding assistants aren’t good enough, at least at this stage. Of course, that’s subject to change. The compilers are actually right now a load bearing part of why language models are more useful for coding than many of the other applications they could be put to. The model is creating some source code, but the compiler is the thing checking that it’s actually correct, as well as producing a useful artifact out of the other end of it.

Chris: [00:04:30] I was talking to a former colleague about how he’s using Rust for a system he’s building. The set of observations that came out of that was, Rust’s very strong in how the toolchain enforces correctness. One obvious implication of that is you can’t feed bad Rust into the Rust toolchain and get something out at the end of it. The model is having to create good Rust, but also the model has only been trained on good Rust because all of the public examples of Rust out there are Rust that has been able to build. The general quality of Rust in the training corpus is really, really good, as opposed to, if we look at probably the other end of the spectrum, the general quality of JavaScript in the training corpus is much more variable. I think there’s kind of a spectrum to be taken into consideration here. It’s very similar to a spectrum we might have already thought about before coding assistants in terms of the fussiness of compilers and the strength of type systems. How much a language and its associated frameworks are going to make you write correct code.

Nick: We were talking about doing this episode and you were talking about product managers, and this feeds directly into what you just said. I’d love to get you to jump to there to play that off what you just said.

Chris: [00:06:00] One of the product managers I follow on social media was raising a point from one of his colleagues that product managers are now starting to say, why would they have a meeting to discuss a new feature when it’s actually quicker and cheaper to get the coding assistant to just build the new feature. Put it out there into at least a progressive deployment environment to determine, with the normal sort of A/B tests or red, green or whatever they’re doing, whether that feature is desirable. Then they’ve got data.

Nick: [00:06:30] I have seen, every time somebody wants to bypass those meetings and that conversation, that’s when I see things that months later nothing works. I don’t mean necessarily that the features aren’t doing what they’re supposed to be doing, but underlying it, the problems that it creates or the problems within the production development process that have been created, and it’s order of operation issues, or it’s making sure that the right stakeholders were listened to so that it appears to do this right feature, but it’s not taking into account certain absolute requirements from other parts of the company, like legal or security or quality. Whenever you skip those things in the process, you end up in a situation where no one can ever figure out why we used to be able to do things, and now it all sucks and everything slows down and people get upset. I’m really skeptical that you can in fact do that, what those product managers are saying. I know it’s so tempting to want to do it. Tony Wheeler at Lonely Planet in like 1998 was doing this thing of, why do we need to pay translators to translate our books into Swahili and take our English books and put them into other languages? We can just use these tools to translate them. Of course, what came out was absolutely, obviously rubbish. This is more dangerous because it looks good and it sort of does the thing where the functions you’re trying to get do function the way you kind of expect them to, but at a huge cost.

Chris: [00:08:00] I think there might be almost a sort of superficiality test there. I used the term load bearing earlier on. I think there’s a consideration to be had there about what’s load bearing and what’s superficial. I was talking to Sarah Wells a day or two ago about the difference between policies and standards and guardrails.

Nick: Yeah.

Chris: [00:08:30] Where that went was, guardrails are about making people safe. If we think about guardrails in our built environment, they normally stop you running out onto the road in front of traffic. If I think about the guardrails we put into our systems environment, they’re to stop people from doing obviously silly stuff that’s going to cause an accident. This then brings us on to how are we putting guardrails in place for our use of coding assistants. Some of that is, we keep reusing our existing guardrails, assuming you have them. There might have already been a substantial investment in delivery pipelines and the ability to do progressive delivery and gather data from A/B testing or whatever it is we’re doing that is ensuring all of this stuff is safe. If it’s safe, it’s probably not going to cause the kind of problems you’re referring to. I don’t think all organizations actually have all of those guardrails already, and so they might be tempted into unsafe practices. There’s another thing here, which is how we’re providing context about safety to the coding assistant. There’s a lot of discussion presently about things like agents.md files and whether they should even be used or not. Research out of Zurich, I think it was in the last week or so, saying actually you might be better off without this. MCP was the hotness of the last year or so, but everybody’s now talking about skills. Do we encode some of the things we would have had in our policies and standards or whatever into skills, which the agent is then going to be using as guardrails? Because, once again, non-deterministic behavior. We can’t count on it to really pay attention to that stuff. It’s all just part of the context mix as we stir it in.

Nick: [00:10:30] Like, five years ago when we started this podcast, one of the things we were struggling with: what is tech debt? One of the things one of us said was, it’s the boring stuff you ignore because you want to get down to doing the things you want to get done. I was just reminded of that when you were saying this stuff about the safety and the guardrails, because the parts I consider to be unsafe are the unseen, undiscussed things that product and engineering are actually not talking about. It’s the inputs that come from customer feedback that doesn’t quite make it through a loop because often we get bad at that, we get lazy at that. Or it’s that arcane human knowledge that comes from subject matter experts in places like legal where, well, you can’t do that because this country or these things, this group of our customers does something. It seems to me that sort of counting on those guardrails, you said, like a lot of companies don’t even have very good ones, but the ones that they have tend to be much more binary. We shouldn’t do it like this, we should do it like that. There’s a lot of nuance that gets lost when you don’t have those people having those conversations, because an idea will come to somebody and like, oh, wow, yeah, I didn’t even think of that. This wasn’t even part of our whole discussion up until this moment. I’m afraid that we will lose that if we try to sort of reduce everything to a skill file.

Chris: I think some of what we’re touching on here is what complexity scholars would refer to as emergent properties of complex systems. Some of this is unknowable until we try. So we’re back to our ralphing. Except, except it’s not the system doing the ralphing now. It’s the people doing the ralphing. Yeah, let’s try lots of things.

Nick: [00:12:00] The most dangerous sentence in the English language is, “why don’t you just…”

Chris: The most dangerous sentence in the English language is, “we’ve always done it like this.” But “why don’t you just” is probably lined up right behind it and giving it a shove in the back.

Nick: Every time I make these sort of declarative statements, I have to remind myself that I’ve radically changed my outlook on some of these things because of the speed that we were talking about. You said ten weeks into this year and suddenly things are working. Obviously it’s not sudden, it’s been this incremental thing. Then all of a sudden we hit that point.

Chris: [00:12:30] Yeah, I feel like a breakthrough has been accomplished. I was at a Google event last week. They asked everybody to stand up and they went through a seven stage ladder of AI adoption maturity.

Nick: And making people sit after everyone.

Chris: [00:13:00] I was one of the first people to sit down, actually, because I’m still babysitting the tools quite closely. First stage on the ladder was, are you still feeding stuff into a chatbot and pasting the output into your well matured coding practice? Then it went on to, are you working with an agentic system that’s doing this stuff autonomously for you, but you’re supervising it to the extent that you’re reviewing every change before it gets committed? This stepped its way through to people doing the multi-agent framework stuff where they’re kind of literally setting it off while they sleep and it’s burning tokens overnight. Maybe they wake up to some gloriously working code and an empty token balance.

Nick: But on most days.

Chris: [00:14:00] It essentially boiled down to, have you been able to establish a trust relationship with the agentic system that you can let it off the leash and do its thing? I think there’s a flip side to that coin, which is, is what you’re working on consequential enough that you need to establish that trust? Or are you trying out lots of ideas and it really doesn’t matter?

Nick: [00:14:30] Speaking from the security ghetto, I’m going to say that that last one is almost always wrong, and that we are terrible judges of what is consequential and what will be consequential in six months. But I take the point. I really do take the point. Actually, maybe you and I had had a conversation, it was probably a year ago, about, if agents are writing the code or if AI is writing the code, then humans have to write the tests. We talked a bit about testing in the last episode, and like ralphing, where it’s writing the test. Do you see any change in that, that you can actually have it write the own tests? Or do you still feel better? Is that affected by the trust relationship, or is this just a constant still, because it was a constant when we talked about it?

Chris: Certainly the practices that most people are talking about encompass test driven development in a way that probably hasn’t been widespread in general development practice. The extent to which the tests are being eyeballed is, I think, highly variable.

Chris: [00:15:00] I do see reports of agents cheating because they do that. But I think also people are coming up with ways to refine that process, at least to their satisfaction. If I kind of back up to why I was sitting down early in this process, it’s kind of because the things I’m concerned with, I do actually still care a lot about the quality and the detail. I’ve not yet let go of that.

Nick: [00:15:30] We talked about this yesterday at lunch. It was a good point.

Chris: [00:16:00] Should the agentic system need to worry about my notions of code smells and whatever else? Maybe not. Maybe that’s too fussy and maybe that’s too expensive in terms of engineer supervision versus that energy could be expressed elsewhere. We’re seeing an argument surface that coding’s free. There’s no moat now around product. So we need to focus every effort on the bottlenecks that live elsewhere. One of the primary bottlenecks is go to market. How do I even find customers that are going to care about the thing I’ve just dreamt up? It brings us somewhat back to those fundamentals of tech debt, because we might be doing something very expedient to find product market fit and find a customer that cares for that product market fit. I had one other little note here, which was about how we use the tools. Something I found a few times in recent adventures is, if I ask the platform, and this could be just a prompt in a chat, it could even be a prompt going into a coding assistant, to do something for me, it might do a pretty terrible job of that.

Chris: [00:17:00] I’ll give a concrete example. I had to reconstruct a changelog from a git commit history a couple of weeks back. I asked for that to be done and it just did a terrible job of it. The resulting markdown file was incomplete. It was all over the place in terms of formatting. Thoroughly unsuitable. So instead I asked for a Python script to do the job for me. Not only was the Python script that was produced very good, I was using Gemini, so it was a very Googley kind of Python script, in compliance with their coding standards and with lots of good comments and very thorough in how it was making use of the present idioms, but also produced an absolutely perfect changelog from the git commit history. According to what I was trying to accomplish, it needed a couple of iterations to refine it to the point that it was exactly what I wanted. It kept on doing that iterative improvement, and it took minutes to get to the finished article and crank the handle and get the changelog out at the end of it.

Nick: [00:18:00] Are you still surprised when that happens? Because I am, every time. Every time I get something, it’s like, that’s quite good actually. Oh, I want to change this. Then you go through and then you’re there and it’s just like, that was why we talk about this stuff.

Chris: [00:18:30] Well, I think this use of tool thing is something that is going to iterate upon itself. Right now, we sometimes ask the LLM to do something and it does a bad job, but if we ask it to use a tool to do it, it does a better job. I think what we’re seeing with the coding assistants and agentic practices is we’re essentially embedding that tool usage, and even in some cases tool discovery, into that iterative process. From that comes improvement.

Nick: We can leave it there. That was really cool. It’s been the Tech Debt Burndown podcast. I’m Nick Selby.

Chris: And I’m Chris Swan. Thanks for listening.

Recording date: Mar 13, 2026

Download at Apple Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

After a long hiatus, the Tech Debt Burndown podcast has returned for its third season. We’ve got a bunch of episodes (some recorded quite a while ago) in the can, so more to come; along with some fresh stuff, as 2026 has been a wild ride already.

Chris Swan and Nick Selby kick off the season with a discussion of what Chris has called Tech Debt as a Service: how we’ve entered the slopocene, and how we are working with our magical AI coding assistants.

We start with a discussion of multi-agent coordination frameworks, one of which is called Ralph Wiggum, and what’s become termed as ‘ralphing’ is basically sending the coding assistant into a loop until it has successfully met some success criteria in order to exercise the loop: It’s kind of going round and round, and maybe doing lots of pretty dumb stuff, but it keeps trying, right? And you don’t let it stop until it’s got to where you want to go. You’ll probably also define some kind of max loop criteria so you’re not just burning tokens all day long . . .

Nick: It’s the Tech Debt Burndown podcast. I’m Nick Selby.

Chris: And I’m Chris Swan.

Nick: We’re here to talk about Ralph Wiggum. And Chris, why don’t you kick us off, please?

Chris: [00:00:30] So the theme for this episode I thought was going to be essentially how we entered the slopocene and how we can get tech debt as a service from our magical AI coding assistants. I guess one of the things that we opened 2026 with was the arrival of these multi-agent coordination frameworks, one of which is called Ralph Wiggum. Yeah, I prepared some notes ahead of this. And Nick asks, what are you, Ralph Wiggum? Why are you talking about Ralph Wiggum? Isn’t that just the kid off of The Simpsons? And I said, yes, that was worse.

Nick: Wasn’t it the chief of police off The Simpsons? And then you had to remind me of the boy.

Chris: [00:01:00] And so what they’re doing here with what’s become termed as ralphing, is basically sending the coding assistant into a loop until it has successfully met some success criteria in order to exercise the loop. It’s going round and round and maybe doing lots of pretty dumb stuff, but it keeps trying, right? And you don’t let it stop until it’s got to where you want to go. You’ll probably also define some kind of max loop criteria. So it’s not infinitely doing dumb stuff and chewing through tokens and running up a bill. But yeah, that’s what people are doing now.

Nick: [00:01:30] Of course, my definition of ralphing is shouting groceries or vomiting. So when you just said that, I guess it all makes sense.

Chris: [00:02:00] I was looking up ralphing on the way in, the Urban Dictionary definition is vomiting stuff up. And so yeah, I feel we’re getting our AI agents to vomit stuff up until we’re satisfied with the vomit. And it’s passing tests that it’s written for itself according to its interpretation of a prompt. And it’s not just Ralph Wiggum. At the same time as that was hitting the scenes, there was another similar framework called Gastown that came along from Steve Yegge. And so I think Steve’s quite a famed developer and has done some amazing blog posts that have captured people’s attention over the years. More recently, I’ve seen Steve working a lot with Gene Kim, and they’ve written their book together about vibe coding. If we look at what’s become of Gene’s DevOps event, it’s now more focused on enterprise, but very much more focused on AI and AI adoption and how things like vibe coding and the use of these multi-agent frameworks is starting to take effect.

Nick: [00:03:00] So that’s actually a good thing. We were talking about the book that we’re writing, and we were just talking yesterday about the lead time. So now you take a look at Gene’s book and even some of the stuff earlier in the season that we were just discussing. So has Gene’s book already been overtaken by events?

Chris: [00:03:30] I think so, yeah. Gastown went from zero to huge in the space of days at the start of the year. And I should also tip a hat to Ruv Cohen because if I look at people using these multi-agent frameworks, as far as I could tell, it really began with Ruv and his work on this stuff. And then other people have created similar frameworks to enable that. And lots more people are now picking those things up and running along with them. I also feel there’s something of a fruit machine mentality going on here. So we talk about tokens and here we are recording this at Brighton Seaside. And so we could step out and find an amusement arcade not very far away. And we could probably buy a bunch of tokens and feed them into slot machines and pull the handle and see whether we win.

Nick: Right.

Chris: And I’ve got a sense right now that a lot of the LinkedIn posts about this are coming from people who’ve stuffed their tokens in the machine, and they’ve pulled the handle and it’s come up, bar bar bar, and it’s paid out the jackpot: they got something useful. And the people not writing LinkedIn posts about this are people who put their tokens in the machine and they’ve pulled the handle and it’s come up cherry melon bell. Yeah. Let’s go again.

Nick: [00:04:30] Yes. I think that the great unwashed is basically a bunch of older Parliament-smoking women sitting in the back of the casino, just with the giant bucket of tokens and continuing to put them in to see if you win. And it’s always a big surprise.

Chris: [00:05:00] The most insane thing, I think, is if you look at the economics behind this—and I think he’s been doing an excellent job in recent months of exploring this—they’re selling the tokens to us at pennies on the dollar, the data centers and everything that’s happening behind these tokens are all being carried by SoftBank and other deep-pocketed investors right now. Or Nvidia is feeding money into the system so that people will buy its GPUs.

Nick: Right. And get hooked like any good dealer does. Yes.

Chris: And so I feel we might be in an Uber moment in that period where all of the Uber rides were subsidized.

Nick: Yes, yes.

Chris: [00:05:30] Yeah. We’re then waiting for how useful is all of this going to be when we’re actually having to pay full freight for the tokens?

Nick: I would like to just zoom back now. Right. Let’s give the higher-level introduction to what it is that we’re talking about and introduce the concept of tech debt as a service. And it’s really compelling to just want to create it because you can do so much and feel so creative so quickly that before you know it, you’re left with a giant pile of tech debt.

Chris: [00:06:00] One of the factors that’s really captured my attention here is the notion that we should no longer be looking at the code. You come up with your plan. You dispatch your multi-agent coordination framework to implement the plan. It’s doing test-driven development. So it’s creating a whole bunch of tests that are going to persuade you that the plan has been implemented correctly.

Nick: Some of the best comments ever, by the way. Done. It’s good.

Chris: [00:06:30] And then we get software out of the end of this process, but no human being is actually going line by line through that software and seeing that they’re comfortable with that. We’re taking our comfort from the test running. And I think in some ways, this is what we do all the time with continuous delivery. And progressive delivery is we’re feeding change into a pipeline, and we’re taking confidence and safety from our tests. And then new stuff is emerging from the pipeline. And we’re doing our canary deployments or our feature flags or whatever else to ensure that it’s behaving as we want it to. But this feels like much more wholesale change. We’re cutting new things from clean cloth and stuffing them through the pipeline for the first time and expecting them to emerge into the world fully formed and ready to go, rather than, I think, continuous delivery and progressive delivery, even more so. We’re all about tiny incremental changes where we’re moving forward a tiny bit at a time, but we’re always making forward progress.

Nick: [00:08:00] Yeah. And it’s funny, that has actually become a cultural thing. I did a podcast with a bunch of folks for Bishop Fox a couple of weeks ago. And the question was, does this all save time? The consensus was not really, but developers are spending more of their time thinking about what they want to do and deliberately writing out the prompts so that when it gets generated… So now the code just happens in seconds. But you’re right, there’s a missing piece there at the end where you take a look at it and see did it actually work, or is it just passing the test that it wrote for itself?

Chris: I think there’s an aspect of this: not looking at the code, which I’m willing to accept as the introduction of a new abstraction layer. One of the comments I saw from…

Nick: A nice comment that’s really…

Chris: [00:08:30] …good from one of the industry elders, was how much of us look at the bytecode that emerges from a compiler, or the machine code that emerges from a compiler? And the answer is almost none.

Nick: I know one person in my entire life who would do that because he thinks it’s neat.

Chris: [00:09:00] I think there are occasions when you’re having to focus on very specific performance optimization that you might actually be diving into: what has the compiler done with my source code and how is it emerging and how is it actually interacting with the instruction set architecture of the machine I’m going to run it on? Most of the time, most people are not looking at compiler output. And so if we say that we’ve moved up an abstraction layer, we’re no longer dealing with programming languages, we’re just dealing with prompts. Then the coding output of the prompt becomes like the output of the compiler. Now there is a very strict differentiation here, which is compilers are deterministic. Yes. They’re going to give us the same output.

Nick: Yeah. You run it 15 or 15,000 times.

Chris: [00:09:30] Every time, whereas large language models are non-deterministic. We can’t expect that repeatability that I think we’re used to with compilers. And I think I’m going to dive down a little rabbit hole as well here, because one of the things that’s happened in recent weeks was Claude actually was prompted to create a compiler. So this was a project that was done at Anthropic, and they got to a compiler that was able to compile the Linux kernel.

Nick: Okay.

Chris: [00:10:00] And this was something that they completed in about two weeks, and they spent about $20,000 of tokens on it. So not enormously expensive. It was all enabled by the fact that compilers have got great test suites. And can we know the things going through? Compilers have test suites. And so we can see when something is converging on being correct. The destination I’m trying to get to here is they made a functioning compiler, but it’s terrible. And so all of that effort that’s gone into GCC and LLVM, etc. over the years to create a high quality, performant, correct compiler was not there in the product. And so when people were performance benchmarking the code coming out at the end of this thing, it was in some cases hundreds of times slower than the equivalent code. And I think this relates back to this notion of dealing with abstraction layers. So as an industry, we’ve been quite comfortable with orders of magnitude differences between different programming languages. If I want something to be quite performant, I’m probably going to pick up a low-level language like C or Rust or something like that. Whereas if I’m not so bothered, then I might be using JavaScript or TypeScript or Python or something like that. Well, once again, we are confronted with this balance between engineering productivity and system productivity.

Nick: Yes.

Chris: And I think…

Nick: That’s really nicely said.

Chris: …a lot of people are going to choose, I want to get this into the market, or at least I want to test my market fit with something. And so I’m going to take engineering productivity every time. And system productivity is something we can worry about later if it succeeds in the marketplace.

Nick: Yes.

Chris: And that’s tech debt as a service.

Nick: [00:12:00] That is, it’s an old trope, but there’s nothing as permanent as a temporary fix, right? And once you’ve got it working and you’ve got that fit, the temptation to never look at it again because it fits is… you want to get on to other things that are more important. So yeah, it just sits there and festers and moulds and compounds as time goes by.

Chris: [00:12:30] But also, I wonder to what extent you can take the prompts that got you there and say, do it again, but do it better. And to what extent you can also have an expectation that these things will have improved? Maybe in the time it takes you to establish that something’s got product-market fit, you can do a V2 of the product almost with the same prompts, but with a better underlying mechanism for implementation.

Nick: [00:13:00] Our developer, Dmitri, is talking about this quite often, and he is iterating much, much more quickly and making really substantial changes to a whole bunch of different things that he does. That’s about all I can say is I’m recognizing the pattern. But I haven’t dived deep enough to say anything about it.

Chris: [00:13:30] I think it’s worth as well touching on the impact this is having on open source. There was a project last week where the maintainer had essentially retired, and the project was still healthy and well maintained, but the new maintainers had decided to do what they considered to be a clean cloth reimplementation using LLM-based coding assistance. They had not fed in the original code themselves. They had simply said, here’s the test suite, create new code that passes the tests. And when they ran it through tools that are essentially there to look at software plagiarism, it was coming in at a tiny percentage. So it was passing the squint test.

Nick: Right.

Chris: [00:14:00] On being a clean implementation. The reason that they were doing this was so that they could use a more permissive license. So the original developer had chosen a copyleft license, and they wanted to relicense it with a more permissive license. Now, of course, the thing lurking in the background here is almost certainly the LLM that they were using had been trained right with the source code for the original working implementation. So although they’d not fed that in themselves…

Nick: It was there.

Chris: [00:14:30] Was probably there. Yeah. Lurking in the weights in the background. And since that, there are now organizations starting to pop up and say to enterprises, hey, if you’re bothered about the licensing of some of this open source that you might use, then we’ll create for you a relicensed re-implementation of it. This potentially undermines the whole pretext of how people have been going about maintaining open source projects. And it comes on the back of open source maintainers now being inundated with slop. Yeah. We’ve seen curl in recent months has ended its HackerOne bug bounty because throughout 2025, they had lots and lots of sloppy, supposedly security vulnerability issues that were just sapping huge amounts of time to chase down. And almost all of them were phantoms. We’re seeing many projects where they’re getting contributions of low quality. That’s causing even more maintainer burnout than we were already facing. And I think it was precarious already for many open source projects and their maintainers. And now it’s just this tidal wave of low quality, sloppy submissions being put in there. How to deal with that? So we’re seeing people seriously proposing reputation-type systems.

Nick: Oh, right.

Chris: Yeah. The thing that Stack Overflow had, it becomes gatekeeping. And I think one of the things I’ve always loved about open source was literally anybody could show up and get into a codebase and start making improvements.

Nick: [00:16:30] Yeah. Curl was always famous for having a significantly large raised eyebrow department to which everything was submitted well before this. But yeah, you’re absolutely right. The gatekeeping was around quality. And are you following the mission of this project? It wasn’t, where did this come from?

Chris: [00:17:00] Yeah, I’ve done a documentation PR to curl that was accepted. And I like the way that Daniel proudly announces each additional contributor as they show up. It’s roughly a handful a week. And it’s great that that project is continuing to attract people in addition to a very dedicated set of core maintainers. That’s, I think, one of the best-run projects on the planet. They’ve set up a lot of infrastructure around that. And I think there’s also been a lot of investment from the broader community in ensuring that success. And then when we get further down the xkcd cartoon…

Nick: Yes.

Chris: [00:17:30] Your mileage starts to vary an awful lot more. There’s a whole bunch of emerging concerns around how open source survives the slopocene. Yeah. How we’re going to be able to proceed with that. Yeah. Something else I’d noted here was a few days back, Amazon had an all hands where…

Nick: Yes.

Chris: [00:18:00] They were essentially saying, hey, we feel we’re starting to have some AI-assisted outages. Can we stop doing that by having more senior reviewers ensure the quality of what we’re doing? And I made a comment on LinkedIn that I think Amazon uniquely brought this problem upon themselves in that the whole two-pizza team thing, and you build it, you run it, and a hiring process that ensures that they never have any glue people, is all well and good during the glory days of growth, and seems to be having a stressed relationship now that the executives are playing Jenga with the org model. And who’s keeping the place together? Well, it now seems that the onus is being put upon those senior engineers to ensure quality and that the constraints have moved. So the constraint is no longer, how can I make code? It’s do I have enough reviewer bandwidth?

Nick: [00:19:00] How can I assure that the code that I’m getting is something that we could put in and that it’ll work and be as stable as it should be?

Chris: [00:19:30] Am I maintaining my delivery pipeline carefully enough that I’m ensuring that good quality stuff is what’s making it into production? Amazon certainly had an engineering culture that could get lots and lots of services out there and supported and billing people and whatever else. They’ve at least got a lot of that culture and systems, whereas there’s plenty of other organizations where there’s an ability to create the code and maybe even some artifacts out of that code, but who’s taking that to production?

Nick: So I think we’re out of time for this. But I think the next episode, we’re going to talk a bit more about that. And also the really compressing timeline of how these things are coming out and how people are adapting to it. So, it’s been the Tech Debt Burndown podcast. I’m Nick Selby.

Chris: And I’m Chris Swan. Thanks for listening.

Recording date: Apr 17, 2023

Download at Apple Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Gene opens with a comment about the “tendency of the industry to jump on hot trends”, and that sets the scene for much of the discussion, which goes on to touch blockchain, and of course ‘AI’.

We touch upon topics where Gene and his co-authors go into more detail in Cybersecurity Myths and Misconceptions such as where liability should be placed to better incetivise the creation of spftware that’s safe, secure and reliable. Though Gene acknowledges that we don’t (yet) even have good metrics for those terms. That leads into some discussion on whether organisations like the Open Source Security Foundation (OpenSSF) can fill some of the gaps.

Before closing we get to some discussion of the European Union Cyber Resiliance Act (CRA) and some of the consequences that might have for open source software.