Acima Development

Episode 102: Distributed Authority


Listen Later

This episode of the Acima Developers Podcast continues a series on a NASA paper about engineering failures, focusing this time on decentralization of authority. The core question Dave poses is where breaking things down and simplifying goes wrong, and the group lands on a shared answer: complexity doesn't disappear when you split systems or teams apart, it migrates into the seams between them. Will frames organizations as multi-threaded programs, arguing that distributed work is inherently less efficient than a single mind holding everything, and that the real failure is that nobody budgets for that inefficiency. Teams plan sprints as if they operate at maximum efficiency, then get blocked waiting on other teams like threads waiting on a mutex.

The conversation moves into war stories about organizational structure. Kyle describes how Acima's DevOps team, once embedded with engineering under one budget, now sits in a separate department requiring alignment several layers up before work can happen. Dave cites a Microsoft study finding that bug difficulty scales dramatically with org-chart distance between the person who needs a fix and the person who can make it, and recounts a previous employer where carving out a DBA team concentrated all database authority in one group while the blame stayed distributed everywhere else. Will's proposed remedy is cross-functional "tiger teams": pull a dedicated body from every needed discipline (DBA, DevOps, security) onto one team for the life of the project, deliberately over-allocating rather than pretending ticket queues between silos are free. He argues for institutional checks and balances over exhortations to "speak truth to power," since pressure, layoffs, and optimistic scheduling make honesty structurally difficult.

The back half turns philosophical, sparked by Vivian's observation that startups work like mesh networks (everyone connected, nothing precious to break) while enterprises are hierarchies protecting a "golden goose." That leads to a warm tangent on being kind to your future self: Dave's story of leaving network diagrams inside junction boxes that paid off 15 years later, Vivian's "second brain" note-taking practice, and the YAGNI principle of trusting tomorrow-you to solve tomorrow's problems with better information. Will counters with a lament about corporate chat retention policies deleting institutional knowledge after three months, which he calls "leadership malpractice." Vivian closes with a fitting meta-observation: without Mike there to keep them on track, the decentralized conversation never actually reached a conclusion about decentralization, inadvertently proving why some centralized accountability matters.

Transcript:

DAVE: Hello and welcome to the Acima Developers Podcast. I'm your host today, Dave Brady. We have got a really good panel today. We're still talking about the NASA disaster stuff.

Today on the panel, we've got Will Archer. We've got Eddy Lopez. We've got Vivian Moore. We've got Kyle Archer. We've got Will's dead camera connection from his previous dropped call, and we've got Ramses Bateman on the call.

So, we've been talking about this NASA paper, if you've been following along at home. You can get this on the Googs or wherever: Elements of Engineering Excellence by J.C. Blair, R.S. Ryan, and L.A...Oh boy, Shutzenhofer, from AI Signal Research in...if you find that, Google for that, you'll find this.

And, basically, it was...they prepared this paper for NASA telling them, "This is why your stuff keeps breaking, and you keep killing people when they try to go into orbit or come back from it." And we've already talked about, like, normalization of deviance and, I don't know, some other stuff. We did a whole show a couple of times, and I kept making it a forum for AI. So, I'm hosting today, so I'm going to be asking questions and trying not to turn it into the Dave show.

We're really excited to have Vivian on the show. Vivian, you're one of our interns for the summer. Do you want to say hi to the audience, just tell them who you are?

VIVIAN: Yeah, sure. So, I'm Vivian Moore. I'm going to be one of the software engineering interns for the summer. I've been actually listening to the podcast for the last, like, couple months or so, kind of getting prepared for the internship, getting to know everybody. So, it's kind of a weird experience being on here now that I'm actually at the company, but it's very fun.

DAVE: Right on. Welcome.

So, Mike always starts with a story, and I usually have 20 of them on hand, but all stories that I have are going to go immediately onto a tangent. So, I kind of wanted to just kind of open the floor a little bit. Mike commented to me...Mike's running late today. He might make it. I'm hoping he does make it. But he sent me the meme from Office Space of, like, when you have eight bosses, nobody's actually in charge when you answer to eight people, right? It's the bystander problem, bystander effect where, you know, if a crisis is happening and there's eight people standing there with their cell phones, and you say, "Call 911," nobody calls 911 because everyone assumes that everyone else will do it.

Paper hones in a little more sharply on this, that it has almost more to do with siloing teams and isolating teams from each other and stopping them from communicating in between each other. And, in the pre-call, I mentioned that the devil kind of lives in that space.

I've seen so many software ideologies come down the pipe over the past decade or two, where people have, you know, like, "I've got the answer, and this is going to solve everything. All you have to do is simplify the piece that you're working on." And they show you this really simplified thing, and it looks good. It looks clean. It's obviously true, but it doesn't solve the complexity. It just pushes the complexity outside the thing you're actually working on. And if the other team on the other side of the wall is pushing that complexity back at you, that complexity now lives in between.

Honestly, that was my big grief with, like, the way Java architected things out by default. I mean, like, Java's great. It's a good plan. But the naive approach to the way Java breaks everything down into tiny, tiny, tiny, tiny, tiny, tiny...Like, I can remember people saying, "You should not have a class longer than 25 lines of code, and you should not have a method longer than 5 lines." And I wish I could write software like that. I know people who can, and, I tell you what, they are very, very good at finding the file they need or the seven files that they need because they're all over the disk, right? Which arguably is probably better than trying to sift through a 3,000-line chunk of code, which actually might be an allegory for this organizational problem.

But my issue is that you start having problems with how the modules get picked up and put together, because complexity starts to live in the seams, not the units, but the integrations between them. What do you guys think about, like, decentralization, breaking stuff down? We all know simplifying things is good, but where does it go bad? If we just take simplifying and breaking things down as an objective good with absolutely no qualifications, where does it go wrong?

WILL: So, whenever I think about, like, sort of, like, these sort of, like, distributed processes, right, like, across engineering organizations, individual engineers, like, I always think about a multi-threaded program, right? So, one of the things that we've seen over the past, like, you know, however many years, like, really since we broke out of the 486, you know, line of computing is more and more and more cores, right? More and more cores: 18 cores, 16 cores, 32 cores, lots and lots of cores.

Well, the problem with lots of cores is there's lots of threads. And one of the sort of dirty, little secrets of, like, this sort of, like, nitty-gritty systems-level programming is your efficiency, as you have these multi-threaded processes, intrinsically, like...And I think this is mathematically provable. It always goes down relative to a linear A to B to C to D to E to F, all the way to Z, like, one at a time, right, maximally efficient. But, you know, then you only have one person working one way.

For people who've experienced, like, the joy of creating a system, or a service, or a program, or a protocol soup to nuts, and it all lives in your head─you made all the decisions; you did every single thing; you know where all the bodies are buried─you will never in your life experience efficiency like that. But you're still only one person of flesh and bone. You only have 24 hours in the day, just the same as everybody else. And if the enterprise is going to scale, you're going to have to scale out to different people. And that comes with inherent inefficiency.

If you did it right...and writing multi-threaded programs is astonishingly complex if you've had the burden. A lot of people don't really do it. It's pretty rare for people to write hardcore, nitty-gritty, multi-threaded programs anymore. All of that heavy lifting has been done by, like, really, really, really, really smart people at several different layers, you know, like, to the database, to the operating system, to the sort of, like...Anyway, the frameworks, all that stuff has been done, but it's astonishingly hard to do.

We have, you know, general organizational principles to do it, and they're all pretty good. They're all easier said than done, but, like, where I feel like things break down foundationally is we don't allocate for the inefficiencies inherent in the system, like, that's the problem.

Every team schedules themselves. They plan their prod. They plan their releases. They plan their sprints, their production quotas, you know what I mean, their timelines for this maximally efficient thing. And it's always wrong because you're always going to have to, like, you know, take the lock. I'm waiting on this mutex, you know? I'm waiting on a barrier, right, which is, like, the release calendar, and all the threads have to wait at this, you know, this barrier. You know, going back to, like, all these, sort of, like, nitty-gritty parallel processing terms.

And so, you don't allocate for that, but it's always going to happen because every team is just like, "Well, how fast can you get it done?" And we all want to say, "Yes." We all want to say, "Yes, I'll get it done. No problem. I can get it done in a sprint. I can get it done in two sprints." And you could, but you can't, and you never will.

And capacity planning doesn't take into account, like, well, you know what? I'm going to spend 20% of my time fielding outages and service requests from other teams that interface with me. And, like, did I blow it? Ah, maybe, maybe not, you know. But we're going to have this inefficiency. And it's baked into the system, but, like, human psychology is such that you always want to downplay it. That's my read on it.

DAVE: Yeah, no, 100%. There are some things that when you put them together, they're too big to hold in one head, right? Like, we have software that customers use to onboard leases, literally to apply for a lease, and we give them to you. We have another team that just services the leases for our existing customers, and they are separate teams. They have an entire business domain between them, and our products have to talk.

I want to say, when Acima started out, like, in twenty-teens, early teens, right? This was all under one roof. It was just one gigantic app. I could be wrong about that. But it was one gigantic monolith. I know our merchant portal app has been calving services. It's like a glacier constantly calving sub-bergs out there.

And you've hit the nail on the head that you can't hold all of MP and all of LMS in your head at the same time as a functioning integrated unit. So, let's build an API interface between them. Now we've decentralized these two things a little bit. We've decoupled them, but there's an inefficiency that now hits really hard, which is that the interface into LMS has to be robust and generic enough that somebody who doesn't know the innards of LMS can use it, and vice versa. When they call back into us to look something up, our interface has to be generic and robust.

And when you're all in one spot, you can run into this thing where it's like, "I don't have to solve the general case. I know I'm solving this case, this case, and this case, and everything else can go hang. Who cares?" But if you make this a blind public API and you publish it on GitHub and say, "You can use my library," now you do actually have to, like, assert your domain and say, "I'm handling this case, this case, and this case, but not that case."

And half the time, if it's a public project, somebody will go, "Well, why not? Why don't you just..." right? And then they hand you a, you know, an issue that's going to be, you know, a 7,000-line pull request or whatever. And that's the answer to why I didn't just. But that's inefficient because that doesn't do anything for my three base cases. I'm done. I've got what I need. My itch is scratched, right? Screw you guys. I got mine. But if I want to decentralize, if I don't want to take on all the business of managing leases after they've been created and, you know, servicing lease lifetime, then I have to support a decentralized team, and that has to give them the necessary communication.

KYLE: You kind of hit on what I was thinking in a little bit different way there, Dave. I was sitting here thinking, like, you see the problems with decentralization evolve as you're going from more of a startup to an enterprise level, in my mind. And that, you know, is that where you've got one boss and three developers to begin with. And there's a lot of efficiency. There's a lot of, you know, speed that can happen with that.

But as you age through the company, you know, now all of a sudden, you need to extend this out to, you now need DevOps; you now need IT; you now need QA. You've got all these independent groups with independent goals. And then it goes further and further, you know, and it's stuff like you have people reporting to different departments with different budgeting, you know, different things entirely. And it's getting those two budgets aligned, those two initiatives aligned for the one goal is no longer an easy thing.

I can even remember here, you know, at Acima, DevOps was in engineering. We were next to the engineers. We had the same budget. We had the same goals. Well, those are two different departments now, and we have to get the people above us aligned before any of us down in the trenches that are doing the work can start doing and communicating together to get anything going. There's so much overhead, so much, you know...we've got too many cores, you know what I mean? Like, exactly what Will was saying. There's too many cores going on, and that has created a lot of chatter.

DAVE: Mm-hmm. And it's layers, right? It's cores of cores of cores, right? We've got threads running in cores, in multiple CPUs, on multiple machines, in multiple clusters, in multiple data centers. And this is the [SP] pitch: My thread needs some data out of that thread, but I have to go up to the core, up to the CPU, up to the da-da-da. I literally have to hop over the internet to another data center and then back down the same, you know, ladder.

There was a fantastic paper that came out of Microsoft. They analyzed all of their bugs in Windows. They had plenty of data to work with, right? [laughter] We had a lot of fun poking at that. And they rated, like, how hard a bug was to fix. They, like, "Let's categorize this data. What do we see from it?" And they said, "Far more, by far and away, the biggest thing that would add story points and difficulty to a bug fix, and also something that would be most likely to cause a bug, is when the thing you need and the person who needs it are separated, not by distance, but by layers of the org chart."

If it's my stuff, I just fix it while I'm typing in my file. If it's somebody on my team, I have to, you know, poke you or go stand by your desk or open a Slack call. But if I have to go up to my team lead and then up to the department head, and then up to the engineering manager to go back down to a different department head, to a different team lead, da, da, da, every time you go up a layer, the likelihood of a problem and the severity of the fix...the severity of the problem and the difficulty of the fix, like, squares. So, give me a bug that's 10 times harder in my own code to fix. It's so much easier than a simple one-point story that is 3 layers away, because that's a 16x multiplier on everything that you do.

I thought that was really, really interesting. And I lived the thing that you're talking about. I lived this at a previous employer where they divvied the company up into horizontal slices. And it's so easy to justify this on a budget. You know, it's like, "Hey, if we put all of DevOps into one pool, and they just serviced all the DevOps needs for the entire company, that would be so efficient because we could buy them one big Grafana license." And it looks really, really good on paper.

But now you've got this org chart where this entire layer services this entire layer. It looks great on a PowerPoint slide until you realize that I'm over here on this end, and you're over here on that end. And when I go down a layer, I'm going sideways across five layers. And I might talk to a DevOps person who has no idea what my problem domain is. And that DevOps person has a different budget. Like, you actually said budget issues, right? You're having the...You know, my laptop is starting to bulge because the battery is going to explode any day now, but I'm waiting in line because IT, they're cracking the whip over them because they have to get the Keycloak integration locked down into our, you know, da, da, da, da, da, right? It's the different masters.

The organization that I was referencing before, I won't name names, we had the database team. We had everybody who was embedded. You know, as a developer, I worked everything from the JavaScript all the way to the database, to the SQL stored procedures, all the way up and down. And they sliced out the database layer and said that we now have a DBA team.

And it went wrong in exactly the worst way, exactly the way that you could predict if you understood one social thing. And the social thing is the CTO of the company was a co-founder. He had put in millions of dollars, which means the CEO outranked him on paper by a few dollars and cents, but the CEO could not go to the CTO and say, "Fix your stuff, or you're fired." Okay? The CTO would just be like, "Screw you. We're... I'm saving money."

So, what happened is all of the authority to do anything moved, in the database, moved to the database team. But all of the blame for anything that goes wrong in the database went everywhere else. So, if I took out the database, it was my fault. But if I needed a column added to the database, I had to go, you know, beg and plead from the DBA team, and it was that Microsoft up five layers all the way to the C-suite. You had to go all the way to the C-suite to get a budget authorization/a command of just do the stupid thing, please, from engineering to data. And it crucified us in exactly the ways that you would think.

In a gigantic enterprise organization where everything is sorted, and everything is turnkey, sure, I think maybe it works. But I've mostly only ever worked in startups, and startups, lately, startups moving into the enterprise. Like, Acima is kind of in that boat as we kind of enterprise-ify. But we still have...we have a lot of startup in our back pocket. And that startup means that we have our three base cases and nothing else is supported.

And if we separate the layers, ugh, the data team or the DevOps team, they're now backfilling all these other use cases because our parent company wants them or because our sister company wants them. We don't need it. Who cares, right? It's...We got ours, but we can't say, "Screw you," because you over in DevOps, Kyle, you are answering to a different master, and, hopefully, we stay in business.

WILL: No problem. I mean, and what I think of when I hear these things, right, which I've heard, like, so many times, and the way I think about it, at least, is, like, well, what do you do about it, right? And you have these sort of, like, nobody wants to talk about the inherent inefficiency baked into the system to get a thing done, right?

And so, from my perspective, like, what I think is you just sort of, like, when you are trying to develop a feature that crosses a bunch of different teams, you make a new team, and everybody on the team, right, like, you have a representative for, like, everything that you're going to need to do, and you can give those guys back when you're done with them. But, like, you actually, like, really, like, you take this inefficiency, and you say, like, "All right, well, I'll need to roll this thing over," right? And so, I need somebody to manage the lease, and I need somebody to, you know, somebody from the lease, whatever, underwriting team, you know what I mean?

And, like, I will get a DevOps person to, like, make sure that this stuff is together. And, like, I'm going to get everybody. Like, I will assemble, like, the Avengers team, and it's going to live as long as this thing is alive. And you're going to do that, but, like, politically, right, like, how many VPs need to sign off on this thing?

DAVE: All of them.

WILL: All of them. All of them, right? And so, it becomes really easy to do this thing. And it's one of those things where there's an argument that can be made that, like, yeah, like, enterprise moves so slow. If you come from a startup background and you see the pace of feature development in an enterprise, right? It's glacial, glacial.

In all honesty, right, it isn't, like, it's glacial because, like, you're doing so much, you know? It isn't that way. I've, you know, worked for a few enterprises, including but in no way excepting Acima. And, like, you know, the actual work that you're doing, right, like, if I had carte blanche and all the resources I need, I could probably do my coding for a month in a day, and I don't know that I'd need the whole day. But I can't do it. And maybe that's just the nature of it.

DAVE: Maybe.

WILL: And I suppose, like, the thesis that I've got is that by really accepting and embracing this model where we are going to over-allocate, I'm going to get a body from every team that needs to have a body contributing to this thing. And we're going to give everybody everything you need so that we're not...I don't have to put a ticket in to another team. Like, I just go in, and you're on my team until we get this thing shipped, and we're done, and we embrace the suck with both hands.

And we say, like, "No, this is what it's going to take." And you could get more things done faster rather than sort of overscheduling this blocking to get, you know, a container allocated, right, to do the thing. But every time I do it, every time I want to hit that thread, I got to take this overscheduled mutex, and I'm blocked until the thing comes out.

It reminds me of, like, in the battle days when we had spinning metal disc drives. When you over-allocate your drive, they had a little light, right, that would go on and off. You could see it every time the drive did a seek, and it would kind of chatter, you know, as the, like, kind of physical spinning head on the disc. And you could tell that when you had exceeded the capacity of your RAM because it would start paging out virtual memory onto that disc because you didn't have any memory.

So, it's like, okay, well, I'm going to page this one out. I'm going to page this one in. And that disc would start chattering, and when that disc started chattering, when the disc started talking to you, what it was saying was, [laughter] you're done. Nothing is getting done from this point forward. You need to...somebody's going to have to get voted off the island if you want any of these processes to finish. You're done now.

DAVE: The sound that spooks anybody over the age of 30 on this call is [vocalization]. That's your hard drive going.

WILL: Yeah. But the hard drive was never going to...the operating system was never going to tell you that. It was never going to...you were just going to feel it, and if you happened to be present, and you weren't, like, you know, tunneling into some server somewhere, then you were going to see it, and you'd be like, oh, no. Oh, they're planning my...they're [inaudible 23:25] process. Somebody's going to have to walk now.

DAVE: I talk about law versus lore. Lore is the stuff that they won't teach you in CS class, right? And I remember, I wanted to get into Linux. I was a Windows developer, C++, and I wanted to get into Linux and open-source development. So, to teach myself Linux, I took an old computer, and I installed it. This is back when you put it in a...You know, Linux came on five floppy disks, because it would fit on that. And I remember typing away and hearing [vocalization], and then a pause, and then a pause, and then [vocalization].

And what it was is it was, I would say, a bot. It was like a Perl script somewhere on the internet trying to password, trying to brute force a password. Three password attempts and then a three-second timeout [vocalization], and then a three-second timeout. And you could hear it. I could hear it because the computer was under my desk. I was physically adjacent to it. As soon as they moved everything into data centers, I'm like, "But how will you hear the paging attack [laughs]?"

Will, when you were talking about getting groups of people, do you mean, like, cross-functional teams, like tiger teams?

WILL: Yeah.

DAVE: Where, we're going to put a DBA on your team, so now your team has the authority to make a database change? We're going to put a DevOps guy in it, so you have the authority to punch a hole in the firewall right now and just get your stuff done.

WILL: Yeah. That's exactly what I mean. That is exactly what I mean. Because, again, I feel like we mostly talk about human problems, rarely technical problems. And I always look for, sort of, like, founding fathers checks and balances, style, organizational principles that you can use to combat these sort of inherent psychological weaknesses in planning, in leadership, in resource allocation, in willingness to have difficult conversations.

You can always say, like, "Well, you just got to be honest." And it's like, yeah, man, that's great, but we're on our third round of layoffs at, you know, big co. Everybody's head is on the chopping block, and you're going to go in and tell your VP, "We're not going to ship this thing," you know? You're an intern, you know, who is two or three weeks into your first internship job in a terrible job market. And it's just like, if I do a good job here, you know, I might, you know, have a chair when the music stops, and I graduate, you know?

And if somebody has just given me a completely ludicrous timeline, am I going to say no, you know? There's a lot of people who feel some pressure, especially in the year of our Lord 20 on 26. And when you increase the pressure, which can happen for any number of reasons, you know, the ability of people to speak truth to power is not something that I think we could just say, like, "Well, do the right thing," because the CEO, in one of those giant all-hands meetings, said, "You can trust me with the bad news." And they meant it, but, like, also it's kind of a cop-out way to run leadership.

So, you introduce these sort of institutional checks and balances so that you eliminate the ability for, like, managers and, you know, all these people to say, "Trust me, bro." Because the optimistic scheduling winds up being, you know, you get your least resource-overloaded team sort of setting the pace and the expectations for the organization as a whole, because they said they could do it in two weeks, and they can, hopefully. But, like, what about your security compliance officer who's six months back? If they're a blocker, they need to be a blocker for a reason, but they're overscheduled.

You have a team, and it's like, "Hey, that security guy's on the team. He's on the team until it's done." And then you go to the security guy, or you go to the security guy's boss, and you say, like, "Hey, I need him for two weeks," and he says, "You could go kick rocks," you know? And that's how it is, right? So, you just sort of, like, organizational principles so that you can't pick your own pocket. Because the temptation is always going to be there. It's never going to go away.

We're always... I've gone to work every single day of my career for decades now under some sort of pressure, internal or external, either because I want to do more, or I'm overscheduled, or I'm just an anxious person. I want to do everything I can, and usually the call is coming from inside the building, but, like, I don't know. Who among us is different? So, yeah.

DAVE: I think it's also...I think of, like, mentality. I can't remember which business book I read years ago. It might have been The Dip. It might have been one of the Seth Godin ones. But he talked about how...the author talked about how, in a startup, you need mavericks. You need boat rockers because we don't have a golden goose. We don't have a cash flow stream. Your job is we're going to turn a whole bunch of hungry programmers loose in the desert. And we're going to ride hard, stay late. We're going to sleep rough. And we're going to catch a goose, and we're going to bring it back. An enterprise company is the opposite. They have a golden goose, and your number one priority is don't hurt the goose, right?

I joke that Merchant Portal is one of the rougher codebases I've ever been on. So, somebody was like, "Man, you're so negative about this." And I'm like, "No. No, I'm not." And I had to go back and explain. It's like, I'm not negative on this. We are a multi-billion-dollar company. This codebase is our cash cow. That is freaking awesome. And so, it's a target-rich environment, certainly, but it's not just, like, messed-up code for the sake of having messed-up code. It's code that made us money, made us money, made us money.

The developers that wrote it rode hard, stayed late, slept in the desert in a sleeping bag, and now we're all in our air-conditioned apartments with, you know, healthcare and, you know, a human resources department, and don't mess up the goose, right? So, our job now is kind of to preen the golden goose and get it, you know, keep it fluffed and safe and healthy.

The thing I found interesting from the book is they talked about when you move from startup to enterprise, there's this pogrom phase, where you have to get rid of the martyrs...sorry, you have to get rid of the mavericks, not the martyrs. You turn them into martyrs. You have to get rid of the mavericks because they, by instinct, want to rock the boat, and rocking the boat is how you tip the golden goose out into the shark-infested waters. That's a weird metaphor.

Yeah. So, it's like, I think sometimes you have people that want to get rid of the mavericks, but then if you go too far with that, you end up with everyone reports to eight bosses, and now nobody's in charge of the project. The only thing that got distributed was blame, and authority has been jealously guarded, and everybody's covering their own butt behind.

VIVIAN: Okay, I just wanted to jump in here. From everything I've heard you guys talk about, as much more experienced engineers than I am, it feels like there's kind of, like, a distributed graph system that we're building. Like, in a standard enterprise environment, there's, like, one CEO that is in charge of the CTO that kind of distributes authority down from there.

And so, at the end of the day, like, whoever is in charge of making the decision is the one responsible for it. But what that results in is a lot of isolation between teams, a lot of disconnect, a kind of inability to move forward and make decisions and actually kind of, I don't know, as you guys are talking about, innovate and make big decisions. And in a startup environment, it's more like a mesh network where every single thing is connected to every single other thing. Each engineer knows every single other engineer, knows what they're good at, knows what they're bad at, knows where they can jump in, knows what they can do.

And because they're not protecting anything, they can, as is the common phrase, like, move fast and break things, because there's nothing to break. When you break something, it just breaks what you're building. It doesn't break the golden goose, that if you lose, you lose the company.

So, I guess I'm kind of curious how you find the ability to continue to innovate, to continue to create these mesh networks that actually drive innovation, that drive this progress forward, while maintaining a level of accountability to where each person knows who they need to be accountable to and why they need to be accountable to that person, while still being able to talk to anyone to get the things done that they need to get done at the end of the day.

WILL: Amy, and, I think, foundationally, you don't. It's a different animal. It doesn't work the same way. You don't write multi-threaded versus single-threaded code the same way. You don't do it. It doesn't work that way, like, I mean, and that's it.

So, what do you build, right? We could talk for an episode or two about, like, taking a majestic monolith, right, a unitary program, right, which Acima was, I've seen it, you know. And then sort of slicing off sub-processes, slicing off business units, breaking it up into pieces that still work, right, and kind of wing walking your way through that as things expand, and leadership changes, and you get bigger and bigger and bigger. All those things are just engineering and people engineering challenges. You know, you need to do things.

And I think another aspect of, like, we talked about, like, this mesh network, right, where everybody can talk to everybody else. One of the things, I mean, in terms of, like, a mesh network of people is, like, one of the most important things you accumulate as you gain tenure within a company is a network of connections to people who will actually do anything [laughs].

And a horrible heuristic, which nonetheless I have found to be true, is the number of people who actually work in any company is the square root of the number of employees of that company. And everybody knows who they are. And one of the things that a manager, a good manager, at the very least, will do is utilize those people to the best of their ability without burning them out because we all know somebody who, you know, if you've got a problem, you call them, and they're going to make something happen for you.

But, you know, everybody knows that person, and they will just burn them out. They will use them up. And, you know, you always want to be one of those people. You know, I'll say, like, you know, as an old senior dev, right, like, try very hard to become known as one of those people because they will always eat. But, I mean, managing that, managing that load, right, managing this sort of, like, this one node in your network that is just getting absolutely dogpiled with requests is, you know, an organizational challenge.

DAVE: I think Kent Beck talked about how to, like you were talking about, Vivian, of taking organizational, like, a mesh. You've got this...We want all the ability to move through it without the kind of the defects of it, right, the bogging down, without just immediately going to centralization and command.

And the interesting thing I find is that if you go read, like, Extreme Programming Explained...this is a book from, like, 2001, like, it's a really, really old book. But he gives, like, 11 principles of extreme programming. And they are all based on taking something and dialing it up to 11. And they all, in my opinion, address these exact problems. It's not just waterfall. It's the organization that is meshed in with the waterfall. So, the, you know, constant communication.

Extreme programming says, on-site customer. An on-site customer literally means the person who wants the software is on the cross-functional team with you 40 hours a week. They sit in the bullpen with you so that you never have that thing of, "Do you think they want this?" It's like, "I don't know. Gary's sitting right there. Ask him," right?

YAGNI, the principle of YAGNI, which is, You Ain't Going Need It. It's the build the three base cases that you need, and then don't worry about the other base cases. Now, if you're building a public API, you have a little bit more of a going to need it than that. But if you're building internally, Kent put it really, really well: trust tomorrow you to deal with tomorrow you's problems. Don't try to solve tomorrow you's problems today.

And, I think, it was Sandi Metz who put the best postscript to that ever, which is, because tomorrow you always has more information than you, and tying tomorrow you's hands is not doing yourself any favors. Find ways to be kind to future you by leaving yourself the ability to change choices without locking yourself...And we like to lock things down and tidy them up and say, it's clean. And then tomorrow you find out that, well, we put this in the wrong place. Really wish we hadn't welded it. Really wish we had just, you know, set...because we didn't need to. We just needed to set it down on the end of the table. That's [inaudible 36:39]

WILL: I don't know, man. I don't know about tomorrow Will and what he's going to be up to, but past Will was a real piece of shit [laughter], and he screwed me over and over and over again.

DAVE: Dude, like, legitimately, legitimately in the last couple of years, I made it a conscious principle of try to be kind to future Dave. I am not kidding. I have experienced a couple of times this year the sensation of, "Well, past me was kind of a nice guy [laughs]." Like, legitimately, like, feeling happy that past me...I did have to start to hold grace for myself though, because you'll get a bug. It's like, go write this, and then you go to that spot, and it's just a blank file. Like, who never wrote this? Oh, I never wrote that.

Well, what moron wrote this without this one...Oh, past me said you ain't gonna need it, and here it is 9, you know, 9 months or 19 months later, and we haven't needed it until now. We might have gone for another year. We might have gone until forever without actually needing this. So, we tend to judge very harshly.

What was the...it's very difficult to predict the future accurately, and, unfortunately, it's very easy to hold someone accountable for not predicting the future correctly. And when I look back at past me, it's the same thing, right? It's like, past me was a jerk. I'm trying to get kinder to future me, and as I do, over time, past me is turning out to be less of a jerk. And that's kind of weird.

VIVIAN: I mean, this is a bit of a tangent, but I've found that, especially when it comes to, like, schoolwork and things like that, the more I treat future me as an entirely separate person, that I'm, like, holding myself accountable, too, like, "Oh, future me is going to be really mad if I don't take care of this," it almost adds a level of responsibility that makes doing things a lot easier.

Like, it reduces that friction of like, "Oh, do I really have to do this? Do I want to do this right now?" It's like, "No, I'm accountable to future me because future me is going to have three more things on her plate that she's not going to be able to have time to do the things that I want to do now." So, that habit has helped a lot in recent years, and especially with, like, taking notes to make sure that when future me comes to look at what past me has done, there's actually a record of it so that I can reference it and move forward growing rather than shrinking.

DAVE: Yeah. I rewired a basement years and years and years ago for a friend, and they sold the house. And one of the things that we did, sorry, not the...Sorry, networking. We pulled network cable. And we were pulling it through the walls, and that meant we were going to switch boxes in various places. And there were spots where we were literally going through the junction boxes in the walls.

So, in every single junction box, I printed out a copy of the network diagram and circled, "This is this node," like a "you are here." I folded it up and put it in the junction box, not next to the wire so that it would...I wasn't putting tinder against the spark gap. But I put it into every single junction box, and I forgot all about it. 15 years later, I bumped into the person who bought the house from me, and they were like, "Thank you. We had a problem go wrong with the network [laughter]. And I pulled the junction cover, and there was a map of the network in there." And it's like, yeah, it absolutely solved it.

It was kind of a pain to do it at the time, but, like, I realized that...I'm not going to...you know, it's...Here's a weird off-skill from this as well. When you get good at writing things down, folding them up, and putting them where future you will find it, you give yourself permission to un-remember that thing. You give yourself permission to forget that thing, and that clears up space.

There is a...Eddy...we were in a meeting years ago, or a year ago, and I asked a really stupid question. And Eddy was like, "Man, you are fearless about asking stupid questions." Like, yeah, it's because I'm real stupid. But it's because I actively pick things to forget. It is a hugely powerful career skill to correctly identify the things in the future that future you will need you to have done in the past, without over-anticipating and trying to, like, complete all the things because that's wasteful.

If anybody is listening to this and wants to pick one thing to improve on, on being nice to future you, don't lock the door unless you are doing security work. Don't weld down the interface and require seven layers of ceremony if you don't actually need it. It feels nice and complete to tidy up. It feels like...in RSpec, drying everything up into your lets up at the top of the file feels nice and tidy, but it's a garden path. You're being led down a garden path because future you's going to get there.

You go to the bottom of the RSpec file, and you're like, "What are all these variables, and what are their roles? Gosh, if I had just left them, you know, in every single example, left them duplicated, then I would be able to read this one example and understand the entire context." My map of the network would be in every junction box in that case. Identifying that is more art than science and possibly more black art than regular art. But it's a huge career booster when you can manage that because it frees up your capacity. It frees up CPU power in the future you because you don't force future you to carry that cognitive load.

VIVIAN: There's a concept in note-taking that I've kind of grown to enjoy quite a bit called, like, a second brain. It's fairly common. And there's a...the YouTube channel, I think, is called, like, No Boilerplate. But he talks about essentially something that you've actually talked about, too, like, if you do it more than, I think it's twice, automate it in some capacity.

And the idea is that with a second brain, you're essentially creating this system where anytime you're thinking of or needing to keep track of something more than twice, you write it down, and then that's future you's job to look it up. But it's always on the responsibility of past you to have written it down. And so, over time, you can build trust in yourself, and in, like, the past you, so that you know that when you write something down, future you is going to have forgotten it and still be able to access that information because you can trust yourself.

And again, like, it frees up so much cognitive load to not be needing to be trying to remember every single piece of a schedule, every single piece of a note from a meeting that was three years ago that has that key piece of information that you need. That kind of decrease of cognitive load frees up so much of the other focus that you need to do.

DAVE: I love that you said trusting yourself, learning to trust yourself, where you go through that discipline. Once you've been on the other end of it, and the thing that you've received from past you has been the exact map you needed in the exact junction box when you needed it, you just get this, "Ah," feeling.

And then the next time you're in that situation, and you're writing something down, you write it down, and that little niggling worry in the back of your head of, "Do I need to remember this?" goes, "No. No, I don't." And just, "Ah." You can feel the CPU cores in your brain drop ten percent because you literally just let it go, and it frees up cognitive load. It frees up CPU power for the problem that you actually have to deal with today.

MIKE: I really rely on that pretty heavily with my to-do lists. Because if you have to remember it all in your head, that takes a ton of effort, and you're going to forget something, and it's stressful all the time. You write it down, you know, you can get to it. You know the place to go. You're good.

WILL: I've got a lot of... I have a lot of notes. I bought an e-paper tablet just to write notes out longhand. But I very rarely read it [laughs], you know, like, I just write it down. Like, occasionally, you know, occasionally, what I am notorious for doing, and one of my deep abiding hatreds of Microsoft Teams and, like, sort of, like, corporate lawyer retention policies, is that, like, I am currently working for a large organization where their lawyers have extorted the engineering team to have their chats expire at about, I think, three months, which is leadership malpractice on a apocalyptic level.

There's so many things that, like, you just... It was, like, a message. [inaudible 44:56] message from that team lead [chuckles] where, like, he was saying that you got to put this phone number in for this system. And it has to be a palindrome because somebody was the weirdo [SP] artist, and that's just...he really liked palindromes, but, like, if you do it like that, then this system, you know what I mean?

And it's just these pure weirdo things that, like, I will remember to write down on the magic notebook. But, like, it's not easily indexable like you hope Slack and maybe Teams because right now, as we were speaking, I was referencing a conversation, which fortunately has not gotten down the memory hole, between us, our head of QA, a third-party vendor, tier three, I don't know, customer service, or whatever's above customer service team, and, like, a project manager who quit two weeks ago.

And I was like, "God, it's somewhere in there." And it's about why we can do this thing in this particular pre-production environment, but not staging and not development, but, like, because of the configuration of this third-party vendor's service. And I just knew it was in there somewhere, and I dug it up, and I added the guy to the chat, but, like, I'm not going to lie to you, like, this happened in May. In August, it's all gone. Tears in the [inaudible 46:33], baby. It's gone [laughter].

DAVE: Yeah. It's gone. Yeah. There was this sci-fi thing that I read or watched years ago about protecting your IP. The accountants and the lawyers would be so happy if when you leave the building at 5:00 PM, if we could force you to leave the laptop and everything you know about the project behind the door. Like, we'd just erase your mind on the way out the door, right?

And when you talk about the...I understand what the lawyer was thinking. The lawyer was thinking, "If we get subpoenaed, we don't want things going back forever," right? I get it. That lawyer does need to be slapped, though, because literally going through and cauterizing the organizational knowledge base to do that, that thing about going out the door and having your brain wiped, this was not aspirational. This was a dystopian warning [laughs].

WILL: Oh, yeah. Yeah, yeah. Well, that's the plot of Severance, right? The Apple TV original TM. Get some sponsors in here [laughs].

DAVE: I love it. I love it. We have ranged pretty far afield today. We got anything else we want to cover on this? It's been an interesting ride.

VIVIAN: I mean, I just wanted to make a point of something that I kind of noticed was a little bit funny. The topic that we were going to be discussing today was decentralization of authority. And without Mike to kind of keep us on track, to hold us accountable, we had a wonderful conversation, but did we end up talking about or coming to a conclusion about decentralization of authority?

DAVE: Right.

VIVIAN: No. So, it kind of proves the point of why, at the end of the day...

MIKE: [inaudible 48:03]

VIVIAN: Although you can accomplish really great things with a lot of decentralization, if you're trying to accomplish one specific thing, that might fail if there's no one to kind of keep you on track.

DAVE: I love that.

VIVIAN: To keep things moving forward.

DAVE: Yeah. Did we have the wrong podcast, though, right? Did we...We didn't have the podcast we thought we would, but did we end up having the podcast we found out we needed, right? Be the podcast they deserve, not the podcast they need. No, it's the other way around, isn't it? So, yeah, get the quote right.

WILL: So, I really...I mean, I feel bad that I didn't get my rant on extreme programming in.

DAVE: Mm-hmm. I didn't intentionally beat you to the post on that one, but I'm with you, brother. Solidarity.

WILL: I mean, I mean, I'm just saying it's a solution to all these problems.

DAVE: Mm-hmm. Mm-hmm. That actually might be a good place to wrap up then. Thank you, guys, for coming out. Thank all y'all, and thank you for listening.

And this has been the Acima Development Podcast.

...more
View all episodesView all episodes
Download on the App Store

Acima DevelopmentBy Mike Challis