Acima Development

In this episode of the Acima Development Podcast, host Mike and a panel of returning guests—Eddy, Dave, Kyle, and Ramses—engage in a reflective conversation about the importance of engineers and technical professionals understanding the broader business context of their work. Mike opens with metaphorical stories about differing perspectives on the same task and how having a broader view—like knowing you’re building a home, not just hammering wood—can significantly impact the decisions one makes. He compares this awareness to navigating without landmarks, emphasizing that understanding the “map” or business direction helps avoid disorientation in both literal and professional contexts.

The conversation pivots to real-life examples from the panelists’ careers, including Dave’s story about a freelance project where a critical bug caused a dangerous hardware malfunction, reminding him that technical choices can have serious, even life-or-death, consequences. The group discusses how engineers often work in silos, focusing on technical excellence while missing business priorities like revenue, customer needs, and risk management. Mike shares a story of how strategic engineering choices—like creating permanent ad spaces on news sites—prolonged a startup’s life. The discussion illustrates that understanding how a company makes money, or what problems are most pressing, empowers developers to make impactful contributions rather than waste time on low-value tasks.

Throughout the episode, the panelists stress the value of communication between technical teams and business leadership. They discuss the dangers of neglecting foundational issues—using examples like collapsing buildings and broken gondola cables—to highlight the long-term risks of ignoring maintenance and risk mitigation in favor of short-term gains. Eddy and Kyle bring up the role of auxiliary teams (QA, DevOps, IT) in cost-saving and risk reduction, further reinforcing the theme that everyone in a company, not just engineers, benefits from understanding the business landscape. Ultimately, they urge technical professionals to pop their heads up, ask “why,” and align their work with what matters most to the organization.

Transcript:

MIKE: Hello and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting today. I have with me a great panel. We've got Eddy, Dave, Kyle, and Ramses. They've all been here before, so, hopefully, you’ve heard their voices. And we're going to be digging into a topic.

Actually, there's a huge windstorm behind me. We've got a great editor. You probably won't hear it, but if you hear occasionally a weird sound, that's what's going on [laughs]. And let me start with some stories. And I'm not going to introduce the topic. I'd like not to introduce it because I think the stories often speak better than a title.

So, imagine three people, well, three people who are at a Habitat for Humanity site. We actually have... that's a charity that Upbound, our parent company, does a lot of work with. I can't remember how many houses they built last year. It was quite a few. Like, wow, they built that many houses [chuckles]?

DAVE: And Habitat is a charity that builds houses, right?

MIKE: Exactly. Exactly. They build houses for people who need houses. You know, imagine you walk in, you know, you're a reporter. You walk into a Habitat for Humanity site, and you find some people working. You know, yes, you walk up to the first guy and say, “Hey, what you doing?” He says, “Well, I'm nailing this two by four to the floor.” Fair enough, right? You’re like, okay, you know.

So, you go, and you talk to the second person like, “So, what are you working on?” She's like, “Well, I'm raising a wall here.” It's a little different perspective, right? They're working on the same job, but she has a little different perspective. She sees that [laughs] nailing something to the floor is part of a bigger project.

You go and you talk to the third person, you know, “What are you doing?” She says, “Well, I'm building a home for somebody in need.” Those three perspectives are very different. And the third person who, you know, this woman who sees the bigger picture, is going to make different choices because of her awareness of that broader situation. I can tell another story here. So, you know, put a pin in that one because I think it's really important. The choices that she's going to make are going to be affected by her understanding of the bigger picture.

Another story. So [chuckles], I lived in the Midwest for a long time [chuckles], and when I grew up as a kid, I was a landmark navigator. I learned to find places by looking at landmarks, and that actually works okay, right? That works okay, unless you get to a place where you can't see a landmark, and then you end up in a lot of trouble. I [chuckles] actually have, in my neighborhood, we got some friends that...a child who was walking around the neighborhood and couldn't see the water tower anymore, and they were completely lost. They were, like, 12 years old, you know, [chuckles]. They were the point where you think, well, really, they got lost?

Well, if you're used to navigating by looking at this prominent landmark and that goes out of sight, you have no point of reference. And so, you just wander around aimlessly, hoping you'll see the landmark again. It's not a great way [chuckles]. It's not [inaudible 03:29]. I've had to learn to think differently. In my head, I always know which way is north if I am going someplace I haven't been because, that way, I know I'm not going to get lost because I know where I'm at on the map. I know which way is north. I know that, you know, even if something goes off a different way, like, that will probably go the direction I need to go. And it makes a huge difference.

And I've become a much more successful navigator as I've learned that, to learn the, you know, the lay of the map before I even set out because it makes a big difference as to the choices I make. And I do a lot of cycling with my kids; I've mentioned that before. And, you know, we do exploring. Sometimes we go places we've never been before and knowing where we're going makes a big difference, especially, you know, if you have to reroute, you better know where you are. And always knowing which way is north, knowing where you are on the map makes a big difference. It's a huge deal because you don't get that situation where you're untethered. Well, where do I go from next? I have no idea because I can't see the landmark.

DAVE: I love that you said change the way you think, that you became aware that you had to change the way you think. I grew up in Southern Utah with, you know, Canyon red rocks. I grew up in Moab, which is right outside of Arches National Park, so all these gorgeous red rocks. So, I always know that when I'm looking, you know, at the Blue Mountains, I'm looking at the La Sal range and that's east. And so, I grew up with very thick, I mean, geologic landmarks. Nobody's going to move that building.

And then, when I was 19, I went to Puerto Rico, and I spent 7 months living in Ponce. And I've got these mountains, which is on the south side of the island, south coast. So, I've got ocean to the south of me and a mountain on the north. And I was absolutely fine, and I was there for about eight months. And then, I moved to Bayamón, which is on the north side of the island, and I was lost for a month, like, actively disoriented every single time, because the sun was coming up in the west, and I couldn't figure out why. It was, like, genuinely jarring. It took me a surprisingly long time to realize that I have a thinking error that has me 180 degrees turned around.

So, I love that you said you became aware that you had to change your thinking. I love that.

MIKE: And I did, and it was hard, to be honest. It took a while. It took a while. I'm old enough that I used to navigate with paper maps, you know, this is pre-internet navigation days. And you would go to the store, and you would buy a booklet, I say booklet, but it's sometimes more bookish than booklet-ish. And you want to go somewhere? Well, you look up the street name in the back, and it tells you a page number and a letter and a number. You turn to that page, and you find the grid, and within that grid, you find the place that you're looking for. And then, you expand out, and you might have to...and then each map, you know, you see the direction of each side. You had to figure out how to navigate [chuckles] maps that way. It was --

EDDY: But of course, Mike, you did that without driving, right?

MIKE: Yes.

EDDY: Like, you had your own copilot. Okay [inaudible 06:32] listeners.

MIKE: Before you drive.

EDDY: [laughs]

MIKE: You'd see a lot of people in those days with the booklet open [laughs], you know, like, on their knee. Where am I going?

DAVE: Or pulled over on the side of the road with the map unfolded on the steering wheel. Yep.

MIKE: Exactly [laughs]. You know, you’ll learn to do things differently when you have to find things in a different way. So, I'm going to talk about software. About, I don't know, 15 years ago, we'll say, more or less, approximate [laughs], I was working at a place. It was a startup, and there was a big shakeup in leadership, and we basically...we went for about a year where we had no leadership at all for our engineering department. So, what do we work on? And we had to make a choice. We had to make a choice.

And, you know, a lot of times we as engineers don't think about making that choice. We don't think about the broader business perspective, right? So, we think, well, what is going to make this startup successful? In the end, it wasn't successful, so maybe it didn't matter. But we actually prolonged the life of the startup by several years because of what we did. We were building a...it was a website builder.

We worked with publishers like newspapers. And we knew that they needed more flexibility in publishing, and we reworked, well, there are two things we did. We reworked the interface for publishing, which, you know, made that a little nicer. And, honestly, that was the worst choice we made, the worst of the choices we made, because yeah, it made it look better, but the functionality didn't change that much. And when you’re a company that's struggling [chuckles], then a little window dressing is probably not the right answer.

We also worked on a project that allowed you to put a permanent ad, allowed companies to put a permanent ad on a newspaper, which is actually a huge deal because newspapers are one of the most reliable sites in terms of reputation from browsers, from search engines, I'll say, from search engines, not browsers, but search engines. So, if you get linked to from the newspaper, well, that makes you look really good.

And so, what we’d do is we’d allow a business...and this is just taking the newspaper model and putting it online. We'd allow them to have a permanent ad, like a car dealership, you know, car dealership puts an ad in the newspaper. Well, now they have a permanent mini-site within the newspaper that they can go to that not only has their advertisement there but also links back to their website. So, now their website when you search for cars in that area, well, they're the first one that comes up because they're linked to from the local newspaper.

We built a whole system for them to build out those little mini sites and that carried the company for, like, years [laughs] because making that choice, again, we didn't have somebody telling us to do this. We had to make the choice based on feedback from the business. We had to suddenly put on our product hat. I learned a lot from that experience about how important it is to know what's going on in the business.

We could very easily have said, oh yeah, we're just going to make this perfect interface on the inside, make it pretty, which does nothing to make money for our customers. And that would have been very much the wrong choice. We did enough of it that, you know, that there was a cost, and we probably shouldn't have done it at all. Thinking about what we should have worked on made a life-or-death choice for the company.

And that's what we're going to be talking about today is, why we as engineers should know about the business. Why should engineers care about revenue? Why should engineers care about any of these things that the business usually is concerned about: prioritization, money, gathering requirements? I'm going to argue it is a big deal, and one of the key differentiators between engineers who are really successful and those who maybe end up lost quite a bit.

Dave, you've already run with this a little bit. Do you have any more thoughts or anybody else? You know, what are your thoughts about this idea of knowing about business, knowing about the business and its inner workings as an engineer?

DAVE: I have so many stories I could go into, and if you know me personally, you know that that's...I can often not be persuaded to not tell them. I'm going to try to stay concise. What seems to me is very essential is that you need to know, what do we need to succeed? And you want to close that feedback loop. And a lot of times, we just trust. We trust in the society, and the society in this case could be just the ecosystem at your business. I trust that my manager is doing the right thing. It might not be. They might not be.

I lost a job once because the accountant couldn't keep the books straight. You had one job, and that was to save mine [laughter]. And so, the reason I have a ton of these stories is I spent about 15 years as a freelancer, contractor, ran my own shop for a while. And we would go in, and we would fire jump Rails projects like firefighters that jump out of airplanes right in the mess.

I was the guy that you would call if you had a website built by somebody, you know, by your nephew who's, you know, a high school student or whatever. And it kind of works, but it's 90% there, and it doesn't...and it's inefficient, da, da, da. And so, I would jump in and then untangle this and make it work. And I usually...because I was a contractor, I often had an expiration date on my hiring time. I would go into a system like I'm already dead. I'm just going to work really hard while I'm alive, you know, fiscally alive or dead at this company.

And because of that time box and because it was...like, this company they've backed into the meat grinder a little bit, and they're like, “Oh, we have this website that we need to make money.” This is going to sound so obvious that it will sound stupid that I'm mentioning it right up until you realize if you look around the room, nobody is paying attention to this. And that is, if the company doesn't make money, they can't make payroll. If they can't make payroll, you don't have a job. It's absolutely critical that you get that sorted out.

I will append something to this, which is Uncle Dave's career hacks. The most powerful trick I've ever learned from my career was look at the person one level up of you and figure out what does that person need to look good and to succeed? I want to look good to my boss. The best way you can look good to your boss is to make your boss look good to their boss. And if you go high enough up the hierarchy, you're going to reach the C-suite. And, at that point, what's important to them is staying solvent, making money, right? There might be a mission. There might be, you know, noble goals and all that, but if there's no margin, there's no mission.

And so, figure out where the money is going, figure out where your business model is, and be prepared to embrace it and engage and be willing to step down if the boss says, “No, we're not doing that way because you're a technical person. You might not understand the business.” But you need to know the business exists and not shoot everyone in the foot with some bad code.

I'm not sure how perfectly this adapts. I don't know if anybody needs to hear this, but I will add this one bit to it. I worked with an engineer, about 20 years ago, who was amazing. He was a senior in college. And while he was working with us, he got out of college and graduated and [inaudible 13:57]. And this guy had a work ethic that would not quit. If you...well, no, he had a work ethic that would not quit from 8:00 to 5:00. And at 5:01, he was out the door, and he'd go home. But you could trust him to write 1,200 lines of code every single day. You could just about set your watch to this person, very, very smart.

In Dungeons & Dragons terms, he had a very high intelligence, not necessarily a high wisdom or not a very high experience level. He could solve any problem from first principles, but he had to solve every problem from first principles, if that makes sense. And I described him to a co-worker at one point saying, “If you tell him to go plow that field, he will go plow that field. You'll get up in the morning, and you go out and you look and that field will be plowed.” But if you accidentally told him, “Plow that field,” and you pointed at the interstate, you're going to come in tomorrow, and there's going to be a field plowed right through the interstate.

MIKE: [chuckles]

DAVE: And this is that. It's the same thing, right? You need to be an expert at what you're doing. But you also need to be paying attention to, you know, am I hammering a nail correctly, or am I building the house correctly?

MIKE: Yes. You know, I'm going to ask a follow-up question. What sort of choices...so you talked about, you know, plowing the wrong field [chuckles]. What are some of the choices you get into? So, this might take a moment to think about. What are some of these choices where it really matters to understand why you're doing what you're doing?

DAVE: Man, I don't know if I have good examples. I think I've told the story multiple times about the time I got a bug report that began with, “Fortunately no one was killed, but...” that happened.

MIKE: [chuckles]

DAVE: I was building a hardware controller that would shake things to vibrate them. I'll tell the story very, very quickly. It would shake things very, very quickly to do squeak and rattle. And Toyota had one of these machines of ours, and it's like a sound card. You play out waves, and you listen to the feedback from the accelerometer [inaudible 15:43], so you can tell how much you're shaking the car. And we had a bug that caused energy to store up in the system and then suddenly be just released all at once.

And we bent the frame of the car, like, we literally launched it. I say we launched it; this was a car. We launched it, like, up in the air, like, six inches and then over a foot and then back down, which was enough that everyone on the shop floor was wearing their brown trousers at the end of this event, right? Nobody got out of this without, like, holy crap, we could have been hurt, or we could have been killed by this.

Up to that moment, I was just designing data transfer. I had these beautiful acoustic wave patterns, and I knew all about frequency distributions, and they were elegant. And I had ways of generating random data that could then be massaged to fit a pattern but still be random, all this lovely, lovely stuff. And then, we got that bug report, and I just realized there are people's lives at stake. If I screw this up, somebody could die, and that sobered me very, very fast, and, interestingly enough, made me that much more excited about my work because, all of a sudden, what I was doing was important, and was relevant, and there was meaning in it.

I hadn't thought of that. Oh my gosh, the value of knowing where the money's coming and going isn't just that you're going to help protect the money. It's going to get you up in the morning and get you out of bed and excited to go to work. And if you've ever been in a deathmatch slog or had a difficult time at a company, you know that meaning is precious. Finding meaningfulness in your work can be very, very precious. And yeah, knowing where the money is, fantastic. And you won't get pushback from the CEO if you're making them more money.

MIKE: Side story. There was an engineer who worked on a side project for a couple of months, just on the side, and came back and said, “I've got this thing that solves the problem.” I'm not going to go into the details of it, so I’m deliberately being a little vague here. And we looked at it like, um, yeah, that does solve the problem. And it's made, like, tens of millions of dollars in less than a year. It was a critical piece that made an enormous difference. And it's because this engineer looked and said, “Hey, there's a problem here with the thing that is most sensitive in bringing in revenue, and I know how to fix that. I know how to address that problem.”

And so, he looked into the most important problem, came up with a solution, and it made a tremendous difference, a small project with enormous impact because he was thinking, well, what do they need most? That wasn't life and death [chuckles], but it sure makes a huge difference to the trajectory of the company.

DAVE: Oh, yeah.

MIKE: And you're talking about meaning, that is it [chuckles]. Getting up in the morning, like you say, it is...some people may be highly motivated to get up and dig ditches every day. But I think most people want to know that they are building something profound [chuckles], that those ditches are the foundation, right, of some great edifice. And, again, you know, you can be hammering nails, and then you go, and you leave, and you say, “Yeah I nailed some two by fours.” Or you can be the person who built a home and that woman who built a home is going to come back, and she's going to build another one because it meant something.

DAVE: I grew up in a small, rural conservative town in the ‘70s and ‘80s, and I was raised on the general concept that police are your friend, like, Officer Friendly. We literally had skits and movies about Officer Friendly. And as I got older and started getting my first speeding tickets, I was, like, not as happy with the police, at least the traffic enforcement part of it. And a friend of mine was the son of a highway patrolman. And I happened to overhear this officer get up and go...sorry, highway patrol, so it would be a trooper.

This trooper got up and on the way out the door, he said, “Well, I’m going to go save some lives,” and walked out the door. And that's all he was doing. He was speed-trapping on the freeway; making people slow down so that they would stay alive. He didn't see it as like, oh, I can't wait to just confront angry people and take their money. No, that had nothing to do with it. It was literally keeping people safe.

MIKE: I have a friend who was a police officer who ended up in a life-and-death situation [chuckles] and made a choice that ended up sparing the life of the person he was in a confrontation with. And the mother of this young man, you know, just came to him in tears, the officer, in tears afterward. “Thank you for saving the life of my son.” Her son was doing something wrong [laughs], but he managed to address the situation without resorting to lethal force. And he had a mission in mind, which was to keep people safe and alive, and because he followed through on that, lives were changed.

DAVE: Absolutely.

MIKE: Now, in software, as you pointed out, a lot of times it's not life or death stakes. Maybe you're working on pacemakers, yeah, maybe that is critical, right, or anything medical. But, you know, doing things with money is a big deal. And it's very important to not disempower people by, you know, separating them from their money in a way that you shouldn't. And having the meaning where you come in and you’re like, well, you know, I'm providing financial opportunity for people in a fair way really does make a difference. It keeps you going.

Again, I’m going to ask the question again. Dave, you've answered a little bit. What do other people think about? Are there times when you thought, oh, I need to make this choice? And I've got some ideas in mind, and I can throw them out there. But are there others who've had decision points where they had to think about what the business is doing and it made a real difference?

EDDY: Well, I think having the domain knowledge is critical in order to know what you're building, right? If you don't understand the business use case of whatever you're being asked to build, right, you're isolated. And so, you don't have the blueprint of, like, how the user experience is going to look like.

I would argue, though, I kind of ask myself a question, if what I'm working on is going to make the company money, and if it isn't, is it going to provide value in any way, right? And so, trying to find a balance between actively working on something that will pay dividends versus actively working on something that will sustain the software so that it does keep making it money [laughs], you know what I mean? So, I wonder if we can sort of dive into that a little bit. And how are we able to find the balance between understanding that the company that you're working for needs to make profits, but at the same time, right, needs to also maintain its libraries?

MIKE: Absolutely. I was going to dig a little bit further there. So, you're finishing up the ticket. You don't have one [inaudible 22:59] line. What do you do?

EDDY: I think we naturally gravitate to CI or bugs.

MIKE: Bugs. [crosstalk 23:09]

DAVE: What needs doing? Go look at Jira. What needs doing? Yeah, what's in the backlog?

MIKE: Your CI, you know, there's a lot of things that that could mean, continuous integration. I think you're talking about continuous improvement. You're going to go pay down some tech debt, or improve something, or work on a bug. And why is that? Why is that your default? And I think that's the answer probably for most people, but why is that?

EDDY: It's a great question. I don't know. I think I normally gravitate to that to maintain busy, like, busy work.

MIKE: So, I think that the reason we do that...I think there are some reasons; work comes in different sizes, and a lot of times the strategic work we're working on. Well, the business says, “This is the big thing we need to build.” We work on that. It's a big thing, right? We get to the end of that, and we’re like, well, I know there's another big thing coming along in the next sprint or, you know, in the near future, and I'm done with that big thing. Now I need a little thing, right [laughs]? I need something to fill in that space. Because if I start a big thing, I'm not going to get very far because then I'll have the real big thing I need to work on next.

So, we try to size it, and we say, “Okay, well, what's going to fit in that space?” And so, we try to go for something smaller, and I think that's the right thing to do [chuckles]. I think we go and we grab, okay, well yeah, I'm going to grab something small that'll fit. And that's just something really important because there are usually a few things in that size, right?

So, do you do something that shores up the foundation, or do you do something that's kind of cosmetic? Do you do something that solves a customer issue? Those could all have valid reasons to work on them. And if you don't know the reasons, you're just going to pick up something at random. And it suggests something about not just the engineers working on it, but, you know, the leaders, product organization, trying to provide priority here and trying to provide a map. You know, we talked about maps, giving a road map so that you know, well, this is actually what's most important. I need to build up the foundation here because foundation is eroded [chuckles], right? There's a weak spot here, and losing the foundation is not good.

You know, there was a hotel, and it was a hotel or a condominium, I think it was a condominium, in Florida a few years back where the parking garage had been eroding for a number of years and then finally collapsed, which ended up taking down the whole building.

DAVE: Wow.

MIKE: It was tragic. People died. And, you know, they'd known about it for years. They were like, “Wow, that is some exposed rebar there with water running over it. That's probably not a good thing [laughs],” but nobody did anything. And there's another life-or-death situation where it wasn't life or death until it was. Shoring up that foundation really is important. So, that may be the right thing to do, and often is the right thing to do because a lot of times the business can't see it, right? They're looking for the grand aspirations and don't always see that water running over the rebar.

EDDY: So, let's go a little extreme here, right? So, you mentioned, unfortunately, people died because of that restructure. Basically, what you're saying is, okay, if we work on fixing up our foundation for this building, it's going to cost money, right? And it's not going to give us a direct or an immediate profit, right? But your justification is, yeah, but then when this building collapses, you're going to lose more money trying to build a new building. And then, you have all these legal litigations that you have to deal with.

So, I think that's sort of how you can justify working on stuff, where you're like, it's not going to immediately make you money, but it reinforces our framework so that we don't have to go into those critical masses and having to invest more. [inaudible 27:11]

MIKE: Absolutely. And it's hard for the company to keep on that, especially if you've been, like, a startup, because if you're a startup, the answer may be, “You know what, we built that last year. It'll hold for a few years. Don't work on shoring that up because if we don't get revenue, we go out of business, you know, we lose that solvency.” And so, it can be a difficult decision. And then, when you do get solvent, and you've got some money, sometimes the default becomes, “Well, let's just keep building on top.” And you have to make that transition to say, “Well, no, we have to make that investment,” and it's important to voice that. Dave, you were going to say something.

DAVE: I was going to say, I was talking to somebody a little while ago...a long time ago, about 10 years ago. They were talking about how when you get a request from business to say, “Hey, I want to do this,” you should ask them “Why?” and you should follow...it's called popping the why stack. “Oh, it's for that? Okay, well, why do you need that?” “Oh, it's for this.” “Okay, well, why do you need this?” And you keep going until you get an answer that matches one of three categories. It's either going to make us money, or it's going to save us money, or it's going to reduce risk of a catastrophic loss of money.

And that third one is the...because nobody wants to pay to shore up the rebar in the garage when the only money to be made is you get to keep making the money you're already making, so that's a siren song to just, you know, not worry about it, not worry about it, not worry about it. And you can get in a situation where you push your risk off and off and off, and eventually, your risk management strategy becomes hope. And, you know, how hard can you hope? Because you're going to have to hope pretty darn hard because that's the only thing we're doing is just hoping it doesn't break. And it's terrible when that happens.

There was an accident...somebody else will be able to find this better than me. A tram car...one of the gondola things that go up the mountain in Italy fell down the mountain. Like, literally, the cable snapped, and it was near the top. And it was the other side, so the car stayed suspended. But it was now the only weight on the cable, and it flew down the thing. It killed everyone aboard. And the people that maintained it came out and said, “This is a terrible tragedy that nobody could have seen.”

And I remember an engineer sitting down and saying, “Let me give you five reasons why this was criminal negligence, starting with that cable cannot possibly fail until it is visibly frayed in...” Steel cable, you can literally break, like, half the fibers in that cable, and as long as you space them out, the fibers adjacent to them will support. You just about have to cut the thing to ribbons or shred it to rags to get those cables to fail. They're incredibly fault-tolerant. And this cable failed. It literally snapped that cable. And you can't have that happen.

If anybody had just gone up and looked at the cable in the previous five years, they would have immediately said, “Oh man, we have to fix this. This is way too many threads broken per foot, or whatever it was.” There were two or three other systems that also failed, and that engineer then proceeded to say, “And let me tell you why I know they weren't checking the braking system, and why they weren't checking the safety arrest line,” because that also broke. And yeah, just all the way down the line.

It is so tempting to just look at that freight cable and go, “Replacing that cable is going to cost the company a million dollars. I do not want to be the engineer that tells the CEO that we have to replace this cable,” but it was necessary. And some people paid with their lives because they didn't want to save that money, or they wanted to save the money, instead of didn't want to spend it.

EDDY: So, what you're saying is, working on that it's justification for saving the company money in the long run, is what I'm understanding.

DAVE: And there's an art form to that, right? There are times when you're polishing code, and you’re really satisfied. Oh, I want to use this design pattern. I'm going to tuck this in. It's going to be so satisfying to have this. We've all written code that you just sit back and go, “That works, and I like it,” and it makes you very happy. But you can end up polishing rivets, and you think you're doing all this noble good. But what you're really doing is something that makes you feel good but maybe doesn't help the business.

And that's, I think, the point of this discussion is pop your head up and look around and say, what direction is the company going? Is this important? Is this essential? You might find, you know, oh, these classes aren't namespaced correctly. That's a technical debt thing that's going to put us all out of business in 3 weeks. No, it's not. You're fine. We don't need to stop development for 9 months to just change around some scoping rules. That's probably not the severity of risk.

EDDY: What kind of scares me a little bit is when you have, let's say, for example, you're a construction company, and you make a bid. And you're like, “Okay, cool, no, we can make this building for you in three months,” right? And then, you're reaching the deadline, right? You're a few weeks away, and suddenly, you're like, “Okay, we're reaching the end, right? This is going to hit the budget that we have proposed.”

So, suddenly [chuckles], you get murmurs saying, "Eh, you know, this base foundation that's going to keep the building afloat we don't really need that right now.” But it would be nice to kind of go back and say, “Hey, by the way, this building can have tenants, and it's fine for the next couple of years. But if we don't go back there and add this foundation, this is going to crumble.” And you finish, and then, suddenly, you turn the cheek. You work on other stuff, right, you get other contracts. You get other stuff, and then you forget.

Suddenly, it gets forgotten about, and you're like, oh, man, right, suddenly, the building is starting to shake, and you're like, see, I knew that was a problem. We documented it as a problem, and suddenly, we have to go back. And so, I guess my concern is, there is a judgment call at that point, right, on whether it's worth taking the extra days that it would take in order to do it correctly. But what are the repercussions for doing that? And I feel like sometimes business tends to do that, you know, so you have to find that balance as well.

MIKE: Well, and you just suggested something. You said, the business wanted to do something different. Well, the business is made of people, and we're part of the business. We have the obligation to communicate. “Hey, this foundation is unstable,” because somebody in a different place in the business can't see that.

Think about that parking garage, the security guard, right, who's wandering around the garage. They're going to see that every day. The remote owner of the building is not and has no way to see it. And unless the person whose job it is to be looking, right, and maybe their job isn't even to look at the foundation, you know, they're just looking to make sure that there's no thieves. They're in a position to see something. And we're in that situation sometimes, where we have to bring up this uncomfortable fact. You know, there's a security issue, and if we don't deal with this, there's a lot of risk.

It's uncomfortable, and it's easy to just not worry about it. But in the end, you know, the financial viability of the company depends on it. And if you've got leadership that cares about the right things, and I'd like to think that most people do, you know, that always can deviate, right? But assuming that you have capable leadership, they will, I think, in general, appreciate the fact that you told the truth.

EDDY: So, what do you do then if you have to tell your client, right, “Oh, you know what, the projection that we told you about, three months, you know, it's going to actually three and a half months.” What repercussions do those conversations have?

DAVE: Coming in from contracting land, I’ve tried to develop a habit of if something's going to go wrong, I want to tell the customer as quickly as possible. Because if I find a catastrophic error, like, if we literally, you know, we did not consider that this architecture literally cannot accomplish the task that we are doing, if we continue, we have thrown away the money up to this point, if we continue, we're just going to keep throwing good money after bad.

But I also try to present solutions. We had that same problem writ small here at Acima. I had a task to make a database change, and it was just a normalization change, shouldn't have been that hard. But we had left that change for seven years, and so we had this massive First Normal Form problem in some of our data. And trying to extract it, there's all these reports from the data team that they're looking at the data in the wrong place, you know, in that First Normal Form. And to extract it, it's going to put it in another table. That's going to break all the reports. So, I had to sit down and think for a minute about, what can we do to keep the system running the entire time that we're fixing it?

And when we had a, you know, when business comes in and says, “Stop working on that, start working on this,” we had one of those. And I had gotten this refactoring to a point where you can talk to the object like it's in the new pattern, but you can keep talking to it with the old pattern. And the First Normal Form continuously sinks into the third normal and vice versa that if you update this other record, it will update the master.

It's inefficient. It's not great. We shouldn't run this way forever, but if we had to, we could because the system works, and that data is in the right place. And I'd love to be able to circle back, and I might not. I'm not anticipating that I won't be able to. It's near and dear to my heart. I would like to see it finished and, you know, I will love that and adore that topic and push on it with people. But if it never comes up in the budget codes again, then, okay, well, it's probably not as essential as we thought. We went seven years on that First Normal Form thing, and we could have just added two more columns to that table and been done.

It would have been a one-day story and, you know, I turned it into, you know, a two-month tilting at windmills. But, and this is your fault, Mike, I was in the meeting where I said, “This is what happened. This is the problem I ran into. This is how we could fix it.” And I think I actually presented, we could just do it the bad way and just be done, or we can clean it out. And, to this day, I remember Mike saying, “Thank you for doing this the right way.”

I was nervous going into that team meeting of, like, man, I might get yelled at because I'm proposing we blow the budget wide open on this, and that's above my pay grade. And it was nice to have other people in the room saying, “No, I absolutely support. Do this the right way. We need this done the right way. We’ve put this off too long.” But I had to run it. That's my point is, I had to communicate it up the chain because I don't like it when technical people make business decisions, and I don't like it when businesspeople make technical decisions.

If you're a one-man company, you have to do it all yourself, but businesspeople know what the value of something is, and technical people know how much something is going to cost to build. And I don't like it when businesspeople say, “I want you to build this, and I want it to take two days.” Okay, you're welcome to want that as hard as you want. But I'm the technical guy, and I'm telling you this is a seven-day task.

I've also had times where the reverse is true. I come in and I say...Michael on our team I walked in, and I said “Hey, I saw this little problem over here. I can fix it in, like, three hours. Let me just cut a ticket real quick, and I'll turn around and burn it.” And Michael just immediately said, “No. No, that's worthless. This is all wasted money. Please, don't even think about this anymore.” And I'm like, “Oh, okay.”

Because from a technical side, it made a lot of sense, and it was easy, and it was going to make our job a little bit easier, eh, maybe. But I found out that business we're not, you know, that this is supporting an initiative that is no longer important to the company. So, you know, please, there are more valuable things you could be working on. I'd love for you to, even just for 2 or 3 hours, just give them over here to where we need the budget. So, yeah do communicate is what I'm saying.

MIKE: Yeah, that was a fantastic question, right? You know, what do you do when there seems to be misalignment? Going back to our central idea, you can't do that unless you have some awareness of what the business goals are, right, and at least some high levels going on. Maybe you don't know the exact financial status of the company [chuckles], and that may not be important. But you do know that this thing is broken, and I can communicate that and balance that with other concerns.

And you may know that the company said, “Well, right now, we care a lot about revenue,” or “Right now, we care a lot about yield on our existing assets, you know, because we've got good things, and we want to make sure that they work well. We're not going to put in a lot of new investment.” Or, you know, maybe we care about risk management. Or maybe we're a startup, and we're desperate to get a new client, and we need to have a pretty web page [chuckles] that makes the client think that we're great. It can be any of a number of things. And understanding those priorities makes sometimes all the difference as to which choice you make.

DAVE: An adjacent thought to that is I realized that what frustrates me the most about work is when the person in charge of the capacity to make the correct decision [SP] lunches the call, makes it wrong. I hate it when I make a technical call and I'm just flat wrong. I waited that database refactoring. I waited into it fully cavalierly thinking I was going to have this done by Wednesday. And, no, you're not going to have it done by March, think again. And I hated that because it was a technical call, and I made it wrong.

But I've worked at a shop, and some of you people have heard this story. There was a shop that I was at that brought in a VP that said, “I want this. I worked at this other company, and we used this thing, and we want it. We want to give it to our customers because it's fantastic. Everybody in the world is going to want this.” And the whole engineering department dropped what they were doing for a year to go work on this, and we shipped it, and nobody wanted it. It was just DOA absolutely. And the VP lost his job for that. The company took a big hit on the stock price. And, like, when they found out where we'd wasted all that money, nobody wanted it.

Amy Hoy calls it kosher ham, where even if you could make it, there's no market for it. [chuckles] You could say, “Here's some ham, and it's kosher. My Jewish friends, come eat.” And they're going to look at you and go “No, we don't do that, even if it's kosher. I don't care. I don't want that.” So, you'll see the same thing, right? It's like, if you're going to build a report for somebody on the data team and they're saying, “Can I get that as an Excel spreadsheet?” And you're like “No, no, no I can do this. I can do this all for you with a JSON transformed in GraphQL.” Even if it worked, they don't want it. They want a spreadsheet. They want their Excel so they can do it.

And it's the same thing, right? Know what the customer wants so that you're not accidentally...I guess the definition of kosher ham is if you did it perfectly and did the best possible version of this, it would still not be what the customer wanted. And it hurts when the business lunches that call.

MIKE: And we can help. Now, leaders who are listening to this, there's an added burden, right [chuckles]? Communicate early and often. Do that market research. Find out whether that's something that actually will work. Think it through. And sometimes we're going to make wrong choices. But try to minimize the impact of those where we can, right? And get feedback all the way along. Be listening so that when the people who are working with you tell you something, you hear that information and don't just go on your solo mission because it really isn't just you.

Well, I think that we're reaching a good stopping point here. We've shared some stories. We’ve shared some figures of speech and metaphors that connect things to, you know, to other aspects of life to show just how important it is to know where you're going, where you are as you build something. Any final words?

KYLE: Just one thing that I was thinking is because this, I feel like, has been a little bit more specific to developers and new features and how that can bring revenue into the company. Just if you're in an auxiliary position, meaning, like, QA, or, you know, DevOps, roles like that, IT, something like that, always keep in mind, what can you save the company? There's, you know, there's always improvements. There's always, what can you save?

And, you know, you might have a great architect or engineer that maybe doesn't know about a newer technology that's coming out that's cheaper or, you know, there might be newer processes that you can put in place to save the company money. And a lot of the time, especially for these auxiliary roles, that's just as important, if sometimes not more important than some of the work that's being done to bring in additional revenue.

DAVE: That reminds me of the central tenet of, like, agile programming when we were first telling people 25 years ago “How about we don't give you an estimate, how about we don't tell you how long this is going to take. We're just going to tell you what we're working on, and what we think is the most important next step.” And the allegory they use or the analogy that they use is, it's like a deck of cards. And everything you work on, every task you work on on this product should be one of these cards. And you want to stack the deck so that the most important thing is always on top, that way, you know you're not wasting your time padding out, polishing rivets when the engine hasn't been finished.

And there comes a point in your development when you hit kind of, like, around the 80% mark, where you start realizing this is a working MVP. This is a minimum viable product. We could ship this if we had to. And it gets you out of that mindset of, no, we can't let the customer see this until every pixel is perfect, and balanced, and we need the thing, and it's got to adapt. Otherwise...look, we can probably get you 80% of this product in 30% of the time if you just let us work on the most important thing first. But all the other parts have to be there.

You know, the feedback loop, understanding the business, talking to the people that are in charge of that the communication is essential. That goes back, by the way, it goes back to what Eddy was saying that if you're in a shop that's very rigidly working from, like, a story backlog, when you finish the task and you don't know what to work on, well, you do. Go look at the backlog and take the top card. If your product owner is prioritizing, you always know what the next most important thing is, and that is very, very satisfying when you can get that, when you're tired and you don't want to think through the whole project. You know, I could just work on that. We already did this work.

MIKE: Well, thank you. Thank you, everybody here for your input. And, hopefully, our listeners will think a little bit more about what's going on, you know, around them. What am I doing? What am I doing here? And based on that, what should I do next? Until next time on the Acima Development Podcast.

The latest episode of the Acima Development Podcast takes a deep dive into the evolution of DevOps, platform engineering, and the concept of ZeroOps. Host Mike, along with panelists Justin Caldwell, Kyle, Justin Ellis, Will Archer, and Eddy, discuss how infrastructure should function like a well-designed vending machine—providing a seamless interface where developers don’t need to worry about the internal mechanics. The team explores how DevOps has transformed over the years, moving from traditional sysadmin roles to cloud-driven automation, enabling developers to focus on delivering code while minimizing manual operational tasks. The discussion introduces ZeroOps, a modern approach that seeks to eliminate the need for infrastructure management by automating processes and providing a frictionless experience for development teams.

Justin Ellis provides historical context on IT infrastructure, explaining how organizations transitioned from manual punch card systems to today’s highly automated cloud environments. The panel highlights best practices in infrastructure management, emphasizing the importance of standardization, automation, and observability. They discuss the shift from managing infrastructure as “pets”—custom, hand-configured systems—to treating it as “cattle”—standardized, scalable, and disposable resources. Tools like Terraform, Kubernetes, Docker, and Prometheus play a key role in creating resilient systems that support modern software deployment strategies. By implementing these practices, companies like Acima have been able to scale deployments from quarterly releases to multiple times per day, improving reliability and developer efficiency.

The episode wraps up with a discussion on future innovations in DevOps, particularly the pursuit of fully dynamic environments where developers can spin up isolated test environments on demand. The team acknowledges challenges like managing stateful applications, dealing with legacy infrastructure, and ensuring effective observability, but remains optimistic about advances in AI, cloud-native tooling, and automation frameworks. As organizations continue to refine their DevOps strategies, the goal is to build systems that are not only scalable and efficient but also reduce complexity and operational overhead, allowing developers to focus on building great software.

Transcript:

MIKE: Hello and welcome to another episode of the Acima Development Podcast. I'm Mike. I am hosting again today, and I've got a panel with me here today of Justin Caldwell, who's coming to us from the cloud engineering DevOps side. Kyle who has been on here a lot before also from DevOps. We've got Justin, Justin Ellis, who has been here before, and we've got Will Archer. And we've got Eddy.

As usual, I'm going to tell a story to lead us in here to kind of launch our discussion. Oh, I don't know how many years ago, a lot of years ago [laughs], I worked at a sheet metal shop where we made bubble gum machines. You've probably seen a vending machine where you put in a quarter, and the bubble gum ball does a lot of things before it makes it down. We made, like, roller coaster-type things. So, the bubble gum ball would drop in the top, and it would roll around and do some loops and stuff, and it would come out.

And it's interesting to actually make one of those. There's a sheet metal around the side. We'd use the wire, and we'd actually stretch it out so it would bend better [chuckles] and curl it around a jig. Anyway, there was a lot of work that went into it. And then when you run it, you go through all the work to see that bubble gum ball roll all the way down the machine and come out the end. It's exciting to watch. People do it not so much for the bubblegum ball, or maybe, but to watch all of the machinery going on inside because it's fun, right? People like to watch that.

And the interesting thing about this is what you get out at the end is exactly the same as if you didn't see the machinery at all [laughs]. In either case, you got a mechanism that drops a bubblegum ball, and it comes out. And the interface is exactly the same. They probably don't have that many coin-operated [chuckles] vending machines anywhere, you know, a lot of times, you put in a card, but same deal, right?

You have a mechanism. You press the button or turn the dial, and bubblegum ball comes out. It really doesn't matter what happens in the machine. What you want to have, and this is really important, you don't want to have a weird vending machine where you have to put in something other than the quarter, right [laughs? You don't want to have some vending machine where it asks you for something in a foreign currency, for example. Like, well, that's kind of cool, but I don't carry pesos, right [laughs]?

And this is kind of the hard thing about switching over to cards, like a lot of things have done. Well, do I have a card with me? Do I have a coin? And now it's kind of flipped to the other side. Like, who carries cash [laughs]? So, it's flipped. You want to have that common interface because, otherwise, it's awkward, and nobody knows how to use it, and then you have to know what's going on inside.

If you have to know how the machine works, you've failed. The idea is to have this as a simple interface. You don't care what's going on inside. Of course, with a bubblegum machine, you care a lot because that's the main reason you're looking at is to watch the ball come down. But in the end, if it doesn't work inside, then that's a fail. What you want to do is have this simple interface that never changes. You just want that bubblegum ball to come out.

And I say this because it applies directly to how, well, to a lot of things, the interfaces between systems and software. Today we're specifically going to talk about the integration with DevOps. You may call it by different names, whatever interface that you interface with to get your infrastructure. Because there's different ways of handling this. There's times when we have people embedded into the team, so each team has their own person, and they build their own stuff. There's a way of approaching this. It's something that's called ZeroOps, where you just press the button and the thing comes out [chuckles].

DevOps provides a common interface that everybody needs to use, and nobody needs to know how that bubblegum makes it down the track because it simplifies systems so much. And we've got Justin here who's been passionate about this idea for...we were just talking on the pre-call. I've worked with Justin for 15 years [laughs], so he's been passionate about this for a long time and has built this more than once and made multiple companies really successful in terms of infrastructure because of this idea.

And I'd maybe ask Justin first to take over from here and give some thoughts, kind of launching from [chuckles] that introduction. And tell us what you think the interface with a cloud infrastructure, however your infrastructure is, maybe you have an onsite infrastructure, what is that interface supposed to look like? And why is that important?

JUSTIN ELLIS: Yeah. Thanks for the introduction. I have thought about this problem really for the past 15 years about how to build systems better, how to make software development easier. On the system side, you're thinking about availability, reliability. You're thinking about security. You've got to think about a lot of different aspects of it.

But at the end of the day, really, our main job is to enable software developers to be able to deploy code, push code out, manage their code. And they don't really care about the mechanics underneath, right? They have a job to do, and then we have a job to do. And I think about if you go back to the very first instances of kind of our field, in the '70s, people used to write on punch cards. And they'd basically create their punch cards. And then, they would create a huge stack of all these punch cards, and then they would pass these punch cards to someone called an operator.

And that operator had a very specific job. They had to, A, keep those punch cards in a very specific order. And then, they had to feed them to the machine very carefully and keep the order. You couldn't drop them. You couldn't have any of those issues. And then, the program would get executed, and then they would get the output, and then they would help return that output back to the user.

Things have changed quite a bit, and I think there's been some major movements. There's kind of those traditional IT systems where you'd have sysadmins. And then, that kind of moved into the cloud world when AWS was introduced to the world. And that also kicked off this new movement called DevOps, where it was like, hey, instead of passing it to the operator and then having the operator do everything, let's get everybody kind of involved in this process and enable developers to be able to pass this code and then also understand what's happening, what's getting tested, kind of move that forward, right?

And then, since the DevOps kind of movement mindset idea, however you want to call that, that's kind of moved into, you know, SRE is still a big style of managing systems, site reliability engineering. And then, that's kind of moved into now this new model which is very similar and kind of inherits from a lot of the previous models, which is kind of platform engineering.

And the goal on platform engineering is to create kind of a standardized platform that developers can interact with. They can put the gumball machine in. They can put the quarter in, and then their code gets built, tested, make sure everything goes well, and then that gets output to a system that you run.

So, long story short, we've gone through a lot of different movements, and I think that we're still perfecting and getting to a better place. And I think every single iteration has had new innovations and new additions to the systems, things such as orchestration, which we have with Kubernetes and all of that kind of stuff.

And so, through that whole journey, I think we've learned a lot, and I think we're still, not early days, you know, we're well into it. And we're trying to provide kind of a standardized platform the developers can develop against. But it has been interesting to kind of see that evolution over time, and then see kind of new processes, new ideas on how these things get going and get worked.

I will say the one interesting part of our job is that there's not a standard way to do things. I guess that's how software development is, too. There's not one golden path that you can just say, "Hey, just take this golden path and do that up." It all kind of depends on your requirements, and the people involved, and all of those kinds of things.

MIKE: Thank you. That was a very comprehensive answer.

JUSTIN ELLIS: [laughs] Okay.

MIKE: But I think a lot of us…you talked about some of these old days with sysadmins and punch cards. My father-in-law's career actually was a computer operator [laughs]. And he would move the tape reels, you know, the magnetic tape from one place to the other and ran this big computer inside a bank. It was a thing [laughs].

JUSTIN ELLIS: Yeah.

MIKE: And the thing is, having to know about those details, having to know specific details. I remember, back in those days, I would be configuring a server. I remember one time configuring a server, actually, over to my parents-in-law's house (It was, like, a two-hour drive.) banging away on the computer on a remote connection, configuring this server along the way.

And every time you do that, it's going to be a little bit different than the next one because you make some little difference, right? And especially if you've got physical hardware, maybe this one's not exactly like the other one. And so, you have this pool of snowflakes. Every one is a little bit special, and every one is a little bit different. And managing that requires somebody with deep in-depth knowledge, who knows how those systems work. And that person becomes a point of failure. But then every system is hard to work with. It's brittle. It's very challenging to expand. And you're talking about a very different way of thinking about it, Justin.

JUSTIN ELLIS: Yep. Yeah. Yeah. Kind of within the whole DevOps world, we talked a lot about pets versus cattle. And the idea here is that you can have a pet, which is your pet server, that you've got to go in and you've got to care for. You've got to patch. You've got to update. You've got to make sure that everything is going well.

And then, once you have 10 servers, once you have 20 servers, once you have 50 servers, it's the same thing as having 50 cats. It's a lot of work to keep up on it. It's a lot of work to continue that forward. With the cattle idea, it's like, okay, what is an application? What are the types of workloads that we run in our system? And can we standardize, and can we build to a specific type of target?

Now, there's been a lot of tools introduced like Docker containers that kind of allow us to kind of define those targets and define where we're going so that every application that we build, whether that’s an application or a service, that could be kind of bundled up and then pushed out in just the same way as all the other services. And then, we can kind of scale those out independently.

So, you might have one service that has a lot of traffic on it and another service that has very little traffic. This allows us to think about those applications very similarly, build the same tooling for how we do that, but then you can kind of scale those independently. So a lot of power there.

MIKE: You know, I'm going to ask a question for not Justin, or not Justin Caldwell here, and maybe particularly to people who've been around for a while, who've worked with systems that do not treat things like the bubblegum machine, like the cattle, like just hand it off and it works. What are some of the failure modes that you've seen [chuckles]? Because that kind of provides some context as to why this is so important.

JUSTIN CALDWELL: So, I worked for Fidelity Investments for a number of years on their identity and access management team. And we would have a team of DevOps engineers who were very, very familiar with their login infrastructure, and that was all they did, and it was their cat or series of cats.

And we would only be allowed to do an install once a quarter, and it was nuts because we'd have to schedule it out. We could only do it from starting at around midnight, our time, and we only have a window until about two or three o'clock in the morning. And if we didn't make it during that time frame, the install, we would have to start the rollback process and try to get it another night.

But the thing about it was that only they knew all of the arcane secrets. And only they knew, like, where all the servers were, where they physically were, and what version of whatever was on there. And we were at their mercy in terms of an application team. And if any of them, God forbid, if any of them left the company, you knew that that next install was going to be crappy [laughs]. So [chuckles], it was nuts. And that's my little story. Yeah.

JUSTIN ELLIS: And I think it's great to have Kyle here because, you know, I'm trying to tie it back to the gumball machine as you kind of watch it go through. But as a developer kind of pushes some code, they also need to understand what's going on in the system. They don't need to know the fundamentals, but they need to understand, okay, how is my build going? How did the deploy go? Once it's deployed, is it successful? Are there problems kind of going forward?

And so, kind of a big part of the platform, a huge part of the platform and something that Kyle has been focused on for quite a while, has been kind of how we observe it once it's running in the system and everything's going well, but giving feedback to the developers. Because developers are best suited to understand what's going on in their application, best suited to understand what bugs come up, or, you know, maybe that last deploy didn't go as well as we wanted to.

So, I do see our job as kind of also completing that feedback loop so developers can understand what's going on in their system and all of that kind of stuff. And that goes back to also this other idea that's kind of fundamental to continuous deployment, which is application ownership, where engineers are really the ones that are running their applications in production because they know their code best. They're the best person to kind of address those. But having engineers kind of involved in the process, again, they don't need to know the fundamentals, but kind of involved there.

So, that's Kyle. Kyle has built an amazing observability stack that allows us to kind of see that gumball as it moves through the machine, which gives the developers the power and the understanding to see what's going on in the system.

KYLE: I would say, with that, on the DevOps side, that's one of those things where it's almost insulting if the developer can come to me and tell me what's wrong with the gumball machine mechanically, right? I want to know, first, that there's something wrong mechanically with the gumball machine.

If their gumball comes out and it's a result of what they put into the machine and they got a funky gumball out that way, that's fine. They can debug that. But if it's something fundamentally wrong with the gumball machine, that's another side of the coin where that type of information needs to come to us directly so that we can get that addressed so that it is not hindering the engineers.

MIKE: It is true that sometimes the gumballs would literally go off the rails [laughs]. It's a little bit bumpy, and there it goes. And that observability is important. You want to be able to look in there and say, "Okay, it is sitting on the bottom. That did not work." Now, it would be far preferable if the machine could automatically detect that and throw a new gumball down, right [laughs]? It makes a huge difference if you can make a machine that can do that.

JUSTIN CALDWELL: So, one thing I do want to address, classically, DevOps were not viewed as engineers in the classic sense. A lot of them were viewed as, like, oh, point and click like ClickOps. That was the phrase and everything, or back to the mysterious arcane people that have a bunch of bash scripts and things like that.

But in modern day, like, in the last five years or so or maybe even a little bit longer than that, with the introduction of Terraform, with the introduction of other IaC solutions out there, it is nuts to me how much engineering code, like, basically, to me, as a software engineer, I look at what the DevOps teams are doing and I'm like, you guys are just coding [laughs]. You guys are full-on engineers. You have your process. You have automated tools that's checking your coding just like application engineers. And it's, you have your history. You're checking in code. You're doing code reviews.

There is no more ClickOps. And if you are still doing ClickOps, I think people recognize that and they are like, oh, let's move this to Terraform or some other IaC. And it's just like, you're engineering complete systems. You're having data flow diagrams. You have the code that controls all that.

And I look at it like, man, I need to go learn that because you're speaking my language, and it just looks awesome. Rather than putting boxes up on the front end like a web interface or anything new, you're dealing with whole infrastructures and everything. And I think it's so cool to see that work being done that way.

EDDY: I was going to say the appreciation that I have towards DevOps has only been the exposure here at the company. But I hear things, like, terminology tossed around all over the place. Like, that was foreign and still is, right? "Oh yeah, we need to run Terraform," you know, or, like, "Our clusters are weird," and, you know, like, "Oh, that container is really odd or whatever." To me, having a team that understands that at a very deep level and a granular level that's just less things I have to worry about as a developer.

So, when the infrastructure is well built and you have a whole team dedicated specifically for that and it goes smoothly, that's just less things on my plate that I have to worry about. So, I just want to say the value that DevOps provides is enormous.

KYLE: To go along with what Justin was saying there, with this infrastructure as code, I know our team, and I hope other teams are the same way, if you run into something that was done, ClickOps, the entire room shudders. We're all just like, "Oh, that's disgusting. Like, why would you do that? You better have a good reason, right?" Maybe a proof of concept. But if you don't have a good reason, otherwise, it's just like you get chastised. You're like, "Why would you even consider doing it that way?"

So, we have gone completely to this infrastructure as code world. And it is like Justin saying: we have linting; we have formatting; we have code syntax. Some of these are their own specific languages. Like, over here at the company, we use Terraform, and Terraform is historically in HCL. However, now going forward, they're creating support for your popular languages. You can do TypeScript. You can do Python. You can do Ruby. To go along with that, yeah, you are. You're putting Terraform functions and Terraform resources inside your code.

I know here we heavily use, what, Bash, Python, Ruby. We even have some Go. That's a CLI. But the other three, like, those are integrated with our HCL code. It's generating HCL code. It's help functions, you know, those types of things. So, to go along with that, I don't think the idea that DevOps is not engineering anymore really fits. And it is kind of like, so, where do you put them? And I think that's kind of where the community is landing on this idea of we need to call them something else. We need to call them, you know, platform engineers, or SREs, wherever you're landing in that community, but they are engineers now. They're part of the engineering department.

JUSTIN ELLIS: Yeah, we call them SREs at my place. And the other thing we think about a lot is the interface that developers have with us. Now, I feel like we should probably be more mature on the interface side of things, you know, maybe even have a web user interface to allow folks to kind of see what's going on in the system. So, right now, it's all CLI-driven, and then also additional tools like GitHub Actions and Jenkins to kind of provide that interface for us.

But we thought a lot about how do we interface with developers? And one of the things that we've done is thinking about if you're running an application or a service on the platform, how do we allow developers to say, "Hey, I need a new bucket. I need a database. I need a new resource for my thing"? And a lot of times this takes a lot of steps, a lot of coordination between those. And I've always said if developers have to understand Terraform code, or are writing Terraform code, or if they have to know what Kubernetes objects there are, or how the fundamentals of Kubernetes works on a deep level, then we’ve failed our job.

So, we've kind of created what we call kind of an application manifest, which is basically very simple. Well, for some applications, a very simple application manifest that allows developers to kind of list out the resources that they need. And I think that does give feedback to developers, too. So, they can see, okay, what databases do we have? Do we have a bucket? How do we communicate with Kafka? All of those types of things.

So, we kind of want to continue to kind of lean into that style of giving developers that feedback and also empower them to be able to kind of build their applications with components that we've added: security, and guardrails, and all those good things, high availability and everything else. But allow them to say, "Hey, I just need a database. Here's some configuration options for that database," and then we can kind of take it from there. We can make sure that scales properly and kind of goes forward.

So, I think we've done a good job so far, but I think we need to continue to kind of lean into that idea of, like, how do we make that interface for developers as easy as possible to give them legibility to the system, understanding of how their application works? And when the new people onboard, they can kind of see, oh, this is how this application is. This is the database and all the configuration that they have for their application.

MIKE: And you're describing a very proven pattern. You look at the Unix, Linux world; you have tools, and you have a configuration file that's around that tool, and that's the interface that you use. You don't point and click. And if you do point and click, it's going to generate the configuration file. It's a very standard way of communicating. And then, it's configuration, right? It's checked into your version control. It is a form of code, but it's something that is clearly well-structured and that you can learn and fill out the config. Much better than having to speak with the arcane wizards and have them build something out.

EDDY: So, I have a generic question, and this might be a little difficult to answer. But how much control, typically, should you give an engineer, like, a developer, over your configuration file?

JUSTIN ELLIS: Yeah, as much as possible, so as much as possible. But then you still need kind of those checks and balances in there. So, every kind of configuration change that gets made, just like with code that gets reviewed, and PR'd and quality checked and all that kind of stuff, we try to do that same system. So, if people can make a configuration change because they want to make that modification, it doesn't go directly to production. An engineer looks at that, like a PR, understands what the diff is going to be, and then approves it for an application within kind of an environment. So, I always like to empower developers as much as possible.

I've worked places before where, as a software developer, I didn't feel informed, or I didn't understand what other applications there were out there. So, I've always felt we need to empower. But then, obviously, you have other things like security and guardrails and all that kind of stuff and then costs as well. You need to understand like, okay, do you actually need this gigantic database to run this new application? So, there needs to be some back and forth there. But yeah, to answer your question, as much as possible, for sure.

WILL: Listen, I just need 10 more instances, okay? It's the last time I'm coming to you, 10 more instances will do it. This is the last time. Everything's going to be 10 more. And I promise, it's the last time, 10 more, and it'll be cool.

MIKE: [laughs] Nailed it.

JUSTIN CALDWELL: I do want to point out to, Eddy, like the mechanism. I mean, so, from a security point of view, the mechanism for controlling that is through the code owner’s file in your repo. And so, you can go ahead and create a PR that has all of this new infrastructure that Will wants. Will wants 10 more instances, so he goes and modifies that. But the code owners will be, you know, one of the teams will be the SRE team or the DevOps team. And they'll be forced to review that and say, "Sorry, Will, you only get five."

MIKE: And, hopefully, you have somebody...and to go back to Kyle and the observability, they say, "Well, we gave him five, and now it's using half the cluster's worth of resources. Something is wrong here [laughs]." Maybe we should go talk to Will, say, you know, "The problem's not on our side [laughs]. We need to take a look at your code. What is the reason the database is running constantly at 90%? And you've got all this queuing for your requests. Something is not right here." And that conversation can happen. Go ahead.

JUSTIN ELLIS: Oh, no, I was just thinking, you know, so kind of a theme of this is ZeroOps. And just to kind of give you some background, Upbound as a company, generally, we haven't really talked about ZeroOps. This is kind of a new initiative. And I actually have just recently changed roles to kind of focus on ZeroOps specifically. And I think just like anything, naming is hard. ZeroOps doesn't mean that you literally have zero operations. In my mind, it means that you have as few operations as possible. So, that means that we're not running sysadmin-style on servers and keeping those VMs kind of managed. This is more and more moving towards container and orchestration.

So, when I think about kind of ZeroOps across our enterprise today, I've kind of worked on developing that strategy and roadmap for where we're going with ZeroOps. And part of that is understanding where our operations costs are coming from. We kind of have a company where there's different lines of business that have different maturities. As far as kind of platform engineering and infrastructure as code, we have a lot of legacy stuff where things still live on VM. We have a big on-premise structure, right?

So, when I think of ZeroOps, I really do think kind of like increasing our capabilities and moving more towards a platform engineering style, moving towards an interface for developers, for data, for security to allow them to write security rules and then having kind of that same PR process across all of the organizations.

So, just like we've treated developers, kind of start to treat those other folks, whether that's QA or security, kind of moving forward and allow them to kind of go through that same process, where they're able to write new security rules, new QA processes. And then, they have a really clean interface to test out their code in non-production environments and then, eventually, kind of roll that stuff out to production. So, that's kind of ZeroOps. And I'm kind of curious, when you guys hear ZeroOps, what comes to your mind? What is the general feeling about ZeroOps?

WILL: It sounds like ops has been running really good, and the gumball hasn't gotten stuck in a while. And so, some very smart person is looking at a spreadsheet, and thinking like, why do we have all these ops people when ops is running so good? And I feel like it’s people maybe being a victim of their own success, you know what I mean, sort of like, well, gosh, that spare tire is so heavy.

MIKE: [laughs]

WILL: And we don't need it. I mean, the emergency parachute, two parachutes? Why do I have to pack two parachutes? So, I mean, I don't know, I spent a little bit of time last week with the gumball stuck in the machine. It's not that I can't do that kind of stuff, but I'm not as good as you guys are because why would I be if I do it once a year when it's, like, the gumball stuck? Who's got the key? Oh, crap. What's going on?

So, I mean, efficiency is efficiency. One developer could do more work, right? And one ops engineer can do more work. So, you're doing more with fewer people because you have the tools that allow you to have that increased productivity and leverage. But it does give me a little bit of anxiety, a little bit of stomach acid just because I know how good I am at it, and I'll figure it out eventually [laughs], but eventually being the operative word.

JUSTIN ELLIS: Yeah, ServerOps, to me, means...and this might be a way of thinking of it as less SSH-ing into servers, less manual interventions moving towards more automations. So, instead of like, you know, you have a problem that pops up, right? And instead of solving the problem kind of on that local level of just like, hey, let's just get this issue fixed and then move on, it's like, how do we provide a system to prevent all of these errors kind of going forward? So, how do we make sure that this problem kind of never pops up again? How do we build a tool to kind of watch this and intervene, maybe do self-healing? Whatever that may mean, but kind of more sophisticated automation.

And automations are tricky because, especially with interventions and all of that, things can go terribly wrong. So, you have to be very careful and very structured and think about it more of a framework when you do that automation. And then, also, there's systems that are much better automated than others.

We think about kind of stateful systems or stateful applications versus stateless applications, meaning that a stateful application would be something on a VM that requires a disk, and that state is kind of on the disk. And if you need to scale up that server, you've got to go in and change the VM, which is going to require maybe some downtime and all those kinds of issues.

When you move towards more of a stateless model, that means that all the storage is abstracted away from the application itself. And so, that just gives you a lot more tooling and a lot better ability to add automation pieces so that you can scale out. You can address problems, and you can fix those kind of going forward. So, I think that kind of stateful versus stateless distinction and moving more into kind of a stateless direction really does help a lot to add additional automations, and then giving those platform engineers the ability to work within a framework and push our technology forward.

MIKE: There's a couple of questions I'd like to ask, because there's something that you touched on, Justin, that there's legacy systems because that's the reality pretty much everywhere. If you're starting on day one, great, but that's almost nobody, right? Because that only happens on day one. And then, after that, you've got a legacy system that maybe you wish you'd redone on day one. And I'd like to hear about the process. But before we do that, I want to start with a little bit of the goal.

We heard before about a company where you could install or deploy once a quarter, and if you fail that, out of luck, right? Maybe you can do it the next day, but you still get once a quarter. So, I'm going to ask our platform engineer folks here, and I'm putting you on the spot a little bit, but how many times do we deploy a day at Acima? I'm going to ask about Acima line of business specifically. So, how many times do you think about Acima deploys a day? And it doesn't have to be exact, but approximately.

JUSTIN ELLIS: Across all of our applications, I would guess we probably...more than 30 times, sometimes 50 times a day. But somewhere around that range in the multiple dozens of times a day across all of our applications. I would imagine we've had days where we've deployed out a hundred times a day. So yeah, quite a bit.

MIKE: Okay, so up to a hundred times a day. I would argue that that is a much better situation to be in than once a quarter. If I've got a bug, I'd sure like to hear that we're deploying a hundred times a day. I'd much rather hear that than, well, we can get that for you in Q4 [laughs]. It's hard to even express the dramatic difference. And those of us who have kind of seen the industry move here, you saw that happen.

There was a time when we thought, deploying more than once a day, oh, that is my worst nightmare because you knew how bad the deployment process was. But if you actually build the infrastructure, it completely changes the paradigm to where deployment is nothing. And the infrastructure is there; you don't have to think about it. You put in the quarter; you get your gumball. Why would anything else happen? The system just works. And it really is a game changer.

So, you've seen that example of how, I mean, I don't know how many orders of magnitude that is. I'd have to start thinking about it, once per quarter to a hundred times a day. That's a lot of zeros of difference . How do you get there? Because it doesn't start there. How do you take a real company that maybe has some legacy stuff and get them there?

JUSTIN CALDWELL: Yeah, so I would say, if you're in a company where you have a lot of legacy systems; you have a lot of stateful systems; you have a lot of systems that kind of break down, or you may be a place that deploys, once a month, right, and you want to change that. I think you really have to start at the fundamentals and kind of the systems level and start to sketch out how you want to handle your kind of cattle going forward. Start to understand what the patterns are across all the applications.

And then, my recommendation is always start small. Start with maybe one cluster that's kind of a new cluster. This may provide orchestration. But use the mature frameworks and platform tools that we have today, and maybe transfer that into a stateless application that you can run, right? Follow that process. Think about the fundamental abstractions, about what is an application? What's a service? How do we think about scaling this out? How do we secure this right? Take that one application and try to modify it to get to that state. And that's not always possible, right?

But there are always applications that you can do that on, where you can start to think about abstracting it out into its fundamental pieces and fundamental parts. And that may mean, okay, let's get this database off of a VM and maybe move to a managed service like RDS because we're not Postgres experts, or we're not whatever else. But we know enough to handle a Postgres instance on something like Amazon, or AWS RDS or GCP database. So kind of extract out, think of it in pieces, use modern tools.

So, I think that would be my general strategy when tackling those legacy projects. And then, sometimes if you're running like, say, commercial off-the-shelf software, you have no choice. You always have to run a VM, right? Think about you might have to migrate to a new tool to get to this more stateless fast-moving thing.

And then, the last thing I'll say, and I know I'm going long, but it's just, when you do have deploys that are spread out by weeks or maybe even months, what you're going to do is you're going to have a lot of changes into that, you know, if you think about all the changes that a developer does across an entire month, and then you think of all the developers across all the teams making changes to an application, those changes build, and build, and build. And then, you're going to deploy that out, which means you're going to have a really good idea of how this works.

Once you have multiple applications talking to each other, it's a complex system, and you're going to get side effects that you never anticipated and you never thought about. So, when you have that deployment style, you're going to run into more failure modes, and they're going to be larger failure modes, where if you deploy once a day, twice a day, three times a day, you're going to get that quick feedback. You're going to make those changes smaller. So, you just see a lot of benefits by moving quicker and having a solid rollback strategy.

WILL: Yeah, but also, I would say that I think rapid deployment, generally, general best practices, and for sure, better than quarterly or even daily. Like, just run it, but what if I happen to put two quarters in the gumball machine? What about three-quarters? What about five quarters? You know what I mean? What about 30 quarters? And here they go.

And this is maybe less of a, you know, if you have your release strategy for production worked out, I think most people going to production still...it's not the Wild West of no matter how good your pipeline is. But for stuff like your test environment, your staging environment, your QA environment, I have seen many very Wild West kinds of situations where it's an absolute free-for-all.

And you lose a lot of time on that, where it's like, why are my GraphQL queries...just no GraphQL today. Sorry. Oopsie. We made an oopsie. There will be no GraphQL today. It's like, okay, all right. So, it's too much of a good thing. How do you manage access to the gumball machine when you don't have an operator, let's say.

JUSTIN ELLIS: Yeah, you know, this makes me think of application size. So, I know there's this big monolith versus microservices architecture, right? Some styles are better suited for certain environments. And I wouldn't ever advocate a really tiny microservices infrastructure where you have these tiny, tiny microservices.

I think there's a right size for applications. I think once applications get too big and you have too many engineers, say, working on a single application, once you get there, then you're starting to think about queuing, and dependencies, and ordering, and all of those kinds of things.

I think once you get to that point, I think it's probably wise to start thinking about, do we need to break this application up? Is this one application that, say, 20, 30 engineers are working on, right, and the amount of code that they run on? Is it time to start thinking about maybe extracting that out into multiple services? So, if you're starting to queue, I would say maybe it's time to start thinking about separating those concerns. I think it's always smart to start with a monolithic architecture and then extract pieces from there kind of as you go.

WILL: Oh no, no, I'm not talking about a queue although that is a problem. I'm talking about I'm developing my microservice, and you're developing your microservice, and Eddy’s got his microservice. And Ramses and Mike they're working on theirs. So, we've got five microservices. So, we're not going to go to production until we have our ducks in a row.

But we're all on the staging environment, the QA environment, and we're all pushing up changes, and things are getting weird because we're all just trucking along hoping, expecting, right? Because all this microservice, these complex applications where we have successfully split them up. But we still interdepend on each other and if everybody is moving in different directions at the same time, like, things are not always…

We've got a microservice architecture, but we're all pushing our changes up to staging and not for nothing. Like, the data environment is big enough and complex enough so that I can't just...it doesn't just fit on a box. I work for a large retailer, and there's a lot of background knowledge around pricing, and availability, and SKUs, and all this stuff which is way bigger than anybody can have. But I have to have it so that I can get my changes out. And I've seen the end state to a lot of this advanced DevOps stuff, and it can be really challenging in a different way.

MIKE: There's something...and I'm not on the platform engineering team, but I've heard something from them [laughs], and I've talked with them a while about this. You're touching on something, and the problem is the shared environment, right? And you talked about the complex data, absolutely. And this is not an easily solved problem, but it's something that we're actually working on. Because if you can come up with a replicable data environment, create this complex data environment, take a snapshot somehow, right? And that may be through some sort of seeding, or maybe you actually do have an actual snapshot that you copy.

If you can get to the point where you can replicate that data and on the platform side you can replicate the environment, then you can get to the point where there isn't a QA environment, spin up your environment. All the other services are known working based on the most recent build that's in production of the other services. And you're just testing your own in that environment. And that seems like the holy grail.

That's something we've worked for for a while. And kudos to the [laughs] platform engineering team for getting there, because we're actually pretty close at Acima. We're actually just kind of right at the cusp. I think we'll be launching that maybe in the next couple of months to where we can have that and then there isn't a QA environment. You get the environment you want and it's isolated. It's a dream [laughs].

JUSTIN ELLIS: I can't tell you how many times...I've been talking about dynamic environments for a long time, but dynamic environments is still the goal. It would be great for QA to spin up a whole new environment, have just your feature branch in that environment. Everything else is on the main branch with data that's there, that's relevant, that's correlated. I agree with you 100%, that is the holy grail. We're working towards it.

But it is a wicked problem. You have to think about the data side. That's one of the primary things. You have to think about how it interacts with exterior systems, say, maybe, like, a third party that you're interacting with, how that data gets all shuttled through the system, so agree 100%.

MIKE: So, there is a way. There actually is a path through this. There's been kind of a through thread through all this is that things that require interdependencies, coupling, dependencies between each other are hard to deal with. Anytime you have something shared like that, it's hard to deal with. And a shared environment, yeah, you're coupled tightly, and you got to deal with that. If you can figure out how to decouple that, so, no, it's not a shared environment. Everything else is not shared. And then, you just got to put up your own thing. It changes the paradigm, right? It switches the way we think about the problem.

Like, you have, oh yeah, well, deployment is hard, so I'm only going to do it every now and again. Well, now I have all of this entanglement of things and all of the interdependencies that I have to deal with. Well, instead of you disentangling, I just put up one thing at a time, then I get rid of that problem. And you have to rethink the way you approach the problem to eliminate the complexity, and then the problem becomes much easier. What were you going to say, Will?

WILL: Oh, well, you know, I had a question. We were talking about stateful versus stateless systems, right? And I think everybody has bad ops in the beginning for the exact same reason, which is a good reason because you can't afford it. You're just getting up however you can. You don't have money for ops. You've got whoever can get their EC2 instance up and running and rolling, and then you go with that. And then, you get ops, hopefully, when you can afford but probably more commonly when it really starts to exert pain and pressure on the organization.

And so, what makes a service stateful, right? Like, this has to be stateful. This is a stateful thing, where it has to be like that. And it isn't so much of like a well, I'm just going to throw up one instance, and I'll just roll the database in with the instance because I only have one monolith, and there you go.

We'll just put it all up on the same box because that's relatively easy to do. But where it's just like, okay, I'll move it off this box. And now I have two boxes, and then we go and do the thing. But what are the really tough nuts to crack in terms of applications that want to be stateful and they don't want to move into that stateless process?

JUSTIN ELLIS: To me, it's all about the disk, right? So, if that specific application and how it's run and that process needs access to, say, one specific disk, if it's like a regular, say, virtual machine that maybe runs Linux that writes files directly to the disk, that's where you run into problems. Because what you want is, you know, stateless is kind of a misnomer in that you are still storing state. It just happens to be in an external system. And a lot of times we'll use managed services for those external systems.

So, really, it's trying to abstract out and push this stuff into a database, maybe into an object storage and kind of getting that out. Once you've got that stateless, then you can kind of scale up that 1 process into 10 processes, and they can all use those same data sources. They can still all use that same state. They can all reach out to the same Redis instance. And then, that Redis instance is the one that's providing that state going forward. And just a lot of applications just generally depend on the disk. I don't know, Kyle, what else? Can you think of anything else (I'm kind of putting on the spot.) that kind of drives the stateless versus stateful system?

KYLE: The one I'm sitting here thinking of is observability-minded, but I'm thinking of Mimir and how they have that, right? They've tried to get out of being stateful. They use an object store, S3. You can use that as your backend, but you still have some of that that is still very stateful. And that's any of the current time metrics that you're wanting to store. But kind of like Justin is saying here, any of that, as soon as you can just move it off into the object store or anything that's managed, then you kind of get it out of the way or out of the way so that you can become stateless to a sense.

I think the other thing that is helpful is generally a quorum. If not everybody can be talking to it, not everybody can be writing to the same table at the same time, or whatever it happens to be, a quorum to know who's talking to it at the current time, because that's another system, or who's going to be accepting even incoming requests.

Maybe you shouldn't be receiving incoming requests to everybody, load balancing. So, anything like that to where that one source of truth, I guess, is kind of your state, anything not to overload that to allow those frontend replicas or whoever's interacting with that stateful process to allow those to become more stateless to a sense. Hopefully, I made sense on that spiel [laughs].

JUSTIN CALDWELL: Yeah. And so, you mentioned Mimir, and Mimir is basically just a way to manage Prometheus and kind of scale it out. So, I'm kind of curious, on the Mimir side, are they using...I know that we use kind of EFS volume. One of the challenges you have with not writing to disk is that object storage can be slow. Disk I/O you got to go over the network and all that kind of stuff. So, I'm curious, does Mimir use it just for kind of the Disk I/O bandwidth generally, or is it just an architectural consideration?

KYLE: Sorry, I guess I'm not understanding your question.

JUSTIN ELLIS: So, when I think about kind of, you know, back to Will's question of what are those main challenges when you are trying to move from a stateful system to a stateless system, what things are going to be difficult to move into kind of more of a stateless model and getting those into, say, object storage or whatever else?

The one thing I can think of there is that having a disk on hand, having these new SSDs available really does give you a lot more bandwidth. So, data proximity really kind of is a big issue. And when you can write directly to disk, that's going to be so much faster, so much quicker than, say, making a network call out to S3 and writing that file. And you want to read those files as well, and that needs to be quick and fast, right?

So, in an application that really does require to consume a lot of data process, a lot of data in a quick fashion, which I would imagine something like Mimir does, that may be one of the reasons that there are kind of stateful bits throughout their general architecture.

KYLE: Right, right. So, on both the read and the write process, to my understanding, you kind of want that local timeframe, meaning that timeframe that you're going to be using the most of. And it kind of requires a little bit more of an understanding of Prometheus itself. Prometheus data itself gets chunked. And so, you can change that chunk, but I think, by default, it's three hours.

So, when you get that chunked, that chunk can then be sent up to your object store. However, this recent one that has not been chunked, that's the stuff that's being currently written to, so you need quick access to that. So, that's very stateful, whereas that chunk can be up there and be the, quote, unquote "stateless" in S3.

Same thing on the read side. We're going to keep anything that's been queried recently we're going to keep that down, and that's when the long queries are happening is when you're having to go past that short time frame. And you're querying for a day, two weeks, you know, a month or longer, then it's going to have to go and pull and this down. And that's either landing, like Justin is saying, on a disk, or there's also caches in the Mimir stack as well to have that in memory, at least for the time being until that needs to be flushed.

MIKE: So, since there's rapid access memory, you say it’s tricky to send out to something managed [laughs].

JUSTIN ELLIS: Yes, right. Yes. Any applications like AI applications need to read a lot of data at a specific time. So, there are unique systems that do require a lot of disk input, output in kind of a quick fashion, and those ones are going to be a little difficult. You're going to have to think in more of a distributed systems. And distributed systems are hard to manage. And going forward, you're thinking about disk. You're thinking about quorums. You're thinking about ordering and all those kind of complex problems.

Some of those are solved. Some of those are still yet to be solved. But we do really do rely a lot on kind of managed services, especially for distributed systems. But we also can run distributed systems in-house as well, and they are just a little different to manage, a little bit more difficult.

KYLE: And we have kind of a new technology, new-ish technology that's coming down the pipe, where we're also investigating the utilization of streams. Specifically, we're looking at Kafka, you know, what can that provide us? Because you can just assign multiple listeners to a Kafka topic, and then get the stream as you need it, and that can make things stateless as well.

EDDY: I was going to say, I thought maybe as we're inching closer to the end of this...we touched a little bit about holy grails in DevOps, so I want to elaborate a little bit on that. And what are some things that you'd like to have in DevOps that isn't quite possible yet but in an ideal world for a holy grails system?

JUSTIN ELLIS: That's a great question. One thing on my wish list, I think it is possible today, but there has been a lot of movement in what's called an internal developer platform. This could be something like Spotify's Backstage, which is an open-source project, which kind of provides the fundamentals of an internal developer platform.

What that is is, basically, instead of having CLIs and having web services like GitHub Actions and Jenkins, you have kind of, like, a central kind of observability coordinator that can tell you the status of your services and your applications. It can tell you what build is out there today instead of having go to a CLI and kind of integrating all the data from the systems to give developers access to be able to see things and perform basic functions. So, I think that is possible today, but we're not there quite yet. Dynamic environments was the other one that we want to get to. And then, that's all I can think of right now, but I'm sure there's probably 10 more [laughs].

JUSTIN CALDWELL: So, I was going to ask, if you guys could just bullet fast, just list through the tools you guys use every day that you love.

JUSTIN ELLIS: Yeah. So, Terraform that's what we use every day, and we do love it. Terraform really is a great system going forward. Alternatives to Terraform are CloudFormation. And there's some even newer ones that work a little bit differently, maybe within Kubernetes, like Crossplane. But I still think Terraform is great for infrastructure as code, and something that's awesome.

Another tool that we use heavily and we use every day, and this is kind of our orchestration framework, and that's Kubernetes. So, Kubernetes really does have some great fundamentals. Kubernetes comes out of Google who originally how they managed their system was a system called Borg. And Kubernetes is kind of an offshoot of Borg and has a lot of the fundamentals that you want when you're running a system as far as ingresses, deployments, applications, and scaling those out. Those are kind of my two favorites today. Kyle, I'm interested to hear what your favorites are.

KYLE: I'm going to surprise everybody here and say, in general, the Grafana Labs tools, specifically. It's a great product. Most everything that they have is a great product, and they are open source. They serve their open-source community really well for monitoring. It has been needed in my tool belt. I would also throw out Prometheus. Prometheus is something that was needed, a high-performant time series database that was needed. To go along with Justin, tools, Terraform, and Kubernetes.

I also find that I have to throw out my IDE, and that's VS Code. VS Code is something I could not live without. I've tried to use some of the other IDEs out there, and I'm not as comfortable with them. So, there's that. And I'm trying to think if there's anything else. Oh, Lens is a visualization tool to see into Kubernetes. Right now, I'm currently using OpenLens, the open-source version, but any visualization tool to see into Kubernetes and be able to manage that. AWS' offering is getting a bit better. But I still really rely on OpenLens in order to do my day-to-day.

JUSTIN ELLIS: And I'll just add, obviously, Docker and containers just in general.

KYLE: Oh yeah.

JUSTIN ELLIS: That's kind of our bread and butter. I know that developing in Docker sometimes can be a pain. But for what we do when we manage clusters and applications, Docker really just gives us a single target to build towards, and then we can basically put that anywhere. And then, I'll just also say AWS, generally, GCP I really like as well. These cloud platforms have become so mature and so great.

It does require a lot of kind of...this is why I don't want developers to necessarily have to understand AWS. When you go into the console, you see all the tools that they have and all that kind of stuff. So, I've always wanted to create kind of an opinionated framework about what tools we kind of use going forward. But AWS is just phenomenal. I really like GCP as well. Modern cloud platforms are really just pretty awesome and just a game changer. And something so fundamental I don't even think about, you know, AWS has only been around since 2008, or something like that, but really just super fundamental to what we do.

MIKE: And that does bring us, I think, really well to a good conclusion. We've talked about a lot of things, about how to bring legacy systems forward, about the critical nature of having the easy plug and play [laughs]. It does what you expect it to do. You have a configuration file and done, so you don't have that complex interface. And then, the issue of reducing complexity all around and how that applies in a lot of different parts of our work.

Any final words particularly from the platform engineering folks that you'd like to share as people finish this podcast?

JUSTIN ELLIS: I will just say, I think it's a really exciting [laughs]...I know it sounds weird, but it's a really exciting time for infrastructure with this new AI tooling and increased capabilities there. I'm just really excited about the future and how we kind of manage infrastructure going forward.

I think a lot of those fundamental pieces that we were just talking about are now in place. There are some really impressive open-source projects. I'm just really optimistic about the future and how we manage stuff going forward. I think we're just in a great spot in that these ecosystems are going to continue to grow.

Kubernetes has kind of spawned what's called the Cloud Native Computing Foundation, which is kind of very similar to the Linux Foundation. But if you go there and you look at the landscape, and all of those tooling, and SaaS, and frameworks, and open-source projects, it's just really an exciting time in infrastructure, and I think we're in for a lot of improvements in the near future.

MIKE: Great. Well, if you're listening [laughs] and you'd like to get here, it's possible. There's so much opportunity to dramatically improve the experience and return on investment of your infrastructure by approaching with some of these tools that are available. And kudos to the team here for having made that happen.

JUSTIN ELLIS: Thank you.

MIKE: With that, until next time on the Acima Development Podcast.

Cross-team communication is one of the biggest challenges in software development, and the latest episode of the Acima Development Podcast dives deep into why it’s so difficult—and how teams can improve. Host Mike kicks things off with a humorous but insightful story about botching his son’s haircut, using it as a metaphor for what happens when people act alone instead of seeking input from others. The panel, featuring experts from different departments and organizations under the same parent company, shares real-world experiences of miscommunication, siloed teams, and the consequences of not aligning on tools, expectations, and processes. The takeaway? When teams don’t collaborate effectively, things go sideways—fast.

One of the key topics discussed is how different teams use different communication styles and tools, often leading to misunderstandings. Some prefer instant messaging, while others rely on email or ticketing systems, creating friction that slows down progress. The panelists agree that social capital—trust built over time—is crucial for overcoming these obstacles. Successful cross-team collaboration often comes from learning the hard way, through repeated trial and error. They emphasize the importance of setting clear expectations early, aligning documentation processes, and making an intentional effort to break down silos. Without this effort, teams risk wasting time, missing deadlines, and struggling with misalignment between business and engineering goals.

So, what’s the solution? The panel suggests several best practices, including embedding engineers into other teams temporarily to build rapport and understanding, structuring projects around clear milestones, and ensuring leadership actively supports collaboration. They stress that no matter how well-designed a system is, success ultimately depends on human connection and clear communication. Whether working in small teams or across an entire organization, the key is to acknowledge the real cost of collaboration upfront and commit to the effort required to make it work. By embracing transparency, proactive planning, and direct engagement, companies can turn cross-team dysfunction into a productive, high-functioning environment.

Transcript:

MIKE:  Hello and welcome to another episode of the Acima Development Podcast. I am Mike, and I am hosting again today. With me, I've got a great panel, and it's going to be real. I'm going to go a little bit deeper into the introductions than I sometimes have because it's going to matter for this discussion.

First of all, we got Kyle with us from the DevOps team. We've got Dylan from Rent-A-Center, and it's like a sister business, I guess you might call it, you know, we're different lines of business, same parent company. And it's great. We're going to want to do more of this. Our parent company is Upbound, and we'd like to get more participation from across Upbound so we get diversity of opinions.

So, Dylan is with us from the Rent-A-Center side. So, it's great having Dylan. I'm probably going to ask him a lot of questions. We'll get to the topic in a minute, and you'll know why [chuckles]. We've got Molina from application support. We've got Ramses and Dave, who work with lease origination. We've got Eddy who works with our partners, merchants. And we've got Will, Will Archer, who works for an unnamed third-party [laughs] retailer.

WILL: Technically, I’m an independent consultant, but I work for a large retailer.

MIKE: And he’s with us, as he usually is as he usually is [chuckles]. And that’s the group we’ve got today. And I’m going to start, as I like to do, with a story, a story of...in this case, it’s personal humiliation [laughs]. So, a couple of weeks ago, my four-year-old...and I can tell a lot of stories about my four-year-old because four-year-olds provide lots of opportunities for stories. Well, he decided --

DAVID: Content generators.

MIKE: They are. He decided he was going to cut his own hair [laughs], and he did, got a big chunk out of the front of his hair. But it was probably still going to be salvageable, you know, I pulled out the clippers and trimmed around the sides, and it was okay, you know, it was okay. It wasn’t great, but it was okay. And I was talking to my wife about it, and we were working together on this endeavor, this creative endeavor [laughs] of trying to fix the disaster he’d made of his hair.

And we looked at it, and we were like, “You know, it’s okay, but it’s not great [laughs]. We’ve got to do something here.” And my wife suggested, “Well, maybe it should be shorter on top. We need to do something.” And I’m like, “Yeah, you’re right,” then we kind of left it. We kind of left it. Something came up. We got busy. And later that day or the next day, I was trimming my own hair [laughs]. And those of you who haven't seen me, I don't really have hair. So, when I say that, I mean I take off the few scraps that are left around the side so it doesn't look like I'm faking it [laughs] and trim my beard.

I was doing that and in walks my four-year-old. And I've already got the clippers out, and then I said, “Oh, well do you want me to trim your hair?” And see what I did there? I didn't go and ask anybody. I thought, you know what, I'm just going to do this myself. This will be just fine, right? So, I put the comb that you put on the clippers to, you know, make it longer, the one that I usually use for my beard, number four for the longest part, then I use, like, a number two. None of this is important, unless you...but if any of [inaudible 03:40] clippers, you know what I'm talking about.

So, I put all this together, and I started trimming, and I started trimming just a little bit. And I thought, wow, that's a lot shorter than I thought it was going to be [laughs]. And I had a moment there to try to rescue what I had done, and I didn't [laughs]. I didn't. I didn't bring in the team, didn't bring in my wife to say, you know, “Hey, what do you think I should do here” Didn't bring her in initially. I just thought, hey, the situation's here. I'm just going to do something about it, and I just went for it. And I finished, and it wasn't good. In fact, it was terrible [laughs]. It was terrible.

And my wife sees it, and she's like...frankly, she cried [laughs]. It was that bad. I felt so bad. I felt so bad. And that was entirely my fault. There were several places where I could have fixed this problem, and I didn't. If I had brought her in to communicate upfront before I did anything at all, she would have probably given me good feedback. If I had started and saw that there was a problem and communicated, I would have gotten good feedback. But no, I just went ahead alone. I thought, I can do this, and boy was I wrong. Luckily, it's been growing out some, and she came in and fixed it some, and it's better [laughs]. It's okay.

WILL: [inaudible 05:05] give him a Mohawk. Like, a mohawk it always sounds sort of plausibly on purpose.

MIKE: Yeah, it's close. It's almost a mohawk. It’s kind of a mohawk-ish sort of look with a bit of a pointing back. It's growing out. It doesn't look so bad anymore. There's recovery. It was a bad incident all around, and a lot of it is my fault, which is my story of personal humiliation to say, you know, maybe you should involve other people on the team, people on other teams, early and well in your communication process before you do something that's hard to fix. And we're going to talk about cross-team communication today and all of the error cases in that and how to make it better.

That's why this panel is such a great one. We've got people across lines of business. We’ve got people who work in different business, right? And we've got people from various different teams that have to work together in order to talk about this recurring, crucial problem that we have in engineering. And in our pre-call, we talked a lot about this that it’s...this is a [inaudible 06:11] problem, probably in software engineering, and it's not a technical problem. It is a human problem, and it's universal. There is the problem statement [laughs].

And I'm sure you all have some stories about problems that you've run into like this in software, and problems, suggestions, as well as, you know, some things in between, problems, suggestions, and just ideas, right? I'm going to stop talking and let somebody else talk for a little bit and take things in their own direction. So, what are y'all's thoughts?

WILL: You know, for me, I think the most interesting thing, I think, cross-team dysfunction is almost the norm. I think that's almost the default. And so, I'm a lot more interested in hearing about stories when things went well, when things went right. What did that look like? What did that feel like? How were those successes set up? To see if there's through lines and threads that we can sort of, you know, take and put in our back pocket and maybe save for a rainy day, because it's only a matter of time before we're going to have to get some teams together and come up with a good solution.

MIKE: Well said. I've got some suggestions there, but I'm not going to jump in yet.

EDDY: I think it's just important to be honest about what broke and what worked. I think a lot of us like to add sugar saying, “Oh yeah, we met the deadline, you know, it was handed in at the time that we were expected to. There were some bumps, but it's fine. It ended up working.” And that's cool, but I think a lot of the times being able to reflect and do a postmortem and just really be honest, you know, like, what went wrong, how not to replicate that, I think is a really good step.

WILL: Did I misunderstand the question? Are we talking about incident reports, or are we talking about cross-team collaborations that went good? I mean, I'm maybe asking for case studies of, you know, this project and these two teams got together, and it went well, and this is how and maybe why.

MIKE: Yeah, you know, you're right. I think we're talking about cross-team collaboration. I think that Eddy would suggest [chuckles]...well, maybe there is no such thing as a -- [laughter]

DYLAN: I almost have to agree with Eddy here that the best teams for cross-team collaboration often have a long trail of failures behind them, and they've learned to work well together through that.

KYLE: I've got a bit of an example for this. As my organization, we kind of merged together to the greater organization. We both standardly used different tools for communication, we'll go with that, different chat communication tools. And I think this is where a lot of issues come in is people always have preferences on how they use their specific tools. Each tool will breed a different culture. And that was one thing that I wasn't understanding when I was going in on my project with communicating with the other portion of my team over at RAC.

Finally, I sat down with one of the RAC individuals, and I was just like, “Hey, I keep pinging you. I keep pinging your team. I'm not getting any response. What am I doing wrong?” And we had to communicate to say, “Oh, well, we don't use the chat client that way. We use it this way, and that's why we're not seeing any of your communication.” And it wasn't until we had that direct one-on-one communication that I learned how they normally communicate. And I feel like that's how you kind of have to go is one side or the other has to kind of pivot to figure out, like, how does the communication have to happen, and who can pivot, and maybe both sides need to pivot. But who needs to pivot so that that communication channel can stay open?

DYLAN: Absolutely. That's not only with text communications or email communications as well. If two teams are sharing process documents between each other, they may have thought they agreed on some process. And one team says, “Okay, we'll put it in our shared drive for our Confluence,” and the other organization does not use Confluence at all.

KYLE: Oh yeah.

DYLAN: They don't have the same document in the same place, and they don't know what the process is later on.

WILL: Oh, brutal. Man, people from the business side keep on trying to make Teams happen. Teams is never going to happen [laughter]. No one's going to use Teams. It's never going to happen. But they keep on just coming in. They're just like, let’s put...Slack. Why are we paying for two? We get Teams for free.

MIKE: [laughs]

WILL: Don't do it.

MIKE: It's true, and there's a cultural...we should probably do a whole separate podcast on communicating with the business side [laughs] because that cultural divide is deep and vast [chuckles] and hard to bridge.

KYLE: Even the culture of communicating by email that's another one that it’s...engineers, I feel like, and maybe I’m wrong here, but engineers, I feel like, they've developed a culture of communicating over tools like Slack, or even Teams, or Google, even. And there are other lines of the business that prefer email. For engineers, really speaking for myself, that seems to slow the process down to a snail's pace, and it's a hard thing to get over.

MIKE: Exactly [chuckles].

WILL: I've always wondered what it is. Every engineer, I think, I've ever talked to has it: the aversion to email. I don't know, I haven't unpacked it, and I have thought that I should accept it as a given. Like, no engineer will email you, unless extreme pressure has been applied to them [laughs].

EDDY: Well, I think part of that is because, as an engineer, you're expected to hit certain deadlines, and so you need answers pretty quickly. So, leaning towards instant messaging, like, it's even in the word, right? Like, you get messages instantly versus having to wait for emails to come. I'm not saying that one's superior than the other, but I think we work in an environment where we're trying to fetch information as quickly as possible. So, having direct communication with that person via a DM, you know, does wonders.

WILL: Yeah, fair.

MIKE: It violates, like, a psychological contract that we have with each other. When you're talking with somebody in real-time, the timing matters. If somebody pauses for a while, you think, oh, what did I say wrong? It breaks your connection. It breaks the emotional connection badly, and email guarantees that. There's no way you're going to have real-time communication over email. I mean, maybe you could, but it's super awkward. Whereas if you're directly messaging somebody, you feel like, yeah, I'm almost in the same room with them, and it's a totally different dynamic.

KYLE: [crosstalk 13:28] you’re in the same room [laughter].

DAVID: There's a trade-off there that didn't become clear to me until somebody dialed it all the way to 11 in a different direction, which is there's a phrase that came out of agile, back when agile was still called (XP) Extreme Programming. One of the phrases --

WILL: [inaudible 13:50] XP.

DAVID: Oh yeah.

WILL: [inaudible 13:52] to do with that.

DAVID: One of the phrases that they circulated a lot back then was, when you're pair programming, they'll tell you “Never type when you can speak. Never speak when you can point,” right? Literally, it's, like, how much information can you communicate in the smallest amount? Now, pointing is very ambiguous. You have to be in the same context. You have to be physically present. There's all kinds of stuff. But if you have that context, you should use it, right? The last thing you want to do is sit down to pair program with somebody and they pull out their phone and Slack you a message, right?

MIKE: [laughs]

DAVID: Unless they're sending you the shockie to get into something. You’re just like, okay, yeah, I do need that written down, but, yeah, it doesn't make a lot of sense. So, when you see the immediacy and the ceremony...and sometimes you want the ceremony, sometimes you want a Jira ticket documenting that I have opened this issue with you on this date, and this was my request, and y'all better do something about it. Versus just, like, “Hey, we're going to let [inaudible 14:54]. You want to come?” Like, you wouldn't do that with Jira. You wouldn't do it with a ticketing system.

DYLAN: Your Slack example highlights, too, that switching from pair programming to Slack that's a context switch, and it can be jarring. Switching to email is a much bigger context switch because you're not just drafting an email. You're confronted with your whole inbox, and you see a subject line from someone and think, oh, I need to switch gears and reply to this now, too. So, psychologically, it throws you off.

DAVID: Serial access, like, accessing one thing at a time, like, as in messages coming in over time into your inbox, is one of the most inefficient ways to do something. And how is that serial port being used? Every marketer in the world is jamming messages into that serial port. It's...yeah.

EDDY: I want to be completely transparent, and I want to go on out here. I'm hoping other people can agree. But switching between two different applications that have two different interfaces kicks my butt every single time, right? It's crazy. It's crazy.

MIKE: Well, it breaks your flow. Anything that breaks flow prevents effectiveness, right? I think we've talked about flow in another, you know, another podcast before, but getting into that state is hard to get into and if you break it, you lose it for a long time. And anything you add that makes it harder to get into that flow just adds another layer of difficulty.

WILL: You know, for me, honestly, when I think about sending or, God help me, receiving an email, I feel like it is the least likely thing to get me any answer that I need. It's almost always something that I don't want to do that also doesn't matter. And my best-case scenario is I send somebody an email, and I'm like, hey, I have this problem with this vendor tool, and it's doing this thing. And please reach me at an actually productive communications channel so I could work it out [laughs]. The email is sent in getting a Huddle or a Slack message from somebody I don't know well enough to ask directly.

EDDY: I don't fully expect people to really have the answer off the top of their head, but just out of curiosity, like, just a rule of thumb, how productive has the experience we’ve [inaudible 17:27] with emails? Like, have they actually provided any sort of value?

DAVID: I'm going to say yeah.

EDDY: I’m surprised.

DAVID: For me, it's a receipt. You've got something, like, I will pull up my inbox and basically say, “Uh, where's the receipt for this thing? Or where's the confirmation code that I got for this?” And I love that. I've just told you something, though. I've just admitted that I am not an inbox-zero kind of guy. You open my Gmail, and the unread messages is well into five digits. I remember when I passed 20, 000 unread messages and still climbing. And it's because I open up email, I scan, I’m like, yeah none of that's...and then just move on. You know, why bother?

I don't deal and set aside and finish my emails. If that were my working system, email would have a completely different flow for me. For me, it's much more asynchronous. There are times when I will go for a day or three without checking my email, but anybody who knows that I'm living in that asynchronous environment knows that well, we'll send this to him. It will get to him eventually, but it will get to him. Whereas anybody who's tried to Slack me knows that I frequently have my notifications not working the way they need to. And so, I think everyone in this call has gotten a message [inaudible 18:43] “Dang it, I missed this notification. Sorry, where were we?”

MIKE: Well, email, what's right there in its name, replaced postal service delivery, some sort of, you know, human delivery [chuckles] involving somebody grabbing a written document and carrying it or maybe fax [laughs]. And, in that respect, I think, and this may be why it's popular on the business side, it’s like, wow, this is so much better than waiting for the courier [laughs], and maybe it's because people like the paper trail that you get.

But it's not that you don't have that in the other tools. I'm not saying it has no value. There is some value, but we're talking here about collaborating across teams, and we've kind of diverged a bit into the tools. You've got to set the ground rules first. And if your engineer is trying to coordinate on something, email is probably not the way to do it.

WILL: I think email is the industry standard for communicating with people you don't know. I've got a bunch of people from completely different departments. I don't know any of them. I don't know how they work. Are they Teams people, Slack people, whatever people? And we're going to go out and we're going to create, effectively, a document of record, right? So, people you don't know, like, when you're sort of managing up, you're talking to people sort of further up the chain. You can't just Slack, you know, you can't just hit people up like that. You send an email where it’s like, all right, I sent it, and you're going to read it or not. But I sent it, and there it is, and if you have a thing, then you can get me. Anyway, it's not going to be a Slack thing.

And I think that right there kind of segues me into maybe one of my answers in terms of the original question I asked which was, what's a successful collaboration look like and feel like? And, for me, when I have collaborated with teams successfully, and, like, the way I've collaborated with teams successfully, is when one, or preferably both parties, have some sort of social capital with each other.

When I know you, when we've collaborated before, when I know, you know, who you are, and how reliable you are, and whether you are going to, like, whether there is a quid pro quo, whether I'm not just getting bombs thrown into my sprint and then I'm never going to see you again, like, are you a pure [laughter], unadulterated problem for me, or are we collaborators where this is going to go both ways, and I'm going to have a question for you, you know, six weeks or a month down the line and, you know, you're going to scratch my back?

And I think sort of cultivating that is one of the bigger hidden, you know, tidal forces that as you get tenure in an organization, you start to build up those connections where you know people, and you know people who can help you out, who will and who can, versus people who are just going to create problems for you and you must [laughs] be coerced into helping them because you know it's just going to be a time sink. And it's not that you won't. It's not that you won't. If you have a problem, I'm going to fix your problem, but I might solve some other people's problems first, or more thoroughly, or more promptly, because, you know, it is a social capital situation, and not everybody's a good actor.

MIKE: This goes right into what Dylan was saying right at the beginning, which was the collaborations that work well are usually the ones that you've had a few tries first [laughs]. You have to establish that relationship first. And, it goes, you know, if you don't establish some ahead of time, you don't have it. I don't have a relationship with you. I don't trust you. I don't know you. This isn't going to work.

EDDY: Well, it's kind of like I instinctively thought of the analogy of a professional sports team, where, like, you just get a bunch of people who claim they're really good, and they are good in their own right. But if they aren't willing to “play ball,” quote, unquote, things are going to go tough. You have other people who want to do it in one approach. You have another person who wants to do it in a different approach.

They're both valuable approaches, but unless they align, and there's trust between each other, you know, it's going to be rough. It's going to take a really hard time to find that happy ground. But once you do and you've played enough games together, eventually, you start to understand, you know, how other people think, you know, it's easier to come up with game plans when everyone understands the way they play, right?

MIKE: Well said. We don't need a bunch of heroes. We need a team.

DYLAN: That can be both very frustrating and very valuable. Often, when two teams are collaborating, one team is managing software infrastructure, some kind of system that is much more dangerous to mess up than the other team, and often they have more robust processes than the other team. And when the other team joins, it can be very frustrating because they have to go through a ton of red tape to do things. There's a bunch of process they haven't done before. They don't really get why it's important. They haven't been there at the incident management meetings. But being able to integrate that process into themselves also makes them perform better as a team.

WILL: Yeah, and, honestly, a lot of the time, almost all of the time, the collaboration, like, within the teams it's not equal priorities, right? There's a big thing for my team, and I've got a big deliverable. I've got a big project, and this is my thing that I'm going to get out. And it impacts and interfaces with other teams. It's like, oh, I need you to update this API, or I need you to provision some hardware for me, or what have you, right?

This is my main thing, and this is, like, something that you're doing on the side. And that can be a source of some amount of friction because I care a lot, and you care to the extent that, you know what I mean, for better or worse. And that's one of those critical social capital times because I can just be like, “Hey man, can you just get this thing in for me?”

And it's not a big piece of your sprint, you know, no one really is chasing this thing down, unless I go to my manager, and then they go to their director, and then their director goes to the other director, and then the other director goes to the manager. And they're like, okay, come on, let's get it done. There's all those steps, right? That's collaboration across teams, but even as I said it, you can see where it's just like, oh, it sounds like a crime. But if I'm just like, “Mike, do me a solid, you know, help me add this Boolean flag onto my JSON, please, so I can get this thing out, then it's a matter of the day.

MIKE: Well, yeah. You tie it in...I've been thinking about a couple of successful projects I've seen. I'm thinking about a couple of really successful big projects that were related to each other. Well, they weren't related to each other. They're totally separate projects that had a pattern that was related to each other. They followed an almost identical pattern in that they were very successful, and I think for the same reason. And it ties into what you're saying about, you know, you've got to talk to the director. They talk to somebody, you know, that horrible chain of command that makes communication just grind to a standstill and play the telephone game [chuckles].

Where it's really worked, where I've seen this work is when you started with not business stakeholders...business stakeholders, at first, figure out the requirements, and then you have the product team come. You know the requirements really well. That product team that knows the requirements sits down with the engineering leaders who have something to this project, who actually know what they're talking about. And you get the group, small group. I'm not talking 10 people; I'm talking maybe 2 or 3.

The people who are most affected by this project get together with product, and they hash out together the order of operations to get this big project done, so you have a series of milestones. And that gives a roadmap which you can then put timeframes on and then start filling out the specific requirements and teams associated with each milestone. I’m seeing there's a lot of planning involved here, you know. But you've taken this big problem, how do I get from here to the other side of the world, and you've broken it down into smaller pieces. Well, how do I get from here to the edge of my state, or how do I get out of the driveway, right?

You’ve figured out the major milestones along the way, and then you start functioning at the level of those milestones, and you know the teams involved. And then, bring those teams in to solving that milestone, you get that milestone done, move on to your next milestone, and so you have the map right from the beginning that everybody can see. Everybody sees the roadmap. They see the milestones along the way. And then, they pitch in to get each of those milestones done, and then the next milestone. And each of those milestones has the teams clearly identified and the requirements clearly identified to get that milestone done.

So, there's some commonality here, one is the really well-documented and well-created roadmap. You can't just have somebody throw something out there. It has to be agreed on by the engineers who know what they're talking about together with product, so you actually get the right business requirements so we know what you're actually building.

And then, you've broken it into those milestones. I feel like those milestones are critical to make it work because then you're not looking at something so big that nobody cares. You're looking at pieces that everybody can engage with. And then, each of us has a clear deliverable that everybody knows how to get solving. Because then you can get engineers thinking, well, how do I solve this problem? You get each of the teams that care about it engaged with that problem, and it matters to them. Then they can collaborate, reach across teams as necessary.

And before you actually have them collaborate, the teams get together and define the API boundaries. How are we going to communicate with each other? That can only happen...you only get a start with those API boundaries so that they can code to that, rather than not knowing what they're doing, if you've had the road map, and you've identified the teams, and you said, “We're going to get together first.” And that's a lot of overhead up front, and it requires product who's invested and the engineering leaders who are invested in working together. If you can get that, I think you can make it work. But there's lots of places that can fall apart.

WILL: I love it.

DYLAN: The way you put it, setting the groundwork is itself an engineering project.

MIKE: Absolutely [laughs].

WILL: I'd pitch you on it. I mean, I love everything you're saying here. I would pitch you on another thing as well and maybe a tactical, small unit level. I really love engineering embeds, where okay, these two teams are collaborating. And, generally speaking, both teams are not fully tasked. But more often than not, I think 80% of the time, most teams are not fully tasked, but it's like, okay, this team is providing this, you know, subcomponent for this larger project. This is our main baby, and there's this subcomponent. Lend the engineer out. Give an engineer to the other team. They're going to be embedded for as long as their piece needs to be there. They're going to do stand up with them. They're going to sit there. They're going to build that social capital.

And most critically, I mean, maybe two things that I think are going to happen that are really important. One, I think it's very easy when you have a sort of like a [inaudible 31:15] engineer for a couple of days, you know, half of somebody's sprint cycle. It's very easy to underestimate impact to productivity with cross-team friction and collaboration and getting things ready to go. And so, when you're dealing with stuff as an embed, this guy belongs to this team, you know, for such and such, you're going to take that more seriously. And, honestly, I think when you have somebody else's project, it's really easy for that to get kind of short shrift.

And it's also sort of...how do I put it? No, actually, those two things it takes a lot more friction than people get credit for. And this will get people to take it seriously from a planning perspective and fully commit from an engineering perspective, like, I am going to get this thing done. This thing is my thing, and I know these guys. They're not just strangers throwing work over the wall at me, and it's hitting me on the head. They're my team for now. Temporarily, they're my team, and this is our project.

MIKE: Oh, I like that a lot. And it's a little different than setting up with people who are, like, a tiger team, where you say, “Okay, this project, we're going to take somebody from each of these teams and put them together and isolate them from everybody else.” I have seen that not work so well because then you've deliberately isolated people. You've shut them off from the rest. But the way you put it, where you're loaning somebody, where you're sharing rather than isolating, I think that works out really well.

DAVID: I think I've mentioned this before that I've done that on teams proactively without a project, where it's not like...we literally called it the hostage exchange program where we would take a sprint, and we'd basically say, "We want one of yours. We're going to give you one of ours." And after two weeks, we'll go back. And so, literally, the whole point is just to pass this person around the team, let them soak in the environment. They're not going to be super up to speed, but that's okay because your guy over there isn't up to speed either. But after two weeks, you swap back.

And it's so beneficial to be the person who was out on the exchange because now anything that happens with that team's product, everyone in your team will turn and look at you. And you're like, “Hey, you were working on that team, you know, last month.” And the really cool thing is when you want to go drop a ticket on somebody's desk. Like, they've got a ticketing system for how they're going to handle their stuff. But before you do that, you just pop open Slack and you're like, “Are you guys on Kafka 2.14 or 2.15?” It's not worth stopping the whole system.

And the other person, because they've sat next to you for two weeks, they're like, "Oh yeah, 2. 7. We're actually on 17. You're way behind." "Cool. Thank you." And if I need something beyond that, I'll go do the ticket and do the thing. But there's that moment, like, water cooler connection of just, like, "Hey, buddy, where's this at?" and being able to do that.

EDDY: Yeah, Dave, but then you run the risk of becoming the git blame of that, right? And then, everyone just comes up to you and says, "Hey, you touched this code before. What does this do?" You're like, "I was just there. I just --"

DAVID: The reason ticketing systems exist is that people don't have a good defense against becoming the git blame guy, and they're like, I need to get work done. I need you to stop interrupting me. Go talk to the tickets. And that's, unfortunately, that's just replacing one dysfunction with another, but yeah, you're right. That’s a very real problem, and it's bad when it's...

I guess if I'm yelling at you "What version of Kafka are you on? Drop what you're doing. Pay attention to me," I'm not building a team with you. I'm not revisiting a good interaction. It's got to be that water cooler, like, social capital like we've talked about of like, “Hey, how are you with this?” Because half of those conversations are equally likely to begin with, "Hey, is your kid feeling better?" And that's a social capitally valuable, you know what I mean, positive social capital in that communication, because you care about the human on the other side.

WILL: I guess what I'd say, to Eddy's point, when you have that guy, you're the ambassador. You're the liaison between these two teams. You've been over there. You've done the hostage exchange. You have some connections. You have some background. You can get things done. And your job is constantly being looped in to, like, oh, this is going wrong, help me. This is going wrong. Help me. This is going wrong, help me.

A, that is a strong signal that the hostage exchange program needs to not just be continued but expanded, right? And B, think about how badly these projects would be going without you. You're saying, "This is bad. We shouldn't do it.” And I'm saying, "This is good. It's saving your ass, and you need to do a lot more." It only becomes an anti-pattern when you have this unmet demand such that your concierge, your ambassador is just getting dog piled with requests for support from these other teams. That's, I don't know, just another way of looking at that. It's not too much; it's not enough.

DAVID: You could have both.

DYLAN: In a crisis, you end up being very happy that you have someone like that on your team because you don't have to play a game of telephone. When, you know, something goes wrong, support figures out. They call a developer; developer realizes it's a DevOps problem; they call DevOps. If the developers also worked with the DevOps team, then they can start looking at the DevOps side, too, and also call DevOps in parallel, and it saves you time. And if you need to respond to an issue in 5 or 10 minutes, then you don't want to lose that 5 minutes of waiting for someone to respond.

EDDY: So, you're saying when things are on fire, don't use emails.

WILL: No, you don't.

MIKE: [chuckles]

DYLAN: You know, that's important, too.

EDDY: [laughs]

WILL: No, nobody sends an email on fire. They have a guy. And they're like, I have a name, in my head, of somebody who's going to put this fire out. And, you know, from a career development perspective, I'll just say that in my career, like, consistently being that guy has been really good for me. And if you're like, “How do I advance as a developer?” being the name that people can call when things are going bad, it's really good for you.

MIKE: You know, talking about the email, Molina, who was here, had to go, but she does application support. So, she handles the bad stuff. When things are on fire, she's the firefighter [laughs]. And she doesn't instant message me. She doesn't email me. She texts me.

WILL: [laughs]

MIKE: And that matters, right? She knows that she can send me a text message, and I'm going to get immediately on that problem, not just me but a couple of other people that she knows, you know, are critical to solve this problem. And I'm going to jump on that problem because, you know, that's going to happen.

Now, I don't have everybody text me, right [laughs]? But the person who I know is doing that critical firefighting, well, yeah, you're going to get the most direct, and engaging, and immediate sort of communication that I have available, or a phone call even, right? Her and her partner there in application support they'll both do that. Yeah [laughs], that relationship matters.

WILL: Yeah, that's the gig, man.

DAVID: And it's respected, too, though, right? You're not going to get a text message at 3:00 o'clock in the morning saying, "Hey, would you fill out your TPS report sometime in the next week?" [laughter] No. I started work before the internet existed. And we had a saying back then, which was “When they come find you in the bathroom, you know it's serious.” And this is the modern equivalent of that.

WILL: Yeah, what do you call it? I don't know. I keep my phone number on my Slack. Nobody calls me if they don't need me, but it's rung before [laughs]. That's just the gig.

MIKE: So, we've talked about a few things, right? We've talked a lot about channels of communication and how some are better than others. And the more immediate the channel, the better. We’ve talked --

WILL: Not necessarily.

MIKE: Well, that's true. You're right that having a...You're right. I did not state that well, that immediacy is good in many cases, but there are some cases where, I don't know you, asynchronous, keep a record of this is the right answer. So yeah, my summary was a little off.

And we've talked about some successful projects where you have social capital and the deep importance of social capital, both at the, you know, leadership level, at just between team level, any way you can possibly establish it that's critical, and also, the planning. The actually doing the engineering project to build out your milestones, to build out the project plans, and get people engaged before you jump into the project because if you just jump into it and I hope it's going to work, well, it's not [chuckles]. What have we missed?

KYLE: I was just thinking about kind of expanding on what Will had pointed out earlier about integrating a person into the team. And I was thinking...because I think a lot of the examples are, you know, devs from one team going and being on another dev team. And I was thinking about my history in QA, and most of my QA jobs, you had a QA team. And you worked with just all of the developers or, you know, all the support, all the developers. And whenever there was a bug that you needed to have questions answered, you got up, you walked over, and you had to confront that person. Now, sometimes, that person was somebody you knew. Sometimes it was, you know, the grumpy Java developer, and you just didn't want to talk to them that much.

However, I was in another position, which was very jarring, for me, and I didn't understand it when I first got into the job, and this was actually at InsideSales. They distributed the QA individuals. We were on a QA team, but the QA individuals were integrated with the developers. So, now I'm in with the developers. I'm testing this specific product’s app or this specific product. And then, I'm going to their planning meetings. I know all of their requirements. And then, when we're getting together as a group in QA, it's, okay, well, what's happening with this service? And that's kind of how the communications are getting distributed that way.

But that was probably the fastest form of communication in a development style that I'd ever experienced, and that was because as a QA individual, I was part of that development team. Even though I wasn't necessarily directly involved in the code, I was part of the team. And that's something that I don't know that every company gives that kind of credit to their QA or QE engineers, that they're actually part of that development team. And so, I think there's a bit of slower communication because of that. That's just been my experience, but yeah, it's something to add there.

EDDY: I can attest to coming from QA.

DAVID: For me, this all seems to break down to context, right? It's like, if I'm up on stage giving a presentation in front of a thousand people, I'm going to adopt a very general context. I'm not going to tell edgy jokes. I'm not going to make fun of people. I like to tease people. I like to be silly. I like to be fun. But in a universal context, you don't know if I'm absolutely just destroying a person willfully, or if I'm just teasing them because it's funny.

But when you're working with close people, you know, I have a co-worker that we literally did the hostage exchange, and we had a third guy on our team. And halfway through the morning, he was just staring at us, like, mouth open, eyes wide terrified because TJ and I had gone 5 minutes about “Your mom,” “Oh yeah, well, your mom.” And I mean, it sounded like a Call of Duty lobby. And then, TJ had to stop, and he's like, "Matt, just so you know, we are best friends. This is fun for us. We are..."And I'm like, “Oh, yeah, yeah. No, this is totally just a bit. We're just...” And he's like, "Oh my God," just the weight coming off of him.

But that context pays off in both directions. When you have a general context, you have to follow policy. You have to be very strict. It's your ticketing system. You're going to follow these rules. But on the other side of that, if you can move into a higher context with somebody...like with Kyle, if I write you up, "When I test this, I need you to load this app. You need this backing up, da, da, da." I give you these three pages of how to set everything up because I don't know you from Adam. You might not even be in Utah. You're on the other side. You're coming in from the Rent-A-Center side.

So, I've got to give you all the context and all the background, where other times I can hand you maybe a PR and basically say, "This fixes the tax freeze in Montana," and you go, “Got it.” And you know exactly what the problem is. You know the history. You know what to check. You know 90% of the app is going to be untouched by this, so you need to focus here. And if you're in that general context, no, you have to do the test on every single piece because you don't know what the intent was. Having that context, for me, is the key element between stilted receipt-based communication and being able to just very quickly just say, "This is what we want," and being able to strike and move at it.

WILL: Okay, so, I mean, I feel like we're all fairly aligned in that, to the degree that you can, to the degree that it's possible, it's good to have people who know each other. Or, at the very least, you have people who are embedded so that this alien, foreign team they're committed enough to your success, and your project, and your priorities to give you somebody, right? There's a level of trust there.

Like, if you're going to give me a senior developer for a month, okay, all right. I mean, you've taken me a little seriously, at least, and so, that's great. But what about when it's too big to be able to put up a bunch of buddies, right? What happens when it's too big to be, you know, something like that? It's like a cross-organization type thing, you know, where it’s this is a big thing.

MIKE: And that goes down to what I was talking about before. The times when I've seen this successful, you put in the work ahead of time to get the relevant engineering parties together with, you know, the requirements people, you can get it from product, to build out that plan, build out that road map, get the milestones in place, and figure out the teams involved. Take the time to communicate with those teams.

With each milestone, you have a kickoff, right? You collaborate together. You figure out what boundaries you need to establish. What are our APIs going to look like? You know, whatever the case may be. You can't not do that. And I'm going to [laughs]...sometimes you think, oh, I've done this before. I know what I'm doing. No, you don't [laughs]. You don't. It doesn't work that way. It just doesn't. If you don't put in that work ahead of time, it doesn't work. And I've never seen it work, I don't think.

But I have seen it work on the occasions when people really have done it right and put in that time ahead of time and held to it. It helps also if you have a project manager who has the milestones, meet regularly with all of the teams, and talk about how each of those are going. And like, oh, there's a gap here. What do we have to fix?

WILL: You know, I'm over my skis here because I've never done this. I've done, you know, management and team coordination of projects and stuff with developers I could count on my finger. But I would almost pitch people to do the exact same embed thing, except with functional organizational units at whatever level is required. Do I need to have a director and all of their teams embedded in another vice president's thing? Okay, that's that but the full-time commitment to the collaboration.

And so, whoever is overarching this thing, whether it's a VP, or a director, or a CTO, or the CEO, if it's the biggest, baddest thing, maybe, to Mike's point, maybe not on a tiger team, but say, like, okay, this is the thing. We're doing this thing. And I'm going to take you, and you, and you, and you and your organizations, and you're doing this now. You're reporting to this structure for a quarter or maybe a year, depending.

I think based on my, you know, experience with corporate organization and bureaucracy, I think that is a hard order in terms of maybe careerist internal politics. But that doesn't really put me off of it necessarily because you're just saying the quiet part out loud because that's the cost of collaboration on this scale. You're going to need to have somebody applying pressure at that level, with that intensity, or else it's going to grind to a halt, or it's going to take forever, or it's going to fail. And so, I don't know, that's what I would pitch you, you know, the embed philosophy, but you're just embedding bigger pieces. And if you don't want to embed bigger pieces, then you should perhaps question your commitment to an endeavor of this scale.

MIKE: You're saying, if you're not willing to actually create that social structure, then you're saying you don't care about the project because that's how people work is in a social structure.

WILL: Well, exactly, exactly. And so, it's some other VP's thing. And I'm a director, and I want to, you know, move up the thing, or at the very least, I don't want to work any harder. And somebody is applying pressure on me to work. And if no one's going to move me over there for as long as it takes, then you're running into some real serious operational risks, you know?

MIKE: That might be a good place to kind of tie this together. You're saying that...you talked about different scales. It all looks about the same. It's kind of this fractal thing [chuckles]. If you get people working together, it's still people working together. And getting together in the same room or the same virtual room, you know, getting together into the same structure so that they are working together has to happen. And that happens when there's two of you pairing together, or you're putting your stake in the whole company on this huge project. Well, you better have some way to communicate and be working as a team.

WILL: And it's going to have a cost, an intrinsic, undeniable, you know, yeah, it's going to have a cost, and you need to commit to paying that cost. That inter-organization or interpersonal friction it's going to cost you something. And in my view at least, and I know I'm out over my skis here, but whatever, hot take.

MIKE: [laughs]

WILL: I think being transparent and upfront and really, like, grabbing that nettle with both hands is going to get you a better outcome, and it's very easy. And we've all seen it over and over again, where somebody's like, “Oh yeah, this should be fine.

MIKE: [laughs]

WILL: This will be fine. This is no big deal.” I hesitate to think of an example where somebody has overestimated the cost of that functional collaboration. I can't think of a time, ever, you know?

MIKE: Me neither [laughs]. You can't make it go away.

WILL: Yeah, so just, you know, pull the band-aid off fast, and get into it.

MIKE: Cool .Like I said, I think this was a great discussion. Hopefully [chuckles], you walk away with some good ideas as to where to go next. And until next time on the Acima Development Podcast.

This episode highlights how unforeseen career opportunities, such as web development, emerged due to exponential technological growth, which is often difficult for humans to predict. The panel notes that AI is the latest technological frontier, poised to impact software engineering in ways that were previously unimaginable, much like the rapid changes brought by smartphones and the internet. They reflect on how technological tipping points often arrive unexpectedly, transforming society and work in unpredictable ways.

The conversation then shifts to AI’s role in programming, particularly tools like GitHub Copilot, which significantly enhance productivity by automating repetitive coding tasks. While AI accelerates workflow by handling syntax and boilerplate code, the panelists emphasize that human expertise is still required to refine and validate outputs. They explore how AI might evolve from merely assisting developers to potentially managing entire software architectures, leading to shifts in job responsibilities. However, AI’s ability to completely replace developers remains uncertain, as it struggles with higher-level orchestration and nuanced decision-making. The discussion also delves into how AI’s increasing context awareness and rapid iteration capabilities could reshape programming roles into more supervisory, product-management-like positions.

The panelists debate the long-term implications of AI on software engineering careers, raising concerns about the diminishing need for junior developers and the future of technical education. While some anticipate a future where coding is more about directing AI through structured prompts rather than writing syntax, others argue that the demand for precise human understanding will persist. They also touch on potential regulatory concerns, the challenges of AI reliability in mission-critical applications, and the broader economic impact of automation. The conversation concludes with a reminder that while AI-driven transformation is inevitable, adaptability and continuous learning will be crucial for developers to thrive in an evolving landscape.

Transcript:

MIKE:  Hello and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting again today. I've got a great panel with me here today. I've got Justin, Will, Dave, and Kyle. And we are going to talk about the future and let me start by going back to the past.

So, a hundred and fifty years ago-ish, we'll say, more or less [laughs], almost everybody had a farm. It's just what you did. Maybe I could go back a little further, maybe 200 years. But if you go back within a reasonable time period, there were people in cities who had businesses, industries. It's not that industry didn't exist, but it was very much on a...basically, a home scale, right? You couldn't really have factories because nothing had been automated to that point, and they didn't have machines to make factories work. So, you might have a lot of people working together in the same building, but that's about as far as you got in terms of a large-scale industry. And, again, almost everybody was in farms.

And if you had said to somebody in that time, “Well, you know, in the near future, nobody's going to be working on farms, and [chuckles] you're all going to be living in the city. And a lot of people are just going to have jobs like standing at a counter and feeding you food,” they would laugh in your face. It was just patently ridiculous on its face. There's absolutely no way that would happen. And then, a few changes to enable automation led to an industrial revolution and World Wars, [laughs] and to the point now where we’re somewhere between one and two percent of people actually work on farms. And the economy has just completely transformed.

To take that a little bit further into the future, when I graduated from high school and was looking forward to my future careers, you know, what career I might take, I never considered web development because that career didn't exist [laughter]. I couldn't have wanted to work that job because there was no web to work for, you know, there was no popular internet. I couldn't have even conceived of it because nobody had. Maybe a few innovators who thought, oh, maybe these early browsers will accomplish something. Yeah, it just plain didn't exist. And so, most [chuckles] of my career has been in a field that didn't exist when I graduated from high school, which brings us to today.

If you look at areas of exponential growth, things seemed normal for a while, and then, suddenly, they're different. And exponential growth is really hard for us to reason about because we tend to think linearly. You think linear growth, you know, it'll happen gradually over time, but that's not how exponential growth works. And technological change tends to be exponential when you have people in a large group together collaborating. You can go even further back, go to the printing press, which enabled the rise of culture and modern innovation and science because people could share information with each other. And the internet has provided a similar set of growth where we can get access to information so quickly. It enables exponential growth in technology.

And there are new technologies, like AI, that have promised to change software engineering. And the thing about this also is it's really hard to predict what it's going to look like in the future, you know, because you follow a trajectory unexpected things happen, right? You have tipping effects that change things.

I really wouldn't have thought in 2005, in a few years, we would have a phone that would make us all just retreat to ourselves, teenagers learned to stop writing [laughter], and a total change in society where people quit sharing the same reality because they watch different news sources. Like, this is going to happen from a phone in the next decade? What? And these things are really hard to predict, but they happen. So, we are going to go out on a limb today and start doing some speculation, and we're going to be wrong. But [laughs] –-

DAVE: We hope. We hope.

MIKE: [laughs] There are some things that we're probably going to see, and this is a chance to talk about what we see coming up in the future. And there's some value that comes from this because we can start planning some trajectories for what we might want to be doing in the next few years. Because there are some things I think we can see, and then there are some things we're not going to see, and we're going to go back, and we're going to laugh at ourselves [laughs]. But it's a chance to lay something down and give ourselves something to laugh at five years from now when we realize we were totally wrong, or maybe see that we saw some things. That's my lead-in.

So, I could ask a leading question. First of all, any immediate thoughts? But then following that up with a question, what do you think is going to be most impactful from these new technologies? I'm going to talk specifically about AI because that seems to be the big thing right now. Five years ago, everybody thought it was crypto. It's a thing, but it's not ‘the’ thing, right [laughs]? And --

DAVE: That's a separate discussion.

MIKE: That is a separate discussion. Probably not going to be relevant here because probably isn't relevant. That's not the [inaudible 05:30] thing, but this AI thing seems like it might be, even though that's been going on since, like, the ‘50s.

DAVE: I'll give the metaphor that I gave in the pre-show, which is that a year ago, people were asking me, well, programmers, “Is AI going to take over my job?” And I'm like, no there's no way AI can understand at the high orchestration level. You can refactor a function, but you can't hold up the whole app. And a year later, like, I slowly changed over a year. I'm like, ooh, the junior programmers are in trouble, but the seniors are okay. And then, I'm like, oh, the seniors might be struggling as well.

And the metaphor that I gave at the top of the call or in the pre-show was it's 1970, and we are welders watching the car industry wheel robots, welding robots off of the truck, and things are going to change. And one of the welders at the company is going to become the robot manager who is in charge of orchestrating everything and keeping everything, and everyone else is going to get jobs in construction. That's what my dad did when the mine shut down in the 1980s. He'd been doing deconstruction on rock underground, and he's like, well, I can go dig stuff up and tear it up above ground.

But he had to pivot because the uranium industry went away, and our industry isn't going away. Like, everybody still needs their web page. Everybody...well, okay, it's 2025. Everybody needs their podcast, and it's still happening, but I genuinely think we're going to see a lot of industry change.

WILL: Okay. So, I’ll say, all right, I'm hearing you. And what I have seen personally, like, I'm not a [inaudible 07:04] right? Like, I'm a technologist. I'm a future-looking dude. That's why I'm still in the business. And I've used ChatGPT...not ChatGPT, I've used Copilot. Copilot is the thing I've used.

I have used it to great success patching up things that I find tedious and unpleasant, and you know I'm talking about unit tests. And I love it. And I love it. I love it. I could say like, "Listen, I want a test that does this, and this, and this." And it will get my mocks right. And it'll get my imports right. It'll write out the thing. And it will give me what is roughly analogous to, like, if I had a very energetic remote developer on fiber that was putting something out, and they didn't particularly care about the quality. They just wanted their 5 bucks.

And I could take that, and I can give you something good because I know what I'm doing. And I could do this myself; I just don't want to. And I’ve found that to be very useful in a healthy 20%, if I'm being generous and optimistic, which is my nature, 20% productivity bump. I'm getting more done faster. I am not sweating. It seems like you've maybe started the year, started last year, let's say, right, 2024 that's where you were sitting. 2025, you're like, okay, I don't know, right, talk to me about what changed and what you're seeing, the specific terms.

DAVE: I resonate with what you just said about Copilot making things easy. There's an xkcd comic, number 1168, I just Googled it, which, basically, they're sitting over a bomb, and the screen says, "In order to disarm the bomb, enter a tar command with all the switches correct from memory on the first try." And he looks up and he says, "I'm so sorry." [laughter] And nobody knows which switch is the tar, right? I do. I do. I read that, and I'm like, ZXVF to unzip and...anyway, yeah, that's me.

WILL: Yeah, David. Yeah.

DAVE: That's me. Like, I got it. And what I use Copilot all the time for now is give me the Rails generate command to alter this table at a unique index, because you can do this on the command line, right? You can do Rails generate migration, and then you can say, add deleted at to customers or add extra columns to customers. And then, you can type space, and you can do delete it at colon timestamp colon index. If you're going to say customer is going to belong to widget, you can say widget ID colon references. And that literally tells the database to set up a foreign key, and it's going to build the model and build it up.

But I can't ever remember the CLI syntax. I just remember that it exists, and it's super convenient. And if you know the change that you want fits in that little DSL, it's so much faster to just write the command, hit enter, look at the file just real quick to double check it, and then run the migration, and you're done, especially when you're, like, greenfield development, right? It's real fast. And that is what I'm loving. I can sit it down, or I can say, “Give me the command to get the URL of the homepage of the project that I'm currently in the directory of in Git. I want to ask Git, Git remote show origin," that sort of thing.

And I love that Copilot comes back and tells me things because I tore Git all the way down to the bolts in 2010. And I wrote all these scripts to help me manage how Git works on Git version 0.7. There's a lot of development that's happened since then. And Git remote show origin Git was the only way to get the URL of the fetch URL push URL, and you could massage that and turn it into the homepage. And Git remote show will contact the origin and say, "Give me all your data or give me the information and give me the name of every branch because I'm going to have to build the remote's origin thing for this guy."

So, it's like a four-second thing on a big project, on a big server. And I was passing along something, and I asked the AI, "How do I do this da, da, da?" And it mentioned, “Oh yeah, by the way, you want to use Git remote," and it was like, "get URL," or something like that. And I'm like, that's a thing? It is a thing. It only gets the URL, and it's instant because it reads your git config. And I'm like, I needed this 10 years ago, but I solved it back then, and I never circled back because it worked. I love Copilot. It knows the switches, and it knows the latest thing.

WILL: Yeah, okay, sure.

MIKE: But what makes you change your opinion? What makes you change your opinion to think, wow, this is going to do more than giving me Git commands; this is going to be able to architect an app and build it out? What makes you think that we're going to go to that next level? Because there's a substantial jump between those two.

DAVE: Oh, absolutely.

WILL: I'm just going to say, give Mike's answer first, and that's fine. But then when we're done doing [inaudible 12:17] now we can do what separates the real pros and the seniors from the other stuff, which is not greenfield. This field is brown and dirty. There are barrels poking out of the ground. Nobody knows what's in them. There are rocks. There are stones. There's rebar poking up, like –-

DAVE: And it's not hygienic either. It’s got [inaudible 12:43]

MIKE: [laughs]

WILL: [inaudible 12:43] The children must play. And I'm in here to make this into a preschool [laughs].

JUSTIN: So, I got a couple of thoughts on that. One is the context window for all of these tools, you know, the LLM tools have expanded to the point where you can get an entire project into the context window. And that allows you to, you know, a number of things, one of which is, like, it can figure out where to add the changes that you want to change. Like, hey, I want to move this widget 30 pixels to the right on every page. Or you can craft your query like that, and it will give you what it thinks that you should do. So, that's getting better.

But having said that, I read a really cool article, probably two or three weeks ago, about how, to your point, Will, LLMs right now give you 70% of the way, and it's that last 30% that shows whether or not you are, like, a senior engineer or not. And it'll give you a project and everything but resolving all the little issues that pop up because of hallucinations that are hard to track down that’s still --

DAVE: Recognizing it's a hallucination at all, yeah.

JUSTIN: Yeah, and handling the edge cases. So, that, I think, is where we are currently, at least where I'm at currently with the tools. And getting there, you know, here we are 2 years into this revolution, and where are we going to be 5 years into it from now? 70% today and in 2 years, and then, in 5 years, are we leveled off. Or are we going to continue to see steady growth? I don't know. If we see any growth at all, that'll go from 70 to 80 to 85, and that last little bit I'm hesitant to say we'll ever get there. But the people who are doing the work are going to get more and more productive, hopefully.

WILL: One, are we seeing AI development accelerate so that we're seeing more progress faster, or are we seeing plateauing? Because I'll be honest with you, what I'm seeing personally is a plateauing and a refining of all the output.

DAVE: I have some data for you.

WILL: I'm not seeing exponentially...well, I mean, --

JUSTIN: We went from here to here really, really fast. But now it seems like it's just barely increasing.

DAVE: Yeah. So, it’s a lot --

WILL: It feels like it’s slowing down, but I could be wrong. And the second question, and this is just really specific, is like, how many thousand lines of code of context can I get into an LLM? Because, I mean, really? My understanding of the LLM is that it's an exponential growth in complexity as your context size increases because there isn't a semantic understanding of what all this stuff means per se, per se right? It doesn't translate into the symbols like we do. And so, you know, just, like, 2000 lines, 3000 lines, 10, 000 lines that's not, like, a linear model increase. Does anybody what that context is?

MIKE: Yeah. So, Transformers, which is the tech that's been used for all of the LLMs, it was a technology that came out, was it 10 years ago or something? And the way that those work mathematically, that is true. It grows exponentially by the context length. However, however, there are workarounds. Those are moving forward quickly because you can take groups of things and compress them and put them together. And that's really how our brains work anyway, right? We think about things hierarchically. And by approaching it hierarchically, you can actually address larger and larger windows. And now they're into millions of tokens. So, yeah, much longer than just a few thousand.

DAVE: Yeah, ChatGPT I just asked it, “What's your context?” And it's like, “4K,” And I'm like, really? I was playing with Mistral last night, and it's, like, 130k. What are you doing? But that was in the last 12 months. When I was first going out to Hugging Face last February, and it's like 2024, so a year ago, 8K was big. And the LLaMA-30b had just dropped, and everyone was freaking out about it. And now they’re like, 405b, I mean, they're just getting bigger and bigger.

So, I said logarithmic, but it's the same thing as the exponential. We're just on the bad side of it, which is that if you want to double the things you can think about, or if you want AI to go up, like, 1%, you got a double its capacity. And if you want to go up another 1%, you've got to double that capacity. So, we're on the bad side of that. But what we're seeing is, like what Mike said, specific breakthroughs. Like, somebody will attack a specific problem. In the last 12 months, they've solved...in image generation, they've solved the hand problem much better. It still happens a little bit, you know, seven fingers or, three thumbs, or the fingers are on backwards, too many arms, that sort of thing, good time. But I'm not seeing that anymore.

I went out and built a character for a comic. There's an AI that will let you generate, like, graphic novel-type stuff, and you can develop characters. And it had trouble generating you the same face two images in a row. You could do four in a row, but they would all look similar in different poses, but your next slot, completely different all re-randomized. They've solved that. You can literally just say, “I like this one. This is the face. Please make everything look like that.” So, I genuinely think this is coming.

The general trend line is, we just keep adding more and more and more and more. And I think, eventually, we're going to get to a point where the width is big enough and the speed at which it tries and tries again to basically cheat...you can have twice as much AI if you just run it twice, right? And the same idea. Cross-intentional stuff. This blew my mind. I watched this a couple of months ago. The way they made it so that you could say, make me an image of, you know, a whiskey bottle made out of yarn, right, is while they were doing all the Stable Diffusion, like the pixels blurred, now resharpen, now fix it, they were doing that.

At the same time, a completely different text-based AI was being told this image has a girl in a red sweater. This image has an angry woman in it. This image has a dog that is happy. And the genius move is somebody said, let's tie these two together. And so Stable Diffusion when it was blurring and sharpening was also being fed, this is what the English...and, like you said, AIs don't think the way they are. It was not, this tells you what the image looks like. No, it's like, this is the statistical probability of the words that you might hear in sequence correlate to pixels in sequence over here.

But the end result, once you've thrown 24 trillion tokens at it is that you can say, "Show me a whiskey bottle that’s, you know..." And you can tell it crazy, crazy things. And if it knows... “Show me a whiskey bottle with irritable bowel syndrome.” You're going to get an image if you do that.

MIKE: [laughs]

DAVE: I haven't tried it. I think it'll be funny if I do, right? That kind of thing. And it'll do it. And so, it's a trend line. I genuinely think...I guess what I'm saying is I am startled by the amount of things that are now trivially easy to do today that I was certain a year ago we wouldn't be able to do. Are we going to be out of a job? No, absolutely not. There's going to be people managing the AI. Until the robot platforms come out, humanity has a chance, but we'll be the operators running stuff. But yeah, we're going to lose stuff.

Real quick tangent, my father-in-law was a Renaissance man, machine operator, machinist. He built engines in his garage as a hobby, like, literally gasoline engines that were...they would hold pressure, spark, flame. I mean, he would machine things to extremely tight tolerances. He made his own bolts on one project, literally turning them on the lathe.

And he had the entire skill set from woodworking to metalwork to engineering to metallurgy. Like, he knew how to temper steel to different properties, like, he knew the full stack. And when I need something built, I go to Amazon, and I order something from China. And the Renaissance people are going away, people who worked in their garage building engines as a hobby don't exist anymore. And they were shade tree...you could find one on every street 30 years ago, and now they're dying out. And I think that's what we're going to see.

The people like us that are crusty, old farts that we actually know what transistor-transistor logic is, we're going to be surplus to requirements. The next generation of programmers are not going to be able to...they are going to detect hallucinations like we are, but they're going to do it on a completely different basis. They're not going to come to it from knowing everything that the software is built on. They're going to know it by recognizing patterns. And like, oh, yeah, last time I told you to do this, it didn't work. Or, oh, you told me this, but I need to test it, and it'll just be a habit, and it will go along. And then, somebody will write an agent to do it, too.

JUSTIN: Go ahead, Kyle.

KYLE: I was just going to say, even without AI, though, we're hitting that right now. Juniors that I've met they don't necessarily know how a computer works. I mean, we're always having to adapt and expand. And that's where I'm listening to some of this, and is it something that will take our jobs and we'll be replaced? And that's where I have a hard time thinking. Well, no, not unless you let it.

And what I mean by that is we're all going to adapt. We're all going to be doing it differently. Programmers, you know, they were in C, or C++, or any of those languages that were not memory safe. Those are not popular languages now. We have languages that handle that...who a few years ago would have used Stack Overflow. We have Stack Overflow. That has sped things up. I think things are just going --

JUSTIN: Actually, Stack Overflow is going away, but anyway.

KYLE: Well, I'm saying, the progression, right? But I'm just saying, if you adapt to the changes, the world does move faster, but the demand really has not decreased. It's just increased. Like, do things faster, do more. And so, I don't know that the jobs that we have today will exist, but they'll be different and plentiful.

DAVE: That's the way, yes, --

WILL: Well, and --

DAVE: The job you have right now is going to go away no matter what. You're not working on the same stuff today that you were two years ago. We're not going to lose our jobs and starve and die because of AI, just things are changing, right? I don't build WordPress blogs in PHP anymore. I don't solder anymore.

KYLE: Thank God.

DAVE: My job has moved on. Those jobs have gone away, but I am still employed and employable, and that will never change until I die, hopefully.

WILL: Well, I mean, until you fall further and further behind the [inaudible 23:58]. I mean, it is a situation where if we all get 20% more efficient, then some of us are going to, I mean, the weak gazelles in the herd are going to get called out, and we're in the middle of that.

JUSTIN: I just have a question for the team. I mean, we're here in a podcast, a development podcast. What can developers do to not be forced out with the next wave of things? Ideally, we all want jobs, and this wave of more efficiency means that maybe there'll be less jobs. But, ideally, we want to know how to use those tools and be one of the people that is advancing and using it rather than being run over by the car.

MIKE: So, something that was said a minute ago, most people aren't writing in C, I think it was Kyle, aren't writing in C anymore. But it doesn't mean people quit coding. Most people aren't writing in Assembly languages anymore either, which is what...people certainly aren't using punch cards [chuckles] very much to program a computer. There might be somebody out there that has to run a legacy machine. It could exist [chuckles], but it's very limited. But we're all still coding, but the nature of coding may, is, may change.

A couple of years ago, I read an ad...there was an ad for Anthropic, a company that's gotten really prominent in the last couple of years, where they were asking for a prompt engineer. And I think they were offering a 300,000 a year salary, and they couldn't find anybody.

DAVE: Oh wow.

MIKE: Because it didn't exist. They couldn't find anybody who had experience doing that because nobody had done that yet. But, a year from now, two years from now? So, here's a prediction that some people can hold me to and see if I'm totally foolish. Five years from now, if you can't be good at prompting an LLM, I think you're out of a job because I think that becomes the new language.

DAVE: In this industry, yeah.

MIKE: In this industry. I think we may even have new programming languages that arise that are designed to be friendly to AI rather than friendly to whatever the language constraints are right now, and they'll compile down to whatever. That will fundamentally change what we're doing. And it may look a lot more like product management, where we're gathering requirements and expressing them really, really well for a good product manager. It may look a lot more like product management than loops, right?

DAVE: I have a corollary prediction which is going to be really, really interesting. Source code is a thing that you run through a compiler to talk to the computer. But it's written to be read and understood by a human which means good programmers write source code that other humans can read and can interact with.

I would not be surprised in the next five years if the prompt becomes a version of source code, and if we actually start seeing people who start to say, “This prompt and that prompt, this one is more efficient, but this one is more robust. And oh, and by the way, this one, a human can't understand it. They need an agent to do it. So, it's actually bad. Let's use the other one.” You're going to see people...you need to write a prompt so that a human can understand it. The AI will turn it into transforms.

WILL: I'm a hard, hard, hard no on that. I think natural human language, English, spoken English, human language is absolutely terrible, like, atrociously bad for the kind of precise interoperability, for the type of precise design requirements. It is atrocious for that. And we have tried to develop sort of specific language around, like, legal contracts, and it's wretched. It's like we have...we spend billions every year arguing about the specifics of legal contracts, which are not, I mean, they're not on the same level of sort of specificity and complication sort of like source code is. We couldn't do it.

So, I think, I actually think source code for your modern high-level languages like Ruby it's actually an excellent example of something that is source code but is very readable on a human level. I think we'll see something a lot like that.

DAVE: I agree. I think a prompt language might be coming, yeah.

MIKE: Yeah, now, that's exactly what I was suggesting. I think we're going to have a language that's up at the next level. We don't write machine...we don't write Assembly code. Most of us don't write compiled, a lot of us don't anyway, compiled languages anymore; we do interpreted languages. Put another layer on top of that.

Now we've got the prompt language, which is just...will have to be exactly as precise as what the lower-level languages were. But it's a specification of what you want from that project. And the people who are very detail-oriented, just like today, the people who are very detail-oriented, and are able to communicate with precision what exactly they want the system to do will be successful. Because, exactly like you said, Will, if it's not precise, it's going to be wrong. And the need for precision is not going to go away, I don't think.

JUSTIN: Okay, so I want to walk through, like, a typical workflow five years from now. So --

DAVE: Before we jump into that, can I --

JUSTIN: Let's put our imagination caps on and suppose we're working for a web retail company, who could it be, who is selling...they're creating a new form to gather data, or something like that. I don't know. They're doing a know-your-customer data entry, okay? So, you have the product manager, and then you have the engineer. And they're sitting down together, and they both have their laptops in front of them, or maybe their headsets.

And the product manager says, I want to collect, you know, first name, last name, date of birth, driver's license number, SSN. And the other hidden person in the room is the AI, the Figma AI, so suppose Figma is listening, and Figma is listening to that. And what happens next? I imagine that Figma will be listening to this and real time Figma has, up in front of it, the web page with their company's UI.

WILL: Design template, right? And one of you got to style that, right?

JUSTIN: And these things appear on the screen. [crosstalk 30:47] And these fields appear on the screen as you go along. And the engineer is not doing anything at this point.

DAVE: He's steering the UI, though, in real-time, right? Like, make the social security bigger, make this field required.

JUSTIN: Yeah, exactly. And so, it's like, hey, make this field required, a little star shows up next to it. So, in real-time, they're having this conversation, and these things show up on the Figma UI with the company's contextual UI as all part of it. So, they go back and forth with the engineer and the product designer or the product owner. They go back and forth until they reach something that looks good. What happens next?

WILL: Then we're talking about it. It's like, okay, this is what I need. Then you've got the senior engineer, right? And the senior engineer has a good understanding of the topology, right? The various microservices that need to get looped in the sort of ops resources that need to be allocated. And so, he is going to talk through the architecture, right? He's going to talk through the architecture and be like, okay, I need to go, you know, the last [inaudible 31:52]

JUSTIN: So, is copilot listening?

WILL: Yeah, yeah. Let me finish [laughs]. I need to go out to Galactus and get date of birth so we can send everybody a birthday card. And then, I need to go to thus and such service that says, okay, this is the thing. And all the while, boop boop, boop, I need to go here and get this. I need to go here and get this. I need to go here and get this. And the AI is saying, “Okay, you need this thing.” And then, I'm going to create an epic for this sub-team.

It's like, okay, boom, I need this from this. It's going to take the meeting transcript and an understanding of the thing. It goes to the sub-team for that engineer. And then, you've got a, I would say, probably an asynchronous kind of a thing where it's like, okay, we need this. It looks like this. This is how this thing goes for you. There's still going to be people involved in the steps. But all of the incessant meetings and sync-ups and stuff like that it knows what the org chart is. It knows what the subsystems are. It knows who the person in charge of those things is.

And then, you have the follow-up meeting where, like, okay, we take it I'm Galactus, right? And I need the customer information, and then it describes that. And I say, like, okay, this is it. This is the...I think a lot of it is going to be just people are going to look a lot more like...they're going to look a lot more like technical program and product managers where you're the engineering manager for the AI. But you're going to code, and you're going to need to code because there's still going to need to be a body who can say, "This is good, bad or indifferent," you know what I mean?

JUSTIN: I mean, you've got to have somebody who knows it. But when you're at the start of the project, you may not need somebody who is actually coding because you can go in. You have the OpenAI, sorry, OpenAPI specs for all the services you currently have, and they could be represented graphically somehow. And you could, as you're chatting, Copilot is going in and giving you the list of those things. And you say, "Hey, I want that one," and it gets dragged over. And you say, "Hey, I want that service," and it gets dragged over.

And so, these connections that happen, I mean, theoretically, the code between the front end and the back end and then the back end into another back end all of that could be done automatically. You do have to have the engineer to make sure that it’s making sense. But it'll go in. It'll check the parameters that exist already. And then, you'll decide, oh, we don't have a data store for this yet. Go make, based on my GitHub template, go make a new project that has a service that stores these in our standard data store and that will follow all our standards and everything else. This is sounding very hypothetical, right? It's crazy. Maybe not 5 years, but maybe 10.

DAVE: You haven't said anything that I don't think we could build right now with a tiger team, legitimately.

JUSTIN: Great.

I've been playing with storytelling LLMs, and they really, like, Mistral is not going to generate JSON. It can't do it. It can't even keep quotes lined up in English text, but it can tell a compelling emotional story, that kind of thing. And it's really, really good at making inferences. This is the last thing I'll say about [inaudible 36:06], and I’ll let you guys take the floor. But the prompt engineer...the stuff that I'm doing right now is I will write a story where husband and wife are arguing, and he gets mad, and he wants to walk away. And she's like, “You can't walk away from me.” She grabs him by the arm and pulls him back. And I'm like, no, she would not do that. That's wrong to the character.

So, I have to tell the AI, “She's not going to grab him,” and then regenerate. “You can't walk away, grabs him by the arm.” I'm like, why did you ignore that? “Well, the narrative tension needed to escalate.” I'm like, oh, okay. And you end up going to the wife and then changing. Oh, I see she's got vindictive as the first trait on her personality. I'm going to move that down three and move supportive up to the top. Regenerate it. And all of a sudden, it's a completely different story. She's like, "Don't walk away from me. I need you to understand me". And it's a completely different tone to it.

You can give the AI 7500 tokens of this is a young boy, and he's got some interesting exotic powers, and he's gone to a school to learn. But you can take a very small AI with very low context, and you can say, "This guy is Harry Potter," and it just got 10, 000 tokens worth of inferences out of that. That's what prompt engineering I think...it's not headed there. It's there right now, at least in the fiction category. And I think when you marry that to Copilot, it's going to be terrifying because you'll have sloppy. You'll accept slop but emit precise. That's going to be a good day.

WILL: I feel like one thing that I think will really...I think a quantum leap in terms of AI's ability to generate software is the ability to sort of...so, you have a self-training model. I build these things, right? And it's like, well, does it render the webpage, or does it not? So, one of my superpowers...this is something that I do all the time because, sadly, I have progressed at a point in my career where people give me just the worst, just the worst jobs. I get nothing but the absolute worst jobs, worst jobs [laughter].

And it's like one of my models, right? This is a simple model, and I got it when I was developing really wretched stuff, embedded system device drivers, which are hell on earth to debug. But this is just a simple intellectual way to make nearly anything work, and you just debug from the known to the unknown. And what I'll do is I'll have some component. I don't even want to tell you what I had to work on today. It would curl your hair. It would be terrible.

But I had this front-end monstrosity. It wasn't working. And I'm like, okay, listen, we're just going to gut it. We're going to gut it down. I'm going to tear this thing down to the studs, and it's going to print, “Hello, World.” That's literally all it did is, like, print “Hello World,” and I got that working. And then, I started adding things in brick by brick by brick.

DAVE: You got a working piece.

WILL: Until I found the thing that was broken, right? An LLM could do that, an LLM could absolutely do that. And then, you have the model, which is training itself. I'm training myself. It's like, okay, I'll try this. I'll try this. I'll try this. I'll try this. And we could find ourselves in a situation where a lot of my job is just wretched, ditch-digging work where I'll get up into the muck and just stay here until it goes.

Well, an AI could do that job, and I could just be like, this thing doesn't compile. Why doesn't it compile? Make it compile. And the AI could be like, okay, I'll just tear it down to the studs, and I'll add things back in line by line until it works. And that's a slow process, but you could just be like, all right, I'm going to bed. Call me back in the morning. Let me see what you found. And I'll say another thing. You would need this thing. But one of the things I'm grappling with right now at my large, big box retailer, which is not Rent-A-Center. It's not Rent-A-Center. I bet it's better than this [inaudible 40:30]

DAVE: So, you're working for a competitor? How dare you?

WILL: Yeah, anyway. So, I'm [inaudible 40:34] that a big box retailer with a lot of legacy problems. And one of the things that I'm grappling with right now is migrations, where you're migrating one bad service to another thing, right? And so, you migrate it over. You run into real problems, real serious issues when you don't have the operational discipline to finish one migration before starting the next migration, then your problems compound in a truly brutal way. Well, those migrations are extraordinarily painful and the source of a lot of brownfield misery.

But if you had an AI that could generate products and iterate itself and say, “Okay, here's the old one, and here's the new one. Are they the same? No? Fix it.” And that's sort of when you have the template, what you could do is you could execute these massive migrations automagically, close to automagically. Then you could really start to...and it is a dumb ditch-digging miserable process, but that's the kind of thing where you could accelerate development tremendously.

MIKE: So, you're talking about what sounds like reinforcement learning from human feedback.

DAVE: There's an AI listening to this podcast a year from now, as it's ingestion, and it's going, I wonder if I need to be thinking about how the whole system hangs together [laughter].

MIKE: Reinforcement learning from human feedback is what allows ChatGPT to work. They train it, and it's off based on terabytes of data, right? And then, they show it to a human. Here's what I think you meant. And they say, “No, you can go this way,” and it learns from that. And they do that through iterations. So, practices generating the text and then gets human feedback and based on that, over a number of iterations, improves to better match what humans want.

You're talking about taking one system matching it to another one, and it gets it wrong. Get some feedback on that and then keep iterating to get it right. There exist algorithms today to help with that. Also, some of the newer algorithms that exist are no longer bound by a fixed amount of calculation. That is, it comes up with a chain of reasoning where it comes up with the next step, and then the next step, and the next step, which has an indefinite number of steps, right?

DAVE: Oh wow.

MIKE: And it will generate based on that approach. The latest OpenAI models, for example, are built that way. And that changes the way that these problems get approached to be closer to the kind of human-driven algorithm that you're describing. So, I think that we, again, on this one, I'm not making a prediction on because I think it's really hard to say whether that really will be the key. But there is a chance, because there are hundreds of billions of dollars being poured into this, there is a chance with all of that mental energy, all of that compute going into it, that somebody is going to find the algorithm that will continue because they're continuing right now to find new algorithms that push it forward.

JUSTIN: Yeah, we're so early in this.

DAVE: And where we focus our attention it goes fast. Like, the grass is greener where you water it, and that's really, really true right now in the AI side of things. They're taking programmers that are complaining about their jobs and saying, “What are they complaining about? How can we help with this?” I love TDD because it's a completely different way of testing. It needs a different name because people think, oh, I'm just going to write the tests that are at the end I'm just going to write them at the beginning. It's not like that. It's a completely different discipline entirely. It's a completely different experience.

But most programmers don't do that, they prefer to test after, and there's all kinds of anti-patterns that start to happen as a result of this. And the Copilot stuff, and Gemini, and stuff that's out there is optimized to solve that pain. It will write your crappy after-tests for you, but nobody's working on the TDD thing. So, hang on, I just had a business idea. I got to write something down [laughter].

MIKE: So, we've talked a lot about what we see things looking like in five years, higher level languages relying on the AI to do a lot of the grunt work. Potentially, developing some of the processes we use to deal with ugly code, with code that is in bad shape, in a bad environment, bad all-around situation, industry, you name it, it'll help you with those things. And if we do, are jobs going away? Yes? No? It seems like, generally, the answer is no. We still need people building stuff. But our expertise will have to change.

DAVE: Where do we need people right now? That's going to shift. We're just going to be sidling side to side to side to side. And a hundred years from now, only 1% of us will be farmers. And they'll be producing the food for the world but in AI, yeah.

WILL: Yeah, I don't worry about myself so much just because I've been doing this for a long time, and I know a lot of things that I don't even know how I know them. I just know them. I just know. And I don't need to know a language. You could drop me in a Go project tomorrow, and if I had Copilot or something like that, I'd be fine. I'd be fine. It's like, how do I set it? Oh, that's the core routine? Okay, that's fine. How do I do a loop?- How do I do a thread? Okay. How do I make my API calls? Okay. I'm done. Moving, moving. Chop, chop chop. No problem.

I worry more about the junior developers that are coming online right now because I know how I learned it and that sort of, like, old-growth, forest free reign kind of a thing that's not going to be tolerated. I feel like we're eating our seed corn on a large degree in this industry right now. And yeah, Justin, go ahead.

JUSTIN: Yeah, so this hits close to home because my oldest child he is graduating from university in a year from now with a computer science degree. And chatting with him, the degree that he got from Utah State University, which is the northern end of Utah, it was very much a classic computer science degree. It had more to do with my computer science degree than with perhaps the prompt engineering that you see.

So, he's deep in algorithms. He's deep in the classics. And he used C++ of all things, memory allocation, all that, you know, close to the metals type of stuff, not terribly close to the metal. He's not writing in Fortran or things like that. But at the same time, I'm chatting with him and trying to figure out, okay, what are you going to come out and be able to do?

And he's having trouble right now getting an internship, whereas a couple of years ago, he'd be snapped up right away. And not only that, he's, like, at the top of his class. He's keeping his scholarship, so he's got, like, a 3.85 GPA, which is hard to do as a computer science major. But yeah, we're trying to get him an internship, and I'm worried about him getting his first job, so...

MIKE: I have a nephew in the exact same situation. He graduated six, eight months ago, still doesn't have a job.

DAVE: I think we have another podcast topic there because nobody wants to hire the core thing. They're looking for people that are amazing on their side hustles. Side hustle is the wrong word, the other things, right? You don't hire somebody who meets the job requirement and says, "I guess I can do the thing you want to do." You hire the person who comes in and says, "Yes, I can do that. Also, I care about this, and I have a degree in, you know, I have a Juris Doctor because I went to law school, or I have this other thing, or I've worked in finance." That's what you hire for.

And when you're very, very young, the side hustle is completely different. You want to hire the kid who stays late without being asked or cleans up his desk or her desk without being asked to. Or one of the best personality tests I've ever seen...I was in church and one of the senior church officials was standing there, and there was a gum wrapper on the floor of the chapel.

And the senior church rep went over and bent down. And I'm in a church where hierarchy is kind of important. Somebody that high, you're not supposed to make that person pick up litter. And he did. And everyone in the room went, “Oh wow.” And that's not in the job rec for the thing he was doing, right? But that's the difference between getting this job and having a career that will go.

KYLE: There was something else that Justin brought up, too, and that was the C++. Now, I went as a computer engineer, so I did C++ as well, but I'm aware that other universities do C++, but others will do C# Java, which is one step higher, I guess.

JUSTIN: That was in there, too, but yeah.

KYLE: But, yeah, no the point that I'm trying to make here is that we might need to, I mean, this might be an institutional problem, too, because a lot of this is theory. They're teaching theory-based learning and it's all algorithms and stuff like that. We're not actually teaching engineers coding. We're not teaching them prompting and stuff like that. So, it might be a problem at the fundamental, like, schooling level before they can get the jobs.

MIKE: This is not a new concern. For years, people said, “Well, we need to give people more practical, hands-on stuff, trade school stuff for engineers rather than the theory.” And I might push back on that. I think it was Will who said we're eating our own seed corn. I think that's true. And I think that's closer to the problem because, honestly, people who understand what's going on and can understand what happens when the prompt doesn't give you your answer are going to be incredibly valuable, I think, in the coming years.

I think that we need to have that pipeline of people who actually get the things that are happening at a lower level. And they're not going to be that useful right up front. They're going to have to take some time to learn it. They're going to have to learn those skills, but you need both if you want to be really successful. I do think that there is a shortage right now in the industry of companies that are willing to invest in their future. There's some uncertainty.

I think it also has to do with the change in inflation. When you get different interest rates, it changes where the companies want to invest. We had really low interest rates for a long time, so companies would invest in their people. Interest rates go up they say, “Oh, we can make money other ways, so they drop all the people.”

DAVE: [inaudible 51:43] operational, yeah.

MIKE: I think that's a lot of the problem in the computer science industry right now. It's just a lot of big tech is firing because they can make money through investment. However,...go ahead.

WILL: I mean, I'd say having gone pretty far past a regular old bachelor's degree, there are a lot of professional fields where you're not working with a bachelor's degree. You don't work as a doctor. You don't work as a psychologist. You don't work all kinds of things without post-bachelor's education.

KYLE: Post engineering.

WILL: [inaudible 52:21] is a master's degree minimum field to be in. You don't know shit with a bachelor's degree. I'm sorry, you just don't. It's just too much. You can be smart, and you can be trained. And most people with a bachelor's degree and a good GPA are smart, and hardworking, and motivated, and absolutely can be trained. And after they get that sort of finishing school, which we've been dumping off on industry, right? Historically, like, the first...it's just no.

The first year out of bachelor's you kind of suck, and you do what you can, and we give you jobs that you can take on, but we're investing in you with the expectation that you will pay us back later. A lot of managers have exploited that thing, and they underpay their people when they start producing, and then that's kind of a self-defeating cycle. But that was how it was always supposed to...how it was always intended to work. And I think that's just how it's going to go.

But, I mean, honestly, we've already seen them come for the boot campers. First, they came for the boot campers, and I said nothing, for I was not a boot camper. Then they came for the undergraduate interns, and I said nothing. That was not [laughs], you know. And then, they came for the senior engineers, and there was no one left [laughs].

MIKE: No one left because the AIs are already doing everything else [chuckles].

WILL: Listen, you know, if I'm that guy...I got to make it...well, no, that's not true. I got to make it in another 20 years, man [laughs]. I don't know, maybe I'll be the guy shutting the lights off, you know, walking out the door. But I'm consoled by the fact that having seen the incredible leaps and bounds the industry has taken over my career, they just want more. The more we give them, the more they want, the more we give them, the hungrier they get. [inaudible 54:30]

DAVE: This stuff is cool.

WILL: Yeah, this stuff is cool. And it's an arms race. It's an arms race, right? Oh, I have a hundred engineers. I double their productivity, right? Now I'm taking ground on my compiler. So, I've got features they don't have. I'm in markets that they're not in. I'm doing better things, right? And then, your competitor has to hire another hundred engineers to keep up because they just want more and more and more and more and more. I think that if we can get really, really negative and pessimistic, that is something to keep in mind. You know, Rent-A-Center, I don't know who you guys...who's you guys’ number one competitor? You guys got a competitor who you’re worried about.

MIKE: There's a couple of them.

WILL: What's that?

MIKE: Oh, there's a couple of them. I guess we could name them [laughs]. There's –

WILL: Well, no, whoever it is. If you guys do something better, well, then they got to do it, too. And they do something better than you, you got to get...and so, it's one of those things where engineering departments they're pretty durable. Head count is pretty durable.

DAVE: Is it though? Because the AI is taking over call centers. We're seeing call centers get gutted by, like, 90%. And that's right below the interns, and then they're going to come for the interns. The key thing, though, is that if AIs take over the interns, we're no longer taking interns and producing junior programmers and that pipeline of improving people. And that goes back to...a lot of the Renaissance people, I think, are going to go away, but also, we're going to get shot in the foot. We're going to have to find a way to backfill those basic skills or to obviate them so that you don't need them.

KYLE: And does that kind of go back to what Will was pointing out? Does that then mean that future engineers have a master's or a doctorate degree in order to get into the field?

DAVE: I don't know.

MIKE: There may be a cycle. There may be a time when we get greater credentials for a while in this kind of arms race. [inaudible 56:36] I'm good. Eventually, you're going to run out of people. They're going to run out of the junior engineers. And I think it's possible that those requirements come back down, and you get closer to the boot camps where, wow, we need to actually backfill these people. We need people who know what they're doing. These tools work, but we need some people to run them. The pendulum could swing the other way.

DAVE: And all the people that are not going to be working in programming in 15 years are going to be working at something else because their programming needs are going to be solved. So, they're going to have an opportunity to do other things. We couldn't have been programmers 100 years ago, even if we had the computers, because we were too busy farming because we were starving. When you say they're coming for my job, that is the same sentence as they freed me up to do some really cool stuff later. But if you don't know what that cool stuff is, it's a negative statement. Like, they came for my job.

KYLE: I mean, the automated checkout stands were going to take over, right?

DAVE: Yeah.

MIKE: It depends on the political --

WILL: All I could say is, to my mind, it is absolutely stunning. So, my wife works in healthcare, right? She works in healthcare. She's a therapist. And it is stunning the lack of regulation and the power that just a regular, old software developer, high school diploma, maybe [chuckles], is just rocking in out the street. I'm not trying to say, like, you have to have this, and this, and this to be good at your job, or anything of the sort. But there is nothing, nothing, nothing whatsoever [inaudible 58:44] you know. People are doing this stuff in India, where it's just like, is this guy even accountable to United States' criminal codes?

MIKE: Right. [laughs]

DAVE: Nope. We know the answer to that one. Yeah, that's actually a good point.

WILL: Maybe, maybe not, you know.

DAVE: It's something I was going to say at the top of the call, but it's relevant. The problem that I think AI doesn't have, and is a long way away from, is being trustworthy for a mission-critical, low-resolution task. I do not want an AI in my pacemaker trying to decide or hallucinate when it should send that pulse. I don't want an AI in my CPAP. There is an AI in my CPAP, but it's not deciding valvular. It's looking at the entire night going, we're coming up on the time when he breathes easier. Let's take the pressure down so that we're not doing it for him. It is in there.

But the mission-critical part of this, right, where you've got machines that will dispense insulin and if it's the wrong dose, you die. That kind of mission critical stuff I would not toss that to a sketchy LLM without a whole bunch of agent LLMs going, is this reasonable? And even then, I'd still want somebody doing post-mortems on the failures.

WILL: But even in that point…so, I've worked tangentially in certified- software doing static analysis tools for making sure that your avionics controllers and stuff like that are going to do the job that they need to do. And I think AI can contribute tremendously to the effectiveness of that stuff.

DAVE: Oh, absolutely.

WILL: Like, through better static analysis tools, through more robust and durable test suites, just, like, really, really , really, really testing the ever-living snot out of all of this stuff and letting people do better work. And yeah, okay, they could generate some code, but we have run it through, and we have to test it. And you're having smart people be more effective, be more efficient. And I think that's fine.

But, man, I don't know. I think, David, I think you might have sold me, I don't know. Because as I'm walking through the problems, and I'm saying like, okay, this is what I'm grappling with, right? And I didn't spend a lot of time thinking about, and here's how I would build a tool, or a Transformer, like, tooling around shaping this problem into something that the AI could mill down.

MIKE: Well, that's actually probably a good place to end this, talking about avionics. Let's bring this in for a landing.

DAVE: I like it.

MIKE: We asked some questions, where are things going to go? I think we kind of all have a lot of agreement that they're going to change a lot. We don't know exactly what they're going to change like, but there's some suggestions as to the direction things are going, more effective tools, allowing us to do some things more effectively, and maybe even fundamentally transforming our industry.

And there's absolutely a need to stay on top of it. That's never changed. It's been the case all along in this industry, but perhaps even more so now. And it may go in crazy ways we haven't thought of, you know, it's not going to be subtle changes. It may be pretty dramatic ones. That's not necessarily something to be scared of. It's potentially something to be excited about, but we need to be prepared and watching. Keep your finger in the wind, which way is this going? Or else you might end up in trouble.

Thank you. Hopefully, listeners, you've gotten something from this and have some ideas to chew on as to where to take your career. And until next time on the Acima Development Podcast.

This episode of the Acima Development Podcast delves into the challenges and strategies of working with legacy systems in software development. Mike begins by drawing a parallel between his past experience in construction and software engineering, emphasizing how businesses often opt to demolish and rebuild rather than work with existing structures. However, unlike construction, software developers frequently lack the luxury of starting fresh, as applications must remain operational while being updated. This leads to the episode’s central theme: how to effectively manage and modernize legacy systems without causing disruptions. The discussion also highlights how software often evolves in ways that diverge from its original vision, making maintenance and scalability a significant challenge.

The conversation explores various approaches to dealing with legacy systems, from full-scale rewrites to incremental updates. Matt likens outdated software to old pants—once a great fit but no longer suitable for current needs. The panel discusses past examples, such as the rise and fall of ColdFusion, and the complexities of working with outdated languages like COBOL and Fortran. They emphasize that while complete system rewrites are sometimes necessary, companies often benefit from gradually modernizing through a “strangler fig” approach—wrapping legacy components with new systems until they can be fully replaced. They also stress the importance of considering long-term viability when choosing programming languages and architectures, avoiding trends that might fade over time.

The episode concludes with insights into best practices for managing legacy codebases. The panel discusses strategies such as isolating components, minimizing dependencies, and maintaining clear APIs to improve maintainability. They also highlight the importance of version control, automated testing, and structured modernization plans to avoid the pitfalls of outdated software practices. The conversation touches on AI’s potential role in modernizing code, though they agree that current AI tools aren’t ready for fully automating legacy system overhauls. Ultimately, the team underscores the value of continuous improvement—whether through refactoring, gradual replacements, or thoughtful decision-making—to ensure software remains functional and adaptable over time.

Transcript:

MIKE:  Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting again today. With me, I have Matt, Kyle, and Eddy. We may actually have some other people showing up later. We'll see [laughs]. We've got some delays out there, but, for now, we've got Matt, Kyle, and Eddy. And well, I'm not going to introduce the topic quite yet. Let me lead with some storytelling here, and I think we'll have some good storytelling in today's session.

Years ago, I did construction work, family business. My dad and his brothers they did building, so I did that some growing up, did it in my 20s, did a fair bit of construction work, among other things [laughs], but I did that. One place I worked was an indoor commercial construction, that is, we'd go into office buildings, sometimes high rises, sometimes smaller, you know, I've worked in banks, high rises, and in between [chuckles].

And we would go into spaces that were being leased by a new tenant and build it according to the new client’s specifications. The interesting thing about that is you have these...particularly, like, you have high-rise buildings. It's basically just concrete walls, right, and [chuckles] flat in between other than support beams to hold the whole thing up, and everything else was kind of open to the discretion of the client. And they did things in a lot of different ways.

And what usually happened, and this is interesting, is that it was cheaper for them to just tear the whole thing out rather than work with what was already there. You think, well, maybe, you know, you can keep some of the stuff. That's not what generally happened. The way that, you know, spent a lot of time just completely demolishing [chuckles] stuff that was already there. Actually, the demolishing didn't take that long, but, you know, a lot of time demolishing and then rebuilding from scratch, you know, dropping it down to the concrete.

And that's really interesting because you think it might save you to use the existing building, but that's not what we did. I can't think of a case where we, I mean, that's not entirely true. We did do...some cases we'd leave some stuff, but, you know, drop ceiling torn out, [chuckles] the walls torn out, carpet torn out, wallpaper torn off. A lot of times just tearing things apart before we rebuild it because it's hard; it is hard to work around something that's already there if it's not quite what you want.

You know, in engineering, we a lot of times don't get that luxury. We have to work with what's there because it's in use. It's very different. When the old tenant leaves, you got to strip things out and rebuild it. We don't get that. Like, I can't think of any time I got that luxury, right? You got an application in production. You're trying to make money, and maybe you're desperately trying to make money if you're running a startup. And you can't have downtime. You can't inconvenience your existing customers. You've got to build around what's already there, and what's already there may not fit at all what you want to be doing.

I've heard it described as starting by building a shed and ending up building...you're thinking you're going to build a high rise and ending up building an airport. It's just [laughs] the way that we build because the way business needs change just diverges over time, so we have to work around it. And that's the topic we'd like to talk about today.

We'd like to talk about what it involves working with legacy systems, which is what almost all of us are working in, you know, every system gets older over time. And this is a chance to share stories, and we'll be sharing, doing some of that, but also talking about the principles. What do you need to think about as you're building around things that are already there? Sometimes they fit, sometimes they don't, but you just got to live with that. That's my introduction. And I'm going to stop talking and see what my fellow panelists here think, just off the bat. Any thoughts about this topic? Do you have any stories? Do you have any ideas about what you need to do?

MATT: Well, it's like buying a pair of pants in the ‘90s, right? Here we are, and now 2025, Happy New Year, everyone. Something we bought in the ‘90s, great at the time, in style and fit. Now, here we are in 2024, many years later, gotten older, put on some weight, styles have changed. And those pants, while we still may be able to cover ourselves up with them, they just don't fit, and I think you run into that a lot with software.

Business rules change, direction changes, and teams change, right, code languages come in and out of style, popularity, hard to find support. So, when we're building something, we generally think, oh, what's new and shiny? This would be great to build this in. And anyone who's been in the industry for any lengthy amount of time has probably done this, and that's, oh, there's this hot, new language. Let's take, for instance, ColdFusion, right? I think that's a great example.

MIKE: That’s the one I was thinking of, by the way [laughs], the same one I was thinking of [laughs].

MATT: Yeah, at the time when it came out, a lot of people jumped on that bandwagon and thought, oh, this is going to be awesome, you know, it's Adobe. They're not going anywhere. It's going to be around forever. So, we build applications in it. Then, a few years later, come to find out, nobody knows the language. It's been abandoned, and here we are stuck with a system that's built in ColdFusion. So, what do you do when you need to update it?

MIKE: Well, that's the question, right?

MATT: Right.

MIKE: Assume you're a new tenant now. Throw out the old system, build a new one? If you had a ColdFusion system, what would you do?

MATT: Yeah, I mean, perfect example, we have this case here at Upbound. We have a system that we use to track our capitalizable time, initially written in ColdFusion. And Acima historically has not used that system, and we had a choice, right? Either we adopt it with it not meeting our needs and try and figure out how to adjust it to make our needs, or we build something new. Well, here, we chose to build something new in hopes that it will be able to scale and support our needs for the future, and that's not always the answer.

MIKE: Oh, it's not. There's something to be said that scale matters. And that's why really big systems tend to be built in really old languages. You want to find COBOL? Go look in the airline, right [laughs], or a bank. It's been there for decades, and it just needed to keep running. You're going to find old tech because it was cheaper to keep it running than to build something new, even decades later. Go ahead.

MATT: Yeah. Well, that's a very, very good point. You know, we also didn't really build things in the past in service-based architecture, right? We weren't using microservices. We were building monoliths. And the bigger the application, the more longevity it's had, the harder it is to be able to rewrite that system. So, maybe an answer is, hey, what can we extract out of here? Can we isolate domains, pull things out, and then start to modernize, that word modernization, then we can start to modernize that way, right? Okay, what's a new language that's popular, has had some longevity, so we know that it's going to be around, and how easy are resources to find for it? And then, you start to make decisions that way.

MIKE: You hit right on it. A lot of times, it's not cost-effective to rebuild the whole thing. But there may be a component in that system that, well, a new component, right, that's attached to it that you can build externally. Or there may be a component you want to rewrite wholesale, right? Just rebuild that component. You're not rebuilding the whole thing. You're just extracting that portion of it out to a service, and then you get a new service that wraps the older one. Have you ever seen a strangler fig tree?

MATT: Yes.

MIKE: They're cool, right? They’re interesting. Birds poop the fig seeds into the upper parts of the tree, and it will start growing way up high in the branches of the tree, send down a root down to the ground, and then root itself there to get the nutrients it needs. And that tree will gradually wrap around its host tree until it completely envelops it. And, eventually, the host tree is gone, and you're left only with the strangler fig. And you end up with this big, beautiful tree that is still there, and the original tree has, you know, completely vanished.

You can build software that way, right? You start with just a little piece, make sure it's well grounded, and build it out around the old section until the old section is no longer needed, and it's just gone.

MATT: Yeah, and it's important, too, when looking at these legacy systems, are we sticking with the same language? If so, that makes a big difference, right? It's a lot easier to refactor, make changes than it is to try and work a new language into an existing system, you know, some can be compiled into other languages, you know, if you're running in the JVM, et cetera. But I think that's a big component where you have to start to make these decisions is thinking, okay, what technology is going to be used for it, and what components can we build with this technology?

MIKE: Well, and there's some pressure there. If you're thinking about extracting components, and this is, I think, a healthy thing [chuckles], it provides some pressure to be decoupled because if you're really loosely coupled between systems, then you can completely change the tech stack on one of them, and it doesn't matter, right? They talk to the API. As long as they both talk to the same API, it doesn't matter. Now, this applies at all scales, even within your app, but between them, if you have really loose coupling, it works out great.

There's the famous email that was written by Jeff Bezos at Amazon (Was it 20 years ago?) where he said “Starting today, no applications get to have any internal private APIs to any other. Everything's all public. You can only communicate over these public APIs, and if you violate that rule, you're fired. Thank you [laughs].” And people were kind of panicked by it at first, but it forced teams to have strong APIs that were decoupled from each other, and that's how AWS was born. It matters [laughs].

MATT: It does matter. And it'll help you build things that will have that longevity, right? Just because something's old doesn't mean it's bad. Maybe your business rules haven't changed in 25 years, and you built this initial system in COBOL, but it does exactly what it needs to do. Why would you rebuild it? There's no point in redoing things just for the sake of redoing things.

MIKE: Right. As long as you are still able to get sufficient resources at a reasonable price to maintain the system. And if there's some core business rules that seldom, if ever, change, and, you know, COBOL [chuckles]...I laughed a little about COBOL. It was such a huge technology. There's still a pool of developers out there. They're kind of pricey, but they're available, right? You could get one if you needed to.

MATT: Yeah. One of my very first jobs in the industry was converting COBOL and Fortran over to PERL and PHP. Now, in hindsight, was that a great idea, eh, you know, probably not. Nobody wants to write in PHP or Perl anymore either. But it's interesting that it didn't really need to change. All we were doing was duplicating the exact same logic in a different language, and it really didn't make sense. You know, maybe they were trying to be a little bit forward-thinking in that those languages were going to be dead languages, but they could have lasted for another 10, 15 years and been just fine.

MIKE: Honestly [laughs], Fortran's still got a pretty active community. That's nowhere near a dead language. I haven't looked at language rankings recently, but Fortran might be ahead of Perl [laughs].

MATT: Yeah, you know, and for a while in the early 2000s, Perl was it. Everyone loved Perl, right? CGI scripts and its file handling was exceptional. You could get a lot done with very little code. It still, to this day, wins Code Golf almost every time.

MIKE: Well, and Perl is actively developed. It's not a dead language.

MATT: No, no, not at all. It has its place, and web applications just aren't it.

MIKE: But there's still a lot of shell scripts out there in Perl.

KYLE: I think there's something to be said, too, about some of these older languages. The programs that are written in those they were written in a different time with different constraints. So, we're talking about some of these COBOL or Fortran programs. Sometimes you don't want to get rid of them because they perform. We're not hindered by the same constraints that we were a few years ago, so we don't take into account that we only have half a gig of RAM, right?

And it may not be worth to switch it over to, you know, Ruby, Node, or whatever you're working with that will use two gigs of RAM because of how different the performance metric is. Maybe you need to write an adapter or another layer around that to take advantage of your old applications.

MATT: Yeah, I think, as engineers, we've gotten a little bit complacent in thinking about those types of things, like memory usage, disk space, because it's so inexpensive these days, right?

KYLE: Yeah, you can just throw hardware at it. That's a common solution these days, isn't it?

MATT: Yeah, absolutely. Just spin up another container.

MIKE: It's not necessarily the wrong one, either [laughs].

MATT: No.

MIKE: Hardware costs, while pricey, depending on the hardware, depending, you know, this is conditional, right? But, you know, they can often be less, even if they're somewhat pricey...developers are really pricey. And if you can write something in a language that is easy to think about and easy to communicate about and you're not going to write bad code, it may save you money, even if it uses a lot more resources.

MATT: Absolutely. And, you know, just in this short conversation we've had already, we’ve already listed, what, 10, 12 things that you really have to take into consideration when doing this.

MIKE: We've talked about this idea of wrapping a legacy application, gradually splitting out components, building them in. I'm going to use...you laughed a little bit at modernize [chuckles] because it's a loaded word. What's considered modern is not necessarily a clear concept. But what meets the current business needs, building in some framework that meets the current business needs that, you know, wraps the older one, gradually replacing components of it until, if necessary, the original is entirely replaced.

Sometimes though, we work within the system, right? You're not going to split out a component, but you're working inside legacy code. And maybe it's been there for a decade or more, and many hands have been in there changing different pieces. There's lots of different design styles, lots of different architectural ideas, all kind of crammed in together, like one of those old farmhouses you see that's had additions put on three or four times, and they all look completely different from the other ones [laughs]. It looks like that inside. What do you do when you find yourself in a system like that? How do you stay oriented and stay productive if you have to do all that context-switching between, okay, what's going on here?

MATT: I think that's where you really have to keep design patterns in mind, right, and see what you can isolate. Can you eliminate dependencies, if possible, and really confine your new code so it isn't dependent on other things? I think that's a really good way to start making your application better and more understandable and just make it easier to work in in the end.

MIKE: You said, if I heard you right, make it so it isn't dependent. Can you elaborate on that a little bit?

MATT: Yeah, when we're a little less experienced—and I'm not saying that every legacy system is written by inexperienced programmers, because that's far from the truth—but if, let's say, I don't know Ruby, right, and I just started writing in this language. Likely, if I'm new to this industry, I'm going to couple things together that probably shouldn't be. You know, I'm going to pass dependencies through all my methods, and Class A may rely on something from Class B, et cetera, right?

And I think that's where you need to think about, okay, how can I isolate this? Is this really dependent on this other thing and could I make a superclass? If they require resources to be shared, let's share them through a superclass or whatever, you know, just eliminate dependencies and try and decouple things as much as possible so that code is usable outside of its current form.

MIKE: Got it. So, you're talking about, you know, detangling the spaghetti, decoupling so you don't have mutual dependencies and, you know, single-purpose code [chuckles] that is reusable.

MATT: Exactly. You're much more eloquent than I at stating that, but yes.

MIKE: [laughs] You said it first, just rephrasing [laughs]. Yeah, absolutely. Improving your little corner of the world, right? And you said, specifically around best design principles, getting each component to be serving a single purpose and to have a clear boundary between itself and the rest of the code so that it's reusable and not, you know, like you say, mutually dependent on other code.

MATT: Yeah, and you also said cleaning up your little corner, right? If I have a really messy bedroom, you have to start somewhere, so let's clean the closet. If I have a nice, clean, organized closet, that's a really good start because it makes room. And then, as you start to get that organized, it makes it much easier to organize other things because you've established a pattern on how you're going to do it and what your approach is going to be.

MIKE: That's great, and it reminds me of something my wife does. She borrowed this idea from elsewhere, but it's a brilliant idea. Every week, you focus on just one area, so you don't have to do everything all at once. So, like, this week, we're going to work on making the kitchen clean because there's always the things, right, the things that you put off, maybe the deep cleaning of the stove, or [laughs] cleaning out the junk drawer, right? And if you try to think about all the things that got to get done, you're never going to get them all done. You just throw your hands up. And the tech debt piles up, and [chuckles] you never get to it at all.

But if you allocate some time on a regular basis and a rotating basis to each of the areas, right? And in the home, maybe this week we're all going to work in the kitchen. Next week, we're going to work in the living room, next week bedrooms, whatever the case may be. Like, in your code, maybe we're going to do some front-end cleanup this week or this sprint, right? Next sprint, we're going to do some cleanup in area X, or maybe it's a group of sprints, maybe a quarter, whatever it takes to get some real meaningful work done. That's not everything you're going to be doing, but it's a portion of what you're doing so that you're continuously paying down the tech debt and improving the code.

MATT: Yeah, what's that show? Marie Kondo, I think, her name is, tidying up and about organization. There are so many things that are applicable from life into software engineering and software design and vice versa.

MIKE: Right. Absolutely. Exactly. There is hugely relevant information there, right? That's why, usually, at the beginning of our podcast, I talk about something outside of engineering because getting ourselves out of the mindset of just what's right in front of us, I think, helps us to see those connections.

MATT: Yeah, for sure. Eddy and Ramses, you guys have been in the industry for a while now, but maybe a little less than us old folks. What do you think on this topic? Have you had the opportunity to really get into some legacy code and take a look and say, “Oh, man, what am I looking at, and why is this such a mess?”

RAMSES: Yeah, every day [laughter]. Well, quite a bit. I’ve worked in a few different projects, I guess, now, and some are more than 6 years old, 6 to 10, maybe even 12 now. But, yeah, when you get in those kind of deep pockets, it's, yeah, it's just, you know, no one's touched it for 10 years, and people are scared of touching [laughs] it. It’s brutal. Yeah, that's always just interesting. You just have to kind of, you know, take a step back, figure out what's going on. And, you know, depending on the severity or the complexity of your change, just try and maybe not poke it too much.

MATT: And I think, too, when companies start out, you know, startups, people have a tendency to get a little more cowboy. And when they build software, they're looking to be first to market, right? Let's just make it work, get an MVP out there so we can get to market, and then we'll come back and fix it later. And, you know, often that doesn't happen.

Also, you know, you have a lot of freedom in that kind of a world where you may have seen a podcast or listened to a podcast or seen a video somewhere, video tutorial on some new pattern that you've never tried. And you think, oh, I'm going to try it inside of our application, even though we don't do it anywhere else. And those are the kind of things, I think, when we come across, we just kind of scratch our head and say, why is this here, and why did we do that? That and if I look back at something I've written two, even three, years ago, if I don't look at it and think, what was I thinking? I'm lying to myself, right?

RAMSES: Oh yeah.

MATT: Because we grow every day, and we're always getting better. We're learning from our peers, you know, other ways of doing things. And it's just, there's always room to make things better.

MIKE: How do you navigate a large system that's, you know, got a variety of different styles and approaches? I've got some thoughts on this, you know, on how you approach that. You're in a codebase that you don't fully know, right, and in a large system, nobody does [laughs]. Nobody knows the whole thing. So, what do you do to quickly figure out where you're located? You know, it's like one of those shows where they drop you off without GPS and no obvious landmarks, and you have to find your way out [laughs]. How do you do that in the code?

MATT: I like to start from the outside in, if I can, you know, you find your entry point then it makes it really easy to navigate your way to where you need to be working, or at least find out if you're in the right place, right? If we're in the Ruby world, lots of Prys. I like to throw a Pry in once I get to a comfortable place, you know, maybe it's inside of a service object a couple of layers in, throw a Pry in, and then look at the methods inside of that service and run them inside of my Pry and console just to see if I'm getting the expected results. That's one of the ways that helps me navigate my way through new code. Run the tests, I mean, look at the tests. First thing, right?

MIKE: You know, you said find an entry point. I was thinking something similar. When you have something you need to work on and no context, you've got nothing, right? It's a blank wall. And finding that entry point is everything. So, if you've got an error, see if they've got a stack trace, right, because that can take you to someplace. Now you've found a place in the code. You've got that entry point where you can put your debugging statement, or your print statement, whatever kind of debugger you are, and start tracing things down.

If you have some UI, then you can go to that page and try submitting some things, right, put in some logging on the server side and see what's getting hit. If you're doing heavy front end, you can kind of follow a similar approach. Start with something that you do know because you've got to know something, right? You've got to know something. If you really know nothing, then you probably either don't have a sufficiently written request or you need to go talk to somebody [chuckles] to get them to give you some context. There's always someplace to start. Getting that starting point is more important than almost anything because once you can pry in, right, then you can start digging down and going further.

I find that new developers often get...they get paralyzed. They get paralyzed by all of the unknowns, and that's totally crippling. Instead, if you don't worry about all the unknowns and start by finding something that you do know, then you can start with that seed- and expand that knowledge outward until you know enough to solve the problem or build out the feature that you're looking at. That having an entry point, as you called it, is most of the effort in many cases, in my experience.

MATT: Yeah, and as long as you have...and maybe entry point isn't the right word, reference point, right?

MIKE: Reference point.

MATT: As long as you have a reference point, you can navigate in all directions out from there. If I know I should be working in some, let's say, user class, right? I can also look and see, what is calling that user class? What's instantiating it? What's using its methods? And work your way out from there, and then slowly, but surely, your circle gets bigger. And you start to learn your way around, and you're able to navigate things.

MIKE: You know, thinking about the geography reference I made before, you know, you’re dropped off. You don't know where you are. If you can find a landmark that you recognize, that changes your world [chuckles]. That changes so many things because now you have, again, you have that center that you can navigate out from and start growing that circle. Otherwise, the Cheshire Cat, if you don't know where you're going, it doesn't really matter which way you go [chuckles]. But if you know where you are and where you're going, it changes everything.

So, storytime, maybe for us older folks here especially, what are some times that, you know, some crazy stories that you’ve found in legacy systems? You know what? I'll lead out here. I'm going to start with a story. I'm thinking back to...I think it was my first full-time dev job, although it wasn't right away. I'd been there for a while. We were given a customer management system, and somebody in the company decided it'd be a great idea to buy this. And the guy who sold it to us was just, like, some person out of his garage.

And the whole thing was built inside a SQL Server as stored procedures. And when I say the whole thing, literally, the whole thing. There was, I think, a little bit of UI written in Microsoft Access that gave it a little bit of forms on the front. But then all of the business logic was written in stored procedures, which were stored inside the database that were not under version control.

There was no source control at all, were not searchable [laughs], and had no, you know, there was no rollback. You know, if you make a mistake, it's permanent, and it's live. There were also some calls out to compiled executables for which we didn't have the source code. [inaudible 31:31] here you go, make this work, and we had to make a number of changes to make it work for us [laughs]. It's hard to describe what it was like doing that [chuckles], you know, but --

MATT: That’s the thing nightmares are made of.

MIKE: [laughs] So, say, we make a feature request. Well, I'd start with the form in the UI and then figure out which stored procedure it was calling to, right? And then, in that stored procedure, it would usually call to four or five others, which themselves would call others. They were nested, so, you know, they were nested all over the place. And then, just step through those again.

It's not searchable, so you have to find its reference in the UI [chuckles] for a SQL Server and go...and this was a long time ago. It may be better now. But go find that other stored procedure, find what it linked to, and you could trace your way down eventually. Again, no debugging statements. That was not possible. You just had to try to understand it. And by looking long enough, you started to understand the geography [chuckles] of it and know what was needed.

MATT: Yeah, that, you know, we think today, what did we ever do without tests in our software? And then, you go back further, and no version control, no tests, multiple people working in the same code. And it was like, let's shove this file up via FTP, and then someone else is working in the same file and overrides your changes, and it breaks everything. Oh, man.

MIKE: [laughs]

MATT: Those were the days where you’d get really good at tailing logs.

MIKE: Yes. Or the system I'm describing where there is no test system. Any changes you make are live in production the moment you make them. You get really good at understanding what you're doing before you make a change. So, there's my story.

MATT: I don't know how we navigated that world.

MIKE: [laughs]

MATT: But somehow we survived it. And you're also in charge of keeping the servers up and the network and everything else at the same time.

MIKE: All at the same time [laughs]. Oh, I can tell stories of mistakes I've made.

MATT: I've brought down a number of production systems throughout the years [laughs].

MIKE: So, I gave my legacy system story. What's your legacy system story?

KYLE: What I'm thinking about right now is back when I was an intern for a company that I won't throw them under the bus. But I was an intern and then a full-time employee at one point. And the issue that we had there was we had multiple versions of their software to serve different customers, all written in different versions of Java. And so, what it was is you had different looks and UIs and different functionality bleeding through these different versions of the applications but also had the same functionality.

And my task at that company is I was a manual QA engineer, so I was tasked to test these. And so, we would be provided with specifications, but we had to know how this was implemented in every version of the application for each of these customers. Just how bad that was, like, just the test matrix on those types of systems was just insane just because I was writing bugs.

And half the time, I was right in the sense that, oh yes, it should work this way in this application, but not in this version of the application. And there was times where I misunderstood the requirements, and, no, it shouldn't work this way for this customer, but it should for this other customer. Choices like that where we're creating different versions and getting customers stuck on different versions like that that's one that really stands out to me.

MIKE: There used to be a popular way of doing things some years back, where every time you got a new client, you'd spin up a new version of the app, like, their own complete, hosted copy of the code, rather than having a shared web app. And oh, what a nightmare [laughs]. That is so bad.

KYLE: [inaudible 35:52]

MATT: Yeah. I worked in the lead gen industry for a long time, you know, spammers. And I worked for a place that did that with education. It was all of the for-profit lead gen education. And every time we got a new advertiser, we had to spin up a new instance of the application. But they would share, like, the cost per click, cost per acquisition, all of that.

So, if it changed when spinning up a new advertiser, we had to go in and change 50, 60 different independent applications for every partner every time we made a change, and no version control. It was all FTP. And this would have been, I don't know, I think, maybe 2001 somewhere about. It was just a nightmare. It was an utter nightmare to try and maintain that and keep it up to date.

The application itself was built in a hacked version of Mojave framework, so maybe it was a little later, 2003, 2004. Someone had torn apart Mojave framework and customized it to do this lead gen stuff. And it was just...it was miserable. Just the folder navigation alone was a nightmare.

MIKE: So, I’ve learned a couple of things from these stories. Lack of version control is...are there any systems out there without version control? I hope not. But if there are, if you're thinking about doing it, don't [laughs]. If you're starting in a new project, it's worth it from day one. Get it up in Git. It's worth it. There is no question. There shouldn't even be a hint of a question. If that's a question in your mind, you're doing it wrong [laughs].

MATT: First thing you do, git init in your local directory. It doesn't have to be hosted. You can do it locally if it's only a personal project but do it.

MIKE: Yes. Second lesson is, don't copy your code. Have a shared version. If you need to have customizations, build those in as some sort of feature flagging that you can turn on and off, and you have a configurable system. It's easy. Now you've got a configuration system that users can use to toggle, and it's great, and everybody is happy. Don't build a new system for every client. That is the path to madness.

MATT: And test your code.

MIKE: And test your code. Test your code. You know, things that are best practices sometimes you don't think about because they've just become so ingrained. Have you ever been in a situation where you don't have those? And, you know, a lot of students doing their coursework haven't done unit tests, right, maybe haven't done version control. Maybe just make another copy of their code when they need to make some changes. I mean, there are some things here that are easy to do when you're just getting started because it seems like it makes sense at the time.

There's a reason that these [chuckles] have become standards in the industry because they will save you. It's like autosave in a document. Back in the days when you could write up 30 pages and then the power goes out, and you lose it all, knowing that that would happen, autosave is the thing you do. You just don't write a document unless you have something persistently saving in the background, right? I mean, this is so universal now. It applies from command line editors up to everything, right? But there was a time when that didn't exist. There are some things that you just do from the beginning, and you just don't consider them negotiable because it matters that much.

MATT: Yeah, and it's a little off-topic, but it's just making me think about some of these horror stories. Looking back on the very first time I ever saw Subversion, [chuckles] and, you know, what was it before that, CVS?

MIKE: Yeah, concurrent version, what is the S for?

MATT: Which just stored the entire file, not just changes, and was a huge memory hog, but it worked. And then, one day, I was working for a place, and our manager came over and said, “Hey, as of today, we're going to start using Subversion.” And that's all we were told, no training or anything. Needless to say, about the first 20, 30 merges that happened always broke everything [laughter] because no one knew how to do merges, no one knew how to resolve conflicts, and it was just...it was chaos.

MIKE: Well, there's also something you said there about gradual transitions, meaningful, planned transitions, and not hard cutoffs, training.

MATT: Yeah, and training.

MIKE: Subversion was a great step forward. I loved Subversion back in the day.

MATT: Oh yeah.

MIKE: And then, Git was a great step forward beyond that. There's a reason why the industry...there are a few others out there, Mercurial, that some people use. But Git has just dominated. I can't remember the last time I saw a repo that wasn't in Git.

MATT: And now it's just second nature, right?

MIKE: Yeah.

RAMSES: Yeah, Git is awesome. It's great.

MATT: I wonder if this new AI boom will change the world of repository management.

MIKE: That's interesting. Totally off-topic [laughs], Nvidia recently came out with their new graphics cards, and they get, like, somewhere near 250 frames per minute, or something. And they do it, you talk about the AI, by predicting what the next few frames are going to be via AI. It takes the frames that you've seen in the past, predicts what's going to happen next, and pre-renders those frames, pre-renders those frames, and then just makes any tweaks as necessary, which seems wildly inefficient and crazy, but it makes a tremendous difference. It pushes things forward so much further.

To your point about...I bring this up because you think, well, with version control, what's AI going to do? Well, sometimes, if you have good enough AI, you can do surprising things that you'd never think was possible before.

MATT: And, you know, it's relevant to these legacy systems, right? A lot of these legacy systems have tasks that could be accomplished integrating some of these LLMs and, you know, even machine learning.

MIKE: Now, as of today, in early 2025, I would argue pretty strongly that you can't just take existing AI tools, point an old repo, and say, “Hey, convert this into something new.” It's not going to happen.

MATT: I second that. I've tried.

MIKE: [laughs]

MATT: Just, you know, like, fiddling around with it, seeing how intuitive it really is. I still would not trust it to write my code, though I do trust it to write base tests. I've been doing a lot of Python work lately, and Copilot is great for writing unit tests. You do have to validate them and make sure that it's testing exactly what you want, but it sure saves you a lot of time.

MIKE: Well, that kind of goes to where I see those tools being right now. Think about having an extremely fast but somewhat inexperienced and clumsy junior engineer, and they’re there at your disposal and can do a lot of things, and they don't mind any task you could throw at them. Big tasks, they're probably going to get wrong, and so you shouldn't. But something like writing unit tests, if you can knock those out, well, that's fantastic. That can save you a ton of time.

KYLE: I’m finding it very good for the idea of a rubber ducky. That's what I've been using AI for is it's a very intelligent rubber ducky.

MATT: Oh yeah.

MIKE: Having that rubber duck to talk to is super useful [laughs]. It's super useful. I think that we're kind of reaching a good stopping point here. We've talked about the fact that we're all working with legacy systems, almost all of us, right? Most people don't have the luxury of always working on something new. And the thing you're working on new is not going to be new forever. Eventually, it becomes a legacy system. And there's a variety of techniques you can take to deal with that. Wholesale rewrite, if it's small, might make a lot of sense. Usually, that's not something you have the luxury of doing, but you can split out components and rewrite them in a manner that is better aligned with your current business needs.

And if you have a system that you're just continuing to work in, you can find reference points within the code to navigate your way. And then, as you make changes, you can gradually improve. Pay down the tech debt. Take the time to pay down that tech debt so that your system stays maintainable or becomes more maintainable. Maybe it's not very maintainable, but it becomes more maintainable over time.

And there's some things we can learn from bad systems, too. Follow best practices. Follow best practices. They're there for a reason. Write unit tests. Use version control. Do the things that play well with others because one of those other people is future you, too [chuckles]. It's all worth doing.

Until next time, on the Acima Development Podcast.

In this episode of the Acima Development Podcast, Mike and Ramses dive into the importance of recognizing and respecting mental and physical limits to maintain productivity and avoid burnout. Drawing a parallel to endurance sports like marathon running, Mike explains how the depletion of glycogen reserves mirrors the mental fatigue that can occur during intense problem-solving in software engineering. He shares a personal anecdote about his experience with low energy and poor decision-making while working in landscaping, emphasizing the value of taking breaks to restore mental clarity.

The conversation explores practical strategies for managing mental capacity in high-cognitive tasks, such as stepping away from work, engaging in light physical activity, or shifting focus to a simpler task. Ramses shares his approach of taking regular breaks and prioritizing tasks based on energy levels, while Mike highlights techniques like the Pomodoro method and mindfulness practices to improve focus and productivity. Both agree that recognizing when to pause, reflect, and reframe assumptions is essential to navigating complex problems effectively.

Mike concludes by underscoring the importance of regular mental “maintenance” to avoid burnout, likening it to routine upkeep for a car. By sharing personal stories and actionable tips, Mike and Ramses provide listeners with valuable insights into balancing productivity with rest, fostering a healthier and more sustainable approach to work. This episode serves as a reminder that taking breaks isn’t a sign of weakness but a key to long-term success and creativity.

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today. With me, I have Ramses. It's just the two of us today, which is kind of nice. We'll have a conversational podcast today.

And we're going to talk about something we've actually been talking about for several weeks. We're going to be talking about how to clear your head. I'm going to share something outside of coding, as I like to do, and then bring it back to the work we do in software engineering.

There's a challenge that people do endurance athletics run into. I'm not a marathon runner, but I've heard about it most from marathon running. People talk about hitting the wall. And in a variety of sports, they talk about bonking [laughs], you know, you hit the wall and bounce off, bonk.

And I've looked into this some. There's a...in your liver, but there's also in other tissues, but particularly in your liver, your body stores glucose, you know, the simple sugar that you use for fuel for your body. And it doesn't store it directly as glucose. It makes chains like branch chains like snowflake sort of chain-on-chain structure that's easy to break apart, but also easy to store called glycogen.

And your body stores some of it so that if you have to get up and go running, you can do that, and it's great. And you can sustain exercise for a couple of hours. I was researching this earlier today to make sure I was telling the truth. I found estimates of between 80 and 100 minutes of about how long the body’s glycogen reserves stay active until you deplete them with intense exercise. You hit that point, and suddenly, your reserves, your ability to quickly take your body's energy and put it into action, basically drops off a cliff. It just stops.

It's not like you can't move necessarily because your body does have other ways. You can start breaking down fat reserves, and so on. But the easy energy is gone, right [laughs]? You hit a wall. And I’ve read about this with marathons. Well, elite marathon runners, we'll say, can finish a marathon in a little over two hours, which is crazy fast [laughs].

RAMSES: Yeah, wow.

MIKE: They're really good. But notice how that's about 120 minutes or, let’s say, 150 minutes. And the glycogen reserves, remember, last for about 100. You can see why, a lot of times, people running marathons hit a two-hour point, and they hit the wall because they haven't replenished those energy reserves, right? And so, if you just try to go...and it's tempting. You want to just go and finish, right? So, you're maybe five miles from the end. If you're an elite runner, you're probably most of the way through the race when you're hitting that line, and you think, well, I can just push through. Well, no, you can't [laughs].

RAMSES: It's deadly, yeah.

MIKE: Yeah, it just does not work. Now, like I said, I'm not a marathon runner. I do quite a bit of cycling, and I'm definitely familiar with hitting that wall. And I've hit it in other contexts, too. I remember I was working in landscaping many years ago [chuckles] and also intense physical work, right? You can deplete that blood sugar.

And I remember working a job, and my boss was out doing something else. And he'd left me and my coworker to finish up. And we had a couple of other...we were going between a few different jobs. And the one we were at was running long, but I didn't want to let my boss down. So, I said, “Well, then let's just go finish up the other job.” This would have been hours since me and my co-worker had eaten. And my thought was, I just need to get this done.

And we went there, and I was just so single-minded. I've got to get this done. Got to get this done. I wasn't really thinking about how foggy my head was getting. And [chuckles] in my rush to get things done, I was digging, and I was...I think I was just digging a hole, but it was hard. It was, like, clay-hard ground. So, I was just swinging a mattock, and the other side of the pick it's a flat bar kind of thing, digging that out, trying to break through the clay. And I hit an irrigation line, water went everywhere, and [chuckles] we didn't have the parts to fix it. And we had to go back and solve that problem.

So, I’d made the problem worse by trying to rush things through. It made the problem worse. And my coworker, she says, “If you don't get your blood sugar up, keep your blood sugar up consistently, I'm not going to work with you anymore,” and [chuckles] she was right. And we were friends, you know, this was not mean-spirited. This was just, no, that was not the right thing to do, and she was right.

I should have taken some time to get my head right. And this was pre-cell phone days, right? So, hard to call my boss, but, you know, I could have found a pay phone or something, found some way to pull this in, or maybe just gone back to the headquarters, called the customer, let them know we were going to be late because we were going to be late anyway, and finish the day. There's a number of things I should have done.

But, luckily, it wasn’t a terrible disaster. We went back the next day with some PVC and fixed the problem, and everything was okay. But I should have known better. And you get that low blood sugar, and you just can't think right. You do not think clearly. None of which has anything directly to do with software engineering, right? But when you're building software, you think. You have to think. And we're probably not doing vigorous exercise while you're doing software engineering. They’re not necessarily compatible, right [laughs]?

But your brain uses quite a bit of energy, and further, you can only sustain any activity for so long, that mental activity included. Sometimes we think, I'm just going to push through this. I just need to keep going. And you can tell your brain is slowing down, for whatever reason. There's all sorts of reasons that you can't think clearly anymore. You've got distractions. You've been focusing on it too long, and your brain just is done.

And you think, well, I'm just going to push through. I'm going to push through. And the next thing you know, you've spent way too long accomplishing nothing [laughs]. You haven't made the situation any better. You've made it worse. Or you go back, and you've written bad code. You've made a bad decision. It is critical to be attentive to those signals in your brain and know, you know, I need to do something different. Have you had similar experiences, Ramses?

RAMSES: Yeah, all the time. You're saying it's about, like, 100 minutes, 120 minutes, that's about my cap [laughter], maybe two hours on a really good stretch. But, at that point, it's, you know, you usually feel it in your back a little bit, and then usually, just have to get up. [inaudible 07:17] so I can't sit down for too long without moving a little bit. But yeah, it's really interesting. Sometimes you just...you're really close to solving the problem, but maybe you're down a dangerous hole. Yeah, it's really easy to fall into the, I'm really there, and, two hours later, you're still working on it.

MIKE: Yeah, exactly [laughs]. Exactly. And, like you said, you're just trying to push through, trying to get it done. It's hard. It's hard to catch yourself. It takes practice to recognize. There's the people who use the Pomodoro technique, named by a guy who had a kitchen timer that’s shaped like a tomato. He'd set it for an hour or a half hour, whatever the length is, turn it on. And after that timer finished, time to get up and take a break.

And forcing yourself to take a break like that sounds like you'd be disruptive, right? Like, wow, I'm never going to be able to get flow time. Like you said, that hundred minutes, you start going long enough, you're going to hit a wall, and you might not notice. You might not notice.

RAMSES: Yeah, it's really easy to just sit down for, you know, 3, 4 hours and crank away. But I think you have to, like, mentally put yourself in a good spot where you can get to a good stopping point, and even if it's not a good stopping point, you just have to get up and take a break.

MIKE: Exactly. Well, and I want to emphasize this point. To all of our listeners, we're giving you permission to take a break [laughs]. You're going to think, oh no, I need to push through it, but that's not how it works. It's okay to take a break. And we're going to talk some about what we do. What do you do, Ramses, when you need to step away from it?

RAMSES: I usually just get up and either go outside, or at least walk around for a while, move my limbs, stretch out a little bit, move my legs, just anything else, then think on the problem, usually.

MIKE: Yeah, so several things. Moving, right? Getting up off of your chair, getting your blood flowing. I don't immediately have it in front of me, but I've seen research that suggests that your brain works significantly better when you're getting some exercise.

In fact, I just saw a headline within the last week that said something along the lines that the cognitive benefits for being a weekend warrior, that is, going and doing intense physical activity on the weekend, whatever that activity might be, are lasting durable. So, even getting up and moving on the weekend can help you throughout the week. It just makes a huge difference to move, to use your body somehow.

RAMSES: Yeah, that's a great one.

MIKE: And you didn't talk about going for a run. You just said, walking around.

RAMSES: Yeah, it's usually [inaudible 10:11]. I mean, outside of the office, it's sometimes a little bit more strenuous, like a bike ride, or a jog, or something. But during the day, usually, it's just casual or light walking, a bit of socialization if I'm in the office.

MIKE: Right. And I think that's important. You don't have to go out and run a 5k, oh, you could [laughs], if that's your thing. If you're an elite runner, you could probably do that in under 20 minutes. But most typically, you're probably just going to go for a walk, go to the bathroom, say hi to somebody, and that's it, right? Just getting away from it for a few minutes is enough.

Do you find that you can read about something else, you know, go and read a news article or something you've been meaning to look at, does that work for you, or does it help for you to just nothing?

RAMSES: It sort of depends. Maybe it depends on how complex the task is that I'm working on. I sometimes find even shifting to a slightly easier or a smaller problem helps to just kind of push the big stuff away. And then, you come back to it, you know, later when I feel better.

MIKE: Yeah, I do the same thing. There'll be some trickier problem I know I need to deal with. And I will, not even fully consciously, but I will, if I'm not in the right space, I'll set that one aside for a bit and work on other things and then get back to it when I know I've got some uninterrupted time and I'm in a better headspace.

RAMSES: Yeah, it's like a sort of procrastination, but I think it's rightly justified, usually.

MIKE: Exactly. You feel bad like, oh, I'm putting this off. And I had the same thought this morning. I had a task I needed to take care of. This one was generating some data just for routine information for the auditing department. So, it's nothing bad, just grabbing some routine information. And I had to pull some data that I wasn't quite sure where it was at.

I put it off because I knew that, in that moment, it would take me longer than was justified. I worked on some other things, got through those, and then came back to it when I knew I'd have some dedicated time and spent some dedicated time for a while. And I was getting close; I was getting close to getting that. I find that doing something different is also, for me, very helpful. That way, I'm still being productive but getting into a different place in my brain so I can come back with fresher eyes.

I try to teach my kids how to deal with it when they run into mental hurdles. There's three tactics I teach them, and I'm sure there are others, but they're the ones that we found are most effective, so they're what I suggest. And they are: deep breathing, so kind of mindfulness, meditation, you know, just focus on your breathing for a bit. Get some exercise, or do something different. And we've already talked about two of those, right? Getting up, walking around a bit. It doesn't have to even be a major exercise, but getting some exercise makes a big difference.

Doing something different helps you to get your brain working on something else so you can come back with fresh eyes. So, helping them with their homework, if it's not working, we'll do one of those things. And sometimes doing something different is the right answer, and there's not shame in that. In fact, you're doing the right thing. You said, Ramses, it feels like procrastination. So, do you want to elaborate that a little bit, you know, like, where you feel like, oh, I'm putting it off and being lazy?

RAMSES: Yeah, sometimes I feel...like, sometimes I might have a hard problem I'm working on, or maybe a lengthier PR that I'm reviewing, some sort of task that takes more of my time than I'm willing to put in currently [laughs]. Sometimes I just maybe sleep on it or, especially if it's towards the end of the day, you get something, and you know it's going to take more than an hour or about an hour, and you're already kind of out of capacity, mentally at least, I find that I usually just, if I put it off until the next morning, then that's my number 1 priority. I've got fresh eyes in the morning, usually. If I treat it as my highest priority, then I know I have to get it done. I do that first.

MIKE: So, that's interesting. So, you just talked about kind of the flip side of stepping away, which is reserving your best hours for the tasks that most need them.

RAMSES: Yeah, and that's kind of hard. At least personally, I find my best hours of alertness is usually in the morning, which usually kind of coincides with meetings. But sometimes after you have lunch, I mean, afternoons, sometimes it’s pretty good. But the last hour and a half before the end of the day is usually not very good.

MIKE: So, you have 2 productivity peaks: 1 in the morning, 1 in the afternoon.

RAMSES: Pretty much.

MIKE: [laughs]

RAMSES: It depends on the task, though. Not all tasks are completely equal.

MIKE: We are not built the same as our computers, right? The computers are built to treat all tasks identical, just be a single-purpose machine that can just keep on executing instructions.

RAMSES: [inaudible 15:49] task.

MIKE: Yeah, but humans don't work that way, right?

RAMSES: No, we're very bad at multitasking, usually at least, especially on very complex problems. It's hard to keep the context in your brain on more than three or four different projects. It depends on the complexity of the project, though.

MIKE: Well, having to switch between projects, the cognitive load there and the cognitive shift always takes something. You can't just drop into something new and expect to be immediately 100% productive, even within your day, right? Even with something you worked on yesterday, it takes some time to get back into the flow state where you're deeply focused. So, there is a cost. There's a cost to switching between things. However [chuckles], there's also a cost of staying on the same thing for many hours at a time.

RAMSES: Yeah, they're probably about equal. I find if I'm on something for too long, it's usually a good sign to move on to something else.

MIKE: Yeah, usually if I get into a deep flow state, I can go for a while, and then I'll recognize I'm starting to hit a hurdle, look up, you know, it's been a couple of hours. And then I need to do something else for a while: change gears, work on something different, well, do one of the things that we've been talking about, right? We've talked about getting up, taking a break, getting some exercise. We've talked about switching gears to working on something different for a while. We haven't talked very much about deep breathing. And it's not just deep breathing although that's the most popular technique for mindfulness.

We all have to breathe, right [chuckles]? So, it's something universal to being alive as humans that we will breathe. So, it's an easy thing to pay attention to. I think it's not the breathing itself that's important but the fact that it's there, and you can pay attention to it. Because if you're paying attention to it, it's focusing on your environment rather than focusing on whatever is worrying you in the moment.

RAMSES: Right, right. It’s kind of shifting your mind away from the problem. Makes sense.

MIKE: Yeah, exactly. You can practice, right? You can practice doing that. You can practice changing that focus. And I'm not an expert practitioner of mindfulness [chuckles], but I can say some that sometimes people worry that, well, I need to focus on the breathing. And the more you try to focus on that, the more the other things come into your mind, right [chuckles]? Like, forcing yourself to try to focus on something is, like, the best way to get your mind thinking about something else [laughs]. Like, I'm not going to think about how hungry I am. Well, you're going to think about how hungry you are. That's exactly what you're going to think about [laughs].

RAMSES: That's all you're going to think about, yeah.

MIKE: Exactly. So, the trick to that mindfulness is not to necessarily be acutely aware of your breathing but rather to switch from your focus state on your problem to just being aware of your environment. Because you can practice doing that, not trying hard to focus but rather letting the stream of whatever's happening pass through your mind and just kind of being a vessel, letting it pass through.

It's actually quite similar to the flow state that you get into when you're working, when you're feeling like it's just happening, right? You don't even have to try because you're so deeply engaged. Kind of flipping to the other side of that coin and letting everything flow by you where you're not engaged is similar and restful, deeply restful. It allows you to flip that around and get back to work. And even the walking around the office is related to that in that you may not have an aim, right? You're just walking around, and [chuckles] you're making a conscious effort to not be deliberate about what happens.

RAMSES: Right. Right.

MIKE: Again, listeners, we're giving you permission to not [laughs] worry about your work for a minute because that is how we work. By disengaging from it for a period of time, you can engage better after you've had that break. And getting good at disengaging is actually a very valuable skill that will prevent that burnout that you get when you hit a wall from just trying, trying, trying, and won't let it go. So, Ramses, can you think of any times that you've been particularly good at managing your mental capacity, stepping away, getting back, and what you did to make that happen?

RAMSES: I do that quite a lot, I think. I guess I haven't really worked on any very complex problems, at least not this week.

MIKE: [chuckles]

RAMSES: In general, though, even if it's not that complex of a problem, sometimes I tend to just want to work to just burn through things, or get things done, but we all have our limits. So, after probably about two hours, it's kind of where my max is at these days. About an hour and a half, two hours is when I start feeling antsy. I'll just get up. I'll see the time; oh, okay, it's been a while.

So, I just get up, walk around, stretch out, clear myself off of that task. I also do that even if it hasn't been 2 hours. It's not, like, 2 hours and then, you know, or work for 2 hours and then take a break, work for another 2 hours, take a break. That'd be insane.

It's usually 2 hours is the max. Or I work on something; I'm at a good stopping point; I'll take a break, even if it's been, like, 20 minutes. Just take a quick, you know, minute break or whatever just before you hop into something new because that something new could take 2 hours, right? And you don't want to be stuck at a 3-hour interval.

MIKE: Right. There are natural gaps in the work that you can use to your advantage.

RAMSES: Yeah, I find that even just a one or two-minute break sometimes is good enough in between, especially shorter sessions, just because you're usually having to switch between context when you're doing either your own code, or reviewing code, or meetings, or whatever. But yeah, if I'm just taking a break, it gives you a little bit of kind of a...it re-energizes you a little bit just so then you have a little bit more mental clarity for the next task.

MIKE: Right. I find particularly after lunch, like, that post-lunch slump, maybe half an hour after lunch, is when I tend to have the hardest time with focus. And if I just do something else for 10 minutes, maybe not even work-related, like the walking around...occasionally, I'll even take a 10-minute nap [laughs]. Don't take a two-hour nap. That's not going to work.

RAMSES: Oh yeah.

MIKE: I'll set a timer for 15 minutes, lay down, and think, I wonder if I'm going to fall asleep. And the next thing I know, the timer is going off, or, actually, a lot of times, my timer doesn't even go off. It's just been 10 minutes later. I wake up, and I’m like, oh, I feel way better [laughs]. And I usually...I actually don't take a nap very often, but I do occasionally. But I will take 10 minutes of doing not work. And I sometimes feel, well, I should be more productive, then I remind myself, this works. This works. And then, I'll go back, and I'll find myself in some of my best flow state of the day for the next hour.

RAMSES: Yeah, that extra 10 minutes is certainly worth it.

MIKE: I've been thinking about times when I was definitely able to work on something for longer than a couple of hours. There was one time, quite some years ago, where I was working at a place that was growing really fast. Our traffic was growing tremendously. And our indexing solution was starting to fall apart, and it had been working. And we were running multiple instances of the index because one of them would sometimes crash, and then we'd fall back to the other one. It got to where even with running several instances at once, there was usually one down or multiple down. So [chuckles], it was not working, which was a bad situation.

So, I had to replace it with a different indexing solution. The one we were using was, like, a rewrite of the popular Lucene engine that's written in Java, and this was written in C to try to be faster, and it was; it was way faster. It was also buggy. And sometimes, it would randomly crash, and we didn't know why, and neither did the author.

And [laughs] so, I switched over to just a standard library, that [inaudible 24:29] Lucene Solr is what I was using at the time, that set up all the infrastructure and all the configuration switching over because they were minor syntax differences as well as some other things. It took me multiple days, so I didn't sleep for a couple of nights in a row [laughs]. And that was one very long day.

It's really weird when you get up in the morning, and you're seeing your second sunrise after that [laughs]. Should this be working? [laughs] I can't imagine that my brain was in the right place. And I thought, how did I make that happen? Well, it took a huge amount of mental energy to have that focus, but also, it wasn't a particularly complex problem.

I was at a small company, and I was the only one who was in the position to do it. So, the job had to get done, and somebody had to just go through all of the work to configure everything. It was kind of straightforward work that just took a lot of time. That's something that you can generally do because you don't hit that cognitive wall because you can maybe make subtle shifts in what you're doing, right? If you're starting to have a hard time focusing, work on another part of the project and keep pushing it through.

If that was a very cognitively demanding task, I couldn't possibly have done it, right? You can't imagine thinking, well, I need to solve this math proof; yeah, I'm just going to work on it [laughs] because there's some places and mental energy, some gaps you have to cross in your mind that you can't force. So, I’m thinking, well, one way that you can work through, you know, if you need to work things through a long job, you can do that if it's not that demanding. But that's different, right? That is only possible if it's not especially cognitively demanding.

RAMSES: Yeah. Yeah, that makes sense. I used to work freight a number of years ago, and I very rarely took breaks...some breaks, you know, just naturally. But it was not uncommon to work, like, 4 to 6 hours straight, just throwing freight.

MIKE: [laughs]

RAMSES: But it's not mentally taxing. It's just very physically...relatively physically.

MIKE: Right. Like the landscaping [laughs], digging holes [laughs].

RAMSES: It was mostly just cold.

MIKE: [laughs] If it's really cold, then the exercise helps [laughs].

RAMSES: It did. It was, no, it was great, fittest I've ever been [laughs].

MIKE: I’ve brought that up as an example to say, well, you know, there are some exceptions, and you just gave one as well. There are times when you can push through, but those times have very specific constraints. They kind of work around some of your mental limitations by maybe just not hitting them. If you don't have to think that hard, you can probably push further. Of course, eventually, you hit physical limitations. You need sleep. Your body needs sleep [chuckles]. You probably are going to run into muscle fatigue if you've [laughs] been doing demanding physical labor for 4 to six 6 hours. At some point, you have to stop, and there's no shame in that. It's just the constraints of the tool we have.

RAMSES: Yeah, the human...the longest anyone stayed awake for is about, like, 19 days or something.

MIKE: Was it 19 days? I thought it was less than that. I thought it was, like, 7.

RAMSES: Yeah, maybe my dates are wrong. I thought I saw something where it's, like, either 11 or 19, something just absurd.

MIKE: So, now I'm looking it up, and now you've got me curious. 11 days and 25 minutes.

RAMSES: That's...okay, wow, that's pretty insane.

MIKE: That is. I've heard...I guess there's a rare medical condition that makes it so you can't fall asleep, and it's a death sentence.

RAMSES: Yeah, that's rough. Wow.

MIKE: Luckily, very rare. That rest is necessary. It's absolutely required for life. That's a theme I really wanted to focus on today is that there is no shame. There shouldn't be any negative feelings around taking the break that you need because that's not being less effective. It's being more effective.

Cars need oil changes, right? Gas motors need oil changes. And you might think that you're using your car more effectively to not get the oil change because you're spending more time using it without doing the maintenance. And I think that that kind of thinking, I think most people would agree, is probably wrong. Because what's going to happen is you're going to destroy the vehicle so that you can't drive it at all.

If you take the time to do the regular maintenance, it does take you off the road, right? It takes the vehicle off the road, but it allows it to keep going. And with that regular maintenance, it can go for hundreds of thousands of miles. Without it, not nearly that much, right [laughs]? You might make it a few tens of thousands, and then something's going to break, and it's not going to work at all anymore.

The best use of that vehicle is going to require that regular maintenance. You can also try running it without fuel, right? Say [chuckles], “Well, I need to get down to as little fuel as possible.” And then, you miss the gas station, and you have to get your friends to push, or you get a tow. Also, not a great way to ride your vehicle.

RAMSES: No, it's not a great way to spend your evening either.

MIKE: [laughter] No, it's not.

RAMSES: I guess it makes for a good story, though.

MIKE: Right, once [laughs]. That maintenance is necessary, and it's not shameful. It's not wrong. It's the correct thing to do to get the use of your vehicle. Likewise, giving your body the maintenance that it needs: taking that break every couple of hours, getting the sleep you need.

And maybe that break can consist of a variety of things, right? Maybe you take a quick nap. Maybe you walk around the office building. Like you say, if you're home, maybe you jump on your bike and do a few laps around the neighborhood. Whatever it is that you do, we all have our different things that we do, do it because that's not avoiding work. It should be embraced because it'll make you more effective.

RAMSES: When you're stuck on a hard problem, you’ve stepped away, how do you get back into it?

MIKE: That’s interesting. Do you have any suggestions?

RAMSES: I think it depends on maybe the complexity of the problem or what you're working on. But sometimes if I'm kind of knee-deep in and it feels like you're maybe not getting anywhere or not getting where you want to be, sometimes I'll just start over or start at a...kind of backpedal a little bit, kind of re-approach the problem in a different way, and I find that sometimes works.

MIKE: When you asked that question, you got me thinking about it, and I came to almost the same answer. I find that what's very helpful, for me, is to go back kind of to the beginning, or a few steps back, and recheck my assumptions. Did I make the right choice at this fork in the road, right? Are my assumptions correct here? Are my assumptions correct here? Are my assumptions correct here? And work my way down. And, generally, when I effectively solve the problem, I'll find the spot where, well, you know what? I made some assumptions here, and maybe they weren't right. And [laughs] it's hard to see that when you're deep down in it and depending on those assumptions.

RAMSES: Yeah, it's really tricky if you got, like, 30 file changes and 500 lines of code. And you're like, okay [laughs], where did we go wrong?

MIKE: [laughs] Yeah, and maybe rebuilding part of it, doing it again makes more sense. A lot of times, the hard problems, the most cognitively demanding problems, for me, are debugging because you have to figure out the context. You have to figure out where the problem is, and a lot of times, it's not your own code, right? It's somebody else's. Rarely is it all your own, right? You're working in something bigger.

And having to understand where you're at in that problem and figure out the assumptions along the way is very demanding. And [laughs] it's easy to miss something somewhere. And sometimes you'll find out...I’ve probably talked about this in other podcasts, but several times, I can think of getting to a point where I thought, well, this is impossible. This is absolutely not possible. And when I think that, I've learned to step back and say, “You're right. It is impossible for this to happen, given the assumptions that I have made. And if that's so, then it means not that the problem is hard but that my assumptions are wrong.”

RAMSES: Yeah, yeah, that makes sense. Yeah, it's just ignorance, right? We just don't know what we don't know.

MIKE: Yeah, and then stepping back and saying, “Well, if this is impossible for it to be happening, then that means that some of my, whatever it is, my assumption is wrong.” The hardest debugging problem I had that I can think of took me a month.

RAMSES: Whoa.

MIKE: Yeah, and multiple times. So, I was using a new library. It was actually a library that's going to be deprecated soon, TikiTorch, which you've probably used.

RAMSES: No.

MIKE: [chuckles] It was a library that was brand new at the time, and it does asynchronous messaging between applications. So, you're dealing with a cross-application problem, so it's a distributed system, and it's multi-threaded on both ends. So, it's concurrent on both ends, and there's no guaranteed order of operations because you have different threads that can launch at different times. So, you're dealing with both concurrency and a distributed concurrent system [laughter].

And twice, I was trying to figure out why it wasn't working. We were getting messages that weren't getting through. And we were also having high memory usage and trying to figure out why. So, it was a long time ago, so I don't remember all of it. But I remember that the way we were using the library was wrong. We were firing off a new thread every time we used it, rather than relying on existing thread, which was bad.

RAMSES: Wow.

MIKE: That's where the excessive usage came from. So, our usage of the library was wrong, which was firing off a bunch of threads, which was slowing down the app. And then, two other places in the code, I thought, this isn't possible. This can't possibly be happening. I had to step back, and I thought, well, if this isn't possible, is the library broken? And it was [laughter].

So, I found two bugs in the library. And the library had been fairly thoroughly tested, but there were a couple of edge cases that got missed, and there were bugs in them. And [laughs] if you reach something that's impossible, if you can prove that you're...you say, "Well, my code can't possibly be wrong," take a step back. Well, maybe that's true. But that doesn't mean that the system underneath it is, right? And that's a hard jump to make because you usually trust the libraries that you're using, right? That's kind of the standard practice. At some point, you've got to trust the system, that the system's going to work if it could just do what you tell it to do. But sometimes, the library's wrong.

I remember my very first full-time development job, we were working with a database adapter that, if you passed it a null character, it would break the connection.

RAMSES: Wow.

MIKE: And our connection would sometimes break, and we didn't know why. And it turned out to just be if somebody accidentally put a strange control character in their input, it was killing our database connection [laughter]. It was a problem in the adapter. My manager I was working with at the time submitted a patch to the maintainers of the library, and they got it applied.

RAMSES: Wow, nice.

MIKE: Yeah, yeah. It was nice. Sometimes, it's the library, which was kind of a long tangent. I grabbed your tangent and ran with it [laughs]. How do you get back into it? Well, a lot of it is going back, right? Going back a few steps and exploring it in a different way for me. For me, a lot of it is challenging my assumptions because usually, where I go wrong, it's because my assumptions are wrong somewhere.

RAMSES: Yeah, yeah, that makes sense. Yeah, it's really easy to have wrong assumptions sometimes based on what you know or what you don't know.

MIKE: Exactly. It’s like the ignorance you were mentioning before. There's no shame in ignorance as long as you're working on it [laughs] because we all have most things that we're ignorant of. And we're going through the process of undoing that a little bit.

RAMSES: Yeah, I think I'm ignorant of most things, which isn't bad.

MIKE: Yeah, no, in an incomprehensibly large universe, we couldn't help but be ignorant of most things.

RAMSES: Yeah, right, right, yeah.

MIKE: Thank you, Ramses. This has been great.

RAMSES: Yeah, this was great.

MIKE: Hopefully, listeners, you've gotten some ideas as to how to manage your day to be more effective by taking some time to be less so, taking some time to do something different than the work you're on so that you can give your brain a chance to rebuild its reserves.

And with that, thank you. Until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, Mike, Kyle, and Will dive into the evolution of DevOps and its emerging subfields like Platform Engineering and CloudOps. They discuss how system administration has transitioned from hands-on server configuration to abstracted, automated infrastructure management. Kyle highlights the ongoing debate about whether these new titles represent genuine specialization or are merely rebrands of traditional DevOps roles. He explains that while the roles share significant overlap, Platform Engineers focus on building tools and workflows for engineers, while CloudOps specialists handle deployment and configuration in cloud environments. Despite these distinctions, Kyle acknowledges the fluidity of responsibilities within modern DevOps teams.

The conversation also explores the complexities of managing cloud environments, especially within Kubernetes ecosystems. Kyle explains how tools like CMOs and YAML configurations bridge the gap between engineering and cloud infrastructure, enabling smoother workflows. They touch on the growing abstraction in networking, including dynamic layers like sidecars and service meshes, which add flexibility but also significant complexity. Kyle emphasizes that while these advancements streamline deployments and observability, they also introduce challenges in maintaining clarity and efficiency across multi-cloud and multi-region infrastructures.

Finally, the team discusses the future of DevOps, including the increasing role of AI, automation, and multi-cloud expertise. Kyle predicts a continued shift towards higher levels of abstraction, with AI potentially optimizing auto-scaling and failover processes. However, he notes that AI struggles to keep up with the rapidly evolving DevOps landscape. Observability, cost efficiency, and auto-scaling remain top priorities, with multi-region deployments posing significant challenges. The episode concludes with a lighthearted reflection on the inevitable "closet of servers and box fans" lurking behind even the most advanced infrastructures—a reminder that despite the abstraction, physical and technical realities still anchor every deployment.

Transcript:

MIKE:  Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting again today. With me, I have Kyle and Will. Today, we're going to lean heavily into Kyle. Kyle is from our DevOps team. Well, I say this...we were just talking in the pre-call that we say DevOps, but even DevOps is kind of falling out of favor as a term because there's platform engineering, other things. But 10, 15 years ago, there was no DevOps; it was system administrators.

And for those of us who were doing engineering at the time, a lot of times, it was just a developer given some system administration tasks, and [laughs] we’d get them done. I remember once I was driving to my...well, I was a passenger in the two-hour drive to my parents-in-law and banging out and configuring a server on the drive on a really weak mobile connection. And I got that done by the time we got there. And I was custom, just shelled straight into the server, configuring it as I went, based on stuff I’d done. And it was always that way. You shelled in; you set up the server yourself, and sometimes you went into the data center [laughs], and you brought in the hardware.

We live in a different world today, and there's a reason that people have done that. If you’ve got 50 servers that have all been hand configured by somebody, and then he decides to go somewhere else, or she, or they [laughs], even if you have a pool of people that have been doing that, you lose so much knowledge. It's just a horrible way to maintain things. Just like you don't want to have your code sitting on somebody's personal laptop, we use a version control system. All of the best practice from software have made it into system administration. Totally changed the world.

I'm talking too much because I'm not the expert. I'm just remembering how it was back then and knowing how much better it is right now. I don't even think about DevOps anymore, about the platform. It just happens. I merge [laughs] my code, and it ends up on production because of what Kyle and his team do. Kyle, how do you see that history of how we got from system administration? And Will was asking on the pre-call as well, is it just a rebrand, or [laughs] is there something really fundamental here that has changed in that process of deployment and infrastructure?

KYLE: So, this one's been a little bit interesting to me because I was talking to a coworker of mine that he went out to do some interviewing. And I asked him, “Hey, if you were to do a mock interview for DevOps, how would that go?” And he was just like, “Oh, well, I've done system admin-type interviews.” And he’s just like, “So, what's the difference?” And even some Google searching yielded, like, there's quite a few similarities. There's some differences, you know, maybe they manage the actual servers more, or they manage, you know, a system admin would manage more of, like, physical hardware more type thing. And DevOps is more of that CI flow, programming flow, and then the cloud infrastructure type scenario.

But then it kind of breaks down more because then you've got your specialties, observability being one of them, and each company has their opinion of what that should be. Should that be SRE? Like, here at Acima, we were the DevOps team, and we are still the DevOps team, but we have rebranded. There's half of my team that's now platform engineers and half that's CloudOps engineers. We're still doing the same skillset, but now we're called something else. I came in on the DevOps naming convention, so I don't have the full history there. And I don't fully understand the difference myself in a lot of them because there's so much crossover.

Even looking at new jobs or job postings at different companies, it's like, okay, well, that's a new name, you know, just CloudOps or cloud engineering. It's just like, oh, those are just DevOps tasks, and it's like, they're wanting a DevOps guy. Where's this new name coming from? But I think that's part of...part of the divide is coming in from trying to divide DevOps as a culture instead of team. We're trying to teach those key concepts, and it's dividing out, like, what a DevOps engineer is actually doing and giving them more of a title that's appropriate to that. So, if you've got somebody that's working on your platform all the time and designing your platform, maybe they're that platform engineer. If you’ve got somebody that's working in one of the cloud spaces, then maybe they're more of a CloudOps or a cloud engineer type person.

WILL: Okay, so, it's not the cloud. Like, your platform engineer and your cloud engineer they're different. Is it a situation where, like, okay, you got to have your own data center, or else you're a cloud engineer? Is that more/less it?

KYLE: So, I'll explain the difference on our team, which is we have a few of us where we actually are writing the code for the platform. So, like me, I'm sending out the observability infrastructure and designing how that flow is actually done. And the CloudOps individual is the one that's taking in the requests and configuring the alerts, configuring the dashboards to deliver to the requesting team. It’s platform would be, you know, your CI/CD Jenkins, setting that up, and then Ops also be setting up --

WILL: Can you be more specific? Like, it doesn't have to be everything. So, like, case study it through for me. Because these are just sort of broad platform technologies and you can cover a lot of ground, right? Can you be more specific about case studies and workflows, and like, this workflow goes here, and this workflow goes there? Does that make sense at all?

KYLE: I'm not quite sure what you're asking me.

WILL: Ah, okay. Ah, who knows --

KYLE: But we still cross. Regardless, I mean, it's one of those things where we're crossing each other's boundaries. And we call ourselves separate teams, but we're still one team. I guess that's where it's a little bit foggy for me is to, like, well, you're wanting a defined line, I think. And even on our team where we have the separate sub-teams, there's not a defined line of who does what.

WILL: Right.

MIKE: Is there, like, aside...do you have some people who are more dedicated to actually interacting with the clouds and a side that's more integrated with interacting with the software side? Because there's the cloud side, right, and then there's the software that goes, you know, you interact with the engineering side. And so, there's two interfaces, right? The cloud interface, the one side that you deal with, and the other side is dealing with the engineering. So, I could kind of see that you've got an API of sorts on either end, and you have people assigned to either end. Is it that sort of division?

KYLE: So, not quite in our setup. We’re doing the legwork. We're setting up the actual platform to be used, so setting up the standard workflow, maybe. Okay, so, like, our team, we've developed, like, a CMOs is a tool that we've wrapped several CLIs with. And in order to use that tool, we have a configuration YAML that will call different components in order to use that, so the platform people would have been the ones designing that, designing the tool, programming it out, and getting that functioning. And it would be the CloudOps guys that then take that from the engineers’ requests, and then they'll take what's been configured in that YAML file and generate the Terraform configuration that would then generate the cloud infrastructure from there.

MIKE: So, that sounds like somewhat like the division I was thinking I was describing, where one side is building the interface that the engineers use, and the other side is actually building the cloud side so that that can be deployed. It's not like there's a singular interface, but there's two ends of the interaction. There's the interaction with the engineering on the engineering side, then the interaction with the cloud [chuckles].

When I say the cloud, obviously, it's not a singular thing. You may have three or four different clouds that you're working with, all of which have slightly different services. And you have to make that work and use the tooling that works with that. So, you know, it requires a team to manage that and make people think about it as just a cloud. And likewise, you need to have a team that actually gives the engineers the interface. They don't have to know, right, they just know your interface.

KYLE: And that model is just what's working for us here. I'm aware of other companies where they have a DevOps team or a systems team that is specific to their Kubernetes environment. They're the only ones that touch that Kubernetes environment, and then they'll have a DevOps engineer on every actual development team that interfaces with that. But they don't have any permission to touch their Kubernetes environment, but they can configure different things.

So, it's kind of dependent on what each company is doing. And it is something, too, that goes through the industry in waves, too. I've seen it both in DevOps and in QA, where, you know, how much of it is a culture? And how much of it can developers themselves do, and how much of it do we actually need teams to do this work as well? And I think that's a size and stuff like that.

MIKE: So, you talked about a lot of companies, [inaudible 10:28] this as well, that have DevOps distributed among the teams. And you all have done some pretty amazing work by creating a tool that kind of replaces those people so that we interact using YAML files and the CMOs you mentioned, too. You know, rather than having to have an embedded DevOps person, you tried to build the tooling so that it's self-service. Kudos on your team for building such an easy-to-use system. But it doesn't mean that...most companies don't have that system yet, right?

KYLE: No.

MIKE: And, in that case, you almost need to have a person from the platform engineering, DevOps, whatever you want to call it, who interfaces with engineering so they know how to use this mysterious...this platform that you've built, it would seem like.

KYLE: We’ve even tried to do an ambassador program a few times to get engineering individuals from each of the teams up to speed on what we're doing so that that is a bit more seamless and easier for them to deal with.

WILL: Yeah, it is tough, man. The bigger the infrastructure gets, the more of a sort of, like, a dark art it is to actually spin up a box and put it onto the web, you know. I'm a [inaudible 11:42] system administrator. But at Acima, or the place that I currently am, like, get a box up there, get it on the load balancer, get an endpoint, assign it to all that. I don't know, man, whatever [inaudible 11:57] people's problem; that's it; I'm out, which is it's just such an odd...it's an odd place to be, if you really think about it. If you sort of came up just spinning up servers, plugging them in, let's roll, you know, like, wiring switches, I literally don't know how I would get a box up and an IP assigned, and get a route assigned to a thing. I'm like, I don't know, but you got to do that.

MIKE: Well, and usually when people are using a system...you've mentioned Kubernetes. Kubernetes is not for the faint of heart.

KYLE: No.

MIKE: [laughs]

WILL: Is it really bad?

KYLE: There's so much to Kubernetes. It's almost got to have its own degree, I feel like. It adds several layers of networking, like, just diagramming it out. We've had a team member that...it's the running joke because every time we add something to it, we have to diagram out the network ops because we're adding in another network ops just about every time we add a feature.

And you've got standard Kubernetes. You've got your gateway. You've got the Kubernetes internal networking. Then you'll have...if you add something like Linkerd, a mesh of any kind, you're going to have more networking layers, and it terminates here, and it terminates here. But outside of Kubernetes, you're also terminating your TLS, or you're doing internal TLS. That's just talking, like, I'm just throwing out buzzwords, I guess, at this point for referencing how large it is.

WILL: [inaudible 13:39]

KYLE: Huh?

WILL: I don't [inaudible 13:41] transport layer security matters.

KYLE: But it's its own ecosystem. And then, on top of it being its own ecosystem, you've got to figure out how to run your servers and/or your code in that ecosystem. And then, with Docker, it's been made easier by, you know, it doesn't really matter your host too terribly much. You can just run an AWS host. You can just run an Ubuntu host, whatever you want to do. But then there's works and different things between any of the languages that you're running, too, as much as you want to think it's just straightforward, oh, I can spin up a Ruby app, and it's going to act just like a Java app, well, no, there's several differences. And is it going to act like a Haskell app? No. Haskell is a world different. So, configuring all of them in Kubernetes it's a lot of investigation and learning just to get it all working together.

WILL: I always felt like that was one of the more interesting aspects of going back to the primordial ooze of system administration. It's like, system administration in the battle days you were administering a Linux or, God help you, a Unix, or a Solaris server. And Google wasn't a thing; Stack Overflow was not a thing. You just kind of had to know. It was like a London cab driver where you just had to drive [laughter] the car around as an apprentice for a couple of years until you just knew.

And I feel like a substantial portion of that is still big thing, right? Where it's like, okay, Kubernetes, that's fine. But if I have to set up a Rails app, and I have to set up a Java app, and, oh, my authentication headers need to look a little bit different here or a little bit different there, and I have to get certificates added to the Java runtime, right, these are just things that I know because I've burned my fingers on the hot stove before.

There is, like, I don't know, like I said, like a level of...because you're responsible for every platform's thing they didn't get quite right, or things that they just kind of zigged when everybody else zagged. And you have to make sure no, no, smooth it all out. There's a level of just witchcraft that's just attached to it.

MIKE: If you want to have your brain bent, explain a network sidecar [laughs].

WILL: A network sidecar. I don't know what a network sidecar is, but, like, hit me.

MIKE: [laughs] I could try. Kyle talked about layers of networking. You can take your normal network stack. You can decide, hey, you know what, I'm going to add something onto my packets. And it's just magically going to work in this infrastructure because I'm going to add a new protocol on top of my networking that I'm going to run between my servers in order to allow me to do monitoring, or security, or you name it. You are augmenting the network protocol by adding a layer on top of it, and you can do that dynamically and use plugins for it. Did I capture that, Kyle?

KYLE: Yeah. That's kind of similar to what that Linkerd that I brought up is doing, kind of your meshing. They've got an injection pod, and if you have the right label on your pod, it will inject a sidecar on your pod to do the communication between pods. And it will terminate before it gets to your app, so that you have that layer taken off of your communication, just in time to communicate with each other then you communicate back. It's putting it back on there until it's the next sidecar.

WILL: That makes sense to me. I don't know; there was nothing about that was all that crazy. It's like a little middleware, where it’s like, oh yeah, I'm going to take a packet. I'm going to wrap it with this other stuff, this other thing. And at the server level, at the server layer, or, I don't know, the switch layer, whatever, okay, all right, that makes sense.

MIKE: Which is cool. But going back to, like you said, the primordial ooze days, you had a network stack, and that's what you got. And now we're talking about a network stack that is dynamic, that is configurable, and that you could plug stuff into. Let's say I wanted to add some fancy security protocol to the communication between servers. Nobody has to know anything about it. You just inject it into this mesh, and it just magically works, which is, of course, sometimes it doesn't work. You just might [laughs] have a whole team that administers it.

KYLE: Well, now it's getting a little bit crazier, too. Speaking of how deep does your networking going, we haven't yet to explore it here at Acima, but there's a new tech, and I'm not sure if there's others, but it's called a vcluster, where you can do virtualized Kubernetes clusters inside of your Kubernetes cluster, which is an idea for similar environments, development environments, QA test environments. Those type things are good use cases for that, but they're even touting it for some production-type loads and stuff, too.

And it's like, how far down do you go? How many layers of networking and systems do you go? It's getting to where we're trying to configure, I’m trying to think of what it's called, what Vagrant and Docker is, but, you know, one button push an entire environment. You know, it can configure an entire Kubernetes environment and, virtualize it and put that in your Kubernetes cluster. Like [laughs], I don't know, it's going crazy.

MIKE: Yo, dawg, I hear you like networking. [laughter] That's interesting. So, you mentioned virtualization and advanced networking tools. Would you say that virtualization is part of what has driven some of the fundamental changes we've had in infrastructure?

KYLE: Yeah, it's created that...just back in the day when I was in QA, you always wanted to verify the binary. That was one of the first things that you always tested. You wanted to verify the binary between the one you were testing and what was being delivered to make sure the same thing was going out, the same thing hadn't been manipulated at all. Virtualization and being able to package that virtualization specifically is making it to where we can deliver entire environments rather than just single packages. And we're just expounding on that, right? Just expanding on how much we can deliver, how much we can keep in commonality because now we can build an entire OS with an application running on it, and that's our binary.

It's a docker image, which is kind of what I'm referring to at this point, but now you don't have to...we spend so much time configuring an environment kind of like Will brought up earlier. It was back in the day that you babysat that. That was your baby. But anymore, it's you build that environment once with the application that you need. That’s your deliverable. Now you're delivering that.

And we're trying to get to the point where we're delivering entire platforms, entire systems. It's just moving faster and faster. And I think that's part of, yeah, like, why the title keeps changing, why our expertise keeps changing is we're doing something different because virtualization and dockerization, those type of things, have allowed us to pivot so much faster.

MIKE: So, it's not just trying to put a few, you know, use better resources by putting a few servers on a single physical server. Now it's completely changed the way you think about deployment, is what I'm hearing. It's nested.

KYLE: Yeah, it's very nested, yeah.

MIKE: So, you've got...there’s not just an advantage in resource utilization, although there is that. It's an advantage in being able to configure your...all the way to the OS level, right? And even which libraries you're including on your OS to be a deliverable. So, you don't have the dependency hell that people have run into forever because it's just configured as part of the binary that gets deployed. So, all of your components, all the way down to the operating system level are specified, and it goes together as a bundle. You remove a lot of the questions there, so it's not...although this does end up using resources more effectively.

You get all this crazy configuration. You say, well, then you just take it into layers, have an entire system of...if you got a hundred services, it's what I'm hearing, is you can configure all of that, and it's all just configured. And you could deploy that in multiple places if needed, so if you want to spin up an ad hoc development environment, for example.

KYLE: Yeah, and it's going deeper and deeper. You said, more effectively, using resources. To some extent, we're getting to a point where you can, but you don't have to run on dedicated hosts. I'll speak to the one I'm aware of is, you know, there's Fargate and serverless options that will run Docker. And in that, as long as your app fits into the constraint of their limits, it'll run it. And you don't even need to manage your hosts anymore. We do over here at Acima still, you know, in some of our production workloads and development workloads. But you don't have to even manage your host at all if you don't want to, and that's where we're at these days.

WILL: Yeah, but how much of that is, like, how do I put it? Like, how much of that is, like, a business model looking for consumer demand? It's just, like, it's a little...you don't even want to have a web host, right? Like, for some stuff, for some specific use cases, okay. But it seems like --

MIKE: It’s a level of abstraction. Once you get away from the idea of thinking, I have to manage this physical server, it changes the way you think about the problem. I'm thinking about back when people wrote, well, before even Assembly language, right? People punched cards, or they had the machine language. And you had to just map those codes over to something that you were going to put in as numbers. It doesn't allow you to think about algorithms in the same way as if you add another layer of abstraction on top of it. And, at first, people thought, well, I can do that better. I can write my own Assembly, and I can do a better loop than you can, and, at first, that was true. At first, that was true.

WILL: Oh, it's true now, right now.

MIKE: I saw an interview with Bjarne Strusup, the guy who made C++, said, if you tried to carefully craft Assembly to do better than the compiler does for a modern compiler for C or C++, they will almost always do worse.

WILL: Oh, no way, dude. I've done it myself, personally, hand-rolled Assembly. It's easy to build. I was a compiler researcher for a while.

MIKE: Okay, well, you were a compiler researcher [laughs] [crosstalk 25:59] compiler researchers.

WILL: Well, you got to know Assembly, right? But, yeah, you can beat the compiler most of the time.

MIKE: I think that for small circumstances, I'm sure that's true. But the larger you get, the more complex the system, the harder that gets to beat because it just takes too much time.

WILL: Well, exactly. It's never worth it, right? And the compiler is pretty good. And if you know how a compiler works and you know how your hardware works...and those two things, especially for a modern operating system, modern processor, modern memory management, you got to know some stuff. But yeah, a compiler is easily beatable. You should have that knowledge. And you care, right? Because you need a CPU-bound hot loop. Yeah, they're very beatable, but it's usually just...it's almost never worth it, right? Because, like, how many million times do you have to run that thing to get to win, right?

MIKE: And I had to do that for a class once years ago, you know, beat the computer by doing some loop unrolling. I know that you can do it, but it's, again, the amount of time you spend on that is almost certainly not worth it because think about the --

WILL: Oh, never [crosstalk 27:18] flow, like, database, bytes on the wire, network downstream calls. End of list.

MIKE: [laughs] There you go. You need to optimize something else. And where I was going with all this is that thinking at that higher level of abstraction. Because the more you have to deal with those low-level details, the less you're thinking about, well, maybe I can add an index to my database [chuckles]. You want to be thinking at a different level of abstraction.

So, I think that what Kyle's describing here is just thinking about the problem from a different layer. Now, at first, it's going to seem like, well, that doesn't make any sense to me. But I think that 20 years from now, people will wonder how you ever thought differently because they will invent stuff that we haven't thought about yet.

WILL: I'm just thinking about, like, how I’d do it if I wanted to virtualize my server. And, basically, what it would be is I would just have to have some sort of routing table thing live, that's like, oh, if I go to this request, if I go to this thing, and then I would have to basically have a web server that I awaken from memory, and then it would do the thing, and it would go back away. So, to serve my request, I would need to go through all the process. It would just be like, I [inaudible 28:46] my infrastructure, right?

And I was thinking like, okay, well, where does that make sense, right? Big stuff that's not used very often, very infrequently used servers, like, places where you don't have a performance thing. But, in the end, it smells, to me, like a pod in cold storage that you could boot up again relatively quickly.

KYLE: They have the different types now. So, you can have warm hosts, warm, like, instances. The boot time isn't nearly as bad.

WILL: Well, exactly. But you have a warm host, but, like, a warm host, but not a hot host, a warm host. But it’s like, okay, what is that? I just took the memory, the memory image, and I flushed it to my quick SSD, so on top of my virtual memory ecosystem. And I was just off to the races, like, weird, you know, it's just odd. It's not like you're not put on ice in one of these big cloud data centers, your virtual host, like, your virtual EC2 instance. But that thing hasn't gotten a hit in a couple of weeks. You really think it's like...it’s warm, maybe not simmering, maybe not [inaudible 30:18].

[laughter]

KYLE: It’s not ready to take the Black Friday load.

WILL: I'm just saying.

MIKE: It's above absolute zero [laughs].

WILL: You'll never catch him. You’ll never catch him paging you out. You'll never, like, there'll be no smoking gun, and you won't be able to subpoena anybody. You'll just be like, hey, I paid for that thing to be up all the time, and it wasn't. It was just warm. And they’re like, you'll never [inaudible 30:47].

[laughter]

MIKE: You're saying the same thing that I was saying a minute ago is that once you've got to that level of abstraction, you don't have to think about all the details that you just said, waking up from memory. And those details are interesting. They're all interesting on their own terms, within that layer. And the engineer who wants to deploy to it they don't want to know about all that. They just want to merge their code and see it go to production.

WILL: Right. Yeah. I publish my code. It makes it to the CI. I get a little object, my little binary that goes in. It gets integrated in the pod, pod goes into the rotation with the load balancer, we do our...what do you call it? Green/blue deployment. It shifts all the old ones out, all the new ones are in, and then, boom, I'm locked. I'm in business. Easy-peasy.

MIKE: [laughs]

KYLE: Easy-peasy.

WILL: Easy-peasy, lemon squeezy.

MIKE: [laughs] If the engineers have to think about it, then DevOps still has some more work to do, right? Where do you see DevOps going in the next few years, Kyle?

KYLE: I've kind of been trying to think about that and, like, how does AI come into play? And then, we're really kind of headed towards this platform engineering type role; at least, that's what I've been seeing and what we end up delivering. And I think we're going to see more systems-based deliverables. I think we're going to continue to grow that out. And I think the next thing is, like, I think you're seeing right now a lot of specialization in the sense of clouds and which cloud, you know, you’re AWS certified; you’re GCP certified, that type of thing.

I would expect that in the next few years, it's not going to matter. They're just going to be one of the tools that gets used in your multi-cloud rotation, and you're going to need to know all of them. You're going to need to know how to use all of them, how to have them interacting and networking between all of them, and how to have either Kubernetes or whatever else it is, how all of that's going to play together between the multiple cloud environments. Because right now, we're kind of eking into multi-region, and [inaudible 33:19] understand that there's teams that are good at multi-region and stuff like that.

I think it's going to go cloud-based, and then how much of that is handled by AI and stuff like that I'm not sure. I'm not sure how much is going to assist us in our world. I've seen hit-and-misses from my perspective as some of the tooling...AI is good at stuff that's known, I guess, is what I'm getting at. And the DevOps world is moving so rapidly that by the time it's documented, it's outdated is what I've referred to it as.

And so, even some of the stuff that, like, I'll ask an AI about, it's like, I have no idea what this tech is called. I've never seen this tech. And it's like, so it can't help me with it. I'm having to use the resources online, other resources online to do that. But yeah, so that's a very long-winded way of me saying I have guesses, but I don't really know. But I see it going towards we're managing at a higher and higher level as we move on.

MIKE: Well, AI, like you said, known things. You can definitely teach a system to understand things like load. So, you could...one thing that machine learning can do, and it doesn't necessarily have to be that sophisticated machine learning, is do things like auto-scaling or failovers. And that seems like something that could definitely be automated and probably better than what we're doing today.

KYLE: Probably, yeah. Like you're saying, it's just a system that's able to take in all the metrics, all the logs, all traces, all the observability information, and is able to make decisions based upon what's coming in and scale up your environment. Yeah, I can see that happening if we get AI to a point where it's able to turn over like that.

WILL: I'd say what AI is great for, writing Nginx config files.

MIKE: [laughs]

WILL: I'm so bad at Nginx config files. I was so bad at that for so long, and I would always get it done eventually, but, like, [inaudible 35:36]

KYLE: What I’ve found AI to be the best, for me, is I hate writing RegEx expressions. [laughter] Just throw it into the AI. It gets me...I'm like, thank you. This is the best tool for that.

[laughter]

WILL: Oh, dude, yeah. And it's so bad, too, because sometimes if you switch languages around, like, everyone they get it, like, just a little bit different, and I’m like, I don't need to know. Just what's the space, man? What’s the space? Backslashes, percentages, whatever, just what is it? Nginx config files, though, man.

MIKE: I'm [inaudible 36:19] ngix config files. And you got, like, some plugins and stuff in there, and you got a...oh man. You configure your security certificate [laughs], some dark magic there [laughs].

WILL: So, let me ask you this, right, so, like, okay, like, DevOps, right? To the degree you can, right? What do you see, like, the business has a need, like, an unmet need from DevOps? What are you guys thinking about implementing? What's the need that you wish you could fulfill or maybe you're getting pressure to, like, can you deliver this thing for me?

KYLE: Like, specifically right now, what are we getting pressure on?

WILL: Well, I mean, to the degree you can. Like, what's on your mind in terms of they really want this, and either I don't know how to do it, or I'm trying to free up resources to do it, or something like that. Like, where are you seeing like, okay, I'm getting heat; the next step for me is this thing?

KYLE: Me specifically, it would be observability, just because that's where I'm focusing in on, and so cutting costs on logging and then the other metrics and traces in that atmosphere. But the company as a whole, auto-scaling, being able to react to load. Pod horizontal auto-scaling is something that we're really looking towards getting nailed down and how we go about that. Do we go about that in a standard load fashion? Do we respond to load? Do we respond to our Ruby apps? Do we respond to Puma threads? And do we have differentiators between our different languages on what we're responding to for that?

Then another one is RDS, Postgres specifically. How do we scale that? How do we make that multi-region? How do we have that single point of data? And then, how do we have that available in multiple regions for DR purposes or multi-region access? And then, on top of that...I feel like I just went brain down on what my last point was [laughs], but yeah.

WILL: I mean, it seems like efficiency, right, and cost management, like, that’s, like, [inaudible 38:47] everybody talks about auto-scaling. What they're talking about is they're talking about efficiency, right?

KYLE: Yes.

WILL: If I need 20 boxes, let's get 20 boxes. But if I only need 10, let's keep it at 10. Man, multi-region, that is such a bear of a problem. Can I ask, there are some people who need multi-region for sure. But, in general, really? Multi-region, really? If California slides into the ocean, we got to stay up, and it's a reasonable expectation because there, you really need to start thinking about is this juice worth the squeeze.

Because, on one hand, like, yeah, okay, you've got a tremendously much more complicated system. But you also, like, you're introducing all kinds of permanent all the time performance taxes, all the time, because those things got to sync up. And they have to, you know what I mean, keeping the East Coast and the West Coast on the same page, that's a long hop. That's a long...

KYLE: And you're saying that, and one of the things that we do keep in mind, or not necessarily battle with, but we do keep in mind and we monitor, is that replication latency because it's a real thing. You're replicating clear across several states, several time zones, and it's like, it’s there. Your data is not available right away. They call it real-time data, but it's not. There's a latency sink to that. And then, we kind of think of the data as a single source, but we're replicating the data in several data lakes even. And then, do you do that multi-region, too? Do you need a multi-region data lake, which is basically a copy of your data anyways, at least for us? And how do you keep that available?

I kind of tend to agree with you on some of your point, which is, is the cost worth it? Do we really concern ourselves with it? Because it's been a running joke on my team, you know, California, Oregon, if something happens to them, is being able to fail over to a new region our biggest concern? There's going to be so much else that's down. Just because we have our stuff in Ohio, Oregon blew up. What the heck? Is everything going to work still?

MIKE: You don't have to have so much an explosion as some contractor with a [inaudible 41:27] who didn't put up the flags [laughs].

WILL: Yeah, I remember back when AWS was pretty new. It was not unheard of, but, like, once a year, US-East-1? Not today, boys. Not today.

MIKE: [laughs]

WILL: Check back after lunch. We'll see what we got for ya. And Amazon never went down, but [inaudible 41:54]

[laughter]

KYLE: Should have done multi-region Amazon [inaudible 41:57]

WILL: Nope.

MIKE: [laughs]

WILL: I shouldn’t have gone US-East-1. I'm just saying.

MIKE: [laughs]. It's always US-East-1.

WILL: It's always US-East-1. Always. Always. It's always the one, man. Ohio is fine. Portland is nice. Virginia? I don't think so. NSA bots or whatever, I don't know. I blame the CIA. It's got to be their fault. But something's happening.

MIKE: [laughs] So, we've talked a lot about the amazing complexity that goes on in DevOps world today: layers, managing, well, you talked about observability. And it comes down to better management, right, better management of complex systems. And, clearly, the industry feels it's important because we've got a whole field, right, that's dedicated to managing these complex systems and trying to optimize cost efficiency is what we're saying. Is there any final things you'd like to say about DevOps? Say, you know, “This is what you should know. This has become a thing. You should know this.”

KYLE: I would just say I was sitting here thinking we focused a lot on where everything lands and how that's being managed. A huge part of DevOps is the entire CI/CD flow. And that on its own is enough of another podcast, but yeah, that's something else that we need to keep in mind when considering DevOps, for sure, is we need to know about how to build everything through that CI/CD flow and validate it and get it out because there's different pipelines. There's different approval processes, and that's its own beast.

WILL: Oh, man. So, I'm working on mobile apps, right? So, mobile app builds have to go through a macOS machine, and we've got a closet full of Mac Studios screaming, just screaming [laughter] for mercy all day and all night. And it's a closet in the office, and if that closet somebody kicks the plug out of the wall, everything stops. And people will drive down there in the middle of the night, where it's like, ah, I got to go. It's ridiculous, truly ridiculous, but it's got to be done.

MIKE: And all of our layers of abstraction are pushed away, and we see reality [laughs], the closet.

WILL: Yeah, the closet. The closet is just a bunch of wire shelves, like, a box fan, I think [laughter]. I think literally a box fan. They put a box fan on the ground and then a box fan out. So, it’s shooting up, and then shooting out [laughter]. God help you if one of those fans goes out, too, because they'll start throttling [laughter]. And this is a multi-billion-dollar enterprise running off a Costco box fan, lying on the ground [laughter]. It's not on the ground. I think it's on a milk crate for airflow, but anyway.

[laughter]

KYLE: To be fair, that ecosystem is not as easy to build on, and it is very expensive to rent virtual machines on. So, the idea that you guys have machines in a closet like that it's probably pretty common.

WILL: Oh, absolutely. It's a pain in the ass. So stupid. But Daddy Apple wants what Daddy Apple wants. It's what it is.

MIKE: Which is probably a good conclusion here. We like the abstraction. We don't want to have to think about that closet. We want to not think about that closet. We want to think about that closet as little as possible. And so, we have Kyle [laughter].

KYLE: [laughs] And so, we have Kyle.

MIKE: Army of fellow DevOps people who take care of that for us.

WILL: There's always a closet.

MIKE: [laughs] It exists.

WILL: [inaudible 46:05] closet. Maybe it's a virtual closet.

MIKE: [laughs] But it exists.

WILL: There’s always a closet.

MIKE: [laughs]

WILL: And a bunch of blinking lights and box fans.

MIKE: Thank you for listening. Hopefully, you got some glimpse as to the value that DevOps provides and how that whole field has evolved. And until next time on the Acima Development Podcast.

The episode of the Acima Development Podcast centers around the challenges of scaling teams effectively, inspired by the principles from Fred Brooks' book The Mythical Man-Month. Host Mike begins by sharing a parable about his toddler's well-meaning but inexperienced attempts to help, drawing parallels to the complexities of onboarding and team communication. The core theme emphasizes that while eager new members can be valuable, onboarding takes time and resources, which can ironically slow down progress on active projects. The discussion highlights that simply adding more people to a late project often exacerbates delays due to the increased communication overhead and coordination required.

The panel delves into strategies for mitigating these challenges, such as the importance of defining clear project milestones and fostering communication between product and engineering teams. They explore how startups often face growing pains when transitioning from nimble, small teams to larger, more structured ones. Effective leadership and the role of subject matter experts are critical in this context, ensuring continuity and knowledge sharing. Prototyping, rapid iteration, and intentional decoupling of work into smaller, manageable units are suggested as ways to enable independent progress and reduce bottlenecks.

The conversation also touches on cultural and organizational aspects, like aligning incentives to encourage behaviors that prioritize long-term scalability over short-term output. The speakers stress the need for cross-training, maintaining organizational boundaries, and fostering relationships across teams to streamline collaboration. They close with reflections on leadership, emphasizing the discipline of narrowing scope and recognizing the inherent trade-offs in scaling efforts.

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I will be hosting again today. With me, I have Matt, Ramses, Kyle, and Justin. And we're going to be talking about something that's been talked about for a long time. I will introduce this in a minute. I'm going to start by telling, as I usually do [chuckles], a little bit of a story and maybe a bit of a parable.

I have a toddler at home, and anybody who's been around a toddler knows some things about toddlers [chuckles]. They love to help. They absolutely love to help. And a recent example is, about a month ago, I was fixing something, and my toddler wanted to use the hammer to help with everything. And he grabbed that hammer, and he started doing what a toddler would do and hitting everything he could see [laughs] with the hammer, requiring me to give him a lot of one-on-one attention [laughs]. And that is true about everything he wants to help with.

His help with the dishes usually involves a lot of cleaning of water everywhere in the kitchen. His helping in the yard involves filling in a lot of holes or stopping, as I'm mowing, to let him pass. In short, he is very eager to help, but he lacks the experience to accomplish that yet. And that's not to say that he doesn't want to help, right? He has all of the right attitude. And I'd love to hire somebody like that with that kind of eager attitude.

But there are limits to the training time. For him, the training time to do my job would be, you know, a few decades [laughs], and then he could have the experience needed. And that's not saying anything about him, or his character, or his abilities. He just hasn't gotten the experience yet. That's just how that works. And I got some older kids, too [chuckles], and the thing is, the same thing applies.

And even if I went over and grabbed a neighbor and said, “Hey, can you come over and help me with something?” I'm going to have to take some time to explain to said neighbor what it's going to do to do that work. Even if they have lots of years of experience in life, they probably are not very familiar with what I'm going to ask them about. And there's this crucial distinction between somebody having technically the capacity with training and being able to do something now because there is a communication gap.

Communication takes time and that is inescapable that communication takes time. And for some people who have closer experience, it's going to take less time, but it's always not zero. Not only is there communication, there's just overhead of coordination. Some jobs can't really readily be shared [laughs] because only one person fits.

There was a book written on this topic in 1975. It's written by an author named Fred Brooks, and he titled his book “The Mythical Man-Month.” You got some nice alliteration there with Mythical Man-Month. Knowing this was 1975, it's a bit gendered [chuckles] because, obviously, there are more than just men in software, but he's using it in a more generic sense. And there have been women in software from far before 1975, so nothing gendered intended here. It's just the title of the book.

He made this incredibly valuable observation that this idea that you can just have one person one month, that they are fungible, and you can just interchange, you can put more people on a project to make you go faster is false. It's just straight-up false. And it comes largely down to this communication overhead. And if you add more people, then you have more people to communicate with and more connections between people on the team. If you have one person, there's no communication, right? It's just you.

If you have ten people, you have ten people communicating each to 9 other people. That's a lot of connections. The complexity within your team has exploded dramatically, and all of that communication overhead is not going to speed things up. And, in fact, if you add it late in the project, it will almost certainly slow it down.

That's what we're going to be talking about today is this idea of the Mythical Man-Month, the idea that you can just add people to a project and make it faster. There's our introduction. I'm going to stop talking and will let my panel here share some thoughts.

MATT: Yeah, I think when you start adding too many people to a problem, communication is key, right? And you start to play the telephone game. So, it's very easy for things to become miscommunicated. But also, there's the lack of context of the problem, and that has to be shared. People have to ramp up and learn that context and gain the knowledge of the problem they're trying to solve, and, ultimately, that's always going to slow things down.

JUSTIN: Yeah, one thing that is kind of a corollary to that is that the mythical, like, the skill needed for startup, for a leader, is the ability to have vision and to do the thing. And, generally, it's a very small team, and you can all go be very fast and very nimble. But once you're past that startup phase, all of a sudden, you want to do more. And just throwing more people at the problem doesn't solve the problem any faster.

And, all of a sudden, your skill set that you need, that you had to do the startup, which was to do the thing and to be a visionary and drive the thing, all of a sudden, your skill set needs to be management, and problem dissection, and distribution. And so, it's making the move from startup to post-startup that requires a different skill set that is difficult, that's like a gate that people run into or that businesses run into that could prevent them from being a more successful business.

MATT: Yeah, and oftentimes, startups, specifically, will bring in a new CEO once they see that kind of growth, right, that actually specializes in those skills. And we always...well, not always, but very often, we see startups who grow too quickly. And I wonder if it's more than just the financial problems that it causes. But this is one of those problems as well is, the communication gaps that it creates, and the context, and people being on the same page, and understanding what they're really trying to accomplish.

MIKE: It takes, I think, always longer than you'd expect to get somebody else up to speed. It's just hard. It's hard to take something that you can see and have experienced in your own mind and hand it off to somebody else. There's a mismatch. You have to convert it into language, and they have to convert it into their mental representation. And that transfer is inefficient and is going to take time. And, I think, over and over again, we underestimate the challenge of that transfer.

JUSTIN: So, we often underestimate it, but what, I mean, we're all experienced engineers here, and we've been studying this for a while. So, what can we do to make that transition smoother?

MATT: Well, I think there's a few things, right? Front loading is important. You need to know that something's coming ahead of time and give yourself adequate space to prepare for it. And I just want to clarify that I support the idea of contractors and bringing contractors in to help with projects, but I think there is a cost. And one of the things you need to be cognizant of is being able to reuse them. Your first project may slow down, and it may take twice as long as initially planned. But if you use those same teams and bring them in for the next project, they're going to have that knowledge and that context and can jump in and out and actually provide some lift like you expected on that first run.

MIKE: I think that's huge. You have to have the [inaudible 09:52] time. Now, that's not an easily digestible message when you're trying to go fast to market [chuckles]. And you could just come up with a new project, right? The answer is going to want to be, well, I want this, you know, already. Why does it take so long to get people up to speed? Well, sometimes that is not a constraint we can mess with, right? Sometimes, we don't get to hire until today.

MATT: Yeah, and I think there's a balance. And to the opposite side of the coin...and we've all heard this, I know, and probably most of us have said this ourselves, and that is, well, I can do it faster, so I should just do it myself. We also, when we do that, tend to create silos, and you can have the opposite effect over time and create bottlenecks that way. So, I think there is a balance, and it's just a skill that you need to learn and find out what works for you, what works for your business, and, if possible, use the same people repetitively, if you can.

JUSTIN: Yeah. Another thing that I think needs to be considered is you have to acknowledge the tech debt that you are accumulating as you go along, and that's fine. I mean, if you are a startup that is trying to get a new product to market, you may not have time to document the things that you are doing, and you may not have the time to go out and help other people understand what you're doing, either.

But you're accumulating this massive tech debt, and you need to make sure that your management or your business partners understand what's going on and the risk that they're taking with you doing all that work. Then, at some point, you got to look at it and say, “Hey, I am putting 80-hour work weeks into this business,” or “I am doing this, and I don't see a way out yet.” And, at that point, you just put on the brakes and say, “Okay, we've got to pay down this tech debt that we've accumulated because it has turned from a we've proven out the business. We're making money, or we're not making money anyway. In some way, there's success. I'm still getting a paycheck.

But you’ve got to pay that business risk down because that tech debt is converting over to business risk at a rapid rate as well. And you've got to make sure that your business partners understand that, hey, there's a business risk here. If I die or go accept an offer in another company, their company that they are depending on is going to face a lot of pain to replace me.

And I think that making sure that, I mean, that's just part of what a senior engineer needs to understand is, hey, you got to communicate these risks back to the business people. And then, they will, hopefully, negotiate with you on ways to address that risk. And that is just the whole list of things that you got to do of onboarding more people, creating documentation, diversifying your products, making it not be a monolith. All of those things are to address that tech debt and that which translates into business risk.

MATT: Yeah. And we're talking a lot about startups. It doesn't have to work perfectly. It just has to work idea, right? But what about the larger companies, or the big, giant corporations, or even a mid-sized corporation publicly traded? We're on the market. We're publicly traded. We're not a startup anymore. And we're facing some of these growth issues where we want to move quickly but decisions have to be made on how does that really happen?

MIKE: Both of you touched on something that I thought was really interesting. You both talked about silos, and interesting, Justin, you talked about silos and tech debt as if they were almost synonymous.

JUSTIN: Yeah.

MIKE: And that says something, that there's some aspect of tech debt that is centralization of knowledge. And, Matt, you talked about developing a technique of sharing. In software, and this has come up over and over again in our podcasts, that we sometimes think about software as coding, which it is, I mean, that's some aspect of building.

But [laughs] it's interesting, I think, in this call, most of us do not spend most of our time coding. In fact, I would think pretty strongly that, looking at who I'm talking with, none of us spend close to most of our time coding because software is about a lot more than just putting down those instructions for the computer, and it's mostly about that communication. And that needs...if that's not central, as you grow, like you say, you can get away with it. If you're small, you can get away with not having that as a key skill. But the more you grow, the more of those connections you have, the more vital that ability to communicate becomes.

And I just heard some suggestions about things that we can do. And that's probably where we're going to spend a lot of the rest of our discussion here is talking about tactics that we can use to deal with that. How do you deal with those communication challenges? Because you can't just scale up. It will slow things down. What can you do to reduce the costs of scaling so that you can maximize the speed of that while minimizing the impact to the software?

MATT: Yeah, I think one thing that is key, and we've been talking about it a lot lately is you need good leaders, and you need subject matter experts. If people are going to be moving in and out of your projects, there has to be a keystone to help guide those projects who has a broad understanding and can lead the people who don't have the knowledge that's required to see it through.

MIKE: So, you say you need to have some continuity, have somebody who understands it and can help people out and can and will, right? Not just can help people out but will. Speaking of will, Will just joined us [laughs].

JUSTIN: So, I've actually got to leave, but Will will take the torch and move it forward.

KYLE: I wanted to add to Matt's comment there. I think it's also, like, along with good leaders, we need to have leaders with bandwidth because if you have too many people trying to communicate with those leaders, then we still get congestion, and I've seen that across several organizations.

MATT: Yeah, I've been the bottleneck a number of times because of that, and that just goes back to that sharing of knowledge. The more people we can empower and the more we can share our knowledge with others, the better off we're all going to be. I think, especially early on in your career, it's a little bit harder for you to share that knowledge because of insecurities. You think, well, if everyone else knows this, what kind of value am I providing? And ultimately, the more you share and the more you empower others, the better off you're going to be, and that provides value.

WILL: And, honestly, I have to say, I mean, in terms of bandwidth constraints, and empowering others, and stuff like that, I mean, I think I came to working a straight job relatively late in my career. And I think, on one hand, like, okay, not great in terms of dealing with corporate politics and maybe expectations of existing in a broader ecosystem, you know, if my commentary in this podcast is any indication. I'm slowly but surely going up.

But one of the good things is bias towards action, and not necessarily any kind of preconceived notions about what I can and can't do and decisions that I can and can't make, where it's just like, oh, the manager hasn't signed off on this. Well, I'm just going to use my best judgment, and I'm just going to do what I think is right, and I'm just going to go. And that mentality isn't necessarily as widespread in corporate America as, I think, even management would like.

I have gotten very, very little pushback. I hesitate to think of a situation where my attitude is just like, I'm just going to do it, and I'm going to do the thing that I think is right. And if it winds up not being the right thing, then I'll just fix it. I can't think of a situation where I've gotten burned on that. But, if you had to come up in a situation where you're just on your own, figure it out; there's no air support; there's no cavalry to be called, so you’ve just got to do it. People don't necessarily think about that.

MATT: That's a great point. I think, most times, some decision is better than none at all because you can always course correct, right, and pivot if necessary. But if you don't make decisions, then nothing's done, and, ultimately, that's bad.

WILL: Yeah, and I think talking about The Mythical Man-Month, I think one of the...and I don't know the context, so I am just sort of stomping through like a child, but I warned you I had a bias towards action. One of the real big stumbling blocks is sort of, I suppose, even when you have leadership going out and making decisions, people being forced serial processes, like a forcing of a serial process rather than people just sort of like...I mean, I hate to use the word wild [inaudible 20:34] because I think that's slanted in the wrong way.

But, like, people just sort of like, well, what can I do to unblock myself to make some kind of progress despite the fact that I have these dependencies, so I will never finish without these dependencies? It's not possible. But what can I do now? And what corners can I cut to continue making progress in the face of constraints, which, you know, a little bit of a trust fall, where it's like, I think this API is coming. But in the meantime, I'm just going to have a fake API that says, “Everything's cool,” where I'm just going to have 200 bot just saying, “You're all right. Everything's cool,” you know.

MIKE: You just hit, I think, on something critical to getting out of some of this Mythical Man-Month problem. And it's come up before in the podcast more than once, the idea of a scaffolding that you're building a building, you're going to put some scaffolding up because you need that support to build it. And that scaffolding is not permanent. It’s going to have to come down sometimes, so it's throwaway work. But you build that scaffolding in order to work around the constraints that you have. You're going to have to build some support structure in order to build

And if you can decouple people to avoid that serial work, so if you can parallelize by having each of those do a little bit of throwaway work to put in some sort of stub, to build some scaffolding around themselves so they can work in an isolated environment, you enable independent work, and that is...the bane of progress is that serial work. If you have to do one thing, and then the next thing, and then the next thing, if you've got something that's going to take 10 years, well, it's going to take 10 years.

And anything you can do to parallelize is going to allow you to scale up quickly, and that only happens when you have teams that can work independently. And you just hit on a key aspect of that is giving teams the ability and incentive to build that scaffolding around themselves and work independently, build separate components that can then be connected once the other components are available.

WILL: I love scaffolding, and I love just doing whatever I want [laughter]. And --

MIKE: That doing whatever you want, too, nobody wants to be a code monkey, to be just a robot. You tell me what to do, and I'll do it. We function as humans when we have autonomy. We want to be able to build our own stuff, which goes to the same idea. If you can break things down into smaller components, then each group or individual has more ownership over their component. And then, they get to have the joy of building stuff, which is why a lot of us got into this in the first place.

WILL: Absolutely. [inaudible 23:32] I'm stirring the pot a little bit, but, I mean, in all honesty, I sincerely cannot think of an instance where a good faith effort, like, where it's just like, I am operating in good faith, and I'm trying to accomplish the goals of the organization, and I'm literally just going to do whatever the hell I want. And I'm going to be flexible if things don't work out my way, or I need to pivot, or I need to revise, or whatever. I can't think of a time it's ever burned me, not a single time, not ever.

MATT: I'm famous for doing this. Well, I'm not famous by any means anywhere [laughter]. But I have a tendency to do that. Often, I just do it on my own time, so I'm not wasting company dollars or whatever. But I genuinely want to see the company succeed because, ultimately, I succeed as well, and my peers succeed because of it. But taking action feels so good and just creating things is an incredible feeling. So, like you said, Will, I do the same thing. I will just get something, some idea in my head, and I will just run with it. And, oftentimes, it's not the right idea, and I've thrown away a lot of work because of it.

But I think the amount of work that I've kept around and that has ended up being used is well worth the efforts and worth the cost of the throwaway work. I do think...I'm not just going to create something and, without any oversight whatsoever, go ship it because that's a bad idea. But I think if you can share your vision with people, especially if you have a prototype proof of concept of something and can help people visualize what you're seeing, it stimulates others’ ideas as well. And they're just going to add to it and make it better, and that's how you grow and find success. I know this is a little tangent from The Mythical Man-Month, but I think that idea still applies.

MIKE: We talk about building a prototype, going out and building something [chuckles]. Throwaway small work is a whole lot less expensive than a major initiative that didn't have some testing first. That prototype, even if you're throwing it away...and you could probably throw away ten prototypes until you find the right one and save a whole bunch of money versus saying, “This is what we're going to go to market with. Let's spend the next 2 years on it,” and not doing that discovery process because, again, it's about scaling down, right? We're talking here about parallelizing work. It's the same idea as scaling down: breaking things up into small pieces that can be worked on independently and tested independently so that they can be brought together later to make a greater whole.

MATT: Yeah, and if you work that way, it becomes much, much easier to bring in people. The smaller the problem, the easier it is to solve and much easier to understand. So, if you do want to add people, it's a lot easier to add someone to broken down chunks of work than it is, hey, we want this monolith of a service built, so here, just go build it for us. You tell me you want to form with these fields on it; I can go build it. You tell me you want something that can intake an application, go underwrite, take payments and service; that's a whole different story, right? It's the elephant theory. If you can do things one bite or one chunk at a time, it makes it really easy to just hand off to others.

RAMSES: You still have the problem of needing to know the full story, though, right? You still need the context of the bigger picture. You can't just work on making isolation on your one component because you might [inaudible 28:08] to get it wrong.

MIKE: Well, that's a really interesting problem [laughs] because that's hard, right, to see the big picture, and there's a lot of challenges with that. So, that's another key aspect of parallelizing work is you have to have some people who understand the big picture, who are empowered to communicate that picture to others. Then, also it talks about how this work needs to be broken down. You have to have decoupled components that can be built independently. And you probably should start by defining what those interfaces are going to look like. What does the API between your components look like so that you don't have to do as much interaction, but you can work towards those APIs; you can code to the boundary and consider that good? Thoughts?

WILL: My biggest takeaway is maybe, as you were saying that, it caused me to reevaluate my sort of preconceived notions, right? I suppose, to some degree, I like to just sort of assume, well, everybody could just do that. Yeah, of course, I want to do this thing, and I'd be like, chopping up workloads, and it’s sort of maybe stepping back and saying, okay, okay, okay. And I’m like, I’ve been doing this for a while. I'm not bad at it. Am I overgeneralizing my own personal lived experience?

And so, maybe the question in my mind is like, how much of that can the average developer in an organization be relied upon to just do? Because I have seen certain types of organizations and certain types of engineers coming in, maybe not necessarily internal to the company, who probably shouldn't be making those decisions because they're trying to rip and run and might not necessarily have the long-term best interest of the project at heart. And so, on one hand, I'm just like, yeah, just go for it, yeah, YOLO.

MIKE: [laughs]

WILL: And on the other, I'm sort of like wait, wait, wait, wait, wait, not for everybody or some people, you know.

MIKE: So, you talked about they want to rip and run, you said, which suggests that there is some incentive structure within an organization that can actually cause bad behavior. If you're rewarding, let's just get out lines of code; for example, you're probably going to push the organization into making poor choices because you're not valuing the decoupling of components. You're using the wrong metric and rewarding the wrong behavior. So, if you want to scale up, you need to think structurally about what your organization is rewarding. Are you rewarding decoupled functional components that could be reused maybe in different contexts and that is a measure of success? Or are you measuring...banging out lines of code? And those are –-

WILL: You said banging out lines of code. I say rapid prototyping.

MIKE: [laughs]

WILL: In the end, it's like, rapid prototyping, move first, and break things, et cetera. You can only do that if there's a lane for you to clean up your messes later. And that's a completely separate issue, right, pretty far afield from the Mythical Man-Month. But, yeah, move fast and break stuff is...I think in a perfectly led and managed organization, the only metric would be get out a prototype, get out a functional prototype that works as fast as you possibly can. Make sure it's not going to endanger the entire business but some kind of [inaudible 32:32] goal. But get it out as fast as you can, validate it, and then build out the real thing, right? An ivory tower, perfect people, perfect decisions, best business outcome kind of a world. But people are messy; incentives are misaligned; maintenance and refactoring a rapid prototype that, like, hit. You got to hit. It's a winner.

MIKE: [laughs]

WILL: Okay, okay, okay, everybody, all right, let's put the brakes on this thing for three months while we refactor it, not to be doggy doo doo, right? That's a tough meeting to be in. That's a hard sell.

MIKE: It is a hard meeting to be in, and it's one that you're going to have to be in a lot because that's exactly how it works. You prototype, wow, this works great. This is enabling a whole bunch of revenue for the business. Matt, you might have encountered something like that recently [laughs].

MATT: I just went through this exercise.

MIKE: And is it ready for expanding to everything today? Oh, well, maybe not. That particular example can probably grow pretty quickly, but there are constraints [laughs].

MATT: Oh yes. “So, it'll be ready by next week, right?” [laughter] I got a lot of that as well.

MIKE: And communicating that is hard. And sometimes the answer is, well, we can use this in a limited scope while we, in parallel, are improving it and building the better thing. And it's painful to communicate, but you have to be honest and say, “Well, this can't do all of the things yet, but it lets you know what we can do in a relatively short amount of time.” We've come back to this a number of times in our podcast, the idea of building prototypes, building small, and failing early because it's a big deal. It's interesting that this comes up in the context of The Mythical Man-Month. We're saying that the only way to be able to scale up is to break down into pieces small enough that they can fail.

WILL: Sure, but that subdivision of the technical labor into small pieces that's really the art of software engineering, software technical leadership, like, real senior individual contributors. That's the real thing. How do you whack that thing up? If you want a pure rapid prototype, there is a lot to be said for just batching it up where it's just like, batch it up, batch it up. There is, I think, an intrinsic trade-off there in that the bigger your prototype is, the bigger the chunk of feature that you need to deliver to test, to validate your theory, to validate this product. The more the demands are going to be on, I'd say, the engineering leadership, the high-level architecture, the engineering managers, or the project managers. People like that coordinate this sophisticated operation.

If you're trying to just hammer out a really stupid, simple prototype, you could just have a couple of dudes kind of just banging things out. I have worked with developers who were not...it's going to sound like I'm shading them, but they're not. They're just sort of like streetwise, get-it-done type devs, and they're wonderfully effective. But they're not necessarily...they don't come up with...so, for what they're doing, they're very, very, very good at rapid prototyping, but they're not really great at multi-billion-dollar scalable systems, right?

And it works the other way, too, right? There's a lot of people who are great at writing very rock-solid scalable code, but they're not fast, right, like a short-order cook versus a Michelin-star restaurant guy. The Michelin-star restaurant guy is probably not going to enjoy the shift at the Waffle House, just slaying hash all night.

[laughter]

MIKE: If you ever watch somebody working a shift at the Waffle House, it's a work of art [laughs]. Amazing.

WILL: Yeah, they're sweating.

MIKE: [laughs]

WILL: And you can watch it, too, because, like, Waffle House it's a...I don't know who made this decision at the Waffle House, but it's a show kitchen. It's all right there.

MATT: Sometimes that's exactly what you need is just some greasy food, whether you love fine dining or not, every now and again, that's just what we want and what we need.

WILL: You had me at want, and then you lost me at need [laughs]. I don't think Waffle House has ever been a good decision for me. I've been there, and I'll be back, but [laughs]...

MIKE: So, I went on a bike ride with my oldest son some years back. Well, this happened maybe more than once, but we were about a hundred miles in. And we went by a McDonald's and got French fries. And I think we also got some sort of sweet drink, and I tell you: need [laughs]. That greasy garbage food was exactly what my body needed because it just needed a whole bunch of bursts of carbs, salt, and grease, right? Calories, carbs, salt; it's the perfect combination. I can't think of another circumstance where that would be an appropriate choice [laughs]. But there is sometimes, there is on occasion [laughs] the right moment.

WILL: Manual labor, like, a full day of manual labor, like, laying brick or, like, concrete, like something just godawful.

MIKE: [laughs]

WILL: Except you want the steak and eggs.

[laughter]

MIKE: Many years back, I used to work landscape. I’ll tell you, I had an appetite [laughs]. I could put down some food.

WILL: That's the truth.

MIKE: Which goes to the fact, sometimes you need that line cook, right? Sometimes you need somebody who can make the greasy food.

WILL: Yeah, I mean, I think a lot of it just really just boils down to that Mythical Man-Month stuff. Like, that's almost all leadership, almost all just people, organizational leadership, product leadership, I mean, really. Like, how do you scope it? How do you organize it? How do you break the thing out? How do you get a prototype through quickly and cleanly? It's...I don't know.

MIKE: Every really successful project I've taken part in started with a conversation between engineering and product, where you said, “What's the problem?” And then you spent probably hours banging away at it, saying, “Well, what are our constraints here? What's most important? What needs to be worked on first?” and, eventually, coming out with a set of milestones. This is what needs to be worked on first. This is what needs to be worked on second, and so on, that can be assigned to teams to work on and probably in parallel, to a degree, right? These are milestones. These are the parts that can be worked on in parallel. Here is our plan for work that can be split across teams and worked on.

And, a lot of times, I've seen the opposite in my career, which is we've got this thing to work on, and here's some requirements, and people just start banging away at it. And then, they reach the end, and pieces aren't matching together yet. And you spend the second half of your project time...right when you thought you were done, you start the second half: getting everything else working, and that first part, clearly defining what your milestones are.

And notice I said product and engineering together. You have to have somebody with the requirements, and you have to have somebody who knows how to build it, and you work together. And this may be more than two people; it often is, but a small group of people who work together to break that problem down and understand where it's going to go.

And when that has been done, those projects have pretty much exclusively been successful. And when that hasn't happened, the opposite has pretty much happened. And I think that it's inescapable. You have to have a map, and, somehow, that's really easy to miss in leadership. It's just you think, well, we're smart. We can do it, right? It's the guy in his car who says, “I don't need a map. I know where I'm going [chuckles].” It works for a while.

WILL: I feel like the biggest ask that I wish I could get out of leadership, and it seems to be like...well, I think I have some ideas around why it goes like this, but I wish there were more product leaders out there that were more disciplined about scope. That's the number one thing out of a product leader, right, where it's like, what's the smallest thing we could do to prove this concept? What's the smallest thing that we could ship? Because complexity doesn't...it's not a linear thing. It's exponential. It's an exponential curve. And it's tremendously that discipline and then the art, really, of saying, “No, this needs to be in there. This is critical”. That complexity is so cutting things out, cutting things out, cutting things out. It's so rare. It's so critical.

MIKE: We started talking about, and we still are, talking about The Mythical Man-Month and how to deal with these communication structures. And we're coming down to needing a product team to be able to define true minimum scope of work and a series of milestones to work on it, which is a strategy for parallelizing the work. Without that, you don't have a strategy, so your work's probably not going to happen very well in parallel. You're going to have just arbitrary work that will not align very well.

WILL: I think product-wise, it's just descoping it as far as you possibly can and no farther, right? And then, it's sort of like engineering-wise, that's our job. It's like, okay, how are we going to get the building constructed? Who's going to lay the concrete? Who's going to run the plumbing? That's us. You can't outsource that. That's engineering. If you don't know how to build a house, you're not going to be able to tell me when the drywall should come in.

MIKE: [laughs]

KYLE: So, I've been kind of thinking as we've been talking. I'm wondering, how far can we go with improving this process? So, that has drawn me to think about hardware. And if you take the human aspect out of the equation, I'm sitting here thinking and looking at hardware and CPU specifically. And when you're working on a task, some tasks, it's very good to just run on that one core: one thread, one core. Other tasks, many threads, many threads, many CPUs, many cores can do that.

Other times, you just start imposing latency between those threads when you start getting too much of that. And if this was a very easy solution, we wouldn't have as minimal core count as we still do to this day. It's quite a bit further along than we were several years ago. But we're still relatively low on our core count and our thread count. And even that industry is they're playing around with solutions. They're introducing higher caches, different layers of caches, different types of core. You've got performance cores, efficiency cores. Where do you spread the workload and hyper-threading? Some of the newer cores they're dropping hyper-threading, and hyper-threading has been a standard on the lower core counts.

And I’m just like, that is something where it's just, like, mechanically, they are equivalent. You don't really get better than that. And that process I guess the controller could be changed. But that process there you can't just keep throwing cores in there and expect to get improvements. It breaks down at some point. And so, we can't really expect...my thinking is, we can't really expect humans to get much better than that at some point.

WILL: I think it's directly analogous, like a parallel program because, as a programmer, that's kind of how I think about...honestly, that's in my mind. I think of it like a parallelly threaded program or an assembly line where I want to keep all the machines busy all the time. And I don't want to have too much work piling up. I don't want to have too much inventory or backlog or whatever or a thread where I don't want my threads blocked.

And exactly as we were talking about the short order cook programmer, which is disrespectful, but whatever, like, the rip and run, fast, efficient, but maybe less scalable solution and maybe less scalable developer that's, like, a single core. It's always more efficient to have everything running on a single core. It's more efficient of the CPU. The CPU will be more engaged and utilized.

KYLE: You don't have to cross-chatter.

WILL: Then the cross-chatter because they all have to synchronize, and they might have to block. And there might be messages passing back and forth, and all of these things add up overhead inevitably. It is always...I want to say always less efficient to have a parallel solution. And I will say it's always more technically challenging to have a parallel solution, always. It's always going to be harder and probably a lot harder. I've done a lot of parallel programming, more than most web devs do. And it's hard. It's tough. It's real tough. The technical bar is higher. The opportunities for bad things to happen are higher. It's just tough.

As people, as we're organizing things, I think your North Star should be what if we could just not? What if we could not do it? Because I am a really good high-performance, real-time, parallel programmer. And I am the first to pass that job off. I will dump that thing off so fast, and I'll be like, what if we just bought more servers?

MIKE: [laughs] Yep.

WILL: Can I spend my way out of this? And I think going back to some of these management things, what if you could not? Can we not? Is there a way that we could not have to do brain surgery and we could just do it the dumb way?

MIKE: Which is a different way of scaling and requires decoupling your work in a different way, independent units that can operate separately. If three work there, oh yeah, it's great. It's not efficient, but as long as I can have multiple units doing the same thing, it works.

WILL: So, we may be out of time. It might be just time to wrap it up. But one question in my mind is always, like, we're tackling and debating the easy part, right? Which is great, but The Mythical Man-Month really becomes a situation where, too late, then what? Because an ounce of prevention is about a pound of cure at least. But [inaudible 49:40] when we're talking about The Mythical Man-Month in those kinds of situations, something already went bad.

KYLE: And it’s already too late.

WILL: Yeah, too late. It’s like, you know what I mean, and that's --

MIKE: What you don't do is double the size of your team when your project is late because now you're onboarding all those people and spending all of the time and cost to build those communication structures instead of allowing the people who already have the context to finish it up. It's in The Mythical Man-Month. I was reading a summary before the call here. It says, puts it this way, "Adding manpower to a late software project makes it later."

WILL: I want to make one very important caveat, right? Very important caveat because, in my experience, when software projects go south, we're talking about sort of these blocker dependencies, and how do we get rid of those. Well, one thing that I have noticed a lot of the time is, you'll have a team, and they'll have these peripheral teams that are building parallel services, other dependencies. It's like, ah, I got to have this, and somebody from this team is going to do this.

And that is a situation when you can and should, and I would argue must, grab their liaison, if there's a liaison, where this team has a liaison in the other team, yeah, you bring them in. You bring them in full-time. You're working on this all the time now. This is all you do. And that has absolutely moved the needle.

And it's a frequent ask of mine to sort of higher levels of leadership. I'll go to a director or somebody who's over a bunch of teams and say, "I need a guy. I need a guy on the pricing team. I need a guy on the AI personalization team to be on my team, to be my embed and be my full-time asset, my full-time resource until we get this thing out," right? That is a situation where you'll have sort of, like, a fractional developer, and pulling them all the way in, that's a needle mover. That's a big needle mover.

KYLE: That's not necessarily, like, a new body, right? That's somebody that has some tribal knowledge, to some extent, that is able to jump in and contribute.

WILL: Maybe. But maybe they haven't started yet, and I need them on my thing now. They could be fully cold and to be like, this is a dependency, and I can't wait to get on your sprint plan and do it whenever. No, you're in here with me now. Pulling in those sort of resources that are blocked that's a big needle mover because inter-team communication it's a sticky point. We're talking about those parallel cores. That's that sort of memory buffer, where it's, like, I need this data in so that I can process it. That's how that happens, torturing that analogy just a little bit more.

MIKE: And you talk about torturing it; I don't know how much that's torturing it. We could say that even in hardware, you would see the exact same pattern to where it really is genuinely analogous. You have units that can do work, and the communication provides so much overhead that the more units you put in there, it becomes less and less effective until you're not getting any gains at all, which comes right back to where we started, which was, it takes time to communicate and to get people up to speed.

And you can't throw a bunch of people who are unfamiliar with the project at a project and expect it to go faster. It will go slower because you have to do all that work. The time to do that is, like, a year ago [laughs]. And if you haven't done that, then it's already too late. Finish what you've got and maybe start training gradually. Incrementally, increase the size of your team so that a year from now, you have been doing that all along, and you've reached the capacity that you'll need with the context you need.

WILL: How do you measure the most that a team can absorb? So, it's like, okay, let's say the real size of the team, that the true size of the team that needs to be...it needs to be double. It needs to be double. You're behind or whatever. How do you measure the rate that you can add people? How do you measure that?

MIKE: Boy, I’ll give you, like, some gut heuristics. That is about all I got. I don't think you could get more than about...the upper bound is probably somewhere close to 25% the size of your team because you get much more than that there's a lot of impact. It depends on how loaded you are right now. If you're pretty loaded, you're probably not going to get over about 25%. So, that's a rough number. If you reach your team by 50%, I think that's going to be a real challenge. And maybe you could do 30% if you've some strong people. But it's going to take you a while to get back up to full function. Now that's my gut sense. What do you all think?

WILL: So, what I'm trying to do is I'm trying to imagine somebody getting up to speed, where it's like, you're going to have some developer, and they're going to need some amount of time let's say a full dedicated pair, dedicated pair, some amount of time. And then, they'll be able to work independently with maybe a half-time pair, and then maybe a quarter-time pair, and then maybe they'll be, you know, asymptotically, they'll hit whatever level of independence which is never 100%. I'm not 100% independent.

So, they'll get to sort of, like, steady state, and then it's, like, how much time can you...and then, eventually, they'll be able to start training somebody else at some later period, right? And so, it's a question of how much capacity can you allocate to getting that done? And, like every developer...let's say every developer adds 5% management overhead, like, 10% management overhead, some amount. And, eventually, the management overhead is going to overwhelm the productivity wins because they'll be stuck. And so, those are the two factors. And I'm like, in my head, I'm trying to think of the equation, the series equation of, okay, how does this work out? Because you're right; I think if you double the team, you're probably going to need two teams.

And then, you're going to need some level of management because you have a manager for both teams and then some level of inter-process communication manager, where you would need a quarter of a director or something like that, some kind of thing, or a product manager. You're going to need something. And so, things will get bigger, and as things get bigger, the overhead kind of starts to get bigger until eventually you sort of clamp down, where if you add a developer, it gives you this much productivity and then takes away that much organizational overhead, and you're just treading water. You cannot do more. That's not a factor of, like, that's just a factor of people, organization, not ramp-up time, right?

MIKE: Right. And it only works if you are able to subdivide in some way to reduce some of that overhead within each part of the unit.

WILL: But, yeah, I don't know, I think maybe you could allocate 20% of your development effort to...I think you could take 20% of your engineering productivity and allocate it to getting the team bigger. I think that's, I don't know, it's kind of stinky from a statistical smell sort of a thing. And I'm like, okay, if I've got a team of four developers, I could have maybe one guy, one guy carrying out building out the fifth guy.

MIKE: Yeah. Well, you came to the same number I did [laughs].

KYLE: I kind of was thinking 10%-20% in my head, too, so right in that ballpark. But it's one of those things where I was hesitant to answer because it's that age-old question: what is a team size? If it's one person, that's a massive impact. If it's two people, it's still pretty big. At what point is it actually feasible to onboard somebody during a project release, you know, you've got a month to go. At what point? Is it four? Is it five? Is it six? Then you actually have somebody that's available to train while the other ones are getting the project done to actually make it beneficial.

WILL: Just going back to Matt and Mike's original statement, which is just sort of, like, at the point that you are adding developers, you have implicitly admitted that your deadline's hosed, right? You're either descoping, or you're pushing it out, right? That's it. It's like, oh okay, well, all right. So, if I'm going to take 20%, 20% and say, “Okay,” then it becomes a situation where you're going to have a certain amount of ramp-up time. And every developer is going to be different. Some people can just hit the ground running, and we're going to go. You're going to throw your strongest dev on there. They're probably going to take, like, a week, maybe.

And then, they're producing at 50% capacity, and then another week 75, and another week 100%, something like that. And so, I was like, okay, well, I take 20% of my productivity out, and then I could map this out. So, what you need to do is you need to be pushing your deadline out, so let's say the deadline goes out by 20%. And then, you need to have a developer that you could reasonably expect to be...how do I put this? If your timeline is infinite, the one-time cost of onboarding a dev kind of goes to zero.

MIKE: [inaudible 1:00:51]

WILL: And then, [inaudible 1:00:54], then that's bonus, 25% improvement, good to go. If it's a year out, probably negligible. If it's a week out, not worth it, right? You need to be able to amortize the one-time hit of adding a developer and onboarding them and have that be worth it, either because the project's going to be longer term, or we're going to bump out the deadline. Or if you want to keep the deadline, you're going to have to descope. There's just no other way, right?

MIKE: Right.

KYLE: I need to --

WILL: And I'm sure I can math this, but I would have to stop talking and draw on a piece of paper.

MIKE: Yeah, and you nailed the math, right? You're going to have to balance the hit versus the productivity you will eventually get. Do the math. There's maybe a little fun that you can, you know, eyeball it at the back of the napkin. Go ahead, Kyle.

WILL: Or, I mean, it would be –

KYLE: I was going to say, I'd have a hard time with all of the different variables because, even then, you know, we're like, okay, cool, it's 20%. But 20% of whose time and what? Because 20% of my productivity might be vastly different than 20% of Will's productivity, and then averaged across the team, each team is going to be different. So, you're going to have to figure this out on the fly on every project that you're on. And that seems, I don't know [laughs], that seems like it's getting very complicated to try and figure out.

WILL: So, it's just art, not a science, or not a science. Because another thing is, the first week, you can have your junior dev on the first week. He just needs to know where all the light switches are. That's all you need. But on their third week, that sort of maybe one day of pairs time, let's say, then they actually need the senior dev because now they're actually going down rabbit holes that they're getting. They need to make good architectural decisions. And maybe you only have one kind of tippy top dev that's like, okay, I know exactly where this thing is going.

And so, that impacts scalability because your engineering team lead, managers, by and large, should not be relied upon to make specific technical architectural decisions because they don't live in it all day. You could rely on them. Hopefully, they're not total idiots. But in terms of what's this class called? Where does this thing go? What does this API look like? Under the hood, how are we coding this thing up? What goes on the task queue? What are we running? It's just all kinds of stuff. That's the tech lead's problem, and they only have so much time. They only have so much time in the day, and they could [inaudible 1:03:40], right?

MIKE: So, I would suggest, to Kyle's point about it being hard, I think it's almost unwinnable. So, I think the best way to do it is you just say, you know what? I'm going to invest 20% indefinitely, and I'm going to realize it's not going to help this project. This is my long-term investment. It's the same as you're paying down the tech debt. You're paying down your tech debt. You're investing in your next generation of engineers, and you pay that 10%, 20%, whatever it is. You just have to have a constant background of somebody new on the team learning the system.

And if you want it growing, you have somebody on the team, not expecting them to help yet. You're doing that so that they will be able to help later. And if you're continually doing that, well, you got compound interest because as they become effective, you got your next 10%, and your next 10%, and your next 10%. You can get to that double size of the team maybe within a year.

WILL: I freaking love cross-training and cross-teams, where it's maybe you lend that 20%. Maybe you lend that guy to the other team, sort of your brother or a sister team, so there's like, uh-oh, we're in it. Can you jump in quick? Do you know our systems well enough so that you can jump on and move the needle for this thing that has gone pear-shaped on us? You could do it because you already did that.

And you have a ready-to-go developer that knows your stuff well enough so that their ramp-up time is a day or two, not a week or two, like a day or two. Because I know where the bodies are buried, and I can help you get on this thing where this deadline is screwed up. Plus, it's just fun. Cost sharing is fun. Business logic, I don't know, man, you have to continually keep yourself re-engaged in the work. And just doing the same old things the same old way, week in, week out it becomes a grind.

KYLE: Well, in cross-training, I realize this is a tangent, but in cross-training, it simplifies communication, too, because I realized today, somebody asked, "Well, do you guys have automation?" Automation what? What kind of automation? What are you talking about? To different teams, that's going to mean different things, right? There's several cases like this, whereas if I've cross-trained with a team, it’s like, that's the kind of automation that they're talking about, and we don't have to take time to figure out what it is they're communicating about.

WILL: Cross-training is great just because I keep a book of people that I have done a favor for, and when I need an answer, I can get one. And they'll just be like, oh, that's Will. Like, he's a bro. He's good for it. I'll help him out. Because if I don't know you and you're just coming up, I'll be like, “Go to the PM. Go to my manager,” because I don't know you, and I don't know if you're good for it. I'm not going to see you again. You just don't put a problem in my lap, and I got plenty of my own.

MIKE: [laughs]

WILL: And people are like that. If I know you, I got you. If I don't know you, then you got to go through channels, and going through channels that adds a day to your turnaround, every single time. I'm not sorry about it because there's a lot of people who will dump on you and give you back nothing. The overwhelming majority of help requests are people who came and did a drive-by, bang, bang, bang out the window. I've never heard from them before or since. And the next time I see their face, it's going to be the same thing. And I'm going to do it, but you're not getting the express lane. You got to take the [inaudible 01:07:37] with everybody else.

MIKE: So, you avoid getting entangled in other people's work and enforce that process so that we can actually have those boundaries. Because as we've talked about this whole time, those boundaries are how you parallelize and make things work, which is --

WILL: I'm not going to try and justify it [laughter] as anything other than selfishness. I'm not going to try and justify it. Because, in the end, nobody wakes up in the morning thinking like, I'm going to wreck Will's day. Something went wrong. Something went wrong, and the broader organization is responsible for it. And, I don't know, man. If I find out who's giving out my name like that, then that'll be a conversation to have but --

MIKE: I'll justify you on your behalf. I think that you are maintaining boundaries, and that's actually a valid and critical part to what we've been talking about. And maybe that's where we end. You have to have boundaries. You have to have boundaries in your organizational structure, and they don't just appear by magic. You have to plan and recognize that it takes time. It takes time to form these units. And if you expect to get anything done, you can't just tear that apart or throw a bunch more people at it, not going to work. You have to plan it out. And it's a problem that goes all the way down to the hardware.

WILL: Yep, yeah, and do favors, and make sure you keep a book of markers, man. Make sure you got favors you could call in. It moves the needle. It gets things done. Otherwise, it would be difficult [laughter]. [inaudible 01:09:29] a little horse trading.

MIKE: With that, thank you. I think that's a great place to stop, and until next time on the Acima Development Podcast.

This episode of the Acima Development Podcast dives deep into the art of effective meetings, exploring both what makes them successful and what causes them to fail. Mike opens with humorous anecdotes of ineffective meetings, such as “death by PowerPoint” and marathon discussions that inadvertently turned into workout sessions. These examples highlight the common pitfalls of meetings that lack focus, preparation, or structure. The episode aims to turn these common frustrations into learning points for hosting productive, engaging, and goal-oriented meetings.

Panelists Tad, Eddy, and Dave contribute insights into the key components of successful meetings. Tad emphasizes the importance of structure, such as having a clear agenda, timekeeping, and redirecting off-topic discussions to maintain focus. Eddy underscores the value of respecting participants' time, avoiding unnecessary tangents, and setting clear expectations for the meeting’s purpose. Dave provides a philosophical perspective, distinguishing between passive, reactive participation and proactive engagement, and stresses the importance of preparation and decision-making as central objectives of any meeting. The group also discusses strategies like giving everyone an active role, employing visual aids, and embracing rules like timeboxing to manage discussions effectively.

The conversation concludes with actionable takeaways for designing better meetings. Key recommendations include establishing rules of engagement, distributing materials beforehand to ensure participants come prepared, and maintaining focus through time management and participant roles. The panel advocates for meetings that prioritize purpose over duration, ensuring decisions are made efficiently. By adopting these strategies, the podcast argues, teams can transform meetings from dreaded time sinks into productive, collaborative sessions that respect everyone’s time and energy.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I am hosting again today. With me, I have a nice panel. We've got Dave.

DAVE: Howdy, Howdy.

MIKE: Ramses.

RAMSES: Howdy, Howdy.

MIKE: Eddy.

EDDY: Hey.

MIKE: And Tad.

TAD: Hi.

MIKE: Today, we are going to be talking about meetings, specifically about how to have effective meetings. And I'm going to present the topic on purpose because then I'm going to talk about the alternate. Because I think that if you're listening to this, and you're old enough to listen to a podcast, you have been in a horrible meeting. It was ineffective. It seems to be the default state of meetings because they are awful. They're famously bad.

I've been wondering about which particular example to bring up. I have a couple from my experience of bad meetings. Once somebody offered...when I was doing a presentation, they had a related presentation, and they had some slides for it. This was some time back. I was teaching a class, and they had some slides. And they said, “Well, do you want some slides for your class?” And I said...and I paused for a moment. And I thought, slides? And I asked him, “Have you heard of death by PowerPoint?” [laughs] And he said, “Yes, actually, I have.” And it surprised me that the question was even asked because it just sounded so opposite of what I thought an effective discussion would be made of. It’s the first example that came to mind.

I had another situation, this was quite a few years ago, where the team I was on discussed things so much. We had such long discussions about every architectural decision because we were building out a new application. It was hours. It was like six hours a day. And this was before people generally had cameras on. This is long enough ago, you know, video didn't always work, so it was audio meetings. And I couldn't stay awake. So, I started doing pull-ups and push-ups [laughs] during my meetings. And I put on 15 pounds of muscle from my meetings [laughs]. It was so bad that it actually improved my health [laughs].

TAD: Nice.

MIKE: So, I don't know whether to complain about that or not, but meetings can be that bad. And, on the flip side, when you're in a meeting that's good, that is on point, and you get your topic covered, and you end early, everybody is just like, “Wow, how did that happen? That was a good meeting.” And I think it stands out because it's so different from what we expect by default. It turns out meetings are hard. It's hard to have a good meeting. And so, we're going to talk at length about that today.

And I can give my ideas, but I've talked about some bad ones. I'm going to kind of step back and ask the panel here, you know, what do you think is important to have a good meeting? Or we can start off on the other track as well, you know, feel free to mention what you think makes for a bad meeting. I have some things that I believe are key to having a good and effective meeting, but I'm not going to throw those out first. I'm interested in what you all have to say, and then we'll build from there. So, what makes for a good meeting?

TAD: One of the reasons why this topic kind of struck me is because I've actually...and I haven't been a member recently, but I was a member of Toastmasters for about four years. And it's primarily known as a public speaking club. But they’ve kind of tried to change the branding a little bit of more just a leadership business public speaking. Like, there's a whole host of skills that they really are trying to promote. And one of them is specifically meeting management because every Toastmasters meeting they require that you have an agenda. They require that you know all the speakers, all the things that are going to happen beforehand, how long they're going to speak for, that sort of thing.

And as the person that runs the meeting, the success of your meeting is determined by did you hit all of the time points, and did you close the meeting out in an hour or less? And that's considered success, right? And they actually have someone in the meeting who's the timekeeper who the person running the meeting has to consult regularly like, okay, are we on time? Oh, did we fall behind? Okay, let me adjust. So, for me, it would be, do you have an agenda? Do you know what's going to happen in that meeting? Do you have a good idea of how long each piece of that meeting is going to take? And did you end on time?

And part of that is, also, a lot of times things will come up in the meeting, and you have to have a way to, like, you’re like, “Okay, that's a great point. That's off-topic.” What's the process to handle something like that, right? Okay, do you take a note? That will be an action item. We'll take that offline. We'll have a discussion about that later, that sort of thing, too. So, those are just a handful of elements that I think make for a good meeting.

MIKE: Okay. So, I think you've given a good list there. I'm going to come back to some of those as host because I think they're worth revisiting. What about others in the call here?

EDDY: I think one thing people need to realize when they set up meetings is that you need to respect other people's time, right? If you're going to ask someone to be there for 30 minutes and then you spend 20 of those minutes about random stuff that's not even part of the agenda or the topic, you're not being effective, and then you're actually causing people to not pay attention, right? So, I think what's really important is that you have to respect the fact that other people have other things to do and to stay on course. I think that's one of my biggest concerns when being part of meetings.

Another one that kind of just top of my head is timeboxing. It's something that we like to do in other meetings, right? Like, set things up in a certain window. If you think that it's going to go longer than that, axe it and then talk about that elsewhere. Again, it goes back to respecting people's time. I think that's really important.

And probably setting some expectations, like, some ground rules, right? Like, this is what this meeting is going to be about. Let's try to not go into tangents and stay on course. So, I think those three core things, for me, is what makes meetings important.

DAVE: When I think about good meetings, there's, like, the how, and the why, and those are two separate things in my mind. Like, the why, like, what is this meeting for, and, you know, why are we doing this? Most people come to a meeting in reactive, passive response mode. Just like, well, I'm just going to go to this room. I'm going to sit down, and then I'm going to see what happens. And then, I'm going to respond or react to it. And that's what most of us do in our meetings. And when you've got 99 people in a meeting, and only one person has a 5-minute agenda, it's going to be an hour-long meeting where it's just chit-chat, and socialize, and discussion.

There's a really good very, very short book. It used to be free. I don't know if it is anymore. It was paid for a while. It was, like, two bucks at the time. It's called “Read This Before Our Next Meeting.” And he takes a very aggressive stance. I mentioned it a couple of weeks ago in the podcast, a couple of episodes ago, that we do stand up with 30 people on Michael’s teams, and we're out in five minutes. And Will got mad.

He’s like, there's no way you're having...and I'm like, there's no way we're getting together, and discussing, and socializing, and screwing around and not having a meeting. You're right; we're not getting any of that done. But we are getting decisions made, and we are getting emergencies addressed, which was the whole point of that particular meeting.

The read this before the meeting guy...I don't want to do a whole book report, but he basically says, “Meetings are to make decisions.” Everyone in the meeting should come ready to make that decision, which means they should have the information they need. Now, we do an architecture meeting where we get everybody together. And we say, “Let me show you this system,” and that's kind of information dissemination. If we wanted to be really, really aggressive, we could say, “You know what? Why don't you go make a 20-minute video? And then, in architecture, we'll just show people where the link is, and they can go watch it on their own time. We don't have to have everybody here.”

I kind of like the socialization side of it. It's not worth, you know, like, really getting it after. But what I noticed is that if you go through any of the meeting etiquette books, they'll tell you: don't have status meetings; just stop. Because you should know what your team is doing. You should know what's going on, and if you don't, you're going to end up one person talking about “Well, I'm working on this,” and half the people in the room aren't on that team. They don't care. It's like Eddy said, “ Be respectful of people's time.”

And I think it's really, really interesting that if you...that's the how versus the why. If you come in ready to say, “We need to make a decision. Are we going to use this technology or that technology, or is there a third-party technology we need to be ready for?” That's a very, very active meeting at that point. You come in. You have an argument, you know, hopefully. Well, hopefully not.

But, I mean, like, if you're prepared to debate and push back, everybody's got their ammo loaded. Everyone's loaded for bear because they came ready to talk about this is the Elasticsearch problem, or this is the Sidekiq problem, or this is dah dah dah. And everybody comes ready and laser-focused; I think we should. And then, the objective is get to a decision. Everybody commits to the decision, and we walk out.

Not to tattle on ourselves, but, like, we've talked about dry monads versus dry transactions for like two years, and we haven't made a decision. And now they're both in the codebase because we have people that want to champion it. And that's fine. I mean, that's one of our functions. That's how we do things, and that's fine. But if we had wanted, if we needed a very strong discrimination of, like, “We want one, and we definitely want not the other.” We would send everybody, “Go study this, go study that, play with them both. Come armed and ready because when you walk out, you're going to be using one of these, and it might not be the one you like.” And that's a very active meeting. You come much more energized and ready to go.

And you can actually have...I've done this before where I've walked into a meeting, and we're all loaded for bear, and then, also, we find out every single person in the room already agrees. Like, there's literally nobody backing the other contender, and it's like, we're done. Let's just not have the rest of this meeting. And you kind of go from there. So, anyway, that's my excited thing.

I've got a bunch of, like, howsies, you know, like, functional things to do. But that was the big thing is, if you come ready and focused and there's an agenda in advance of, this is what we want to accomplish, and as soon as we accomplish it we're done, oh man, you motivate people by saying, "Would you like some of your life back?"

TAD: Yeah, I think I want to say Jeff Bezos does that at Amazon, where he requires you to have a full written up memo before the meeting of, what's the core discussion topic? Why is it important? What are the details of it? And if you show up and you haven't read that, he'll be like, “Okay, who hasn't read it? Oh, some of you haven't read it. Okay, we're going to take five minutes at the beginning of this meeting or whatever. And everyone has to read through the document and familiarize themselves with all of the things we're about to discuss.”

So, the idea being that you have to do your homework before the meeting or else what's the point of the meeting? Because I think I've been in meetings and you've all been in meetings where you get there, and you realize, oh, we've just got a bunch of questions. We've got a bunch of stuff we don't know. Okay, well, now it's time to assign homework to people and come back and have the real meeting later because this meeting was just a realization that we don't have anything prepared [laughs]. We don't know what's going on yet.

EDDY: So, you're saying that if a meeting leads to another meeting because that meeting wasn't productive [laughs], that's a problem.

TAD: Yeah.

DAVE: I've worked with a manager who took the extreme, draconian other end of that, which was “Has anybody not read this?” And, you know, somebody would be like, “Oh, I'm sorry, I need to catch up on this.” And he just canceled the meeting on the spot. And, like, “No, we need everybody in on this, and I'm not going to waste everybody else's time. We'll meet tomorrow.” And they came ready the next day. I mean, there were some scalded paws, and this particular person didn't mind conflict. He didn't mind upsetting people. But, boy, we all came ready the next day. And --

EDDY: And, again, you're setting up expectations, right? I think that's why it's important. Like, we come back to that again is that you're respecting people's time, right? You're setting up expectation of what that meeting is going to be about, right? And if you're not going...basically, if you arrive late, or you're not participating, you didn't do your homework, right, then you're not respecting other people. You're saying that your time is not worth --

DAVE: I wouldn't go so far as to call it disrespect. It absolutely can be considered disrespectful. But I think a lot of us don't realize that, oh, I’ve got this stupid meeting. Fine, I'll just go and find out what it's about, right? And you show up in reactive passive mode. And those are the same thing, right? If you willfully show up unprepared, you're a jerk, yeah, so...

Mike, you talked about being in six-hour meetings, and I had a thought about that, which was that was not a meeting. That was a mob programming design session, right? The objective was to come up with the design. What are we going to build? And if you tell people that's what this is going to be, it's much more palatable.

EDDY: You know, what I'm actually really interested in is kind of a tangent from that, is, if it really is true, what you said, and I'm hoping it's a bit of an exaggeration, Mike, but if you're in six-hour meetings a day, right, I for one struggle to pay attention for longer than, like, 15 minutes on something before I start wandering off and, you know, start doing stuff. So, I think I have some pretty gnarly ADHD, right, and if I'm not being focused, if you're not actively engaging me, I'm just going to start to drift, and then I'm not even going to listen to you anymore. So, I guess as we progress, I'm really curious to kind of dig your mind on how the heck.

MIKE: Well, I wasn't exaggerating so you know [laughs]. That was literally what was going on, and it went on for months, and it was brutal [laughs] for all the reasons you mentioned. That's why...and I'll actually say that...and I think we're going to talk about this in a future podcast. One thing that is important to realize is we do have finite ability to concentrate. And if you don't have some way to get through that, yeah, everybody's going to be disengaged [laughs].

TAD: Well, I think it's especially bad, well, you know, like going back to when I was team lead, I would go from meeting to meeting to meeting. And people would just put stuff on my calendar, and I'd just show up in a meeting. I'd just click next meeting, next meeting, next meeting, and they're back to back to back. And I didn't even know what the meeting was about or why I was in it. I'd just kind of be there.

But I think ours is especially tough because a lot of times our meetings are very complex, technical, think real hard kind of meetings. And those kind of meetings really drain your battery a lot more than just a regular meeting, right, like trying to work out very complex systems and systems interactions. I feel like we need a lot more gaps between those types of meetings just to let your brain decompress than just a regular business meeting, right?

DAVE: I've worked with people, and fortunately, not many. It was, like, 10% of the population are psychopaths that tend to be on that dark triad thing. And I think we've all probably had this experience where there's that one guy who wants you to come to his meeting because he wants to control your time. And so, he's not going to give you an agenda. He's not going to tell you what the objective is. He just wants you here.

And you can tell because he's the guy that when you start looking at your laptop and thinking, you know, I could probably kick off another build, he's the guy that will pounce on you and say, “What do you think?” and will deliberately not tell you what they were just talking about. That's somebody who's a predator at that point. We've all worked with jerks, but it gets weaponized. And if you have people that are reactive and passive, it snowballs, and those people end up running the company, which is why they do it.

MIKE: So, you touched a little bit on wrong [laughs] meeting management for people. Well, also, on something that tends to just really destroy meetings, which is when you have a single individual who takes control, and that can be the host.

DAVE: If there's an objective, yeah, let's have a leader. Absolutely.

MIKE: But there's a difference between being the host to get toward the objective and being a host because you're the focus of the meeting [laughs]. And those are two very different things, right? One is self-aggrandizing, and the other one is to try to facilitate. And we should recognize that facilitation is valuable. Self-aggrandizement is horribly disruptive to meetings; you know, generally, self-aggrandizing is not acting in good faith. They'll say something to draw attention to themselves without necessarily providing value.

DAVE: Which, again, that's the objective of the meeting, right, is, daddy, come watch me swing basically. Yeah.

EDDY: But what I feel like really works with me is to try to have a discipline to not multitask and do other things, right, when it directly doesn't involve me. If someone's going and talking about something else, their updates or whatever, you know, and I'm like, okay, cool, I don't need to listen. And I do [laughs]...I go, and I do another task, right, I end up losing all the context.

And then, when I get brought back in, I'm like, oh, crap. I'm like, now I don't know if I should ask them if they need to [laughs] summarize again because I wasn't listening. So, I hate to say that I know stuff when I don't. So, if I have to have them course correct and kind of summarize everything they talked about at that point, I feel really bad. So, I really do my best to, like, try to not do other things while other people are talking because I'm also trying to active listen.

TAD: So, you need to reduce the distractions. Is that the [inaudible 18:36] there?

EDDY: Because it's so easy, right? If someone else is doing something, I'm like, oh, they're just talking. I'm going to go ahead and check my build, or something. Oh, I'm going to queue up something. It's really easy [laughs] to --

DAVE: But it’s also, especially if you've got ADHD, it's impossible to track a meeting where there's nothing interesting, nothing novel, nothing challenging, nothing urgent. If somebody else is talking about their project and it's not even your team's interface, and you start wearing down. And especially if you’ve got ADHD, you're like, I need a rabbit to chase. And so, you start looking at things.

So, I agree with you that, as much as possible, it is courteous and powerful to be participating. So, I will do several...I have a bunch of tricks in my back pocket for that. Two very quickly are one is I will go look at...when I was a very short kid, I learned very quickly to never say, "I'm bored," because somebody would hand me some work if I did that. And so, I very quickly learned to entertain myself.

So, if I'm sitting in a meeting and we're talking about a thing and I'm like, I have absolutely no interest in this, but they're using that API. I'm going to go look at the API. So, now I'm doing a distraction, but I'm actually trying to contribute by, you know, I'm literally trying to invent questions for the test and go look up the answers for them. It's a great way to study, right?

The other one...and I think I've done this to all of everybody in this meeting, and it's not disrespectful. It is literally a coping strategy I use. And it takes guts the first time, and it's always kind of a risk. But if I get caught flat-footed, I don't hem and haw. I just go, “I'm sorry. I was looking at a thing on my laptop. Could you repeat the question, please?” Just own it. Because now what you're saying is I respect you enough to admit that I was distracted and that I wasn't paying attention, and you now have my full attention. And I apologize to everyone for making the question be repeated.

And I don't think I've ever had somebody say, "Fine, I don't want your answer," [laughs] and turn on. But if it was a thousand-person meeting, I wouldn't feel, actually, I would feel really, really bad, but I would agree it was an okay thing to do.

EDDY: Would you agree, Ramses?

RAMSES: Yeah, I think so.

EDDY: Okay, cool. He was paying attention [laughter].

TAD: So, a strategy that I use is I'll usually turn on subtitles or something. Because I'm like, oh, cool, like, if I'm listening but I'm not totally focused, oh, I need another level of stimuli to help me out here. And so I’ll turn on sub-styles, and I'll read what they're saying as they're saying it and that sort of thing. But I think that's also just probably a good meeting design is that you have lots of...it's not just droning for a PowerPoint. It's audio-visual discussion, variety, that sort of thing, to kind of break things up so that it's not as easy to just zone out in the meeting, right?

EDDY: Fun fact: part of the symptoms that I have with my ADHD is, if I'm watching a movie or a TV show, and I just try to hear what the dialogue is about, I can't distinguish between, like, sound effects and people talking. It gets mumbled [inaudible 21:57] and stuff like that. So, when I put subtitles on, it helps me keep focused, right? And it helps me be more engaged, and I can actually zone out all the rest of the noise because I'm able to correlate, you know, what they're saying versus what's being written on screen. So, subtitles, yeah, I should really --

DAVE: I love how fast live captioning went from being space race, futuristic, impossible. Like, in COVID, I don't think I had any meetings where I had live captioning. And now it's in Teams; it's in Google, their Hangouts, they'll do it. All these people will do it. And now we're all just sitting there, and we’re like, oh yeah, that's old hat. You need that. Turn that on. I love it.

EDDY: Yeah. But you also have to have it going live as you're in the meeting because if you try to retroactively go in and read the transcript, yeah no [laughs].

DAVE: I want to point out that I'm watching our live captions, and I said, “You need that. Turn that on.” And they transcribed it as turn that off. So, whoever's reading this podcast, pity!

[laughter]

MIKE: So, luckily, we have a professional transcriber [laughs], and not the digital version. I provide them with a digital transcript, mostly to see who's talking when so they can pick out the voices.

So, I'm going to come back and draw out some themes here. We’ve spent quite a bit of time talking here about engagement and how engagement is a challenge. And I think that goes back to the beginning of why many meetings are so bad is because it is hard to try to maintain focus for something that is not engaging you. And doing that for multiple hours is [laughs]...the thing is, you're not going to. So, you're either going to feel bad, or you're going to be working hard and trying to make it work. Or you're going to...you're going to fail somewhere, right? And somebody's going to call on you, and you're not going to know...like, there's all these problems that come from there.

We're talking there about meeting etiquette and how we stay engaged. But if you're thinking from the side of how to present, as mentioned, well, you need to have an engaging presentation that if you go in with an open-ended meeting where there's nothing visual, there's nothing audio, there's no agenda, what you're going to have is either people are going to chat. And if your goal is to socialize, well, maybe you've met your goal.

DAVE: Absolutely.

MIKE: And that has merit. So, if that's your purpose, fine. But, typically, you're trying to accomplish something else, and the engagement needs to be handled elsewhere and some proposals there. And I'm going to, again, I’m trying to pull all these threads together. You have an agenda. You have an agenda prepared beforehand and, you know, Tad started right off the bat with that, and not only you have agenda, but you have things time blocked. Time blocking has been mentioned several times.

Dave mentioned that that preparation is not just having, you know, like, the names, you know, these topics we're going to talk about but to have preparation that happens before the meeting. So, an effective meeting comes when informed people come together to make a decision. And if you don't have the information and you don't have the preparation, then you're going to be spending your time getting people up to speed instead and maybe recognizing that. Maybe you're going to have a meeting where you're having a training meeting. Well, that's, again, that's a different purpose because you can't have an effective meeting unless people are prepared. So, that's something else that's come up.

And then, another thing that I caught there that we haven't talked too much about yet is that topics almost always tend to explode. They go longer than you would expect. And when that happens, you need to have a mechanism, some sort of rules in place to say, “Okay, great, this sounds like this is bigger than what we can fit in this meeting. And we're going to ruin the rest of the meeting, unless we take that and give it its own place.” And you need to have rules in place, some sort of mechanism to be able to do that. That's another thing that I'm hearing here.

And, you know, so we gave a checklist. You have your agenda, and you have the time boxes. You have advanced preparation that people are able to take part in before they make it to the meeting, and you have a mechanism to take distractions and lay them somewhere else so they can go be handled elsewhere. Now, if it's truly tangential and has nothing to do with what you're talking about, then you just set it down like, well, we don’t need to...let's get back to the topic. Or maybe it's important, say, “Okay, let's note that. We’ll bring that up in a more appropriate form.”

DAVE: And it's consciously deliberately...because everyone in the room is thinking, oh, we're trying to get this done. It's good to say, “Okay, we are going to do that,” or “Let's do that later.” Or “Not now.” Everyone should see, okay, yeah. And the gears are still turning for the objective we want.

MIKE: Another theme here is we're engaging more than just audio, so you have some visual representation. If you’ve got your agenda, you table something. You put that on a list that everybody can see you're putting it on the list. This is an action item. We're taking this elsewhere. You're keeping people engaged in a multi-sensory fashion wherever possible.

TAD: One thing I touched on, but I feel like I maybe should delve into a little bit deeper is, in Toastmasters, everyone at that meeting has a role and has a job at that meeting. There's no passive anything, right? Meaning that there's the person who's in charge of the meeting. You also have someone who's a timer, who's keeping track of how long everything took and keeping the meeting on track. You'll have someone who's like an ah master who's listening for ums and ahs and hums, right?

And they're keeping a tally, and they're actively going to give a report at the end of the meeting. These are the things that I picked up, you know, terrible, little noise things we add to all of our speech that we're trying to reduce. These are the things I noticed in the meeting. Someone is doing grammar checking, and they're making notes on that thing. And so, everyone in that meeting has something they're actively doing in that meeting.

And just, in a general meeting, you could have someone who's like, okay, I'm in charge of timekeeping, making sure we stay on task. I'm in charge of the action items. I'm going to keep a running list of every action item that comes out of this meeting, right? You could give jobs to all the people in the meeting and have something for them to do other than just passively sit and listen in a meeting. And I find that to be fairly effective that if I have something that I know I'm supposed to be doing, then I'm more likely to be paying attention, especially to the role that I've got in that meeting.

EDDY: That's a good point.

MIKE: Well, and that goes to the engagement that we've talked about. If you're in a situation where maybe one person is going to be presenting, then everybody else is going to be likely to disengage unless they have an assignment.

TAD: Yeah. Well, in Toastmasters, they actually have a role called listener who will ask questions at the end, you know, ask a question and say, “Oh, you know, so and so talked about there are three things that do this. What are those three things?” And the rest of the people are supposed to say like, “Oh, number one was this, number two was this, number three was this.”

And so, you know all these roles. You know the things that are going to get reported on at the end of the meeting. You know what you're going to get asked about. And so, just all those mechanisms make it more engaging because you know you're going to get asked about things. You know there's going to be a report on, like, a little summary report on all these aspects at the end of the meeting.

MIKE: And it's interesting because that's a training for meetings, right?

TAD: Yes.

MIKE: But general training tries to follow best practices. This is the ideal for a meeting. And so, we can learn from that, that in meetings where we're probably not going to have somebody counting the number of times, the presenter says, “Um” in a business meeting, but that idea of a designated listener who, you know, maybe he’s not asking the group. Maybe you have a designated...sorry, a dedicated listener who will ask the presenter at the end, you know, somebody who's looking for holes. And that could be done if you could have more than one person assigned to do that. They could be given assignments. Potentially, everyone in the room could be given the assignment to have a list of questions to ask at the end. There's things that could be done in order to enable that.

TAD: I think so.

EDDY: How short should meetings be is my next question. Or rather, another way to twist it, how long should they be before you [laughs] say, okay, after X amount of time, no one's paying attention, and everyone's done, right? Considering the --

DAVE: It’s a good point.

TAD: I just think there's a problem in that all of our calendars have one-hour blocks. And so it seems like if I have a meeting, I need to fill that one-hour block. And I would love to instead see, oh, this is a 30-minute meeting. This is a 45-minute meeting. This is an hour meeting. And more than an hour maybe is this should be more than one meeting because I don't know that any time you go over an hour, people are really staying focused on that meeting. I think there's a limit to how long a meeting can go and still be a good meeting.

EDDY: I want to say a threshold is 30 minutes tops, but that's [laughs] just me.

TAD: 30 minutes?

EDDY: Yeah.

MIKE: Well, you could have a 30-minute threshold for a topic. If you had two extremely engaging topics, each went 30 minutes, that might be an engaging meeting. But if your 30-minute topic starts to climb into 60, 90, then you're probably in a lot of trouble. And it probably depends also on the level of engagement. I've been in architectural discussion meetings on a whiteboard before, where, you know, time flies and three hours are gone. And everybody is like, “Oh, wow, we need to go eat lunch. That was a while ago.” Because everybody is up there drawing pictures, you know, and involved in that discussion and deeply involved in those technical details.

And then, there's other ones where you have the presenter up on the stage, and everybody checks out after five minutes max [laughs]. And those one meeting works and the other one doesn't. So, it seems like it does depend somewhat on the level of engagement from the group.

EDDY: I've drawn some stuff on a screen, on a laptop. And I've been like, cool, this is awesome, and I get distracted. As opposed to if I get up, get on the whiteboard, basically get a sharpie, and start drawing stuff, I'm engaging the same way, I think, but I think part of that is just nostalgia, right? Maybe. But I do tend to pay more attention if it's in a whiteboard in front of me [laughs]. It could just be that I'm just old.

MIKE: Well, and thinking about the sensory aspect of it, if you're looking at a screen [laughs], yeah, you don't smell the marker, you know, you're not moving in the same way. It’s genuinely, I think, a different experience. A lot of people say they like paper books more than they like digital books. You're not seeing this, listener, but Dave is nodding wide.

DAVE: Oh yeah. Oh yeah.

MIKE: And, you know, I think that having that multi-sensory engagement, even if that small of a marker has nothing to do, and it's probably bad for you, right, has nothing to do with the meeting itself, but there is some engagement. And also standing up...and this is something that I wish was more socially acceptable, is to move in meetings because your brain just works better when you're moving.

TAD: Well, I've had stand-up meetings that are literally we all stand up and walk to the corner of the room and all talk to each other, and then we go back and sit again, right?

EDDY: A stand-up meeting was a stand-up meeting [inaudible 34:36]

TAD: Yeah, like you literally stand up, and you literally move to somewhere not right there at your desk and have the meeting, and then you walk back. And you're just standing the whole time, right? Because I think if you do that, they tend to go a little faster because people don't want to stand around so much [laughs].

DAVE: You don't want to stand, yeah.

MIKE: Hence the name, right? If you force people to stand, they don't want to do it, so it's going to be a short meeting.

DAVE: The counter is true as well, right? You get people that are like, I don't want to do anything in this meeting. I just want to kick back and sit. So, they're the ones that are saying, “I don't want to do stand up. I want to sit down for stand up.” And it’s like, well, it's because you want to check all the way out. You don't want to just not stand. You want to just, like, you know what, fine, you guys do your thing; it's fine, right? So, when you have a stand-up that is three minutes long, all of a sudden, anybody can focus for that long, right?

TAD: So, I hypothesized earlier that meetings expand to fill the time. Does that tend to be true?

MIKE: Yes. Almost always [chuckles].

DAVE: If everyone in the room is reactive and passive, your goal is to pass the time of the meeting. So, you literally have nothing to do, and nobody...especially if you've got like a little bit of organizational pressure, nobody's going to have the psychological safety to go, “Hey, guys, can we just quit? Can we just leave? Can we just be done?” Because there's going to be one person who's like, “Well, but I have a topic, right?” Like, if no one's engaged or if your goal at that...Sorry, I'm an extrovert. I think by talking, and I just hit on this. Thank you, Tad. If you don't have a reason to be there, you're reactive and you're passive, your meeting objective is now to get through the meeting. And you have no objective standard other than the clock.

MIKE: But, on the flip side, if you have a goal and you've accomplished that goal, that changes things. And it may even be a metric. Going back, also, to Tad's point about having an effective meeting, you have those time boxes. And you measure the effectiveness of the meeting based on how well those time boxes are honored. It's not about an arbitrary box. It seems like the goal there is to effectively contain those decisions. Well, let me say it differently; it's not to contain, but rather to engage. And if you're meeting your purpose, then you're done with that timebox, and you're ready to move on.

I would say that compressing the time is generally to be considered a success. One metric you could use of effective meeting, and I've often seen this, if you get out early, it's a good meeting because it meant that you were on point. You reached your objective, and you stopped. Your goal is not to fill that space but to solve a problem.

EDDY: Or the individual who tends to prolong the meetings is not there [laughs].

DAVE: And the counterpoint is true as well, right? Goodhart's law is that if the metric becomes the goal, it ceases to be a good metric. There will be that same psychopath that wants to drag the meeting and control everybody's time. If you tell that person we want to cut people off at time, they will eagerly do this, not because they want to save people time, but because they like cutting people's toes off. Great.

And as soon as that happens, everyone else in the room will say, “I'm not getting into that timebox because we lose toes in there.” That's Goodhart’s law writ large. All of a sudden, nobody wants to participate in that metric because it's a terrible goal. But if you have another goal, then it's a great metric for measuring, did you achieve it?

MIKE: But you keep on touching on something here, Dave, which is that an actor not in good faith can...and, you know, sometimes maybe they are even in good faith. They are earnest and just really care about something, and they want to talk about that, nothing else, that a single actor can easily destroy a meeting, and that's hard. That is hard as a meeting host. If you host a meeting where there's somebody who is very interested in getting what they have to say out there, and they're not going to let anybody else stop that, it is a challenge. What can we do to address that problem? Because you get into that situation, it's going to be very hard to have an effective meeting, ever.

DAVE: Stop inviting them to meetings [laughs]?

TAD: Robert's Rules of Order.

DAVE: We had a similar rule to that. It's related to this. It's not a direct answer to that, but we had a rule that we use called ELMO. It stands for Enough, Let's Move On. And, basically, if you've made a point and somebody else makes a point back at you and then you make your same point again and they make their same point back, there's nothing else. We're done. We're done. Let's move on, right? You can weaponize ELMO. I've had somebody shout ELMO at me who very clearly just wanted to hurt me and wanted to just be a jerk. But other people would do it, and it's like oh yeah, I need to focus because I know I get distracted. I get excited.

MIKE: One thing that got mentioned earlier was having rules of discussion, rules of engagement in your meeting. And if everybody has engaged in the rules before the meeting and has agreed to those rules, then it's not the host who is causing the trouble. It is the rules, right? The rules are what's ending this. And it's really nice to be able to point to the rules and say, okay, well, we decided ahead of time that we're going to timebox this for 15 minutes. We've reached the end. We're going to table this and bring it back another time. Now, you can't blame the host for following the rules.

The person who is disrupting agreed to those rules as terms of being in that meeting. So, they cannot, you know, it's not personal. You can't make it into a personal attack. They can ask to change the rules, but that's a group decision, and you have to get consensus. So, it seems we haven't talked very much about those rules of discussion, but they can help the host tremendously. And all of us are going host a meeting sometime, probably [chuckles] by giving control to the rules. And you think, well, no, I want to have control; no, you don't [laughs]. You want to be able to direct the conversation where you can, but you also want to be able to have that structure.

You want to have a constitutional meeting where you have a structure that defines the engagement, and your role is to facilitate within that structure. And you can lean on the structure for that rule of law rather than having to invent the law as you go. You cease to become the king. You don't have to have that on your shoulders. King or queen.

DAVE: It helps when everyone in the room feels like the rules are there to help us get to the thing. Then it becomes, like, if somebody slams the door on your toes, you realize, oh, I'm sliding out of the box that I was supposed to stay in, and we all need this box. One of the most powerful things that happened to me...I did Toastmasters. It's been 20 years, but I loved it.

And I remember watching the president of our club speaking something, and the timekeeper said, “You're out of time.” And he stopped mid-sentence. You're allowed two words when you go over time, and those words are “Thank you.” And he stopped mid-sentence and said, “Thank you,” and he sat down. And I'm like, “Wait, what? You just stopped?” And he’s like, “Yep.” And everybody followed that example after that, and we got out on time every time after that. It was amazing. Again, not because we like chopping people's toes off but because we like the utility that the box gives us. What is the meaning of this?

MIKE: If people can be bought into the system, then, you know, it's a group effort. It's a tool that people are using to accomplish their goals.

DAVE: I could do a whole nother hour on allegiance to systems, versus victimization, versus yeah...we have 3 minutes left in our meeting today, so I'm not going to get into that.

MIKE: We've covered a lot of ground here. This isn't one of our longer episodes, but it has covered some key points repeatedly. This is one I think I may revisit because there's some things that I can reuse here. I've definitely learned some things from the panel here today. We've talked about the rules of engagement and getting everybody working with those, right, using those as a tool, employing those to make the meeting effective, the whole group.

We've talked about engagement by giving people assignments, and we're there for a reason. Give people a purpose so that they are engaged, not to waste time but to actually provide utility. And if you can't give somebody an assignment, maybe they don't need to be there [laughs], and they can be excused. Or you need to change what your meeting is because you're not engaging people appropriately.

We've talked about timeboxing, making sure you have an agenda and a mechanism in place for being able to end topics when you reach those timeboxes so that they can be discussed elsewhere or can stop.

We've talked about some metrics for effective meetings. Ending early suggests that you are trying to solve problems. We've talked about having a discussion ahead of time, the pre-read being very important to enabling people to be able to come in and make an effective discussion and decision. I'm trying to make a summary here as we come to a landing. Anything that I missed?

DAVE: I don’t think [inaudible 44:15] good.

MIKE: We had a meeting with a purpose [chuckles]. We gave ourselves a timebox actually on this one, which we're going to hit in about two minutes, and we're going to nail it. We had advanced preparation by giving out the topic ahead of time enabling us to have a discussion. I think we nailed it. I think we did a good job here. Thank you. And until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, Mike kicks off the discussion with a story about his father’s woodworking journey, which leads him into a reflection on the craftsmanship seen in Michelangelo’s David. Drawing a parallel to software development, Mike explains how Uncle Bob’s philosophy about structuring code emphasizes that frameworks shouldn't dictate structure; rather, it should reflect the application’s purpose and business logic. The panel, comprising Eddy, Kyle, Ramses, and Will, dives into this idea by comparing software architecture to traditional architecture, stressing that applications should be designed to clearly convey their business domains rather than merely reflecting the framework they are built upon.

The conversation shifts to how different development roles approach structuring business logic. Will, primarily a mobile developer, shares his preference for keeping business logic close to the data layer, ensuring it remains adaptable and maintains integrity across varying platforms. Kyle adds a DevOps perspective, highlighting the value of configuration management and the necessity for disposability in infrastructure components, ensuring business logic stays modular and replicable without becoming burdensome or redundant. The team explores the tension between creating a reusable layer for business logic while managing caching, persistence, and the pitfalls of “fat controllers” in frameworks like Rails. Eddy brings up how MVC can sometimes obscure domain logic, advocating for clean architecture practices and encouraging separation of interface and business layers.

From a security angle, Justin emphasizes the importance of validation before any business logic is processed, advocating for clear layers to handle inputs and outputs independently from core logic. The team also discusses how integration and unit testing benefit from this layered approach, allowing developers to isolate and test business logic without impacting UI or data layers. They conclude that a middle layer for business logic—distinct from the UI and data layers—is crucial for maintainable, secure, and efficient code, reflecting that, much like in traditional craftsmanship, thoughtful structuring and separation of concerns lead to better long-term results in software.

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today. I’ve got a great panel with me here today. I've got Eddy, Kyle, Ramses, and Will. And, as usual, I'd like to start off with a story that's not about software development [laughs] and tie it in.

And I’ve mentioned my dad a couple of times in the podcast. I'm going to be again today. He's an interesting guy. Like, we all think our dads are interesting. Well, I think my dad is interesting, too [laughs]. Most of his career is building, building things, but he's also a tinkerer. He makes his own tools. So, he does woodworking, right? And he makes his own tools for woodworking, and woodworking ends up doing a lot of things. One of the things that woodworking led him into is, you know, making stuff, then he's done some sculpting.

Just in the last month, my mom and dad traveled to Italy and Greece. They hadn't done a whole lot of traveling. Did it there and went and saw some of the sculptures in Europe. And my dad said when he saw Michelangelo's David, it made him cry [laughs]. Because he's done sculpting professionally, you know, he's been building things his whole life. When he saw the craftspersonship, right, that went into that, it was just overwhelming. And, you know, it really affected him, which is pretty cool [laughs]. I haven't had that reaction seeing, well, I've never been there, so I really can't say. But, you know, when you devote yourself to making stuff so much for a long time, it, you know, it really affects you when you see something that's done well.

Now, we build software [laughs], and you don't see the structure very much. I heard an analogy made by Uncle Bob. Look him up. If you don't know what I'm talking about, he's a luminary in the software community. That's not his full name, but it's what he goes by. He’s one of the signers of the Agile Manifesto. He's a prominent guy. I heard a talk from him once, and he said that in software, we tend to structure our applications around the framework instead of structuring them around what they do, and that idea has really struck me. And he made the comparison to architecture.

I'm going to start with this, going back to this statue of David. You know, if you're going to make a big marble statue, you are constrained by the chunk of marble you start with. If you don't follow that shape, you're going to have some edges, right, where it does not work. And it drives everything that comes afterwards. You know, everything has to work around the constraints of that stone. And somehow, when we build software, we sometimes don't think that way. If we're writing a Rails app, we think, okay, I've got models, views, controllers. Everything should look like the framework.

And I'm glad we've got the panel we have here today because not all of us are, you know, we're coming from different kinds of frameworks, which is kind of cool. This idea, though, of structure coming from your framework rather than what your code is it's a big one because if you look at your code and say, okay, yeah, this is a Rails application. I've got models, views, and controllers. I have no idea what this application does. I have no idea what the business domains are. I don't know how to find anything [chuckles]. It's all just lumped in there together.

I'm going to take that further and say that if that's all you see, then there's something probably horribly wrong. Because, in any large application, you've got business logic that has nothing to do with presenting out to the customer. It's probably running in background jobs. It's running...you're probably sending out emails to people. You are, you know, writing reports, whatever the case may be, although you shouldn't write a lot of reports in your app. That's a bad idea. Use an outside tool [laughs]. If you’re writing reports, you're doing it wrong.

Anyway, but there's all these things other than just, you know, your web app, you know, you probably got a mobile app somewhere. You've got an API. You've got to talk to it, talk to that with. So, if all your business logic is structured around your display code, then you've got a serious problem because it's not reusable.

Going back to what Uncle Bob said, he used the example of not a sculpture but of architecture. He wasn't thinking about that marble you start with, and you get down to the sculpture. He was saying, well, if you look at a floor plan of a building, you know what that building is for. If you look at a school floor plan, you're going to look, and say, okay, here's the gym, here's the, you know, lunchroom, and here's all the classrooms. It's really obvious that it's functionally designed to match that. If you look at an office building and you see your cubicles, you know, and you see your meeting rooms, it's going to make sense.

You're not going to see every single building look exactly the same because we wouldn't do that because the building should follow the function. We have domains in our code. If you've got something that's, you know, a major component in your code that can make up a...if you've got blogging software, you're probably going to see something around leaving comments, right? And that's a major domain of your application. And if it doesn't look like that, then there's something wrong with how it's structured.

That idea of thinking about where your business logic goes, thinking about how your application is structured, I think, is a potent one. And that's our topic today. Where should you put your business logic? And we're going to talk about this in kind of a framework-agnostic way. Because I think these principles apply universally regardless of what framework we're working in.

I'm wondering...so, we've got some...so, Will, you're not at Acima. I'm curious what your thoughts. I'm going to start there before we, you know, kind of talk about the Rails side or some of the in-house stuff. I'm also curious what you think, Kyle, and what your thoughts are because you’re looking at this from a DevOps perspective. Where should your business logic go? How do you structure things so that you end up with your business domain and it’s recognizable? And you don't just look like, oh, this is a React app, and that's all it is. I have no idea what else it does.

WILL: I’d say one thing I am doing a lot of mobile development, you know, that's probably 80% of my work these days. And one thing about mobile app development is you can't roll it out with the speed that you can roll out a web app, right? I can have a web app. I could roll the whole server, I don't know, even a big server I can do, you know, in under an hour, you know, little stuff. You can do it in minutes, if not seconds.

So, one of the things that you really need to be mindful of as a sort of, like, a mobile app developer, is pushing business logic as close to the web server as you possibly can. Sometimes you got to do...I'll do in terms of the data, like, what am I going to render and when, where, why, and how? I always want to push that back onto the server because I can fix it. I can change it. I can get different stuff going. And so, you know, for a mobile app and for a web app, too, I really like to keep that view layer pretty, you know, I mean, in terms of logic, I want to do layout. And I want to do a last line of defense sort of, you know, sanity checking, right?

Because if something, if for whatever reason, I'm the last line of defense to keep something from being screwed up on the screen, that's what you want, right? That's your goal. That's what you want. But, you know, it's hard to hold that line because you're also sort of managing your L3, 4, 5, like, your last layer of caching, so there's performance considerations there. But, I mean, in terms of that, like, that's as far as I want to go. I want, like, the logic of what I'm being presented to be as far down as possible.

And, I mean, in terms of, you know, business logic, you know, once you start crossing over into web land or server land, I should say, you know, I guess what I'd say is I want to keep the view of what I'm thinking about in terms of logic as small as I possibly can. And maybe I think about it more from the other side because I pick up a lot of, you know, legacy, you know, projects. And so, when I'm trying to make sense of a legacy project, where do I sort of start wrapping my head around it? And I almost always go down to the database schema, like, as low as I possibly can in terms of the data, right? So, I'm looking at database schema. I'm looking at the downstream APIs that are feeding data up to me. It's like, that's how I start to untangle, you know, the logic around some other application.

So, for me, I want to keep it, I think, in general, I want to keep business logic as low as I can get it without duplication. It's like, how do you know, oh, I went too low, you know? Is when I start repeating myself, and I have different subsystems that are doing the same job over and over again. So, I think, you know, broad philosophy-wise. That's the best I've got for you.

MIKE: So, you've got a layer of business logic that lives close to the data but is not right there. It's a little bit above, so you don't have the same thing replicated everywhere or, you know, in all of your database table interaction probably using ORM. It's not duplicated in all those places. It's a layer above. So, it's reusable as widely as possible.

WILL: Exactly. Yeah. Yeah. Push it down. Push it down as far as you could push it until you start to repeat yourself, you start duplicating the same logic, the same functions, the same functional units over and over and over. And then, you know, and then you start bundling them into a module so that you can reuse it. And then, you know, you just kind of continue on as you are.

MIKE: I like that because there's a principle there because it applies both on, you know, you're in your mobile app. You're pushing it as close to the web app as possible which is, the web app is your data layer there, right, that, you know, you're pushing down to there. So, you've got this layer above your data layer for that site that has got your business logic. And then, on the web layer, you've got a similar structure. You want to push down as far as possible, as close to the data layer as you can, so that it's reusable. That sounds like something that can be replicated. Makes some sense.

So, Kyle, I said I'd ask you as well. So, I'm deliberately going through the non-web app route first before we get into the web app. So, I'm curious what your thoughts are here from the DevOps perspective. Because you write code. You write a lot of code, I'm sure, or at least work with configuration as, you know, as if it was code. What are your thoughts about business logic from a DevOps perspective and where that should go?

KYLE: I'm trying to think how this would relate in the DevOps world. And what's coming to mind is, specifically here at Acima, we deal with a lot of languages, just to start with. And it's kind of impossible for a small DevOps team to learn every language that all the engineers at Acima are dealing with. And just being able to go in and visualize to see, you know, what is this code doing is very helpful.

So, depending on that structure, which each language is going to have their nuances, if it's going to be in Python versus Ruby, you're going to visualize what's being presented to you in very competing ways just because of the way those languages decided to do things. But kind of like Will is pointing out, no matter what, a language generally has a data schema of some sort. And so, we can kind of see, you know, how is that logic put into a database? You know, how's that being stored, and where does that live?

And then, the other thing I was thinking about is where we offload the business logic. Where is this offloaded to that's, you know, can it be replicated? Is this in Redis, RDS, those types of tools? You know, does this get put into a data lake where it then gets replicated across? And kind of trying to think about that.

And then, the other thing I was thinking about, too, is at what point do you start making or putting your business logic into executable functions? And I might mean, you know, Lambdas or the like. When is your code, your app code, just your app, and then the business logic itself gets offloaded onto something else that's kind of executing it? Not necessarily something that is done everywhere.

I'm just saying, like, when you offload that work where it's constantly repeatable, constantly something that's a small unit that can be thrown away if need be and then rebuilt, and then, like, you're not, you know, necessarily relying on one framework or another, but you're relying on a small piece of code that you can just kind of get rid of and recreate if need be. Those are my initial thoughts on the subject.

MIKE: Let me dig a little bit more. I know [inaudible 14:28] he uses the analogy a lot of wanting to treat virtual servers, pods, you know, containers like cattle instead of pets.

KYLE: Yes.

MIKE: [laughs] And it's a vivid image there [laughs], you know, they're disposable, right? This is something that you're not going to have any qualms with getting rid of.

KYLE: Yes.

MIKE: And spinning back up. And so, I'm thinking you don't want to have little bespoke components that you have to rebuild every time for every new service that comes up. But rather you want to have a lot of reusable [inaudible 15:07] in your infrastructure config and building, rather than having this fragile thing where you have to build every one separately.

So, I would ask, you know, how do you go about doing that? Because that's logic in building something. You have reusable components for building the infrastructure out. How do you avoid that reuse or avoid the lack of reuse, right? How do you avoid the situation where you end up having to build everything alone? What does that look like in terms of how you structure what you set up?

KYLE: I'm thinking just everything is config. Everything is config management in that case, especially in the orchestration world. So, you're saving a copy in one place and able to replicate it in several others. And you're taking out those pieces that you aren't relying on that you're able to replicate. So, the server itself, you can replicate that as many times as you want. But the data set, which will probably be the business logic here, has to go into a database that doesn't necessarily...it can have replicas, but you're referencing one data set. And so, everything is kind of going down to that one pipeline.

And then, you know, depending on what these other services that you are actually utilizing, would depend on whether or not, you know, are you putting anything? Are you storing something in Redis per se? And is that something that can be thrown away? Because if we're not putting business logic in Redis, basically, if we can treat Redis as a cattle, now we can scale Redis out, and we get a lot more speed. However, if we're putting something that's a pet into Redis, well, now we've got to be careful with how we scale out Redis and how we worry about the HA availability of that. So, does that answer your question, what I'm going with?

MIKE: I think it's an interesting answer. You said a couple of things. You said that we need to structure our applications whenever possible to not have special needs. That is, we need to have data, a single source of truth but try to make everything else disposable. And if you're having a caching layer, try to make that disposable. Like, everything disposable if you can, and only have a single source of truth and [laughs] one and only and not any others because that's costly. That's an interesting idea in that it kind of reflects a little bit of what Will was saying that if you go and understand what the data looks like, it can help drive what comes above it.

WILL: Well, but you said...you made an interesting assumption, right? And as I was sort of, like, thinking about mobile apps, right? So, what I'm doing now is effectively just a glorified web app. I have my single source of truth, right, which is a server, and I'm presenting the data, you know, I'm giving data back. I'm sending it to the data...I'm sending it back and forth.

But that isn't actually how it came up. It came up doing, you know, like, a fitness app. So, specifically, I was doing a line of fitness apps, and they would generate data. They would generate speed data, location data, like, calorie burn data, heart rate data. All this data was like you had a mobile app, which was a lot like cattle in that I have a very capable, fully-featured computer that is reading data and processing it. It's doing its own thing. It's its own process that can do a lot of stuff, right? Easily comparable to one of many server threads generating data.

But it's also, like, it can do anything. It's just like, oh, I ran under a bridge. We don't have any cell phone service now. I ran out of battery, so this thing's done. The system's going down right now. I ran out of space. This thing just crashed. Oh, you got a virus, and it took down your whole phone. Apple, for whatever reason, didn't like your process anymore and decided to squash it. Anything can happen to these things. They could stop any time at all, for any reason. You know, like, oh, I put it on airplane mode. Oh, I was uploading this big data set, hmm, not no more. We're done now.

And then, you have to sort of catch that, and recover from it, and process it, and get the data back up. It's like, oh, I was sending that thing, but it didn't work out. Oh, and then then I ran out of space. So, I have to delete something. I’m like, they just deleted the whole app, and I had to recreate myself from whatever fragments of state I have available to me. The single source of truth, that's complicated, you know.

And if you're talking about your processes like cattle and not pets, it’s like, oh, well, that job? That job just failed, too big. I remember you know, scaling days at Acima where it was just like, sorry, guys. We made too much money today, and we literally can't count it all. And so, we uploaded these giant CSV files to the bank, like, all of our takings for the day. And I'm like, it will crash [laughs]. It will crash.

Kyle, you and I, like, I remember you and I were chasing memory leaks in this, you know, giant CSV bank transaction processing thing because we made too much money. We made too much money, you know. And then, we were sort of chasing it down. And so, you have to have sort of like a fault tolerance and checkpointing and, you know, callback steps where it's like, "Okay, did you get that?" "Yes, I got it." "Okay. I'm going to mark this job as finished," you know, and then, oh, is the job idempotent? Can I run the job twice? Oh, maybe not. You don't want to double charge. There's where the pigs get fat, hogs get slaughtered. I'm not trying to double my winnings. That would be bad. And so, when there isn't a single source of truth, that's when things start to get interesting.

MIKE: So, you’re saying that the structure of things, because we're talking about structure here, becomes challenging when your data is, well, that source of truth, you know, your data is not very reliable, and, well, anything is not reliable. That's interesting. You think about caching, and caching is a mess. Nobody wants to do caching. It's horrible. And it's also the only answer for a whole bunch of problems because of that same problem. There's so many ways, so many ways.

You can't get all the instructions to your processor. You can't get them in there fast enough. So, there is caching in there to get that push down, and you can't upload transactions fast enough sometimes. You're getting a stream of data from an app on the other side. You know, even within the company, right? They're sending you data as things are going on. And you could rely on them and send stuff, you know, make a request, like, a REST request every time. And now you've got a single point of failure, and every time they deploy, you go down, too. So, you've got to have that caching layer. You've got to have redundant data, which affects your structure a lot.

WILL: Dump it on the queue. You know what I mean? Dump it on your...I misremember the queue flavor, you know, the AWS queue flavor. The SN –

MIKE: SNS is the notification, and then you subscribe to it with SQS.

WILL: SQS, yeah. SQS or Kafka. Then what happens if you blow out your queue? Uh-oh. Some of these transactions they need to go out and persist to the database. But they don't just, you know, some of them you just persist it out to the database, no problem. But, again, we're talking about people getting paid. You want to commit that atomically. That goes in one time and only one time. And it only commits when all the [inaudible 23:38] are dotted, and ts are crossed, right? And that takes time, especially if you have multiple data centers.

Again, you don't want to just commit it to one database. You want to make sure everybody got it, and everybody knows they got it. And, like, if you get it, okay. If you get it, okay. And I'm waiting on you [laughter]. Uh-oh, this guy, he's not coming back. So, then I'm dumping my transaction. Transaction is over now. Oh, well, what do we do about that? Now I'm going into the naughty box, where we're going to have to come back to that. Oh, I'm going to retry. How many times do I retry, right? Oh, I've retried it three times. Like, okay, now I'm sending emails. Now pagers are going off, you know [laughter].

KYLE: Well, and it's how do you know when to retry, too? Because you're in the middle of processing a job, and you get a SIGTERM. Okay. Well, did your application handle that gracefully? Is it done? Did it finish? Do we retry now? There's so many cases there.

WILL: Do we all retry at the same time, right? Like, oh, I just lost a whole...I'm kind of falling behind. And so, then I have a big batch of jobs. Like, I was already not quite doing so good, and then I have a big batch of jobs. They all terminate at the same time. And now I'm just like, [inaudible 25:07] my server because all my retries hid in this giant tsunami, and now we've got pagers going off [laughs] once again.

MIKE: And that's a general, yeah, and that's not the only place that happens, you know, cache stampedes where you [chuckles] clear your cache, you know, it didn't come through, so I'm just going to wipe it. And a specific problem, I had a different company. We had a content management system, and we were getting a lot of requests, like, a lot, a lot of requests on the order of about a billion a month or something, you know, so a lot of requests coming in.

And, yeah, if you're trying to rebuild, the content for every one of those, oh, you will just die. And [chuckles] so, you have to have layer after layer of guarantees that, you know, when a request comes in, oh, wait, this content is stale. Well, I'm going to go build that in the background. I'm not going to serve you the stale content. And then, when that background content is ready, we're going to stick it in place so that the next person who comes in gets fresh content. But I can't fill up my caches; got finite space. So, I canceled everything forever. And [chuckles] it's messy. And if you ever just take the cache and expect that to rebuild, bad things happen.

KYLE: Well, and it's how much cache do you have, right? Because then it's do you go with a small amount of cache? And, okay, that helps, but a larger amount of cache would be better. But if we lose that large cache, I mean, we're in for even more of a world of hurt.

MIKE: [laughs] It's interesting we're talking so much about caching and messaging, you know, we're talking about structure and business logic. But we're saying, well, data is messy. And so, inevitably, a lot of what we build is going to be around dealing with imperfect persistence and, you know, imperfect connections, all of the weaknesses in the system because things are going to break. What about just your straight-up reusable business logic? Say you've got code that's going to take an input from your user and split it up into its parts and save it to the database.

Eddy and Ramses, I deliberately went through Will and Kyle first [laughs]. So, I'm going to come up to the web developers and Rails developers specifically. Rails gives you models, views, and controllers. Should you stick your business logic in one of those places?

EDDY: Just dump everything into your models and then just call it a day [laughs]. You can't go wrong with that. Or also, put everything in your controllers, too, just make them fat, you know, put everything there.

No [laughs], I think when you started mentioning domain business logic, I started to really write some notes down. And I think what it really comes down to is how large your project is, right? If it's like a simple blog, you know, things like that, I think you're inherently going to have your models and views and everything tightly coupled. And I think that's fine, you know, if it's something that doesn't have much strain in your service or whatever.

But think in terms, like, merchant portal. DHH, I think, is what we call it. It might be still sufficient, but you need to optimize it in such a way so that you're not putting much constraint in your project. So, I think one way to probably think about that is probably just extracting domain objects, things like that. You may want to mock or stub out tests, you know, just so that things are really simple to follow. And you're not waiting for builds and stuff. But still, you're going to run inherently into a problem where your controllers are really coupled with models and stuff.

And [chuckles] so, I think what we really want to consider is extracting data, right? And I ran into this issue. I ran into this terminology a while back called fat controllers, right? Has anyone heard of that term before?

MIKE: [inaudible 29:45] as bad thing [laughs].

EDDY: Yeah, it's kind of like in dumping ground, right, where you do a bunch of business logic controllers, but that's not really the design for it in MVC, right? Not to mention, I'm a huge subscriber to Singleton, so, like, single responsibility. I think that's really important to also implement into your service. Specifically, if you put stuff in your model, for example, the models, I believe, inherit from ActiveRecord. And they're going to be responsible for a bunch of domain logic and persistence. And I think that's bad, right? So, we want to avoid putting domain or, specifically, business logics into models [inaudible 30:35] controllers, right? Because you're asking for a really bad time. So, something to consider, I think, maybe PORO might be a good alternative, right?

MIKE: So, what you're saying is, don't put it quite in the data layer but have a layer right above that, where you have reusable components as close to your data layer but not quite there. Notice the pattern, which is what Will said should happen in both the mobile app and the web app is that you'd have a layer of business logics. Now, you had talked about having a plain Ruby object. And the specific kind of object really doesn't matter so much. So, you said, well, your business logic should be in its own reusable layer, close to but not quite the same thing as your data layer.

EDDY: Exactly, yep. You mentioned Uncle Bob's Clean Architecture [laughs], so I looked it up really quick. And something that really caught my eye was something...and I quote, “Software dependency should all point towards higher level business rules. The database, web, and other external interfaces does become the lowest level of implementation details.” And I think that's really key to what we're talking about, and I think it's kind of funny because Rails sort of doesn't follow this design, like, we usually think of Rails as, like, database design is probably considered first. And then, you have ActiveRecord models are created by one by one mappings [laughs]. And I think that sort of happens inevitably when you have a CRUD app, right?

MIKE: Yeah, so there's some pushback from him to rethink about that. Think about what your application looks like, possibly independently of what your database structure looks like, because they should not exactly align. You know, maybe you have a concept that doesn't exactly map to what you have in your database but makes sense as a concept. You can build around it, but it isn't necessarily a thing that maps to an exact table.

So, we've talked to, you know, somebody doing mobile development, somebody doing DevOps, somebody doing web development. And joining us a little late today, we've got Justin. Thanks for joining us, Justin [laughs].

JUSTIN: Better late than never.

MIKE: And I'm curious, so we've been talking about how we should structure an application so that the structure of the application reflects its purpose. And you're doing security, and I'm interested in your perspective on that. And I'm bouncing over you, Ramses, because Eddy spoke up first, but I'm still interested in your opinion if you've got one. I’m going to jump over to Justin because we're getting a wide variety of perspectives here. So, from a security’s perspective, what are your thoughts about the structure of applications? Specifically, where should business logic go and why?

JUSTIN: So, that's an interesting question because I have never thought of that from a security perspective. Is there a best place for it from a security perspective? If I'm looking at it the way that something is structured, I'm always interested in validating all the input, validating, you know, doing all the checks before any business logic actually gets executed.

So, you need to be able to do the authentication, do the authorization, you know, validate the inputs. And that's basically it, actually [laughs]. But you got to do those things first, and then you can do business logic. And then, you can, you know, then it gets stored away, you know, whatever happens, happens, the actions, whether it's storing it in a database or calling another third party, or anything like that.

Another thing that's interesting is the inputs and outputs are always interesting, to me, from a security perspective, because, you know, validate the inputs, validate the outputs. And do that for every layer or third party that you're doing, and that makes it a secure environment. I think there are several models that that could fit within, you know, classic MVC, or it should fit within every model, really. But those are my main concerns there from that point of view.

MIKE: Well, you said something really interesting there. You wanted to have a layer that handles your input, and that's not your business logic layer.

JUSTIN: Yeah, you got to validate that before it hits the business logic because if you have unvalidated data going through your business, it could hit all the branches, and it may be doing something that you may not want it to.

MIKE: So, from a security perspective, you're advocating for a structure where you have a well-defined layer that does input handling. You talked about Rails, you know, and you mentioned MVC with Rails MVC frameworks. We'll talk about that because we've done a lot with that. Your controller layer is positioned to do that input validation, and I would argue is badly, horribly positioned to do the business logic because it's designed to have that request come in, do the input validation, and, hey, are you logged in [chuckles]? Are you giving me valid data? Are you giving me something huge that I shouldn't be doing anything with? You know, whatever the case is that they're handing off. It's designed to do that.

But then, if it's doing that, thinking about it, well, that's a separate concern, right? That's a different thing. So, you should be handing it off to do something. So, listeners are not seeing visualization. I'm talking with my hands [laughs]. You've got a layer that speaks to the outside world and handles that. But then the business logic you're pushing back down to a deeper layer that sits in between that interface layer and the database. And this is interesting because we've got a recurring theme again [laughs].

So, Will talked about, well, I'm going to push it as close to the data layer as I can, but not so far that it starts to repeat itself, separating it from the UI concerns, right? Separating from those interface concerns. And he said the same thing about the web application. Well, you're saying the same thing from a security perspective.

And Eddy kind of said similar things from a usability perspective for a developer happiness perspective [laughter]. You don't want to have everything in a place that's not reusable. You want to have it in a reusable place. And all of these are kind of pointing to the same idea that you have this interface layer that handles those interface things. But that's a separate concern from your business logic. Your business logic should go somewhere else. It should go in this deeper layer that's not your data layer but is closer to it.

WILL: My original thesis, I still believe in strong, right? Which is that it should go, you know, as deep, as close to the data layer as you can get before you have duplication. So, form validation on the UI side actually sucks. It totally sucks from a software architecture thing because there's all kinds of stuff. You're, like, half-ass verifying, and you're half-ass validating, where it's like, I could check maybe a credit card CVC, or, you know, did you even fill this form in? I could do that stuff.

But then I got to do all the other stuff on the data layer, right? So, the credit card form. Like, I got to check this card. Is this card actually...is it valid? Is it even on, right? So, I got to do all this, you know, messy stuff. Like, it doesn't get me out of everything. I'd have to write it twice. And it's actually a giant pain in the ass to revalidate the thing just to be like, I went all the way down to, you know, calling the bank, you know, virtually. And the bank says, “Yo, no fricking way,” right? Then I have to parse that data and then update the UIs. But you want to do it because it's better UX, I mean, it's better user experience, right? You want to hit it as fast as you can.

But in terms of software architecture, awful, just terrible, terrible. Like, from my perspective, for my benefit only, do it all on the server side, all of it on the server. Because I got to do it that way anyway. So, like, why do it twice? Well, because it's not fun. So, anyway, I mean, it's just, you know, we sacrifice; we bleed --

EDDY: I would counter-argue that you do simple validations on the client side, and then do all of your logic validation in your backend; just saying. But I'm also –-

JUSTIN: But that's what he's saying, right?

WILL: Yeah.

EDDY: Oh, gotcha. I thought you said it in reverse. Okay, never mind. Never mind.

WILL: Yeah, well, yeah, you have to do it twice, right? I'm saying that it sucks to have to do it on the frontend, and then you have to do it all over on the backend. Because I don't trust anything coming from the user, right? And so, purely for my benefit, do it all on the backend, but it's just not good UX, right? Because you've got to wait for the round trip, and then the whole thing, and it's just, you know, it's a giant pain that nobody likes, you know, as a consumer.

MIKE: So, you just have to kind of forget about the fact that there's another layer that's doing all this somewhere else, and just live in your own layer [laughs]. The other layers don't exist. Then, you don't have to stress about it because you didn't write the validation again. Somebody else did. Except you probably did because you're probably doing both [laughs].

WILL: Probably, you're doing both.

MIKE: [laughs]

WILL: And, at the very least, you have to come back, and you have to parse out the, you know, this is the error, and communicate that, you know, even if it doesn't necessarily fit in your form anymore. Anyway [inaudible [40:55]

KYLE: I was thinking, when Justin was talking, something that we haven't necessarily hit on is actually the testing side of this. And it is one of those things where if you get your business logic too close to your UI level, or if you get it too close to your data level, your matrix on your testing scenarios gets a lot higher, and it gets a lot harder to do. And so, if you're able to isolate your business logic, you can mock and stub a lot of what you need in order to test just your business logic and validate that. And that would be helpful, you know, regardless of whether or not it's QA or security, I would imagine. But that just kind of occurred to me. If we treat it as that separate layer, then we can just focus on that, and curate that, and then curate the controllers and stuff that we're using independently.

EDDY: You know, in the past, I probably would have agreed with you cold-heartedly. But I, more often than not, have wished I had integration testing because it turned out I didn't truly understand, you know, what I was actually touching. So, there is a layer to both.

KYLE: Well...yes.

MIKE: They can both be true, though. They can both be true. You can have integration tests that do full end-to-end.

KYLE: Yes.

MIKE: And they take a hundred times as long. And then, you have your unit tests that don't have to touch the data layer and actually go fast. They're separate concerns that are both important. And if you intermingle those, you'll get in trouble because if you try to do all your tests with integration tests, yeah, they'll never finish, ever [laughter]. And –-

EDDY: Unit tests are fine. You better be damn sure you know all of your branches. Like, that's the only thing. And it's bit us more than once, you know, at Acima [laughs]. And so, all I'm saying is past Eddy would have said, "No, dude, mock, stub, everything. It's fine. You're good." But if you're not actually, like, making calls, you have to be 100% sure, you know, like, you understand how that works, what the response is, everything.

MIKE: So, you know, I think coming back to the theme we're coalescing around here, from a security perspective, you want to have this middle layer that's had stuff validated before it gets there, right? From a testing perspective, you want to have a business layer that's able to be tested independently that's not all entangled with your data layers. Your test runs way slower [laughs]. And I wasn't kidding with 100x. That's probably actually low [chuckles]. It is way slower to have tests that have to go through [inaudible 43:38], and then they're fragile.

And from a code usability perspective, you don't want to have everything. It all pushes towards...and then, going back to your block of marble, your beautiful block of marble, it gives you some constraints. You've got to have a layer in the middle. You've got to have a layer in the middle. And it's kind of inescapable that that business layer should be somewhere in the middle and not in your interface layer and not in your data layer, but in between. It's important.

JUSTIN: I hate to bring this up at the end of the podcast, but I'm just wondering, have anyone had experience with any of the cases where they store the business logic in the database as data and then they load it into the middle business layer that way? And if it's stored in data in the database, you can do various things with it. But that dream is always more complicated to implement than people would imagine.

MIKE: I talked to somebody who worked for a company that spent a couple of years building this beautiful system that did that until they went bankrupt because it took so long to build it they never made a penny [laughter].

JUSTIN: So, I worked at a place that tried to do that, and it was painful for everyone involved. But I'm just wondering if other people have run into a success story because that's the ideal, right, is you just define it with words in a database, and there you go.

WILL: I think in terms of the program logic and stuff like that, eh, I think what you can do is sort of...you can have metadata. And one of the things that I've done is I've had sort of metadata around a record, which could be, you know, fairly sophisticated, I mean, really. Like, I'm trying to think if I could characterize this easily.

So, I'll give you an example, right? So, I used to work on this series of fitness workout apps, right? So, everything was modeled as, you know, exercise. And, you know, it's like, okay, we're going to do chest exercise. We're going to do 20 reps, right? And it would coach you through that in a different way than it would be like, oh, we're going to do 20 minutes on the StairMaster, right, or something like that.

And so, like, the way it would communicate to you, the way it would log data, stuff like that, you could think of it like a style sheet, like a CSS style sheet. It would interpret things in this way or that way. It would do this thing in this way or that way. And so, it was metadata that did affect the way things are processed, but it wasn't really actual code. I think if you're writing actual code, I mean, because I've seen, you know, kind of PHP-type backends, where they're doing, you know, kind of weird stuff like that. Like, we have databases in databases in databases, you know, where everything is kind of Siamese twinned together.

And I find that stuff pretty...it's hard to work with. It's hard to think about. Sometimes there are sort of technical reasons, performance reasons, or sort of generality reasons that you would do something like that. But generally speaking, it's more of a means to an end to that goal, you know. I know if you go and you open up Core Data, which is, like, Apple's native mobile data layer, right? It's their version of, you know, SQL, or something like that, right? It shifts with all the iPhones. It's a SQLite database, and every row is the same, right? Every row contains all of the columns for all of the rows, right? And it'll populate, you know, these set of columns just based on the data type that it is, and that's just how they did it.

And I don't know the internal details around the technical basis for it, but I know some very smart people got paid a lot of money to write a hot, you know, SQLite interpreter for iOS. And I have every reason to believe they had good reasons for choosing what they chose. But if you ever look at it in a SQLite editor, it is just a mess [laughs], you know.

And so, I don't know, I try to keep it as dumb and as simple and as easy to read as I can, unless I can't, and then I do what I got to do.

EDDY: Well, I'd like to see that sometime.

WILL: No, you don't [laughter] [inaudible 48:34]

EDDY: Hey, just to be clear, I don't want to write it. I don't want to contribute to it, but I'd like to see it. It seems a bunch of spaghetti.

WILL: Well, I mean, it's exactly what you think. So, if you have, say, think of your random database schema, right? So, you have 10 different data types, right? Every row contains the columns for every data type and some metadata that says, "Oh, this is a product," right? So, it's going to have these columns in it, and they're going to be populated. Or, oh, it's a special sale, and it'll have these, you know, things. It's just a hash. But, I mean, you know, hey, whatever, man. That's the gig.

And they wrap it up in an only mildly hostile RM, like, you know, APIs, so that you don't have to know about that kind of stuff unless you really, really, really, hate, you know, yourself and what you were going to think about doing that day. Or, and this is what happened to me, something bad happened, and you need to figure out how to undo what you just did. Nobody --

JUSTIN: Hopefully, you can roll back that database update.

WILL: Exactly, exactly. Rolling back a database update that you screwed up in SQLite is...let's just say anybody who knows how to do that had a real bad day [laughter].

KYLE: [inaudible 49:59]

WILL: All right, well, I don't have anything to add. Should we wrap it up?

EDDY: Awesome. Oh yeah. This was a great conversation, like always. Super good. You guys are awesome. Thanks everyone for joining.

WILL: Thank you, gentlemen. Y'all have [inaudible 50:16]

EDDY: Thank you. Another great session of the Acima podcast. Bye, everyone.