Acima Development

In this episode of the Acima Developer Podcast, host Dave Brady and the panel, including Mike Challis, Kyle Archer, Eddy Lopez, and Will Archer, delve into a satirical discussion on how to ensure time estimates for software projects go wrong. They explore various humorous yet insightful approaches, such as using aggressive negotiation with engineers, ignoring edge cases, and focusing on points as extrinsic values rather than consistency. The conversation is full of playful examples, with stories from past experiences, all illustrating how poor communication, over-optimistic estimates, and leadership pressure can derail projects.

The team highlights how over-reliance on tools like Pivotal Tracker or Jira without considering the actual complexity of tasks can lead to skewed results. They also mock the practice of placing value on productivity metrics, like story points, while ignoring the consistency and accuracy of those numbers. Throughout the discussion, the panel pokes fun at the missteps that can occur when management overrides technical teams' decisions, fails to understand project requirements, or implements unrealistic planning methods, using examples from both personal and industry-wide experiences.

Towards the end, the group turns their satire to broader topics like testing practices, infrastructure management, and project launches. They humorously suggest that skipping QA, bolting on security post-launch, and involving high-level executives in day-to-day decisions can lead to even more dysfunction. Ultimately, the conversation serves as a playful critique of common issues in project management and development, with a clear undertone of how these behaviors should be avoided for the sake of successful project outcomes.

Transcript:

DAVE: Hello and welcome to the Acima Developer Podcast. I'm Dave Brady, and I'm hosting today. We've got a good panel today. We've got Mike Challis, Kyle Archer. We've got Eddy Lopez and Will Unverified. I don't think that's your...

WILL: It's Will Archer.

DAVE: It's Archer. Archer? Yeah. Welcome, Will. So, last week, we talked about some structure, and we thought we would carry over, and I think we're going to table that for a week. Today we came up with, actually, it was Will. You came up with a really interesting question phrased in kind of a devil's advocacy. Do you want to phrase that up for us and tee us up?

WILL: Well, so, if I wanted the worst possible time estimate for my project, how could I do it? How could I really screw up my time estimates?

DAVE: I like it. I may or may not have some stories related to this. I'm seeing knowing smiles around the board. Okay, so I'll give two stories really, really quick. And the second one is actually the counter to the first one.

Negotiate aggressively with your subject matter experts in a threatening manner. Specifically, negotiate story points down because you don't want to spend extra time dealing with the problems that your people are thinking of that might be coming down the pipe. And a great way to do it and sound like you're not doing it is to begin every sentence with, "Why don't you just...?"

And so, you can take something really super-duper complicated...I mean, throwing the ring into Mount Doom that's a one-point story. How long does it take you to drop a piece of jewelry into lava [laughter]? It's super easy, right? Super-duper easy. Why don't you just do that? It's fine. And yeah, it's that kind of thing, right? Where the actual thing, you know, getting to the site to deliver it, you know, OSHA injuries where he gets his finger bitten off, right? That whole thing.

WILL: Take the Eagles, guys! It's [crosstalk 02:00] a one-hour flight. Take the Eagles, duh.

DAVE: Yes.

EDDY: I was going to say, don't you like the idea of having an extra cushion to your points?

DAVE: [inaudible 02:08]

EDDY: Yeah, because if you over-deliver and you get it done, like, two or three weeks ahead of time, the only thing that you get is just applause, right? Like, you got it done earlier.

DAVE: Well, so, I think this is kind of the other way around, right, where it's like dropping this in. We're going to budget two hours for this, which is plenty of time to drop a piece of jewelry into lava. And the project, as we know, is going to end up taking three months from, you know, hindsight. And that's the kind of stuff where...like, engineers often get paid to think about edge cases, and weird things, and, you know, what about this thing, and da da da da da.

And very, very smart people who are low in trait openness...openness is one of the big five psychological traits for gauging a personality. People who are low on this are very certain they're the J type on the Myers Briggs. They're judgy, not perceptive. What's in their head, they will press that down on the world around them.

And this leader that I worked with would negotiate aggressively, and because he was the CTO, every discussion was an uphill push, so, okay. And we used Pivotal Tracker, and Pivotal didn't care. Tracker doesn't care what you think your velocity is. It will tell you what your velocity is. And because the stories were getting smaller and smaller numbers on them, all of Lord of the Rings became a one-point story, which means it took us three months to deliver one point of value.

MIKE: Wow.

DAVE: And he did not like that. We literally watched Tracker go sub-integer. We had a velocity of 0.2 at one point. And that did not make the CTO happy either because he wanted the points to be big, which might be another evil thing is, like, place extrinsic value on points.

The counter to it is the second story, which I'll just tell in just one quick sentence. I had a manager who looked at me, and he said, "I don't care what your velocity is. I don't care what the number is. If it's consistent, I can bank on it." If the team velocity is 180, that's great. If it's accurate, I'm going to ship on time, and I'm happy. If the team velocity is 1.8, that's fine. Points don't matter, remember? And if I can bank on you shipping on this day, and I can queue up the next thing, I'm happy." So, if you want to mess up your schedule, focus on the points themselves as extrinsic things, and ignore consistency in your estimates.

EDDY: So, Dave, for the people that may not be aware, what's Tracker? Can you give an overview?

DAVE: Pivotal is a company that makes software, and Tracker is their time-tracking thing. It's Jira, or VersionOne, or Rally. These are all competitors. Jira is the really big gorilla kind of taking over the scene. Basically, you do this, deliver it, and then it goes into the backlog, and you put points on it. And then, Tracker gives you your to-do list against a calendar. It's like, no, no, this is your...I've been measuring your velocity. This is how many stories you have. This is when you're going to be done. And after you've been using it for about three or four months, it starts getting spookily accurate. And if you don't like what it says, you can change the numbers to be wrong.

MIKE: I had a couple of stories I was thinking of as you were sharing them [chuckles], but I'm going to go a different way. I mean, we were talking in the pre-call about the first line in Anna Karenina, which is all happy families resemble one another. Every unhappy family is unhappy in its own way. Like, there's many ways to ruin your estimate [laughs].

And I saw two kind of opposing examples. In one, product came to our team, and they had a big, international expansion project, and they had been burned by some optimistic estimates in the past. I said, "Okay, what's the worst case that can happen?" And so, the team talked about all of the worst cases and all of the unknowns we had [chuckles] and everything that could go wrong.

And we got an estimate that it was going to be this huge project that was going to take, I don't know, like, eight months or something. And then, the estimate was so big that management decided to...they said, "Oh, that's just...that's crazy." And they quit trusting our software. And this is, like, new management at the time. And said, "We're just going to outsource this." And when they looked at what was going to be involved in outsourcing and how expensive it was, we said, "Well, no, we can actually do it a lot faster than that." Ended up actually getting the project done in 2 or 3 months, way below what the previous estimates were.

But, you know [chuckles], there was, like, a year that that project didn't get worked on because nobody wanted to invest that huge amount. And engineers actually didn't even know that that was going on. We just did what we were asked to do. It was the worst case, so we gave it to you. And, of course, the worst case is not the average case. It's going to be some of the cases. So, you want to probably aim high, but that one failed.

And then, on the flip side, we did some annual planning and didn't have a whole lot of information. So, we planned out a lot of stuff quickly [chuckles], and then somebody went through it and said, "No, I don't think it actually will take that long." And they just chopped the number of our estimates in half [laughs] without any input whatsoever. And not surprisingly, a project that had that happened to has gone about double budget because when that project finally finished, it was about double budget because the earlier estimates were actually pretty good because we compared it with previous work and matched. It can go either way. If you don't trust the past experience, a similar future experience, then you're probably going to have some bad, bad, bad numbers.

DAVE: There's another symptom in what you said that I think is really interesting, which was management sent people off to go make a decision. They didn't have a discussion. They said, "Go away, discuss and decide, and then give us the decision." And then, out of context with no dependencies, or corollaries, or caveats, they're just like, "Here's the number." And because they have completely different assumptions, they made a completely different decision than they might have had they been in the room saying, "Well, what about this? Well, what if we cut this feature, or what if we brought in contractors, or what if..." you know, what if. There's no what if. It's just, this is how it will be, and it's unqualified. This is how it will always be in every universe.

MIKE: So, first key to give a really bad estimate is give numbers with no context whatsoever [laughs]?

DAVE: Mm-hmm. Mm-hmm.

EDDY: Well, I do want to say, just because the idea sounds easy doesn't mean the implementation will be.

MIKE: Well, that's an interesting one as well. Have you all ever been in a situation where product did the estimate and then told you what the estimate was?

WILL: I tell you, man, like, I think a really good idea to get these estimates really messed up...so, what you got to do, right? Engineering is so expensive, right? It's really expensive. And so, really, you got to have a business case for it. You got to have a business case for it, and this case is dependent on impact and timelines, how much money.

And so, really what you got to do is you got to go to the very highest level, like, the business folks with the spreadsheets. And you got to present your case for your engineering idea, your engineering idea to them, right? And you really got to get them, like, you have to get them aligned with sort of an outcome before you even talk to engineering because, like, you know what I mean, it's such a...you got to justify this line item expense, right?

So, you really got to go from the business case, top, top, top-level approval of budgets, and then you go down to engineering. Because, as we know, if something went wrong, it's always easier to piss off the C-level executive because they're really, I mean, they're disciplined, emotionally intelligent. They know all of that stuff. And it's easier to piss off your CEO, who maybe made some promises, even to the board, to the public markets, to all that stuff, rather than line-level engineers that might know things that you don't know. Like, as high as you can go, that is just absolutely critical.

DAVE: There's a phrase that weirdly has come up, like, three times in conversations for me this week. And so, this is number four, which is people are sometimes blind to the fact that if you have a problem and you start percolating it up the chain, your team lead can't help you with it. So, you go to the, you know, the department head, and they can't help you with it. So, you go to the engineering.

Once you're about three levels up, it is very, very hard for that person to solve your problem. It is much easier for them to solve you. And it's the same kind of thing, right? It's like, if I go to the CEO of Rent-A-Center and say, "I need this," he's going to say, "Stop needing that. Go away," right? Because that's the fastest way to solve that, especially if he's got five layers of people that he trusts, which you're going to do if you're in the C-suite. So, whether they're trustworthy or not, you've built them to trust. So, yeah.

MIKE: Will touched on something, the person on the ground who's familiar with the system. A great way to have a really out-of-whack estimate is to completely ignore what they say [laughs]. It's very easy if you're in a leadership position to say, "Well, I know what I'm talking about." You get inflated ego. It's a malady that has been present throughout history [laughs] that people will think, if I'm in charge, that means I know what I'm talking about. And a great way to get to get the estimate that you want to hear is to ignore what people are telling you and just don't even ask, right? Like, well, you know, this should be [chuckles], take the eagles, right? There's an easy way to get to Mordor.

DAVE: The thing that I'm hearing a lot is, like, committing to the map instead of to the terrain. We see this a lot. There's a specific case that I will tell people. Sorry, I need to phrase this in the evil way, right? You should make your decisions as early as possible and stick to them because, later on, you're going to get more information, and that's just going to annoy you. So, make your decisions at the point of minimum information. Never, ever, ever defer a decision, even if making it now would be hard. That's a good one.

WILL: Ooh, I got one, just to pile on. Hard launch everything. It's so cool. You have this grand unveiling, right? I mean, it's like you're showing somebody a finished product. They can get excited rather than a bunch of Tiki-tac incremental stuff you can't hardly do a press release on it. Hard launch, always hard, hard, hard launch, big unveiling. Think Apple, you know what I mean? Like, just the biggest possible spectacle. That's the way to do it.

MIKE: Nailed it. Nailed it. And let me take that a step further. You know you're going to have that big launch. You better plan everything you're going to launch on day one. Make sure you have all of your plans up front.

DAVE: Waterfall. Yep.

MIKE: Plan that --

DAVE: Don't start designing anything until you have planned everything.

MIKE: Everything. Yeah. Absolutely. And follow that plan to the T. Bonus points if you can get out ahead, you know, that means that you're really good at planning. So, if you've got, like, a two-year plan, make sure to do all that planning upfront.

WILL: And the people who did the plan should really be in charge of the estimates, too. And when you give those estimates, right? So, you lay out, this is the product; this is the timeline; this is our hard launch. Get all that stuff done. And when you present it, you should present it with a timeline, right? And this is really important, right? It's like the intuitive mind, right? When you present the thing, you want people's gut instinctual reaction, preferably in a public meeting with everybody there.

So, you just want to get people right around the table. And you want to be, like, you just want to shoot from the hip, boom. First thought, best thought. You want that creativity. And you want people...like, the accountability is critical, right? So, you got to make sure that you say like, "I think this is two months. What do you think?" so that people feel empowered to speak up in front of their peers and say, "I'm going to disappoint you and your boss. I'm not going to be pressured. I'm going to give you the real estimate because this is an art, not a science." You got to let people just go with their feelings. Let it flow.

MIKE: The more public, the better, right?

WILL: Absolutely. Absolutely. It's their time to shine, right? So, the bigger the hats you can get in the room, you know, get a CTO in there. Get some VPs in there, really, like, you know, elevate your team so that they can show what they're capable of because everybody wants to move up. And you don't get to see these guys all the time, right? So, they can really get a feeling for your ability to say, "Yes [laughs]."

DAVE: 100%. I had a scrum master that I worked for 15 years ago who loved that step 11 is the commitment. The team must commit. And we had all these blank checks. We had tickets that had been pointed that literally were, "Click to add description." Stop me when you've heard me bang on about that one, right? And it was like, no, we've got this many points in the sprint. Does the team commit to this? And I said, "No. The team commits to work as hard as we can and give it our best level effort, but this is the plan, and it's not going to go that way."

And I got a lot of pushback. So, make sure you push back on the people like that. Call them disloyal. Make sure that they do not feel psychologically safe pushing back on that [laughter], and whatever you do, never, ever, ever consider the possibility that you have not refined tickets well enough to commit to making it work.

EDDY: I do want to say that some of the things that I've noticed in my two years or so is that it's really easy to overestimate how fast a project is going to get completed based on the number of cooks you have cooking, right? Because you could add more talent, but you spend more time bringing that talent up to speed than it would take to have people who already know the context to just do it themselves. So, in a sense, sometimes it slows down the process by adding more people who don't have the context to do it. So, from experience, anyways, it's like, help is great, but you got to be smart about it.

MIKE: Well, no, throw more people at it. It'll always speed it up, right? Like, a good example is pregnancy.

DAVE: Brook's law.

MIKE: Nine women, you can have a baby in one month.

DAVE: One month. There was...I worked at...

EDDY: I think someone told me, as a response, sorry, Dave. But I think someone told me --

DAVE: It's okay.

EDDY: They're like, "Oh yeah, you can't have nine women cook one baby quicker." And I'm like, okay, yeah, but the retort was, "Yeah, but what if you impregnate all nine women? Suddenly, now you have [laughs]" –-

DAVE: Parallel development, yeah [laughter]. I worked at Acclaim Entertainment, which was not my proudest moment. They're not a great company. But we were working on a video game, and they always get into crunch time as the Christmas season approaches because you got to get your games manufactured to be on the stores before Black Friday and that sort of thing.

And they had brought in a whole bunch of contractors from our London studio, flew them over to Salt Lake, and literally just throwing bodies at the problem. And this was when I read Brooks. And so, I wrote, "How long does it take to make a baby if you assign nine women to the problem?" And I came back the next day, and somebody had written, "Two weeks, because we're going to hire nine more contractors."

[laughter]

WILL: I'll tell you what's another real needle mover in terms of team efficiency is, when the deadline starts to slip, the more people you can get on a project asking for status reports, and timelines, and when that's supposed to be [laughter], it motivates. It allows team members to have that sort of clarity of purpose because they know how important it is. Otherwise, I mean, developers, I got to be honest with you; sometimes, when we're behind on a project, we can kind of drift. We start taking long lunches. We go for walks around the office, playing a lot of ping pong.

Without that focus, you could kind of...honestly, sometimes we just forget that we're behind on these commitments. And sort of if you can have somebody every hour, every 30 minutes or so really getting the status reports, it's such a weight off everybody's mind. I forgot. I was like, "What? Oh my God. Stand-up was two hours ago." I already forgot that this project is way behind, like; oh, thank goodness. I could have wasted the entire day not updating you on where the thing was just because, like, oopsie [chuckles], what was I doing?

KYLE: I think having as many of those status updates as meetings that really speeds things up, too. So, make sure that all of those are meetings and scheduled throughout the day.

MIKE: And you want everybody to be aware of it. So, make sure to bring the whole staff in, right?

KYLE: Yeah. The entire staff.

EDDY: I was going to say, make sure everyone who's able to attend to attend; that way, it's more efficient.

WILL: The really critical people, the people you've got to have even more so than product managers who...I know they're coordinating the tickets, but I'd say the people you really, really need to get in those meetings, as many meetings as possible, is the senior technical leaders like your senior engineers, your team leads, the people who are the people who really know the critical bottleneck issues that need to get plugged to enable maybe the junior level team to do work. It's like, you got to know what's really, really going on, and you got to have technical expertise to get those real serious moment-by-moment updates.

DAVE: Yes, if you're going to clean the supply closet in a hospital, make certain that every surgeon in the building is required to be in that meeting, every single one of them.

WILL: Are we going too dark? I feel like we might be going too dark [laughs].

DAVE: I'm here for it, but yeah.

[laughter]

EDDY: I do want to also just say, someone who didn't pick up on it, just for our listener out there, all that was satire, by the way [laughs].

DAVE: Oh yeah, this is a sarcastic episode.

EDDY: [laughs] Please don't take this to heart.

DAVE: Are you being sarcastic? No [laughs]! Yeah [laughs], good times. I think we touched on this a little bit, but make sure your businesspeople are making technical decisions. Make sure your technical people are making business decisions. This is important. If you're at the keyboard typing away, writing some software and that last function just won't work because of this problematic workaround to it, just cut the feature. It's too hard to do; just change the feature to the way you want. If it doesn't comply with some stupid law, whatever, that's somebody else's problem. Businesspeople, at the same time, don't tell people how much stuff is worth. Insist on that you know the price. And we had talked about that. Make sure you're telling people what their estimates are, so...

WILL: I will say, I think, it's really important to make sure you go up the entire chain to product if you want to do something like what color should this button be, or the copy should it be, like, select, or okay, or see all. I'm going to need to check in with marketing, sales. I can't make that decision on my own. I'm just an engineer. I got to have sign-off from VP of marketing on what the okay button text should be. What is the design thing? Should it be white or off-white? I don't know. These are critical questions. Completely above my pay grade. I can't use common sense. I'm an engineer. I'm half autistic, you know what I mean [laughs]?

DAVE: I have worked at a shop where the art team would hand us comps, and they had to be pixel-perfect, or they would just stop the whole team. It's like, "No, no, you rounded this button with a three-point fillet, and I specifically specified a 3.5." Yeah, I can slice pixels. I don't like it, but I can do it because I had to do it.

MIKE: Well, if you really want to make sure your project gets done on time, you don't want to ever have your developers idle. So, make sure you get them started working on the project immediately while you're gathering the requirements.

DAVE: Maximize utilization. Absolutely.

MIKE: Absolutely.

DAVE: Absolutely.

MIKE: It might take you several months to get the requirements for the project. So, you need to make sure that they're really busy working so that when you get those requirements, they already have most of the work done.

EDDY: You know you're being efficient when you're reverting a bunch of the work you're doing because you're releasing a lot of code.

DAVE: I may have put that on a pull request this week. I love it when I can add features by deleting code, yep.

KYLE: [laughs]

DAVE: I think it was yours, Eddy. I think it was your PR that I did that on, so...yep.

WILL: I'm not going to lie. I actually...

DAVE: It wasn't your code that we were reverting. For anybody listening, it was some legacy code that was no longer relevant, so...

WILL: Anyway, I don't know, like, unironically, not devil's advocating, I like just sort of getting to work, you know what I mean?

DAVE: It's very satisfying.

WILL: I like the aspect of, like, well, just agile sprinty. I mean, unironically, I'm not being the devil's advocate. Let's just throw something up and let people look at it, not release it. But it's like, "Here. Do you like this?"

So much ambiguity can be resolved by giving somebody something that they can touch and feel and put their hands on. I'm no longer being ironic. I like giving people prototypes, and they can click the buttons and be like, "Oh yeah, it clarifies so much." I don't know; I've found there's, generally speaking, a difficulty among people who haven't sort of managed a lot of software projects to think about things in a fully fleshed-out way in the abstract. If I showed you a form, and a button, and a widget, and a web page, even if it's connected to nothing, I can get much better feedback, much faster, you know what I mean? I like that.

EDDY: I would argue that's part of your UX design person. It doesn't really fall on the developer.

WILL: You go to war with the designers you have, man. They're art school kids. Like, that's not their gig. It's okay. They're going to make it look good. That's their job. Like, I don't know, if I can make it easier, if I can work harder technically to get a better product, then I'm going to do that, and I'm not going to complain too much about it. Like, that's my job.

MIKE: And, unironically [chuckles], right? I agree with you. I was thinking of an example of, you know, go spend three months working on backend without requirements, without providing a single prototype. And yes, I've seen that kind of thing happen [chuckles] in the past. And it goes about as well as you'd imagine. The [chuckles] services you're building don't match what's actually required because you need to do...to your point, Will, again, unironically, that quick feedback is so critical that if you can't loop on it, it's terrible, which is a good reason to build out one small domain out of your project first so that you can have something that actually works.

WILL: Yeah, yeah, that is an important caveat. I would say that this is kind of frontend only. You can't show your VP of sales a slick API. I don't know who the VP of sales is, but I have a pretty clear idea of, like, that is some slick JSON, you know what I mean? [inaudible 28:00]

MIKE: [laughs]

WILL: But you can show people interfaces. And if you show people a frontend, you know, the backend exists to serve the frontend. Like, the backend is your problem. Get the dorks on it. Anyway.

DAVE: There's a web mockup tool that was designed...it's called Balsamiq. And I don't know if it's still like this. I haven't touched it in years. But when it first came out, it looked like napkin sketches. Like, on your computer screen, it would draw somebody with a ballpoint pen on a napkin, and they did it very deliberately.

And I have had this happen to me where you mockup a quick prototype, business looks at it, and they go, "That's great. Ship it." You're like, "No, this is literally a prototype. There's nothing wired behind this. This is 4% of the system." And they're like, "That looks like 90%. Ship it. You've got a week," and you're like, "No."

And so, Balsamiq literally shipped sketch stuff to stop that, to let the businesspeople understand that this is nowhere near complete. And it's related to what you were just saying, Will. I think knowing your trade-offs is super important. There's a time when you want to spike, and there's a time when you want to just knuckle down and trudge. And we've got good estimates, and we think we're right, and there's a lot of work to do. Let's sled along on this, or let's...versus jumping in ahead and working on the wrong things, but, spikes, sure. And they do happen frontend, backend.

I have literally spiked a mockup of an entire website, full-stack, database to frontend with just Rails, generate, generate, generate, generate, in a car on the way to an investor meeting, because my boss had said, "No, we're not switching to Rails. We're not switching to Rails". And, in the car, he's like, "How long would it take?" And I'm like, "Probably mock it up in the car." And he's like, "No, you can't." I'm like, "Bet." And I did. And so, we switched to Rails.

But that spike there was nothing shippable in that spike. I built it to throw away. It served its purpose, which was to say, "This is possible. This legacy codebase that we spent five years building, here's the prototype. And based on the prototype and our existing codebase, I'm thinking 18 months." So, I was able to give him a realistic estimate off of it. So, knowing where your trade-offs are, I think, and --

EDDY: If that doesn't tell you how powerful Rails is for a start-up company [laughs], I don't know what does [laughs].

DAVE: To be fair, this was Rails 3.0 when you could still write a blog in 15 minutes on it without having to know Stimulus and a front-end packer, and a pixel shipper, and the, you know, the asset poop line, and all that messy stuff, so...

MIKE: So, if I'm going to go...I'm going back into the snark.

DAVE: Yes.

MIKE: Advanced warning [chuckles]. One great way to make sure that a project gets done on time is to bring in as many teams as possible. But make sure they work independently and never coordinate with each other because you wouldn't want them to have any inefficiencies of communication. Also, let them design the interfaces independently, and then we can just fix that communication layer at the end because that's not the important part. The important part is that we all make sure that they're all working, and they're all heads down. We can handle any and many communication issues later. Make sure the APIs are in line. That's not a big problem, right?

DAVE: Yes.

EDDY: Well, if you want a way to speed up your service, just don't write tests.

DAVE: Oh yeah.

EDDY: Just ship your code.

DAVE: Fair point. Yeah. And there's a generic form of that, which is never check your work, right? Never cross-check. In fact, we should put that as part of the waterfall. Make sure coding is 100% done before you start debugging. That's literally part of the original SDLC when they were saying, "Stop doing this."

EDDY: Yeah, also, let your developers be the sole QA. That's good, too.

DAVE: Make sure your creators are editing at the same time that they're trying to create.

EDDY: 100%.

WILL: I mean, software QA exists. The entire career exists because I cannot be trusted.

MIKE: [laughs]

WILL: That's it. That's it. There's nothing a QA can do that I can't and don't. You cannot trust me. I cannot be relied upon. Like, I can't do it. You can't, like, jump. Oh, wait, wait. Hang on. No, no, hang on. I said that wrong.

DAVE: You got it backwards, yeah. How dare you be candid?

WILL: Bro, why do you even have a QA? Devs know how it works. Like, they understand it better than anybody. QA is just a waste of time. Like, you have an extra person? They don't even know how to code, bro. Like, What? Why even? It's a nonsensical position to have. Like, the devs could do it. They did it. Just shut your work, guys, duh.

DAVE: If they can't code, we could have them doing tech estimates for us, though. There's that option, so...

MIKE: [laughs]

EDDY: And you can have your product guys go ahead and help you commit a bunch of stuff. When you're short-staffed, you know, just rely on other departments.

DAVE: Yep. We talked a little bit earlier about Fred Brooks, about how many women and da da da da. Fred Brooks wrote a book called "The Mythical Man-Month" that's all about this. And Brooks' actual law that he quotes is, "Adding people to a late project makes it later."

EDDY: It's true.

DAVE: That's where it comes from, yeah.

WILL: And I'd say another note on QA. I'd say the best people to do the QA are the high-level executives when you're doing your demo, your final presentation, preferably even after it's already been out to production and you got your VP out there.

DAVE: [laughs]

WILL: They're pretty loose people. Like, they're not detail-oriented. They don't have real visions. And even if they did, they found something; it's not like you're going to look like an idiot when your VP of product is checking stuff in prod, and your links are broken. And, honestly, those guys, they don't even notice half the time. They're just like [inaudible 34:30], whatever. They're just so chill. They don't even care.

MIKE: They don't have any relationships outside of your business either, right? And they're too busy working with people in the company. They don't have the CEO of your partner's company on their speed dial because, yeah, they don't have time for that. There's no way anybody would ever, you know, important business partner would ever reach out to them at 3:00 in the morning wondering why the service is down. That's just not their job.

EDDY: You know, one way to really ramp up the way wave to meet the deadline is to just ship it, even if it's broken. You met the deadline. Just merge --

DAVE: Yeah. We can ship DLC later.

KYLE: I've got a controversial one, I think.

DAVE: Oh, please.

KYLE: DevOps is a culture, not a team. I've seen where, you know, that's on the developer. They can do their own DevOps and manage their own infrastructure. And --

DAVE: It's called DevOps for a reason, Kyle. We're the devs. We do the ops.

KYLE: Yeah. So, we don't need a specialized team for any of that.

DAVE: Yeah. And all you need to know as DevOps is just chmod A plus X minus R. Just go nuts, just 777 for the entire hard drive. Yeah, it's fine.

WILL: It's called CI/CD, bruh, continuous deployment. I'm just going to throw that thing up. New endpoint? I don't know. There's probably no scaling requirements. The server was fine before. I'll just throw this new API up there. Let's roll, YOLO.

DAVE: Ooh, related to this, Mike's going to wince a little bit, but don't do any tech debt. If your CI machine catches on fire, get some marshmallows. That's the best thing you can do for it, yeah. So... [laughs].

EDDY: Well, you don't have tech debt if you just delete [laughter] [inaudible 36:16] you've broken.

DAVE: Mike is covering his mouth. There may have been an incident at work this week, so...

EDDY: You can't have [inaudible 36:36] that's broken, Dave. You just delete the code.

DAVE: That's right.

EDDY: Just delete it. It's not broken no more.

DAVE: Yep. Actually, touching on what Kyle said about DevOps, if you really want to mess everybody up, divide your company horizontally. Make sure that the database team are in their own spot, servicing all the database clients, because then you've got all your experts in one place. And anybody that needs anything from the database team now has to go up through their boss and their boss, and then come down through the data management people, and they've got objectives that they need done. Same thing with your sysadmin, same things with your DevOps, same things, you know, if you've got anybody that's writing backend and frontend, get them on separate teams. Whatever you do, do not under any circumstances have a full cross-functional product team.

KYLE: I think another good one is launch your service or whatever deliverable you have before you consider security.

DAVE: Yeah. Yeah. Yeah, you can just bolt that on later, come on. There's no security. That mockup that I showed you that, you said, "Ship it," and gave me a week; we can just do security later. It's fine. 100%.

EDDY: Mockups don't have permissions. Dave, are you crazy? You don't need to scope permissions to your users. Just whatever --

DAVE: If you can see the mockup, you can access the thing, yeah.

MIKE: You'll never have a user who will try changing the URL to see, you know, another user's data. Our users are never clever like that.

DAVE: Ooh, slightly related to that, I'm not sure how to phrase this, but as you're developing, if you trip over something, don't flag that. Just step over it every single time. Definitely don't put rules in your engine. Don't write an RSpec thing to check to make sure that there's no implicit scopes or default scopes on your Rails models. And you want default scopes everywhere. You want to tenant your customers at the default scope. That's really, really important --

EDDY: Make sure you unscope, Dave. Like, that's –-

DAVE: And then, what you'll also do is you'll have your scope that says, "Valid," which is going to select for things that have not been deleted so that then when you say "Unscoped," we throw away the deleted at so that you can pull all the records, and we also throw away the tenanting. And now you're looking at every single customer in the system. I had that happen at a company where showing somebody else somebody else's data involved federal oversight. Like, you messed up. There was a department that you had to report to and say, "We did that."

EDDY: I was going to say, Dave, that's kind of oddly specific [laughs].

DAVE: And if you press me further, I will deny it, yep.

EDDY: [laughs] [inaudible 39:13]

DAVE: I've never worked with an entire company full of idiots. I self-select out of that environment. I've seen all of these design...most of us have, right? All these design hell problems, and they usually come about based on really good partial information.

EDDY: I was going to say, something that can really speed up is just always, always, always going against your framework's convention. Like, it works 100% of the time.

MIKE: Well, I can one-up you there. You want to get out quickly. You don't have to deal with the constraints of a framework. So, make sure you implement from scratch because, that way, you can...

DAVE: Roll your own.

MIKE: Yeah, roll your own. That way, you won't have to memorize the conventions. You won't have to deal with, like, libraries that will fight back against you when you're trying to do direct database manipulation. Just build everything yourself. It's a much faster way to work.

DAVE: This week, I am maintaining some code that has some custom snippets of SQL wrapped up in a custom DSL that are wrapped up in the source code. And it's just a joy to work with. It's absolutely fun. I have not ripped out any of my fingernails. I have not shaved my head defensively to save tufts of hair from being ripped out. It's, yeah, it's just been a dream, absolutely a dream. So...

EDDY: You know, another thing that's helped me, Dave, and I want to say, like, [inaudible 40:41] loading has been wonders. Like, you want me to ship something? Yeah, copy and paste someone else's work, you know, that's done before. It doesn't matter if it's being used in that context or not. Just [inaudible 40:53] is always --

DAVE: Yes, going back to the good intentions, end up in hellish places; drying things up to the point that they become chaffy is fantastic. You should compress everything and especially, you know, that SQL DSL that I'm working on, get into RSpec, and there's an include, not, like, include shared context. It's an include module, and the module was in the support directory. And I opened that up. And when you include the module, it creates a shared context and includes it. And the shared context creates lets. I have told that nightmare story here at work multiple times, thinking certainly that I would never see it. We saw it this week, so it's pleasant. It's kind of fun.

EDDY: Mine is a controversial one.

DAVE: I will say that I am sincerely saying that I'm enjoying working on this because I feel a little bit vindicated when I work on it. Also, in defense, because I did actually say something about a real company that is doing real work, and I don't want to get yelled at for this, I do want to point out that the developer who did it that way did it in 2015 when it was a best practice. I've written a lot of code that had shared context and riffling the deck to shuffle all the pieces so that they, like, laminates on an overhead projector with multiple layers to make them line up and that kind of thing.

It's very satisfying to dry that up in your text editor because you've got it all in your head, and you're like, oh, yes, why should I duplicate this? Let's put this over here. It's not until you come back later that you realize that you have just burned off every entry point for intuition and for someone else to be able to follow your path.

MIKE: I was thinking about a real-world story as well about avoiding the framework. I'm going to go back a lot farther, I think, like, 2003 [laughs]-ish.

DAVE: Ooh, okay. So, probably not Rails. It might have been Rails, but probably not Rails.

MIKE: Not Rails. I don't think Rails was out there. Anyway, this was actually some sort of Visual Basic. I don't remember the exact flavor of it. And there was no database library used whatsoever. It was all just raw rights to the database. And it only takes one unsanitized database input. It only takes one to lose a million dollars [laughs] of your customer's money.

EDDY: Yeah. Just don’t --

DAVE: What's the .NET... .NET has a SQL thing, like linker, SkLink. There's a name for it. But yeah...that would've been coming into existence about two years after 2003. So, yeah, if you're going to write SQL, it's going to be in the code. It's going to have to be in the code.

MIKE: This is back...I remember Hibernate was just getting started in the Java world and, like, wow, this is great. You can use some library and convention to write your SQL rather than hand-coding it all [laughs], but not so in the application that we took over [laughs].

DAVE: Oh, never switch. There's some people on this call that are going to feel seen and attacked at the same time, but this is actually not. But if you're going to add a new technology, add it. Don't change the existing stuff. You want people to have to maintain both. It's even better if they don't know which one they're working on until it's too late. That's a good one.

MIKE: As many text stacks as possible?

DAVE: Mm-hmm. Mm-hmm.

EDDY: Ship --

DAVE: For real, I used to walk into stand-ups...if I wanted to terrify everyone, I would give my stand-up report as, "So I did a quick spike, and I rewrote this in Rust." And, if you can sell that with a straight face, you can get, like, the blood to drain out of other people's...they're like, "You're doing what?"

MIKE: [laughs]

EDDY: Also, make sure you support multiple versions. I think that's --

DAVE: Oh.

MIKE: Simultaneous.

DAVE: Arbitrarily, arbitrarily. Don't just do it for the interface stability reasons. Do it arbitrarily.

MIKE: Well, if you don't have three versions of bootstrapping in your codebase, you're not a real project.

DAVE: What are you doing right? Yeah, and Angular 1 to Angular 2, they redesigned everything. So, you're going to need both, or you're going to be missing stuff out.

EDDY: It's a good thing that you can't support multiple versions of Ruby, though, like, satire aside, right? Like, you can only have one true Ruby version for the project. You can't support multiple. Is that correct? I see you smirk, Dave. Like, I wonder [chuckles], is that possible?

DAVE: It's absolutely possible. Like, Ruby 1.0 shipped with a library, dRuby? Distributed Ruby. And it made remote procedure calls very, very easy. You could write your program in two places. And if they knew how to find each other, you could call the function here, and execute it over here, and send it back, and nobody uses it. It was an interesting experiment, and nobody uses it. But 100%.

And, honestly, we do this here, where we've got service-oriented architecture. So, this team is on one version of Ruby, and this team is struggling because they've got this really important science library or finance library that isn't up on the latest. So, they have to stay back on an old version. Yeah, you do see that. So, what you have to do is you have to get out of the processor. You can't share memory with each other. I'm not going to say that super confidently. Somebody will say, "Here's a proof of concept."

MIKE: [inaudible 46:34]

DAVE: But, to my knowledge, if you're going to have multiple versions, they're going to have to drop to a protocol and communicate with each other like separate programs.

EDDY: Not ideal.

MIKE: We've come up with a lot of great ideas. I think that we could come up with an estimate that's pretty much whatever we want.

DAVE: We might be able to bring down the whole system.

[laughter]

EDDY: I was going to say something controversial, but some people may or may not agree, really just depends on your perspective. But I was going to say, as a satire, make sure you always do integration tests, and use factories to make sure you're testing the real data so that it gets committed to your database, but --

DAVE: And committed without any trade-offs.

EDDY: But I feel like some people do agree with that, and that's an interesting [laughs] topic.

DAVE: As long as you're ignoring your trade-offs so that you've got your test where you need to go really, really fast, as long as they're writing to disk and doing everything slow, and exercising the full-stack, and the stuff where you just need to test an accessor method that doesn't even talk to anything else that should be talking to the database, too, then, yeah, you'll be fine. You'll be fine.

MIKE: Well, you need to make sure that everything works, right? So, make sure --

EDDY: Make sure you test Ruby and Rails, too, like --

MIKE: Well, yeah, you can't trust the framework, and you definitely can't trust your third parties, so make sure that you make live API calls to their sandbox in your integration, you know, in your CI environment.

DAVE: Oh, yeah. If you come up with a workaround to a problem, solve it as far up the tech stack as possible. Like, I worked with an engineer who would recompile Ruby to add features to it, and we set him straight pretty quickly because [laughter] we're like, "We're not shipping your copy of Ruby to all of our servers. It's not happening. Stop asking." To be fair, he was a genius. He could shatter a human mind at 50 paces just by thinking about it, and he was very, very, very clever, and sometimes he was too clever for all of our good, so...

KYLE: I'd say –-

DAVE: I had a funny idea for how to end this podcast, which is to say, "Okay, now we estimated this much time," and then cut off right in the middle of the sentence, just no outro.

[chuckles]

EDDY: Kyle, you were going to say something?

DAVE: Sorry, yeah.

KYLE: I was just going to say, and don't worry about the performance of your code. We can always throw more hardware at it.

DAVE: Right [laughs].

EDDY: It's your computer that's slow, dude, [inaudible 49:11] [laughs].

KYLE: And we can ship that into the cloud if we need to.

DAVE: Yes.

EDDY: Develop to the cloud; it has unlimited resources.

DAVE: And I will see you in raise. You should care about maximizing performance at the start.

MIKE: Yes [laughs].

DAVE: You want to make the code as hard to read and as difficult to maintain as early as possible. And unrolling anything that's readable into something that's just hyper-efficient, go for it. That way, you're guaranteed to optimize things that aren't important. So, we will still need to scale up into the cloud to get around all the bottlenecks that we didn't even look at.

MIKE: And make sure that you do lots of planning around that optimization during the beginning of your waterfall process.

DAVE: Yes.

MIKE: Because that's what's going to matter the most is that optimization.

EDDY: I was going to say one thing, too. I was going to say, don't add comments and be super brilliant to what you're doing, so it's complicated for everyone else.

DAVE: Yes.

EDDY: So that you --

DAVE: Never ever pave a path through the code. You need to vanish into the jungle and be lost without a trace. Anybody who needs to follow you needs to solve it from first principles. That's important. It builds character.

EDDY: 100%. Like --

DAVE: And you need to prove your worth.

MIKE: So, if you don't have tricky code to read, well, then you're probably doing it wrong, right? I mean, how smart are you if everybody can read what you're writing?

DAVE: Yeah. We can go into a whole tangent about, like, your ego is more important than right and wrong, and that should be protected at any cost.

KYLE: I think one thing that goes really good on readability is the largest PRs as possible.

MIKE: Oh yeah.

DAVE: Definitely.

MIKE: You wouldn't want to waste time with QA having to review a bunch of small PRs. So, make sure you release the whole feature in one, that way, they can just test it once –-

KYLE: Save testing time.

EDDY: And test in production. That's the best way --

KYLE: Oh, that's the best one. Yeah, we can go to production first.

DAVE: Yes --

EDDY: Who better to tell you that your feature sucks than actual users, you know what I mean?

DAVE: I have a suggestion for a good final one, which is, if you're listening to this podcast and you think we're making fun of you, ignore it. It's fine. Don't take any lessons away from this. Remember to just stick by your guns, no matter what. Nobody has done any of these things in real. We're not talking about you; it's fine. Yeah, definitely don't take a hint from this to maybe try to improve steadily and stably. If you are going to react, make sure you overreact. I think that's the important takeaway here.

MIKE: We'll take the same advice because, of course, we've never run into any of these things ourselves.

DAVE: Mm-mm. Mm-mm. That's fantastic. Are we at a good stopping point, gentlemen?

EDDY: I think so.

DAVE: This has been a lot of fun. Thank you, guys, for coming out: Mike, Kyle, Eddy. Will had to drop off for a family issue. But so glad you guys came out. This was a lot of fun. And thank you all for listening and watching The Acima Developer Podcast. We'll see you again in a couple of weeks.

In this episode of the Acima Development Podcast, Mike kicks off the discussion with a few personal stories before diving into a technical debate. He recounts his recent epic hike in the Rockies with his son, which turned into a grueling trek due to altitude sickness. He also shares a nostalgic memory of visiting the massive Edmonton Mall in Canada as a teenager, noting the stark contrast between the pre-internet shopping experience and today's online marketplace like Amazon. The conversation then shifts to a comparison of governance structures in various countries, which sets the stage for the episode's main topic: software architecture and the debate between monoliths and microservices.

The panel explores the evolution of software architecture, with a focus on the trade-offs between large monolithic systems and distributed microservices. Mike and the others reflect on how early approaches to software development often favored building large, cohesive systems, but over time, the trend shifted toward breaking these systems into smaller, independent services. They touch on the challenges of communication and synchronization among microservices, and some express a sense of regret over the movement toward extreme modularity, suggesting that, while microservices have their advantages, they can introduce significant complexity and overhead when overused.

The conversation also delves into the practicality of maintaining and managing distributed systems, particularly from a DevOps perspective. The panel discusses the fine balance between creating boundaries in code and allowing flexibility for different teams to work on separate components. There is consensus that, while microservices can improve scalability and modularity, they can also lead to what one panelist dubs "micromonoliths" if not properly managed. Ultimately, the panel agrees that the choice between monoliths and microservices depends on the specific needs of the system and the organization, underscoring the importance of choosing the right tool for the job.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting today. We've got a great panel with us here today. We've got Will, Eddy, Adam, Dave, and Kyle. And today...well, I'm going to hold off on what we're talking about. I'm going to give three stories; three things to start us off.

WILL: [laughs]

MIKE: But the first one is a story. I'll start there. So, last week, I was on vacation, drove out west, saw the Rockies, went on a, I'd say, an epic hike, which is true, but it was also just, like [laughs], in some ways, a brutal death mark hike [laughs]. Went on a hike with my oldest son, and we did, like, 8,000 feet of elevation gain, and, like, 20 miles, like, 16-hour hike, just a big, brutal hike.

Loved it, but it was tough, particularly because we got up to the high elevation, and my son got altitude sickness, and he was just sick. I'll spare you the details, but there were some digestive issues going on that were serious [laughs], and that cost us quite a bit of time [laughter]. And he was not having a good time. Everything's fine. We got down to a lower elevation, and he felt a lot better.

And it was great, I mean, the view was amazing. Didn't see very many people. It was a gorgeous hike, but [laughs] it was rough. We've been back home this week, and he's been doing some online classes, but, in between, he's been running up and down the stairs for exercise. So, like, you know, go one flight of steps up, and I think he's been doing 60 times up and down the stairs, seeing how low of time he can get it, and no altitude sickness [laughs]. It's working out better for him. So, there's my first story.

Second story, when I was a kid, and this was a long time ago, I went up to the Edmonton Mall up in Canada. I think it's the largest mall in North America. And I don't know how old I was, like, 14? So, I'd say it was a long time ago. So, I really can't speak to them now, but I can when I was 14. It was this huge mall, and there were several stores. There was, like, three of that store within this mall. It was so big that there was, like, a region where you go to a store and then go to another region of the mall and go to the store again. And you just get lost in this place [laughs].

I don't know how many, you know, fast food places they had. I remember there's some places that had at least three stores. It was crazy. And they have, like, an indoor amusement park, and water park, big mall. And, you know, thinking about the era that I'm talking about, that was, like, the pinnacle of shopping experiences because you could get all the places, you know, all the stores in one place. Since that time, times have changed. That was pre-internet era.

Well, now, if you want to go to something similar, you just pull up your phone, and you go to Amazon, right [chuckles]? Not that people don't still go to the big mall in Canada; I'm sure they do. And they probably enjoy that experience, but they probably lean more into the experience part of it, your amusement park and water park. Because you go to Amazon, instead of having everything in one place, they have their little stores all over the world, right?

And by distributing that work, and just making it a central, like, bazaar, you know, where you come and see all the stores in one place, you get way more selection. And the world's kind of fallen for it. We're not going to debate the pros and cons of Amazon here [laughs] [inaudible 03:54] that they have been widely embraced, that they have been widely embraced with that different model.

Third story. And this one is not so much a story as a comparison. Let me talk a little bit about governments. Now, in most large...I say most, I'm going to mention the United States and the European Union. You do have a central federal government, but you also have a contentious group of states within that central government that don't always agree and regulate themselves internally. And sometimes it works, sometimes it doesn't, but [laughs] it does allow for some local rule and for those independent states to run things, you know, to their own liking. And then, they federate for the big decisions that need to be made together.

There are nations that don't work that way. I was thinking about Saudi Arabia. Very much, we've got a monarchy [laughs]. We say what you do, and that's it. The interesting thing about that, though, is it's not like those fractious internal debates don't happen in Saudi Arabia. If you read anything about the internals of the ruling family there, it's scary [laughs], murderous sometimes, and I don't want to mess with that. So, it's not like those fractious internal debates go away. They just get handled very differently when you've got kind of a monolithic rule, which brings us to the topic of our discussion today.

We're going to talk about software architecture and about monoliths versus microservices. This has been an ongoing discussion for a long time now in the community. It used to be kind of everybody went the [inaudible 05:33] think [laughs]. You'd build a big thing and then deal with it. And people start saying, "Well, no, big is bad. We need to have lots of small things." And then, some people pull back and say, "Well, no, maybe big wasn't so bad after all because there's a lot of issues with communication among lots of small services and keeping those all in sync."

I brought up a few examples outside of the software world to give us some fodder to chew on, and I'm going to start off by just asking you all in this spectrum, saying, hey, you know, putting everything in one big application versus breaking everything up into small apps. Where do you stand? Where do you think that right balance should be?

TAD: Can I get a definition of what you would say a monolith is and a microservice? Because I remember way back when we were just...everything was services, and then this huge trend came along of, oh, what if we made, like, really, really small, very, very narrow services and did everything with Sinatra and never touch Rails again because it's too big and bloated? So, what makes something a microservice, versus a service, versus a monolith, versus anything?

MIKE: There you touched it, right [chuckles]? I used the word spectrum deliberately [laughs] because I don't think that anybody has a perfect definition there. And you could certainly, say, "Hey, we're going to have an architecture where there's no service that has more than one end point." Sure, right? I don't think most people would like that. Just going out on a limb --

DAVID: It's arbitrary. That feels like an arbitrary decision, right?

TAD: Well, I've seen that. I've seen places that they're like, oh, we're going to have a routing number validation service. We're going to have a credit card validation service. We're going to have, for any particular thing that you need to do, you've got a service that you hit. And it's divided up amongst...it's almost like functional level [laughs] services.

MIKE: But how did it work? [inaudible 07:43]

TAD: Not great. Not great. You can probably tell that I'm like...every time I hear the word microservice, I just [vocalization] shudder and just, I don't know.

WILL: So, one, I feel like a monolith versus microservices architecture it's like pornography, you know what I mean, it's hard to define, but you know when you see it. And, two, and what I've seen, I have some direct experience, and then I have received...maybe a year and a half, a year, year and a half ago, there was this top-down mandate at the large electronics retailer that I work for that all of the teams are going to be cross-functional teams, right? So, they said like, "Nah, we're not doing monoliths. Everybody's doing everything," right?

And almost immediately, all of the cross-functional teams immediately spread out into their, you know what I mean, on a micro level, so instead of having a team of mobile developers, or a team of backend developers, or a team of frontend developers, everybody sort of split out and said, "All right, I know how to do the mobile app stuff, so I'll do mobile app stuff. You know how to do back-end stuff, so you'll do back-end stuff. And you know front-end stuff, so you'll do front-end stuff." And everybody just sort of self-organized  into their silos of choice.

And even if it was one giant repo with one giant Windows desktop application, right? Like, if you had a team of developers...and they will naturally, instinctively, like honeybees, self-organize into functional groups. It's like how even if you have an open seating plan, everybody will kind of pick a chair, and they will sit in their chair automatically. And they will get familiar with this one siloed piece of the codebase because it takes time to boot up. And so, it's an emergent sort of organizational thing. Microservices will happen instinctually. And they're like, oh, maybe we don't have separate GitHub repos. Even if it's all in just one big pile, humans will instinctively do that, you know what I mean?

MIKE: I think that there's a lot of truth to that. Well, we have limits as to what we can fit in our brains [chuckles]. There's finite limits to human capacity to think through a problem, to think through some domain within a problem. And once something gets outside of that space, you know, once it outgrows that, it becomes really hard to think about. And, you know, you probably have to map something out and think about sub-pieces of it. So, you just naturally look at sub-pieces of it.

And there's a few ways to approach that. You can have it within the same repo, right? Not the same repo but the same application. You deploy it all together. And then, you have, you know, lots of units, functional units within a single codebase, or you can just blow it up and have separate, you know, fully independent. If you're out on the web, if you have the advantage of doing so, you can have all these separate units, you know, deployed separately, managed separately. You don't even have to think about the other ones, except via API. And really kind of harden those boundaries to where, you know, you only talk to each other via API.

And by enforcing that, now you don't even have to think about those other things, maybe [laughs]. Maybe you don't have to think about them. And I think that's kind of the microservices or the, you know, the small services advocates would say, "Well, yeah, no, you should definitely make a hard boundary. That way, you're not tempted to cross those boundaries." Where people who've seen that go too far tend to say, "Well, you know, maybe we can maintain boundaries in code, and we don't have to put it, you know, deploy separately." You're saying, "Well, this happens automatically." And I think that there's a lot of truth to that. Do you think there's value in hardening those boundaries to enforce an API?

WILL: You know, if I'm being totally honest with you, what I see, like, the difference between sort of a monolith versus a microservice thing is, I see it as analogous to sort of serial versus parallel programming, right? So, with serial programming, you do it right, and then if you do everything right in a row, procedural, like, tick, tick, tick, tick, tick, tick, you know? Add it all up in a line, and then there's my result. I return my result, and I'm good. But it's not as fast. It's not well, you know, broadly.

Like, if I want to write some slick parallel algorithm thing, that's a harder job to do. But you can get that 16 core, you know, server. You can keep them hot, and that's great. But it takes higher skill. And so, similarly, the spectrum you're choosing your path on in terms of microservice versus monolith, if you have a monolith, you're leaning really, really hard as things get bigger. As the monolith becomes larger, you're leaning really, really hard on super high-skill, super domain knowledge engineers that can wrap their heads around everything and understand how to deal with the monolith without knocking it over. Because we think of it like a pyramid, but really, it's like a pillar just swaying in the breeze.

And so, when you go over to the microservices thing, then you're leaning on sort of managerial organizational capacity to manage this sort of 16 core. Instead of 16 core CPU, you've got 16 dev engineering team. And you need to make sure that everybody is coordinated and pulling in the same direction and all those things. And so, that's sort of the spectrum. Like everything in engineering, it's not good versus evil, angel versus devil things. It's just sort of where are you on that spectrum, and what's the right tool for the job for where you are right now?

DAVID: And, Mike, you mentioned at the top of the call you were talking about monoliths versus services and whatnot. And do we sit down and write a big thing? And I think, no, when you start out, you always write a small thing that, you know, every monolith started out small somewhere. And, for me, it's always down to a trade-off, right? If you've got services, you now have very expensive communication between those services. But the things that are in those services are now decoupled, which is nice.

In a monolith, the communication between them is very fast and very easy. I just want to look at that module, just go look at that module. The drawback is that if that other module wants to look into your stuff, they can just do it because they're inside the firewall, and so there's that coupling back and forth.

So, I made a laundry list as you guys were talking. Tad, you mentioned definition of microservices. What I saw in the teens, the last decade, was a lot of people doing like, oh, monolith's bad; SOA good. And what they went out and wrote was every service was its own monolith. It's like, oh, instead of having one, you know, 100,000-line code base, you now have 500,000-line codebase. This is not moving in the right direction.

TAD: Now you've got two problems.

DAVID: Now you've got two problems, exactly. And so, yeah, every problem in computer science can be solved with another server in the microservice cluster, except for the problem of too many services in the cluster. And I heard a really good definition of what a microservice is, and that is something that you could rip out and rewrite in two weeks. And I've never [laughter] seen somebody write...yeah, exactly. That is the correct reaction. And if you're saying, "We're doing microservices," and your services are all 30,000 lines and have 73 models, those are not micro. You couldn't rip them out and rewrite them.

The interfaces between things are very expensive. Well, they're much more expensive because you have to negotiate with both sides, and, often, there's, like, three or five different deploys that you have to do to make it possible, and then use it, and then require it, and da da da. And you have to work out that dance. And if you need that secured, and hardened, and guaranteed, then you want that to be formal. You want it to be in a service. So, you want it to be locked down on a contract. But if it's fluid and it's flowing, you're just paying through the nose for something that you could have spiked out with an agile team overnight. But now it's a six-month initiative because you made it as expensive as possible.

EDDY: I'm actually curious to hear this from a DevOps perspective. Do they prefer to maintain microservices over a monolith?

DAVID: Do we have DevOps on...Kyle?

KYLE: Yeah, I was going to say, at least from my perspective, I'm all about orchestration tools, and something like Kubernetes is extremely powerful. So, I would much prefer a microservice, but I do feel like we've kind of talked about it. A lot of microservices end up being micromonoliths, is what I've called them in the past. And it's still one of those things, smaller units that can be divided out. And then, it's, at what point do you continue to divide? Because you can have quote, unquote, "microservices," or you can go [inaudible 16:54] the functions because there's stuff like AWS Lambda or serverless, right? So, at what point do you stop...each rendition complicates the next one. With functions, at what point do you keep them hot? Because that's something with Lambdas.

Originally, they weren't kept hot, so you had to wait for them to spin up and be able to function; now, they have hot functionality, and the same thing with even running microservices on servers. You have to have hot servers. Otherwise, you're waiting to deploy. And then, you go to the smaller units, at least in my experience, and you've always got that one server that you can deploy to because it's always up. But you start going out. It is nice in the sense that you can divide your workload amongst smaller machines and cut costs and stuff that way at times. But that's a long way of me saying it really depends on the implementation, I guess. I prefer microservices, but not to an exaggeration.

EDDY: So, micromolothiths.

WILL: Well, I always felt like, from a DevOps perspective, you've got really heterogeneous workloads, in that you've got boxes that are like, okay, I'm doing this thing. I need this amount of stuff. And I have other things that, like...and so, the more heterogeneous jobs you have running on the same machine, I feel like, correct me if I'm wrong, because this isn't really my trade, right? It's like, it makes it hard to correctly provision a machine. Because it could be doing anything under the sun, then you have to create a larger machine, larger buffer because the workload could widely vary.

KYLE: Yeah, yeah. And even within inside of our company, we've got services that are doing different things that are different sizes. And we will make it so that they won't land on the same nodes as themselves or the other large services, and sometimes we don't even want some of the small services to land on there. And kind of what you're talking about, we do have more general-type workload servers, along with we have more scheduled workload like our data science teams, where they've got crons that sit there and run more so than anything, maybe serverless is what it would be closer to, where it's just very spiky workloads. It's just, throughout the day, just bam, bam, bam. And, like you're saying, those need very different machines to support them.

WILL: Right. Right, yeah.

DAVID: One of the things about Kubernetes and Docker—and its predecessor Vagrant, and Chef, and Puppet—this idea of, like, oh, you say you're doing microservices, but you need an entire operating system now? Your microservice needs...you got to configure how many CPUs it has? That's not a microservice anymore. If you need to install Ubuntu, all of Ubuntu [laughs], to run your service, it's not a microservice anymore.

WILL: I don't think I track.

DAVID: Oh, just something you can rip out and rewrite in two weeks. Oh, but you're going to do a Docker container. So, you have to provision an entire operating system drive space. That used to be called full-stack dev. And it's like [crosstalk 20:17]

WILL: And maybe I don't understand because, like, I don't know, I'm stupid, you know what I mean? I thought Docker was cool. I was like, oh, I could just do a Docker. Like, I'll fill up a pod, and I don't have to do a full-on EC2 install. This is so great. What should the microservice, I mean, if it's a proper microservice, how should we be doing it if not [inaudible 20:38] a Docker image? [inaudible 20:40]

DAVID: I want to be careful to not walk into the trap of proper because I think the way we do it now is kind of proper. But 15 years ago, all the hardware in the colo was the same. Like, everyone had the same processor. Everyone had the same hard drive. And everyone would say, "Can I please have more RAM on the server?" And ops would say, "No, you can't have another gig of RAM because another gig of RAM on your box means we have to provision every machine in the colo. It's a 20 million upgrade." And Docker gets us away from that. And I'm 100% on board that that's a feature.

I'm just saying when I see Docker scripts where it's like, okay, yeah, well, let's go get this thing, oh, and now let's download Python and patch it, and da da da da, and it all becomes part of the entrails that are actually part of what you're working on, suddenly, it's very, very complicated.

Mike, you mentioned story time at the top of the hour, and there was a specific story that I think is actually relevant to this. When I was at a company that shall remain nameless because I might still have friends there and I want to keep it that way, they wanted to do a big push for SOA. And we had over 700 separate services at one point, and it was trending up. So, when I parted ways with the company, it was 700 different services, and there was incredibly painful fatigue around adding a new service. Like, if you wanted...oh, I want to do a new service to this thing. You would get...like, everyone would just automatically just...

Sorry, Eddy just asked in the chat, "What is SOA?" service-oriented architecture. So, take your monolith. Instead of a single architecture, it's spread it out. And then, most people, when they do SOA, they'll take...instead of doing micro, they start tiny, but they get bigger and bigger and bigger. And then, you've got complexity inside each of these distributed services. So, that's where that came from. We had 3 different services that were putting records into the God record, right? Every startup has that one table, right? For us, it's like the contracts that we do. And in a medical company, it'll be your prior authorization request, that kind of thing.

And we had three separate services that were creating new items, new documents, which they needed to create, and I can't remember...they could not just stick it in the database and read it back to find out what the primary key was. They were, like, physically disconnected from each other, which means they had to come up with primary keys without colliding with each other. And for whatever reason, it wasn't GUIDs. So, these services had this...I want to point out this was in the last 5 years of my life or 10 years of my life, and I've been here for 4, last 10 years of my life.

Every time you created one of these documents, it would try to insert, and then we'd go, did it work? Uh, no [laughter]? That primary key was taken? Okay. Here's a new one. Does it...no? Okay. And, yeah, and it was fun because we were running out of key space. We had meetings about exhausting the key space. We had used all of the keys randomly through the entire search space. We had over 50% of them, and so we had this system that would retry 5 times, and it was failing 0.01% of the time, then 0.02%. And we're like, you know what? We're going to retry six times. We're like, this is not solving the problem. We are running out of places to randomly roll dice and try to hit a blank spot somewhere in the key space.

WILL: [laughs]

DAVE: It was a nightmare. And so, I came in and I said, "Guys, this is the textbook example for a microservice," a random number generator that can look at the database and see, is it colliding? No, no, it's not. Great. Oh, and, by the way, we could make the random number generator deterministic. They couldn't have ordinal because they were worried about ID prediction attacks. So, they had to be random.

And so, yeah, little microservice. It was five lines of code, baby. Fantastic. And everybody would talk to this to get their primary key and insert it in the database, and it was guaranteed to be available to them. Life was good. And everyone dug their heels in and said, "No. No more new services. No more." And I'm like, primary key on your God object table, if there was a reason for a microservice, this is it. And because we get away from why are we doing this and into this is the right way to do it, we stuck to the doctrine rather than to the principle. And, hmm, yeah, good times, good times.

MIKE: So, they'd done so many microservices that people refused to do another one, even when --

DAVID: Yeah, there were over 700. Somebody came back from DEF CON...I won't name names, but his initials are David Brady. He came back from DEF CON...I told my team, "You got to see some of this stuff." And I started talking about war gaming and red team versus blue team, and, you know, what's our disaster recovery. And the company was in the process of getting a second data center. So, all of a sudden, we went from 700 servers to 1,400 servers because we had to duplicate everything. And the war game plan was, if this colo gets nuked from orbit, we have to be able to fail over to this one. It was a nightmare.

And it took over a year to get to a point where we could get everyone together, and we had a war game night. And it was literally when traffic died down on Saturday night. We all lined up, and we started...we basically simulated an outage as politely as possible. We let everybody shut their services down politely so that you didn't have to do data recovery on the restart. But we took everything down out of the Atlanta data center and started bringing everything up in the Chicago data center, and those war games took 9 hours. We started at 6:00 p.m. And we figured this will probably go till midnight. No, it took till 4:00 a.m. It was a nightmare.

And the plan was to bring it down, move it over, bring it up, test it, bring it down, come [inaudible 26:17] back. And once we got up on the new data center, we were, like, "This is our new primary data center. Nobody ever move this again." And, a year later, they said we needed to war game again. So, every year, that company swaps data center. I don't know if they're still doing it that way, but that's...yeah. So, too many...literally, just the act of saying, "Where is the next service going to be?" It gets expensive. And it's a low exponent, but it is exponential. And if you get enough, that line will start to curve up on you.

MIKE: Well, I think that your story captures something, and it's been touched on a few times in this call, and it goes to the idea of coupling. If your services are tightly coupled, you have not separated out your system. You've just made the same system with a more challenging interface. I've heard it put that a tightly coupled system that's distributed is just a distributed monolith.

DAVID: I like it. I like it.

MIKE: You've got all of the problems of both monolith and a distributed system. And the coupling there is, I think, absolutely the key. If you can't have one of those services go down, then you haven't really extracted it. Anything that's a single point of failure is not really meaningfully extracted. It might as well be inside your system because you depend on it just as much as you did internally. And you have all of the cost of network overhead, and managing that service, and everything else that is involved in the cognitive overhead of having that separate. You really haven't separated it out.

There are tools, though, I mean, you can have...if you build your system right, you can, many times, have a service that can go down and nobody notices [laughs]. I mean, hopefully, you're monitoring it, and you notice, but the other services can keep running. And there's challenges there. You have to have some data redundancy. You have to have, you know, whatever. It's very situational-dependent, right? So, I'm not even going to drill into that.

DAVID: So, kind of a weird...I was reading on some stuff on how Amazon did their architecture way, way back when. They did things in Lisp because they had a bunch of crazy gearhead PhD doctorate candidates that loved to do stuff in Lisp, which is wildly inefficient, but they were writing this highly distributed, super scalable system, right? And it was because they had everything super, super tiny. The weird thing that they said that I have taken away and I just keep it in my back pocket...and I've forgotten the why. So, I'm going to throw this out there undefended and let you guys just pick it apart. And anybody listening to this can pick this apart but just think about it.

When you're building two Rails models, if you do a belongs to in one direction, it's just instinctive, right? You automatically go to the other one and create a...if it belongs to, then you go to the other one and do a has many, right?

TAD: Has many. Sure.

DAVID: Yeah, and the guy that was writing the...I want to say this was the microservices book that came out of Amazon from this guy. He said, "Resist that temptation." Bidirectional communication in CS is often ten times harder than single direction. And he gave a very specific example that if your customer has a shopping cart, okay, if customer, you know, has shopping cart, that's great. If I want to pull up your order page, I have to loop in the customer server, and I have to loop in the shopping cart page. But if it's bidirectional, then if you turn around and you look at the leaf node, as long as you know the customer name or stuff that you can go look up later, that's fine.

But if it's indexed and it requires this thing to be in the other present, like you said, you now have just duplicated monoliths. You have to have...that's what it was. He was specifically talking about having a database where you might have shopping carts and no customers. Like, you literally don't have customers in that database. You have to go to the service to get it.

But in the other system, the customer is there, and the shopping carts are there because they're linked, and they have to be efficient. So, that was the rule that I tried. It's like, I try not to just immediately do...and I want to say we actually have a RuboCop rule that says, if you put the one side in, always put the other side back in. And that's not always a bad idea, but it's not always a great idea.

TAD: I think it's pretty surprising to not have bidirectional stuff. Like, if you encounter something, you assume that that relationship goes both ways. So, --

DAVID: I could give you a better example. If you want to...so, we do leasing here, for those listening online. You need the customer record in order to service the lease. If you're in the call center, you need the lease, and you need the customer. But if you are at the contract origination system, you don't need any of the servicing stuff on the account because the account has...but you do need the customer information. And that was where the argument was.

And I agree with you that if you're down over in the servicing side, it should probably be bidirectional because you always need both. But if you're at the other end, single point, and that was the guy's push, was, like, by default, go single directional until you need bidirectional. And I've been caught in the neck with, you know, it's like, I just need this item, and it's over here, but there's no way to propagate. And so, you have to do a refactor PR to add that, so...

TAD: Well, I could understand you don't want to load everything up, but it seems surprising to me that you wouldn't want both things to know about their relationship with each other.

WILL: I'm with David. I'm with David. I'm strongly with David because, like, I don't know whether you guys saw the lightbulb turn on, right? Because one of the easy patterns that I've seen in this sort of database-centric things, I mean, we talked about a Rails application, but there are many where the lines of ownership get really muddy because you can get to anywhere from anything. And you'll find these situations where you'll get sort of, like, hop ons, right?

A better thing is I do have this sort of aggregator object. I've got, let's say, a lease, and I have things attached to a lease. Or I have a customer, and I have things attached to a customer. And it promotes very clear lines of ownership, where if I've got a customer that, I can get what I need from the customer. Or I have a lease, I can get what I need from the lease.

A way to visualize the thing that I'm thinking about, at least, is, like, think about an unstructured graph and traversing this unstructured graph where things can just go topsy turvy, any old place, versus a tree where I go from the node, and I could go to the leaves, but I don't go up, right? There are no cycles in this graph. And think about sort of traversing and just writing sort of traversal algorithms on an extremely structured tree, where it has rules, and the rules must be followed, like a bee tree or whatever.

TAD: But doesn't a tree have a node? Like, the parent node knows about its children, and the children know which node they belong to.

WILL: It depends on the tree, [inaudible 33:13]

TAD: It seems super weird to me that a leaf, you can say like, "Who's your parent?" And it says, "Oh, that guy." But I can ask that guy, and he's like, "I don't have any kids. I don't know of any relationship to my children." That just seems really surprising to me.

WILL: I'm thinking about really sticky, complicated, little data graphs, data associations because I've seen, I don't know, I've seen them go off the rails, where there are all these things snarled up together because there isn't clear ownership. There's not a clear, I am the container node that everything sort of spins around. And what I'm thinking about specifically is these situations where you need to put in...so let's say I want...I want to make a service call, and I have to attach a bunch of different extraneous stuff to my thing because it's like, oh, well, I have the shopping cart.

The shopping cart has orders, but I have to have the customer because the customer has the rewards account index, right? So, I can give them their rewards points. And then, I have to have the associated store because the order is going to be fulfilled from their preferred store. And you get all these things that are just sort of, like, there isn't a reason that they all have to go together, but because these things have not been organized...and, I mean, I don't want to be, like, I'm not a zealot on this. But, as a rule of thumb, making things a one-way hop, unless there's an overriding need for it not to be, that's really compelling as a guiding principle.

DAVE: I may have explained it poorly. Actually, I may have achieved exactly what I said when I started this story, which is I'm defending this poorly, and so it'll start discussion on it. But yeah, it's interesting to kind of push and think about, like, if you've got literally two services, one for orders and one for customers, one of them needs to know about the other one, but the other one might not always need to know, and that's, yeah, even though they do always have that relationship, yeah.

MIKE: Well, I think that's actually kind of important. Let me take your idea a little further. Let's say you have customers, and you've got leases, and you've got locations, and you've got merchants, and you've got shopping carts, and you've got, you know, any number of things. I think that if the customer has to know about all of those things, then it's just a big ball of mud. You've got all of that complexity tied up in the customer. But if you have a customer and it's just customers and your shopping cart is in some other service that handles shopping carts, and they have a reference back to the customer, so they know who the customer is.

But the customer knows nothing about shopping carts. Doesn't have to know a thing about shopping carts and doesn't even have the ability to look it up because he doesn't know about shopping carts and doesn't have to. But your customer stays simple, and you don't have to deal with that complexity. Shopping cart knows what the customer record is and could look it up if it needed to, right? But that's a one-directional relationship, and it doesn't need to be otherwise.

And where those shopping carts are very ephemeral, right? I mean, they come into existence; they pop out of existence; you have many, many of them, and they really have nothing to do with the core idea of the customer, that actually disentangles, I think, your architecture dramatically. And when you have all kinds of things, you know, and I could probably come up with 100 things that need to reference those customers, if the customer has to know about all of those, then your customer gets extremely complex, where I think that you haven't really decoupled, unless the customer doesn't have to know, right?

You've got something that maintains, here's your customers. And somewhere in a data warehouse, you can map everything together, right? Or maybe even in the frontend, you have some ability to tie things together because you can make the, you know, you can [inaudible 36:56] backend and put things together. But I don't think that whatever is holding, you know, managing your customers should know about what payment you made, you know, what your payment method was that was used three days ago. It gets entangled with something that has nothing to do with that. And that's the idea, I think, in the best sense of what Dave is trying to say.

DAVID: I think I might see a distinction here as well that if you talk to the entire system, you're going to have to know about both and the system output. But if you want to have a conversation about shopping carts, you can't just go talk to the customer service. You have to go talk to the shopping cart service. It's allowed to talk to customer, so there's that direction. But the other direction, there's no link. There's no way to link customers down to their shopping carts because that service is whittled down and kept very, very small. That's the piece that, at the system level, you absolutely can go both directions. I'm not saying sever the link in one way. I'm just saying this particular tiny service we sever it just for that service.

WILL: Well, I like this shopping cart customer kind of thing, right? Because there's a clear ownership thing, right? Customer has a shopping cart, right? But maybe case study, like, what I'm imagining here. It's like, okay, I've got, let's say, I'm trying to do some new feature from this shopping cart, and I want to go out, and I want to say, "Based on the stuff you've bought, I'm going to machine learning up some things so that when you're in your shopping cart, I can show you some other stuff because you've got cash in hand." And I'm like, okay, I got a hot buyer. Let me see if I can upsell them on some stuff based on the things that I know about them.

And so, then as developers start developing out the feature, then you start to tunnel through the shopping cart, then through to the customer, and then through to their purchase history. And now you're going from this shopping cart thing, and you're tunneling through, and you're getting into all this other stuff, which is like, whoa, whoa, whoa, whoa, whoa this is a little bit, you know, this is a little funky, and now you're sort of injecting these dependencies by going through the path of least resistance. Like, I have a shopping cart. I'm checking out, right?

So, rather than pulling in the stuff maybe kind of a top-level way, you're side-loading this relationship, and you're using that to tunnel your way through the system to avoid doing it on a straight-up way, where it's like, hey, I want you to, like, you know, like, you're bringing in a shopping cart, and I want you to bring in some recommendations for me, you know what I mean? Is this analogy coming off the rails? I'm freestyling a little.

TAD: I think what I'm kind of thinking about now is, I think that you need to know the relationship, but you also need to respect the relationship, meaning Eddy and I are coworkers. And he's like, "Hey, let's go for pizza," or something like that, and I'm like, "Hey, yeah. Can I borrow some money? [laughter]" He'd be like, "Sure," right?

But if I reach into his pants and pull out his wallet [laughter] and then grab that money out of his wallet, I've violated that relationship, right? I think it's important that I know like, oh, Eddy and I...Eddy is someone I could borrow money from, right? But if I'm going past that interface of that relationship, I'm violating the boundaries of that relationship. So, I think, that's kind of how I'm seeing it right now.

WILL: Well, yeah. I think we're all seeing it. I think we actually are vigorously agreeing because, like --

DAVID: Yeah, violent agreement, yeah.

WILL: We all understand that this is maybe a guideline, but not a hard and fast rule, which is why it would be totally fine. And you were like, "Hey, I need to borrow a couple of bucks for lunch." And Eddy was like, "Yeah, yeah, yeah, just go grab my wallet out of my jacket. It's over there," right? Like, obviously, that's not a blank check to just get into Eddy's pockets anytime you'd like, but, like, reasonable people can make exceptions.

And the thing that I like about not doing these bidirectional relationships by default is it sort of puts guardrails on the relationship where you need to do it on purpose. Like, I'm doing this on purpose because these things are so coupled, and I actually think shopping cart to customer might be. In the end, it might be one of those such tight relationships, where it's like, yeah, we should probably go both ways.

MIKE: Well, it depends on your business.

WILL: Yeah, exactly. But, on the other hand, I don't trust you guys with nothing. Like, a big piece of my life is just hiding the sharp objects from the junior developers.

EDDY: Jeez, well, I don't know, man, if someone's reaching into my pocket, I want to be able to reach into theirs, too, you know?

[laughter]

DAVID: That's a bidirectional relationship, yeah [laughter].

TAD: That's a violation.

WILL: [laughs]

DAVE: I just want to say I'm so happy to be on this call with y'all and to not be the person who launched the sentence about reaching into somebody else's pants. That's normally Dave Brady brand but thank you. You did give me a...I'm sorry, go ahead.

MIKE: If HR might need to get involved, then it's a violation of best practices.

DAVE: Yes. Yes.

WILL: That's why I work remote, man.

[laughter]

DAVID: I had a light bulb moment when we were talking about this, that if customers and shopping carts are coupled...and stop me when you've had this...we've had this pain. Everyone in this room, I'll wager...I know, maybe not Will, and maybe not Kyle. I don't know how much coding you're doing lately. But everyone else on this call, I know you've had this problem where it's like, oh, I need to, Will talked about the shopping cart, I need to calculate the shipping and, in order to calculate shipping, I need to know how many boxes and how big the items are. So, what boxes so that I've got to do the packing on it, and then I need the weight on this.

We can go to a shipping calculator. It needs to know what state we're going to send it to, and that's going to come from the customer. But other than that, the shipping service doesn't need to know anything about the customer. It just needs...and, honestly, the only thing it needs to know from the shopping cart is what items are in it. It needs an inventory list.

And if you can say, "Here's the state, and here's the list of items," that service now doesn't need to know anything about the shopping cart or the customer. But we've all written specs, where you're like, okay, I want to do this. Oh, I need...this is a shopping cart. Oh, I need a merchant. Okay, in order to have a merchant, I need a location. And to get a location, I need a merchant, to get a merchant...oh, I need the legal paperwork that says, "The merchant has been onboarded and has signed the paperwork." Because they're a valid merchant, they're supposed to go through the system, so they're going to be checked at every point.

EDDY: You don't have to worry about that if you use factories, Dave.

DAVID: Right? Exactly. Exactly. So, just, you know, we'll just do everything with factories, put it all in the database. And do you want half-hour spec suites? Because this is how you get half-hour spec suites [chuckles].

WILL: And also, the other thing about shipping is like, maybe it's not for me. And I also said it, like, because I could speak as a large electronics retailer. We do a lot of shipping, and let me assure you, shipping is much more complicated at the state. There's lots of places [inaudible 44:17] you can get stuff the same day, and there's a lot more places in Utah you cannot [laughs]. Like, my dad lives in Michigan, but he lives in the part of Michigan where Prime is not a thing. You'll get it next week, period [laughs].

DAVID: The last mile problem we were all talking...like, in 2008, we were talking about the...whoever solves the last mile problem...this is before Uber existed, right? I might have that year wrong, but, you know, before Uber existed, solving the last mile problem was the reason why there were only three big, you know, FedEx, UPS, and DHL were it for shipping stuff, and now it's all super federated, and everybody's distributed.

I tracked a package on Amazon yesterday, and I saw a new status update, which said, "Delivery appointment scheduled," And I'm like, "What?" I'm used to carrier picked up package, package in transit, received at facility, left facility, you know, out for delivery. That's all you need. And, all of a sudden, we've got a delivery appointment. And I'm like, somebody's working on that last mile problem, and it's not well...what is it? The future is here; it's just not well-distributed, right?

There's still places...there's a place in Nevada that does not have cell phone services. There's only four pairs of copper going in and out of the town. Great Basin National Park, that's what it is. There's a little town there and a little national park. And yeah, I remember pulling in there, and I'm like, oh, my phone doesn't work. Let's have a modern horror movie now. We've now solved that problem.

WILL: Yeah. I mean, Great Basin, I want to write that down. Like, I want to go camping. I only want to go camping places if there are no bars.

DAVID: It's a...not to turn it into an ad for this, Great Basin National Park is a glacier in Southern Nevada, in the middle of the desert, right adjacent to the Mojave Desert. It's a glacier because it's on a great, big, volcanic talus cone. And you're just driving across the desert, and then, all of a sudden, there's this mountain sticking up out of nowhere, and there's a glacier on top of it. It's crazy. And Lehman caves, which is gorgeous, so...I may have vacationed there recently.

WILL: I'm underlining it [laughs] --

EDDY: So, I kind of want to...Sorry, Will. I just kind of want to go back to where we initially started. So, I guess, essentially, there is no rule of thumb to follow, right?

DAVID: Maybe not a rule of thumb. Well, there's lots of rules of thumb. You just have to remember that they're all based on principles, right? You can distribute something. And if you distribute it in such a way that the distributed nodes have very slow communication between them, and they don't do very much communication, you're great. You can do all your high performance in your silos now, and everything's great. And if you do it wrong, you're going to take the high-performance stuff and then stretch it over a boundary, and now everything is over HTTP that used to be in the CPU cache. Ugh, been there, done that.

WILL: I don't know, I've got a rule of thumb. I've got a rule of thumb in terms of, like, monoliths versus distributed services. And this is going to be on the record hot take right now: You should keep it as monolith for as long as possible.

DAVID: I agree.

WILL: Because, as I see it, the fundamental trade-off that you've got between monoliths and distributed services is you're levering engineering output versus sort of management and the ability to communicate, you know what I mean, to coordinate and keep everybody pulling in the same direction and keep the children from fighting with each other because it's just natural human nature. And, in the end, this is probably my engineering bias, but I've run into a lot more effective engineers than I have effective engineering leadership and [inaudible 48:12], you know what I mean? I see you, Kyle. You can hide behind your hand all you want, but I saw that [laughs].

EDDY: Isn't there a cost, though, to having a monolith, though? Like, if you say, like, "Hey, no, like, I have a monolith that was the first service that we developed for their startup. It's been developed actively for 10-plus years," right?

DAVID: Sure.

EDDY: And you're like, oh, go and develop for this feature, right? Suddenly, isn't there, like, a higher ceiling to come in and get acquainted with, versus if it had a microservice, a true definition, to Dave's point, to a microservice, right? Wouldn't it be easier for you to hit the ground running because you know how to scope or comb through all this [inaudible 48:57]?

TAD: Well, the company I worked for before, and it's out of business, but I won't say their name anyway, they had the entire company was an entire single Go codebase. And you're talking 60-70 developers all working on a codebase that has to compile. And so, every subdirectory had a README file, and it's like if you want to touch the code in this folder, you got to get sign-off from these three people [laughs].

DAVE: The stakeholders.

TAD: And that's how they --

WILL: Stakeholders, David, yeah.

TAD: Like, took care of it. And it's like, oh my gosh, like...

WILL: So, what you run into doing that stuff is you'll have these sort of rather than management fiefdoms, you'll have these engineering fiefdoms, where you have this increasing important hardcore set of engineers that know everything and everything has to run through them. It's just like parallel programming.

TAD: Well, I'm just saying, like --

WILL: People will max out, and you'll feel it, you know?

TAD: It felt like the worst parts of a monolith, where I want to change these three files, therefore, I have to get, like, eight people to sign off on it. Oh, two of them are on vacation right now, well...Whereas if it were a single service with five people, any other four people on my team could probably sign it off, and we could roll it out that day, right?

DAVE: Yeah. To Will's point, there's an agile principle, which is to try to defer your decisions as far as possible. The idea is to defer the decision to the point of maximum information because the information increases monotonically over time. So, if you make a decision early before you've got all the data in, and that decision commits you to a path, and then you find out you were wrong, it can get very, very expensive.

In SOA, once you split, if there's a decision that has to be made on both sides, that affects both sides, that's twice as...well, more than twice as expensive, probably four or eight times more expensive because they both have to change, and they both have to communicate. We've all had that thing where you have to do a tandem deploy because my service wants to change this, and it'll crash you if I do it before, and da da da, and that whole game.

So, to Will's point, yeah, I think you should start small and start monolith, until there is pressure to do. At Acima, we have a great, big monster service that has been calving icebergs off of it. It's not even a monolith now. It's a continent on it, and it's just, you know, calving little subcontinents across. And that's a great way to do it. I don't know if it's a great way to do it. It's exhausting, and it's painful. And we have architecture meetings every week where we scream about it. But it worked. It started small. It got big. It got too big, and then we broke it down.

And if we had started out eight years ago breaking things into, okay, you're going to be this service; you're going to be that service...the first time we tried to change a low-level thing, like, the first time you try to move outside the United States to do business, and, all of a sudden, you need postcodes instead of zip codes, right? If you have to change that in every single service and there's validations that are hard that will crash your service if that's wrong, in every single service, it's going to be a nightmare. If it's a monolith, just update the zip code table, right? Just, right? Less expensive if it's all in one place. Yeah.

TAD: How do you know when to start slicing off baby services from your service? Is it complexity? Is it team size? Like, what's your rule of thumb?

DAVE: I let pain be my guide. Like, when something hurts long enough, when something hurts, I kind of make a mental note, and if it hurts twice or three times, I start going, maybe we should fix this, right? I personally am constantly driven to just polish out the tracks where I'm frequently going. So, if I'm doing something, like, pairing with somebody and I opened up a PR, and then I went back to my terminal, and I typed git set PR number, like, why did you do that?

And I switched over on another branch, and I did git open PR. And it opened a browser to the pull request for that thing. And I'm like, well, it's because I wrote a little thing that ties branches to PR numbers. And I'm like, yeah, that's hard work to do that. I'm like, yeah, but it's really painful to not have this. And, for me, it was less painful to stay up over a weekend writing a little PR management, you know, a little number associator.

WILL: I think if you talk to your senior engineers, you'd be like, "What sucks?" Like, everybody –-

DAVE: You'll get answers [laughs].

WILL: Yeah. And they're going to be...I'm saying there's a whole, like, how do you chip away your icebergs? That's a [inaudible 53:56] question that we could spend a series of podcasts on. Honestly, I'm more interested in, how do you maintain the communication and collaboration among teams? Because I have seen the distributed services architecture is a tremendous leadership challenge.

And I've seen so much just...we could just call it deadlocks, just like parallel programming. I figured it exactly like parallel programming and sort of deadlocks, and race conditions, and stuff like that with these sort of distributed things. It becomes a leadership problem. I mean, everybody knows the reasons why you should start branching out services, why you spawn a new threat. But how do you keep these things together? How do you keep the teams working with each other, right?

DAVID: Let's do that next week. I think that could be a whole call. How do you keep parallel teams productive? Yeah.

EDDY: I guess to kind of –-

WILL: Absolutely. To keep people from fighting, you know, keep people accountable.

EDDY: I guess the answer really is...it was a long-winded conversation to just say, "It depends."

DAVE: That's the other name of this podcast, a long-winded conversation to say, "It depends," yeah.

WILL: Well, yeah, I mean, it always does. There's no right or wrong in engineering. It's just the right tool for the right job. And, believe me, there can be a wrong tool for the job, and you're going to feel that [laughs].

DAVID: Mm-Hmm. Cool. Well, we have lost our host. Mike got called out to a production issue. Thank you for listening and watching our podcast. We appreciate it.

In this episode of the Acima Development Podcast, Mike shares a personal story about his father, a custom staircase builder, to illustrate the importance of balancing speed and thoroughness in work. Mike’s father, known for his meticulous craftsmanship, often took extra time to perfect even the smallest details of his projects. Mike contrasts this with a quick-fix job he once helped with, where speed was prioritized over perfection to meet a deadline. Both situations were appropriate for their respective contexts, highlighting the need to adjust the level of care based on the type of project at hand.

The hosts dive deeper into this theme, exploring how the right balance between velocity and thoroughness can differ depending on a company’s stage of growth. Will points out that startups often need to prioritize speed to survive, while established companies must focus more on quality and performance. They discuss the challenges of maintaining speed when stakes are high, particularly in large organizations where even small inefficiencies can have significant financial impacts. The conversation emphasizes the importance of adapting processes and expectations based on the specific goals and constraints of the work.

Finally, the episode touches on the crucial role of communication and knowledge sharing within development teams. Pair programming, mentoring, and asking questions openly are all identified as valuable practices for ensuring that team members can move quickly while maintaining quality. The hosts stress that leadership should stay involved in the coding process to maintain a connection with the work, encourage a culture of learning, and foster an environment where mistakes are seen as opportunities for growth, ultimately helping teams strike the right balance between speed and quality.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I am hosting again today. With me, I have Dave and Will.

DAVE: Howdy, howdy.

MIKE: I'm going to start with a story, actually, well, two kind of connected stories. I'm going to talk about my dad. He's a builder. Well, he's retired, but [laughs] he spent most of his career as a builder. And most of what he did was custom stairs. Most people don't want custom stairs [chuckles]. It's the people who can afford the custom stairs who ask for the custom stairs. So, he did a lot of work for, you know, people who can afford to pay to have custom stairs done. And the people who pay to have custom stairs done are looking for a certain kind of work, right?

WILL: Gone with the wind, grand entrance, duh. Yeah, I didn't get the Scarlett O'Hara dress for nothing.

MIKE: [laughs] So, circular stairs, you know, he did the spiral stairs, and that was kind of his specialty. And it's not just my dad; he did it with his brothers. He's got two brothers he did with. And they, you know, built stairs for years. And in my...well, I have personal experience. So, my dad built a house to sell, and this was in the late '70s. Everybody knows a big economic issue happened then, and people weren't buying. So...

DAVE: That was like the gas crunch, right?

MIKE: Yeah, exactly. So, the house didn't sell. He ended up moving in, and he's still there. [laughter] So, he probably over-housed, but he's been there. But, you know, he just can't...he spent his career doing this custom work, this custom, you know, the best quality you can get, right? And he's continued to work on the house. Can't help himself [laughs]. And so, everything he does, the floor and the kitchen, it's amazing. But you wouldn't pay for it because you couldn't pay for it.

He found an apple orchard where they were getting rid of their trees, and he cut the trees into layers and dried the wood, and then sanded it down, and cut it into the right shape. Well, cut it into shape, then sand it down because it warps after you've cut it. And so, it had to be cut thick, and then process every one of those pieces to get it to the right shape. And it's like every one is, like, at the heart of the tree. Once it was sanded, laid it all down, put epoxy on it instead of, like, your standard polyurethane, so it will last forever. It is just absolutely stunning.

But it's the thing you can only get by doing custom work, right? And when I've helped him...and I've helped him a lot. I've done some work there, and I, of course, worked with him when he was working outside the house, and [laughs] his standards are exacting. He's not a jerk, but [laughs], you know, he does quality work. And people work for him; he wants that work done.

And so [laughs], the work is great. Everything he does is done, exacting perfection. A lot of the work he does himself because he knows he's the only one who will give it that kind of attention [laughs] to actually get it done the way he wants it done. The one time when I've been doing a lot of work with my dad, I went and helped a cousin of mine who was moving out of the trailer he'd been living in. Though it was kind of a dump, and they wanted to make it look nicer before they moved out.

And I remember I was putting in some molding along the floor, just some cheap, little, thin stuff, and they asked me if I could do the corners. I was like, "Yeah, sure." And the speed at which we did the work in that trailer...I was getting together some, you know, some other family folks. And we got together and quickly did a whole bunch of work, you know, got together in three hours and redid all kinds of stuff in this trailer.

The floor was totally out of true, right [laughs]? So, trying to get that corner right was nearly impossible. I think, eventually, I put some sand under the linoleum to get it [laughs] leveled so [laughs] [inaudible 04:11]. And we got it all together. It looked a lot nicer, so we could sell it. And we were done. And here's the thing: in both situations, the right work was done. If you're going build a custom stair for somebody who wants a custom stair, well, you're going to give that every little bit of attention to detail that it deserves. If you were getting that trailer ready to go so you can quickly sell it, you're going to do that quick because that's what it deserves.

And that's not to say that the trailer doesn't deserve care, but it would be incongruous [chuckles] with the rest of it. It wouldn't be worth the value of that whole structure. You know, there's people who actually make a small home look really nice. I've seen that before. This one was not. This was not one of those homes [laughs]. This was not [laughs] one of them. But both of them were right.

There's a balance that you have to strike based on the work at hand, and sometimes you got to get that done. And, in fact, most of the time, the work we do is probably more like that trailer, right [chuckles]? We need to get the work done. And I'd say this: I say more like, I don't like to do shoddy work. I want to do quality work, but some things need to get done.

But then there are other things, you know, if I'm building something that has anything to do with money [chuckles], I want that to be done right. And I want somebody inspecting that [laughs], going over with the white glove to make sure there's no dust on it, you know, because you do not want to mess that up. And, you know, there's a time and a place for both.

What we're going to talk...I'm leading with a story. I haven't even talked about our topic today. We're going to talk about the trade-off between velocity, you know, between working fast, getting things done, and being thorough. And I thought about, you know, the stories because it is a trade-off. If there was a clean answer, then we'd all be doing it.

And the problem is we often tend to err on one side or the other, right? We tend to maybe go the wrong way for the work at hand, and we get in trouble. And sometimes, you maybe get...and we've probably all worked with somebody who had the work to do, and then they just didn't do it at all, you know, they're off [inaudible 06:37] [laughs].

I did some construction work in New Orleans area. I used to work for a construction company in New Orleans. And there was one job we'd worked on it for months, and, basically, it was done, right? It was done, but we weren't ready to move on to the other contract. And there was a couple of days where there was nothing to do. So, they said, you know, "Go around and look busy." I spent two days polishing the outlet covers [laughs]. So, when somebody came in, it looked like we knew what we were doing.

And there were some people who probably spent more time polishing those outlet covers on other days [laughs] than they should have. There's that problem, too. I think we may delve a little bit into that, but starting with the topic of balancing this work of, you know, this balancing between being thorough and being fast. I've been talking a lot. I heard that Will had a story. I'm interested in what y'all's thoughts are and how you'd link it up.

WILL: So, more I have a thought to share because I think a big piece of the thoroughness is sort of the phase of the company that you're in, right? But when you're in a startup and you're throwing features out the door as fast as you possibly can, and maybe you're going to take down the app; it'll come back up, there's not that much...you have different set of risk, right? Like, the risk you have now is you're fundamentally insolvent. You're going to run out of money. You're going to run out of time. And you have to figure out how to make a product that is worth buying. And that is your existential risk.

However, if you're working for a large, multi-billion dollar company and you screw something up, and your page loads 5% slower, you know, there are direct correlations to sort of website performance and click-through rate, and conversion, sales conversion, right? Like, Amazon did studies on this stuff, and if your stuff is too slow, people don't buy. If your site is not fast to navigate around, people will leave. And so, that stuff is really, really important.

And so, I'm talking about sort of a thoroughness-fitted finish kind of a thing. I'm not even talking about your stuff crashing. I'm talking about your stuff not being fast enough. It's like, oh, well, what if I lose five basis points worth of sales because I just didn't do as good a job as maybe I could have? Five basis points. A basis point is a percent of a percent, right? I just burned my salary for the year, just like that.

This is the phase of the company, and there are indisputable properties of just having a going concern that makes a lot of money. The impacts to the bottom line are substantial. You do need to be thorough, and I would...I always advocate for my developers. Like, if you've got a billion-dollar business, you should pay more for the people working on your site because it makes a difference, whether they're good or bad.

And one of the things...and maybe I'm going to answer your question with a question, or maybe I'll just add on another one. How do you maintain velocity in a situation where you have a big enterprise; you have a going concern; you have stakes? If you screw something up, how do you keep that velocity? Because it's easy to analyze yourself into a state of paralysis. You still need to innovate. You still need to develop products. You need to be, you know, delivering results on a quarterly basis. Because, on every level of the company, they demand that, and that is what you're getting paid for. How do you balance that, right? Because both things are true. You can't screw up, but you got to get things out.

MIKE: That's a good question.

DAVE: I keep coming back to this idea of, what are you really trying to do? What are you trying to accomplish? If you're in a startup, you're not going to be taking on IBM or Amazon. You're not competing at that level. And there's a phrase that you'll hear a lot which is, "The ideas that got us here won't get us there," when you're trying to scale. And the ideas that got you out of startup mode into productivity those ideas will not get you to scaled enterprise-level stuff. But the reverse is true.

 When you transcend the old ideas, don't discard the old ideas because what gets us there won't get the next cohort here. You end up transcending, and then pulling the ladder up behind you. I don't know if I'm making sense with that, but it's like, you start saying, "Hey, we need to have SOX compliance. We need to have peer review. The person who writes the code should not be deploying the code."

And then, you bring in a scrap team that are going to do a skunk work thing of like, "Hey, we want to stand up a quick, little server that will just check this thing on this rating service." And you're like, "Okay, cool. Where's my SOX team? Where's..." no, no, no. You have 30 days to ship this, and the right solution is to just ship it. Cut features until you ship. When you have to play at that level, like, if that little service is going to connect to the accounting database, okay, no, you cannot ship this in 30 days. That is not a working solution. But if you're just trying to do something like, oh, I just want to check your membership loyalty points, that's easy. It doesn't need auditing. It doesn't need review. Just ship what they need.

I'm sure you've all heard this. Growing up, I grew up in a kind of rural, you know, redneck part of the country. And I always heard all the time, "If you don't do it right the first time, when will you have time to do it over?" And then, I got into the software business, and I realized, no, the mantra in startups is, if you don't do it wrong fast, when are you going to get the cash flow to fix it and do it right?

And so, yeah, the ideas that get us to here are not the same ideas that will get us to there. And you need both, but not at the same time. You have to stay alert to, which mode am I in? Am I in enterprise, secure mode? Am I trying not to lose the eggs that we've collected into our basket? Or am I desperately trying to forage quickly and get as many eggs into the basket as I can, and then we'll sort out if I accidentally stole your eggs, or whatever; we'll sort that out later? But at the enterprise level, I don't need that level of security. I have to get the eggs collected. That's 17 metaphors rolled up in a weird ball, but there you go.

MIKE: You know, you're talking, Will, yes, you know, how do you keep the velocity going when you have those constraints? Another story [laughs]. I ride bikes a lot with my kids, bicycles, not motorbikes, but, you know, pedal-powered. And I discovered something great [laughs], and it took me years to find this, and I'm glad I did.

You can attach a tow...it's not a rope. You can attach a tow cable. Basically, it's a bungee cord that's wrapped in ripstop nylon that's been scrunched, so it's tough. You got the bungee cord inside. So, you don't have all the problems with a bungee cord, where it's going to have that hook, and it's going to tear your eye out [laughs]. But it, you know, it expands and contracts.

So, I attach it to the back of my bike, and then I attach it to the bike of one of my kids, right? And then, I've actually got two of them, so then I attach it again in a daisy chain [laughs]. And I've got my three-year-old that I'll put on a seat on my rack. So, I've got three kids, you know, all in a line behind me, and I'll go for a ride. And I've been doing that a lot over the last year.

And I'll tell you, the first time was hard [laughs] because that adjustment from riding, you know, riding with no constraints on you to riding with three people pulling behind you is a shift [chuckles]. And I even had to make a change. The bike that I was riding a lot...well, I've got a fat bike. So, I've got fat tires that are, like, four and a half inches wide, so I can ride in winter right on the snow, packed snow. It still doesn't work in too deep snow [chuckles]. And I've done a lot of riding on that and some mountain biking stuff. Even though it was made to go up some steep stuff, it wasn't geared low enough. So, I've got a gravel bike that's got a...lots of stories about bike, but we need it for context here [chuckles].

DAVE: It's okay.

MIKE: It's got a huge gear ratio. It's got a gearbox. It's got a huge gear ratio. And I can go really slow. And I can go pretty fast, too. But I can drop that thing down into an incredibly low gear. I have never gotten to the bottom gear yet, even pulling three kids, even going up a steep hill. I've never gotten to the bottom gear. I rarely get below...like, the lowest I ever get is, like, fourth [chuckles] out of 12. So, I'll give you a sense that, you know, this thing goes way low. It's awesome.

But [laughs] I had to change the way I rode my bike so I could keep moving forward, right? I had to change. And then, the reverse is also true. On the occasions I go out alone, I, like, almost fall down [laughs] because I'm used to having all of this mass behind me. And I have to rethink my riding style. We've got the same kind of thing, you know, thinking about this velocity. One time, I went on a longer ride with my kids, and I went up this, like, mile-long hill. And there was a headwind, a really strong headwind. And I was going mostly up that hill at, like, between three and five miles an hour, so, basically, at a walking pace.

DAVE: Just fast enough to keep the bike from tipping over.

MIKE: Exactly. Fast enough to keep the bike from tipping over. And if you were trying to do that, like, most people on a bicycle would just agonize over that because you're just not getting...that spot in the distance is not getting there very quickly. But the important part was not to get there quickly. I had a headwind. That was just the reality, right? I'm going up a hill. That's the reality. I've got all this weight, and this is the reality.

The important thing is that I keep pedaling. And as long as I'm keeping on moving, you know, your brain, you can adjust to it, right? You can adjust to these new constraints, and you just keep on going, right? You're not going to go as far. You're not going to go as fast, but it doesn't matter. The important thing is that you don't stop. Because you could think, well, I'm not going very fast. What difference does it make if I just stop? Well, it makes a huge difference [chuckles]. Stopped is very different from moving forward.

We're going to be in different situations, right? Sometimes we're going to have the headwind. Sometimes we're going to have a bunch of constraints we have to deal with, a bunch of weight we pull along with SOX compliance, you name it. But you can still set up the processes and run basically the same, right? If you're moving forward, you're moving forward.

The key thing is that you never get in that situation like, well, we're not going as fast as we were over there. So, what we're doing doesn't work. We've got to break everything, either by stopping because, like, well, this doesn't even matter. Very bad choice. Or saying, "Well, we need to completely reorganize our process. We need to get our velocity up to exactly this." Well, that's not realistic either. Because if you've got all these constraints, I think they're there, and you got to just deal with that. And you can adjust to it. Like, you can adjust to a lot, as long as you're keeping on moving. And it's really critical not to try to compare different kinds of work because they really don't correlate.

DAVE: You just connected two epiphanies that I've been carrying around for 20 years. This is exciting. When you mentioned that the important thing is to keep peddling, to keep moving, I had a job very early on in my career. I was moving out of, like, minimum wage jobs and into, like, my career. I was actually writing software for a living. I was on the networking side, so I was pulling cable as well. I was literally, like, crawling around behind people's desks and dragging cable. And if there was a hardware problem, I had to deal with the hardware. I ran a soldering iron at one point, right? I was all over the map.

And my job was to be the network guy. My job was not to break down boxes in the shipping room. My job was not to, you know, manage the sales cycle. My job was the networking stuff. And I came in and the stockroom in the back was getting piled and piled up and piled up with supplies of stuff. And I was in this mentality of, hey, I got to get this network stuff. It's not my job to do this.

And I walked in, and my boss was in the back room cutting down boxes and breaking everything down. And I'm like, "Wait a minute, if it's not my job, it's really not your job. What are you doing?" And he just looked at me, and he's like, "We need the space in order to get our job done. I'm an expensive stock boy, but I'm going to get it done."

And that was the epiphany for me of; sometimes the objective is, am I doing the job that I'm supposed to be doing? And sometimes, the focus is what do we need to accomplish? The how doesn't matter, but the what is what matters. So, I've literally carried that around in my head: I'm an expensive stock boy. I carry that around, and it has availed me very good over the years.

There's been times when it's like, there's this big, messy part of the project that nobody wants to touch, and I don't want to touch it either. But then I look, and I'm like, well, that is between us and done. So, I'm going to go be an expensive stock boy and just take care of it. And, usually, somebody higher up on the chain will go, "Ooh, that's an expensive stock boy. Let's get a stock boy." Or they go, "Yeah, we just needed that room cleaned out once, and it needed to be done. So, that was the right price to pay for it once."

WILL: I think that's a big...I've seen it a lot as sort of like, you know, teams specialize, right? And they sort of fragment into this archipelago of little islands, you know, that make up this country. And then, you have, you know, directors and VPs and stuff that they have their little region, their little fiefdom.

And I think that is one of the hardest things in terms of scalability, in that you still need people who take ownership of the entire organization, the entire app, the entire end-to-end experience. And, honestly, like, one of the things...I was just talking about this today. Like, where I think that breaks down really badly is sort of in the corporate hierarchy, like, the people you'll have are, like, VP or a director that isn't in the code every day, doesn't know where the bodies are buried, doesn't know how all these things interoperate. They have the visibility, but because they're not in it every day, they're very detached.

And I think there's a little bit of a black spot in the industry for somebody who is VP level or whatever your broad organizational level leadership. But they're an engineer. They're doing work. They will cut down boxes. Oh, this ticket was screwed up? I'm going to clear the ticket, and if you can't...I mean, bluntly, I know a lot of smart people, and I know a lot of smart people who've moved into management, you know, leadership positions. But you have to pay your dues. You have to pay the toll. If you want to ship code, you got to ship code. You got to put in your [inaudible 22:46]. Nobody gets an out.

If you are rusting, you know, then you're rusting. I mean, just like after a year on the bench as the manager of the team, like, you're going to have to ramp up just like a new hire would, maybe not that bad. I've seen it fixed places, you know, with things like principal engineers and stuff like that, but it's a systemic problem. And I haven't really seen a systemic solution other than senior, senior leadership recognizing the problem and being proactive and making sure those people are there and empowered.

Because a lot of the stuff you'll see is, they'll go around peeing in some Cheerios. There are a lot of people who have pet projects. They have things that they're doing. They have deliverables and stuff. And somebody comes in and says, "Hey, like, the interop between this department and this other department is all screwed up. Our codebase is all screwed up. We have technical debt. We've got performance and security considerations. And I'm going to have to like, you know, I'm going to screw up your quarterly results because this broad thing happens at like..." you know, there are a lot of smart people with interests in kneecapping those kinds of systemic engineering initiatives and accountabilities and stuff like that, you know, it's a ferociously difficult problem.

And I was contending with that earlier today, like, wanting that person in that role to come save me like Superman. I need heads cracked, and I don't have the juice to call these people, you know, behind the [inaudible 24:44] I can't do it. I know it's screwed up. They know it's screwed up. Anybody who's looking at it knows it's screwed up, but, like, the people that can can't [laughs], you know what I mean? I suppose a tangent, but a thought that was on my mind.

MIKE: Well, you said something. It's something I've thought about a lot. I've heard a lot of times, people who do hands-on, you know, typically blue-collar work, working with their hands, talk about how much the leadership doesn't get it. Like, "They just don't understand the work we do." And I can't count the number of times I've heard that.

And I think it's generally, like, I almost take it as a truism because I've heard it so many times that if you have somebody who hasn't done the work or is not connected to the work, they're going to miss things that are important. They're not actually going to understand what's going on. And, to your point, it doesn't matter how much you're looking at it if you don't know what to see. And it seems like there is a critical need for leadership to actually have either gotten their hands dirty or be willing to listen to the people who have and actually respect what they have to say.

WILL: Well, I don't want to be a jerk about it, but I will say this directly. You got to keep on paying the toll. Rust never sleeps. Every...I don't want to say every but, virtually, every senior engineering, leadership, management person that I have worked with I know in my bones started out as an extremely competent individual contributor. At one point, they were all good. They were all really good. I mean, once you get to the director and above level, present company accepted; once you get to that level and above, it starts to catch up with you because you know what your schedule looks like. Mike's shown me his calendar.

MIKE: [laughs]

WILL: When are you going to get your hands on the iron and do it? It's a luxury. That's like a day off. And if you don't do it, it catches up with you because everybody has to pay the same toll. If you want to be good, you have to continue sharpening the saw. Rust never sleeps. And that's when you run into these sorts of difficulties, the very difficult line to walk.

DAVE: It works backwards as well. I remember the first time I worked for a manager who had a string of managers in the early noughties and mid-noughties. They were, like, senior developers. And cowboy coders didn't get along with anybody, but they were very, very good. And they ended up in, like, CTO positions, and nobody wanted to work for them, right? Because they had no people skills. And the ones that had people skills weren't necessarily applying themselves to what type of battles do I need to fight at this level? They were still trying to focus down here.

And the first time I worked with a manager who came in and said, "Yeah, I used to be a developer, but now I'm a manager, and this is my job," and, like, the first three or four months working for him I hated because he would actually make me do one on ones.

WILL: [laughs]

DAVE: And I'm like, no, everyone knows that one-on-ones are uncomfortable, and nobody likes doing them. And after three or four of them, I'm like, this is really productive. I actually feel like I'm driving my career in a great direction because I'm actually doing this.

And that was the time that I finally realized that...and it goes back to what I said earlier. The ideas that will get us there are not the ideas that got us here, right? You need that reference. You need that empathy for the people working under you. And you need to be able to look beyond and say, "What..." It's the same thing. If you're not focused on what it costs to write code, you're going to have some struggles.

But if you are the manager and your job is to be aware of like, oh, AI is coming, and we need to be ready, or we're going to wake up, and our industry is going to be gone, you know. And do we want to have gone with it wherever it goes? And, I mean, that's one example. Somebody will react too strong to the mention of AI, but you get what I mean, right? There'll be something that, like, we're going to do this initiative.

We talked about this in a meeting earlier today about a manager who said, "We're going to pursue this initiative." And it was the wrong initiative. Turned out we got there, and there was no opportunity there. And it cost some people their jobs as a result, and that was no fun. And it was failure in both directions. Like, this person didn't understand the engineering, and they also didn't understand...they had a view of what ecosystem they wanted to put in place. And they managed to solve the wrong problem in both directions.

WILL: [laughs]

DAVE: Somebody said, "We're leveraging the inefficiencies of both approaches. [laughter]" That was, like, anti-synergy.

WILL: That's almost impressive.

DAVE: And I've been there. I've been on a team where I had a cowboy coder working with me, and I was, like, the meticulous, make sure your tests pass; make sure your code is expressive; make sure to actually dah, dah, dah. And there were times when I'm like, "You're going so fast, and you're breaking things." And he was like, "You're dragging an anchor when we need to get stuff done." And there were times when we got so much at loggerheads that we ended up shipping too little, too low quality and too late because we leveraged both of us. We both dug in our heels, and we ended up with nothing.

And I think I've told this story here, but I worked with a guy named TJ at CoverMyMeds who was astonishingly good at picking his battles. And he and I were really good friends, and he got really, really good at seeing when I was about to turn and go down a rabbit hole. And he would just kind of put his hand on the back of my belt, and say, "Whoa, whoa, whoa, whoa, wait, we don't need to go down that rabbit hole." And we fought a few times over [inaudible 30:55]. And I finally said, "You know what? You're right. We don't need to go down that rabbit hole."

He was one of the first people I've ever worked with where pair programming delivered on every promise pair programming has ever promised anyone. And we were also linearly faster. Like, the two of us could complete a sprint's worth of work for two people in, you know, a week and a half. We literally...every way we went faster because the synergy worked so, so well because we were solving the right problems, and we were leaving the wrong problems alone. And that...oh, so good.

WILL: I'm a pair programming seller, and I always evangelize pair programming.

DAVE: Amen.

WILL: I think it's good. It's not a silver bullet. It shouldn't always be everythings all the time, but I think it should be...I think we should do a lot more of it. You know, pair programming now, pair programming, you know, forever. Like, that's all. That's all I'll say on that. That's all I'll say right now.

[laughter]

MIKE: I got to say, it's one of my favorite things to do because the amount that you learn on both sides of a pair [inaudible 32:11]... a lot of times, you have a more senior, a more junior person, but it tends to flow both ways. The amount of knowledge transfer that happens there is unlike anything else that I've experienced in software development.

Like, you can read documentation, and a majority of that is just going to pass right through your brain [laughs], not that it's not technically possible to go read some dry technical documentation and get lots of clarity from it. But when you're watching somebody work, or they're watching you work and seeing how you make each of those decisions, it just changes that equation typically. You see all that weren't captured in the documentation that you just wouldn't see otherwise.

WILL: Yeah, well, I mean, and if it's your stuff, you know you didn't write everything down. You know you didn't do it. But, anyway, a lot of stuff you're talking about, I really do feel like a lot of those that, as line-level managers, I'm going to say flat out, I'll take my hot take; this is your hot take of the session, I think all line-level engineering managers should spend 25% of their week, maybe 20%, right? Eight hours a week, a full workday, every single week should be pairing with the people on their team so that they have both an understanding of like, okay, here's what's going on.

If you're a line-level manager, you should be a very proficient coder. You should be a senior-level engineer. You should be good at that. You can't be like, "I'm a VP. I don't care anymore." No, no, no, you don't get that out anymore. You should be providing that kind of judgment, that kind of insight, that kind of understanding. You should be working with your team and figuring out where everybody is. Don't [inaudible 34:15] like a one-on-one. Get in there and feel it.

All right. Are you writing tests good? Are you thinking about parallelism? Are you thinking about error handling? Are you thinking about design? You know, all this stuff. You get paid for that. That's why people pay you to be around. And it keeps your hands, you know, in the iron, and you have a feeling for how things are going.

I'm actually on a Slack thread right now with three, four staff engineers who are like, "Why are my builds so slow on these laptops?" And I have been fighting the good fight. There is a security team that has installed real-time virus checking on all engineering workstations. And it is virus scanning intermediate build products in real-time. It's a substantial CPU hit. I have been wrestling with these guys for a long time, many months. And these guys didn't even know. They're just like, "Wait. Wait, what? It's doing what?"

And I'm like, "Yeah, man." I've been arm-wrestling these guys to try and find a more economical accommodation to satisfy their security concerns while I can still ship these features. And if you never code, okay, you know what I mean? You're a director. You're a VP. You're up the levels. That's not actually your job anymore. You probably are wasting your time doing that.

But, like, line-level managers, I want your hands...I want you in the muck with the rest of us, at least a little bit, so you know what the hell's going on. Because even when that front line, you know, your NCOs or whatever you want to call them, you know, the people who are on the front line, if they've checked out, how's the information going to filter up? You know what I mean? Anyway, my two cents.

DAVE: It's how do you get feedback? Yeah.

MIKE: I've got one pushback against your hot take. I don't think 20% is enough [chuckles].

WILL: Yeah, sure. I mean, you're right. I agree. I think 25%...I think 50% is probably where I would cap it, but it's somewhere in between there. But I don't think...how do I put it? Like, I'm not asking for 30% and expecting to get 20%. This isn't like a negotiation where I know I'm not going to get everything I want. Like, I'm just saying what I need. This is the real number. I want a real 20%, and getting 8 hours consistently every week out of a manager's schedule to pair program is a substantial ask. That's hard to do. You need support. That is a rare bird to make even that happen.

MIKE: I stepped into leading an internship once, where the person who was leading it...I say an internship, it was an apprenticeship. There were some people that we wanted to bring in to be full-time engineers. They weren't coming straight out of college. These were generally people who'd maybe been in another profession, and they were ready to come and learn what they're doing. And I'm not the one who started it. The one who did left, and I stepped into it. This was a number of years ago.

My manager at the time said, "I think you should be spending 50% of your time writing code." And I laughed at him [laughs] because I was spending, you know, like, 80% of my time pairing with those apprentices. Because how else were they going to learn the ropes unless they were working with somebody? And --

WILL: Well, but that counts. Pairing time is coding time.

MIKE: That's true. Fair. Fair.

DAVE: I remember pairing with Eddy Lopez here at Acima when he was fresh out of it. He was fresh on the engineering team from QA. He might have still been on QA. And he and I were pairing, and I asked him something, you know, "Where's this in the source control?" And he wasn't sure how to...and he's like...And I'm like, "Okay, hang on." And I cracked open a terminal, and we spent an hour playing with Git. "And I'm like, here's where to put it. Here's where your sandbox is. Here's the distributed nature. I can be the server. You can connect to me, and I've got all the changes because da da da da da." And I was explaining where the deltas go and all that stuff.

And we were on the podcast a few months ago and Eddy pulled this story from his perspective, which is that he has paid that forward so many times. And I'm like, that was probably one of the most effective hours of my entire career because I invested in someone who then has now paid in, you know, bazillions of dividends. And I'm not trying to blow my own horn. I'm really happy that that happened. When Eddy was like, "Oh, yeah, you did this, and I love it," I'm like [vocalization], I'm going to go for a long time on that compliment. So, it's awesome.

MIKE: We landed into the value of sharing information. We've rerouted a little bit because [laughs] it was just such an important topic. Have we really hit how you keep that velocity going, how you keep moving? Because you asked, I think, a vital question, Will. And I gave kind of a metaphor, but I don't know if I got really into the technical details, how you do it on a development team. How do you keep that motion going?

WILL: And, you know, actually, like, so there's another issue. Like, I have a lot of meetings on Friday. I'm trying to put my meetings on Fridays, because, listen, man, I'm going to show up to work on Friday because they're paying me, and I will do it, but they're not spending their best [laughter]. It's a fairly linear degradation of my engineering effectiveness. I'm still not bad, okay? You got your money's worth out of me today, but yeah, you know.

And so, I was talking with my boss, so my boss long-time consultant. And he sort of, like, you know, went full-time and moved into engineering management. But he spent a lot of time as a consultant freelance, you know, developer, you know, that's me. Like I'm, you know, technically unemployed. Like, I've got a long-term contract with my large electronics retailer.

And it's sort of like my team, you know, that I run sort of runs things my way, which is sort of very like, what have you done for me lately? Like, because I'm just used to the mentality of, like, people are writing me a substantial check for my services every week, and I want them to know what they're getting all the time, and that gets my contract renewed. I got another year. And it's a mentality that sort of the boss shares, and I share. And he's working with another team, and they don't necessarily think about things the same way because they were not brought up in an environment of existential fear [laughs] the way we were.

So, I have a story, right? And it's my mom. She had dogs for a long time. She had greyhounds. Greyhounds were her favorite dogs. She had greyhounds. And she liked to get, like, rescue greyhounds, like, retired dog track racers. I don't know if that's such a thing anymore. I think we've kind of phased out dog racing, which is probably good. But, you know, they would retire. They couldn't run anymore. They tore their ACL. Whatever. And they'd just be like, well, we're just going to dump them in a landfill. But I guess if you want them, you can have them, right?

So, she would get these dogs. And they were the laziest animals that have ever lived. You'll roll home, and you'll have a golden retriever, and it'll just be ecstatic beside itself, jumping on you, giving you kisses in the face. Almost, if not literally, like, wetting itself because it's so excited that its people are home. It's so happy. The greyhound version of that was...they would lift their head off the ground as they were lounging [laughter], look at you, and their tail would wag three times [laughter], wag, wag, wag. Boom.

DAVE: Head back down.

WILL: Head back down. But every time you would take them out in the backyard to feed them, you would get the most magnificent, beautiful, joyous, absolutely astounding run. They would tear around the backyard in a figure 8 at 30 miles an hour. I swear to God.

MIKE: [laughs]

WILL: It was magnificent. Magnificent. The most beautiful thing you've ever seen in the world would be just to see the beautiful athletic power and grace of a greyhound, wholesale. Because they had grown up in this environment [laughs] of existential fear, where literally, if they wanted to eat, they needed to run like their lives depended on it because they kind of did.

And sort of like, you know, so my boss and I were talking about how can we...it's not that this other team doesn't have...it's not that they're not smart. It's not that they're not hardworking. It's nothing of the kind, but they don't necessarily have that mentality. And so, the question was like, we've got these sort of deadlines, you know what I mean? Economy is tough. Macro is tough. The business is like...it's not bad, but it's certainly challenging. Everybody is pushing really hard to keep their head above water. Everybody's working hard. And how do we get this team to do the thing?

And what he settled on, and it's been on my mind since we had this meeting, and now that we have this conversation, it kind of brings it full circle, what he settled on was he just sat down with them, and he's like, "Okay, you know, this is our thing." And he just started coding it up.

And he exhibited the mindset of like, okay, we've got to move fast, so let's move fast. Let's do this then. Let's just put it together. Let's get the scaffold up. Let's get the skeleton, the framework. We're going to make some...we're going to build a jig. We're going to make some operating assumptions that allow us to move forward. And we're going to sort of start the rough sketch, and then we start filling in the outline, filling in the details. We're going to paint the color, and the shading, and all those things.

And the only way he could really get this mindset onto these people, you know, which comes from decades of high pressure, high production consulting is to just get in there and show it to them, show them how it feels. And then, as he did it, the clouds parted. And they were like, "Oh, okay, I see what you need. I see what you want to do." And that kind of knowledge transfer, endemic to the work we do. That's why I'm a paring zealot because I don't know of a better way to do it. I don't know of any other way to do it. And I am fully aware of how fantastically indulgent it seems on the surface, and the sort of visceral, like, oh my God, I'm going to pay two of you to just sit there on a single keyboard all day?

MIKE: [laughs]

DAVE: That's twice as expensive.

WILL: Do you have any idea how much this is costing me, you know, to do this? But I just don't have a better --

DAVE: And then, you let that stop you out, and you're like, okay, well, I'm going to work alone. And you just kind of sit there and go, do you have any idea how much it's costing you to have me work on this without a second brain? It's, you know, yeah.

MIKE: Well, that's it, right? That what works is what works. It doesn't matter how much you don't like seeing 2 people sitting next to each other on 1 keyboard. It's what works [laughs].

DAVE: And if you're banging out 30 controllers with just RESTful actions, then one or two for pairing to make sure everybody understands this is how we roll this out. And then, yeah, go to your office and just I'll do these 15. You do those 15. Let's meet back up at 5:00, and we'll review each other's stuff because you're just banging out boilerplate at that point. But you do the first ones together because there's knowledge transfer going on, and yeah.

WILL: Yeah, absolutely. The question that we had was sort of, like, how do you get that mentality? Because it wasn't even a technical thing. These guys don't know how to do the work. They understand the work. They're very competent programmers. They're really good. But that sort of, you know, that piece wasn't something that they really understood until they saw it. And then, once they saw it, they're like, "Oh, oh, okay, I get it. I understand now. All right, let's go." And then, you know, boom, that's how I [inaudible 48:00]

MIKE: You know, the kind of meta stuff around software development is so vital. It ends being most of what we talk about on the podcast because [chuckles] it's critically important. But you're probably not going to learn it in school because it's...I don't know if it's...it's probably hard to communicate about but it also sometimes, I think, seems trivial.

You think, well, we need to teach you the hard stuff. We need to teach you about writing code, and writing code is critical, right? Learning to express yourself in that way is a change in mindset that does not generally come naturally. And you have to bang on that for years to get good at it. But if you just focus on that, you're missing half the picture.

Like, Dave, you talked about spending an hour teaching somebody Git. I mean, that is important, learning how to use those command line tools, you know, whatever tools that you use, you know, that are on the command line. Like, learning the tools for your job —that's important. And without that, you're not going to be able to work very effectively. And there are so many things like that.

WILL: I thought so many people like the debugging, you know what I mean? Like debugging. Like, hey, how to use a debugger. Like, some people don't like using debuggers. Like, you should know how because a well-configured debugger where you could just stop the code, it's like, whoa, whoa, whoa, hang on. What are we doing? Why is that X instead of Y? That's critical. Just the idea, the mentality of debugging.

I taught some kid just today, you know, another one of my golden bedrock principles, which Mike has heard out of my mouth many times, and I'll repeat it a thousand times more with no shame at all: Always debug from the known to the unknown. I had a guy, he was like, "Oh, my analytics, they're not working. I thought I turned this analytics thing off, but I'm still seeing [inaudible 50:17]. This is all screwed up." I'm like, "Known to the unknown," you know what I mean? And I'm like, "Okay, I wish you'd check out the whole component."

And he's like, "Okay." But the analytics are still fine, but known to the unknown, take out more, delete more code. You got version control. Rampage. Don't do the thing. And he's like, "Oh my God, you know, 15 minutes of just taking a [inaudible 50:40] at this thing until I finally made the noises stop." And I was like, oh, somebody duplicated it. Like, this thing exists in two places from, like, known to the unknown, always go from the known to the unknown. However, you can make it work, make it work, and then move from there.

I'm like, that's just the kind of thing where it's just like, I think, hopefully, if I got it into his head, that was another one of those, like, one-hour conversations that he'll be like, "Old man Archer said, 'Known to the unknown.'" And I'm going to pop up over his shoulder like Yoda, if I did it right. And I will save him a thousand times [laughs].

DAVE: Amen.

MIKE: It's interesting the arc this conversation has followed because we talked about velocity and balancing that. And, certainly, there's a balancing act. But we didn't really linger much on that idea because maybe it's clear in all these different kinds of work. We've talked about how to communicate with each other, and share information, and keep organizational structures effective.

WILL: Going fast is a feeling. You got to feel it. When you're on a team, and it's clicking, when you're writing code, and it's clicking, when you have the feeling of like, you know, mastering the command of your tools, understanding the organization, the needs, the codebase, it's a feeling. When it all comes together, you feel it in your gut. It feels good. I can't explain it to you, but I can show you how it feels.

DAVE: It feels like flow, right?

WILL: Yeah.

DAVE: You get into it, and you're excited, and then, all of a sudden, it's 6:30, and your wife is like, "Are you coming home?" And I'm like, "Oh, did I miss lunch?" And she's like, "Yeah, you missed lunch. Come home. You're missing dinner."

WILL: Like, you know, the mentality of like, you know, we need this thing out. We are going to make compromises. If we are going to cut some corners, we're going to cut the right corners, not the wrong corners. We're going to give an appropriate amount of spit and polish to this thing. How do you make those judgment calls? I mean, you just got to know it, and you can get it through osmosis by sitting here with me, or with my boss, with whoever.

DAVE: Sure.

WILL: But I can't give you a book and just be like, "Here you go." And that's the weird part of engineering; you got to feel it.

DAVE: I think it's interesting that we're in a phone call talking about velocity versus thoroughness and all three of us are bringing communication. And what I realize is I think at intermediate to senior levels in your career, interpersonal communication becomes the most difficult technical skill that you want to scale. And if I was having this conversation with you 25 years ago, I would be like, you got to know your text editor. You got to know how to do da da da da. You got to know this tool and that tool, and you got to know this other thing. And, oh, you need to learn this other thing.

And it was all the kind of things that a junior to intermediate level developer would be focusing on, mastery of what's right in front of me. And then, you start realizing, oh, I spend all my time tackling problems that are bigger than 12 people together. We need to learn how to communicate. And I'm no longer in a spot...

And the great thing is, we end up hiring juniors who are trying to master their tools and, like, don't bother me about communication. I got to figure out how to use this tool to do this thing. And you're like, yes, you need to keep mastering your basics, but while you're doing that, I need you to come talk to me about...And I think it's interesting that the higher up in the seniority you go, the larger the problems you take down, and then, all of a sudden, the problems that you've experienced are at that level instead of one level down.

WILL: Well, I mean, I also say this. I think one of the real bear traps in the industry, another [inaudible 54:48] difficult problem in the industry is not knowledge transfer from juniors. Juniors are actually pretty comfortable, usually. Usually, they're very comfortable. The really tough ones are the seniors.

MIKE: [laughs]

WILL: That's leadership with a capital L because, like, you know what I mean? You're going to take Ls. Things are going to move. Things are going to change. The needs of the organization are going to change. The needs of the industry are going to change. Like, things are going to move, and there's a natural human reluctance to look foolish to, like, be a beginner again, and again, and again, and again, and again, and again, and again. Knowledge transfer among seniors, and keeping seniors current, and keeping seniors flexible and empathetic, and stuff like that, that's the real challenge.

DAVE: Convincing somebody that this style of code is more readable than this other style when they are utterly familiar with that other style, and they're like, "Well, clearly, it's readable." And I'm like, "Nobody else on the team agrees with you. It's familiar to you. It is unreadable to us. Would you consider doing it this way?"

And you get that spot where, as a junior, you're humble, and you're willing to change anything. And you get to intermediate, and you're like, "What I know works." And then, to get from there to senior, you have to go, "What I know works, and what you know will also work, and we're going to do it your way because we need to move."

MIKE: And that is hard. I read an article the other day by a scholar of the Hebrew scripture. The specific text isn't that important. The important thing is that it was written a long time ago. And what he said was that there's this recurring theme that you have a leader who starts out humble, and they apply that humility, and they become very prosperous, and they get set in their ways, and it causes all kinds of trouble [laughs]. And it's not a new problem. It's not a new...but we still have to fight it, you know [laughs]? And we have to be willing to do some introspection and say, "I'm that guy now. And I need to be willing to humiliate myself."

WILL: Well, I mean...so, go ahead.

DAVE: I'll just say the biggest compliment, I think, I have ever been given in the last year was also from Eddy, ironically. I like Eddy. He says nice things about me.

WILL: [laughs]

DAVE: But we were in a meeting, and I asked, like, a bone stupid...I mean, like, the kind of question that everyone in the room was like, you're asking that? I won't go into the question now because if I don't give you enough context, it's going to sound like an offensively stupid question.

WILL: [laughs]

DAVE: And it was, it was borderline offensively stupid. But as soon as I asked it, all the juniors went, "Yeah, I want to know the answer to that, too." And so, we realized we had zero knowledge share. Like, everyone on the team thought everyone else on the team knew the answer, so they didn't want to ask. And Eddy turns to me, and he says, "You are fearless about asking stupid questions."

[laughter]

And momentarily, I was kind of offended. And I'm like, no, he meant that as a compliment, and he's sincere, and he's right. I'm willing to ask really, really dumb questions. If I feel guilty, if I'm self-conscious, I will say, "I know the answer to this, but just so everyone else knows, you know, it's like, how is this working?" And you got to be willing to do it. I've had people get impatient with a stupid question, but I've never had somebody think I was stupid for asking a stupid question. I've never had somebody think I was a coward for asking a stupid question, to put it that way.

WILL: Do you guys want a life hack for embracing your inner idiot?

DAVE: Oh yeah.

MIKE: Yeah.

WILL: All right, so this is what I do. This is, like, actionable stuff. This is the Slack life hack. I won't discuss ticket work in a DM, and I won't tolerate other people discussing ticket work in DM. If you got a question about how to do your job, it goes in a channel. Maybe the team channel. It doesn't have to be in a public channel. You don't have to put it on general, or whatever, but everything.

If it's like, "How does this work? Hey, do you remember how to do this API?" Anything. Anything at all. I mean, if you want to have a private conversation with me, it's totally fine. Let's talk about our weekend. Let's talk about the dumb thing that thus and such said in thus and such meeting. Like, "I can't believe David had to ask that, Jesus, that guy [laughs]." But, like, none. Ever. None. Zero. No chance.

DAVE: There's a countervailing force that you have to be very vigilant for, which is all it takes is one VP to say, "Just get it done," when somebody asks a question. Those questions will evaporate because, all of a sudden...we've talked about psychological safety before on the podcast, right? Where it's like, if you have someone out there...when somebody asks a question like that, the response from the people who know the answer should never be, "You should know this."

The answer should be, "Yeah, you should know this. Let me tell you. Thank you for asking this because I should have told you this." And that is so, so helpful is to be welcoming of the dumb questions because if somebody's not willing to ask a dumb question, you got a lot of people walking around with a dumb question unasked, which means they don't have the obvious answer, which means they're dumb. It's...yeah.

WILL: And, I mean, the other thing that is like, you know, to be perfectly honest with you, I mean, there's always teams shifts and teams ebb and flow, and people come in. And conventional wisdom that everybody knew six months ago it's like, well, you got three new devs on the team. They don't know that. How could they? They weren't there.

And sometimes I just forget, okay? Like, I'm lazy, okay? I am. I'm as lazy as I can get away with being. And also, also, also, it becomes, like, an unofficial Wiki. I'll search a Slack, you know what I mean? I'll search a Slack Channel. Oh, weren't we talking about that thing the other day? And then, I'll find it, and I'll probably post it again. I'll probably repost that link because that's just how it goes.

I mean, in terms of VPs just being like, "Just get it done," you want to be out of this thread because you got other stuff? The response to that is, "No problem, Jim Bob. We're going to work it out." I'm going to start another thread, and Dave and Mike, and I are going to figure out exactly how this thing gets done, right?

If VP is saying, "I'm delegating this," that's fine. The only thing I want to hear is, "We got you." And then, the team figures out how to do it, and, like, they don't pinged every 30 seconds as we're hashing it out. That's fine. If he wants to go cook, let him cook. It's fine. This is my job. I got it.

MIKE: I think that's a pretty good stopping spot.

DAVE: It is. This was awesome [laughter]. You know it's a good podcast when you want to do two more hours on the tangents that you've dug up part way along the thing, yeah.

WILL: Well, happy Friday, gentlemen.

DAVE: All right.

WILL: You all be well.

DAVE: Have a good one. Cheers.

MIKE: Thank you. Until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, the panelists join forces and discuss the things that scare them in code and how to avoid these pitfalls. Mike begins by recounting his experience with a fast-growing startup that used a poorly managed customer management system. This system lacked version control, searchability, and a test environment, which forced developers to tweak code in production. Additionally, Mike shares another experience with a codebase riddled with security vulnerabilities, illustrating the dangers of inheriting poorly written third-party code.

Will adds to the discussion by highlighting the broader organizational issues that can lead to scary situations in coding. He emphasizes the importance of maintaining a strong chain of trust within a distributed workforce and ensuring diligent code reviews. Will warns that when organizational control breaks down, it becomes a significant concern, especially with large volumes of foreign code entering the system. He argues that fostering a sense of ownership and responsibility among team members is crucial for maintaining code quality and preventing issues.

The conversation then shifts to strategies for cultivating ownership and responsibility in coding teams. Mike and Will discuss the importance of giving developers genuine control over their work to foster investment and care. They touch on the challenges of balancing team freedom with organizational needs, noting that allowing developers to make decisions, even if it means making mistakes, can lead to better long-term outcomes. Eddy also emphasizes the value of training and mentoring junior developers to ensure consistent code quality and adherence to best practices.

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I am hosting again today. With me today, I have Eddy and Will. There's a chance we may have some additional folks coming and going. We're kind of in an influx today [laughs], but we're going to start here. I am excited about what we're talking about today because I think it'll be fun.

And I'll just lead with the topic. We're going to talk about things that scare us in code. And how do we avoid having some of those things happen? Because I'm hosting, I get to start with some stories [laughs], and I'm going to start with stories that are long enough ago that the victims are not obvious [laughs]. And the companies are likely, well, more than likely; the companies are out of business, so we don't have to [laughs] be too sensitive about it.

There's one example I think illustrates kind of everything, maybe not everything, but many of the things that scared me about code. I worked at this place early in my career, a fast-growing startup, really fast-growing startup. They were growing like crazy, and they needed to have a customer management system to manage all of the new customers they had. And somebody knew a guy who [laughs] had built something, and they paid for it, and we got it.

And this system consisted of a Microsoft SQL Server having a whole bunch of custom extensions added to it and a whole bunch of queries so that the entire system ran inside of the Microsoft SQL Server. And some of the components they called out to were compiled code for which we didn't have the source code. And the areas for which we did have the source code, that source code was just stored procedures inside the database, and that's where most of the code was.

Some things about that. There was absolutely no version control [laughs], which means any code there is, the only chance you have to do anything about it is to tweak it in production because there was no test system. There's only the production system, and there's no code repository. There is only the production system. If you want to change something, you go change that stored procedure live and hope for the best. And [laughs] further, because it's just stored in the database, all you have is those stored procedures, and you want to find anything, there's no search system. You just have to try to figure it out. And they referenced each other all over the place. So [laughs], you had to go find them.

And sometimes you drilled all the way down, and you found a compiled component for which we didn't have a source code, and you just hoped it did the right thing [laughs]. Maybe it was sending out emails. Who knows what it was doing? You didn't have access to it anyway. And we used that thing for years. Oh, that was an interesting piece of software. I ended up making a number of changes in there that we needed. But I could go on. I'll mention some of those key characteristics again. No version control. No searchability. No test system. You can only test it in production. And it came from a third party. It came from a third party. The development team didn't have any choice over it.

At that same company, we inherited a codebase. Our company decided to bring it on to try to bring in some customers. And I didn't know about this codebase ahead of time, and none of us did. But apparently, they had some security holes before we acquired it that were actively being exploited. At least there was something like version control in this system. However, this was their SQL injection. They were handcrafting all of their queries. In this case, they hadn't sanitized their SQL. And this was in processing financial data. And they'd already lost, I think, over a million dollars before we found the hole. You imagine that didn't go over very well [laughs], especially this small startup. And probably led to the demise of the company shortly thereafter.

Some things in common between those. Notice the third party. Developers having no access until afterwards, so inheriting a codebase. And a lot of times, you don't acquire code from a third party, but it's handed off, and you don't have the insight, and lack of adherence to anything like best practices and difficulty in searchability. There's a couple of worse stories [laughs], but they also illustrate a lot of the things that I think are worst in code, scariest in the code, which are stuff that I don't understand, and I have to go dig in because I'm inheriting it, and I have to go figure it out.

Poor adherence to good practices [laughs]. Lack of searchability. Lack of test environment. Man, all of those are horrible. And both of those projects kind of checked all of the boxes. And I could go on, but I'm going to give you all a chance. Will, you've been around in this industry for a while. You've probably got your own stories. What have you seen that scares you in the code, and what have you learned from it?

WILL: If I'm talking about things that scare me, it really honestly goes [inaudible 05:11] time. I can remember the time where I deleted the prod database or the various things I screwed up. But really, they all had this one centralized theme, I think, which is when sort of organizational control kind of breaks down. That's really the thing that concerns me the most with sort of bigger organizations, like if you're an organization where they're really pushing sort of a distributed outsourced workforce, right? Like, I might be experiencing some of that right now. And there is...a lot of people have, like [inaudible], is minding the store.

Code reviews are really, really difficult. And it is still very difficult to find people who are diligent in their code reviews and making sure that people are really looking after stuff. And that's when things start to go bad, especially when you get this sort of injections of a large volume of foreign code into your organization, your distributed system. So, that's really what keeps me up at night, where it's like, people are checking things into production. People are checking things into the app all the time. Who is minding the store?

And it becomes, you know, past a certain point, you're doing a trust fall by necessity. You physically cannot manage all of the code. You can't do it. It cannot be done by any single person. And so, you're just sort of...you trust the people and that sort of chain of trust is a very fragile thing and shockingly easy to break down.

MIKE: Yeah, I think my experience is totally --

WILL: It's just sort of like this organizational chain of trust.

MIKE: Yeah, I was just saying my experience is totally consistent with yours. And [laughs] the way you say that organization breaking down, and it doesn't have to be inheriting a bunch of that outside code, like you say. You can have just something as simple as a change in organizational structure so that you don't have the same team cohesion that you had previously. And say the people watching the store were out taking care of something else; stuff starts to fall apart.

WILL: Let's talk about this. Let's talk about maintaining that chain of trust and how do you do it. Because I'd say the best model maybe that I've seen that is scalable, not without faults but scalable and practical is, you know, each sort of section of the code, you know, major subsystems are owned by a team. And they have a watchdog who is responsible and proactive enough to keep this subsystem under their thumb. And I don't know if I have a better working model, really.

MIKE: Well, that watchdog you talk about, I feel like if somebody feels like they own it, that is [laughs] not told that they own it, but genuinely has some opportunity to get invested in something, it makes a huge difference. I know when I build something, I care about it, even if it's something, you know, that's not that important. If you make a pile of rocks, you can get a little fond of that pile of rocks because [laughs] I built that. That act of creation binds you to it.

I think that some of that applies in our jobs as well; if we feel we had say in what we're building, feel because we actually did. I'm trying to make the distinction here. Because you can have kind of faux control where somebody says, "Yeah, you own this. You're on call tonight." But you don't get to make any of the decisions. Well, you don't really have control. You have responsibility without control in that case.

And that's where I think is kind of an anti-pattern because if you have the responsibility without any control, it just burns you out because you're disempowered. On the flip side, if somebody gets to actually make decisions, creative decisions of what they're doing because, you know, they own it, they get to make those decisions, then they become much more invested. You know, it sounds like I'm talking about a trick like, here's how you get your team invested. Rather, I'm saying you've got to surrender some control.

You've got to, as leadership, be willing to say, "I don't get to call all the shots here because I'm not close enough to be able to make the good decisions." And if you're making those decisions, you may make the wrong ones. And there's always cases like this where there's more art than science, where it could go either way. If you don't have enough trust in your team to make their own decision, then you're seizing that ownership for yourself for no gain. It's just hogging the ball, right [laughs], and not letting anybody else play. And that, I think, really poisons things. I think that in order to get the ownership that Will was talking about, to have somebody really claim the code and be willing to care enough about it, they have to actually have it genuinely be theirs.

EDDY: There's a few key things that both of you said that I kind of want to elaborate a little bit on. And losing control of the codebase is a little interesting because if the staff is trained right with the expected conventions and what we have planted for team rules and such, that can be mitigated. But also, there was a joke that was said [chuckles] a few weeks ago. I'm not going to name-drop the person just because he's not here.

But it basically said, I'm paraphrasing, but it basically said, as a senior developer, if you take a junior developer under your wing and cultivate their decision making, you can then, at that point, have a proxy battle from them in your behalf in order to enforce some of those decisions that you want. It's kind of funny. But the more that I've adapted to that ideology, I've found myself also enforcing some of the ideologies from my peers, which is another benefit, I guess, to that fear.

MIKE: You know, a few hundred years ago, for any skilled trade, maybe not any, but a large number of skilled trades, people learned the trade by apprenticing with a master in the field. And, to your point, one of the best ways to learn something is to watch somebody else do it and you apprentice with somebody. Well, and you can see this in the works of, say, artists, where artists who apprenticed with the master end up having work that looks very similar. Of course, they diverge in their own way, but the apprentice does tend to follow the master. And I don't think that we should shy away from that.

Certainly, people are going to diverge and go their own direction. But there's real value in not having to make some of the hard decisions yourself when you're getting started. And you're going to rely on the expertise of the master while you're still learning. That's what kids do with parents. And, of course, parents sometimes get it wrong. And a lot of times, you know, the basics of adulting [laughs], you're going to pick up by watching. And they're probably going to have at least somewhat right, and you're going to pick up on that and learn the things to do. That's how we learn as humans. We learn from more experienced members of the tribe and don't have to make the decisions: hey, should I eat this berry? Is this going to kill me? And look [laughs] to somebody else who might know that answer already.

And so, you know, I think that it's worth leaning in, leaning in there. And if you are a leader, it's okay to tell people, "Hey, that berry, I saw that make somebody really sick the other day," or maybe "20 years ago. I've been doing it for a while. And I don't want that to happen to you all, so don't eat those berries. There's a reason we don't eat those berries." Maybe there's a reason we don't test in production. And follow the best practices of setting up a test environment, setting up a staging environment that you can test in so that you can test things. And put a lot of investment into that so that you can do solid testing before you get to production. You're not going to get poisoned that way, in the way that you would if you go straight out...

EDDY: I want to say that it's kind of funny because you say if you've lived that pain, you're saying, "Don't eat it. It's bad for you." But sometimes, your body acclimates to that substance that you're eating, right? And then, you just don't notice that it just tastes awful. And that's kind of just what happens in code. It's like, you might be doing a bad practice, but you're just so used to doing it. Your body just got so used to doing that that you just don't realize, you know, that it's bad [laughs] until you have someone else who's trying it for the first time, and they're like, "Oh, dude, this is pretty gross, dude."

MIKE: [laughs] Yeah, it's true.

WILL: There's a lot of things that were, you know, they were good until they were bad, or even worse, more commonly, they were relatively benign diseases. They weren't going to kill you until you got to a particular stage. I've done all kinds of things in a small team, small, high velocity, tightly integrated team, but we're fine. They were all fine. But on a more, call it, enterprise scale, you know, organization, they would kill you dead as fried chicken. So, it's not a cut-and-dry thing. The argument would be that for those small, high-velocity teams, if you're not moving fast, if you're not cutting corners, you're not going to make it to enterprise scale where you can go so slow.

MIKE: That's a really interesting point. You can get away with a lot of stuff when it's just yourself. You can get away with a lot of stuff.

WILL: And you have to. Let's talk about the things that are going to kill you when it's just you out there on a wing and a prayer. Like, you need to ship, and you need to ship today. And like, oh, you don't know how to do that? Well then, you're just going to do the best you can. Overwhelmingly, the thing that is going to kill you is running out of time or money, right? Like, 10 [inaudible], if not higher. And so, it's like, oh, security is not so tight here, and I'm like, no, it's not [laughs], you know? It's not.

EDDY: There's an estimation of good time for half a day. So, you need prep time to get there. But sometimes the party is in the next hour, right?

MIKE: [laughs].

EDDY: So, you're like, well, dang [laughs]. Well, I guess I'll just put in an extra cook to turn on the heat a little bit, you know, get it done in an hour, and hope that it tastes the same.

WILL: Exactly. I mean, being worth robbing, you know, by cybercriminals, being considered worthy of being a ransomware target is, you know, that's first world problems, baby. You made it [laughs]. It's like getting sued by a patent control. Like, hey, there you go, like a disturbing truism that, like, a rite of passage to be like, oh, I've got an actual business now. I'm actually a businessman now, is getting sued for the first time. Your first legal letter where somebody tries to shake you down. It's like, oh, you made it, baby. There you go. Look at us now.

MIKE: [laughs]

WILL: I'm worth robbing.

MIKE: So, we're digging into this idea that the consistency that is absolutely critical for managing a larger organization can be less so when you're working with a smaller organization because your circle of trust with a smaller organization is so easy to maintain. You all know where things are bad when you've got the small team. I'll say, "Oh yeah, there's that bad thing. We all know it's terrible. Don't step there."

But as you grow, that shared knowledge becomes harder and harder to maintain, and it becomes more and more important to have standardized practices, standardized ways of sharing information, ways to replace people when they move on or, you know, otherwise removed from the organization. And that's a challenge as you grow, and it's going to chafe a little bit. And you're going to be like, "Hey, I miss my freedom to do whatever I wanted." And to have the maturity [chuckles] to say, "You know what? I need to change. I need to change what I'm doing, or else I'm going to cause harm because being a cowboy is not going to work anymore."

EDDY: There was an off comment, you know, that was kind of relevant to what you're saying but I kind of want to expand on a little bit. It was, you can tell someone the direction and tell them, "Hey, don't step there, right? There's a trap door. You're going to fall." But what do you do in the instance when you tell someone, "Don't step there." And they respond with, "Oh, it's okay. I can fly," right? Like, what do you do [chuckles] in that circumstance? You're like, "No, dude, you can't fly. I've been there." And they're like, "Oh, no, no, it's fine. I'll be fine." Like, what do you do? How do you keep quality when the individual believes that they're not affected?

WILL: It depends on the blast radius of the problem. I am a believer in sort of you break it, you bought it. If somebody is willing to say like, "I want to do this thing. This is my thing. I will take ownership of it," and the blast radii is small enough so that if it goes bad, then...you know what I mean? Like, it is a human-sized error with a human-sized sort of level of consequences, and somebody comes to me and says, like, "No, I'm going to do it my way," I'm like, "Okay, do it your way," you know? I believe in sort of agency like that. You have to be mindful because, a lot of times, things can be slow-growing systemic problems. Make a judgment call whether you can afford to let somebody go and try it their way.

In my estimation, the way I make those decisions is, is it a really big deal for you, and can you shoulder the consequences of it yourself? And so, if it's a manager, it's like, okay, the manager's team can shoulder it. If it's an individual developer, then the individual developer can shoulder the consequences thereof. I don't know. If I can say "Yes," then I will, honestly, because that's how people learn. And if I can, I try to explain why. Or I'll come up with a sort of rubric to be like, "Okay, try it your way, and these are the conditions under which I will say, you were right, and I was wrong. These are the conditions under which I will say, ah, you know, I mean, this is just tomato, tomahto. Both are, you know, more or less equivalent. And these are the conditions under which, like, can we all agree that I was right and just do it my way now, you know?"

EDDY: I agree. You're like, at what point does it start tasting bad? If you're saying add more sugar but without the added sugar it still tastes okay and you're okay with the taste, you're like, "Yeah, yeah, it's fine. It still tastes all right."

MIKE: [laughs] It goes back to --

WILL: And I'll say there's a lot of stuff out there. The overwhelming majority of decisions fall into that, I would say, broadly, you know, that tomato, tomahto kind of a situation. It's very rare that you'll have a black-and-white this is wrong scenario.

MIKE: I was going to say that goes back to the ownership discussion that we were having before. If you don't give the team...so this is opinion of me, but I think it's right [laughs]. If you don't give people ownership, then they won't have any reason to take responsibility, and you get ownership by making those choices. You're committed now. You've made the choice. You see how it works, and now you care how it turns out. And, yeah, maybe you've made the wrong choice, and you learn from it. Maybe you make the right choice, and now you've got something defensible, and you'll care about that the next time.

Giving people that chance, you know, the tomato tomahto chance to do it their own way, it may sound superficially important, but I think that it's actually critical for giving your team the opportunity to actually make consistently good code. Because if you take away that control, well, then nobody cares. I don't want to say it's like slavery because you can leave the job [chuckles]. But if you don't have any agency in it, then you just don't care.

EDDY: Well, and I think what's more important is if you have a new cook in the kitchen and you're constantly holding their hand, they're not really going to learn anything. However, if they end up burning the kitchen, they are going to remember. And they're going to be like, "Dang, I had to clean all this up. And all this mess, I had to clean it up." But then, guess what? The odds of them burning the kitchen again for the same recipe probably isn't going to happen anymore. So, I think there is some benefit to putting trust in your cooks.

MIKE: I was building on what you said before, where you have to have that watchdog that keeps going. And the way I say you develop that, I think you're right that you need to have somebody who cares enough to keep people in line. And I think you only get that, and this is why I've been talking about the ownership: if that watchdog actually feels like they own the code, feels like because they do.

They're only going to care enough to keep the code good if their actions matter, you know, if they actually get to make those choices, which means you got to let go of the hand and let them do it, which is going to be uncomfortable. But then they will own it, and they'll learn, and they'll become that fierce owner, you know, it's their thing, and care enough to keep the code solid. And I think that having a culture of giving people the chance to actually own the code is hand in hand with having quality code, that you can't take away control and expect the best practices to continue because it rubs against human nature. That was my high-level idea.

WILL: Yeah, no, I totally get it. And so, the question that I have for you guys, this is kind of a broad thing, but, like, how do you cultivate that kind of [inaudible]? How do you value people who have been around long enough to take ownership? What are engineers like that worth to you? How do you build that stuff up? How do you keep...I would say, I love institutional knowledge. I love people with ownership. I love all that stuff. I am very fairly stridently against the idea of seniority as if, like, sort of years on the job is a suitable proxy for these desirable qualities. So, how do you make that happen?

I won't bury the lead, but one of my great bugbears of the industry and one of the things that I...I really hate the fact that you've got to move around to move up, and I understand the countervailing forces there. But, like, if you didn't want to do that...because all these things that we're talking about, these watchdogs, this ownership of code, this isn't the kind of thing that you can have with people moving every two years. But if you're not moving every two years, there are consequences for your career, which can be fairly significant. How do you manage all that stuff? How do you create these cultures that we all agree we need but there are so many headwinds against?

MIKE: Well, Netflix, if I remember right, so I'm going from memory here, but my understanding is that for years, they had a policy that if you got a job offer somewhere else, they would match it. No questions asked. They cared enough about that continuity. They put their money where their mouth was. And so, they had some of the best-paid engineers in the industry by some margin, sometimes, like, you know, 30%, 50%, 100% more than some of their peers.

But I tell you, you do that, and you get a committed team who cares enough to stay around. I think they did the things because the company cared enough to actually do what they said and value their team members. The result was they got a team that made them extremely successful. So, I think that there are examples in the industry of people doing right. But there's a lot of examples of being short-sighted because there's a lot of incentives to be short-sighted, to say, "Well, we need to save money. We need to make the stock price go up., And we're going to get that by reducing costs," which is a reasonable thing to say. But then what happens next is dicey, right [laughs]?

You say, "Well, what you do next shows what you value. And if what you do next is continue to value your team and pay them appropriately, well, that shows what you value." If what you do next is say, "Well, let's get rid of all of our senior experts because they cost a lot, and let's hire a bunch of new folks without any context and, you know, it'll work, right?" Well [chuckles], you've shown your values. And there's a hazard, I think, to leadership to make decisions that will probably turn out well in the short term but will lead to a steady decline over the long term.

EDDY: It's like what do you do when the only two people in Coca-Cola suddenly die and they're the only ones that know how to make your recipe, right? You can get pretty close to the genuine taste, but you just know that it's going to be a different taste, and it's not going to be the same.

WILL: Well, like, the great [inaudible] are full of indispensable [inaudible 27:49], right? And I think there's a counterargument to be made that your initial team is the people you could get. But as things go on, as the company gains legitimacy and financial power to compensate and hire and retain good developers, that original team maybe, like [laughs], uncomfortably accurate definition of a technical co-founder, is the best developer you can get to work for free.

MIKE: [laughs]

WILL: So, you could start bringing in more heavy hitters as time goes on. And a lot of the decisions, engineering decisions that were, I think, defensible, if not absolutely correct at the time, are no longer so. There's a flip side to that as well. Everything is a golden [inaudible] [laughs], right? You're always trying to find a balance.

MIKE: Sure. What you probably don't want to do, though, is hire these great, new folks, keep them around for six months, and then send them out the back door. And maybe you're deluding some of the people there originally, or maybe you do ask them to leave. But you're still looking for continuity with people who know what they're doing. There's still the respect for expertise and, again, not seniority per se but expertise.

WILL: Well, yeah, and then it becomes a sort of a leadership challenge in that you need to sort of walk people down where they were, at one point, a law unto themselves, and now, you know, maybe they have a manager. And I think, arguably, the current stock option regime mirrors that, I think, pretty well in that, you'll have your founding teams, and they'll get a big slug of equity. Like, your initial hires they'll get a big slug of equity, and that equity will appreciate and appreciate and appreciate.

And people will come in. You'll get your series A round. And you'll start to bring in some real hitters, and they'll get more compensation. But the people who came there blocked the stock, and it's worth a lot more now. And so, they have, you could argue, golden handcuffs on one hand. But you could argue they're paying for institutional knowledge and continuity, even though they might not be continuous in the same role. Everybody sort of gets a...it's a fair shake there.

MIKE: It's interesting that we started by saying, "What scares us in the code?" And we're talking about stock options [chuckles]. And I don't think that's off. It seems like a tangent, but I think it's inseparable in that we're talking about the practices that lead to ownership and people caring enough to make the code solid.

WILL: Well, exactly. Well, we talked about the code for 5 or 10 minutes, and then we immediately jumped to the real problem, which is sort of maintaining the team, maintaining that culture of engineering excellence, maintaining a sense of ownership and responsibility as things scale, as things get bigger, as the problems get more complicated. I mean, that's it. That's the issue. There's no two issues around it. As long as everybody is coordinated, and focused, and disciplined, you can sleep pretty soundly. I mean, every once in a while, something's going to go bad. But, man, dude, nothing makes me happier than a problem that got successfully opened, closed, and resolved, and I didn't have to lift a finger. I'm just like, ooh, ooh.

MIKE: [laughs]

WILL: Oh yeah, oh yeah. What's up? Oh, that was Saturday.

EDDY: For those who are fans of --

WILL: You guys had a rough Saturday, boys.

MIKE: [laughs]

EDDY: For those who are fans of Emperor's New Groove, there's a meme where he goes like this, and it's like an okay because it's just so perfect. And that's what you made me think of when someone reports a problem, and it just goes fluid, and it just goes perfect. Awesome moments that you sleep like a baby.

I do want to say that there's a couple of situations or circumstances that sort of keep me up at night, and one of those are having a tower, where the foundation of that tower on the right side is held by a twig. It's so freaking scary because that's legacy code. And touching something where you just don't know where it is, you can make the change that's super scary. And I feel like a lot of us sort of try to avoid that and find another alternative because it's like your whole app is rooted off of it [laughs]. So, that's one of the things that keeps me up at night.

MIKE: That goes back to the continuity we've been talking about. Hopefully, you can keep the continuity so that you have somebody who knows how to work with that twig and can share it, sometimes, though, you don't. And in an older codebase, you probably don't have people, and if they do, they're probably spread thin and don't have much time to talk to you about it. And that's a difficult problem that –-

WILL: I don't know, I'll say that just, for me, I don't spend a lot of time thinking about pieces of the codebase that have been running without complaint for many years. Those are not high on my [inaudible]. Nothing like battle-tested code. Although, gentlemen, if you can't hear, I am being paged. I think it's time. I think it's my time. I will talk to you guys. See you guys.

MIKE: As Will was saying, if it's working, you don't have to worry about it. Sometimes, though, it stops working [laughs] or needs to be changed. And then, you got to have somebody who can go in there and fix it. I can bring up one example. She's not here, so I'm not going to name. But I worked with a really good engineer who was obsessive about detail, and she'd see something, and she just couldn't help figuring out why it worked and how it worked.

And it meant that it took her a long time to get things done initially because she would understand what was going on before she pushed up the code. But it also meant that once she'd worked in the area, she understood it, and she could get that job done. And anybody else who needed to work in that code would go to her because [laughs] they knew that she could help them understand what was going on.

Having somebody who can have that level of focus where they will take the time to actually figure something out is so valuable because it solves your twig problem, Eddy [chuckles], because they'll take the time to figure out exactly how it's working. And there's a hazard there because you might think, well, their output is not as high. They're not cranking out a bunch of code because they're going to take longer to get things done generally because they're not going to push out their changes until they're sure they work. But they provide immense value to the team [chuckles] because having somebody who actually gets it is just so valuable.

EDDY: I agree. I agree. Now, there is another thing that keeps me up at night. And this is when you have a long chain, and they're all connected to one another. And in this chain, it calls another chain, which calls another chain, which calls another chain, which calls another chain. And, suddenly, you have 17 chains that are hooked up, right?

MIKE: Link after link [chuckles].

EDDY: Oh, that was so bad because then what you have to do is...and they tell you, "Oh, modify this chain." And you're like, ah, see, I don't know. I don't want to modify the middle. That's too scary. I don't want everything to just split open. So, what I find some people do it's like, oh yeah, we'll modify the tip. We'll modify the end. We'll modify the other end. And that's why you get a long series of chains, and that really keeps me up at night any time I have to touch any of that code.

MIKE: Well, and if you're doing that also without doing any refactoring, you probably end up with 20 parameters to the end of the chain, where the chain began to try to promote reusability. And then, oh yeah, well, there's this component that could be reused. But if you haven't figured out what's going on in that chain, the reason why you have it, then you're not going to reuse one of the components and kind of branch the tree where you need to. You're just going to tap in at the end and add another parameter to make it do what you wanted, rather than going deeper into the chain. The part that does do what you want, you could have [inaudible] in there.

EDDY: Yep. Which is why I sort of adopted the ideology of singleton classes. I really do enjoy them a lot. They're much easier to work with.

MIKE: Singleton can have a few different meanings. Could you clarify exactly what those parameters are that you're --?

EDDY: Sure. So, for me, a singleton class is just a single hand of responsibility for that class, and that's all it does. [crosstalk 37:45], right? Like, don't branch off of that. It's possible that you can go beyond that if you want to, but don't violate that. It's part of...there was a book that we read on Ruby, which was...I don't recall the exact name of it off the top of my head. You might remember it, Mike. But the whole –-

MIKE: On "Design Patterns"?

EDDY: The "Design Patterns," yeah. So, part of the idea of the design pattern from that book was don't go beyond of what the class is intended to do because you risk maintainability and reliability from that class. So, if you say, oh, this class' whole purpose is to structure a hash, and that's all it does, and you're like, oh man, but it would be really nice if it would also do this, and you're like, yeah, see, it works. And you can do that, but don't do it, right?

MIKE: Single responsibility, single responsibility. So, I wasn't sure where exactly you were going because sometimes singleton can mean that you've got something that's self-contained, so it's both...there's only one instance of it in all of your running application because the class will only create one instance of itself. And so, you have that single shared instance across the application, so you avoid issues with duplicates of it.

And so, if you're managing a connection pool, for example, you don't want to have two managers of the connection pool. You want to have one, so you don't...only one person in charge. But what you're saying is something a little different, which is single responsibility principle, your unit of code. And if you're functional programming, you don't have classes, but you should have your function. It should do one thing. It should do that thing well, and it should do nothing else. And if you've got something else to be done, well, some other unit of code should do that.

EDDY: Agreed. Now, you run into issues where you implement modules or concerns that essentially do the same thing, but they deviate slightly. And that's also a problem that's really hard to address.

MIKE: Are you talking about people writing code that's almost the same but not quite?

EDDY: Yes.

MIKE: [laughs] Yeah. And so, they end up diverging.

EDDY: And, suddenly, instead of maintaining just one, you're maintaining two. You're maintaining three [laughs].

MIKE: Well, that gets to the don't repeat yourself. Don't repeat yourself is actually kind of subtle because there's times when you absolutely should repeat yourself. And here's where I take that. In cases where the repetition is trivial and makes things easier to understand, then you should repeat yourself such as in a display component. To take something to its logical extreme, let's say you said, "Well, all markup uses tags, so I want to make sure that every single tag is written by shared code." And so, you have to go through this shared code to render a tag.

And maybe you have some sophisticated component. Maybe you're writing a front-end component where that actually does make sense. But if you're just writing out some markup, then that's probably a really bad idea because you're going to have all kinds of complexity, or you could have just thrown some text out there, and it becomes far harder to manage.

Likewise, unit tests. People sometimes think, man, I'm going to make sure that I have no duplication. So, they take anything that might possibly be shared and put it into a shared example that's in some other file that you have to go digging [laughs] to find. It makes it so hard to see what it is that you're testing, where if [inaudible] the details and repeated them, then it would be a lot more comprehensible. So, I'd say don't be so DRY that it chafes [laughs].

EDDY: I've received code where I've reviewed where they've obscured all the dependencies to top level in the specs, and they just call one subject, right? Boom. And then, like, that's the whole context for that test. And I'm like, no, what? What's going on here? And so, I have to go and I have to dig to figure out what's all the dependencies for that. And I'm like, sometimes I wish, man, can we make this a little bit more DRY? Just a little bit more so that we have...because tests, in general, should document themselves, right?

MIKE: Yes.

EDDY: And if you're obscuring that for your reader, then you lose context and it's really easy to keep all that in your head. So, I have tried to push a bunch of new tests that I've written, right? And I've applied a bit of dryness to it.

MIKE: Well, less DRY [inaudible], like, in this case, less DRY. You are going to repeat yourself. It's important for legibility. But then, on the flip side, you say modules will do the same thing. That's where it actually matters a lot to not repeat yourself because you are going to have to maintain two things, or then three things, or then four things that are slightly different that basically do all the same thing. And now you've got to maintain all of them. And, in that case, for important business logic, yeah, put it in one place and one place only. For unimportant display issues, repeat yourself at will.

EDDY: I have pretty much bent the knee, and I've said, all right, I'll settle with damp [laughter].

MIKE: You like your code moist.

EDDY: Yeah, it's a little moist, you know, best of both worlds. It's fine. You know, not too extreme one way or the other. But you have ideas from both is really where...because you have to be able to differentiate between syntactical opinions, right? That doesn't really, like, it has the same functionality, you know, same everything. But the holdup really just becomes from that individual's opinion on how the code should read versus how it functions, right? So, at that point, you really just...you have to give in. And you're like, eh, it was just syntax, not a big deal.

MIKE: Let people make their own decisions a little bit [laughs].

EDDY: Agreed. And it came back.

MIKE: It did come back. That's maybe a good stopping point, actually.

EDDY: Agreed.

MIKE: So, we looked around a lot and didn't talk a lot about specifics in the code but a whole lot about practices that tend to drive good code. And so, we sleep well at night when we give people the agency or ownership to make their own decisions. It sounds like it might make you sleep worse at night [laughs]. What's my teenager out there doing [laughter]? And if you've given that teenager the chance to make good choices since they were three, then maybe you are sleeping at night because you trust them to make the choices you've already seen them making.

EDDY: I'm not a father personally, but I do attribute my tests to do the parenting for me [laughter]. So, I sleep better at night knowing, you know, that my tests are holding me accountable.

MIKE: That's great. Well, thank you, Eddy, and thank you [inaudible]. It's been great and until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, the panel delves into the topic of changing programming languages in applications. They reflect on their experiences transitioning between languages like COBOL, Perl, PHP, and Ruby, highlighting the complexities and costs associated with such endeavors. They stress the difficulty of converting large, established applications to new languages due to the potential for high costs and disruptions, as exemplified by legacy applications still running on COBOL.

A significant part of the discussion revolves around the value and challenges of rewriting applications in new languages. Will presents a strong argument against rewriting code, citing examples where organizations have suffered due to poorly executed language transitions. The group agrees that the decision to switch should be driven by practical needs, such as performance improvements, scalability, or security, rather than the whims of new leadership or a desire for modernization. They also emphasize the importance of mentoring junior developers and avoiding language specialization that can lead to organizational inflexibility.

The podcast concludes with reflections on the current trends and the future of programming languages, referencing the Stack Overflow Developer Survey. They note the rise of languages like Python and the enduring niche of Ruby. The hosts acknowledge that while modernizing or changing languages can sometimes be necessary, it should be approached with caution and clear rationale to avoid disrupting productive teams and established workflows. They suggest using opportunities like refactoring to evaluate and implement new languages gradually without jeopardizing existing systems.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting today. I have with me Matt, Justin, and Will. We're happy to have Justin with us. So, Justin and Will are not Acima employees. They have been in the past. And it's great to have both of them with us to provide their insight in their current endeavors. Great having you back here, Justin.

JUSTIN: No problem. And I do have to say that I went from one Utah startup that is an A startup with five letters to another Utah startup that was an A startup with five letters.

MIKE: [laughs]

JUSTIN: Both of which were acquired by big corporations and both of which have interesting relationships with the parent corporation.

[laughter]

MIKE: Does it --

WILL: I know, like, one guy at a startup number two. I'm very curious. I'm very curious. Like, you know what I mean, I'm always curious about like, you know, the hot goss which we won't get into.

MIKE: Exactly.

WILL: I went from a Utah retailer to a big box electronics retailer, that one [laughter]. And, you know, it's a different environment.

MIKE: Well, and this is perfect. But today we're going to...I'm just going to jump straight to the topic today. We're going to talk about changing languages in your apps. Now, this applies in, you know, languages that you're doing, but we all have applications that have been around for a while on some languages, and, eventually, that doesn't work very well anymore.

Now, there are still applications out there in COBOL. I don't think that anybody says, you know, "I want to create an application in COBOL," [laughter] because it's a language that hasn't been actually [laughs] developed for, like, what, 50 years or something [laughs]? I don't know. I have to go look at exactly how long it's been since COBOL was, you know, like, a popular language.

But you know what? There's still COBOL developers out there [laughs], and there's still COBOL applications out there. Because it is so hard to take a big working application and so expensive to convert that into a new language, which will itself eventually fall into disuse, and then you have to do it again. And, you know, the bigger the application, the more that costs.

MATT: ...Started my career.

MIKE: Did you? COBOL?

JUSTIN: COBOL?

MATT: Converting COBOL and FORTRAN applications to Perl and PHP web applications.

MIKE: Wow.

JUSTIN: Wow.

MIKE: To Perl. So, I wonder if that's happening. So, I was thinking about leading out...I thought, you know, I'm going to lead out with COBOL. But I thought about talking about my [laughs] first year as a professional developer, where I wrote in Perl, [laughs] right, you know, as well as other languages and, you know, comparing that. I'll say, I worked in Perl and then went away from it. I came back to Perl, and it was, like, starting from scratch [laughter]. It was a challenge.

MATT: I do still have some love for Perl, though.

MIKE: It's a cool language.

JUSTIN: [inaudible 03:24] as well on my first programming job, also.

MIKE: Oh.

JUSTIN: It was great.

MATT: PHP, not so much love, but...

JUSTIN: No, [inaudible 03:32] PHP.

WILL: I don't know, I like...like, Perl's, like, the Vi of, like, programming languages. I mean, like, in that, like, I could do it, but I never really got right with it. But I saw people who were, like, old, grizzled, wizard beards, sys admins just cooking on some, like, Perl scripts. And I was just like, okay, okay, like, this is, like...like, I'm seeing what's happening here, and I could recognize that, like, this is a power tool for, like, serious people to do serious work. This has utility.

MATT: You can accomplish a whole lot in Perl with very little code. It's the king of obfuscated code.

[laughter]

WILL: Ah, it is, yeah, yeah.

MIKE: Perl was the inspiration for the name of the Ruby programming language because they have a lot...

JUSTIN: Ooh.

MIKE: Yeah. Ruby would not be called Ruby if there had not been a Perl beforehand. But I got to say, when I started Ruby, it made sense, and that never went away [laughs]. Like, Ruby is like Perl, and it has still some weird Perlisms in it, if you go looking for them. It's like Perl intended to be easier to read.

WILL: Yeah, with an emphasis on sort of, like, readability, and, like, actually object orientation and stuff like that, yeah, completely. I got a hot take for this one, but I don't know whether anybody wants to get in here.

JUSTIN: So, I've got a hot take. I mean, I'm sure we all have hot takes for this one [laughs]. Why don't you go first? I have a good, lukewarm, and a bad experience, so three different experiences.

WILL: My hot take on, like, when should you rewrite your program in a new programming language is don't. Don't. I have seen...I don't even want to say 10 bad examples to 1 good example, you know what I mean? But I think it's actually even higher than that. And I have seen entire organizations, entire companies ruined with just this sort of thing. It's almost like an organizational anti-pattern, where you'll get some new hotshot CTO. He comes in. He has a tech stack that he favors. I say he; it could be a she, but it never is. It's always a dude with an ego and a tech stack, an area of expertise, a team of, like, people that he trusts.

JUSTIN: Yeah, and he brings --

WILL: And then, they come in like a bull in a China shop. They rewrite everything in their tech stack, bungle the job, screw up the product roadmap, and, like, take the entire organization down, you know what I mean? And then, he sails off in his golden parachute to do the exact same thing again. I recognize that, like, there are no such things as absolutes in engineering, but I am an overwhelming no. Absolutely no. Like, there's almost always ego, programmer laziness, and sloth.

I can't think of a language, a modern production language, that can't be reformed instead of replaced with 10 times less cost and a better outcome. Even PHP, which is a dog, gutter terrible language, has been remediated. It's fine. You can do PHP well now. You can do it, so do it. And, like, if you're a programmer that just doesn't want to learn another language, [inaudible 07:12]. Sorry. Like, I mean, it's laziness, sloth, and arrogance. Like, there's no...you know what I mean? Yeah, that's it. That's my hot take. Don't. If you want to do this, you're wrong. Don't do it.

MATT: So, you said a couple of things there. One, you used the word modern.

WILL: Yes. Important [inaudible 07:34].

MATT: I am wholeheartedly against the idea of modernization for the sake of modernization. If it's not broke, don't fix it. You know, we all face this. Everywhere we go, we face this, right? Some hot, new language comes into town these days. It's Node.

JUSTIN: Java.

MATT: And JavaScript backend.

MIKE: It's some new language that'll be transpiled into JavaScript. That's what it'll be [laughter].

MATT: Yes. Yes. And --

WILL: And has transpilers all the way down [laughs].

MATT: And everyone wants to jump on that bandwagon when they have a stable, expandable, scalable system.

JUSTIN: Okay, I've got one example where it worked, and that was in mobile, in the Android world. It was originally written in Java, and it still is in Java. About 2017, 2018, Kotlin came on the scene, and Google and JetBrains, no, IntelliJ, whatever, IntelliJ.

MIKE: JetBrains.

JUSTIN: They did it right. They were able to convert Java programmers over to Kotlin programmers. And they were able to convert codebases from Java to Kotlin with minimal issues. And it helped. I mean, there was a lot of things there about why it worked, but a couple of the main things that I could think of were that Kotlin, at the time, was just plain better than Java.

And then, two, Kotlin was compiled down into the JVM. And then, three, to convert a Java file to Kotlin, IntelliJ had a button that said, "Convert this Java file to Kotlin," and it converted it into working Kotlin. It may have been a little wordy, but it worked. And you can do that class by class. And it would work alongside the Java files and the Kotlin files. They would just work alongside. They were in the same codebase. And when you hit compile, it all worked.

MIKE: I've also seen a success story that's even different than that. It was a Java application. And this kind of goes back to Will's [laughs] saying before...the incoming people...it was an acquisition. The new people said, "You know what, we're not going to use Java anymore. We need to use a scripting language." And their preference was PHP. And this was a long time ago, but still [laughs].

WILL: Oh, man. Oh, oh.

MIKE: And I had been to a Java users' group where they had talked about Ruby on Rails and showed how, hey, you know, this is a, you know, a structured framework that is responsible and uses scripting language. You got to think about it. And so, I thought, you know what? This is old PHP, right? This is before all of those problems that you talked [laughs] about, Will, had been fixed. They were all still very much there. But I was like, you know, I like my life. I don't want it to go [laughter] in all sorts of bad directions.

So, I went, like, in a weekend, I learned Ruby, wrote a prototype for the new version of this, and said, "How about we try this? It's already starting to work," and got them on board with trying this other, you know, language: Ruby. And we converted over. We ended up with one-tenth of the lines of code and more features.

MATT: I've had similar success. And when I said that, I didn't mean you should never do this, right? Because there is a time and a place. And one of our successes was a big PHP monolith into Ruby, so much better, so much cleaner. But there was reasoning behind why it happened, you know, new lines of business. It was in healthcare. So, we now needed to support claims and imports from health plans, EDI imports, and, you know, all sorts of things that the old system didn't do. And it justified, hey, it's time to version this and roll out a new system.

WILL: But I want to...hang on, I want to get in here because I do take your point, Justin, and I agree. And I have experienced the same thing with sort of, like, legacy JavaScript stuff moving over to TypeScript. However, I don't think those are rewrites. Those aren't rewrites. And I don't actually think they're new languages. I think Kotlin is just JavaScript 2.0, you know what I mean?

JUSTIN: Java 2.0.

WILL: Yeah, Java 2.0. Similarly, like, TypeScript is JavaScript, like, 1.1, right? Where, like, you have...no, well, you have strong interop. You had the same project. You had the same codebase. And you're just sort of like, I know that they're different languages for, I think, mostly legal reasons, you know what I mean? Like, there's two big organizations butting heads, and like, so, like, the Kotlin people just forked it, you know? TypeScript is the de facto standard. Like, TypeScript took over JavaScript. Like, I don't even think they should call it TypeScript anymore because if you're writing JavaScript-JavaScript, like, you should stop [laughs].

But that's it. That's my point. Like, I agree, but I actually think it's more of a...that's more of an upgrade, you know what I mean? Like a major version upgrade of your programming language rather than, like, redoing it, you know what I mean? Subtle point, but yeah.

MATT: Now, there's ways also to approach this, right? You have a monolith. You decide you want to start extracting microservices. Perfect time. You know, maybe there's technologies that will support the job you're trying to accomplish better than what that monolith was written in for this particular task. And I think those things are super important, as well as how hard is it to find engineers for the languages you're supporting. That's a big one, right?

WILL: I think that's a mistake mostly in hiring, if I'm being direct. Being that, like...how do I put it? Like, I think if you're hiring engineers with language expertise, you're probably doing it wrong because I find, like, language expertise to be...for a good engineer, it's not that big of a deal. You're going to spend...it's a 10 to 1 ratio around learning the codebase, and the company, and the systems, and all that stuff, versus it's this much figuring out how things are done in Acima, and, like, this much learning Ruby. And Ruby is probably one of the hardest languages just because there's so much magic, you know what I mean, associated [crosstalk 14:30]

MATT: Yeah, I agree, when we're talking about senior engineers, right? I've always considered myself language agnostic because I don't care. I mean, I do care a little bit, right? I'm not going back to PHP. I'm just not. But when we're trying to mentor junior engineers, you're moving people over from other departments, that becomes a little more difficult, right? Because the learning curve is already pretty steep. When they start to get used to one language, to throw them into a new one, I think, provides more challenges.

WILL: Yeah, but they get more support.

MATT: Once you have some experience, absolutely, I agree, right?

WILL: I think it's more impactful on a junior engineer because it's like learning a different, like, human language, in that, like, you learn a different language, and you think in a different way, you know? You learn Ruby, and you think in a Ruby way. You learn JavaScript; you think in a JavaScript way. You learn nasty, old C, and you think, in, like, a different way, and you have a different perspective. I think for junior engineers, actually moving around in ecosystems is actually even more important because they'll learn better habits and they'll have more informed opinions.

I think when you have a junior engineer, God help them, right, that makes it through all the way through to senior without ever having sampled, you know, from a combo platter, that's when you get this sort of, like, senior people that are not language agnostic. And those are the people that come in because they don't know any better, and they're like, "This is all I'm good at. All I'm good at is .NET." So, you're a .NET shop now. And those people are dangerous.

MATT: Yeah, well, I agree with that. Point being, it becomes more difficult at that point, right? Because, as a company, we still have to focus on delivery, and it makes it harder to deliver. I agree. And we're all ADD, right? Every single engineer is ADD. We love to learn new things.

WILL: That's because engineering is boring [laughs].

MATT: We love to make things easier on us because we're lazy, and we're very inquisitive. So, we like to solve problems. I love learning new languages. It's one of my favorite things to do, just, you know, that exploration and experience of something new is awesome. But there are some challenges behind it.

JUSTIN: So, Will, you mentioned earlier about the refactoring. And, you know, Martin Fowler wrote the book on "Refactoring." One of the things I vaguely recall him talking about, when you are going in to refactor something, and that's basically what you're doing, is, when you're rewriting something in another language, you are deciding that you are going to refactor it, even if it is two totally different languages or as you say, Matt, like, you're changing...you got a monolith or something, and you have to break it apart. You have to identify the parts that you're going to move over to the new language and do that in an organized manner. And you have to be very cognizant and purposeful about what you are doing.

And those types of things mean that, I mean, help you, if you have to do this anyway, help you on your road to success. So, I don't know, big hot shot coming in and mandating stuff, sometimes you just have to go in and do what they say. And so, if you're going to do it, you might as well do it right. And that involves, you know, being an expert at refactoring and being very purposeful about it, so...

MIKE: You suggested something really interesting because if you extract out portions of an application by one, eventually, you know, you may end up with nothing of the original application because all the pieces have been extracted. So, you've refactored by breaking up into different services. And there was replacement, rather than rework, by extracting services. That's a way you can do this without actually rewriting the application, per se, so much as, hey, you know, this needs to be refactored enough. I'm going to rebuild this component. Once you've built enough components, eventually, the old codebase is gone. There's a kind of a strangulation technique, like a strangler fig. Eventually, the old --

JUSTIN: Yeah, yeah, exactly. And that's the example, the classic example, right? So...

WILL: Well, I mean, and I would also say, like, as I was thinking about it, like, I'm in JavaScript land now, right? And, like, one real serious problem with JavaScript land is framework fever. Man, like, that tribe of developers, they love a framework, God help them. And one thing that I encourage people who are, like, sort of, like, thinking about picking up a new framework to do is, if you want to grab a new framework, grab it as a refactor. Don't grab it for, like, the greenfield stuff. Grab it from, like, this old way, right, where, like, oh, I have this old thing. I'm going to refactor it using this new framework, this new idea, right?

Because what people will so frequently try and do is they'll so frequently try and do, like, okay, well, we're going to do a new thing. And I want to play with my new toys. And we'll do both of them together, which I think is an exponential increase in complexity. And it makes it very difficult to discern whether it was a good idea in the first place, right?

MATT: Yep. You understand what the problem is if you're doing an extraction, right?

WILL: Yeah.

MIKE: And you can A/B comparison.

WILL: Yeah. Like, is this better, or is it worse? And you can also, like, how do I put it? Like, a lot of times, especially with new frameworks, like, you'll have, like, you won't have a lot of the grime that real-world long-term requirements sets entail. You know, like, you'll do it, and you'll do the simple version, right? Versus, like, the actual reality of like, okay, this is a complex line of business app, and a lot of stakeholders have a lot of asks for it.

And you'll sort of, like, you'll skip over all that stuff, you know, and you'll do the tutorial, like, the demo version of something, where it's just like, it's not going to be that simple, and especially for, you know, new, fancy frameworks that haven't been, you know, road tested for a long period of time. Like, a lot of that stuff, like, start getting into the details of like, oh no, I need this. I need this. The security team needs this thing.

And like, well, no, I mean, like, the security team, like, often, they will have asks, like, specific asks about like, "Oh no, I need...anytime you read this thing, I need a log. I need a log of who, you know, even read this document," crazy stuff like that, which, like, well, no, for financial stuff, like, that's for real. Like, I need that. I need, you know, I need to encrypt all this stuff on the server, you know, for whatever thing. You know, all kinds of requirements where it's just like, oh, this is the new, hot JavaScript library. But, like, grimy old Java, or, you know, crusty Ruby on Rails, like, they had to fight these battles already.

MATT: Yeah, I don't like change for the sake of change. I always want to know why. Why do you want to switch everything to Node and TypeScript, right? Is it for performance? Is it for concurrency? I don't think so. But you know what I'm saying? I would like to know the reason for these changes because, oftentimes, people are misled by hype. And you're making change simply for the sake of change. And I don't think that's healthy for anyone, especially not a company whose financials are at risk, you know?

WILL: Okay, so let me ask the counter, right? When is it a good idea to change? Let's say I had, you know, like, kind of a core piece of the company that was written in a programming language that was, you know, like, modern. It was current, but it was, like, say it was something like Haskell, right, where it's difficult to write. It's difficult to hire for. It doesn't offer, like, meaningful performance enhancements, and, like, the team is sort of, like, notably unproductive, right? You know what I mean? Like, when is it worth investing in, like, moving away from the thing, right? It's like, oh, I have COBOL written in the 1950s. Okay, that's an easy layout. That's fine.

MATT: Sure.

WILL: But, like, when does it start to make sense? When is it okay?

MATT: I think you just gave the wise with your question, right? Engineers are really hard to find. Team is unproductive in that language. It's hard to support. You can't move engineers from other teams to go support it, those types of things, right?

JUSTIN: Also, it's like, if it's an older language, all of a sudden, the community is not updating it. You know, I'm putting my security hat on here. They're not updating it for security fixes that are on there and, you know, especially, I've seen that with older libraries that are included in whatever language, right? You're depended upon this library, and, all of a sudden, there's a security hole there, and the community has gone on from that library or that language.

MATT: Absolutely. We run into that all the time with gem support, right?

JUSTIN: Yeah.

MATT: You start using these gems, and, all of a sudden, there's no longer support. Nobody's doing security patches. They're not being updated. And now you're stuck. Either you have to take over support of that gem, write your own, build it into your codebase, or find something different.

MIKE: What if you have, like, a super performant Haskell team that's knocking it out of the park? They've got a good connection to the university. You've got, like, a pipeline of people coming in where you don't have a problem. That would change the parameters of the equation, right?

JUSTIN: Yeah, totally.

MATT: Absolutely. Absolutely.

WILL: If I walked into a high-performing team in, I mean, nearly any language, if they were writing COBOL but it was working, everything was clicking, you know what I mean, like, it wasn't just maintenance mode, it's like we are writing modern COBOL and everything is kicking butt, like, I wouldn't change a thing. [crosstalk 24:57] I mean, I don't have a problem with Haskell, you know what I mean? I actually like Haskell. I've never seen it productive in practice, you know?

MATT: Well, you could throw any language, right?

WILL: Like, one of my favorite languages, one of my favorite languages in the world is, I programmed in, like, five years, is OCaml, right? It's object-oriented ML. It's a very...It was like Haskell or like ML if they were, you know, it was designed by people who were trying to get things done. It's [inaudible 25:29] native. It's really fast. It's like, you know what I mean? And it's practical. Like, you can just sort of like...you can downshift and just do some imperative programming and just get it done if you need to. There's one and only one company in the world that uses OCaml.

MIKE: Jane Street Capital [laughs]?

WILL: Jane Street Capital. [inaudible 25:48]. OCaml wouldn't even exist outside of academia if it wasn't for Jane Street. But, like, it works. It works. Should anybody else use OCaml? Probably no. Like, I love it, but I would not advocate anybody to pick it up for anything, ever, you know.

MATT: You said something that I think is really important and that is, you wouldn't disrupt any team that is productive, just meshes, clicks, and gets things done, and meets business needs. However, we see it far too often. Back to your first example, some new hotshot comes in says, "Hey, we're going to start using this language. We're going to change everything. Turn your world upside down."

WILL: I'm a consultant. I will come in, and I will write this platform for you, but I'll only do it if you let me do it in Haskell.

[laughter]

JUSTIN: Well, going back to the situation I'm in right now –-

WILL: I might get emails for this. I know I am.

[laughter]

JUSTIN: So, the company I'm working for right now was acquired by a big bank, which is a Java shop, and they were acquired in 2023. They came in and dictated the Ruby application, which made the company, the startup company, the money, and made it all attractive; they dictated that, hey, this needs to be rewritten in Java.

MIKE: Oh wow.

JUSTIN: And they spent six months trying to do that. You know, there were zero Java engineers. And the big company did not want to give the resources or, you know, provide the expertise and the guidance and everything. Six months later, they said, "We're not doing it. You guys –- "

WILL: Who said it?

JUSTIN: The company, the smaller company that got acquired they, said, "If you want us to keep on making new features and everything and, you know, improving this product, we can't convert this to Java like you want us to. We got to stick on our platform that we want, I mean, that we're on, that we have the expertise on, that we have the, you know, the domain knowledge and everything." And, you know, so, basically, six months of development time went down the drain. Luckily, all that happened before I got there.

MIKE: [laughs].

WILL: That, like, I mean, so I guess the big question is, like, you know, what is the big bank going to do? I mean, because, like, you know what I mean? And so, the dangerous part about it is like, that's such a transparently bad idea, right?

JUSTIN: Yeah.

WILL: You've got technical requirements that, like, a Ruby on Rails application cannot achieve, right? Then, okay. But if you're just sort of, like, saying it just to say it, like, you are already making bad engineering decisions, right? And, like, are these guys going to be able to turn the corner and say, like, "Oh, I'm sorry. We were wrong"? You know what I mean? Like I said, risky thing to do. And if the big bank had the flexibility to say, like, "Okay, okay, okay, just do your thing then," I mean, [inaudible 28:56]. We all make mistakes. It takes a lot to learn from them. I'd say that's bleeding edge, big bank agility, and flexibility.

JUSTIN: It was, and they accepted it. And they said, "Okay, you guys can come in as you are." They have to come into the environment, right? And then, they were like, "Okay, if you come in, though, you do have to use our compliance framework and everything like that." And we have to add support for Ruby, which wasn't there before. And so, all of that is a pain, but it's all doable. And it's all fitting within the, you know, within the existing architecture, so...

WILL: And I'd say, well, I'd say, like, it's interesting to me, right? So, okay. So, let's get you in trouble at work, right? So, here's the question that I've got, right? So, all right. You've got Ruby on Rails app, you know what I mean? It is functioning. However, right, it's the odd man out. It's through an acquisition, and you're doing stuff differently from the rest of the bank, you know what I mean? Like, you know, it's an acquisition. It wasn't a merger. Like, Daddy is using Java, and you're using Ruby. And that means you are wrong, not them, right? Because you're swimming against the current, against this entire big bank, and it puts you on an island, right?

JUSTIN: It does.

WILL: And you're in Salt Lake City. You're not in New York. You're in Ruby. You're not in Java, right? Like, I think it is incumbent on sort of like, you know what I mean, like...and, like, there are advantages to it as well, you know what I mean? Because you want to be able to have this interoperability, this interactivity with the rest of the organization. Like, how do you, as the smaller party, how do you integrate into the larger thing?

JUSTIN: So, I'm glad you mentioned that because there is an initiative, and they have hired a few Java engineers to do microarchitecture, the microservice architecture. And so, they are moving over...I'm a little unclear if it's, like, net new or existing functionality, but they are going to be moving piece by piece things over to Java. But it is down low on the priority queue.

Our number one thing on the priority queue is, you know, continued growth, which is one of the reasons why the bank bought us. And the continued growth, and the new features, and everything there, all that, where we can serve our customers, is all still in Ruby. And so, it's a separate small team that are doing very clearly defined success story, you know, microarchitecture success goalposts that they could focus on. And then, they'll move piece by piece over to that. And, all of a sudden, our timeline actually went from, "Hey, you have to do this in the next six months," to, "Hey, you've got a couple of years."

WILL: Well, I mean, not for nothing. I actually, like, sort of, like, I don't know whether you guys are doing this at all, but, like, there is JRuby. There's JVM-compatible Ruby. Like, you can run in the JVM. The JVM can sort of, like, swallow you whole, like, with sort of, like, Java interop. And you can start, like, shading off routines, shading off business logic, you know what I mean? Like, I have moved away from JRuby because, like, it was kind of a lot of work to make it go.

I had an application that ran on JRuby in olden times because, like, the JVM had, like, kind of...I don't want to say the bad, old days of Ruby but, like, bad-er days of Ruby when, like, there were kind of memory stuff and scalability issues. Ruby's been working on it for a while, but, like, it was a hobbyist thing that kind of blew up. And there were funkiness. There was, you know what I mean, like, memory management, especially Ruby had some real issues, you know what I mean. And so, I was restarting a lot of servers a lot of the time.

MATT: It's funny [laughs] --

WILL: And so, JRuby is in there, you know, and you can have that call an in interop. You're not going to be able to just push a button and [laughs] convert the file. But, like, you can call...if you can get over to it, you can call Java, you know, from Ruby, you know, pretty sleek.

JUSTIN: I'll have to take a look at that.

MATT: It's funny --

WILL: Anyway [inaudible 33:19], sorry.

MATT: It's funny you say memory issues while talking about Ruby and Java is in the same sentence [laughs].

WILL: Well, I'm sorry, but, like, Java's [crosstalk 33:29] in ways that Ruby wasn't for a long time.

JUSTIN: Java has gotten a lot better over the last couple of years. And Java has almost, I mean, the last three years, Java looked at Kotlin and stole the best parts of Kotlin. And so, all of a sudden, Java's, you know, picked up a lot of the nice things that Kotlin had, especially, like, the functional programming idiom. A lot of the null pointer exceptions have gone away, things like that.

WILL: Did it get the coroutines? Did they pull the coroutines in?

JUSTIN: I'd have to take a look, so...But, in any case, it'll be interesting to see how it goes forward, but, you know, they are recognizing, like, even if they were to stay in Ruby, I mean, we're currently on a monolith, and so going in that direction anyway we're breaking apart the microservices because we're growing the teams. We're growing the engineering teams. It just makes a lot more sense. And so, it's a natural breaking point anyway that we can move forward to the parent org desired architecture, desired language.

MATT: Yeah, bottom line is we have to roll with the times, right? Unless we're the ones --

JUSTIN: That's very true. I'm glad you brought that up. It's like –-

MATT: And unless we're the ones making these decisions, then we have to adapt, and that's just the bottom line.

MIKE: If anybody is making the decision, though, don't go and destroy your teams and your precious goose that lays the golden egg. Don't do that [laughs].

WILL: Well, I mean, like, you know, it's the classic, like, you know, the classic, like, joke, right? I mean, the difference between a heart surgeon and a car mechanic is the heart surgeon works on a car while it's running, you know. And similarly, like, you know, if you want to be an engineer working on this enterprise stuff, making them enterprise bucks, you don't get to do it the easy way. You have to do it the hard way, where you can maintain the uptime and you maintain, like, continuity. And you're working on the engine as it's turning. I mean, sorry, you know, like you wanted the big check; go and earn it, you know? Like [laughs], stuff's hard.

And also, I'll say like, you know, I'll throw another bomb because, you know, I like stirring the pot. One of the more disastrous clinging to languages that I have seen has been in the Ruby community, where, like, people are trying to force Ruby onto, like, mobile apps, right? It's just not going to go. And some of that is not a technical thing, right? Like, it isn't that I don't think Ruby could make a perfectly, pleasant mobile application. It just won't be tolerated by the powers that be.

And sort of people are just like, well, we're a Ruby shop. We might be the developers and maintainers of Rails. And we're committed to Ruby in a way that, like, is maybe slightly personal. And so, we're going to try and force our way onto the iPhone. And it ain't going to go like that, you know? Write some Swift; write some Kotlin; write some JavaScript, you know. It's going to be all right, guys. Sorry [laughs], you know.

MIKE: Well, and [inaudible 36:46] example, you can back yourself into a corner. The recurring theme here is that, yeah, you got something new, build it in the language you want to do, the one that fits. Just don't destroy what you already have. But if you don't already have it, well, it's a really good time to reevaluate because those choices change over time [chuckles]. And, well, we're talking Ruby, so, you know, Ruby on Rails was really huge 20 years ago. Almost, not quite, but almost.

WILL: God, yeah, almost. No, no, you're right. No, that math is correct. You really bum me out, Mike. [laughter]

MATT: That really hurts.

MIKE: And it's still got a solid community. Like, I'm not saying that Rails is going away anytime soon, but Python has taken over a lot of that space as a very similar language. And --

WILL: You really don't need both, you know?

MIKE: Exactly. And so, you know, you've got to do some serious soul-searching. Maybe you've got some brilliant developers who are really good at what they're doing and, you know, stick with the language that works. But you better ask that question and think about how that question is going to be answered 5 years from now and ten years from now because you don't get that opportunity very often. Because once it's built, that's when you don't get to change it.

WILL: Yep. Yeah, absolutely.

JUSTIN: So, Stack Overflow, of course, does the Developer Survey every year. And so, there you can see the directionality of language usage. So, not necessarily that should make your decision for you, but it should be a consideration. And I'm looking here at Ruby. Let's see here. I'm trying to figure out.

WILL: Share your screen I want to see.

MIKE: Sorry, listeners, you don't get to see the screen.

WILL: I can Google this. I have the Google. Like, I got the whole internet. Stack Overflow. What was the search? Like, language survey, is that right?

JUSTIN: Developer Survey, yep.

WILL: Yeah, Developer Survey. All right, 2023, let's see it. Oh man, here I go. Oof, yep, I see it.

MATT: Python's, like, 30% right now.

WILL: Yeah, Python's, wow. Oof, goes bigger.

JUSTIN: So, here you have desired and admired. And I'm trying to think of...Rust is the most admired language, with more than 80% of developers want to use it and they want to use it again next year. Compare this to the least admired language, MATLAB [laughs]. I don't even know if MATLAB counts as a language.

WILL: It sadly does.

MIKE: It totally does. I've used...there's an open-source equivalent called Octave. I've used it before. It's actually very similar to the R language. Have you ever used R?

JUSTIN: Yeah, I have. But the thing is, I've only ever used that in college, so...

WILL: There's a lot of work that gets done, like, in MATLAB, like, big E engineering, like, type stuff, where you, like, do sort of, like, you know, anyway, yeah, it's a thing. It's a thing that does real work. It's for real.

JUSTIN: Yeah. So, the way I read this is admired versus desired. So, a good example here is Rust. 85% admire Rust, and they want to continue to use it, versus 30% of people who aren't using Rust want to use it. So, it's like, once you start using Rust, you love it. But if you aren't a Rust person, you don't even know what's going on here, 30%. This one here, JavaScript, this one's a really short bar, you know, 40% desire it, but of those who are using it, only 60% want to continue to use it, so...

WILL: I like the TypeScript one. That's the one I like. I'm looking at the same thing, right, where, like, people are just like, no, we really want to...Wait. So, desired versus admired, I'm curious. So, explain the difference between desire and admiration to me one more time because I think I missed something. I think I misinterpreted.

JUSTIN: Okay, so the question itself was, "Which programming, scripting, and markup languages have you done extensive development work in over the past year, and which do you want to work in over the next year?" So, you can answer twice, I guess.

WILL: Okay. So...

JUSTIN: So, which one are you using right now, and which one do you want to use next year? And then, they could be the same answer.

WILL: Okay, so what do you want...something that you want to try and people who want to keep using it. Is that admirable? Like, I tried it, and I liked it is admired. And I want to try it is desired. Is that right?

JUSTIN: So, desired is the second one. And admired means that you answered the same for both languages, I believe.

WILL: Wait, no, because, like, have used it and want to continue. Admired is have used it and want to continue. And then, desired is, want to use. So, like, I want to try it versus I've tried it, and I want to keep going. So, that makes sense to me for Rust because, like, I came up programming nasty, old C. And, like –-

JUSTIN: [laughs]

WILL: Yeah, you could do better. You could do better, and later they did. And, like, a lot of C developers, you know, really want to keep the good parts of it.

MATT: Look at that gap with Erlang.

WILL: Erlang. I'm looking --

JUSTIN: Erlang. Is it even on here?

MATT: Just scroll down, yeah. [crosstalk 42:14]

WILL: I see Elixir.

JUSTIN: So, the people who use it generally admire it, but the people who have never used it don't list it as something that they want to use.

MATT: And Elixir matches.

WILL: I'm looking at C++ and C, and I just don't understand...those don't make sense to me [laughs]. What? Like, I don't get it. Is it people who are like, I'd like to learn how to program in C, and, like, people who want to keep going? I'm unclear what this is because --

MATT: Those are your game programmers.

WILL: Okay.

JUSTIN: Is Unity in C, C#?

MATT: Unity is C#. C#. That is where I learned to write code in C# is because of Unity because I wanted to start building some games.

WILL: Interesting.

JUSTIN: Nice

WILL: I have never done any game programming. I feel like I'm the...well, not since I was in, like, high school, you know?

MATT: It's a lot of fun.

JUSTIN: So, unfortunately, I need to drop. But I suggest people take a look at this. I love this survey because it does help understand, like, here for web frameworks is, you know, the number one thing is React and then Node.js, Next.js, Vue.js. So, you mentioned earlier the JavaScript frameworks.

MATT: Think about those top three as you use them all three together.

JUSTIN: Yeah, you do use them all three together. And then, there's the folks who use Angular [laughs].

MATT: I felt really burned when I built out a huge application in Angular 1, and then [inaudible 43:56]

JUSTIN: Oh yeah, and then Angular 2 is like a whole new language [laughs].

MATT: Yep. Didn't even work.

JUSTIN: Didn't even work.

MIKE: Well, I think that we've reached a good kind of set of conclusions here, which is don't break it if you don't have to. But every time you refactor, which you should be doing, and you should be extracting things, it's a great opportunity to rethink what you're using today. And that's your opportunity. Use it while you have it. You know, it's don't miss the opportunity of a lifetime because you've missed the lifetime of the opportunity [laughs].

MATT: Yep. [inaudible 44:33]

JUSTIN: That's the first time I've heard that. I really like that.

WILL: I still feel like Ruby has enough of a critical mass to make it viable. I think the people who love Ruby really, really love Ruby. They're really good programmers. Like, almost all really good programmers I know that have tried Ruby are like, "Oh, I love Ruby. And, like, in the end, you know, I mean, like, and this is, like, purely personal bias, so I'm contradicting myself, like, rather severely, I know how to turn products around in Ruby really quickly, and if you can do it real fast, real well, you should not screw with it.

But, at the same time, like, I mean, like, everybody can see the direction the wind is blowing, and I wonder whether there's, like, an analogous, like, you know what I mean? I mean, like, what's the analogous, like, legacy language, you know what I mean, where it's like, it was good, and it had a hardcore community, but, like, we shouldn't have...like, what was it? Flash, you know what I mean? Like, can you compare it to Flash?

MIKE: I think Ruby is more like Perl, where it's really good at its niche, and, you know, 20 years from now, people will still be using it, but it's just going to get smaller and smaller and smaller, and you're just going to have...well, I actually saw, like, a comparison once for cars, comparing programming languages to cars. And they said Perl was like an old Volkswagen bus. And I think it was Ruby or Python, it doesn't matter, is like a minivan [laughs], you know? It's the same basic thing, right? It solves the same purpose, and it fits in the same niche. It's going to be the same thing. I think that there's a lot of equivalence there.

WILL: Yeah, I don't know, maybe I should learn Python, you know. Let's just switch over. I mean, it doesn't really matter. I mean, I don't know, I could [inaudible 46:34] really. I'm like, what's the difference between Python? Like, you've got, like, the significant white space, and, like, it's got...it's a little more typier. Is that right or wrong?

MIKE: It's not really even all that typier, no. It's just...so; Ruby leans heavy into, you know, its block. So, anonymous functions are, like, central to the way the language works. So, you don't have loops in Ruby, you know, people don't do loops. It's all built in the language. So, it's very functional in that respect. But Python has much less of a language. It's just much less language. Python just tries to be pseudocode written down as code. So, it doesn't have all of these cool features that you get in Ruby, but it gets the job done.

WILL: I don't know, maybe I should just do Kotlin, you know, just stinky, old Kotlin. Kotlin is fine. I've never done a web app in Kotlin. I just do, like, mobile apps. Anyway, sorry, a little bit of a tangent. We should wrap it up. You were trying to play us out, and I ruined it.

MIKE: I was, but there we are. And I'm with you. I love Ruby. And if it's working for you, keep using it by all means. I'm not pushing back against that. Think about your environment that you're in. Make your choice when you can.

WILL: But I've already abandoned them for, like, front-end stuff. Like, the way Rails is going with the frontend, I don't agree with it, and I don't have to use it, and I don't [laughs].

MATT: No, I'm not a fan of Rails frontend, personally, either. I love Ruby and Rails for APIs. You can work so quickly. It's nice. But is it the future? I guess we'll see.

WILL: No, it's, I mean --

MATT: I mean, that's not the future, is it?

WILL: But that's the reason I'm so frustrated with it because, like, they're just sort of like, they're sort of like, they're digging in their heels. And they're just saying like, the thing that we were doing in 2005 when we came out...anyway, sorry, tangent. We can fight about this next week, maaaybe.

MIKE: [laughs]

WILL: But nothing needs to be said. Like, everybody knows the score with Rails, you know?

MIKE: [inaudible 48:44] Thank you all, until next time on the Acima Development –

WILL: See you guys.

MATT: Thanks, guys.

In this episode of the Acima Development Podcast, host Mike starts by sharing an anecdote about his three-year-old son, emphasizing the importance of guidance and support in learning. This sets the stage for a discussion on onboarding new employees, highlighting the similarities between guiding a child and mentoring new hires. Mike notes that new employees, like his son, have potential but require proper guidance to become productive members of the team.

The discussion then delves into various strategies for successful onboarding. Matt praises Mike's analogy and underscores the importance of mentorship and guidance for new employees. Eddy adds that providing the right tools and environment allows new hires to realize their potential. The group agrees on the significance of assigning buddies to new employees and promoting a supportive culture where asking questions is encouraged. This not only helps new hires learn faster but also fosters a collaborative environment. They also stress the importance of documentation and how it can be a valuable resource for new employees when mentors are not available.

Towards the end, the conversation shifts to the role of tools and technology in onboarding. The hosts discuss the benefits of standardizing tools to streamline workflows and make the onboarding process more efficient. They also touch on the challenges of remote work and how it can impact the onboarding experience. The episode concludes with a reminder that onboarding is not just about processes and tools but also about building trust and relationships. By treating new hires with humanity and fostering a culture of openness and support, companies can ensure a successful and smooth onboarding experience!

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting today. And with me, we've got Eddy, Ramses, and Matt.

And I'm actually going to start today by talking about my family. I have a three-year-old, and [chuckles] he is very much three years old [laughs]. He is figuring things out, and sometimes, he does the wrong thing, like chasing you around and hitting you with a stick. Like, no, that's the wrong thing to do [laughs]. And you tell him that, and he'll be upset and think that you're trying to hurt his feelings. Like, no, I love you. I just don't want to get hit by a stick [laughs].

And, you know, three years ago, he just laid there and cried if he was hungry, right [laughs]? He had a lot of abilities, but none of them were developed at all. And I honestly don't expect him to be a contributing member of society for another 15 years [laughs]. And that's pretty normal because that's true for all of us. We all get this start, you know, we've got this potential, but not really any knowledge about the situation and got to figure out as we go. And it takes a long time before we're productive.

I mention this because today, we're going to talk about onboarding new employees. And, hopefully, you're not putting your three-year-olds or your infants to work [laughs]. But we all start, to some degree, you know, in ignorance. You know, we come into a new job and there's a lot of stuff we don't know. We have a ton of potential, but there's so much that we don't know and things we don't even know we don't know that make it very hard to get up to speed.

And, you know, depending on how well that onboarding process goes, depending on how much support is given to those new employees, it may take them a fairly short period of time to get up to speed or...hopefully, not 18 years [laughs], but it can take a while. I mentioned the support they get, and that's something we can influence. To some degree, there's also the intrinsic complexity of the systems that we're working with that we don't control. There are some things we can do, you know, we can do refactoring, try to improve our systems, but, you know, some problems are hard, and it's going to take people a while to get up to speed. These are real problems, and they're worth talking about.

And that's our topic for discussion today is, how do we onboard? It's also timely because we've got some interns starting on Monday, and we'd really like to give them the support that they need to quickly get up to speed. Now, I have some thoughts about the things that are most important for helping somebody in their onboarding, but I'd like to open the floor. What do you all think about what's important for helping people do that onboarding?

MATT: I'd first like to say you are always so great with your segues and analogies because this one makes perfect sense, right? You have a three-year-old who is learning as he goes. And would he probably figure out life on his own? Yeah, eventually. But without your guidance and without some leadership, it would take much, much longer than it would instead of you providing it for him, right? And I think that's kind of what we have to look at when onboarding new employees, whether it be new to the company, new to a role, or switching departments, you know, any of those things. I think everyone requires a little bit of instruction and guidance until they get up to speed.

MIKE: Absolutely.

EDDY: You mentioned individuals having potential but not the right guidance, kind of paraphrasing a little bit. And any individual that you hire, you interview, and you say, "Yeah, this person's got the right attitude. They got potential," but if you expect this person to hit the ground running, they're going to fall over, or they're going to drown. If you're patient and you provide the right reinforcements, and the right help, and tools, utility, et cetera, the person will grow into their own, and they will show you what you perceived during the interview.

I'd like to believe that some of the individuals who are engineering now are showing that on a day-by-day basis, right? Individuals, you know, who were ambitious, you know, who had the courage, you know, to push forward and be given the opportunity to do that shift in career have now been able to provide all that throughput, you know, that was initially observed during the interview. So, I think that so long as you do provide the patience and the right atmosphere, people will grow into their own, and they will surprise you.

MIKE: I can say that's true of you, Eddy [laughs]. You've done amazing work. And, honestly, I would say that's true of each of you. I know many of you listening are like, oh, I don't care about that. I do. I'm going to call it out publicly. I'm working with three people that I'm very happy to be working with.

MATT: And I second that. One of the things that I think is really important and maybe sometimes we overlook when we're onboarding people is that in this process, we can also empower them to be mentors and help them be able to onboard the next people that come on. And I've found, at least throughout my career, that one of the important things as the mentee or new person being onboarded is to ask questions and not be afraid to ask those questions. And as mentors, we can help guide that with our new people and encourage that. That can go a long way.

MIKE: I think that culture does make a huge difference. If you shut down any question quickly, there's not going to be very many more questions, and you do a lot of harm. Back to the three-year-old, you know what three-year-olds do? They ask questions. "What's this? What's this? What's that? What's this? What's this [laughs]?" And you can shut that down and not respond. And the result is that curiosity will be lost, and that's an awful outcome. You wouldn't want it to happen to your three-year-old. I don't think you want it to happen to your new employee, either. That curiosity is probably a lot of why you want them there—is that hunger to grow.

MATT: Yeah, and it creates isolation if you do shut it down. As annoying as those "What's this?" questions might be, that's how they're learning, and that's how they're gathering all this new information. And will they retain all of it at once? Probably not. But that's why they need to continue asking these questions.

You know, I've been in the industry forever now, but here with the company, I'm in a new role, and Mike is present in most of the meetings I am. And I am one of those people who's always asking questions. And it may feel a little disruptive at times, but I try to ask the question once and remember those things so I don't have to again, and that way, if someone asks me about those same things, I have the answers for them.

EDDY: Can I say how much talent it takes to understand something that's been directed to you the very first time [laughs]? I will have the individual reiterate three different ways for me to really understand that before they say, "Do you get that?" "Yeah, yeah, I get that [laughs]."

MIKE: Okay, so let's talk about that. So, going, you know, with the analogy we've got here, this morning, my toddler asked me, "What's this?" And he was pointing at a cell phone tower. And I said, "It's a cell phone tower [laughs]." And he now has, like, some label for it that he's probably going to forget [laughs], and that's probably about all he has. He has no concept of what's going on there. He's three. I'm not going to explain to him electromagnetic radiation and, you know, low-frequency waves that can travel through a lot of distance and through even solid materials sometimes, and how those resonate with your antennas.

I can just say, "That's a tower," right? If you don't have something to hold that information onto, to kind of pin it to, it's like trying to remember 20 numbers in a row. You're not going to remember it. Your brain won't do that. You can only hold about seven. But if you have that narrative, if you actually have the story, you have something to pin it to. So, you're going to need somebody to repeat and repeat and repeat with greater depth each time. Because the first time, you might, if you remember it, learn the name. The second time, maybe you have a little more story around it, and so on. And with each iteration, you're going to internalize more information.

MATT: Yeah, and let's face it, when you're starting somewhere, you're drinking from a fire hose, and you definitely can't retain everything that you're being told. You know, you just have to try and pick the things that you think are important and hold on to those, and maybe they aren't the right thing sometimes. But you're right; things do take more than once to really ingrain.

EDDY: In order for me to reassure myself that I've been able to internalize something new, I try to explain it to someone else who has no context, right? I read this article a few weeks ago that basically says if you're having a hard time explaining it or dumbing it down, you probably don't understand it. And so, I've gotten into the habit I'm like, oh yeah, cool, I think I get it. But let me try to explain it to someone else and see if they can at least get the concept of what it is. It has done wonders because if I'm able to be like, oh, huh, wait, I don't know if I'm sure about this analogy; let me just make sure, it really harps that ideology down.

MATT: Yeah, if you want a better understanding of something, teach it. And you're probably going to pick up something along the way while you are.

MIKE: Absolutely, and that connects back to the onboarding. One thing I found, so kind of the first tactic I'm going to throw out here that I have found to be extremely helpful for onboarding, is to pair people together who are starting at the same time or starting near the same time. If somebody started three months ago, they have everything fresh in their head. And they're going to be able to probably more easily help the person who just started than actually somebody who's been there a long time. And it has the added benefit is now they're explaining it. Now, they're finding where the gaps are and deepening their understanding.

And if people are starting at the same time, it's kind of a back-and-forth, right? It's this pairing that allows them to do that, you know, in a very tight loop with each other, and it works out really well. There's another tactic I'm going to get to in a minute. But I think that that's one of the most helpful things I've seen is to pair people together who have started at around the same time. Have you all seen that to be similarly effective?

MATT: Yeah, absolutely. And each of them are going to retain different things. So, if you have them together...in fact, we are just going through this on one of my teams. I have two new hires, and it worked out that I could start them both the same day; much, much easier that way because then they can work off each other, as well as it frees up some of our engineering time, right, to be able to do that with two people at once. But it's a much broader conversation when you have three people involved versus two.

MIKE: So, in short, it works, right [laughs]? Put people together. You know, to bring up another tactic that I've found is extremely valuable, is give somebody a buddy. Give whoever is coming in an assigned buddy. Like, "You, this is your friend [laughs], and you're going to work with them." In the absence of that, people don't know who to ask, right? You're like, well, I should ask somebody, and they'll go and try to ask whoever they saw first because [inaudible 12:22] familiar face. You don't know who to talk to. And you can go talk to the person next to you, like, oh, I don't know, right [laughs]? And then, you drop right into that isolation. You need to have a line of contact, somebody that you can ask.

Now, they may not be the right person to answer your question, but they can point you to the person who can answer that. They can point to the documentation. They can bring you down to IT to get you a new laptop. You know, having somebody have a person is just one of the most important things I've seen.

I think I mentioned in a podcast, a recent podcast that we were on, I remember a job I had over 20 years ago, comfortably over 20 years ago now, well over 20 years ago, where I was doing some construction work. And my first day, they set me up with somebody else, and there were some language challenges. We didn't understand each other very well at first, well, mostly me not understanding [laughs] him very well. This was in Louisiana, and he had a French background, but we still communicated. And, man, having that buddy made such a difference. It made such a difference. And pretty soon, I was able to understand. And we worked well together. I learned the ropes, and we actually got to be really close friends.

RAMSES: Do you speak French now, too, Mike?

MIKE: Not a word [laughs]. Well, probably a word but not meaningfully [laughs].

EDDY: How's your Spanish?

MIKE: Not a paid promotion, but I use Duolingo, and it has been helping. I've been studying Duolingo for almost four years daily.

RAMSES: Oh, nice.

MIKE: It has been very beneficial. I can read Spanish pretty well now.

MATT: I do love that we can practice those languages here at this company. I've been trying because I went so long without speaking any Spanish, and Eddy and a number of the other team members have really helped out with that.

EDDY: To be fair, this is a side [inaudible 14:11], but I've always associated programming terminology in English. And I've never once thought, huh, how do I say array in Spanish, right [laughter]? So, like, I can say everything else except array, and I'm like, oh, man, how do you say that, right? But, like, it's kind of funny, right? You're like, yeah, I understand, but not all the lingo.

MATT: Have you looked? Is there a direct translation?

EDDY: There is, and if you're interested, I can tell you.

MATT: I am.

EDDY: Yeah, array in Spanish is regla like a ruler.

MATT: Great. Interesting.

EDDY: It is, yeah.

MATT: So, one of the other things I think is really important for onboarding is to have a plan instead of just flying by the cuff every time we bring someone on, and I know I'm guilty of that. But having a plan, having a schedule, assigning these buddies, assigning a pair, those things, if you have them lined up up front, will really help improve the process.

MIKE: About seven years ago-ish, more or less, Acima was running their first apprenticeship program. And the one who had been running it actually took a position in a different company after they were about two weeks in. And I was brought in to lead that program and a team two weeks in [laughs]. That was a challenge [laughs], I've got to say, because that continuity was lost, you know, and the preparation was [inaudible 15:45]. And I put a lot of time into getting on top of that. And I really wish that I'd had the preparation ahead of time.

In subsequent years, I learned my lesson [laughs] and tried to get well out ahead of that. You know, for an internship or apprenticeship, you know, we've got a schedule. We've got a project. We've got the person they know to cling to and talk to. We've got the buddies lined up. And having those things lined up means that you can come in and have it go much more smoothly.

MATT: I think another new challenge we're facing this year, in particular, is as we're operating under the corporate umbrella, some of these interns are in an entirely different state now, and we're used to having them in office historically. So, that, I think, presents some new challenges that we need to overcome.

MIKE: One thing we're doing with the interns is dividing. So, in general, the Draper office, we should have local folks primarily. But, you know, the pandemic made things really interesting because we were all remote, no face-to-face contact, and that brings with its own set of challenges. Now, we're getting farther away from that. Most companies have at least a hybrid connection where some people are seeing each other sometimes. But, you know, it was all on camera or voice, and you do lose something. You do lose something, some connection. And you have to...I'm not saying that you shouldn't do it. But it takes additional work to be connected when you're not physically in the same space.

MATT: Absolutely. Even working with some of these people that we've worked with that are in our Plano offices for a few years, you don't really feel like you know them until you actually meet them in person and get a little face time. I mean, it improves those relationships so much. I think it's super valuable.

MIKE: Well, even things like...actually, Eddy and I were talking about this before we started the call. Turning on the camera [laughs] when you're working with remote people (It is a little off-topic from our main subject.), but it makes a difference. It gives you more connection. I've done a lot of remote work, decades, actually, of remote work [chuckle], so I'm very familiar with it. But one thing that I used to years ago, partly out of necessity, not have a camera on, you know, it was all just voice or text chatting. But over the years, I've become much more...well, I'll say I've developed a habit of turning my camera on, even when I don't feel like it [laughs] because it can be exhausting to have people staring at you all day. But I try to do it anyway just to establish that greater human connection.

EDDY: We were pretty lucky, Ramses, myself, and Tad, just to put names out there, sorry, guys. We were given new laptops, and we got to upgrade from Intel to Mac, right? And that was awesome. But I was also a little skeptical at first because I'm like, dang, I haven't onboarded in a while. Like, do I still remember how to do this? And we got out pretty quickly, but it's only because I had a bunch of the context already that I didn't know I had, you know, until I had to reapply that again.

And I'm like, oh crap, it is not working. What was that again? Oh yeah, cool, cool, cool. And I was able to fill stuff in. I cannot imagine doing that with this being the very first thing. Like, if I'm an intern coming in and like, hey, set up Kubes. Set up AWS. Set up your bundles. Like, it could go over your head. And I realized that maybe documentation could be a lot better. But in order to help facilitate new onboarding, I think documentation is critical to how fast an individual gets up to speed.

MATT: Absolutely. And, yes, it's really hard without some assistance and guidance setting up, especially our services. We have a pretty complex ecosystem. And I experienced that when I started with the company. It was right when the pandemic hit, and we all got sent home, and I didn't have all of my services set up yet. So, I just kind of had to fumble my way through it, reaching out via Slack where I could. It was tough. So, you're absolutely right. And I think something probably everyone, at least through my experience, can improve on is documentation, especially with onboarding new employees.

MIKE: Well, let's dig into that. We've talked about having a human connection, but that person can't be there all the time. That's just not possible unless you're some privileged person who can pay for a personal mentor [laughs], but, you know, generally, you're not going to get 100% of somebody's dedicated attention. And you're going to need to figure stuff out on your own sometime. How much do you think we should invest into documentation? Because documentation can be very expensive because it's an artifact, just like code is.

One idea that has really stuck with me is the idea that code is debt. Usually, we think, no, we're building this great thing. Well, yes, you built something, but now you've got to maintain it. It may provide value. But what's providing value is that you solve the problems, not necessarily the code itself. And if you have that code, that artifact requires maintenance. Likewise, documentation is the same way, and it doesn't even provide value. It doesn't pay for itself by, you know, having users use it. It indirectly pays for itself by making things go faster. It doesn't mean it's not valuable. But it's expensive. It's expensive and hard to maintain. So, to what degree should we invest in documentation to make onboarding easier?

MATT: Well, I know that you've said this before. I think that it's worth the upfront investment just for the cost savings you're going to get by, you know, increasing the rate at which people can get onboarded and up to speed. But something you've said a number of times, and it's stuck with me, is while we're onboarding, we have those new employees help maintain that documentation and make sure it gets up to date. That way, you're kind of killing two birds with one stone. You're getting people onboarded. They're getting more familiar because they have to go through and help maintain this documentation. So, it will bring up questions that maybe they haven't thought of before. It'll identify gaps in that documentation that they can help fill. So, I personally think it's invaluable.

MIKE: So, you just said something, I think, important here. A way you can keep your documentation fresh is everybody who uses it, particularly this onboarding documentation, updates it and make that formally part of the, you know, the responsibility. It builds on something we've already talked about of, you know, teaching and explaining [inaudible 22:38] the documentation.

EDDY: Actually, I've experienced that firsthand over the past couple of weeks. Part of the project that I'm working on and part of the team is integrating with a third party, right? And a bunch of that has been through the course of APIs and, like, what kind of response, what the shape looks like, what they take, you know, what attributes are required, which ones are optional. And having a robust documentation on that goes leaps and bounds, right, to answering your own questions, and so instead of waiting for individuals to respond and give you the understanding, not just there; it's just in general, right? I just want to harp on, like, how crucial it is to have a robust onboarding, right, that just grows.

MATT: Yeah, and on that, I know the project you speak of. On that project, you guys have also been creating documentation on the flows, things like that. And I'd be willing to bet that you understand it much, much better because you've had to go in and update and create that workflow and help maintain the documentation on our side.

EDDY: Agreed.

MIKE: So, we've talked about pairing people together, putting the new folks together so they can help each other out. We've talked about assigning an experienced buddy. We've talked about documentation. Are there any other key tactics that are helpful in onboarding? And we're speaking kind of generally, but this kind of applies maybe even across industries.

MATT: Yeah. I think something I try to do is help with also building soft skills. And when you're onboarding people, one of the key things is building trust. It's much easier to work with people that you have trust in. And if you can establish that upfront and help build that trust, not only will it help them trust you, but you're going to help build some trust in them as well because you're opening up those communication channels and really establishing that interpersonal relationship that I think is important to work well with people.

MIKE: That's interesting. And I had a thought about building trust. I mean, what is it you do to develop trust? You know, in dogs, you see a dog that wants to show that it's trustworthy, you know, it will, like, roll over on its back, show its belly. It shows vulnerability. It makes itself vulnerable. And one of the most important things, I think, that's maybe a key thing that we do to establish trust is show that we're willing to be vulnerable, show that we make mistakes, show that we're willing to, you know, listen to feedback. Establishing trust, I think, begins largely with yourself being willing to listen. What are your thoughts about establishing trust?

EDDY: Gosh, just having someone you can reach out to when you're in doubt that does wonders, right, to helping you feel better about your job.

MATT: Yeah, I think also, too, let them know they're doing a good job in picking up on things, you know, reinforce that. Help them build some confidence as well.

RAMSES: Related to the documentation point, I think having a system that promotes searchability. Whenever I'm learning a new thing or a new process or working on a new feature, if I don't have a point person immediately available, I like to be able to have a good tool where I can search and find out how the feature works myself. So, it's related to documentation. And I think if our system is documented well, you know, and if they don't necessarily have that point person immediately available, they can at least be able to search and try to find an answer.

MIKE: I actually got a story about that from my first dev job. That searchability is a big deal. I remember that I was writing some code. It was some Windows code accessing libraries. Was it the MDC? Whatever it was back then, a long time ago. And I was just looking through the documentation. And, you know, I looked in the index, you know, looked through the relevant sections of documentation. It was posted, but I couldn't find in the documentation an API that did what I needed to do. And I was kind of giving up. And the guy I was working for Googled it and found the answer.

And this was a big deal because Google was brand new at the time, and I didn't think to use it because I hadn't used Google before. And he pointed this out to say, "You know, I'm not trying to make you feel bad. Google is this amazing tool." Now, there are other search engines now, but that was the one, at the time, that became available and fundamentally changed the way that the documentation was searchable, fundamentally changed the way software development was done.

Because instead of poring through documentation, you could search and actually get an answer. That discoverability was just...I don't know how I would have done, you know, had a career in software without that searchability. And the productivity gains that you get from adding that searchability are amazing. And documentation tends to be pretty opaque. It is hard to get through there and find what you're looking for. That's a really great point, I think, you're making, Ramses, that doing what you can to try to make it searchable is a big deal.

EDDY: I think it's cool. Google is this new thing in our arsenal, in our utility belt that we can use in order to be more productive. Now we have AI [laughs], and that takes it even a bit further. You're like, cool, now I have a catered, like, bot that I can interact with and get even faster results.

MATT: I'm on board the AI train for sure. But you said something there, Mike, that I think I would at least like to ask everyone's view on, and that is tools, right? Google, this new tool, can get you everything you need. What are you guys' thoughts on tooling, such as standardizing IDEs? You know, we use Vi. We use Visual Studio Code. We use the JetBrains tools in our company. And they all are different, very similar, but different. Do you think there's a benefit in standardizing some tools so workflows can become much more similar? Because then you can help with shortcuts, macros, things like that, code generation. Would that benefit in the onboarding process?

EDDY: If we can have everyone just use Visual Studio Code, the whole department will be really happy [laughs].

MATT: Except for a JetBrains user like myself, and I think Mike's a Vi user.

MIKE: I am [laughs]. And, you know, a lot of that's because I had a mentor who taught me Vi, who was doing a lot of server administration. And it got in my tool belt and became [laughs] really useful because I took the time with it. And that's a tricky question, right, because if you've already got people on different technologies, there's going to be a cost to making that uniform.

MATT: And I asked because the same thing, like you, you know, I started in Vi and Vim, moved over to some other tools. I think, at the time, I was using Sublime. And then, we had a project that we did a lot of paired programming on, and everyone I was working with used RubyMine. And I saw their workflows and all the cool shortcuts they were using that was really speeding things up. So, I thought, hey, I'm going to get on board, and I'm going to switch to this because this is what everyone I'm working with is using. And I've kind of just stuck with it since. But I think that was a big part of the onboarding process for that particular project that really helped me out.

MIKE: It's almost inevitable that you'll probably end up using the tooling of your mentor. And even if you don't choose to standardize, and maybe that's a discussion for another day, getting somebody set up in the tooling that you're using so that you can do something in common is probably going to help them move a lot faster.

MATT: We're all going to go back to Macromedia.

[laughter]

MIKE: Oh boy [laughter]. The olden days of yore.

MATT: Yes. It's interesting things to think about.

MIKE: Well, so, we started scratching on tooling. I mentioned, you know, search engines have changed things. I don't know how much AI is going to change things in the future, but I'm sure that it will. And the next generation of AI that goes beyond, you know, text generation, you know, maybe they add reinforcement learning; maybe they add a new algorithm we haven't thought of yet may fundamentally change the game again.

And our industry is like that. It's quickly moving, and there's new tools. And it's easy to miss that and end up getting behind. I think we need to be careful about that. And going back to the idea of trust, maybe your new hires have some good ideas. And we shouldn't get so ingrained in our flows that we're not open to improving our process, and probably should actively, you know, seek for that, look for ideas from new hires. What do they have that they can offer that might change the way we do things?

We recently hired somebody who has a lot of experience with PostgreSQL and using some of the maybe less-used features like search for doing full-text searching. And he's got some great ideas as to how we might employ those. It might change the way we're doing things. But if we don't consider that, then we might end up stuck and never end up doing it better.

A few months ago, we had a new hire who improved our unit test suite, made it go significantly faster by going and making a variety of changes from experience that he had, tooling that we hadn't used before. It made a big difference because rather than just relying on what we knew, we let the new guy show some of his experience. You know, you give a new person, let her, let him do their thing, you may end up learning a lot more yourself than you expected, as well as just teaching.

MATT: Yeah, well, I mean, we hired all of these people for a reason, right? And everyone has something to offer. We just need to be open to listening and seeing those things that they do have to offer us. Yeah, I mean, could you imagine being able to have a conversational chat and just look for something and have it go through all of our Slack history, all of our Confluence docs, all of our Wikis, all in one go, and just be able to ask it a question? "Hey, how do I patch this particular thing because my system isn't running?" And someone's asked that question before in Slack at some point. And it just responds and says, "Oh, well, here's the solution." Powerful.

MIKE: So, we've talked, you know, having a group to work with, having a buddy, documentation, discoverability in that documentation, and flexibility of the tools, taking time to think about tools and work together on tools. Again, these things aren't even necessarily industry-specific, not even work-specific, because a lot of these things could apply to a three-year-old. You know, without thinking about it, we can easily overlook it and not take the time, not invest the time, to make that onboarding process work well. Are there any other key things that we're missing that you all want to bring up?

MATT: As you are summarizing things, I am also taking notes to make sure that not only are we talking about it, we're putting some things into action and helping to improve our process here.

EDDY: Would you think that the individuals who are local can get some benefit with being desk buddies with the individual who's mentoring them?

MIKE: I do. I remember starting at a job a lot of years ago. It was a startup, and they were in a tiny, little office, and [laughs] me and three other people basically just crammed in a corner of this [laughs] space, kind of almost on top of each other. It was hard to even get out of the little corner we were in. And later, we moved to a bigger, nicer office space where we were separated from each other. And then, we ended up just...we ended up very much more isolated. Because we all just ended up messaging each other, doing some sort of text messaging, rather than talking to each other because, you know, you don't want to talk across the aisle and interrupt everybody else around you.

So, we ended up not talking nearly so much as we did when we were kind of forced to be in a closed space. And I'll tell you, when we were in a closed space, I learned so much. I learned so much, and part of it is that I was new. But my understanding of what I needed to do just leaped forward because I was just so closely exposed all the time as to what I needed to learn. Have you had that experience?

MATT: Yeah. You know, what is the likelihood, even though it's right at our fingertips, of someone who's sitting at home working remotely, spinning on something versus sitting next to one of their peers in conversational range just turning their head and asking the question? I personally think that the likelihood of the question in person is going to happen much, much more quickly than if you're isolated at your home, even though we have Slack right here and everyone available. I'm going to ask that verbal question much sooner than I am going to reach out via some form of messenger and ask the question.

MIKE: It requires work. You have to develop a habit of becoming...it feels annoying when you're not used to it. You have to make a real effort to be much more assertive than you would otherwise be when you're in a remote position. And that takes time, and so that's a gap. That's a barrier that absolutely is there that takes time and effort to overcome. Now, maybe you're a remote-first company. That absolutely happens in a lot of cases. If you're in that situation, you have to make that effort. You have to encourage that culture strongly to encourage that inquisitiveness.

MATT: Yeah, and I think by just general human nature, we don't want to be invasive, and we don't want to bug other people. But I think in order to be successful, we have to ask those questions. And the likelihood that the person on the receiving end is perfectly fine with it is pretty high, you know, I mean, people ask me questions, and they think they're annoying me. I am not annoyed whatsoever. If I am available, I'd love to have those conversations. I'd love to answer questions and help people. And I think most people really feel that way.

MIKE: Absolutely. I'm usually buried in questions [chuckles], to tell you the truth. If I'm not in a meeting, I'm answering questions somewhere else. So, I'm answering questions in a meeting, or I'm answering questions in some sort of messaging tool. I don't find that annoying. That's, like, my job, right [chuckles]? To go and help people. It's what I expect people to do.

And if people are not asking the questions, that's worse because then I feel bad. You know, then I'm thinking, well, now you're unable to be effective, and I'm unable to help you because you didn't ask. So, I love the fact that people be proactive. But, again, you have to establish that culture. That's a critical thing for this onboarding is that...and this is brought up a couple of times now, that you have to establish that culture and encourage the safety.

MATT: Yeah. If you're not asking me questions, then I'm asking questions, you know [chuckle], entirely different types of questions than I would be asking. So, I encourage it. And I think we all want everyone to grow and be successful.

MIKE: You know, and that's probably a good place to finish this conversation is that it comes down to that relationship. You know, we're talking about solving business problems, but, in the end, we're people working together, and we should treat people with that humanity. If you want somebody to be successful, treat them like a person [chuckles] and develop that kind of human relationship. Think about what the human needs are and be responsive to them. And I think you would be far more successful and happier as a result, too.

Any final words from anyone?

MATT: As always, I always enjoy being a part of this. So, thank you all for letting me participate.

MIKE: Of course.

EDDY: It's always a treat when we can get Matt [laughs].

MATT: Thanks, Eddy.

MIKE: Well, thank you, Eddy, Ramses, Matt. Until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, Mike and the team delve into the importance of breaking down projects into manageable iterations to avoid stagnation and inefficiency. Mike shares a story about a past experience with a developer who struggled to produce releasable code, which led to the realization that breaking projects into smaller milestones can significantly enhance workflow and productivity. This approach allows for continuous integration and deployment, preventing the team from getting stuck on incomplete tasks. Mike emphasizes that this skill of breaking down work is crucial yet often overlooked in engineering education.

Will adds to the discussion by highlighting the value of creating "jigs" or scaffolding in both woodworking and software development. He explains that in his experience, especially with mobile app development, creating temporary structures or mockups can facilitate progress even when back-end systems are not fully ready. This method allows for continuous progress and testing, making it easier to adapt and iterate on the final product. Will’s anecdote about using a reverse proxy to simulate API endpoints underlines the practicality and effectiveness of this approach.

The conversation wraps up with broader reflections on iterative development and its applications in various fields, including aerospace and automotive technology. The team discusses how companies like SpaceX have revolutionized their industries through iterative processes, emphasizing the importance of starting with a functional but imperfect product and continuously improving it. They conclude by stressing the necessity of having a clear roadmap and the willingness to "cheat" by using tools and frameworks to eliminate human error, thus ensuring consistent and reliable outcomes. The episode ends with the advice to always build a jig and embrace iterative development for better efficiency and success in engineering projects.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting again today. With me, I have Will, Ramses, and Eddy. I have a story I'm going to tell before I introduce the topic, and I think about this a lot. I was thinking about it this morning. I think I thought about it yesterday. I think about it probably at least monthly about..., and I've probably even shared it on the podcast before [laughs].

But several years ago, I worked with a developer who was assigned to a project. We had talked about it as a team. We were all really excited about it. We had this new idea. It was kind of a developer-driven initiative. They worked on a little bit here, a little bit there, and a [chuckles] little bit over there. And this kept on going for, like, a month, and we thought that, you know, in a month, you'd have some code out. And looked at it, and there was kind of a framework or something there but there was nothing that was releasable. There were just pieces here and there, and it really felt like it stalled.

Now, they'd been working on this day in and day out for weeks, but nothing had made it into production. And in a continuous integration, continuous deployment environment where you can get stuff out there piece by piece, that was unusual. And I thought, you know, what's going on here? And I consider this a failing on my side. And I thought about it, so what am I doing wrong here?

But I think there's some maturity in career experience here for this developer. They hadn't learned how to take a project and break it down into iterations. I worked with the team to do that. And I say I worked with the...we kind of talked about this idea, and the team jumped in, you know, and I was just a minor player. But as they became aware of that, it completely changed the workflow of the team I was working in, where everybody focused on, well, how do I break this up into milestones? And, on a large scale, like, having a big project, how to break it up into milestones, and then, a small scale, developers are breaking up.

I had one developer come onto the team who made it like a singular focus. They'd come from another company where they spent, like, months on something [laughs] before they released it, and now they spend, like, multiple pull requests a day. And it's like their life is totally changed, and they're completely happier [laughs] because they've completely changed their workflow. They really embraced this and became kind of an evangelist. And I've seen it; I've seen this just be so successful for people who do this. And even for teams that get good at breaking things up, versus people who just stall for weeks or months. I've seen other...and that's certainly not the only instance I've seen. I just saw that one change, and then that developer became successful.

I've seen other scenarios where somebody jumps into something, you know, a challenging problem, and they spin, and they spin, and they spin, and they get weeks in, and then farther in, and just never get anywhere [laughs]. And it's frustrating for the person who's doing it. And there's a few things that feel worse than being deep in a problem you just can't solve it. And you just feel like there's no light at the end of the tunnel. It's frustrating for everybody who's working around them. It's frustrating for everybody.

Like I said, I think about this over and over again because I feel like it's one of the big skills in engineering that nobody really teaches, and that's a problem. It's just breaking up work. How do you break up work? And that's our topic today is, you know, how do you break up work, and why?

WILL: The North Star, for me...I do kind of want to pull on this a little bit because I've done stuff that just doesn't work like this, right? I mean, like, most of the business development, product ticket stuff that we work with can be addressed in this way. And then, basically, it's just sort of, like, what can you get out into someone's hands? And I would say, like, I don't know how to translate...I don't know how to segue in this, right? But, like, get it out into people's hands, right? And the thing that I think tends to jam people up is, maybe it's just me, but I think other people as well, is, like, you got to be okay with sort of like, let's call it, like, call it actual construction, right, where we're woodworking, or something like that.

You have to be okay making a jig for something, right? You have to be okay making scaffolding for something, right? Where I'm going to build this entire structure, right? Entire structure. But it isn't actually ever going to go out. Like, I'm building it so that I can do the small steps. So, one of the things I'm working on, right, not at Acima, at another very large retailer, is that I work on a mobile app, right? And so, the thing about the mobile app is a lot of the time, we're front-running the back-end API stuff, right? And so, what I'll do is I will scaffold out all kinds of stuff. I'll go and I'll negotiate back and forth with the back-end team, and I'll say like, "Okay, okay. Well, what's the data going to look like?" And they'll say, like, "Okay, it's going to look like this," right?

So, we'll do a contract, right? Which is just a contract. It's just aspirational, right? This is pure vaporware. But I'll get the contract going, and then I'll start charging ahead, right? And I'll just be like, okay, mock data, you know what I mean? Promise that's going to result to this thing, right? And then, I use a tool. It's a reverse proxy. And I can use that to create, like, a fake API endpoint, right? Where it's like, if you go to this API, this URL, it's going to give you back [vocalization], right? It's going to give you just...it's going to serve up this static file, right?

And now, oh, okay, now I'm making, like, real API requests to fake API, right? And all this stuff is junk, right? It's junk. It'll never go live. But I can actually release the thing. It is real stuff. It might even really be on the server hidden behind, like, some feature flag wall, right? And so, it's out. It's in production. It exists. It's a real thing. But, also, no, it's not, right? It's walled off, and it requires, like, all this fake stuff. I'm like, hey, hey, you know what?

And I'll tell you, like, honestly, like, I have hit on a contract before, but I am much more likely to miss on something. And it's like, oh, well, I have to go to the pricing and availability service, so it's going to be here. And I have to make another call. This JSON's going to...and you know what? It's fine. It's fine. I will have to go back and rework, but parsing Jason is really not that hard. In the end of things, it's really not that hard to parse some JSON. And so, like, you know, and then you'll go back.

But, like, if I wanted to do it, like, the quote, unquote "right way," and get it a hundred percent, then I would have all of these sort of, like, blocking features, where I can't move until the feature's done. I can't move until this is done. I can't move until this is done. And I'm just like, eh, F it. Let's roll. And so, you just do it that way, right? And so, you're making assumptions, making trade-offs, debating stuff, and scaffolding, and, like, you know what I mean, going with prototypes. But you're always putting something out into the world. And that has been very, very, very successful. And it's allowed them to get things out at a pace that they ordinarily just...there's no way [laughs]. There's no way.

MIKE: Yeah, it's interesting you mentioned the woodworking. My dad spent most of his career as a professional woodworker. That is exactly what [laughs] they spent most of their time on was: building those jigs, building the tooling to actually do what you needed. I don't know how much time my dad spent building tools. He still...he makes tools. Like, he'll see a problem. He'll go and make a tool because that's how he thinks now [laughs].

About ten years ago, my brother-in-law was showing him some grips for a handgun. My dad's never been, like, a huge gun enthusiast, but he saw that as a problem. He got curious about it, and he went and built a jig and made grips. And it turns out they were better than most of the stuff on the market. And he ended up doing it as kind of a side project hobby that kind of made him a living for several years [laughter] because he took the time to make that jig. Like, I don't really even hear that word outside of that industry, you know, outside of woodworking, but that's the word they use. And it's not exactly a tool, right? It's like a template.

WILL: Yeah. Yeah. Like a template. So, for people who are not necessarily, like, wood butcher nerds...so, I'll give you an example, right? So, a guy I know he made this really beautiful grill table, right? And you could think of it like an outdoor kitchen table with this circular hole, and his grill goes in the middle. And you've got, like, your whole cook surface out there, and it's super, super cool. And I looked at it, right, and it's like, it's this beautiful table. His grill goes right on the side, and you could do all this stuff.

But, like, what he did is he had this beautiful three-foot circular hole for his grill to go on, right? One does not just cut a perfect three-foot circular hole. Like, one does not just walk into Mordor or cut a perfect circle [chuckles] in a butcher block countertop. And so, I was like, "Oh, what did you do," right? Well, he created this, like, not exactly, like, a router. The router was a part of it, but, like, it was this thing to hold the tabletop in place. And he had, like, a thing, and it had an arm that it swung out, and the router cut this perfect circle. But he built, like, a scaffolding machine so that he could cut this perfect circle because you couldn't possibly do that by hand precisely.

I might have dogged it and done it with like a, you know, done it with, like, a jigsaw and then just, like, put a little trim over it, like, a little metal, [laughter] like, a fat metal piece of trim, like, that big. And it's just like, ah, it's bad, but you can't see it, so who cares? So, that's what a jig is, you know what I mean? Like, it holds the work in place so that you can work on it effectively and precisely a lot of the time. Because if you just try and freehand it, you're screwed. That's why [inaudible 10:07] shop projects, you know what I mean? Like, when you did woodshop, if you did woodshop, they all look terrible because...

MIKE: [laughs] Right.

WILL: Because you tried to freehand it. You're just like, oh, well, let's just cut this. And I'm like, no, no, no, no [laughs]. You get something to hold it precisely so that you have a template, and you can do everything perfectly.

MIKE: No, that's exactly it. The professionals, it's not that they're better at freehanding it, although they might be, is they don't trust themselves to freehand it.

WILL: Exactly.

MIKE: And so, they don't [laughs]. They build a system so that they eliminate the human mistake. That completely changes it, and that's how you end up with a quality piece of work. It's such a perfect analogy. You take the time to build the jig, you know, to build the scaffolding, to build the framework that allows you to test and deploy independently a component without necessarily having the other pieces in place. And without that, though, you're blocked, right? Because there's always going to be some other thing that you're working with.

What we do [chuckles] is we read data from one place, and we transform it, and we write it somewhere else. That's what we do for a living. If your data sources aren't ready yet, then you're blocked, unless you define a contract and you code to that. And then you say...you have some sort of test harness that you can test against that contract, knowing what they have may not be ready yet. And you might even have the wrong contract. You might have to make some changes later, but at least you'll have something that you can make tweaks to rather than having to start from scratch.

WILL: I think it's a good rule of thumb. I think it's a good rubric. Although I have been involved in projects where, like, you know what I mean, things were...you had to do...actually, you know what, honestly, I think I've always done that. I haven't always had the latitude to, like, front-run as much as I wanted to. Like, one of the things that I did, like, way back in the day, is I did a bunch of audio processing stuff, right, for phones and stuff like that back when, like, you know, the world was new on the iPhone. And I would do audio processing.

And the thing about audio processing is audio processing, like, there's not a lot of, like, intermediate steps. You really got to, like, you got to kind of, like, do the whole thing. You know, like, you have to have, like, the machine has to, like, do finished products. But, like, honestly, like, I still did the same thing. I mean, in that I would just make it play a song. I was like, oh, I can't play a song, right? Because, like, okay, oh, you want to play a song, right? Well, I had to have all kinds of stuff.

I had to have, like, a wave decoder, and then a wave, you know what I mean, like, a wave consumer, and stuff like that. And so, I was like, oh, that's too hard. I'll just make a sine wave because I could do that in math. I could do that mathematically. That's a mathematical function, right? Then I don't have to have codecs or anything like that. And then, I did that. And I was like, oh, okay, I did that. Okay. And now, I'll do a wave. Okay. Now I'll do a wave, right? It's like, okay, now I'll do an MP3 into a wave into, like, the speaker. I was like, okay, now I did that.

Now, I will take the MP3 into a wave. I'll capture the audio and do nothing, right? Nothing, just a hand, just passing it through, right? And it'll go out. And then, I'll, you know, and then I'll just do, you know, something else, and then, you know what I mean? And so, I would, you know, just kind of do stuff like that.

MIKE: But the process, like, even if you're not releasing it, in order to be able to...and this ties into testing, it seems like, right? Like, you mentioned the testing. At every step, you're doing something that you could verify. You wanted something that was kind of tangible. And I know none of this is purely tangible, but it was...the closest you're going to get to tangible is something...like, it was making it into sound that was making it into your ears. There was something that you could measure that you could validate on. And once you have that, you could build around it.

WILL: Yeah. Yeah. And I built all kinds of jigs. One thing that I did is I was...so, we were working on, like, continuous DJ mixes, right? And so, it was a bunch of MP3s that were encoded and recorded, you know, in such a way so that there would be a continuous stream of music. So, it'd go play from track one to track two to track three to track four to track five. You would never hear a transition, right? You could do that. But I was transforming and manipulating the stuff, right?

And so, like, when it would go over, it would pop. It would be a hitch. And I was trying to debug that one particular thing. And I was like, ah, like, my ADD brain won't let me sit and wait for the thing to roll over. So, I was just like, okay, I'm going to make a jig so that I just jump in five seconds before it ends. And then, I'll play through the transition, and I'll go and do five seconds again, and then I'll, you know, and that's how that goes through the song.

So, I'm only doing, like, the specific part that I want to see because I don't have time for all that, you know? It's just jigs. Just that's a way of doing it, I guess. My dad always taught me...my dad was an engineer, not a woodworker, but he always said, "Go from the known to the unknown." You always want to be working from a known good state, you know what I mean? It can be as stupid as you want it to be.

MIKE: [laughs]

WILL: As dumb, as useless, like, I've looked at a lot of blinking, red lights, bright in my heart.

MIKE: [laughs]

WILL: Like joy. Like, where I'm just like, look at that. Let's blink it, baby. Yeah, we're blinking. We're blinking hard. And I'm just like, is that it? That's what you did today? It's like, yeah, baby. Look at it [laughter]. If you're a real engineer, you're just like, ooh, that's a good blink.

MIKE: [laughs]

WILL: What does that mean? What [laughs] does that mean? What does that light flashing on and off signify in your brain? You know, what is that? It represents progress towards some goal, you know? But, like, yeah, just that stupid, easily that stupid. All day [laughs].

MIKE: You mentioned a blinking light. You say it represented something, signified something, right? You've got that [inaudible 16:18]. You've got something that you can actually wrap your head around. You can verify there is blinking light. I have power [laughs], and it is making something happen.

WILL: We're blinking hard. We're blinking hard today, Mike [laughs].

MIKE: There is a project that was challenging to test because [laughter] there's a...and probably [laughs] anybody who's done this for very long, you probably have encountered this situation before. It's not unique. And you have a partner who doesn't have a sandbox, or they do, and it doesn't work very well. So, you try to test against this third party, and it just doesn't work right. Like, there's not a good ability to test against the API. And when we're running unit tests, we run into this all the time, and we just say "No," right? There's no way you're going to run your unit test suite against the third-party service. And if you do, you're suffering because [laughs], you know, it's so unreliable.

You can't have reliable tests that actually test something if you're running against an unreliable service. And they'll be, like, 100 times slower, and all the problems with that. You know, you build. You mock it out, right? You build a fake service so you can test against that, and that's what you work with. But sometimes we don't think to do that when we're doing manual testing. Like, well, my unit test does it [inaudible 17:39] manual test. I need to test against the real thing. You know, I want to do an integration test. It has to be perfect. Instead of building the tool to test against.

And, in this project, I regret, in retrospect, not pushing hard to say, "Build the test service. Build the test service. It's going to save you so much time." People spun their wheels for hours and hours trying to get stuff done, trying to share a staging environment to test against this sandbox on the service that was sometimes up and sometimes down. Or if they had just had a service running locally that maybe wasn't perfect but it was pretty close to the third party, they could have saved themselves a lot of grief.

WILL: Mm-hmm. Mm-hmm. Yeah. Yeah, absolutely. I mean, I will say that, like, a little...one of the things I've learned at Acima is the many blessings of the VCR gem. If anybody's listening who doesn't know about VCR, basically, all you do is you run your test suite against a, you know, a third-party, or external or, like, whatever, external API source, and you go out, and you hit the real API. And you save it.

And then, you're testing against the real call, but you don't have to, like, go out and do all that every single time, you know. And you could configure it in various ways so that, like, you refresh the stuff, or you make sure it doesn't get stale, and stuff like that. Because, like, yeah, I want the real, real, you know. I don't trust any of you people. You guys are all liars. Engineers cannot be trusted [chuckles].

MIKE: It does give you...you get exactly the response, the real thing. But then you can save it so you don't have to hit them again. So, you get your reliable test. And you can build a service that will respond with that data and run it in your test environments. You can do some manual testing and stuff through the process. Why not?

WILL: Like, I love my reverse proxy for mobile development, right? Mobile development against unstable APIs, shifting APIs, you know, like, I found something today, just today, and I just patched it. If every single thing failed on me, right, I'm hitting a bunch of endpoints, and I'm aggregating together, and, like, none of them should fail. And almost always, at least one of them would come back. But if every single endpoint that should never fail actually did fail, then it was taking down my whole...not my whole app, but my whole, like, page within the app.

I made a jig, right? And I made a jig because I was in a hurry. And I was just like, I'm just going to take this API call and I'm going to, like, I'm going to butcher it so it could never fail, right? So, it's just going to fail all over the place. I'd, like, dug into the source code, and I'm just like, and you will die. Promise dot reject abort, you know what I mean? And then, fixing the problem was really easy because I was sort of refreshing, refreshing, refreshing, but I couldn't get this intermittent failure to go. Unfortunately, I work for a very large retailer, and I will get those millions of requests in the wild, you know, very quickly, you know. And so, anyway, make a jig [laughs].

MIKE: I think Will hit on something critical, which is, you know, you build the support structure so that you can build these incremental pieces. One thing that we really haven't talked about is you get a project, and this happens at any scale, you know, whether it's something that takes two hours, something that takes two days, something that takes two months, something that takes two years. You've got this big thing, and you got to start somewhere. And I think that sometimes we want to see progress, so we just start somewhere, right [chuckles]? Let's chip away at something. And that's one way to do it.

And going back to our starting, where we started this podcast, you can chip away here and there. You end up with a lot of this and that, but the whole thing doesn't move forward. I think you need to be really methodical about choosing what you do first. And it goes back to that jig idea. You have to have some sort of framework, mental framework, to map out how you're going to do the work.

I'm thinking of another project that I worked on a few years ago. I had spent several weeks in meetings with product and business, and they were laying out these ideas. And mostly, what they had was mockups. It's like, this is what it's going to look like. But nowhere in there was anything about how it was actually going to work [chuckles]. And there were a lot of really big questions. You know, here's how it looks like in the user interface. But how on earth are you going to make this work behind the scenes? You know, go and work with whatever providers, and leave out the details. But, you know, you're going to need third-party data to make this work, and it's not trivial.

And after doing this, I got together with a skilled product manager, and we sat down for a few hours and hashed out a roadmap and said, "These are the milestones we need to hit." And that roadmap, more or less unchanged, ended up being how that project got done. It was on a really tight timeframe. We got it out in about three months, made it before Christmas, and you get the release. It wasn't the greatest product [laughs], but we got the thing done.

And the problem was actually not our implementation as much as that we were trying out a new business idea that wasn't as good as, you know, as they'd hoped. It happens. The important thing is we got it done. And the only reason we got it done, instead of spending the next two of those three months in endless meetings talking about the user interface, is that we sat down and said, "Well, what is the practical plan? What could we do first?"

And the thing that we could do first...and then, we went and actually talked to the business and said, "What do you need first?" "We need to be able to accept a credit card payment," or whatever it was. And having that first thing opened the gate to do another thing. It's like Will's blinking lights or being able to play a pure tone sine wave [vocalization] [chuckles].

You get that. You get something to play. You get something to start with. That zero to something is so much more than even the subsequent steps. Something to hang on is just wildly better [chuckles] than nothing. The zero to something is the biggest, hardest part. And getting that tangible sort of thing made a huge difference. And then, other people can start talking about, where do they fit in the process? You know, you can start coordinating across teams to figure out where they fit. And pretty soon, you have a project running. And you do that a lot.

I've worked many projects like this. I'm working on one right now, where the very first thing we did is we sat down, laid down the roadmap, and the project's going well. It's going to be...other than some challenges dealing with a government entity that's just sort of slow [laughs], like, we're going to be on time, close to on budget, you know, it's just going to work out because we've mapped it out like that.

You see, the difference is really not that much. You know, you take a few hours and grapple with the problem and think about how to break it up, rather than getting overwhelmed and just chipping at something and getting one little piece. You're probably never going to make it very far. Now, chipping one little piece is better than nothing, but I think...well, maybe [laughs]. But I think that taking some time to analyze the problem and it's work, and figure out what comes first, and then what comes next, and then what comes next...well, where are your most risky pieces? Maybe you have some government entity you have to get something approved by. Well, you better get that done first.

You know, laying out an order of operations and doing those is just, I think, one of the most valuable things that we do. And I don't think I was, you know, I don't know that it's something they teach in school. They just expect you to know it. But I feel like that's something that ought to be just repeated over and over and over again.

WILL: I couldn't possibly agree more. And I'll tell you, like, also, like, I mean, it's like, you know, sort of, like, in terms of, like, the tactical sort of, like, thrust and parry dealing with, like, business product, stakeholders, stuff like that. It is...how do I put it? Like, if you can distill a feature set down to, like, the absolute, absolute, absolute minimum, like, when you're faced with sort of, like, you know, a lot of demands, a lot of shifting demands, a lot of crazy deadlines, a lot of pushes, and stuff like that...And I understand that, like, what I'm saying here is just, like, plain vanilla, basic agile. But it's the kind of thing it's that's easy to say but very, very difficult to do.

But, like, the more you could distill it down to, like, just the basic core, raw functionality, and then, like, you add stuff on it, I found that when you get that sort of, like, that core workflow finished and you find yourself in a situation where it's like, hey, I'm not going to make this deadline, most product people, like, that I've dealt with, have been very practical, and pragmatic, and reasonable. If you give them something and you say like, "Listen, I can do this other stuff, right, but it's going to push it by a month. Do you want me to do that?" And then, they'll just be like, "No," like, they can make a purchase. They could buy a widget. They can do this thing that they want to do.

And, like, really distilling that down, finding the stuff, where it's like, oh, well, you know what? I'm not going to handle errors gracefully. Like, I'm just going to be, like, is it right? Cool. And then, you get the green check. Is it wrong? No. I'm bailing you out. I'm dumping the whole thing. I'm just dumping, right? No slickness, cleverness, anything. If you can give product scenarios like that, then I've found my ability to have an untroubled existence as an engineer while still maintaining a reputation for actually putting out products has, like, it's gotten so much better.

I don't know, I mean, they get it, but, like, when it's not done, when they don't have a done thing that they can either invest more in or they can ship right now, then, like, that's when the sort of, like, the wish list grows, and grows, and grows, and grows, and grows because it's not finished, right?

And this is real tactical stuff, but if it's done, and they have it, and they can either go to market, or they can delay. If you give them the choice, they almost always go to market. If you could give them something that works, they'll take it, but if you don't, then it's just like, oh, I got to have that. I got to have this. I have to have this feature. We couldn't possibly go without this feature because it's not something or nothing. It's nothing or another flavor of nothing, whatever. That's your problem, you know.

MIKE: That, what you just said, you know [laughs], if you've got nothing, like, you imagine any kinds of nothing, and what you have is nothing. But if you have something, something, right, you can work with that. And maybe it's not enough to go to market with, but it's at least something you can talk about.

Like you said, this goes back to the agile process. The core idea of agile software development is around communication loops. You want to have an iterative cycle rather than no cycle, you know, just planning upfront, build it, and then done. And you can only have that iterative loop if you actually have something to iterate on. And we've talked about this before a lot of times.

You look at aerospace [laughs] because it's big, and things blow up, and it's easy for us all to see. You look at what SpaceX has done to just absolutely dominate, not just, like, the U.S. market, but they launch more rockets than, like, the whole world put together. Like, the nation-state, right [laughs]? And it's because they have this very iterative process. You know, they're like, well, we're not going to build a perfect rocket. Let's build something that's maybe going to blow up, but at least it should be able to get off the ground, and we can measure some stuff on it. And they do that, and it probably blows up.

But they got those measurements, and then the next one gets a bit higher. And the willingness to let things blow up and to be interested in that interaction, right, in that iteration more than in the perfection, drives the costs wildly down.

WILL: I'm actually really interested in, like, so one of the other Elon projects that I'm personally extremely interested in is, like, the full self-driving mode on the Tesla. And one of the things that I'm very, very interested in, in that capacity, is willingness to...I hate to say, like, blow up some rockets, but, like, willingness to put out a full self-driving automobile that is just a little bit better than a human, right? Because, I mean, we're all right. We're not great, you know, we're all right, you know.

But I think if General Motors, you know, or one of the large institutional players was going out and putting together a self-driving car, or even somebody like Google that has a lot to lose, I think they would and do really struggle with, sort of, like, something that's just better and not perfect, right? Like, no, I don't want it to be worse. I don't want it to be worse. That's not cool, you know what I mean? Like, we can't have, like, flying, like, battering rams just barreling down the freeway murdering people. Like, that's not okay. However, what if it's just, like, 20% better than a human, 20% safer than a human? Isn't that better?

MIKE: That is better.

WILL: You know? Anyway, I mean, and so I think that that mindset, I think is, I don't know, I'm interested to see how it goes out. Because, like, it's also one of those sort of things where it's like, you know, you're kind of, like, gambling with somebody else's life there a little bit, Elon. And move fast and break things is not meant to be taken quite so literally.

MIKE: [laughs] That's an interesting one because to drive better than a human, you have to have superhuman skills, and let me explain what I mean. I'm going to have a very specific meaning that may not be what you're interpreting me to mean. And they call this the March of Nines. Have you heard this Match of Nines? If you want to have 90% accuracy, it takes a certain amount of work. And then, to get to 99% accuracy, you've only got one digit further, right? It's like ten times as much work. And to get one digit further, you know, 99.9, you got to do about ten times as much work, and so on.

And each one of those is harder, and part of the reason for this is that you think about driving, and most of the time, driving is pretty boring. Most of the time, driving is pretty boring until the nearby zoo has an error, and then a kangaroo ends up jumping across the road in front of you. And to deal with that situation, you have to have the concept...well, maybe it's not even a kangaroo, right? You know, maybe it's something that you've never even seen before. I almost hit a possum the other day, literally almost hit a possum on my bicycle. And [laughs] had to slam on my brakes, or I would have run over the possum [laughter]. That's not something that happens every day, right? You don't every day go riding along and almost hit a possum.

And I didn't necessarily need to recognize it was a possum, although it took me a moment before I realized what it was. You just have to know that there is something that is walking across the road, and I need to stop. But having that concept of an animal, and that it has direction, and knowing which way to steer because you can tell which direction it's going, and this is very much out of the norm, requires an understanding of a lot more than driving, right? You have to understand something about biology, and something about physics, and some other problems.

What if the street sign's broken, and there's a police officer standing in the middle of the intersection, and they're waving people with their hand, and doing hand signals to say which way to go? Now, your car needs to not just understand traffic lights. They need to understand human gestures. And maybe the police officer is just yelling at you, "Hey, please! Your turn. Go." Now, you have to understand human speech and have a microphone.

Like, as you deal with more and more of these situations, there are situations that require human understanding because our world of driving is built around humans. And I've actually heard this talked about by self-driving engineers. It's on my mind. These roads are not designed for the machines. And so, the problem is a challenging one and challenging in ways, like, it's hard to measure.

Now, this doesn't really go against what you were saying before. You may be able, like, because humans aren't that great at drivers, make the car 20% better, and then have, like, a data center somewhere where people are sitting there, and they'll take over. And they'll have a little camera and see, "Oh, the police officer is waving. Okay, make the car go left, you know, and take over manually." I mean, you might have something like that where you actually take human control or give it back to the human and say, "Hey, you know what? I'm not going to self-drive anymore. Your job." There are solutions like that where you just bail.

WILL: I mean, honestly, like, what I think ought to be done, I mean, like, I don't know. I mean, like, everybody's sort of, like, dumping money into this intention of, like, you know, like, carving out their, like, little digital fiefdom monopoly thing. And I don't think transportation is, like, the kind of thing that can really support that kind of a capitalist enterprise because it's public good, you know?

And so, I think it should be a public-private partnership, where we start creating roadways that have, you know, affordances built into them for self-driving cars. And, like, this is just me, but it goes right back to, like, what we've been talking about. What if your self-driving car was just good on the big streets? That's it. It's good on the big streets, you know what I mean? Like, get it to, like, a four-lane road, you know what I mean? Like, whatever your four-lane stoplight road is, you know what I mean? And you push a button, and you say, "Bang it out. I'm watching Netflix," you know what I mean? And that's it.

MIKE: [laughs]

WILL: That's cool. That's cool. We're good. We're good, man. This is a good gig. Like, I could watch Netflix. I could check my email. I can do my stuff. I don't have to sweat the traffic jams on the highway. I don't have to sweat stop-and-go traffic. I'm cool. We're all right. And more over than that, you know what I mean? I could do even better than that. And I could just say like, hey, you know what? I got my e-bike. Like, I'm just going to go to the city bus...and my city bus, instead of being the bus, you know what I mean?

So, I have to, like, pedal, you know, a block or two to, like, my transit point, throw my e-bike on there. And then, it just takes me right to my office, and we're good to go, you know what I mean? And cars have gone away. We didn't solve the hard part. I don't need to know what...like, my car doesn't know what a zebra is. It doesn't care, you know what I mean? Like, I just built, you know, I built the car to deal with, like, you know what I mean? I bounded the problem, right, so that the self-driving stuff works. And I made the roadway in a way so that the self-driving stuff can, like, process it and, like, we're all good. Any true engineer is always looking to cheat.

MIKE: [Chuckles] And that's been kind of the theme of the conversation, right? A good engineer as Will is interested in cheating. Don't freehand it.

WILL: Don't freehand it, yeah.

MIKE: Because that's a solvable problem [laughs]. But you can take away the human element, and then you have a different problem that's solvable.

WILL: Exactly. Yeah. My hand shakes just as much as yours does. That's why I made a jig, and that circle is flawless every time. I can make a thousand of them, and each one would be just as perfect as the last.

MIKE: Well, that feels like it kind of ties things up, doesn't it [laughs]?

WILL: I love it [laughs].

MIKE: We went from solving our everyday problems to dealing with the big problems of the world. And they have the same constraints. We will close this up. Thank you, all, for listening. And, you know, build a jig [laughs].

WILL: Build a jig.

MIKE: Build a jig.

WILL: Good engineers cheat. Build a jig [laughs].

In this episode of the Acima Development Podcast, Mike and the panel talk about the intricacies of project estimation, sharing personal anecdotes and professional experiences. Mike begins with a story from his childhood, illustrating how his father's ability to estimate distances accurately influenced his understanding of estimation as a skill that can be developed with practice. He draws parallels between estimating physical distances and estimating project timelines, emphasizing the importance of breaking down tasks into smaller, manageable pieces to improve accuracy.

The conversation transitions to discussing the best practices for project estimation within engineering teams. Matt and Justin highlight the effectiveness of comparing new tasks to similar past projects and the necessity of having a stable team to ensure consistent delivery. They stress the importance of planning in small increments and maintaining clear communication among team members to mitigate risks and handle scope changes efficiently. Mike adds that understanding and identifying unknowns early in the process is crucial for making realistic estimates and avoiding common cognitive biases that can lead to significant project delays and budget overruns.

Towards the end, the team explores the challenges of maintaining estimation accuracy during organizational changes, such as team reassignments and company-wide reorgs. Will raises concerns about the impact of these changes on planning bandwidth and overall productivity. The group agrees that preserving team dynamics and ensuring clear lines of communication are essential to minimizing disruption. They discuss the importance of having documented rules and metrics at every planning level to measure effectiveness and ensure alignment with business goals. The episode concludes with a reflection on the necessity of disciplined planning and continuous improvement to successfully manage complex engineering projects.

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I have with me here Will, Justin, Eddy, and Matt. And we're going to be talking today about strategies and pitfalls in estimation. I'm going to start with just a story from my childhood.

We were talking in the pre-recording for a bit about math. I was remembering that my dad used to ask math-related questions to me and my siblings and make a game out of it. Like, he wasn't, like, grumpy, or pushy, or anything. He was like, "What do you think of this?" And then, we'd think about it. This is on road trips. He'd always do it on road trips. Like, that's the only time I ever remember this happening [laughs]. You know, you're on a road trip back in the long ago era before there was internet in your car [laughs]. And you had to do something to fill the space.

I remember that there was mountain ranges, and he'd pick a landmark. You know, there'd be, like, this mountain. "How far away do you think that is? Guess." And my dad was always right. And all of us were wildly wrong [laughter]. He was always really close. My dad's a carpenter. He's very, very good at recognizing distances. Like, he can look at something 15 inches long. He'll look at it and say, "That's 15 inches [laughs]." And he's right. You know, he's as good as a ruler [laughs].

He's just really good at that perception, you know, distance perception. But it applies even to larger scales. So, you know, he'd say, "How far is this away?" And I'd say, "Oh, that's three miles." And my sister would say, "Oh, that's two miles." My dad would say, "Oh, I think that's 13 miles." And it'd probably be, like, 12.5 [laughs]. And it's something that you have to work on. I've seen this.

Most people, you ask them, you know, "How long is three inches?" And they don't know because they don't work with it every day. Like, how would you develop that skill unless you're a carpenter [laughs], right? Unless you're practicing with it many times a day. And estimation of how long something takes, I don't think, is any different than that. It's something that can be developed as a skill, but it's hard. And it's especially hard when we're doing that with projects that you're working on.

You don't see the mountain out there, right [chuckles]? So, you don't really have a good landmark. You're basing it on where you think the landmark's going to be. So, you're not just guessing how far is the landmark, but you're guessing where that milestone is. And it makes for a really difficult problem that hits some of our cognitive biases really badly. And we just get it wrong over and over and over again. So, that's kind of framing of the problem.

Just in general, distance estimation is a tricky problem that you have to develop as a skill and estimating projects. And I think this, you know, we're talking about engineering, but it probably applies to other kinds of projects as well. It's a hard problem because the scale is hard to gauge, and, again, it hits some of our cognitive biases.

WILL: So, I've got a question for you guys, maybe for the group, right? Like, in your experience, right? We've got some miles under our belts. What's the best situation in terms of estimating projects and delivering on those estimations consistently that you guys have all individually experienced firsthand, not somebody that you read about in a book, or a friend's thing, or somebody's plan? Like, what's the best setup that you've had yourself?

JUSTIN: So, the best way for me to estimate something is to get asked to do something that I've done before.

[laughter]

WILL: No, no, no, no, no, no, no. Hang on, hang on, no, no, I'm talking about, like, the best setup, like, doing the work. The best setup where you've been like, "Hey, we're going to have to do this thing." And you're like, "Okay, I'm going to do this thing," you know what I mean? And, like, "I think it's going to take me six weeks." And it takes you, you know, only eight, you know [laughter], or whatever it's going to be, like, the best system that you've worked in yourself.

MATT: I find what works the best for myself, anyway, is to do a comparison on something I've done that's very similar in nature. You know, you have that mountain that Mike spoke about, and it's a lot easier than just conceptualizing it. If you can relate it to something else, then it's a lot easier to break down into the problems. And I find the smaller you can break those problems down, and if you can estimate the problems one at a time, then you have a really good idea of what the larger scope is going to be. That's where I've had the most success.

WILL: Have you had the opportunity to consistently work in that fashion? I mean, like, has that been, like, an ongoing experience for you, or is just, like, what you would like to work on if you could but you can't?

MIKE: I've seen this on teams I've worked with and teams I've worked closely with. And Matt hit on a few of the things that are critical. First of all [chuckles], you have to break it down into small, discrete pieces that are small enough you can wrap your head around them. That is absolutely critical, and that is not free. It takes time.

You have to break down the work into a map and, you know, say, "I'm going to do this, and I'm going to do this, and I'm going to do this." When I say "I," a collective I, you know, the we. We're going to do this, and we're going to do this, and we're going to do this. And, you know, we usually use story points in software development, which is this unitless measure, right? You say, "This is how complex it is." And if there's anything above a three, then you're not good. It has to be small, discrete projects that somebody can say, "Oh yeah, I know how to do that."

Unless you've broken down everything into, I know how to do that, and they're in order, then you're going to be off. And if you do that, though, if you have everything broken down into, like, nothing bigger than size three, and you have a team that you've worked with before, because unless you have some sort of history to know how...you have to have a history. There's a big caveat here. You have to work together as a team for a while so that you know what you can do as a team over time in terms of those points. Then, it tends to land very close. Maybe not exactly perfect, but on average, pretty good.

WILL: I agree with all of that. And you can't mess with the scope midstream.

MIKE: So, you can't change the team. You can't change the scope. Those have to be fixed. You change the team, well, now you've changed the dynamics, right? You've changed the dynamics of how you work together, and work gets done as a team. It doesn't get done as individuals in that way because the team dynamics are relevant. So, you have to have a stable team. And, yeah, if you change the scope, well, that has to be planned just like everything else, and it has to be additive to it.

MATT: Scope changes will happen, but if you tackle them the same way you tackle the initial problem, then you can also estimate, hey, it's going to add X number of weeks to this particular project, right? Or whatever piece of time that is. The key is small, workable pieces.

JUSTIN: Clearly identifying the unknowns.

MIKE: Well, if it's unknown...so, I'm going to dig a little bit more on that because I think it's critical. Identifying the unknowns, that's not a one-liner [laughs] because...so, I mentioned that, and I said this when I was talking about it, you're measuring the complexity. I don't think the story points should ever be used to measure time when you're estimating. They're not a measurement of time.

MATT: Absolutely not.

MIKE: They are a measurement of complexity. And if there is unknowns, it's not truly estimable because that means that you don't understand the work yet. Like, the rules for this plan, right? This only works if you can do everything that I said, which is that you know what those pieces are. I mean, you don't know exactly what you are, but you can wrap your head around it, right? And you know what you can do. If there are unknowns, that means that you haven't refined it down enough. And it's a measurement of complexity. As for those, you'd bump it up to, like, eight or, you know, something outside of what your boundaries are. Because unless it's down to three, then you don't really know. And here's a critical reason why.

I read an article about this, and I don't have it up in front of me, years ago, but it's brilliant. It says that we're very good at estimating the median time to finish, but the mean is way higher than the median. So, if you line up all the projects, and you pick the one in the middle, we're pretty good at that. So, talking about these cognitive biases, we're actually pretty good at picking that one.

The problem is the complexity is very non-linear, and we have a hard time wrapping our heads around non-linear things. So, anything to the higher end of that median doesn't just go up linearly. It goes up exponentially or worse. So, the projects that are harder, that are worse than that median, might be ten times worse, or a hundred times worse, and so they drive the mean up really high. And then, you have a project that's three years late and over budget because it's really hard to predict where those unknowns and complexity are going to land.

MATT: And you can't predict things that aren't known. That's the whole thing. Generally, yes, you're going to bump into a few unknowns. Part of it is trying to account for a rough percentage in your complexity. I mean, and it's really, I would say, experience. There are certain things that you encounter often. You just really have to think through and understand your problem to be able to accurately represent it.

JUSTIN: Yeah. And it's almost like going back to what you said, Mike, you've got to resolve those unknowns before you can give an estimate, and so that's separate work. It's, like, a spike to resolve the unknowns.

MIKE: Yes, absolutely. Listeners can't see this, but I just posted in our chat together. There's an xkcd comic that I think is fantastic. Boss is coming in, talking to the developer. He says, "User takes a photo. The app should check whether they're in a national park." She's like, "Sure, easy GPS lookup. Give me a few hours."

JUSTIN: [laughs]

MIKE: He says, "Check whether the photo's a bird." She says, "I'll need a research team in five years [laughter]." In computer science, it can be hard to explain the difference between the easy and the virtually impossible, and that is so true. And there's this thing that things that are easy for humans are hard for machines, and vice versa. Like, yeah, fold the laundry, sure, I'll do that in five minutes. Have the machine fold the laundry? Good luck [laughs]. Good luck. In your mind, that little mental shortcut that says, "Oh yeah, then I'll do the next step," may not map very well to how you actually make that happen.

JUSTIN: As an aside, that comic, the second comment from the lady programmer, says, "I'll need a research team in five years." Nowadays, it's like, "Let me make an API call to ChatGPT." It's done in five minutes.

MIKE: Well, what's even better is the [inaudible 11:24] message, which, [inaudible 11:27] xkcd is perfect. It says, "In the 1960s, Marvin Minsky assigned a couple of undergrads to spend the summer programming a computer to use a camera to identify objects in a scene. He figured they'd have the problem solved by the end of the summer. Half a century later, we're still working on it [laughter]." That's how it goes.

WILL: So, I'm curious, what is this, like...so, this is an organizational problem, and I think we all sort of, more or less, like, understand, like, how to manage this, right? But on an organizational level, it's really, like, sort of, like, interfacing with, like, the broader teams and, like, where's it gone right and where's it gone wrong and how. Because I myself, like, I've experienced some pretty functional teams in that regard in that, like, you know, we come in with, like, sort of a product plan. We plan the tickets before the sprint starts. We hold to the sprint scope. We plan everything out. Boom, boom, boom, right down the line. And we do MVPs, things like that.

And, you know, I mean, I'm sort of, like, experiencing right now in real time a degradation of that process, where product doesn't want to come into the sprint with a defined scope. Our product wants to give me...they want us to start work when I don't have, you know, a Figma design document that tells me where all the buttons are going to be and what they might want to do, you know what I mean?

I'm going out, and I'm developing the frontend when there is no backend at all. And I'm sort of like, and I'm like, "Well, do you have a schema for what the backend might eventually look like?" "Well, not really," you know what I mean? You could work on all that kind of stuff, but it increases [inaudible 13:16]. It increases rework, inefficiency. And, quite frankly, like, it really kind of pisses me off, you know, like, where, like, I have to, like, my job gets three times harder because people couldn't be bothered to, like, you know, make an effort at the outset of the project. Now I'm kind of curious about, like, sort of like, I've seen it go good and, like, where you've seen the process break down and how.

MIKE: You just hit on, like, all the failure modes, right [laughter]? Requirements not gathered before the work starts. Now, in the real world, sometimes you don't know everything. But, like, you should at least have the overall direction defined so that...certainly, by the sprint, you know what you're going to be working on. If you don't have the requirements, that's just a recipe for burnout and over budget, over time, over everything.

And cross-team, you talked about different teams that are not in alignment. And you're like, maybe the backend's not working yet, but the frontend is. And so, now you have all the misalignment between teams, and getting those coordinated is always, like, the hardest part [laughter]. And this sounds like you're also talking about some team dysfunction, where people are like, "No, I don't feel like doing that." Like, you don't have the sense of collaboration between the parties. And, man, if you don't have engineering and product working together, you're in for a world of hurt.

And why does this happen? I think it tends to happen for the same reasons repeatedly is that unless we're working at non-profits, and I think it even applies there, I mean, we're trying to make money, or, you know, we're trying to maximize our utility, and something comes up. They want to get it done. They being leadership at some level, wherever that is, it can happen at any tier, wants to get something done. And you don't have a great process for aligning that to reality. Then you're going to have more than you can get done.

And there's going to be this misalignment of reporting because you said, "Oh, I gave a prediction." But now you've added to the scope. You've thrown some other things in there, or you changed the team. You've done a reorg. Whatever it is you've done, you've broken the rules. And when that happens, things get out of kilter, and it's hard to get them back because you need to make a conscious effort to say, "No, we can't do this." The only way this works is when we follow all the rules. And there's a lot of times when somebody is going to not want to follow those rules. And it's hard to explain why those rules are important. It involves math and [laughter] discussion, yeah.

WILL: I mean, I suppose so, and I agree with what you're saying. But, I mean, like, I'm a lot more interested in sort of the fundamental psychology of, like, the planning. And the reason I'm sort of most interested in it is because if we were all sort of, like, very rational, logical actors, even on an upper management leadership level, the better you plan the stuff, the more efficient things run. And, honestly, I mean, like, really spending time in investing up front in your product planning and development that will make you more money.

We do better when we're very, very, very deliberate about when, where, why, and how we're taking on projects. That lack of discipline, like, it not only hurts, like, sort of the engineering efficiency, but, in my view, it's extremely hazardous in terms of, like, business impacts.

JUSTIN: So, Will, I mean, you're going more towards the clearly defined problems and everything, but, classically, that's called, like, the waterfall solution, which is a bad word these days, right? So, where do you draw the line between planning everything and planning just enough?

MATT: But planning being the operative word here, right? You can be extremely agile. You know, I've worked where we were extreme programming weekly sprints. We planned every single week. We had our work ready every Monday for that week. You can break problems into smaller problems and then just plan those smaller problems. But you do have to have a really good idea of what the primary problem is, right? And the thing you're trying to solve for and your ultimate goal. But once you understand that, you can break it into small problems, and then you plan pieces at a time, and you can be super effective.

WILL: I do agree that, like, waterfall is a dirty word, and it should stay one, frankly. Because where I see things go, like, most badly, is shipping these grand visions versus tiny, little, agile, incremental pieces. Maybe analogize it with the sort of, like, the Apple iOS grand reveal of the new version of their operating system. Versus maybe, like, the Google Android, like, things are coming out all the time. Like, what's even the current version?

JUSTIN: [laughs]

WILL: Is it Oreo, or Nougat, or Zeta [laughter], or whatever? I don't even know. Like --

MATT: But do you care?

WILL: Come on, who cares? Stuff's coming out just helter skelter pelmel, all the time, right? Versus, like, this, like, this grand, like, unveiling that happens once a year. It's cool that Apple can pull that off, you know, but, like, --

MIKE: Will, do you think that, internally, they're actually having a grand pull-off? Or do you think that they're doing the same thing that Android is doing and they're just tidying it behind the scenes doing these incremental builds, and when they finally have it ready, then they push it out? I think it's the latter. Like, I think that --

JUSTIN: Yeah, I think it's the latter, too.

MATT: Yeah, I mean, when you're incremental, you're mitigating risk.

MIKE: Exactly. There's huge risk when you're doing it. So, there's aerospace example, and it's easy because it's big, expensive projects people know about. There's the Space Launch System recently launched successfully. After years, you know, they finally have a launch. I don't think they...[laughter] they, like, don't even launch a real payload, and it costs, like, two billion dollars, or something, just to launch, not the development but, like, the launch costs that. And then, I read the other day that one of the SpaceX rockets, the Falcon 9, did its 20th launch, reused, like, the same rocket, launched its 20th time.

[laughter]

And, you know, I'm sure it costs some millions, but orders of magnitude difference in price. And SpaceX has blown a lot more things up, but they did it on purpose because they said, "We're going to do this incrementally, and we're going to start small. And we're going to iterate." And now they're just the steamroller that is launching more rockets than, like, all of the nation states put together.

MATT: They're miles ahead.

JUSTIN: Yeah, if only they weren't led by a crazy guy, but anyway.

WILL: Well, you have to be crazy.

MATT: That's why they are so successful [laughs].

WILL: Because, I mean, like, there's...I --

JUSTIN: Okay. So, at the start, he was crazy, but he was focused on his stuff. Now he's crazy and focused on other things; at least, that's what it seems like.

WILL: A lot of those guys are difficult people. Steve Jobs, God rest his soul, [inaudible 20:24], you know what I mean, had his halo shined considerably in his death. And, at the very least, he was disciplined enough to keep his mouth shut and most of his sociopathic behavior behind closed doors. But, I mean, like, the stories abound of that guy doing terrible stuff, and he never really stopped. His PR team got better, but, like, he was a jerk till the day he died. God bless him, you know?

MIKE: One thing that they have in common, though, is that they honor the engineering process. And I'm not going to even touch Elon Musk as a person, right? That's a topic I'm just not going to bring up here. But he came from the software world and brought that engineering process and cultivated it among engineers.

MATT: That is very true.

MIKE: It really doesn't matter about him, right? What he brought was the process. He brought the agile engineering process with him from software world and just applied it to aerospace, and it works. If you do it, it works. If you don't do it, a recurring theme here, it doesn't work.

WILL: I think it is a validation of that process, you know what I mean, in a lot of different, you know what I mean? You're moving that into other industries and avenues, provided that it is permissible to explode the rockets, and there's nobody [laughter] on them.

MATT: And we all agree, waterfall, bad word. But I would argue a waterfall process is better than no process at all.

MIKE: Sure. So, I think a lot of people they think, Agile, hey, we don't have to plan anymore.

JUSTIN: [laughs].

WILL: You can't be farther from the truth.

MIKE: Exactly. All you're doing is you're tightening your planning loop, right? You're tightening your feedback loop. And instead of doing all of your documentation upfront when you know the least, you are choosing to actively plan, and document, and iterate the whole time. So, instead of doing it once, you never stop doing it. And if you think that being agile means, oh, wait, you know, I'm just not going to plan at all; we'll just wing it, then you're missing the whole idea, which is that these interactions, the communication is hard, and that interaction is critical. And you need to have interaction with your hardware, with your stakeholders, with your software. You know, you're testing it. You are evaluating it. You're improving it continuously rather than once.

And it's actually more planning work in the end because you're doing it along a line. But because it's distributed, it's more efficient because you're doing it when you have the most information, which is after you've already done some of the iterations already.

MATT: And it becomes easier with experience because then you have that association. If you've been doing this for a while, which all of us have, there's very few problems that we have never seen before. They just are in a little different context. But, generally, it's the same problems over and over, and we know how to solve them. I think --

MIKE: You're taking text out of the database and transforming it and showing it to somebody?

MATT: Exactly [laughter]. It's only data, right [laughter]? But, you know, there are new things coming out, AI, for instance. GenAI is new to a lot of people, and we're going to learn that as well. And it may be a little harder to plan those technologies because we don't have the experience yet. So, we may not be as accurate with our estimates on those types of projects. But, for the most part, most of the work we're doing, we have done before in some form. And if you break it down, isn't any different than other problems we've already solved.

JUSTIN: So, I do want to add a caveat to that. Well, not a caveat, just, like, an addendum. It's interesting the organizational changes that are happening within Acima, right? Well, Upbound, right? And last year was, like, a record year for Acima and RAC. And this year, we've got a big organizational change where teams are shifting. Directors are shifting. Product owners are shifting. There's a lot of moving parts here. And I'm afraid that they're going to expect the same record level production from an engineering standpoint that they had last year this year because it's, like, oh, we're reorganizing in a more efficient manner. That means we're going to do even better. And it's like, maybe long term, but that short term is really going to suck.

MATT: Well, and maybe it's up to us, right? Well, it is up to us. I really think, just highlighting what I just said, if we look at it as these aren't different problems; they're the same problems, whether I was on Team A or Team B, I'm still facing the same types of problems today that I was yesterday, and we approach it in that manner, I think we can be absolutely as efficient. Change is hard for people. When you are shifting so many things around all at one time, it's up to us to kind of get our bearings and have that compass to remind us that we're still doing the same thing.

MIKE: Well, there's a couple of things that need to happen. First of all, we should work very hard to preserve team boundaries where we can. Like, let's not disrupt high-functioning teams. Second, leadership needs to be focused on using this as intended to improve lines of communication and not just change them. You know [laughs], let's just shuffle things around, and it looks better on a whiteboard. That's just going to slow us down.

If we see that there were...you know, in a large organization, there are difficulties with communication; of course, there are. And if we reorganize to try to improve that while allowing the engineers to keep doing their work with minimally disrupting the teams, then we could actually become more effective. But that is a, you know, to Matt's point, it's up to us. Like there is a decision point here, and we have to be aware of that destination. We have a choice, right? We make things better, or we make things worse.

WILL: So, I mean, in regards to, like, sort of, like, these major, like, company-wide reorganizations, how do you account for and even measure sort of, like, impacts of the reorganization, right? Like, impacts on sort of, like, planning bandwidth, right? I mean, you've got people who are, like, coming up with a plan for the next sprint, writing the tickets, like, measuring the stuff out, right? And so, like, they have a certain amount of planning that they can do, right? And that's affected by their familiarity with, like, their operating environment, right?

Like, you bring a manager into a brand new team. They haven't touched any of the code. They have, you know, a certain amount of planning bandwidth, right? And, like, that's going to be constrained by them getting up on the stuff, right? You have a new team, right? Okay, you know, like, you don't want to disrupt high-performing teams. But, honestly, like, maybe you do want to give your A players your biggest impact priority work, right?

So, you move those people in, and they have to acclimate, right? How do you account for impacts of reorganization? Because this is something that I have seen as well. You know, the large retailer that I work for they love their Q1 reorg. They love, you know, rearranging the furniture. But they're going to have a cost, and I don't know that the cost is accounted for organizationally.

MIKE: And because it's hard to measure, it doesn't get measured, and --

WILL: If it doesn't get measured, then they can't plan for it, you know what I mean? Upper management, you know, those executives that are sort of, like, rearranging the furniture they don't have that. Do they have that in their heads? Do they have it in their mind? There's a cost. We've all felt it.

JUSTIN: I think you treat it as you would, you know, you're onboarding any new member to your team. This is something that happens all the time with engineering teams. People leave; people come in. And you have to get to know this person and, you know, put them through the same onboarding and really get to know their capabilities and stuff. And so, I think your estimates are going to suffer a little bit until you've had a couple of sprints with them. You treat that as an unknown, and make sure you communicate that up. That's kind of what you would do at the team level.

And then, I would think that the folks in the engineering leadership, if they make changes at team levels like that, they just need to get those lines of communication going such that you can set expectations and really understand, "Hey, we just on-boarded two new people, you know, our estimates are going to suffer for the next sprint or two. And then, we'll be able to give a more accurate estimate after that."

MIKE: And there's that bigger question. What metric can you use to measure, aggregated across teams, across an organization? What metric can you use to capture effectiveness as you're doing a reorg? Because we know every metric captures a slice of something and has its own biases. So, what would be an effective metric? Does anybody have an answer to this? What's an effective metric to see how...when you do a reorg, what do you use to measure it? Did this work?

MATT: Velocity over time. I mean --

MIKE: The tricky thing is velocity is team-specific. You rearrange all your teams, velocity changes. If you measure that, then everybody's going to say, "Oh yeah, we're just going to have..." this is five points. That's not three. That's five. Like, you're going to bias it toward higher story points, and velocity ceases to become effective as a number. So, it's hard to measure that precisely. And business initiatives aren't exactly equal to each other. So, you can't necessarily say, "Oh yeah, we've done five initiatives per year [laughs]." It's a tricky problem.

JUSTIN: Yeah, I think if you had an answer to that, you could write your software engineering estimation book, Mike, and retire on that, so...

[laughter]

MIKE: But it's not a new problem. It's not a new problem. Everybody's dealing with it. And then, we fall into these gut, you know, we don't have an answer, so we just kind of look the other way and go with our gut and don't measure it very well. And then, maybe you have your annual reorg [chuckles] that does more harm than good.

WILL: Well, I mean, I don't know. I mean, I feel like you ought to have, internally, a general rule of thumb for, like, okay, if I have a new hire, this is what I expect the onboarding to look like. At week 1, I'll get X. At week 5, I'll get Y. At week 10, I'll get Z, right? Like, you'll have an arc of their productivity, at their level, right? Like, you know a junior will onboard. You know a senior will onboard, you know, et cetera.

And then, like, I think you should probably also have, like, a rough idea of what an internal transfer is going to look like, right? Like, if I move from, like, you know, Team A to Team B, it's not going to look like a new hire, you hope. Although, depending on the team, you know what I mean? Like, I know I'm familiar enough with Acima that there's some teams that you don't just, like, walk into [laughter].

MIKE: One does not simply walk [laughter].

WILL: One does not simply walk into a --

MIKE: An underwriting -- [laughs]

WILL: Yeah, underwriting. Yeah, underwriting.

MATT: Or the [inaudible 31:48] service.

JUSTIN: [laughs]

WILL: Oh no, that one is fine. Like --

MATT: Oh yeah, you worked on that team.

WILL: Yeah, the Haskell. The Haskell, the Haskell folks, you know what I mean? Like e-comm.

MATT: Oh, Lambda.

JUSTIN: Lambda. Yeah.

WILL: That one's there. You know, there's stuff you don't just walk into. So, like, you have that, right? Then you say, okay, reorg. Then I have, like, X number of, like, people moving teams, and probably the same for managers, although, I mean, I will say that, like, I don't think we think enough about management sort of, like, the bandwidth, right? You know, like, we spend a lot of time thinking about, like, okay, coordinating engineers, right? But, like, management bandwidth in terms of planning bandwidth I think it's, you know what I mean, I think it's a little bit unsung. We have a general feeling for, you know, how many engineers a manager can be sort of overseeing, but beyond that, it gets pretty mushy, wouldn't you say? Or am I wrong?

MATT: No, I think about that every day. I'm new to the management team over here on the Upbound side. And it's true that bandwidth gets spread a little thin, right? And that's part of being able to organize your team, having the right people to lead those teams. And everyone has been on...well, not everyone. But a lot of us have been on very large teams versus, like, 3-to-5-member team and seeing the difference in efficiencies, right? And how those teams are managed.

And, ideally, smaller teams, in my opinion, are better. So, you need to do that with management as well and whether there's levels. We have a new concept here at our company of delivery managers. And they've kind of taken over a lot of that responsibility to kind of help spread that bandwidth for planning and let people focus where they should be focused. That's the way think about it.

MIKE: So, we've wandered far and wide here a bit [laughter] as to...we started, like, well, yeah, we know what it takes to run a team well. And then, we started getting into all the complexities when you increase in scale, you know. You go to the next tier, and you've got managers and managers over managers over managers, and it can fall apart. Because, at any place in this hierarchy, if somebody isn't paying attention to the rules, then it can fall apart.

It seems to say something about what you do to run an effective engineering organization is that everybody has to be aware of what the constraints are, be upfront about them, and be willing to stick to some simple rules, right? The planning happens. It happens every time. Every time you can. You don't give it a short shrift, you know; you keep on it. And you don't skip it at any tier.

You know, up at the top level of management, you know, they have to make some estimates based on incomplete information. That's something we all have to do. But they do everything they can to follow the best process they can: gather the information, make the best estimates they can with the information. It has to be all the way down. And then, you have a really well-run organization, right?

WILL: Mm-hmm. Is there [inaudible 35:06] where, like, people who are just sort of guessing, because the company...and what I mean by that it's like, sort of if you say, like, okay, I've got a corporate, let's call it, like, five-year plan, right? Which is not a crazy thing, like, you don't get a business corporate, you know, strategy kind of a thing, right? But on a software, you know, estimation sort of a thing, that's [laughs] a joke. That's comedy, you know what I mean? Like a five-year, like, I can't plan that, you know what I'm saying? And so, it does need to be done on some kind of level, right? Where does that transition happen, and how?

MIKE: There's a really good thing that we've done for a few years now that's spreading throughout the organization, which is that we recognize explicitly that there are tiers of planning. You can map these to a couple of really useful things. You can map them to groups of employees, or you can map them to Jira issue types. And it works either way. So, tier one, you know, planning level one is what the executives do, and it's associated with an initiative in your project management software.

Tier two, planning level two, is associated with milestones within that initiative. I'm going to expand to Columbia, so I've got an initiative. And then, when it comes to milestones, well, maybe you need to negotiate with the government contracts, you know, maybe you need to build some sort of new tax integration. You know, figure out whatever those pieces are. You're developing the milestones, right? But now you have something that's a bigger shape.

And then, there's a tier-three level of planning. Now you're talking with your team leads. Those team leads are breaking this, and that's associated with, like, an epic in your project management software, where now you're coordinating between teams. You know which teams are going to be working on it and where those boundaries are, you know, who's going to be working on what.

And then, you have a tier four, you know, planning level four, which is within the team. That's your refinement, your sprint planning, and so on, where now I know what my team needs to do, right? So that I know what those boundaries are. And I have the epic, so I know what the overall story is I need for my team. Now, I need to break that down into the two and three-pointed stories.

And that works really well because, at every level, you're doing planning based on the information that you have. And you recognize that the accuracy of that planning goes up dramatically as you go from levels one to four. By the time you've done a level four planning and you've completely pointed something out, you know, you can say pretty well when that's going to land.

When you're just at the initiative level, you're like, I'm going to expand into Columbia, right? You really don't know. You really don't know very much, but you know something, right? You know something. You know that that is a business priority that you can shuffle around with other business priorities and say, "Well, this is something I want to do first. And I can start to gather information on it.

WILL: Well, okay, sure. But, I mean, like, so, like, you know, a company like the size of Upbound, right? It's a public traded company. They need to have, like, sort of, like, forward-looking projections about, like, when is this stuff going to land, you know?

MIKE: Yes.

WILL: And what are the business impacts and, like, because, like, because they're beholden to their shareholders. Their shareholders want regularly scheduled updates about, like, okay, you've got an initiative. When am I going to get the initiative? And, I mean, like, is the best we can do on these initiatives just sort of, like, we've done it before and this is about how long it took, you know what I mean? Like, where it's like, okay, you know, I did this, you know, when I expanded into Cincinnati. Like, this is what I got and how. And if I'm doing Columbia, then it's going to be, you know what I mean, ten times harder than doing Cincinnati, right? Like, you got a whole other regulatory language business entity, et cetera. Like, is that the best you could do?

MIKE: Well, I think you can even put time frames on here. So, I'll say, also, that this map roughly to divisions of time. Annual plan, initiative. Quarterly plan; it may be milestones, right?

WILL: Right.

MIKE: And then, you can slot your epics, perhaps, into months. And by the time you get to your stories, you can get that down to about weeks. It kind of maps loosely. Again, this maps to some of these divisions. The precision you get gets more specific. And, generally, at the executive level, you're usually communicating at an annual or a quarterly level. You say, "Well, this year, we intend to expand into Columbia. By quarter three, we expect to be operational at some stores," and so on. You know, you speak in those kinds of broader periods, right? Your resolution is a lot less fine-grained, and that's okay. That's actually expected.

WILL: How do you enforce discipline? I mean, I know, like, on the tier 4 level, right? You're sitting like, okay, I want two or 3-point tickets, right? Like, I want two or 3-point tickets. I want, you know, that's all I want. And, like, if you're doing five or, God help you, an 8 [laughter], you know, things are probably going to go bad, right? And I see how that fundamental process it's got to scale up. But it's a lot less clear how you enforce it, you know, as you go from tier 4 to tier 3 in planning or tier 3 to tier 2 planning or tier 2, God help you, to tier 1 planning. How do you keep a lid on that? Well, because it matters. Like being off by a quarter, that's a big problem.

MIKE: Right, right.

MATT: It is a big problem.

WILL: And for a tier 1, being off by a quarter is actually doing a pretty good job, in my opinion.

MIKE: I agree.

WILL: You're delivering an initiative, and you're only off by a quarter? Not bad, y'all. Not bad.

MIKE: [laughs]

WILL: You know? Like...

MIKE: So, one thing I think you can do...and we talked about the rules down at tier 4, right? That's what we think a lot about. But there's no reason you can't have documented rules at every tier. This is the output. This is the data that went into it. And these are the requirements. Like, you don't say that this initiative is going to get done this year, and unless you have done this, this, and this [laughs], being off by a quarter, that's...it's going to be hard to be within a quarter resolution.

But you can do the things, the executive things [laughs] that executives need to do: gathering information, delegating that out to people that you trust to go and gather the necessary information, and establish a culture that demands honesty and rewards it, rather than one that rewards saying, "Yes." I mean, there's things that can be done there. And now we're getting into business management.

To bring it back to the beginning, we've all got some landmarks we're trying to move our way towards. Hopefully, this has provided some food for thought, how to better figure out when we're going to get there.

WILL: Cool. Thanks, y'all.

MATT: Yes. Thank you, everyone.

The panel delves into the challenges and strategies of onboarding new employees to discuss the importance of communication, mentorship, and understanding context. Mike shares a story from his construction days in New Orleans, where he learned the value of being paired with an experienced team member. This buddy system helped him overcome communication barriers and adapt more quickly to his new environment.

Effective onboarding involves more than just providing documentation. While documentation is valuable, it must be up-to-date and supplemented with hands-on guidance and practical experience. The panel advocates for giving new employees context about their tasks and the organization. This can include having them use the software as end-users or working directly with customers to understand the system and its impact better. Regular check-ins and skills clinics are also recommended to help employees stay updated and improve their skills.

Additionally, the podcast highlights the importance of continuous learning and mentorship for all employees, not just new hires. Regular mentor-mentee interactions and collaborative updates to documentation ensure that onboarding remains practical and relevant. The panelists also note the pitfalls to avoid, such as not following through on commitments and assigning unsuitable onboarding buddies.

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today [laughs]. With me right now, we've got Justin, Dave, and Matt. We're going to talk about onboarding new employees. It's a tricky problem. I don't know that anybody has it perfect, and I don't think we have it perfect. So, we're going to talk about some ideas, some things that we've seen go well and maybe not so well. I've been remembering onboarding in days past.

I tell a story about the late '90s. I was living in New Orleans doing construction work. We would go in and remodel office space. So, sometimes, we'd tear out everything and then rebuild it. You know, we had teams all over. The company I was working for had teams all over the city doing this. And the team I started on, I started just as a laborer. I was not doing anything fancy to begin with.

But what I remember is a couple of things. The day I started, the foreman I was working with set me up with a partner. His name was also Mike [laughs], and they called him Big Mike. And they sometimes would compare him to George Foreman because they looked similar. He was a big guy. He previously worked out on the oil rigs out in the Gulf of Mexico and a big, strong guy. And we got to be fast friends pretty quickly.

However, the day I started, I had not been living in Louisiana for all that long, and he grew up way out in the Bayou somewhere. And when he started talking to me, I picked up about one word out of three of what he was saying [laughs], maybe. And it was that way for a few days. You know, maybe started out with one word out of three. After a while, I got to two out of three. There's a mix of accents there. If you've got a Southern accent and a heavy French accent...

DAVE: Mm-hmm. The Creole.

MIKE: Plus, there's some African-influenced accent as well, and you mix those all together. And I'm a White guy from farther North [laughs]; I did not pick up very quickly. But we got to be friends. And he was patient [laughs]. He was patient with me. And, actually, after a few months, we'd hung out outside of work. We were actually quite close friends. We've lost touch over the years, but I'd love to get back in touch with him. After the storms they had down there, that drove out a lot of the population. He's probably moved, and I don't know where he's gone.

[laughter]

But there are things about this story that were relevant. First of all, the foreman did a great thing. When he brought me in, he paired me with somebody so that I didn't have to know everything. I just had to be able to ask Mike, and [laughs] we could get some stuff done. The second thing is communication is hard [laughs]. We were speaking the same language, and we still couldn't understand each other. He could probably understand me. I was the one who was struggling [laughs].

But, you know, it took us a while to get things right, and we had to work on it. It had to be an active, active effort, especially on my part, to learn to understand. And working on it worked, though. You know, over time, we got to work really well together. We were frequently called on to get a lot of stuff done because we had more effective partnerships [laughs]. We both had enough carpentry skills that we could get a lot done besides just being laborers.

I lay this out, and it's outside of software engineering. I think a lot of times, it helps to think about something outside of software engineering to lay some context. We bring people in, and we let them loose. And you've got to figure stuff out. You may be talking with somebody [laughs] who has a different background, and he's going to use the same words and mean different things, and you're in an unfamiliar project. There's a lot that has to happen for you to onboard successfully.

And there's a few things that I think make a huge difference to be successful and one of them is that regular communication with somebody. And another is, you know, well, getting that communication, but having a buddy, right? Having that lifeline, I think, are some of the key aspects I've seen be effective in onboarding. I think there's a lot else as well, but there's my story to launch this off. And what do you all think?

MATT: Well, you hit on communication, and while I think, you know, trying to dissect a Creole Cajun dialect is tough, [laughter] there's also communication differences in the same language and dialect. For instance, the four of us all have very different communication styles, and I think being able to pick up on that and how people communicate, and their intent is a very important aspect of that.

JUSTIN: I like how they assigned somebody to you who was already part of the thing. They had, you know, knowledge that they've built up as kind of a veteran working there. And they were able to answer your questions, and you were able to be much more effective because you had somebody to ask questions to, and they had to answer right away. You weren't, like, left alone. "Go do your thing [laughs]."

Yeah, I've had experiences with both. And a lot of how you communicate culture to a new team member is, you know, whoever is introducing you, and whoever your onboarding buddy is, whoever you have the most exposure to, that is how they view the whole company almost. And if you don't have somebody, you know, doing that correctly, you don't have a good communication in not just, you know, knowledge, but also culture, and friendships, and there's a whole bunch of stuff there.

DAVE: But, for me, it's, like, context. There's so much out-of-band stuff you can pick up by, like, a personal interaction, where I was talking with Adam a while back, and I'm like, "I want to pair with you on this problem." And he had been thinking through this problem. And he's like, "Well, I know it's not the login. I know it's not this. I know it's not this conditioning thing. I know it's not this sequence in the system." And I'm like, "Adam, I need you to take me with you to hunt badgers." And he's like, "What do you mean?" And I said, "I don't need you to show me how to shoot the badger once you've found it. I need you to take me through the hills and show me every place you check for badgers. That's what I need."

Because he was thinking, I don't want to show you the login system because that's not going to be the problem, and I don't want to show you the pipeline because that's not...I'm like, no, show me that you checked the login, right? Like, literally, the gift in the box was not what I wanted. I wanted the box, and the wrapper, and [laughter]...right? And all the trappings that went with it.

MIKE: That's interesting. You wanted the box.

DAVE: Mm-hmm.

MIKE: Again, this is a little bit outside of engineering, but it's related to that. You know, you think about math class. You can memorize the rules for solving a problem. And as soon as you forget any one of those or you make one little mistake, you're just completely lost. But if you can have somebody who takes the time to help you understand why you're doing what you're doing...and it takes a lot longer. It absolutely takes a lot longer, and in a classroom setting, that's hard. You know, it's a gifted teacher who can bring everybody to that level, and that's, you know, a difficult thing to do.

But if you can do it, if you can help somebody get deep mastery of why those things are there, you know, teach the intuition and understanding, if somebody gets lost, they can backtrack. You know, you can figure out what you're doing or even come up with your own way of doing things and be creative about it. It completely changes your perception of that problem-solving. We're in a problem-solving industry. That same idea is directly applicable. If you could show somebody this is what you do and, you know, tell them to just do that, it's not very useful at all.

MATT: The journey is just as important as the destination.

JUSTIN: I think, to kind of summarize, if we were to put, like, points on a board about, you know, how do we onboard someone, so, right at the start is, like, choose an onboarding buddy who's going to be very helpful for this person in whatever. And I think that's part of leadership's responsibility, you know, whoever is in charge of onboarding this person, whether it's the tech lead, or the director, or whoever the team lead. So, they need to make that decision of who's going to be this person's onboarding buddy. And then recognize that that's going to take time, and that's real work. And you can't, like, you know, expect them to do the same level of work as they did before, perhaps, because they are, you know, if they're going to do it right, it's going to take time.

MATT: And based on the information we have about these new hires, who is going to be the best fit, not only in experience, but personality type, communication type, those things?

MIKE: That investment can go a long way. If you do your onboarding poorly, then you're going to have somebody who doesn't know what they're doing. And somebody can sit around for months and get very little done. They'd be busy, busy all of that time, and not get very much done because they don't know what they're doing.

DAVE: I can't tell if I feel seen or attacked there, Mike.

[laughter]

MATT: But that's important, right? Encouraging people to ask questions and not just sit and spin their gears, ensuring that they understand the things you're going through and leading them. You know, I could sit and write code for hours and have someone paired with me. But if I'm not ensuring that they're understanding what I'm doing, communicating my thought process of why I'm doing it, I don't feel like I'm really helping them. And they're probably not learning a whole lot during that time, right?

MIKE: One thing about our brains is they don't tend to stay connected to something they're not actively engaged with. You show me somebody who can watch somebody do a detailed technical task for a long period of time, and [chuckles] I'll show you something imaginary [laughter]. It's not the way our brains work.

DAVE: There's a really great experiment that demonstrates this, which is that if you just start reciting numbers at somebody and say, "Memorize this. You don't get to write it down. Just memorize it," we tend to be able to hold about seven numbers. Like, some people are down to five. Some people, as many as nine...can hold this in their working memory. And what we miss is that if you can tie these things together with a story...the people that memorize Pi out to, like, a million places, they have a story that ties all the numbers together, and where you are in the numbers, where you are in the story, and, to them, it's just one story.

And that is the essence of, like, "Oh yeah, you know, sign into this. The database is over here. The server is over there." And I'm like, okay, that's three things that aren't connected [laughs], right? And there's 23 more to go. You better start making a story out of this for me [laughs].

MIKE: You know, that says something, right? Without that story. And it also says something about communication. You know, a lot of the best communicators will dip into stories that don't directly connect to the material they're talking about. But then they'll make the connection like, oh, that's great. That was entertaining. And it's entertaining, sure, but it's a lot more than that. Without that story, you would never have learned the idea in the first place.

It may superficially look like they're just trying to make things fun, but that's really not what...making things fun is not what really good communicators do. What really good communicators do is connect you with the material they're talking with. That could be fun, and it's going to be engaging, but the goal is different. They're not trying to force something to be entertaining. Let's make math fun. No, they actually see the fun in it. They see the story, and they want to share it.

DAVE: I had a co-worker...oh gosh, this was around Y2K. This was a while back. And we were struggling with the inheritance rules. I had some co-workers that didn't understand, like, protected versus private versus public versus, you know, friend was the thing. I blurted out the rules of C++ inheritance in terms of intimate encounters, right? Your children cannot touch your privates. Your friends can touch your privates, right?

I got sent to HR, which is fine; I deserved it. But a week later, I'm typing away at two cubicles over. I hear, "Damn it, Dave!" [laughter] And I'm like, "What?" He's like, "I know the difference between protected and private, you jerk!" and just kept typing. And I'm like, yep, that's in your brain rent-free for the rest of your life [laughs]. I've gotten better at little, you know, more tasteful stories since then. You're all going to remember it now, too.

[laughter]

MIKE: So, reinforcing the point about stories, having to have that connection and the importance of context, the importance of context.

JUSTIN: Related to that story there and onboarding new people, the best places I've seen that do onboarding is where you come in, and you're actually onboarded as a user of whatever system that you are going to be doing, and so you know how the users are using it. And you can see, you know, oh, okay, here I'm a customer service rep, and here's me logging in, and here's me helping a customer, looking up the customer name. And then, here's me looking up their lease number, and their payments, and their history, and things like that. And I can, you know, see there. And here's the customer service rep complaining about something.

And so, you go in, and you spend a day or two just immersing yourself in that system. And then, you come back, and you have a story in your head of how this thing actually works from a front end, and then you can dive into, oh, okay, this is how the app works. And this is what is actually happening, and this is where the data is being stored. And I understand what's going on here. And here's this third party that we're calling, and things like that. So, that right there is key, for me, to understanding, you know, see how our users are using it, and then I can see behind the scenes.

DAVE: There's a fundamental problem in computer science, and I call it the A versus B problem, which is where this system over here, you got A matching up against A, and it works. System B matches up against B, and it works. And you get a bug that comes in. Your trouble ticket is, I did this. I expected A, but I got B. And you're like, okay, should you be getting A, or should you be expecting to get B? Which is the right answer, right? It's like, oh, plus four on the zip code. This piece over here only has room for five digits in the printing label. Don't ever put a plus-four on there. That's a bug, right? Anybody else working with postal codes is going to be like, no, you always want the plus four. Why would you want to throw away data?

One of these is right. But if they don't match, you have to know which one of the two is correct, and it's the context. Seeing a user use it is what, you know, it's like, if this is checked or unchecked, if it's checked, we always do the thing. And if it's unchecked, does that mean we can do the thing, or does it mean that we never do the thing, right? There's that context to it.

MIKE: You know, you talked about sitting in the place of customer service and using the software. I'd take that even further. Some of the most successful onboarding I've seen is when the new person is actively assigned to work with a user of the software. It's not always possible. It depends on the context. But you can have them sit with customer support and help them field calls for a couple of days, or, you know, go out in the field and use the software with the users. You know, actually using that software with real users provides a depth of understanding that I think is almost impossible to get otherwise.

DAVE: I can think of a time when I have been sent to work with the end users that...In 2006, I worked on a system for a year, and I finally got to go sit with the users. And this has happened every time I've sat with users. And I was surprised after a year that this still happened to me. I walked back into the bullpen with my co-workers the next day, and I said, "Guys, we're building the wrong thing." Just half a day with the people using it, and you realize we are going around welding shut every door that you need to access. The way that we're trying to get you in and out of the system is like crawling up and down the chimney. We have built this completely wrong. We need to put a lock on this door and make it swing easily instead of messing up your life, yeah.

MIKE: So, we've talked about a few things: having a buddy [laughs], taking the time to get context, assigning somebody to actually use the software, and, if possible, pairing them with real users. Saying that all of those are important for onboarding. How valuable do y'all think documentation is?

JUSTIN: I wouldn't want to dive into documentation to start off with. I'd want to go to those other things first. Because documentation is great. I love documentation, but without the context, I don't know if that documentation is dated. I don't know if it's, you know, even applicable for the system I'm on. And so, without that context or without asking my buddy, it's almost worse than having no documentation [laughs]. You could be looking, like I said, at outdated stuff, or you could be doing stuff like that.

But later on, when I'm more experienced in the system, I love documentation because I have the context there, and I'm just looking for details, or maybe some other stuff when I already, you know, have a good, solid base of everything. So, yes, documentation, but I don't want to push a bunch of documentation at somebody who just started yesterday and say, "Read all this [laughs]," so...

DAVE: If it's a part of the system that is very static, very stable, and we've been using it for five years, then yeah, hand you a piece of documentation that says, "Start here. When you're done, you will have this. Here's the steps." And that's kind of fun sometimes where you hand that to them and say, "We're trying to improve this documentation. Please go through this. And as soon as you get stuck or lost, come find me, and we'll fix the documentation." That's a lot of fun.

And, again, there's a trick to that, which is now they're engaged looking for the problems in the documentation. And now they're paying attention to the system. Instead of "Memorize these seven things and plug them in," it's like, "Okay, I'm in step three. I don't have access to Postgres. What should I do?" right? "Oh, let me add that step."

MIKE: You know, I think we all wished we had perfect documentation. But documentation is a lot of work, both to create initially and to keep up to date. And I'm not complaining about documentation. You know, any documentation is almost necessarily out of date unless you keep it in lockstep with your changes. Unless any changes to documentation go out exactly with any changes to your software, it's going to be out of date kind of the moment it's written.

So, it feels like there has to be another layer, either you put a huge amount of effort into your documentation, which I think is unpalatable for a lot of organizations because the work is hard; it's expensive, or you have some system outside of the documentation to communicate some of that knowledge. And that's hard. That's a tricky balancing act. Because there's probably things in the documentation that somebody is going to forget to tell you.

So, to ask another question, I asked about documentation: how do you avoid those gaps? And we've talked about the value of partnering with somebody. What happens if you never happen to walk by that hole in the ground? You say, "Don't step in that." [laughter] How do you avoid that? What are some things you can do to avoid some of those gaps in knowledge?

JUSTIN: Document. Oh, wait [laughter]. Find it out and document it. I like what you said, though, when you have your mentor there, and you have your new onboarding checklist, you know, to get permissions for all the things and to get all your environment set up. If you're both doing it together and then you update anything that's missing or, you know, common missteps on that onboarding document or series of onboarding documents, usually, that's really helpful, and it kind of updates itself as you onboard new people. And that way, you aren't dealing with an onboarding document that was last updated in 2021.

So, I don't know if that answers your question exactly, but that, you know, goes back to having your mentor and you going through it together and updating anything that you find is missing.

MATT: And things change, right? Something that was relevant even as recent as, let's use our company for an example, five months ago isn't relevant today, right? Because we're doing a lot of things differently. So, if we can make these things a living document and adjust as we go, and maybe we find a better way to do something that we have documented or a different way that's more efficient, then we can update it and keep that document living. And it provides a lot of value for the next person coming through the line.

MIKE: So, we've got another item in our checklist here [laughs]. Have the new person work with their mentor to update the documentation as they go so that it does become that living document as they're using it. One thing...I still think that sometimes you miss those things, the things that you didn't think to document that everybody just knows. One thing that I've seen among people who've been really successful, a lot of times, they'll say, "The time I learned the most was when I didn't have anybody else to help me, and I had to do it on my own."

And [chuckles] maybe they're at a job where they're the only engineer, or they took over DevOps. You know, whatever the case may be, a lot of times it's when [chuckles] somebody had to take care of things themselves, and there's nobody else to go to. You have to figure it out. And, boy, that burns [inaudible 22:22] your brain.

MATT: Trial by fire.

MIKE: But, you know, we don't want to lay everybody off once a month just to give somebody a chance to do that. That's not a good way [laughs] to run a company. So, we're not going to do that. So, I think that setting up some sort of environment that gives people an opportunity to learn and more of a sandbox is really useful. One thing I've seen for security training that was extremely helpful is we rotated weeks. And every week, somebody would add an insecurity to the application from the OWASP Top 10. And then, the rest of the team, knowing that there was that kind of vulnerability in the app, would look for it and try to exploit it.

JUSTIN: In production?

MIKE: No.

MATT: No.

[laughter]

JUSTIN: Okay.

MATT: No.

[laughter]

MIKE: We built a...We built a...

DAVE: So, they're playing. They're just not playing for keeps.

JUSTIN: Yeah. I was like --

MATT: I believe, Justin, this goes back to when you first came aboard. I remember I was doing the OWASP Top 10. Prior to that, the trainings were an application that Mike actually had hosted on his AWS instance called Insecurity. And then, we moved over to the Juice Shop when I took over. But prior to the Juice Shop, it was this Insecurity app that we did exercises on.

JUSTIN: Oh, that's really cool.

MIKE: The thing about that is when you're introducing the vulnerability, you had to understand it well enough to break the code [inaudible 23:54] to it. And sometimes that's harder than it sounds. Like, the frame will fight you [laughs]. And it's hard to get that in there. And so, you really learn to understand the guardrails and how you can break through them. And then, from the flip side, it's fun for other people to try to figure out what it is you've broken. The teams I work with doing that learn...They universally said they learned more about security working through those exercises than they did in years of trainings because that hands-on makes a huge difference.

DAVE: It makes it alive.

MATT: Yeah, I mean, try adding the SQL injection to Ruby on Rails ActiveRecord. It's not easy.

[laughter]

MIKE: It's not. It's possible, but very difficult [laughs] [inaudible 24:45]

MATT: Yeah, you have to try to make that vulnerability.

DAVE: And you can tell when somebody's up against that because they'll start saying oddly specific things like, "How do I serialize a lambda to the database?" [laughter] Yeah, yeah, whatever you need that for, stop it.

[laughter]

MATT: But, you know, something that maybe I want to touch on just for a second, we were talking about the documentation and the living document. Another thing that this does is, you know, the reason we hire people is because we want to hire smart, good people who are going to provide benefit, right? Hopefully, we're hiring people who are smarter than us and better than us at our job.

MIKE: Probably, yeah.

MATT: So, it also provides them the opportunity to give their insight on the things we do. And they may have a different viewpoint on something like patterns or generally things we do that they may have done differently, and we've never really thought about doing in a way that they have. And it opens that conversation, and it gives us an opportunity to not only help them onboard but to improve our systems as well.

MIKE: I can think of a few cases where a capable person came on, and during the onboarding, they're like, "Well, you know, your test suite is running kind of slowly. What if you..." and, you know, fill in the blank. And, like, "No, that's a good idea. Why don't you try that?" And they did. And now everything works better.

DAVE: We brought somebody on a while back. And a couple of months ago, in architecture, they were like, "Yeah, we've mostly got the database in memory now, and everything is, like, so much faster." I'm like, I'm really glad this guy's here.

[laughter]

MATT: Yeah. And they're going to feel really good about their new job, right? Like, they came on immediately adding value. That's really fulfilling, and it's going to give them a really positive outlook on the [inaudible 26:45]

JUSTIN: I'm glad you mentioned that, Matt. One of the things is to give them a small task or something that they can be successful at in a small amount of time and have them do that thing such that they know the system end to end, whether it's, like, fix a small bug, or here is a small, new feature. Those types of things you could probably do. As a tech lead, you could probably do that in an hour or two. But as a new engineer in a new team, you can give this to them and say, "Okay, here it is. Here is our process. And here's how you do everything. Work with your mentor on solving this issue. Here's something that you could be successful at. Go for it."

MATT: Yeah. Quick wins are important, right? Going back to when I started with the company, it was...I came on, and I was told, "Well, we're expecting you to provide lift within two weeks."

JUSTIN: [laughs]

MATT: I said, "Okay." And then COVID hit. [chuckles] And I got sent to work from home with no training, and it was a little rough, you know. But something that did, for me, was make me really work to provide that lift because I had a goal. And I wanted to say, "Hey, you guys need to see that I'm going to provide value, and I'm going to do it quickly." And it gave me that motivation. So, yeah, I totally agree; giving someone tasks early that they can get in and have a win that's a really big morale boost.

DAVE: I worked at a shop where the development process was you shepherded your PR all the way to production, even if you were on another team, or if you were collaborating across the building. If you submitted a PR, you had to...and that meant deploying somebody else's server, which meant that everybody's deploy process had to be pretty well documented and sort of...like, if somebody else could not deploy your service, you had to stop and improve your documentation and your process, which I think was a pretty great thing. But I sat down on day one, and my mentor said, "We need to have you deploying to production by tomorrow night." And I'm like, "Oh."

They walk me through everything on that first day. And he's like, "Okay." And he pointed me at the...we've got a board here at Acima we call the Scooby Snack board, I think, which is basically just tiny, little things that somebody new could just pick up, fix it, and ship it, right? And it was that. They had a board of, like, tiny, little tasks that somebody could do.

The difficulty had nothing to do with changing this one thing on the website. The difficulty was I got to figure out how to deploy this to production and make sure that I haven't taken anything out and make sure that it all still works. And yeah, being able to focus laser tight on I got to write this. I got to get it through review. I got to get, you know, all the things super, like, clarity. And anytime I got spun off into the weeds, I had that story of, I have to deploy. What is stopping me from getting back on track to that? And that's my next obstacle to solve.

MATT: Yeah. Then you have some purpose. And I don't know about you guys, but everyone in this group today has a lot of experience and years behind us, right? But to this day...

DAVE: It's kind of an old guy podcast today, isn't it?

[laughter]

MATT: To this day, when I start somewhere new, there's some nerves involved. No matter how experienced you are, there are some nerves involved. And I think helping people get those wins early alleviates those nerves and gives them some confidence, and then you can really see productivity out of people once they have some confidence.

DAVE: How do we onboard people continuously? What can we do to onboard Matt, or Mike, or Dave on the new stuff that's coming through or the way we're doing stuff next?

MATT: That's a great question. And I could have used a little bit recently [laughter]. For those listening, I have moved my role here at the company. And there are a lot of things that I'm doing now that some of that continuous onboarding really would have helped.

MIKE: Let me answer that question with a question. Would we expect the process to be any different than it would be for a new hire?

DAVE: I think we do because if we bring you on board to a new thing, we don't sit you down and walk you through all the setup of everything and talk you through, like, you know, "Here's the lunchroom, here's..." it's more like, "There's the database; there's the cluster; there's the deploy. Let me know if you need anything." And, generally, if we're seeing you're...and not even senior. A lot of people that have been, you know, done this more than once or twice know what it's like to be abandoned in the middle of a desert and go, "Okay, let's start walking." You know, a lot of times, we just drop you into a target-rich environment and say, "Good luck."

MIKE: You say a lot of times that happens. And just because somebody can doesn't mean that that's necessarily the best way [inaudible 31:37] about continuous improvement, and not that I've thought about this. I'm thinking out loud here based on the question. What if everybody had a dedicated mentor that when they want to give them a, you know if you wanted to add a question, you knew that you had that person who was there to help you and who is going to give you opportunities on a regular basis to go stretch and do something you hadn't done before. Maybe spend some time pairing with you, or maybe you have periodic times when you go, and you work out on the floor, you know, you go and actually...software. All these things we've talked about, I don't see any reason why they would not provide value ten years in.

MATT: I really, really like that idea. And sorry, Mike, you got stuck with me because I just kind of use you for that all the time.

[laughter]

MIKE: Well, good [laughs].

MATT: Yeah, you know, I was lucky enough to end up on your team. And now that I have ended up in management and you are a director, I lean on you all the time. And I greatly appreciate your support in that. But having someone that you can go to and just ask those questions and will walk you through the things that you've never done it's invaluable.

DAVE: I'm kind of in scan mode. I'm like, how best to refine this? Yeah, I'm just thinking about, like, what I'm going to do on Monday, right? It's like, who do I need to be mentoring? Who do I need to mentor me on some things? Yeah.

MIKE: That's exactly what you got me thinking as well. Like, what can I be doing in my role today to make that effective for the people I'm working with? Maybe I can find somebody to latch onto and say, "Hey [chuckles]. Will you be my buddy and help me out here?"

DAVE: Yeah. And kind of related to this, sometimes it's like, okay, you decide this is what we're going to do. And then, you kind of have to trust the process a little bit, right? It's like we used to do...here at Acima, we used to do skills clinic every single day for like an hour. And it was fantastic. And the skills on the team went just through the roof, and then, you know, COVID hit. And business priorities and things got, you know, rougher, and attendance in skills clinic is way, way down.

I was talking with Tad. He's running them right now. I've had lunch with him today, and I was talking with him about that, and I'm like, "You know what? I have not been trusting the skills clinic process, and I have not been going to skills clinic. I need to start going to skills clinic just for the process of taking time to sharpen and being able to share with other people while sharpening." It's like, "Oh, hey, you know, we're doing SQL right now. You can do this with a sub-query. Have you tried doing it with a window function?" And, you know, people around the table going, "You can do that?" "Let me show you something." Yeah.

MATT: Well, and regardless of how many years of experience we have, everybody has something to offer, and everybody sees things from a little bit different perspective. So, I've learned so many things from what we have labeled as, like, juniors because they're thinking about things differently than I do, you know. I've done the same things for so many years that sometimes you get a little bit tunnel-visioned, and you don't look at things from a different viewpoint. And they really help along those lines and open up your eyes really, right?

JUSTIN: Help keep your skills fresh.

MATT: Yeah, absolutely.

DAVE: And you can be with somebody who's very, very junior, and they are probably thinking, there's nothing I can teach this guy. But what they don't know is that you're sat there going, if I don't learn anything from you, it's because I've failed as a student. I'm going to watch you until I learn something. And, invariably, I learn something amazing when I do that.

I worked with a girl who, halfway through the discussion, I found out she was a lawyer. She was a junior programmer. She had passed the bar in Colorado. And I'm like, "Why are you in here and not practicing law?" She's like, "Because I hate law. I don't want to do law. I want to be a programmer." And I'm like, "Okay." And sure enough, that afternoon, we're typing along, and something came by. It was like TCPA or something like that, and I'm like, "A or B?" And she was like, "It's B. It's B. A is against the law. Do B."

[laughter]

DAVE: I'm like, "Oh, okay."

[laughter]

MATT: Yeah. We had a contractor who was also practicing, the attorney, in a different country. But...

DAVE: Yeah, somebody who shows up with that, you know, they can think.

MATT: Yep. Again, everybody has something to offer, and we all have something to learn.

JUSTIN: So, I got a couple of really quick, rapid-fire questions. One that I've noticed was really good was as a, you know, a person leader, the manager is, like, having regular check-ins with your new employee, like, at the start, especially. Those would probably be, you know, weekly, that sort of thing at least. And then, of course, like, a big lunch meeting to introduce everybody at a restaurant, or something like that, celebrate the onboarding, but also, you know, weekly check-ins. And then, what are you guys' thoughts on a 30, 60, 90-day plan, like creating one for your new employees? Is that something that works for engineers, or is that something that is mainly something that exists outside the engineering world?

DAVE: I've only ever seen it done once, and when it was done, it was every bit as awkward as we all thought it would be. And it was profoundly effective, and I have missed it ever since.

[laughter]

JUSTIN: Wow. That did not go in the direction I thought it would go. I was like, [vocalization].

DAVE: Yap. It was with the first time I ever saw a manager who felt that being a manager was a full-time job and required him to study and learn management. And so, he was literally learning things like, when you have your one-on-one meeting, you have to quickly triage the meeting into, are they looking for training? Are they looking to just check in? Or are they melting down, and they need to vent? Because you have to handle those very, very differently. And if you try to go and fix and train when they're venting, it's going to go horribly. I'm like, I never had thought about that. And this particular person would come and say, "Hey, we need to go do our one-on-one."

And it was just...I'm like, this is awkward. This is dorky. "Okay, Carl, fine." And they'd get up...and we'd go to lunch, and it would start awkward. And then, five minutes in, we're just chatting, shooting the breeze. And we're talking architecture and where does the team go. And I grew a lot. And this was 10-12 years ago. It was kind of middle of my career. And my skills leapfrogged every single month.

That was the first place where I put on, as my personal goals, to lose weight or to be able to run a mile under a certain time. And Carl was very, very proud of me when I put that on because he basically...I said, "If I do good cardio, I'm going to think more clearly, and I'm going to be more effective as a programmer." And he's like, "I think that's a fantastic goal. Put it down. Let me know what your time is in 30 days."

MATT: Interesting. I like that.

MIKE: I think we have rightly done so, talked about things that work well. One final question I want to ask is, is there anything that poisons the process? Is there anything that makes it just work terribly? To maybe illustrate the opposite, right? You know, as to what you can do well.

MATT: Yes, not following through. That's key, right? As people get busy, people get sidetracked, things like that, I think it's extremely important to follow through on the things that you promise to deliver, and not only is that relevant for onboarding. I think it's relevant just in business, in life, et cetera. But, you know, let's say I'm onboarding someone, and I say, "Okay, next week, I'm going to spend two days with you going through x," and I get sidetracked. I don't follow through. That's a really big failure on my part and, letting them down and not giving them the room to succeed. So, I --

DAVE: And it's critical. It's critical at that time, too, right?

MATT: Yeah.

DAVE: Because that one failure is 100% of their interactions with you to date and that set the direction that they're going to expect everything from them. You could never miss again, and they'll always wonder, is this the time that he's not going to follow through?

MATT: Yep. You paint that picture.

JUSTIN: One thing I've seen is when they've chosen the wrong person to be the onboarding buddy. And if they choose someone who is bitter about their current position or is not optimistic, it kind of can make that new person follow that persuasion, which is probably not what you want. So, --

DAVE: It's important to distinguish between bitterness and pessimistic, I think, maybe. Because I've worked with a programmer who just he was very much, like, leave me alone, don't, you know, da da da da. And the onboarding was basically...Our boss, I remember on day one, the very first thing she said was, "That's Terrence's office. Stay off his radar. He likes to fire new guys." I'm like, "Whoa, okay!"

JUSTIN: [laughs]

DAVE: And it was a government contractor. You could play that kind of politics and kind of stuff. And that was really, really good advice. And I didn't follow it, and exactly what she told me would happen to me happened to me at the next layoff [laughter]. It didn't make me bitter. It didn't affect my attitude. It wasn't office politics. It was office sociology. It was just, you know, be aware. That's a live wire. If you plug it into the power system, it'll run the whole city, but if you lick it, it's going to ruin your day. And so...

JUSTIN: [laughs]

DAVE: And she wasn't bitter. She loved the company. She loved her job. But she was very, very realistic about, like, that's awesome, and that's garbage and, you know --

MATT: Pragmatist.

DAVE: Pragmatic. Pragmatism. Yeah, that's exactly it. If it works, it's true. That's pragmatism. If it works, it's true.

[laughter]

MATT: But yeah, we definitely want to avoid people poisoning the well, right? Because it does, it carries over, and it kind of just...it spreads as well.

MIKE: We've talked before about psychological safety. You're bringing somebody in. It's not just code. And this is a critical thing. Software engineering, even somebody who's just starting and they're going to be writing a lot of code, they likely will not spend most of their day writing code, at least not every day. Software engineering is a lot of things that aren't code. It is a group endeavor. And you cannot build large projects effectively in isolation. It doesn't work that way. So, you're not just getting somebody to understand a codebase. You're helping somebody understand and build a culture. And you can't forget that it's more than one thing you're building.

DAVE: And I think I've mentioned this before: my favorite epiphany that I took away from skills clinic is that we think of development as a process where the developer is in the box, and we take a problem, and we put a problem in the box, and the solution comes out of the box. And that's not a programming career at all. The box is where the effort takes place, and we take a programmer at a problem and we put them in the box. And what you get out is a solution and a smarter programmer.

And if you focus on investing in that programmer, they're going to grow, and they're going to get smarter. And if you focus only on just product, product, product, product, product, some people will thrive, and some people they just miss...and it's just bad luck. It's not that they're not, you know, good or hardworking. It's that they got so focused on optimizing the process that they end up...it's like not stopping to sharpen the saw, right? You get so busy creating the output that you never increase your ability to create output.

MATT: I wanted to make a Beyond Thunderdome reference there, Dave. Two men enter, one man leaves.

[laughter]

DAVE: Right. That's right. Two problems enter, one solution leaves. I like it. I like it.

MIKE: To come back to a story, several years ago, I was riding bikes with my kids. And my oldest son was pulling somebody [laughs], one of my other kids. And he was back behind. And he was really having a hard time keeping up, which is unusual for him because of who he is [laughs]; I'll say that. He usually does not have a hard time keeping up. I usually have a hard time keeping up with him.

So [laughs], he was just dragging, and he said, "Man, maybe I'm tired today." But he was just trying to muscle through it, and muscle through it, and muscle through it, and muscle through it." Finally, he said, "Can we take a break?" And he looked, and the bike he was pulling, you know, the trailer he was pulling, the tire was completely flat. So, he'd been, like, dragging a plow [laughs] through and was riding [inaudible 44:31] for miles [laughs].

DAVE: Just pulling [inaudible 44:33] the whole time.

MIKE: Yeah. If you don't take that time to look and make sure your tools are in good order and keep things going, you maybe can muscle through it for a while, but it's not going to work the long term.

DAVE: Tying it back to earlier, that's all about the hunting badgers, right? It's like the great thing that can come out of that is not, oh, I needed to fix the tire. The takeaway is sometimes I should stop and check my equipment. And that's the [laughter] aha moment of it. And that's the difference between having a good job and making a whole career, right? It's like that. You'll come back to that.

MATT: Instead of fixing the flat, prevent the flat.

DAVE: Or just be aware that --

MIKE: Be aware.

DAVE: You know, check your equipment, yeah.

MIKE: Sometimes flats happen [laughs], and you got to be aware of that.

JUSTIN: I think from a security point, the guy that thwarted the recent attack that was going to be on one of the base systems in Linux was checking his equipment because, all of a sudden, part of it was, like, going slower, and it was just a fraction of a second slower or something like that. But he went in and, like, discovered the bad code because he was checking his equipment.

DAVE: I've never believed so strongly in the a billion eyes makes all bugs shallow [laughter] from Linus Torvalds from the '90s. I'm like, okay, I believe you now, Linus, so...

JUSTIN: [laughs]

MIKE: Any other final words?

DAVE: Be good to each other. Do code reviews.

MATT: Yeah, right. [laughter] Our interactions are always a pleasure.

JUSTIN: Gentlemen, it has been a pleasure. Yeah, thank you.

MATT: Yes.

On this episodes, the panelists dive into common myths surrounding software development. Kicking off with a reference to Fred Brooks' "The Mythical Man-Month," the conversation challenges the notion that adding more developers to a lagging project will speed up its completion. The discussion quickly expands to explore contexts where adding skilled, senior developers might indeed be beneficial, drawing analogies with emergency services during crises where more hands on deck can mean saving more lives.

As the episode progresses, the group delves deeper into the complexities of software development projects compared to other types of engineering and construction tasks. They discuss the unique challenges of software development, such as the dependencies and the need for sequential progress in certain areas, which can create bottlenecks regardless of the number of developers. This leads to a broader conversation about the necessity of strategic planning, the value of experience, and the pitfalls of assuming that increased manpower translates directly to increased productivity.

The dialogue culminates in a reflective discussion on the philosophical and practical aspects of software development beyond mere coding. The group touches upon the ongoing nature of software projects, the critical role of stress management, and the adaptability required in the tech industry. They highlight that, unlike more tangible engineering projects, software development is often about continual iteration and improvement rather than reaching a definitive endpoint. This conversation underscores the blend of technical skill, strategic thinking, and emotional resilience necessary to navigate the evolving landscapes of modern software development.

Transcript:

TAD: Hello. Welcome to another episode of the Acima Development Podcast. I'm Tad, and I'm going to be hosting this episode today. I'm joined by Will, Justin, Mike, and Ramses. The topic we're going to be discussing today is Common Myths About Software Development. And when I heard this, I thought of one of the classic myths.

There's actually a book called The Mythical Man-Month by Fred Brooks, and he talks about how adding extra developers to a late project isn't going to make that project go any faster. Or one of the interesting examples he gives, I think it's from that book, too, is you can't have nine women produce a baby in a month. You know, human development process, nine months, there are no shortcuts. Have you ever worked on a project where they thought, oh, we'll add more developers to this project, and it's actually worked out?

MIKE: There's a very specific context where I've seen adding developers to a project actually helps, and that's when they are developers who are already very experienced in the industry and deeply familiar with the code at play.

WILL: Yeah, we write it small, right? Like, your team lead or your engineering manager, you know, your senior engineers they kind of fill that role, right? And they sort of, like, parachute in firefighter. If things aren't going well, you need to deploy special forces to, like, help get this thing over the hump. Yeah, I think that's kind of standard, right? And, like, you know, you could deploy as many senior resources as you have available and familiar, you know, which is usually not as many as you'd like.

MIKE: You had one phrase in there: as many as you have. Those three words are doing a lot of work [laughs]. Because if you've got an army, you have very few, like, Navy SEALs, right [laughs]? The elite forces are a small subset of your total workforce.

JUSTIN: It's also worth noting that it's very expensive, both in time and treasure.

MIKE: Ah, yes.

JUSTIN: But going back to your original point, Tad, I think it's like, it is something that is not the norm. Like, you just can't hire contractors to come and solve your problem.

TAD: It's a little counterintuitive, though, because in a lot of cases, adding more resources to something does increase the productivity, right?

JUSTIN: I don't know. Like, can you cite some examples?

WILL: Well, I mean, if I had an emergency room, right? I had an emergency room, holy cow, I'm so backed up. There was a train crash, and I have a lot of, like, patients coming in. I could bring in doctors, you know, I could bring them in or on call. And if I had to the degree that the facility could accommodate them with operating rooms and stuff like that, you know, I could bring in ER doctors, you know.

TAD: More nurses, right?

WILL: Fairly --

TAD: And you should probably increase the throughput in your ER if you did that.

WILL: Yeah, fairly linearly. And you could treat more people, right? And that's a high-skill profession, right?

MIKE: Exactly. Exception proves the rule because you're talking about one of the most notoriously highly paid professions out there [laughs] because it does take that deep expertise.

JUSTIN: So, another example that I'm familiar with is I worked in construction for a while, and, you know, this was back when I was in college and such. And you're working on a big project, and you're building a large building. Like, in this specific example, we were building a library, like a college library. And you can paralyze some things, but other things they have to be done before you can work on other things. And so, there's these natural choke points that you literally can't throw more people at or resources at because, like, the concrete has to set, or the dirt has to be, you know, prepared, and packed, and everything.

WILL: Exactly, yeah, yeah.

JUSTIN: And so, it's the same sort of thing when you're developing software. You have to decide on architecture. You have to decide on, you know, the third-party APIs you're calling. You got these things that you have to do, and you all have to follow the same pattern, hopefully, as you do the various things that you need the new thing to do. And if you have everybody doing different things, you might finish it, and it may be functional, but it's not going to be fun to maintain.

WILL: Yeah, and, I mean, I think it goes back to like, sort of, like, running projects. I mean, I think there's really deep parallels between, like, sort of, like, you know, sort of sequential and parallel programming. It's like, you know, multi-cores, right? We've all got eight cores on our desktop. If you want to keep those cores hot, all of them hot, like, that is ferociously difficult programming, right? So, like, to create, I take a process and parallelize it. That's a fantastically difficult programming challenge.

And I think analogous to that is okay; I've got eight people on my team. We all have to get this thing done. You know, how do I keep all these keyboards hot at the same time to, like, not choke point out, right? Because it's exactly as you say, there are natural choke points. And, like, you know, do you set up scaffolding? Do you set up, like, a fake API server so people can do the front end while the back end is still cooking? You know what I mean? All of these things. And that's a very difficult engineering challenge to, like, maximize the productivity of a squad with these natural in-built choke points. It's tough to do, really tough.

MIKE: Well, and that's a really good...thinking about getting parallelized work on your processor, some of that algorithm is to guess, right? To make a best decision and have it start doing some work that you might have to throw away. To get that parallel, that level of parallel work, you're probably doing a lot of throwaway work.

That is part of the price of trying to get that parallelism is that sometimes you have to go down, you know, rabbit holes that you just have to completely drop. And, you know, that may be true even when it's serial load, but even more so when it's parallel. If you don't know which way you're going to go, you have to do both. And, again, there's cost that comes from that. And if you want to massively parallelize, you're probably going to be doing a lot of throwaway work.

WILL: Absolutely, absolutely. I mean, it's a fact, an absolute fact that parallelizing, multi-threading a process is always less efficient, you know what I mean? If for no other reason than the job-switching infrastructure, you know what I mean? If you're real dorky, you could spend a lot of time on that. But, like, you got to do stuff every time that thread switches context. Every single time. You know, say nothing of deadlocks and all that stuff, so, you know, yeah. Absolutely.

But I do actually think developers are too afraid of scaffolding that's intended. Going back to construction again, right? You're building infrastructure with the explicit end goal of tearing it all out. You know, it gives somebody a platform to work on while the stairs don't exist. I think embracing that with both hands winds up being a pretty big net win, in my opinion.

MIKE: That's an interesting observation there. You say you deliberately build something you know you're going to throw out because it expedites the other work getting done.

TAD: Because it's pretty disheartening to throw code away. As a developer, you take an emotional hit when you've been working on something for two weeks, and they say, "Ah, we're going to go in a different direction." You're like, "Oh, sad [laughs]. Well, into the trash goes all my efforts."

WILL: Yeah, absolutely.

EDDY: And it hurts more when you have it tested. Like, when [laughter] you have all your tests [inaudible 07:55] out, and it turns out you don't need it anymore, and it's like, ah.

MIKE: There's a mindset thing here, though, of what value we're trying to provide to the business. We're not trying to provide code. We're trying to solve problems. And [laughs] if we approach it from that perspective, we say, "Yeah, this was a step in working towards solving that problem." You know, I think we have to be kind of relentless at ourselves that way. The code is not the end product here. We're talking about myths. Our job is to code? Boy, that is not our job.

TAD: Well, it's a component of our job, right?

MIKE: It is.

TAD: Well, maybe we should flesh that out a little bit. What are the other aspects of software development other than code?

MATT: Number one, communication.

TAD: Right, because you're talking about what you're going to build before you ever write a single line of code, usually.

EDDY: Stress management.

MIKE: Think about big construction like a bridge. You would not consider hiring an engineering firm that wouldn't take a year planning before they ever put a shovel into the ground, right [laughs]? Because it's recognized that for big projects, planning is a big part of the work. You've got to coordinate. You've got to figure out what you're doing. You've got to figure, you know, you've got to do surveying [laughs] to figure out what's going on.

You may not have that if you're doing small-scale construction, but absolutely for the big stuff. And even for small scale, I mean, you're going to have a blueprint, hopefully [laughs]. Even if you're building a shed, you probably have a drawing, right? That is work. There is a great deal of work that goes into that construction project that is not the hands-on laying down the materials.

MATT: Then it becomes like putting Legos together, right? If you plan everything well, you know the pieces that need to be put together, and then you just need to put them together.

WILL: Yeah, but, I mean, that's the standard. That's the standard. That's your waterfall model, right? That's waterfall development, which, I don't know, I mean, like, I'm pretty partisan, you know, in terms of I like smaller pieces of work, even if the overall strategy is like, I want to get this big thing done, you know what I mean? Like, I like small pieces of work because I just, you know, and we touched on it, I think, the last session.

Like, I am deeply...I'm not skeptical or cautious of overestimating how smart I am and how good my strategy is and stuff like that. With construction, if I'm building a dam, that's where it is, you know? Like, there's a lot of stuff where it's like, you can't go back, or it's prohibitively difficult to go back. But with software, you do have great flexibility for most projects.

MATT: Yeah. You've called out a good point. I think remaining agile is super important in today's software development world, right? It's necessary. Preloading the planning, yeah, sounds very waterfall-ish. If you can do that, maybe it's just the time before it hits engineering that happens, you know, and by the time it gets to engineering, then it's broken down to smaller problems. And you break it down into simple stories, and you can work from there. That's, I mean, to me, that's agile programming.

TAD: Is there a myth, though, that programming is predictable? Because what's that quote? You know, everyone has a plan until they get punched in the mouth [laughter]. I've never worked on a software project that has gone exactly according to plan.

JUSTIN: If it's a very simple well-known use case, you can do it very quickly. So, like, you know, I want to set up a blog, you know, have comments and everything, and that's, like, every single tech demo ever. So, you got every single language that you could do it in, in datastore and such. I was thinking of a really good example of something that is well-defined, and that can be put together in the real world.

And there was, like, a news story about the U.S. Navy putting together a port in Gaza to unload supplies for the refugees or the people who live there. And they basically could put together this pre-assembled port that the Navy uses on any combat situation, and they could put it together very rapidly and people who know exactly what to do. And all of a sudden, you have this port where huge ships can unload supplies on.

And it's just an example of, hey, if you have a well-defined problem and you have the tools to solve it, you can do very big things very fast. But if you go off that path, then you start having to, like, call in the experts and decide, okay, now that we're going off this path, what should I do in such a way that I'm building something that will last and that is durable? So...

WILL: Right. Well, I mean, and that's the nature of software sort of consuming itself in that, like, when you have that sort of, like, pre-built port roadmap, that's a software product, you know what I mean? Like, when it comes to software, like, when you have things defined to that degree, then it's just, like, push a button, you know what I mean? Like Shopify, right? If I want an e-commerce store, Shopify can get me up in an hour, you know, and they do a pretty good job at it. And so, it becomes, well, what else do you want to do? And fortunately enough for us, they always want more, just a little more.

[laughter]

MATT: And if you want to integrate with Shopify, that's another story.

WILL: Sure. Absolutely. But, like, if you wanted to integrate with Shopify, well, that's something, you know, new that Shopify doesn't do. And then, we're into custom software land, and I'm making my [inaudible 13:37] payment this month.

TAD: That does touch on one of the myths I was thinking about earlier today that a lot of times we parallel software development with things like building a dam, or construction, or things like that, and those things get finished, right? They complete the project, and then they walk away and move on to another project. And software, just because it got released, doesn't mean that you're done.

WILL: Oh yeah.

MATT: Never done.

WILL: Yeah, yeah. I think that's a myth.

EDDY: But when you --

WILL: So, the software is released, right? It's done [laughter]. Like, it's –-

MATT: [crosstalk 14:16] that you're finished, right?

EDDY: Sure, but you got to have people to go back and maintain whatever it is you built, whether it's software, whether it's a building, a car; it doesn't matter. There is a level of maintenance that still is required.

TAD: Even saying built makes it sound completed or finished.

MIKE: Yeah, and to take that further, some software products that we use were created for some utterly different purpose. Didn't Slack come out of, like, an internal messaging video game or something?

WILL: Some, like, in-game chat or something. Or was it just inside the company?

MIKE: I think it was just inside the company. So, they went to build a video game, and they ended up building a business messaging platform. Like, when we say build and release, like, yeah, I went to build a video game, and I built business software. And if you were going to build a bridge, you say, "I went to build a bridge, but I actually built a skyscraper with an airport on top." It doesn't work that way, but we have to do that all the time.

Think about that planning that was brought up before. Yeah, that planning, that cost still has to happen. But recognizing that we have to do these iterations, we still do the planning, but we do it on a smaller loop. We don't spend two years doing it upfront because we know that work is going to be thrown away, right? But we still have to do the work. It's just done in a smaller loop. So, you're spreading it out over the two years rather than doing it upfront.

You still have to have that feedback, and there's still the work involved. There's still the job of communicating, figuring out where we are now and where we go next, and that never goes away. Whether it's upfront or whether it's later, it's still there. And if you try to pretend it's not there, you'll get in trouble quick, I don't know.

And that brings me to another myth. I pointed...it's xkcd number 2030, two zero, three zero. I think about this all the time. It's about voting software, which is kind of irrelevant. There's a line from it where the character says, "I don't know quite how to put this, but our entire field is bad at what we do. And if you rely on us [laughter], everyone will die." And if there's one myth I see about software, you expect that people just know what they're doing. No, we don't [chuckles]. We don't know what we're doing. We don't even know what we're building.

We're building a video game that's going to end up business software, right? And that's not even necessarily a bad thing. We're building something out, and then we'll have something of value and see if we can use it. You know, it's this weird product that's kind of, like, other things and kind of unlike, I say, other things in that it's flexible, and sometimes it meets different needs. And it means that it's kind of hard to talk about, kind of hard to deal with. And we make a lot of foolish mistakes.

I've seen this with new developers, particularly people who change career later in life. They come in and they think that we all know exactly what we're doing, and it takes a while to be disillusioned. You got to kind of break that early and say, "It may look like we know what we're doing, but a lot of our job, maybe not most, but a good portion of our job is trying to figure out what we're doing. We're researching it as we go. We don't know how to use the tools. We don't even know what tools we're using, and we don't understand the requirements. So, a big portion of our job is figuring all that stuff out so that we can get something that works out in the end."

TAD: And a lot of times, you can't know what is needed, and what you need to do, and what the requirements are until you actually jump in and start poking around and doing the coding, doing the research, doing the things, getting both hands into the project. And you're like, oh, now I'm starting to see what shape this needs to take.

WILL: I mean, that ties into something that Eddy mentioned in passing, and it's a myth that I think...let me see how to put it. So, what Eddy was saying was, like, stress management is, like, most of what we do. And I think the myth of...and, I mean, I agree with that, and I think I'd maybe, like, broaden it out to say, like, I think people think of software development as this intellectual exercise. And there are sort of fairly specialized slivers of it that can be that. But by and large, you have to be smart enough and willing to sort of, like, bulldoze your way through thinking about the problems, thinking about the requirements, thinking about the legacy source code and how it works, thinking about, like, you know, how you do the stuff.

I mean, if you even look at, like, sort of, like, these programming puzzles that some people like for interviews, right? LeetCode medium, right? It's far in excess of anything that we actually have to do on an intellectual problem-solving level. What we need to do is maintain, like, a sort of, like, emotional equilibrium and focus on this fairly mundane, ditch digging, systems integration, team collaboration, requirement, negotiation, source of problems. Like, that's what will make you...

Like, if you're, you know, slightly above average intelligence but you have exceptional drive and emotional equilibrium, you know what I mean? Then you're going to be a very successful programmer. But, like, if you have exceptional intelligence and, like, you're a little bit squirrely in terms of, like, just grinding through, like, the necessarily frustrating tedium of, like, professional software development within a business organization, you're going to have a very difficult time, you know what I mean?

TAD: Yeah, and I think there's a myth that all software is kind of the same and that the skills needed are kind of the same. For example, I hear a lot of people ask, "Oh, are you really good at math? You really need to be good at math for software development". And then, I have to say something like, "Well, yes, if you're writing software for a nuclear collider, then you're going to need a lot of math. But if you're a front-end developer wanting to make some really cool interactive thing, maybe you need to know a little bit more design, and you don't really need that much math." Or the example earlier, set up a blog. I don't think you need any math to set up a blog probably, right? But those are all software development.

MATT: Yeah. And maybe this is just me, but I think the people who are just naturally good at math make good software engineers, just that mindset and the way they do things. They may not make the best front-end people, but there's a place for them, right? I think a myth also could be that, hey, I should just switch over to software engineering because the pay is great. I've seen a lot of people make that mistake. It's not for everybody.

WILL: They're having their day right now. I think there's a lot of people who came in for the bag, you know what I mean? And now we're in tech winter and, like, those doctor money FAANG jobs are a little short on the ground. And so, it's just like, "Would you only do this for a paltry $120,000 a year like a peasant?"

MATT: [laughs].

WILL: Like, "Whoa, whoa, whoa, whoa. I'm going to go back to finance," or whatever, right? And those kind of people. And then, you know, you have to wake up every day and want to do it, you know what I mean? I mean, just because, like...I'm pretty good at programming, but every project, you know, like, I got to wrestle the pig every time. It's a brand-new pig, and you're going to have to get in the mud every time. Every time. It doesn't get easier. You just get, I don't know, stronger, you know?

I was having a conversation with somebody who was talking about like, you know, like, needing to hire somebody and the difficulties associated with hiring people and how you'll have these people who, they'll go in, and they'll crush their interview. Like, they'll do really good on these algorithmic problems, you know. And then, you'll take them into the actual workplace where they have to go out and, like, you know, do the actual nitty-gritty nuts and bolts jobs. And they'll struggle or, like, they'll do good for a while. And then, sort of like the morale will fall off, and they'll struggle. And, you know, and, like, you'll have smart people who just can't maintain the focus of, like, staying after it, you know. And then, they will be bad because it's a nonstop grind and, like, yeah, it's tough.

TAD: I've heard it compared to a treadmill.

MATT: That's one of the reasons, Will --

TAD: If you enjoy the exercise, then a treadmill is not bad. But if you're constantly learning and you're constantly working on things, maybe you're frustrated some of the days, right? Not everyone enjoys exercising on a treadmill.

MATT: That's one of the reasons I don't ask the mapping of matrices and things like that in interviews. I just don't. I don't ask really computer sciencey questions because that isn't the real world most of the time.

EDDY: Okay, but the real question is if someone came up to you from your family and said, "Can you fix my computer?" Can you? Can you fix that computer?

MATT: Ugh. I hate that. That is the worst question ever, Eddy.

EDDY: [laughs]

TAD: Yes, that's why I have a master's degree.

EDDY: [laughs]

MATT: Sometimes. I mean, you know, sure, we know that stuff, but it's not what we do.

TAD: Yeah. I mean, I can Google as well as the next person.

WILL: I can Google much, much, much better than the next person.

TAD: Actually, that's true.

MATT: Which is what we spend a lot of our time doing, right?

MIKE: That's one of the most critical development skills is Googling.

TAD: Finding the right information, right?

MIKE: If you think that managing hardware and setting up something with, like, the TV is what we're good at, then you should see any engineering meeting trying to figure out the presentation, like, the hardware in the meeting room [laughs], just trying to get a meeting going. It never works.

MATT: Let's not even get started on spreadsheets.

WILL: I'm terrible at spreadsheets. I don't know how to do them. Like, as soon as...whenever I've had a spreadsheet, like, challenge, like, I Google it. I learn everything I need to know for the next, like, 15 minutes of work, and then immediately cash dump it. I'm just like, pfft, flush. That's out. I'm out.

MATT: I do the same thing.

WILL: Pivot table, whatever. I know the word pivot table is usually the answer to my problem.

TAD: [laughs]

WILL: And then, if you put a gun to my head and ask me to write up a pivot table in Excel, you better have a bucket and a mop.

MIKE: You know, you said something about we're focusing on the, you know, the grind and the treadmill. I read a quote from a bikepacking racer. I don't remember which guy it was, but I read it recently. And he said that that kind of racing, like endurance racing, and it could also apply to marathons or ultra runners, that kind of thing, he said that it's mostly about fatigue management. And you think, oh, you know, it's about building strength. And he's like, "No, it's, like, about eating the right food and managing your sleep and not pushing yourself too hard."

None of those are, like, things that you might necessarily think about have to do with racing. Like, eating? Racing? What? No, it's about managing your body and your equipment so that it can keep going. And that sustainability, you know, trying to think about that metaphor, that sustainability really is, like, a central aspect of our craft. And you can get off, right? You can get unsustainable, and that puts you in a bad spot.

MATT: You end up in burnout. Be good at compartmentalizing, right? I know you and I have talked about that before, Mike.

WILL: It's legitimately tough to do. I think there's a lot of places that are in...I will call them...I mean, honestly, I'd say, like, go at doom loop, you know what I mean? And I was talking about this with Mike a little bit, like, before we got on. It's one of the things that I've been seeing because, like, you know, we're in tech winter now, right? So, things are rough. You know, layoffs are happening, you know what I mean? Like, teams are squeezed pretty tight. I think it's across the industry. Even if you made it, even if you're still working, like, you still have, like, the reaper's hand on your shoulder being like, "How are the tickets going?"

[laughter]

And so, you have these situations where, like, you know, you have a business downturn. You have business pressure. And then, they'll have layoffs, you know what I mean? So, like, that impacts morale. You lose head count. Maybe you don't have nine women to make the baby in a month. Maybe you only have eight. And then, that impacts productivity, and then...you know what I mean?

And a lot of it really is sort of, like, it impacts people's emotional equilibrium, their feelings about their work, you know what I mean? Like, I think it's very understated, you know because it all exists in your brain. And without that focus, that willingness to focus, that willingness to, like, really concentrate, things become extremely tenuous. And you get into the burnout mode, you know.

But you can't stop peddling because, you know, you got business pressure. And, like, you got to make your deliverables. You got to make...you have to make money so that the business can give it to you. And you find yourself with these doom loops, these ugly, ugly, ugly spirals. And I see a lot of that in a lot of places when I talk to folks in the business right now.

MIKE: Which comes back to the same thing, that tenacity [laughs] and fatigue management is going to be the thing that gets you through that winter. My grandfather lived through World War II in Norway. And his family survived through the winter on a barrel of pickled herring [laughter]. [inaudible 27:54] through the winter on a barrel of pickled herring [laughs] is how you get through the winter. Recognizing that it may not always be very nice. There probably will be spring on the other end.

WILL: Oh yeah, and it always will be. Nobody's throwing their computers at the dumpster. I don't see it. It's like people would just be like, "Ah, the internet. We're over it. We're done [laughter]," right?

MATT: At least in my experience, there's always a light at the end of the tunnel. You're going to experience burnout. It happens to everyone. Like Mike's grandfather, you just power through it, and you do what you have to do to survive. And, eventually, you know, this can be the greatest job in the world. It can also be the hardest job in the world. And you're going to experience both of those.

WILL: It depends on the day [laughs]. It depends on the day. Sometimes, it depends on the hour. I mean, I will say that, like, that grind...I don't remember where I heard this, but, like, tenure of a software developer is they usually make it, on average, to about 36. That's not a lot better than playing in the NBA.

[laughter]

MIKE: Well, that's interesting.

MATT: That is very interesting.

WILL: And I believe that number could be out of date, at the very least. I remember it very clearly that I definitely read that, and it could be, you know, maybe a shaky study. But, I mean, I remember the number because it struck me so viscerally. I'm like, oh my God. Like, if you look at somebody like, I don't know, like me, I'm 44, right? If I was a physician, an MD, I'd be, like, just entering into my best years, entering it. It's like, well, you know, he's still a little green, but he's starting to get his feet under him. You know, I think he's ready to go.

But, like, as a software developer, it's just like, you know, people looking at you like Old Yeller, you know what I mean? [laughter] Where it's like, you're walking into the office with a limp, you know what I mean? Like, let's take you back behind the shed.

MATT: Well, just with this group here, we're doing pretty good with those odds, right?

MIKE: Yeah [laughs]. I'm thinking people I've known who have gotten past that or maybe the people who did have the ability to keep going. I've worked with some experienced people. I do work with some experienced people that I find their experience to be indispensable. I don't know how they could do their job, or I could do my job without being able to rely on those years of experience that they wouldn't have had at 36.

WILL: I've found that older developers tend to be pretty bimodal, you know what I mean? Like, there's not a lot of, like, developers, you know, like, north of 40 that are in the middle. Like, it's not really a bell curve. It's kind of like you're either --

MIKE: [laughs]

WILL: Awful or you're fantastic. I'm trying really hard to head towards the light because I have met some awful ones, some truly, truly bad ones where it's just like, you gave up several years ago.

MIKE: So, that can happen when you step off the treadmill for a while [laughs].

TAD: It's something that you got to keep working on all the time because technology keeps advancing, and you got to keep up with the advancements, right? Not necessarily all of it everywhere, but at least the stuff that's relevant to what you're working on.

MATT: We're all going to be prompt engineers before you know it.

WILL: All ready. All day.

MIKE: [laughs]

WILL: I think that's where things are going. I think that AI tools I think they are going to be a fantastic force multiplier, but they're not taking anybody's...well, I wanted to say they're not going to take anybody's jobs, right? But, like, if they double your productivity, you know, the head count's going to go down by 20% probably.

MIKE: There was an agricultural revolution back, what, in the 1800s? And that doesn't mean we don't have farmers. We still have farmers. But they generally operate at a much larger scale, and they're a much smaller percentage of the population. And interestingly, new fields of labor opened up because everybody wasn't farming anymore. I don't think our jobs are going away. Like, there's still people developing software.

And it took a while, you know, but agriculture still exists. It's an important industry. And, you know, there are farmers. Farmers exist, but the way they operate is different. And there's new industries that didn't exist, like I was saying. You know, they've enabled different ways of working. And so, a lot of people who would have been farmers are now doing stuff that's adjacent to that or maybe totally different to that because it's enabled by the farmers being more effective.

I think we'll have similar kinds of transitions that are a little hard to think about. You couldn't have explained a software developer to a farmer back in 1830. Conceptually, that work didn't really exist, and we couldn't even conceive of it. But I think that we'll have that similar sort of transition where, yeah, there'll still be people developing software, but it'll be a different format. And the jobs people have it might not be prompt engineer. It's going to be something that is going to be kind of incomprehensible, I imagine, to us right now. But it would make sense if you saw the evolution over time.

TAD: But I think it's kind of a different beast, right? Because I don't know of any organization that's like, "Oh, you guys have doubled your productivity? Oh, well, we're going to keep the same number of feature requirements that we had," right? They're like, "Oh, you've doubled your productivity? Sweet. We can, like, have twice as many features as we wanted before," right?

Well, I haven't experienced it yet where they've run out of ideas or new software projects or things like that, where they're just like, "Oh, well, we're kind of running low on places to spread our software or new places..." you know, it's like...because I imagine, like, even if we suddenly, like, 10x our productivity overnight, they'd be like, "Oh, sweet, we're going to move into, like, more countries than we were planning," or "We're going to have more features per year than we were planning."

WILL: And I'd also say, like, even more so that because of the nonlinear increase in complexity as software scales, right?

TAD: Yeah.

WILL: Let's say everybody doubles developer productivity, which is, like, a pretty bold statement, but, like, attainable, attainable. But that's probably the top edge of attainable, in my view, with, like, the AI, like, stuff. That's, like, 20% more features because, like, it isn't linear, you know what I mean? 20% more features. Because if you have 20% more stuff, like, I mean, so, within Acima, I mean, think about, like, it would be maybe one other big initiative, you know what I mean? You know what I mean? Like, you'd add one more team feature thing, right? I don't know how many teams you guys have now.

MIKE: I think you're onto something really important there. We talked about the planning and the non-coding aspects of the job. The AI stuff really helps mostly with the coding aspect, right? Does it help with getting requirements from stakeholders? Probably not. If 50% of the job is coding and you completely eliminate that [inaudible 34:59]

WILL: And the style, right?

MIKE: 100% productivity will increase in coding, right? Like, it all goes away. You only get to work on twice as many things --

WILL: Absolutely, absolutely. So, you know, honestly, like, I've been really having a tremendous amount of fun with it just because, like, generally speaking, I know what I want done, and I can express it in a reasonably concise, understandable way. I just might not know, like, all its syntax, you know. And so, like, that has just been, like, cleared from my past. It is an absolute delight, as far as I'm concerned, because a lot of the, like, I just [inaudible 35:37] BS ticki tack jobs are just, like, I have an assistant, you know, with a broom, you know what I mean? Like, I could build a house, and I have a helper handing me boards and, like, doing the sweep up for me. I'm like, ah, this is...yes, all day. I love it.

TAD: Yeah. I think that's a great place to kind of stop. So, thank you, everybody, for participating. Thank you for all of your comments.