Acima Development

In this episode of the Acima Development Podcast, Mike and the team delve into the importance of breaking down projects into manageable iterations to avoid stagnation and inefficiency. Mike shares a story about a past experience with a developer who struggled to produce releasable code, which led to the realization that breaking projects into smaller milestones can significantly enhance workflow and productivity. This approach allows for continuous integration and deployment, preventing the team from getting stuck on incomplete tasks. Mike emphasizes that this skill of breaking down work is crucial yet often overlooked in engineering education.

Will adds to the discussion by highlighting the value of creating "jigs" or scaffolding in both woodworking and software development. He explains that in his experience, especially with mobile app development, creating temporary structures or mockups can facilitate progress even when back-end systems are not fully ready. This method allows for continuous progress and testing, making it easier to adapt and iterate on the final product. Will’s anecdote about using a reverse proxy to simulate API endpoints underlines the practicality and effectiveness of this approach.

The conversation wraps up with broader reflections on iterative development and its applications in various fields, including aerospace and automotive technology. The team discusses how companies like SpaceX have revolutionized their industries through iterative processes, emphasizing the importance of starting with a functional but imperfect product and continuously improving it. They conclude by stressing the necessity of having a clear roadmap and the willingness to "cheat" by using tools and frameworks to eliminate human error, thus ensuring consistent and reliable outcomes. The episode ends with the advice to always build a jig and embrace iterative development for better efficiency and success in engineering projects.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting again today. With me, I have Will, Ramses, and Eddy. I have a story I'm going to tell before I introduce the topic, and I think about this a lot. I was thinking about it this morning. I think I thought about it yesterday. I think about it probably at least monthly about..., and I've probably even shared it on the podcast before [laughs].

But several years ago, I worked with a developer who was assigned to a project. We had talked about it as a team. We were all really excited about it. We had this new idea. It was kind of a developer-driven initiative. They worked on a little bit here, a little bit there, and a [chuckles] little bit over there. And this kept on going for, like, a month, and we thought that, you know, in a month, you'd have some code out. And looked at it, and there was kind of a framework or something there but there was nothing that was releasable. There were just pieces here and there, and it really felt like it stalled.

Now, they'd been working on this day in and day out for weeks, but nothing had made it into production. And in a continuous integration, continuous deployment environment where you can get stuff out there piece by piece, that was unusual. And I thought, you know, what's going on here? And I consider this a failing on my side. And I thought about it, so what am I doing wrong here?

But I think there's some maturity in career experience here for this developer. They hadn't learned how to take a project and break it down into iterations. I worked with the team to do that. And I say I worked with the...we kind of talked about this idea, and the team jumped in, you know, and I was just a minor player. But as they became aware of that, it completely changed the workflow of the team I was working in, where everybody focused on, well, how do I break this up into milestones? And, on a large scale, like, having a big project, how to break it up into milestones, and then, a small scale, developers are breaking up.

I had one developer come onto the team who made it like a singular focus. They'd come from another company where they spent, like, months on something [laughs] before they released it, and now they spend, like, multiple pull requests a day. And it's like their life is totally changed, and they're completely happier [laughs] because they've completely changed their workflow. They really embraced this and became kind of an evangelist. And I've seen it; I've seen this just be so successful for people who do this. And even for teams that get good at breaking things up, versus people who just stall for weeks or months. I've seen other...and that's certainly not the only instance I've seen. I just saw that one change, and then that developer became successful.

I've seen other scenarios where somebody jumps into something, you know, a challenging problem, and they spin, and they spin, and they spin, and they get weeks in, and then farther in, and just never get anywhere [laughs]. And it's frustrating for the person who's doing it. And there's a few things that feel worse than being deep in a problem you just can't solve it. And you just feel like there's no light at the end of the tunnel. It's frustrating for everybody who's working around them. It's frustrating for everybody.

Like I said, I think about this over and over again because I feel like it's one of the big skills in engineering that nobody really teaches, and that's a problem. It's just breaking up work. How do you break up work? And that's our topic today is, you know, how do you break up work, and why?

WILL: The North Star, for me...I do kind of want to pull on this a little bit because I've done stuff that just doesn't work like this, right? I mean, like, most of the business development, product ticket stuff that we work with can be addressed in this way. And then, basically, it's just sort of, like, what can you get out into someone's hands? And I would say, like, I don't know how to translate...I don't know how to segue in this, right? But, like, get it out into people's hands, right? And the thing that I think tends to jam people up is, maybe it's just me, but I think other people as well, is, like, you got to be okay with sort of like, let's call it, like, call it actual construction, right, where we're woodworking, or something like that.

You have to be okay making a jig for something, right? You have to be okay making scaffolding for something, right? Where I'm going to build this entire structure, right? Entire structure. But it isn't actually ever going to go out. Like, I'm building it so that I can do the small steps. So, one of the things I'm working on, right, not at Acima, at another very large retailer, is that I work on a mobile app, right? And so, the thing about the mobile app is a lot of the time, we're front-running the back-end API stuff, right? And so, what I'll do is I will scaffold out all kinds of stuff. I'll go and I'll negotiate back and forth with the back-end team, and I'll say like, "Okay, okay. Well, what's the data going to look like?" And they'll say, like, "Okay, it's going to look like this," right?

So, we'll do a contract, right? Which is just a contract. It's just aspirational, right? This is pure vaporware. But I'll get the contract going, and then I'll start charging ahead, right? And I'll just be like, okay, mock data, you know what I mean? Promise that's going to result to this thing, right? And then, I use a tool. It's a reverse proxy. And I can use that to create, like, a fake API endpoint, right? Where it's like, if you go to this API, this URL, it's going to give you back [vocalization], right? It's going to give you just...it's going to serve up this static file, right?

And now, oh, okay, now I'm making, like, real API requests to fake API, right? And all this stuff is junk, right? It's junk. It'll never go live. But I can actually release the thing. It is real stuff. It might even really be on the server hidden behind, like, some feature flag wall, right? And so, it's out. It's in production. It exists. It's a real thing. But, also, no, it's not, right? It's walled off, and it requires, like, all this fake stuff. I'm like, hey, hey, you know what?

And I'll tell you, like, honestly, like, I have hit on a contract before, but I am much more likely to miss on something. And it's like, oh, well, I have to go to the pricing and availability service, so it's going to be here. And I have to make another call. This JSON's going to...and you know what? It's fine. It's fine. I will have to go back and rework, but parsing Jason is really not that hard. In the end of things, it's really not that hard to parse some JSON. And so, like, you know, and then you'll go back.

But, like, if I wanted to do it, like, the quote, unquote "right way," and get it a hundred percent, then I would have all of these sort of, like, blocking features, where I can't move until the feature's done. I can't move until this is done. I can't move until this is done. And I'm just like, eh, F it. Let's roll. And so, you just do it that way, right? And so, you're making assumptions, making trade-offs, debating stuff, and scaffolding, and, like, you know what I mean, going with prototypes. But you're always putting something out into the world. And that has been very, very, very successful. And it's allowed them to get things out at a pace that they ordinarily just...there's no way [laughs]. There's no way.

MIKE: Yeah, it's interesting you mentioned the woodworking. My dad spent most of his career as a professional woodworker. That is exactly what [laughs] they spent most of their time on was: building those jigs, building the tooling to actually do what you needed. I don't know how much time my dad spent building tools. He still...he makes tools. Like, he'll see a problem. He'll go and make a tool because that's how he thinks now [laughs].

About ten years ago, my brother-in-law was showing him some grips for a handgun. My dad's never been, like, a huge gun enthusiast, but he saw that as a problem. He got curious about it, and he went and built a jig and made grips. And it turns out they were better than most of the stuff on the market. And he ended up doing it as kind of a side project hobby that kind of made him a living for several years [laughter] because he took the time to make that jig. Like, I don't really even hear that word outside of that industry, you know, outside of woodworking, but that's the word they use. And it's not exactly a tool, right? It's like a template.

WILL: Yeah. Yeah. Like a template. So, for people who are not necessarily, like, wood butcher nerds...so, I'll give you an example, right? So, a guy I know he made this really beautiful grill table, right? And you could think of it like an outdoor kitchen table with this circular hole, and his grill goes in the middle. And you've got, like, your whole cook surface out there, and it's super, super cool. And I looked at it, right, and it's like, it's this beautiful table. His grill goes right on the side, and you could do all this stuff.

But, like, what he did is he had this beautiful three-foot circular hole for his grill to go on, right? One does not just cut a perfect three-foot circular hole. Like, one does not just walk into Mordor or cut a perfect circle [chuckles] in a butcher block countertop. And so, I was like, "Oh, what did you do," right? Well, he created this, like, not exactly, like, a router. The router was a part of it, but, like, it was this thing to hold the tabletop in place. And he had, like, a thing, and it had an arm that it swung out, and the router cut this perfect circle. But he built, like, a scaffolding machine so that he could cut this perfect circle because you couldn't possibly do that by hand precisely.

I might have dogged it and done it with like a, you know, done it with, like, a jigsaw and then just, like, put a little trim over it, like, a little metal, [laughter] like, a fat metal piece of trim, like, that big. And it's just like, ah, it's bad, but you can't see it, so who cares? So, that's what a jig is, you know what I mean? Like, it holds the work in place so that you can work on it effectively and precisely a lot of the time. Because if you just try and freehand it, you're screwed. That's why [inaudible 10:07] shop projects, you know what I mean? Like, when you did woodshop, if you did woodshop, they all look terrible because...

MIKE: [laughs] Right.

WILL: Because you tried to freehand it. You're just like, oh, well, let's just cut this. And I'm like, no, no, no, no [laughs]. You get something to hold it precisely so that you have a template, and you can do everything perfectly.

MIKE: No, that's exactly it. The professionals, it's not that they're better at freehanding it, although they might be, is they don't trust themselves to freehand it.

WILL: Exactly.

MIKE: And so, they don't [laughs]. They build a system so that they eliminate the human mistake. That completely changes it, and that's how you end up with a quality piece of work. It's such a perfect analogy. You take the time to build the jig, you know, to build the scaffolding, to build the framework that allows you to test and deploy independently a component without necessarily having the other pieces in place. And without that, though, you're blocked, right? Because there's always going to be some other thing that you're working with.

What we do [chuckles] is we read data from one place, and we transform it, and we write it somewhere else. That's what we do for a living. If your data sources aren't ready yet, then you're blocked, unless you define a contract and you code to that. And then you say...you have some sort of test harness that you can test against that contract, knowing what they have may not be ready yet. And you might even have the wrong contract. You might have to make some changes later, but at least you'll have something that you can make tweaks to rather than having to start from scratch.

WILL: I think it's a good rule of thumb. I think it's a good rubric. Although I have been involved in projects where, like, you know what I mean, things were...you had to do...actually, you know what, honestly, I think I've always done that. I haven't always had the latitude to, like, front-run as much as I wanted to. Like, one of the things that I did, like, way back in the day, is I did a bunch of audio processing stuff, right, for phones and stuff like that back when, like, you know, the world was new on the iPhone. And I would do audio processing.

And the thing about audio processing is audio processing, like, there's not a lot of, like, intermediate steps. You really got to, like, you got to kind of, like, do the whole thing. You know, like, you have to have, like, the machine has to, like, do finished products. But, like, honestly, like, I still did the same thing. I mean, in that I would just make it play a song. I was like, oh, I can't play a song, right? Because, like, okay, oh, you want to play a song, right? Well, I had to have all kinds of stuff.

I had to have, like, a wave decoder, and then a wave, you know what I mean, like, a wave consumer, and stuff like that. And so, I was like, oh, that's too hard. I'll just make a sine wave because I could do that in math. I could do that mathematically. That's a mathematical function, right? Then I don't have to have codecs or anything like that. And then, I did that. And I was like, oh, okay, I did that. Okay. And now, I'll do a wave. Okay. Now I'll do a wave, right? It's like, okay, now I'll do an MP3 into a wave into, like, the speaker. I was like, okay, now I did that.

Now, I will take the MP3 into a wave. I'll capture the audio and do nothing, right? Nothing, just a hand, just passing it through, right? And it'll go out. And then, I'll, you know, and then I'll just do, you know, something else, and then, you know what I mean? And so, I would, you know, just kind of do stuff like that.

MIKE: But the process, like, even if you're not releasing it, in order to be able to...and this ties into testing, it seems like, right? Like, you mentioned the testing. At every step, you're doing something that you could verify. You wanted something that was kind of tangible. And I know none of this is purely tangible, but it was...the closest you're going to get to tangible is something...like, it was making it into sound that was making it into your ears. There was something that you could measure that you could validate on. And once you have that, you could build around it.

WILL: Yeah. Yeah. And I built all kinds of jigs. One thing that I did is I was...so, we were working on, like, continuous DJ mixes, right? And so, it was a bunch of MP3s that were encoded and recorded, you know, in such a way so that there would be a continuous stream of music. So, it'd go play from track one to track two to track three to track four to track five. You would never hear a transition, right? You could do that. But I was transforming and manipulating the stuff, right?

And so, like, when it would go over, it would pop. It would be a hitch. And I was trying to debug that one particular thing. And I was like, ah, like, my ADD brain won't let me sit and wait for the thing to roll over. So, I was just like, okay, I'm going to make a jig so that I just jump in five seconds before it ends. And then, I'll play through the transition, and I'll go and do five seconds again, and then I'll, you know, and that's how that goes through the song.

So, I'm only doing, like, the specific part that I want to see because I don't have time for all that, you know? It's just jigs. Just that's a way of doing it, I guess. My dad always taught me...my dad was an engineer, not a woodworker, but he always said, "Go from the known to the unknown." You always want to be working from a known good state, you know what I mean? It can be as stupid as you want it to be.

MIKE: [laughs]

WILL: As dumb, as useless, like, I've looked at a lot of blinking, red lights, bright in my heart.

MIKE: [laughs]

WILL: Like joy. Like, where I'm just like, look at that. Let's blink it, baby. Yeah, we're blinking. We're blinking hard. And I'm just like, is that it? That's what you did today? It's like, yeah, baby. Look at it [laughter]. If you're a real engineer, you're just like, ooh, that's a good blink.

MIKE: [laughs]

WILL: What does that mean? What [laughs] does that mean? What does that light flashing on and off signify in your brain? You know, what is that? It represents progress towards some goal, you know? But, like, yeah, just that stupid, easily that stupid. All day [laughs].

MIKE: You mentioned a blinking light. You say it represented something, signified something, right? You've got that [inaudible 16:18]. You've got something that you can actually wrap your head around. You can verify there is blinking light. I have power [laughs], and it is making something happen.

WILL: We're blinking hard. We're blinking hard today, Mike [laughs].

MIKE: There is a project that was challenging to test because [laughter] there's a...and probably [laughs] anybody who's done this for very long, you probably have encountered this situation before. It's not unique. And you have a partner who doesn't have a sandbox, or they do, and it doesn't work very well. So, you try to test against this third party, and it just doesn't work right. Like, there's not a good ability to test against the API. And when we're running unit tests, we run into this all the time, and we just say "No," right? There's no way you're going to run your unit test suite against the third-party service. And if you do, you're suffering because [laughs], you know, it's so unreliable.

You can't have reliable tests that actually test something if you're running against an unreliable service. And they'll be, like, 100 times slower, and all the problems with that. You know, you build. You mock it out, right? You build a fake service so you can test against that, and that's what you work with. But sometimes we don't think to do that when we're doing manual testing. Like, well, my unit test does it [inaudible 17:39] manual test. I need to test against the real thing. You know, I want to do an integration test. It has to be perfect. Instead of building the tool to test against.

And, in this project, I regret, in retrospect, not pushing hard to say, "Build the test service. Build the test service. It's going to save you so much time." People spun their wheels for hours and hours trying to get stuff done, trying to share a staging environment to test against this sandbox on the service that was sometimes up and sometimes down. Or if they had just had a service running locally that maybe wasn't perfect but it was pretty close to the third party, they could have saved themselves a lot of grief.

WILL: Mm-hmm. Mm-hmm. Yeah. Yeah, absolutely. I mean, I will say that, like, a little...one of the things I've learned at Acima is the many blessings of the VCR gem. If anybody's listening who doesn't know about VCR, basically, all you do is you run your test suite against a, you know, a third-party, or external or, like, whatever, external API source, and you go out, and you hit the real API. And you save it.

And then, you're testing against the real call, but you don't have to, like, go out and do all that every single time, you know. And you could configure it in various ways so that, like, you refresh the stuff, or you make sure it doesn't get stale, and stuff like that. Because, like, yeah, I want the real, real, you know. I don't trust any of you people. You guys are all liars. Engineers cannot be trusted [chuckles].

MIKE: It does give you...you get exactly the response, the real thing. But then you can save it so you don't have to hit them again. So, you get your reliable test. And you can build a service that will respond with that data and run it in your test environments. You can do some manual testing and stuff through the process. Why not?

WILL: Like, I love my reverse proxy for mobile development, right? Mobile development against unstable APIs, shifting APIs, you know, like, I found something today, just today, and I just patched it. If every single thing failed on me, right, I'm hitting a bunch of endpoints, and I'm aggregating together, and, like, none of them should fail. And almost always, at least one of them would come back. But if every single endpoint that should never fail actually did fail, then it was taking down my whole...not my whole app, but my whole, like, page within the app.

I made a jig, right? And I made a jig because I was in a hurry. And I was just like, I'm just going to take this API call and I'm going to, like, I'm going to butcher it so it could never fail, right? So, it's just going to fail all over the place. I'd, like, dug into the source code, and I'm just like, and you will die. Promise dot reject abort, you know what I mean? And then, fixing the problem was really easy because I was sort of refreshing, refreshing, refreshing, but I couldn't get this intermittent failure to go. Unfortunately, I work for a very large retailer, and I will get those millions of requests in the wild, you know, very quickly, you know. And so, anyway, make a jig [laughs].

MIKE: I think Will hit on something critical, which is, you know, you build the support structure so that you can build these incremental pieces. One thing that we really haven't talked about is you get a project, and this happens at any scale, you know, whether it's something that takes two hours, something that takes two days, something that takes two months, something that takes two years. You've got this big thing, and you got to start somewhere. And I think that sometimes we want to see progress, so we just start somewhere, right [chuckles]? Let's chip away at something. And that's one way to do it.

And going back to our starting, where we started this podcast, you can chip away here and there. You end up with a lot of this and that, but the whole thing doesn't move forward. I think you need to be really methodical about choosing what you do first. And it goes back to that jig idea. You have to have some sort of framework, mental framework, to map out how you're going to do the work.

I'm thinking of another project that I worked on a few years ago. I had spent several weeks in meetings with product and business, and they were laying out these ideas. And mostly, what they had was mockups. It's like, this is what it's going to look like. But nowhere in there was anything about how it was actually going to work [chuckles]. And there were a lot of really big questions. You know, here's how it looks like in the user interface. But how on earth are you going to make this work behind the scenes? You know, go and work with whatever providers, and leave out the details. But, you know, you're going to need third-party data to make this work, and it's not trivial.

And after doing this, I got together with a skilled product manager, and we sat down for a few hours and hashed out a roadmap and said, "These are the milestones we need to hit." And that roadmap, more or less unchanged, ended up being how that project got done. It was on a really tight timeframe. We got it out in about three months, made it before Christmas, and you get the release. It wasn't the greatest product [laughs], but we got the thing done.

And the problem was actually not our implementation as much as that we were trying out a new business idea that wasn't as good as, you know, as they'd hoped. It happens. The important thing is we got it done. And the only reason we got it done, instead of spending the next two of those three months in endless meetings talking about the user interface, is that we sat down and said, "Well, what is the practical plan? What could we do first?"

And the thing that we could do first...and then, we went and actually talked to the business and said, "What do you need first?" "We need to be able to accept a credit card payment," or whatever it was. And having that first thing opened the gate to do another thing. It's like Will's blinking lights or being able to play a pure tone sine wave [vocalization] [chuckles].

You get that. You get something to play. You get something to start with. That zero to something is so much more than even the subsequent steps. Something to hang on is just wildly better [chuckles] than nothing. The zero to something is the biggest, hardest part. And getting that tangible sort of thing made a huge difference. And then, other people can start talking about, where do they fit in the process? You know, you can start coordinating across teams to figure out where they fit. And pretty soon, you have a project running. And you do that a lot.

I've worked many projects like this. I'm working on one right now, where the very first thing we did is we sat down, laid down the roadmap, and the project's going well. It's going to be...other than some challenges dealing with a government entity that's just sort of slow [laughs], like, we're going to be on time, close to on budget, you know, it's just going to work out because we've mapped it out like that.

You see, the difference is really not that much. You know, you take a few hours and grapple with the problem and think about how to break it up, rather than getting overwhelmed and just chipping at something and getting one little piece. You're probably never going to make it very far. Now, chipping one little piece is better than nothing, but I think...well, maybe [laughs]. But I think that taking some time to analyze the problem and it's work, and figure out what comes first, and then what comes next, and then what comes next...well, where are your most risky pieces? Maybe you have some government entity you have to get something approved by. Well, you better get that done first.

You know, laying out an order of operations and doing those is just, I think, one of the most valuable things that we do. And I don't think I was, you know, I don't know that it's something they teach in school. They just expect you to know it. But I feel like that's something that ought to be just repeated over and over and over again.

WILL: I couldn't possibly agree more. And I'll tell you, like, also, like, I mean, it's like, you know, sort of, like, in terms of, like, the tactical sort of, like, thrust and parry dealing with, like, business product, stakeholders, stuff like that. It is...how do I put it? Like, if you can distill a feature set down to, like, the absolute, absolute, absolute minimum, like, when you're faced with sort of, like, you know, a lot of demands, a lot of shifting demands, a lot of crazy deadlines, a lot of pushes, and stuff like that...And I understand that, like, what I'm saying here is just, like, plain vanilla, basic agile. But it's the kind of thing it's that's easy to say but very, very difficult to do.

But, like, the more you could distill it down to, like, just the basic core, raw functionality, and then, like, you add stuff on it, I found that when you get that sort of, like, that core workflow finished and you find yourself in a situation where it's like, hey, I'm not going to make this deadline, most product people, like, that I've dealt with, have been very practical, and pragmatic, and reasonable. If you give them something and you say like, "Listen, I can do this other stuff, right, but it's going to push it by a month. Do you want me to do that?" And then, they'll just be like, "No," like, they can make a purchase. They could buy a widget. They can do this thing that they want to do.

And, like, really distilling that down, finding the stuff, where it's like, oh, well, you know what? I'm not going to handle errors gracefully. Like, I'm just going to be, like, is it right? Cool. And then, you get the green check. Is it wrong? No. I'm bailing you out. I'm dumping the whole thing. I'm just dumping, right? No slickness, cleverness, anything. If you can give product scenarios like that, then I've found my ability to have an untroubled existence as an engineer while still maintaining a reputation for actually putting out products has, like, it's gotten so much better.

I don't know, I mean, they get it, but, like, when it's not done, when they don't have a done thing that they can either invest more in or they can ship right now, then, like, that's when the sort of, like, the wish list grows, and grows, and grows, and grows, and grows because it's not finished, right?

And this is real tactical stuff, but if it's done, and they have it, and they can either go to market, or they can delay. If you give them the choice, they almost always go to market. If you could give them something that works, they'll take it, but if you don't, then it's just like, oh, I got to have that. I got to have this. I have to have this feature. We couldn't possibly go without this feature because it's not something or nothing. It's nothing or another flavor of nothing, whatever. That's your problem, you know.

MIKE: That, what you just said, you know [laughs], if you've got nothing, like, you imagine any kinds of nothing, and what you have is nothing. But if you have something, something, right, you can work with that. And maybe it's not enough to go to market with, but it's at least something you can talk about.

Like you said, this goes back to the agile process. The core idea of agile software development is around communication loops. You want to have an iterative cycle rather than no cycle, you know, just planning upfront, build it, and then done. And you can only have that iterative loop if you actually have something to iterate on. And we've talked about this before a lot of times.

You look at aerospace [laughs] because it's big, and things blow up, and it's easy for us all to see. You look at what SpaceX has done to just absolutely dominate, not just, like, the U.S. market, but they launch more rockets than, like, the whole world put together. Like, the nation-state, right [laughs]? And it's because they have this very iterative process. You know, they're like, well, we're not going to build a perfect rocket. Let's build something that's maybe going to blow up, but at least it should be able to get off the ground, and we can measure some stuff on it. And they do that, and it probably blows up.

But they got those measurements, and then the next one gets a bit higher. And the willingness to let things blow up and to be interested in that interaction, right, in that iteration more than in the perfection, drives the costs wildly down.

WILL: I'm actually really interested in, like, so one of the other Elon projects that I'm personally extremely interested in is, like, the full self-driving mode on the Tesla. And one of the things that I'm very, very interested in, in that capacity, is willingness to...I hate to say, like, blow up some rockets, but, like, willingness to put out a full self-driving automobile that is just a little bit better than a human, right? Because, I mean, we're all right. We're not great, you know, we're all right, you know.

But I think if General Motors, you know, or one of the large institutional players was going out and putting together a self-driving car, or even somebody like Google that has a lot to lose, I think they would and do really struggle with, sort of, like, something that's just better and not perfect, right? Like, no, I don't want it to be worse. I don't want it to be worse. That's not cool, you know what I mean? Like, we can't have, like, flying, like, battering rams just barreling down the freeway murdering people. Like, that's not okay. However, what if it's just, like, 20% better than a human, 20% safer than a human? Isn't that better?

MIKE: That is better.

WILL: You know? Anyway, I mean, and so I think that that mindset, I think is, I don't know, I'm interested to see how it goes out. Because, like, it's also one of those sort of things where it's like, you know, you're kind of, like, gambling with somebody else's life there a little bit, Elon. And move fast and break things is not meant to be taken quite so literally.

MIKE: [laughs] That's an interesting one because to drive better than a human, you have to have superhuman skills, and let me explain what I mean. I'm going to have a very specific meaning that may not be what you're interpreting me to mean. And they call this the March of Nines. Have you heard this Match of Nines? If you want to have 90% accuracy, it takes a certain amount of work. And then, to get to 99% accuracy, you've only got one digit further, right? It's like ten times as much work. And to get one digit further, you know, 99.9, you got to do about ten times as much work, and so on.

And each one of those is harder, and part of the reason for this is that you think about driving, and most of the time, driving is pretty boring. Most of the time, driving is pretty boring until the nearby zoo has an error, and then a kangaroo ends up jumping across the road in front of you. And to deal with that situation, you have to have the concept...well, maybe it's not even a kangaroo, right? You know, maybe it's something that you've never even seen before. I almost hit a possum the other day, literally almost hit a possum on my bicycle. And [laughs] had to slam on my brakes, or I would have run over the possum [laughter]. That's not something that happens every day, right? You don't every day go riding along and almost hit a possum.

And I didn't necessarily need to recognize it was a possum, although it took me a moment before I realized what it was. You just have to know that there is something that is walking across the road, and I need to stop. But having that concept of an animal, and that it has direction, and knowing which way to steer because you can tell which direction it's going, and this is very much out of the norm, requires an understanding of a lot more than driving, right? You have to understand something about biology, and something about physics, and some other problems.

What if the street sign's broken, and there's a police officer standing in the middle of the intersection, and they're waving people with their hand, and doing hand signals to say which way to go? Now, your car needs to not just understand traffic lights. They need to understand human gestures. And maybe the police officer is just yelling at you, "Hey, please! Your turn. Go." Now, you have to understand human speech and have a microphone.

Like, as you deal with more and more of these situations, there are situations that require human understanding because our world of driving is built around humans. And I've actually heard this talked about by self-driving engineers. It's on my mind. These roads are not designed for the machines. And so, the problem is a challenging one and challenging in ways, like, it's hard to measure.

Now, this doesn't really go against what you were saying before. You may be able, like, because humans aren't that great at drivers, make the car 20% better, and then have, like, a data center somewhere where people are sitting there, and they'll take over. And they'll have a little camera and see, "Oh, the police officer is waving. Okay, make the car go left, you know, and take over manually." I mean, you might have something like that where you actually take human control or give it back to the human and say, "Hey, you know what? I'm not going to self-drive anymore. Your job." There are solutions like that where you just bail.

WILL: I mean, honestly, like, what I think ought to be done, I mean, like, I don't know. I mean, like, everybody's sort of, like, dumping money into this intention of, like, you know, like, carving out their, like, little digital fiefdom monopoly thing. And I don't think transportation is, like, the kind of thing that can really support that kind of a capitalist enterprise because it's public good, you know?

And so, I think it should be a public-private partnership, where we start creating roadways that have, you know, affordances built into them for self-driving cars. And, like, this is just me, but it goes right back to, like, what we've been talking about. What if your self-driving car was just good on the big streets? That's it. It's good on the big streets, you know what I mean? Like, get it to, like, a four-lane road, you know what I mean? Like, whatever your four-lane stoplight road is, you know what I mean? And you push a button, and you say, "Bang it out. I'm watching Netflix," you know what I mean? And that's it.

MIKE: [laughs]

WILL: That's cool. That's cool. We're good. We're good, man. This is a good gig. Like, I could watch Netflix. I could check my email. I can do my stuff. I don't have to sweat the traffic jams on the highway. I don't have to sweat stop-and-go traffic. I'm cool. We're all right. And more over than that, you know what I mean? I could do even better than that. And I could just say like, hey, you know what? I got my e-bike. Like, I'm just going to go to the city bus...and my city bus, instead of being the bus, you know what I mean?

So, I have to, like, pedal, you know, a block or two to, like, my transit point, throw my e-bike on there. And then, it just takes me right to my office, and we're good to go, you know what I mean? And cars have gone away. We didn't solve the hard part. I don't need to know what...like, my car doesn't know what a zebra is. It doesn't care, you know what I mean? Like, I just built, you know, I built the car to deal with, like, you know what I mean? I bounded the problem, right, so that the self-driving stuff works. And I made the roadway in a way so that the self-driving stuff can, like, process it and, like, we're all good. Any true engineer is always looking to cheat.

MIKE: [Chuckles] And that's been kind of the theme of the conversation, right? A good engineer as Will is interested in cheating. Don't freehand it.

WILL: Don't freehand it, yeah.

MIKE: Because that's a solvable problem [laughs]. But you can take away the human element, and then you have a different problem that's solvable.

WILL: Exactly. Yeah. My hand shakes just as much as yours does. That's why I made a jig, and that circle is flawless every time. I can make a thousand of them, and each one would be just as perfect as the last.

MIKE: Well, that feels like it kind of ties things up, doesn't it [laughs]?

WILL: I love it [laughs].

MIKE: We went from solving our everyday problems to dealing with the big problems of the world. And they have the same constraints. We will close this up. Thank you, all, for listening. And, you know, build a jig [laughs].

WILL: Build a jig.

MIKE: Build a jig.

WILL: Good engineers cheat. Build a jig [laughs].

In this episode of the Acima Development Podcast, Mike and the panel talk about the intricacies of project estimation, sharing personal anecdotes and professional experiences. Mike begins with a story from his childhood, illustrating how his father's ability to estimate distances accurately influenced his understanding of estimation as a skill that can be developed with practice. He draws parallels between estimating physical distances and estimating project timelines, emphasizing the importance of breaking down tasks into smaller, manageable pieces to improve accuracy.

The conversation transitions to discussing the best practices for project estimation within engineering teams. Matt and Justin highlight the effectiveness of comparing new tasks to similar past projects and the necessity of having a stable team to ensure consistent delivery. They stress the importance of planning in small increments and maintaining clear communication among team members to mitigate risks and handle scope changes efficiently. Mike adds that understanding and identifying unknowns early in the process is crucial for making realistic estimates and avoiding common cognitive biases that can lead to significant project delays and budget overruns.

Towards the end, the team explores the challenges of maintaining estimation accuracy during organizational changes, such as team reassignments and company-wide reorgs. Will raises concerns about the impact of these changes on planning bandwidth and overall productivity. The group agrees that preserving team dynamics and ensuring clear lines of communication are essential to minimizing disruption. They discuss the importance of having documented rules and metrics at every planning level to measure effectiveness and ensure alignment with business goals. The episode concludes with a reflection on the necessity of disciplined planning and continuous improvement to successfully manage complex engineering projects.

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I have with me here Will, Justin, Eddy, and Matt. And we're going to be talking today about strategies and pitfalls in estimation. I'm going to start with just a story from my childhood.

We were talking in the pre-recording for a bit about math. I was remembering that my dad used to ask math-related questions to me and my siblings and make a game out of it. Like, he wasn't, like, grumpy, or pushy, or anything. He was like, "What do you think of this?" And then, we'd think about it. This is on road trips. He'd always do it on road trips. Like, that's the only time I ever remember this happening [laughs]. You know, you're on a road trip back in the long ago era before there was internet in your car [laughs]. And you had to do something to fill the space.

I remember that there was mountain ranges, and he'd pick a landmark. You know, there'd be, like, this mountain. "How far away do you think that is? Guess." And my dad was always right. And all of us were wildly wrong [laughter]. He was always really close. My dad's a carpenter. He's very, very good at recognizing distances. Like, he can look at something 15 inches long. He'll look at it and say, "That's 15 inches [laughs]." And he's right. You know, he's as good as a ruler [laughs].

He's just really good at that perception, you know, distance perception. But it applies even to larger scales. So, you know, he'd say, "How far is this away?" And I'd say, "Oh, that's three miles." And my sister would say, "Oh, that's two miles." My dad would say, "Oh, I think that's 13 miles." And it'd probably be, like, 12.5 [laughs]. And it's something that you have to work on. I've seen this.

Most people, you ask them, you know, "How long is three inches?" And they don't know because they don't work with it every day. Like, how would you develop that skill unless you're a carpenter [laughs], right? Unless you're practicing with it many times a day. And estimation of how long something takes, I don't think, is any different than that. It's something that can be developed as a skill, but it's hard. And it's especially hard when we're doing that with projects that you're working on.

You don't see the mountain out there, right [chuckles]? So, you don't really have a good landmark. You're basing it on where you think the landmark's going to be. So, you're not just guessing how far is the landmark, but you're guessing where that milestone is. And it makes for a really difficult problem that hits some of our cognitive biases really badly. And we just get it wrong over and over and over again. So, that's kind of framing of the problem.

Just in general, distance estimation is a tricky problem that you have to develop as a skill and estimating projects. And I think this, you know, we're talking about engineering, but it probably applies to other kinds of projects as well. It's a hard problem because the scale is hard to gauge, and, again, it hits some of our cognitive biases.

WILL: So, I've got a question for you guys, maybe for the group, right? Like, in your experience, right? We've got some miles under our belts. What's the best situation in terms of estimating projects and delivering on those estimations consistently that you guys have all individually experienced firsthand, not somebody that you read about in a book, or a friend's thing, or somebody's plan? Like, what's the best setup that you've had yourself?

JUSTIN: So, the best way for me to estimate something is to get asked to do something that I've done before.

[laughter]

WILL: No, no, no, no, no, no, no. Hang on, hang on, no, no, I'm talking about, like, the best setup, like, doing the work. The best setup where you've been like, "Hey, we're going to have to do this thing." And you're like, "Okay, I'm going to do this thing," you know what I mean? And, like, "I think it's going to take me six weeks." And it takes you, you know, only eight, you know [laughter], or whatever it's going to be, like, the best system that you've worked in yourself.

MATT: I find what works the best for myself, anyway, is to do a comparison on something I've done that's very similar in nature. You know, you have that mountain that Mike spoke about, and it's a lot easier than just conceptualizing it. If you can relate it to something else, then it's a lot easier to break down into the problems. And I find the smaller you can break those problems down, and if you can estimate the problems one at a time, then you have a really good idea of what the larger scope is going to be. That's where I've had the most success.

WILL: Have you had the opportunity to consistently work in that fashion? I mean, like, has that been, like, an ongoing experience for you, or is just, like, what you would like to work on if you could but you can't?

MIKE: I've seen this on teams I've worked with and teams I've worked closely with. And Matt hit on a few of the things that are critical. First of all [chuckles], you have to break it down into small, discrete pieces that are small enough you can wrap your head around them. That is absolutely critical, and that is not free. It takes time.

You have to break down the work into a map and, you know, say, "I'm going to do this, and I'm going to do this, and I'm going to do this." When I say "I," a collective I, you know, the we. We're going to do this, and we're going to do this, and we're going to do this. And, you know, we usually use story points in software development, which is this unitless measure, right? You say, "This is how complex it is." And if there's anything above a three, then you're not good. It has to be small, discrete projects that somebody can say, "Oh yeah, I know how to do that."

Unless you've broken down everything into, I know how to do that, and they're in order, then you're going to be off. And if you do that, though, if you have everything broken down into, like, nothing bigger than size three, and you have a team that you've worked with before, because unless you have some sort of history to know how...you have to have a history. There's a big caveat here. You have to work together as a team for a while so that you know what you can do as a team over time in terms of those points. Then, it tends to land very close. Maybe not exactly perfect, but on average, pretty good.

WILL: I agree with all of that. And you can't mess with the scope midstream.

MIKE: So, you can't change the team. You can't change the scope. Those have to be fixed. You change the team, well, now you've changed the dynamics, right? You've changed the dynamics of how you work together, and work gets done as a team. It doesn't get done as individuals in that way because the team dynamics are relevant. So, you have to have a stable team. And, yeah, if you change the scope, well, that has to be planned just like everything else, and it has to be additive to it.

MATT: Scope changes will happen, but if you tackle them the same way you tackle the initial problem, then you can also estimate, hey, it's going to add X number of weeks to this particular project, right? Or whatever piece of time that is. The key is small, workable pieces.

JUSTIN: Clearly identifying the unknowns.

MIKE: Well, if it's unknown...so, I'm going to dig a little bit more on that because I think it's critical. Identifying the unknowns, that's not a one-liner [laughs] because...so, I mentioned that, and I said this when I was talking about it, you're measuring the complexity. I don't think the story points should ever be used to measure time when you're estimating. They're not a measurement of time.

MATT: Absolutely not.

MIKE: They are a measurement of complexity. And if there is unknowns, it's not truly estimable because that means that you don't understand the work yet. Like, the rules for this plan, right? This only works if you can do everything that I said, which is that you know what those pieces are. I mean, you don't know exactly what you are, but you can wrap your head around it, right? And you know what you can do. If there are unknowns, that means that you haven't refined it down enough. And it's a measurement of complexity. As for those, you'd bump it up to, like, eight or, you know, something outside of what your boundaries are. Because unless it's down to three, then you don't really know. And here's a critical reason why.

I read an article about this, and I don't have it up in front of me, years ago, but it's brilliant. It says that we're very good at estimating the median time to finish, but the mean is way higher than the median. So, if you line up all the projects, and you pick the one in the middle, we're pretty good at that. So, talking about these cognitive biases, we're actually pretty good at picking that one.

The problem is the complexity is very non-linear, and we have a hard time wrapping our heads around non-linear things. So, anything to the higher end of that median doesn't just go up linearly. It goes up exponentially or worse. So, the projects that are harder, that are worse than that median, might be ten times worse, or a hundred times worse, and so they drive the mean up really high. And then, you have a project that's three years late and over budget because it's really hard to predict where those unknowns and complexity are going to land.

MATT: And you can't predict things that aren't known. That's the whole thing. Generally, yes, you're going to bump into a few unknowns. Part of it is trying to account for a rough percentage in your complexity. I mean, and it's really, I would say, experience. There are certain things that you encounter often. You just really have to think through and understand your problem to be able to accurately represent it.

JUSTIN: Yeah. And it's almost like going back to what you said, Mike, you've got to resolve those unknowns before you can give an estimate, and so that's separate work. It's, like, a spike to resolve the unknowns.

MIKE: Yes, absolutely. Listeners can't see this, but I just posted in our chat together. There's an xkcd comic that I think is fantastic. Boss is coming in, talking to the developer. He says, "User takes a photo. The app should check whether they're in a national park." She's like, "Sure, easy GPS lookup. Give me a few hours."

JUSTIN: [laughs]

MIKE: He says, "Check whether the photo's a bird." She says, "I'll need a research team in five years [laughter]." In computer science, it can be hard to explain the difference between the easy and the virtually impossible, and that is so true. And there's this thing that things that are easy for humans are hard for machines, and vice versa. Like, yeah, fold the laundry, sure, I'll do that in five minutes. Have the machine fold the laundry? Good luck [laughs]. Good luck. In your mind, that little mental shortcut that says, "Oh yeah, then I'll do the next step," may not map very well to how you actually make that happen.

JUSTIN: As an aside, that comic, the second comment from the lady programmer, says, "I'll need a research team in five years." Nowadays, it's like, "Let me make an API call to ChatGPT." It's done in five minutes.

MIKE: Well, what's even better is the [inaudible 11:24] message, which, [inaudible 11:27] xkcd is perfect. It says, "In the 1960s, Marvin Minsky assigned a couple of undergrads to spend the summer programming a computer to use a camera to identify objects in a scene. He figured they'd have the problem solved by the end of the summer. Half a century later, we're still working on it [laughter]." That's how it goes.

WILL: So, I'm curious, what is this, like...so, this is an organizational problem, and I think we all sort of, more or less, like, understand, like, how to manage this, right? But on an organizational level, it's really, like, sort of, like, interfacing with, like, the broader teams and, like, where's it gone right and where's it gone wrong and how. Because I myself, like, I've experienced some pretty functional teams in that regard in that, like, you know, we come in with, like, sort of a product plan. We plan the tickets before the sprint starts. We hold to the sprint scope. We plan everything out. Boom, boom, boom, right down the line. And we do MVPs, things like that.

And, you know, I mean, I'm sort of, like, experiencing right now in real time a degradation of that process, where product doesn't want to come into the sprint with a defined scope. Our product wants to give me...they want us to start work when I don't have, you know, a Figma design document that tells me where all the buttons are going to be and what they might want to do, you know what I mean?

I'm going out, and I'm developing the frontend when there is no backend at all. And I'm sort of like, and I'm like, "Well, do you have a schema for what the backend might eventually look like?" "Well, not really," you know what I mean? You could work on all that kind of stuff, but it increases [inaudible 13:16]. It increases rework, inefficiency. And, quite frankly, like, it really kind of pisses me off, you know, like, where, like, I have to, like, my job gets three times harder because people couldn't be bothered to, like, you know, make an effort at the outset of the project. Now I'm kind of curious about, like, sort of like, I've seen it go good and, like, where you've seen the process break down and how.

MIKE: You just hit on, like, all the failure modes, right [laughter]? Requirements not gathered before the work starts. Now, in the real world, sometimes you don't know everything. But, like, you should at least have the overall direction defined so that...certainly, by the sprint, you know what you're going to be working on. If you don't have the requirements, that's just a recipe for burnout and over budget, over time, over everything.

And cross-team, you talked about different teams that are not in alignment. And you're like, maybe the backend's not working yet, but the frontend is. And so, now you have all the misalignment between teams, and getting those coordinated is always, like, the hardest part [laughter]. And this sounds like you're also talking about some team dysfunction, where people are like, "No, I don't feel like doing that." Like, you don't have the sense of collaboration between the parties. And, man, if you don't have engineering and product working together, you're in for a world of hurt.

And why does this happen? I think it tends to happen for the same reasons repeatedly is that unless we're working at non-profits, and I think it even applies there, I mean, we're trying to make money, or, you know, we're trying to maximize our utility, and something comes up. They want to get it done. They being leadership at some level, wherever that is, it can happen at any tier, wants to get something done. And you don't have a great process for aligning that to reality. Then you're going to have more than you can get done.

And there's going to be this misalignment of reporting because you said, "Oh, I gave a prediction." But now you've added to the scope. You've thrown some other things in there, or you changed the team. You've done a reorg. Whatever it is you've done, you've broken the rules. And when that happens, things get out of kilter, and it's hard to get them back because you need to make a conscious effort to say, "No, we can't do this." The only way this works is when we follow all the rules. And there's a lot of times when somebody is going to not want to follow those rules. And it's hard to explain why those rules are important. It involves math and [laughter] discussion, yeah.

WILL: I mean, I suppose so, and I agree with what you're saying. But, I mean, like, I'm a lot more interested in sort of the fundamental psychology of, like, the planning. And the reason I'm sort of most interested in it is because if we were all sort of, like, very rational, logical actors, even on an upper management leadership level, the better you plan the stuff, the more efficient things run. And, honestly, I mean, like, really spending time in investing up front in your product planning and development that will make you more money.

We do better when we're very, very, very deliberate about when, where, why, and how we're taking on projects. That lack of discipline, like, it not only hurts, like, sort of the engineering efficiency, but, in my view, it's extremely hazardous in terms of, like, business impacts.

JUSTIN: So, Will, I mean, you're going more towards the clearly defined problems and everything, but, classically, that's called, like, the waterfall solution, which is a bad word these days, right? So, where do you draw the line between planning everything and planning just enough?

MATT: But planning being the operative word here, right? You can be extremely agile. You know, I've worked where we were extreme programming weekly sprints. We planned every single week. We had our work ready every Monday for that week. You can break problems into smaller problems and then just plan those smaller problems. But you do have to have a really good idea of what the primary problem is, right? And the thing you're trying to solve for and your ultimate goal. But once you understand that, you can break it into small problems, and then you plan pieces at a time, and you can be super effective.

WILL: I do agree that, like, waterfall is a dirty word, and it should stay one, frankly. Because where I see things go, like, most badly, is shipping these grand visions versus tiny, little, agile, incremental pieces. Maybe analogize it with the sort of, like, the Apple iOS grand reveal of the new version of their operating system. Versus maybe, like, the Google Android, like, things are coming out all the time. Like, what's even the current version?

JUSTIN: [laughs]

WILL: Is it Oreo, or Nougat, or Zeta [laughter], or whatever? I don't even know. Like --

MATT: But do you care?

WILL: Come on, who cares? Stuff's coming out just helter skelter pelmel, all the time, right? Versus, like, this, like, this grand, like, unveiling that happens once a year. It's cool that Apple can pull that off, you know, but, like, --

MIKE: Will, do you think that, internally, they're actually having a grand pull-off? Or do you think that they're doing the same thing that Android is doing and they're just tidying it behind the scenes doing these incremental builds, and when they finally have it ready, then they push it out? I think it's the latter. Like, I think that --

JUSTIN: Yeah, I think it's the latter, too.

MATT: Yeah, I mean, when you're incremental, you're mitigating risk.

MIKE: Exactly. There's huge risk when you're doing it. So, there's aerospace example, and it's easy because it's big, expensive projects people know about. There's the Space Launch System recently launched successfully. After years, you know, they finally have a launch. I don't think they...[laughter] they, like, don't even launch a real payload, and it costs, like, two billion dollars, or something, just to launch, not the development but, like, the launch costs that. And then, I read the other day that one of the SpaceX rockets, the Falcon 9, did its 20th launch, reused, like, the same rocket, launched its 20th time.

[laughter]

And, you know, I'm sure it costs some millions, but orders of magnitude difference in price. And SpaceX has blown a lot more things up, but they did it on purpose because they said, "We're going to do this incrementally, and we're going to start small. And we're going to iterate." And now they're just the steamroller that is launching more rockets than, like, all of the nation states put together.

MATT: They're miles ahead.

JUSTIN: Yeah, if only they weren't led by a crazy guy, but anyway.

WILL: Well, you have to be crazy.

MATT: That's why they are so successful [laughs].

WILL: Because, I mean, like, there's...I --

JUSTIN: Okay. So, at the start, he was crazy, but he was focused on his stuff. Now he's crazy and focused on other things; at least, that's what it seems like.

WILL: A lot of those guys are difficult people. Steve Jobs, God rest his soul, [inaudible 20:24], you know what I mean, had his halo shined considerably in his death. And, at the very least, he was disciplined enough to keep his mouth shut and most of his sociopathic behavior behind closed doors. But, I mean, like, the stories abound of that guy doing terrible stuff, and he never really stopped. His PR team got better, but, like, he was a jerk till the day he died. God bless him, you know?

MIKE: One thing that they have in common, though, is that they honor the engineering process. And I'm not going to even touch Elon Musk as a person, right? That's a topic I'm just not going to bring up here. But he came from the software world and brought that engineering process and cultivated it among engineers.

MATT: That is very true.

MIKE: It really doesn't matter about him, right? What he brought was the process. He brought the agile engineering process with him from software world and just applied it to aerospace, and it works. If you do it, it works. If you don't do it, a recurring theme here, it doesn't work.

WILL: I think it is a validation of that process, you know what I mean, in a lot of different, you know what I mean? You're moving that into other industries and avenues, provided that it is permissible to explode the rockets, and there's nobody [laughter] on them.

MATT: And we all agree, waterfall, bad word. But I would argue a waterfall process is better than no process at all.

MIKE: Sure. So, I think a lot of people they think, Agile, hey, we don't have to plan anymore.

JUSTIN: [laughs].

WILL: You can't be farther from the truth.

MIKE: Exactly. All you're doing is you're tightening your planning loop, right? You're tightening your feedback loop. And instead of doing all of your documentation upfront when you know the least, you are choosing to actively plan, and document, and iterate the whole time. So, instead of doing it once, you never stop doing it. And if you think that being agile means, oh, wait, you know, I'm just not going to plan at all; we'll just wing it, then you're missing the whole idea, which is that these interactions, the communication is hard, and that interaction is critical. And you need to have interaction with your hardware, with your stakeholders, with your software. You know, you're testing it. You are evaluating it. You're improving it continuously rather than once.

And it's actually more planning work in the end because you're doing it along a line. But because it's distributed, it's more efficient because you're doing it when you have the most information, which is after you've already done some of the iterations already.

MATT: And it becomes easier with experience because then you have that association. If you've been doing this for a while, which all of us have, there's very few problems that we have never seen before. They just are in a little different context. But, generally, it's the same problems over and over, and we know how to solve them. I think --

MIKE: You're taking text out of the database and transforming it and showing it to somebody?

MATT: Exactly [laughter]. It's only data, right [laughter]? But, you know, there are new things coming out, AI, for instance. GenAI is new to a lot of people, and we're going to learn that as well. And it may be a little harder to plan those technologies because we don't have the experience yet. So, we may not be as accurate with our estimates on those types of projects. But, for the most part, most of the work we're doing, we have done before in some form. And if you break it down, isn't any different than other problems we've already solved.

JUSTIN: So, I do want to add a caveat to that. Well, not a caveat, just, like, an addendum. It's interesting the organizational changes that are happening within Acima, right? Well, Upbound, right? And last year was, like, a record year for Acima and RAC. And this year, we've got a big organizational change where teams are shifting. Directors are shifting. Product owners are shifting. There's a lot of moving parts here. And I'm afraid that they're going to expect the same record level production from an engineering standpoint that they had last year this year because it's, like, oh, we're reorganizing in a more efficient manner. That means we're going to do even better. And it's like, maybe long term, but that short term is really going to suck.

MATT: Well, and maybe it's up to us, right? Well, it is up to us. I really think, just highlighting what I just said, if we look at it as these aren't different problems; they're the same problems, whether I was on Team A or Team B, I'm still facing the same types of problems today that I was yesterday, and we approach it in that manner, I think we can be absolutely as efficient. Change is hard for people. When you are shifting so many things around all at one time, it's up to us to kind of get our bearings and have that compass to remind us that we're still doing the same thing.

MIKE: Well, there's a couple of things that need to happen. First of all, we should work very hard to preserve team boundaries where we can. Like, let's not disrupt high-functioning teams. Second, leadership needs to be focused on using this as intended to improve lines of communication and not just change them. You know [laughs], let's just shuffle things around, and it looks better on a whiteboard. That's just going to slow us down.

If we see that there were...you know, in a large organization, there are difficulties with communication; of course, there are. And if we reorganize to try to improve that while allowing the engineers to keep doing their work with minimally disrupting the teams, then we could actually become more effective. But that is a, you know, to Matt's point, it's up to us. Like there is a decision point here, and we have to be aware of that destination. We have a choice, right? We make things better, or we make things worse.

WILL: So, I mean, in regards to, like, sort of, like, these major, like, company-wide reorganizations, how do you account for and even measure sort of, like, impacts of the reorganization, right? Like, impacts on sort of, like, planning bandwidth, right? I mean, you've got people who are, like, coming up with a plan for the next sprint, writing the tickets, like, measuring the stuff out, right? And so, like, they have a certain amount of planning that they can do, right? And that's affected by their familiarity with, like, their operating environment, right?

Like, you bring a manager into a brand new team. They haven't touched any of the code. They have, you know, a certain amount of planning bandwidth, right? And, like, that's going to be constrained by them getting up on the stuff, right? You have a new team, right? Okay, you know, like, you don't want to disrupt high-performing teams. But, honestly, like, maybe you do want to give your A players your biggest impact priority work, right?

So, you move those people in, and they have to acclimate, right? How do you account for impacts of reorganization? Because this is something that I have seen as well. You know, the large retailer that I work for they love their Q1 reorg. They love, you know, rearranging the furniture. But they're going to have a cost, and I don't know that the cost is accounted for organizationally.

MIKE: And because it's hard to measure, it doesn't get measured, and --

WILL: If it doesn't get measured, then they can't plan for it, you know what I mean? Upper management, you know, those executives that are sort of, like, rearranging the furniture they don't have that. Do they have that in their heads? Do they have it in their mind? There's a cost. We've all felt it.

JUSTIN: I think you treat it as you would, you know, you're onboarding any new member to your team. This is something that happens all the time with engineering teams. People leave; people come in. And you have to get to know this person and, you know, put them through the same onboarding and really get to know their capabilities and stuff. And so, I think your estimates are going to suffer a little bit until you've had a couple of sprints with them. You treat that as an unknown, and make sure you communicate that up. That's kind of what you would do at the team level.

And then, I would think that the folks in the engineering leadership, if they make changes at team levels like that, they just need to get those lines of communication going such that you can set expectations and really understand, "Hey, we just on-boarded two new people, you know, our estimates are going to suffer for the next sprint or two. And then, we'll be able to give a more accurate estimate after that."

MIKE: And there's that bigger question. What metric can you use to measure, aggregated across teams, across an organization? What metric can you use to capture effectiveness as you're doing a reorg? Because we know every metric captures a slice of something and has its own biases. So, what would be an effective metric? Does anybody have an answer to this? What's an effective metric to see how...when you do a reorg, what do you use to measure it? Did this work?

MATT: Velocity over time. I mean --

MIKE: The tricky thing is velocity is team-specific. You rearrange all your teams, velocity changes. If you measure that, then everybody's going to say, "Oh yeah, we're just going to have..." this is five points. That's not three. That's five. Like, you're going to bias it toward higher story points, and velocity ceases to become effective as a number. So, it's hard to measure that precisely. And business initiatives aren't exactly equal to each other. So, you can't necessarily say, "Oh yeah, we've done five initiatives per year [laughs]." It's a tricky problem.

JUSTIN: Yeah, I think if you had an answer to that, you could write your software engineering estimation book, Mike, and retire on that, so...

[laughter]

MIKE: But it's not a new problem. It's not a new problem. Everybody's dealing with it. And then, we fall into these gut, you know, we don't have an answer, so we just kind of look the other way and go with our gut and don't measure it very well. And then, maybe you have your annual reorg [chuckles] that does more harm than good.

WILL: Well, I mean, I don't know. I mean, I feel like you ought to have, internally, a general rule of thumb for, like, okay, if I have a new hire, this is what I expect the onboarding to look like. At week 1, I'll get X. At week 5, I'll get Y. At week 10, I'll get Z, right? Like, you'll have an arc of their productivity, at their level, right? Like, you know a junior will onboard. You know a senior will onboard, you know, et cetera.

And then, like, I think you should probably also have, like, a rough idea of what an internal transfer is going to look like, right? Like, if I move from, like, you know, Team A to Team B, it's not going to look like a new hire, you hope. Although, depending on the team, you know what I mean? Like, I know I'm familiar enough with Acima that there's some teams that you don't just, like, walk into [laughter].

MIKE: One does not simply walk [laughter].

WILL: One does not simply walk into a --

MIKE: An underwriting -- [laughs]

WILL: Yeah, underwriting. Yeah, underwriting.

MATT: Or the [inaudible 31:48] service.

JUSTIN: [laughs]

WILL: Oh no, that one is fine. Like --

MATT: Oh yeah, you worked on that team.

WILL: Yeah, the Haskell. The Haskell, the Haskell folks, you know what I mean? Like e-comm.

MATT: Oh, Lambda.

JUSTIN: Lambda. Yeah.

WILL: That one's there. You know, there's stuff you don't just walk into. So, like, you have that, right? Then you say, okay, reorg. Then I have, like, X number of, like, people moving teams, and probably the same for managers, although, I mean, I will say that, like, I don't think we think enough about management sort of, like, the bandwidth, right? You know, like, we spend a lot of time thinking about, like, okay, coordinating engineers, right? But, like, management bandwidth in terms of planning bandwidth I think it's, you know what I mean, I think it's a little bit unsung. We have a general feeling for, you know, how many engineers a manager can be sort of overseeing, but beyond that, it gets pretty mushy, wouldn't you say? Or am I wrong?

MATT: No, I think about that every day. I'm new to the management team over here on the Upbound side. And it's true that bandwidth gets spread a little thin, right? And that's part of being able to organize your team, having the right people to lead those teams. And everyone has been on...well, not everyone. But a lot of us have been on very large teams versus, like, 3-to-5-member team and seeing the difference in efficiencies, right? And how those teams are managed.

And, ideally, smaller teams, in my opinion, are better. So, you need to do that with management as well and whether there's levels. We have a new concept here at our company of delivery managers. And they've kind of taken over a lot of that responsibility to kind of help spread that bandwidth for planning and let people focus where they should be focused. That's the way think about it.

MIKE: So, we've wandered far and wide here a bit [laughter] as to...we started, like, well, yeah, we know what it takes to run a team well. And then, we started getting into all the complexities when you increase in scale, you know. You go to the next tier, and you've got managers and managers over managers over managers, and it can fall apart. Because, at any place in this hierarchy, if somebody isn't paying attention to the rules, then it can fall apart.

It seems to say something about what you do to run an effective engineering organization is that everybody has to be aware of what the constraints are, be upfront about them, and be willing to stick to some simple rules, right? The planning happens. It happens every time. Every time you can. You don't give it a short shrift, you know; you keep on it. And you don't skip it at any tier.

You know, up at the top level of management, you know, they have to make some estimates based on incomplete information. That's something we all have to do. But they do everything they can to follow the best process they can: gather the information, make the best estimates they can with the information. It has to be all the way down. And then, you have a really well-run organization, right?

WILL: Mm-hmm. Is there [inaudible 35:06] where, like, people who are just sort of guessing, because the company...and what I mean by that it's like, sort of if you say, like, okay, I've got a corporate, let's call it, like, five-year plan, right? Which is not a crazy thing, like, you don't get a business corporate, you know, strategy kind of a thing, right? But on a software, you know, estimation sort of a thing, that's [laughs] a joke. That's comedy, you know what I mean? Like a five-year, like, I can't plan that, you know what I'm saying? And so, it does need to be done on some kind of level, right? Where does that transition happen, and how?

MIKE: There's a really good thing that we've done for a few years now that's spreading throughout the organization, which is that we recognize explicitly that there are tiers of planning. You can map these to a couple of really useful things. You can map them to groups of employees, or you can map them to Jira issue types. And it works either way. So, tier one, you know, planning level one is what the executives do, and it's associated with an initiative in your project management software.

Tier two, planning level two, is associated with milestones within that initiative. I'm going to expand to Columbia, so I've got an initiative. And then, when it comes to milestones, well, maybe you need to negotiate with the government contracts, you know, maybe you need to build some sort of new tax integration. You know, figure out whatever those pieces are. You're developing the milestones, right? But now you have something that's a bigger shape.

And then, there's a tier-three level of planning. Now you're talking with your team leads. Those team leads are breaking this, and that's associated with, like, an epic in your project management software, where now you're coordinating between teams. You know which teams are going to be working on it and where those boundaries are, you know, who's going to be working on what.

And then, you have a tier four, you know, planning level four, which is within the team. That's your refinement, your sprint planning, and so on, where now I know what my team needs to do, right? So that I know what those boundaries are. And I have the epic, so I know what the overall story is I need for my team. Now, I need to break that down into the two and three-pointed stories.

And that works really well because, at every level, you're doing planning based on the information that you have. And you recognize that the accuracy of that planning goes up dramatically as you go from levels one to four. By the time you've done a level four planning and you've completely pointed something out, you know, you can say pretty well when that's going to land.

When you're just at the initiative level, you're like, I'm going to expand into Columbia, right? You really don't know. You really don't know very much, but you know something, right? You know something. You know that that is a business priority that you can shuffle around with other business priorities and say, "Well, this is something I want to do first. And I can start to gather information on it.

WILL: Well, okay, sure. But, I mean, like, so, like, you know, a company like the size of Upbound, right? It's a public traded company. They need to have, like, sort of, like, forward-looking projections about, like, when is this stuff going to land, you know?

MIKE: Yes.

WILL: And what are the business impacts and, like, because, like, because they're beholden to their shareholders. Their shareholders want regularly scheduled updates about, like, okay, you've got an initiative. When am I going to get the initiative? And, I mean, like, is the best we can do on these initiatives just sort of, like, we've done it before and this is about how long it took, you know what I mean? Like, where it's like, okay, you know, I did this, you know, when I expanded into Cincinnati. Like, this is what I got and how. And if I'm doing Columbia, then it's going to be, you know what I mean, ten times harder than doing Cincinnati, right? Like, you got a whole other regulatory language business entity, et cetera. Like, is that the best you could do?

MIKE: Well, I think you can even put time frames on here. So, I'll say, also, that this map roughly to divisions of time. Annual plan, initiative. Quarterly plan; it may be milestones, right?

WILL: Right.

MIKE: And then, you can slot your epics, perhaps, into months. And by the time you get to your stories, you can get that down to about weeks. It kind of maps loosely. Again, this maps to some of these divisions. The precision you get gets more specific. And, generally, at the executive level, you're usually communicating at an annual or a quarterly level. You say, "Well, this year, we intend to expand into Columbia. By quarter three, we expect to be operational at some stores," and so on. You know, you speak in those kinds of broader periods, right? Your resolution is a lot less fine-grained, and that's okay. That's actually expected.

WILL: How do you enforce discipline? I mean, I know, like, on the tier 4 level, right? You're sitting like, okay, I want two or 3-point tickets, right? Like, I want two or 3-point tickets. I want, you know, that's all I want. And, like, if you're doing five or, God help you, an 8 [laughter], you know, things are probably going to go bad, right? And I see how that fundamental process it's got to scale up. But it's a lot less clear how you enforce it, you know, as you go from tier 4 to tier 3 in planning or tier 3 to tier 2 planning or tier 2, God help you, to tier 1 planning. How do you keep a lid on that? Well, because it matters. Like being off by a quarter, that's a big problem.

MIKE: Right, right.

MATT: It is a big problem.

WILL: And for a tier 1, being off by a quarter is actually doing a pretty good job, in my opinion.

MIKE: I agree.

WILL: You're delivering an initiative, and you're only off by a quarter? Not bad, y'all. Not bad.

MIKE: [laughs]

WILL: You know? Like...

MIKE: So, one thing I think you can do...and we talked about the rules down at tier 4, right? That's what we think a lot about. But there's no reason you can't have documented rules at every tier. This is the output. This is the data that went into it. And these are the requirements. Like, you don't say that this initiative is going to get done this year, and unless you have done this, this, and this [laughs], being off by a quarter, that's...it's going to be hard to be within a quarter resolution.

But you can do the things, the executive things [laughs] that executives need to do: gathering information, delegating that out to people that you trust to go and gather the necessary information, and establish a culture that demands honesty and rewards it, rather than one that rewards saying, "Yes." I mean, there's things that can be done there. And now we're getting into business management.

To bring it back to the beginning, we've all got some landmarks we're trying to move our way towards. Hopefully, this has provided some food for thought, how to better figure out when we're going to get there.

WILL: Cool. Thanks, y'all.

MATT: Yes. Thank you, everyone.

The panel delves into the challenges and strategies of onboarding new employees to discuss the importance of communication, mentorship, and understanding context. Mike shares a story from his construction days in New Orleans, where he learned the value of being paired with an experienced team member. This buddy system helped him overcome communication barriers and adapt more quickly to his new environment.

Effective onboarding involves more than just providing documentation. While documentation is valuable, it must be up-to-date and supplemented with hands-on guidance and practical experience. The panel advocates for giving new employees context about their tasks and the organization. This can include having them use the software as end-users or working directly with customers to understand the system and its impact better. Regular check-ins and skills clinics are also recommended to help employees stay updated and improve their skills.

Additionally, the podcast highlights the importance of continuous learning and mentorship for all employees, not just new hires. Regular mentor-mentee interactions and collaborative updates to documentation ensure that onboarding remains practical and relevant. The panelists also note the pitfalls to avoid, such as not following through on commitments and assigning unsuitable onboarding buddies.

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today [laughs]. With me right now, we've got Justin, Dave, and Matt. We're going to talk about onboarding new employees. It's a tricky problem. I don't know that anybody has it perfect, and I don't think we have it perfect. So, we're going to talk about some ideas, some things that we've seen go well and maybe not so well. I've been remembering onboarding in days past.

I tell a story about the late '90s. I was living in New Orleans doing construction work. We would go in and remodel office space. So, sometimes, we'd tear out everything and then rebuild it. You know, we had teams all over. The company I was working for had teams all over the city doing this. And the team I started on, I started just as a laborer. I was not doing anything fancy to begin with.

But what I remember is a couple of things. The day I started, the foreman I was working with set me up with a partner. His name was also Mike [laughs], and they called him Big Mike. And they sometimes would compare him to George Foreman because they looked similar. He was a big guy. He previously worked out on the oil rigs out in the Gulf of Mexico and a big, strong guy. And we got to be fast friends pretty quickly.

However, the day I started, I had not been living in Louisiana for all that long, and he grew up way out in the Bayou somewhere. And when he started talking to me, I picked up about one word out of three of what he was saying [laughs], maybe. And it was that way for a few days. You know, maybe started out with one word out of three. After a while, I got to two out of three. There's a mix of accents there. If you've got a Southern accent and a heavy French accent...

DAVE: Mm-hmm. The Creole.

MIKE: Plus, there's some African-influenced accent as well, and you mix those all together. And I'm a White guy from farther North [laughs]; I did not pick up very quickly. But we got to be friends. And he was patient [laughs]. He was patient with me. And, actually, after a few months, we'd hung out outside of work. We were actually quite close friends. We've lost touch over the years, but I'd love to get back in touch with him. After the storms they had down there, that drove out a lot of the population. He's probably moved, and I don't know where he's gone.

[laughter]

But there are things about this story that were relevant. First of all, the foreman did a great thing. When he brought me in, he paired me with somebody so that I didn't have to know everything. I just had to be able to ask Mike, and [laughs] we could get some stuff done. The second thing is communication is hard [laughs]. We were speaking the same language, and we still couldn't understand each other. He could probably understand me. I was the one who was struggling [laughs].

But, you know, it took us a while to get things right, and we had to work on it. It had to be an active, active effort, especially on my part, to learn to understand. And working on it worked, though. You know, over time, we got to work really well together. We were frequently called on to get a lot of stuff done because we had more effective partnerships [laughs]. We both had enough carpentry skills that we could get a lot done besides just being laborers.

I lay this out, and it's outside of software engineering. I think a lot of times, it helps to think about something outside of software engineering to lay some context. We bring people in, and we let them loose. And you've got to figure stuff out. You may be talking with somebody [laughs] who has a different background, and he's going to use the same words and mean different things, and you're in an unfamiliar project. There's a lot that has to happen for you to onboard successfully.

And there's a few things that I think make a huge difference to be successful and one of them is that regular communication with somebody. And another is, you know, well, getting that communication, but having a buddy, right? Having that lifeline, I think, are some of the key aspects I've seen be effective in onboarding. I think there's a lot else as well, but there's my story to launch this off. And what do you all think?

MATT: Well, you hit on communication, and while I think, you know, trying to dissect a Creole Cajun dialect is tough, [laughter] there's also communication differences in the same language and dialect. For instance, the four of us all have very different communication styles, and I think being able to pick up on that and how people communicate, and their intent is a very important aspect of that.

JUSTIN: I like how they assigned somebody to you who was already part of the thing. They had, you know, knowledge that they've built up as kind of a veteran working there. And they were able to answer your questions, and you were able to be much more effective because you had somebody to ask questions to, and they had to answer right away. You weren't, like, left alone. "Go do your thing [laughs]."

Yeah, I've had experiences with both. And a lot of how you communicate culture to a new team member is, you know, whoever is introducing you, and whoever your onboarding buddy is, whoever you have the most exposure to, that is how they view the whole company almost. And if you don't have somebody, you know, doing that correctly, you don't have a good communication in not just, you know, knowledge, but also culture, and friendships, and there's a whole bunch of stuff there.

DAVE: But, for me, it's, like, context. There's so much out-of-band stuff you can pick up by, like, a personal interaction, where I was talking with Adam a while back, and I'm like, "I want to pair with you on this problem." And he had been thinking through this problem. And he's like, "Well, I know it's not the login. I know it's not this. I know it's not this conditioning thing. I know it's not this sequence in the system." And I'm like, "Adam, I need you to take me with you to hunt badgers." And he's like, "What do you mean?" And I said, "I don't need you to show me how to shoot the badger once you've found it. I need you to take me through the hills and show me every place you check for badgers. That's what I need."

Because he was thinking, I don't want to show you the login system because that's not going to be the problem, and I don't want to show you the pipeline because that's not...I'm like, no, show me that you checked the login, right? Like, literally, the gift in the box was not what I wanted. I wanted the box, and the wrapper, and [laughter]...right? And all the trappings that went with it.

MIKE: That's interesting. You wanted the box.

DAVE: Mm-hmm.

MIKE: Again, this is a little bit outside of engineering, but it's related to that. You know, you think about math class. You can memorize the rules for solving a problem. And as soon as you forget any one of those or you make one little mistake, you're just completely lost. But if you can have somebody who takes the time to help you understand why you're doing what you're doing...and it takes a lot longer. It absolutely takes a lot longer, and in a classroom setting, that's hard. You know, it's a gifted teacher who can bring everybody to that level, and that's, you know, a difficult thing to do.

But if you can do it, if you can help somebody get deep mastery of why those things are there, you know, teach the intuition and understanding, if somebody gets lost, they can backtrack. You know, you can figure out what you're doing or even come up with your own way of doing things and be creative about it. It completely changes your perception of that problem-solving. We're in a problem-solving industry. That same idea is directly applicable. If you could show somebody this is what you do and, you know, tell them to just do that, it's not very useful at all.

MATT: The journey is just as important as the destination.

JUSTIN: I think, to kind of summarize, if we were to put, like, points on a board about, you know, how do we onboard someone, so, right at the start is, like, choose an onboarding buddy who's going to be very helpful for this person in whatever. And I think that's part of leadership's responsibility, you know, whoever is in charge of onboarding this person, whether it's the tech lead, or the director, or whoever the team lead. So, they need to make that decision of who's going to be this person's onboarding buddy. And then recognize that that's going to take time, and that's real work. And you can't, like, you know, expect them to do the same level of work as they did before, perhaps, because they are, you know, if they're going to do it right, it's going to take time.

MATT: And based on the information we have about these new hires, who is going to be the best fit, not only in experience, but personality type, communication type, those things?

MIKE: That investment can go a long way. If you do your onboarding poorly, then you're going to have somebody who doesn't know what they're doing. And somebody can sit around for months and get very little done. They'd be busy, busy all of that time, and not get very much done because they don't know what they're doing.

DAVE: I can't tell if I feel seen or attacked there, Mike.

[laughter]

MATT: But that's important, right? Encouraging people to ask questions and not just sit and spin their gears, ensuring that they understand the things you're going through and leading them. You know, I could sit and write code for hours and have someone paired with me. But if I'm not ensuring that they're understanding what I'm doing, communicating my thought process of why I'm doing it, I don't feel like I'm really helping them. And they're probably not learning a whole lot during that time, right?

MIKE: One thing about our brains is they don't tend to stay connected to something they're not actively engaged with. You show me somebody who can watch somebody do a detailed technical task for a long period of time, and [chuckles] I'll show you something imaginary [laughter]. It's not the way our brains work.

DAVE: There's a really great experiment that demonstrates this, which is that if you just start reciting numbers at somebody and say, "Memorize this. You don't get to write it down. Just memorize it," we tend to be able to hold about seven numbers. Like, some people are down to five. Some people, as many as nine...can hold this in their working memory. And what we miss is that if you can tie these things together with a story...the people that memorize Pi out to, like, a million places, they have a story that ties all the numbers together, and where you are in the numbers, where you are in the story, and, to them, it's just one story.

And that is the essence of, like, "Oh yeah, you know, sign into this. The database is over here. The server is over there." And I'm like, okay, that's three things that aren't connected [laughs], right? And there's 23 more to go. You better start making a story out of this for me [laughs].

MIKE: You know, that says something, right? Without that story. And it also says something about communication. You know, a lot of the best communicators will dip into stories that don't directly connect to the material they're talking about. But then they'll make the connection like, oh, that's great. That was entertaining. And it's entertaining, sure, but it's a lot more than that. Without that story, you would never have learned the idea in the first place.

It may superficially look like they're just trying to make things fun, but that's really not what...making things fun is not what really good communicators do. What really good communicators do is connect you with the material they're talking with. That could be fun, and it's going to be engaging, but the goal is different. They're not trying to force something to be entertaining. Let's make math fun. No, they actually see the fun in it. They see the story, and they want to share it.

DAVE: I had a co-worker...oh gosh, this was around Y2K. This was a while back. And we were struggling with the inheritance rules. I had some co-workers that didn't understand, like, protected versus private versus public versus, you know, friend was the thing. I blurted out the rules of C++ inheritance in terms of intimate encounters, right? Your children cannot touch your privates. Your friends can touch your privates, right?

I got sent to HR, which is fine; I deserved it. But a week later, I'm typing away at two cubicles over. I hear, "Damn it, Dave!" [laughter] And I'm like, "What?" He's like, "I know the difference between protected and private, you jerk!" and just kept typing. And I'm like, yep, that's in your brain rent-free for the rest of your life [laughs]. I've gotten better at little, you know, more tasteful stories since then. You're all going to remember it now, too.

[laughter]

MIKE: So, reinforcing the point about stories, having to have that connection and the importance of context, the importance of context.

JUSTIN: Related to that story there and onboarding new people, the best places I've seen that do onboarding is where you come in, and you're actually onboarded as a user of whatever system that you are going to be doing, and so you know how the users are using it. And you can see, you know, oh, okay, here I'm a customer service rep, and here's me logging in, and here's me helping a customer, looking up the customer name. And then, here's me looking up their lease number, and their payments, and their history, and things like that. And I can, you know, see there. And here's the customer service rep complaining about something.

And so, you go in, and you spend a day or two just immersing yourself in that system. And then, you come back, and you have a story in your head of how this thing actually works from a front end, and then you can dive into, oh, okay, this is how the app works. And this is what is actually happening, and this is where the data is being stored. And I understand what's going on here. And here's this third party that we're calling, and things like that. So, that right there is key, for me, to understanding, you know, see how our users are using it, and then I can see behind the scenes.

DAVE: There's a fundamental problem in computer science, and I call it the A versus B problem, which is where this system over here, you got A matching up against A, and it works. System B matches up against B, and it works. And you get a bug that comes in. Your trouble ticket is, I did this. I expected A, but I got B. And you're like, okay, should you be getting A, or should you be expecting to get B? Which is the right answer, right? It's like, oh, plus four on the zip code. This piece over here only has room for five digits in the printing label. Don't ever put a plus-four on there. That's a bug, right? Anybody else working with postal codes is going to be like, no, you always want the plus four. Why would you want to throw away data?

One of these is right. But if they don't match, you have to know which one of the two is correct, and it's the context. Seeing a user use it is what, you know, it's like, if this is checked or unchecked, if it's checked, we always do the thing. And if it's unchecked, does that mean we can do the thing, or does it mean that we never do the thing, right? There's that context to it.

MIKE: You know, you talked about sitting in the place of customer service and using the software. I'd take that even further. Some of the most successful onboarding I've seen is when the new person is actively assigned to work with a user of the software. It's not always possible. It depends on the context. But you can have them sit with customer support and help them field calls for a couple of days, or, you know, go out in the field and use the software with the users. You know, actually using that software with real users provides a depth of understanding that I think is almost impossible to get otherwise.

DAVE: I can think of a time when I have been sent to work with the end users that...In 2006, I worked on a system for a year, and I finally got to go sit with the users. And this has happened every time I've sat with users. And I was surprised after a year that this still happened to me. I walked back into the bullpen with my co-workers the next day, and I said, "Guys, we're building the wrong thing." Just half a day with the people using it, and you realize we are going around welding shut every door that you need to access. The way that we're trying to get you in and out of the system is like crawling up and down the chimney. We have built this completely wrong. We need to put a lock on this door and make it swing easily instead of messing up your life, yeah.

MIKE: So, we've talked about a few things: having a buddy [laughs], taking the time to get context, assigning somebody to actually use the software, and, if possible, pairing them with real users. Saying that all of those are important for onboarding. How valuable do y'all think documentation is?

JUSTIN: I wouldn't want to dive into documentation to start off with. I'd want to go to those other things first. Because documentation is great. I love documentation, but without the context, I don't know if that documentation is dated. I don't know if it's, you know, even applicable for the system I'm on. And so, without that context or without asking my buddy, it's almost worse than having no documentation [laughs]. You could be looking, like I said, at outdated stuff, or you could be doing stuff like that.

But later on, when I'm more experienced in the system, I love documentation because I have the context there, and I'm just looking for details, or maybe some other stuff when I already, you know, have a good, solid base of everything. So, yes, documentation, but I don't want to push a bunch of documentation at somebody who just started yesterday and say, "Read all this [laughs]," so...

DAVE: If it's a part of the system that is very static, very stable, and we've been using it for five years, then yeah, hand you a piece of documentation that says, "Start here. When you're done, you will have this. Here's the steps." And that's kind of fun sometimes where you hand that to them and say, "We're trying to improve this documentation. Please go through this. And as soon as you get stuck or lost, come find me, and we'll fix the documentation." That's a lot of fun.

And, again, there's a trick to that, which is now they're engaged looking for the problems in the documentation. And now they're paying attention to the system. Instead of "Memorize these seven things and plug them in," it's like, "Okay, I'm in step three. I don't have access to Postgres. What should I do?" right? "Oh, let me add that step."

MIKE: You know, I think we all wished we had perfect documentation. But documentation is a lot of work, both to create initially and to keep up to date. And I'm not complaining about documentation. You know, any documentation is almost necessarily out of date unless you keep it in lockstep with your changes. Unless any changes to documentation go out exactly with any changes to your software, it's going to be out of date kind of the moment it's written.

So, it feels like there has to be another layer, either you put a huge amount of effort into your documentation, which I think is unpalatable for a lot of organizations because the work is hard; it's expensive, or you have some system outside of the documentation to communicate some of that knowledge. And that's hard. That's a tricky balancing act. Because there's probably things in the documentation that somebody is going to forget to tell you.

So, to ask another question, I asked about documentation: how do you avoid those gaps? And we've talked about the value of partnering with somebody. What happens if you never happen to walk by that hole in the ground? You say, "Don't step in that." [laughter] How do you avoid that? What are some things you can do to avoid some of those gaps in knowledge?

JUSTIN: Document. Oh, wait [laughter]. Find it out and document it. I like what you said, though, when you have your mentor there, and you have your new onboarding checklist, you know, to get permissions for all the things and to get all your environment set up. If you're both doing it together and then you update anything that's missing or, you know, common missteps on that onboarding document or series of onboarding documents, usually, that's really helpful, and it kind of updates itself as you onboard new people. And that way, you aren't dealing with an onboarding document that was last updated in 2021.

So, I don't know if that answers your question exactly, but that, you know, goes back to having your mentor and you going through it together and updating anything that you find is missing.

MATT: And things change, right? Something that was relevant even as recent as, let's use our company for an example, five months ago isn't relevant today, right? Because we're doing a lot of things differently. So, if we can make these things a living document and adjust as we go, and maybe we find a better way to do something that we have documented or a different way that's more efficient, then we can update it and keep that document living. And it provides a lot of value for the next person coming through the line.

MIKE: So, we've got another item in our checklist here [laughs]. Have the new person work with their mentor to update the documentation as they go so that it does become that living document as they're using it. One thing...I still think that sometimes you miss those things, the things that you didn't think to document that everybody just knows. One thing that I've seen among people who've been really successful, a lot of times, they'll say, "The time I learned the most was when I didn't have anybody else to help me, and I had to do it on my own."

And [chuckles] maybe they're at a job where they're the only engineer, or they took over DevOps. You know, whatever the case may be, a lot of times it's when [chuckles] somebody had to take care of things themselves, and there's nobody else to go to. You have to figure it out. And, boy, that burns [inaudible 22:22] your brain.

MATT: Trial by fire.

MIKE: But, you know, we don't want to lay everybody off once a month just to give somebody a chance to do that. That's not a good way [laughs] to run a company. So, we're not going to do that. So, I think that setting up some sort of environment that gives people an opportunity to learn and more of a sandbox is really useful. One thing I've seen for security training that was extremely helpful is we rotated weeks. And every week, somebody would add an insecurity to the application from the OWASP Top 10. And then, the rest of the team, knowing that there was that kind of vulnerability in the app, would look for it and try to exploit it.

JUSTIN: In production?

MIKE: No.

MATT: No.

[laughter]

JUSTIN: Okay.

MATT: No.

[laughter]

MIKE: We built a...We built a...

DAVE: So, they're playing. They're just not playing for keeps.

JUSTIN: Yeah. I was like --

MATT: I believe, Justin, this goes back to when you first came aboard. I remember I was doing the OWASP Top 10. Prior to that, the trainings were an application that Mike actually had hosted on his AWS instance called Insecurity. And then, we moved over to the Juice Shop when I took over. But prior to the Juice Shop, it was this Insecurity app that we did exercises on.

JUSTIN: Oh, that's really cool.

MIKE: The thing about that is when you're introducing the vulnerability, you had to understand it well enough to break the code [inaudible 23:54] to it. And sometimes that's harder than it sounds. Like, the frame will fight you [laughs]. And it's hard to get that in there. And so, you really learn to understand the guardrails and how you can break through them. And then, from the flip side, it's fun for other people to try to figure out what it is you've broken. The teams I work with doing that learn...They universally said they learned more about security working through those exercises than they did in years of trainings because that hands-on makes a huge difference.

DAVE: It makes it alive.

MATT: Yeah, I mean, try adding the SQL injection to Ruby on Rails ActiveRecord. It's not easy.

[laughter]

MIKE: It's not. It's possible, but very difficult [laughs] [inaudible 24:45]

MATT: Yeah, you have to try to make that vulnerability.

DAVE: And you can tell when somebody's up against that because they'll start saying oddly specific things like, "How do I serialize a lambda to the database?" [laughter] Yeah, yeah, whatever you need that for, stop it.

[laughter]

MATT: But, you know, something that maybe I want to touch on just for a second, we were talking about the documentation and the living document. Another thing that this does is, you know, the reason we hire people is because we want to hire smart, good people who are going to provide benefit, right? Hopefully, we're hiring people who are smarter than us and better than us at our job.

MIKE: Probably, yeah.

MATT: So, it also provides them the opportunity to give their insight on the things we do. And they may have a different viewpoint on something like patterns or generally things we do that they may have done differently, and we've never really thought about doing in a way that they have. And it opens that conversation, and it gives us an opportunity to not only help them onboard but to improve our systems as well.

MIKE: I can think of a few cases where a capable person came on, and during the onboarding, they're like, "Well, you know, your test suite is running kind of slowly. What if you..." and, you know, fill in the blank. And, like, "No, that's a good idea. Why don't you try that?" And they did. And now everything works better.

DAVE: We brought somebody on a while back. And a couple of months ago, in architecture, they were like, "Yeah, we've mostly got the database in memory now, and everything is, like, so much faster." I'm like, I'm really glad this guy's here.

[laughter]

MATT: Yeah. And they're going to feel really good about their new job, right? Like, they came on immediately adding value. That's really fulfilling, and it's going to give them a really positive outlook on the [inaudible 26:45]

JUSTIN: I'm glad you mentioned that, Matt. One of the things is to give them a small task or something that they can be successful at in a small amount of time and have them do that thing such that they know the system end to end, whether it's, like, fix a small bug, or here is a small, new feature. Those types of things you could probably do. As a tech lead, you could probably do that in an hour or two. But as a new engineer in a new team, you can give this to them and say, "Okay, here it is. Here is our process. And here's how you do everything. Work with your mentor on solving this issue. Here's something that you could be successful at. Go for it."

MATT: Yeah. Quick wins are important, right? Going back to when I started with the company, it was...I came on, and I was told, "Well, we're expecting you to provide lift within two weeks."

JUSTIN: [laughs]

MATT: I said, "Okay." And then COVID hit. [chuckles] And I got sent to work from home with no training, and it was a little rough, you know. But something that did, for me, was make me really work to provide that lift because I had a goal. And I wanted to say, "Hey, you guys need to see that I'm going to provide value, and I'm going to do it quickly." And it gave me that motivation. So, yeah, I totally agree; giving someone tasks early that they can get in and have a win that's a really big morale boost.

DAVE: I worked at a shop where the development process was you shepherded your PR all the way to production, even if you were on another team, or if you were collaborating across the building. If you submitted a PR, you had to...and that meant deploying somebody else's server, which meant that everybody's deploy process had to be pretty well documented and sort of...like, if somebody else could not deploy your service, you had to stop and improve your documentation and your process, which I think was a pretty great thing. But I sat down on day one, and my mentor said, "We need to have you deploying to production by tomorrow night." And I'm like, "Oh."

They walk me through everything on that first day. And he's like, "Okay." And he pointed me at the...we've got a board here at Acima we call the Scooby Snack board, I think, which is basically just tiny, little things that somebody new could just pick up, fix it, and ship it, right? And it was that. They had a board of, like, tiny, little tasks that somebody could do.

The difficulty had nothing to do with changing this one thing on the website. The difficulty was I got to figure out how to deploy this to production and make sure that I haven't taken anything out and make sure that it all still works. And yeah, being able to focus laser tight on I got to write this. I got to get it through review. I got to get, you know, all the things super, like, clarity. And anytime I got spun off into the weeds, I had that story of, I have to deploy. What is stopping me from getting back on track to that? And that's my next obstacle to solve.

MATT: Yeah. Then you have some purpose. And I don't know about you guys, but everyone in this group today has a lot of experience and years behind us, right? But to this day...

DAVE: It's kind of an old guy podcast today, isn't it?

[laughter]

MATT: To this day, when I start somewhere new, there's some nerves involved. No matter how experienced you are, there are some nerves involved. And I think helping people get those wins early alleviates those nerves and gives them some confidence, and then you can really see productivity out of people once they have some confidence.

DAVE: How do we onboard people continuously? What can we do to onboard Matt, or Mike, or Dave on the new stuff that's coming through or the way we're doing stuff next?

MATT: That's a great question. And I could have used a little bit recently [laughter]. For those listening, I have moved my role here at the company. And there are a lot of things that I'm doing now that some of that continuous onboarding really would have helped.

MIKE: Let me answer that question with a question. Would we expect the process to be any different than it would be for a new hire?

DAVE: I think we do because if we bring you on board to a new thing, we don't sit you down and walk you through all the setup of everything and talk you through, like, you know, "Here's the lunchroom, here's..." it's more like, "There's the database; there's the cluster; there's the deploy. Let me know if you need anything." And, generally, if we're seeing you're...and not even senior. A lot of people that have been, you know, done this more than once or twice know what it's like to be abandoned in the middle of a desert and go, "Okay, let's start walking." You know, a lot of times, we just drop you into a target-rich environment and say, "Good luck."

MIKE: You say a lot of times that happens. And just because somebody can doesn't mean that that's necessarily the best way [inaudible 31:37] about continuous improvement, and not that I've thought about this. I'm thinking out loud here based on the question. What if everybody had a dedicated mentor that when they want to give them a, you know if you wanted to add a question, you knew that you had that person who was there to help you and who is going to give you opportunities on a regular basis to go stretch and do something you hadn't done before. Maybe spend some time pairing with you, or maybe you have periodic times when you go, and you work out on the floor, you know, you go and actually...software. All these things we've talked about, I don't see any reason why they would not provide value ten years in.

MATT: I really, really like that idea. And sorry, Mike, you got stuck with me because I just kind of use you for that all the time.

[laughter]

MIKE: Well, good [laughs].

MATT: Yeah, you know, I was lucky enough to end up on your team. And now that I have ended up in management and you are a director, I lean on you all the time. And I greatly appreciate your support in that. But having someone that you can go to and just ask those questions and will walk you through the things that you've never done it's invaluable.

DAVE: I'm kind of in scan mode. I'm like, how best to refine this? Yeah, I'm just thinking about, like, what I'm going to do on Monday, right? It's like, who do I need to be mentoring? Who do I need to mentor me on some things? Yeah.

MIKE: That's exactly what you got me thinking as well. Like, what can I be doing in my role today to make that effective for the people I'm working with? Maybe I can find somebody to latch onto and say, "Hey [chuckles]. Will you be my buddy and help me out here?"

DAVE: Yeah. And kind of related to this, sometimes it's like, okay, you decide this is what we're going to do. And then, you kind of have to trust the process a little bit, right? It's like we used to do...here at Acima, we used to do skills clinic every single day for like an hour. And it was fantastic. And the skills on the team went just through the roof, and then, you know, COVID hit. And business priorities and things got, you know, rougher, and attendance in skills clinic is way, way down.

I was talking with Tad. He's running them right now. I've had lunch with him today, and I was talking with him about that, and I'm like, "You know what? I have not been trusting the skills clinic process, and I have not been going to skills clinic. I need to start going to skills clinic just for the process of taking time to sharpen and being able to share with other people while sharpening." It's like, "Oh, hey, you know, we're doing SQL right now. You can do this with a sub-query. Have you tried doing it with a window function?" And, you know, people around the table going, "You can do that?" "Let me show you something." Yeah.

MATT: Well, and regardless of how many years of experience we have, everybody has something to offer, and everybody sees things from a little bit different perspective. So, I've learned so many things from what we have labeled as, like, juniors because they're thinking about things differently than I do, you know. I've done the same things for so many years that sometimes you get a little bit tunnel-visioned, and you don't look at things from a different viewpoint. And they really help along those lines and open up your eyes really, right?

JUSTIN: Help keep your skills fresh.

MATT: Yeah, absolutely.

DAVE: And you can be with somebody who's very, very junior, and they are probably thinking, there's nothing I can teach this guy. But what they don't know is that you're sat there going, if I don't learn anything from you, it's because I've failed as a student. I'm going to watch you until I learn something. And, invariably, I learn something amazing when I do that.

I worked with a girl who, halfway through the discussion, I found out she was a lawyer. She was a junior programmer. She had passed the bar in Colorado. And I'm like, "Why are you in here and not practicing law?" She's like, "Because I hate law. I don't want to do law. I want to be a programmer." And I'm like, "Okay." And sure enough, that afternoon, we're typing along, and something came by. It was like TCPA or something like that, and I'm like, "A or B?" And she was like, "It's B. It's B. A is against the law. Do B."

[laughter]

DAVE: I'm like, "Oh, okay."

[laughter]

MATT: Yeah. We had a contractor who was also practicing, the attorney, in a different country. But...

DAVE: Yeah, somebody who shows up with that, you know, they can think.

MATT: Yep. Again, everybody has something to offer, and we all have something to learn.

JUSTIN: So, I got a couple of really quick, rapid-fire questions. One that I've noticed was really good was as a, you know, a person leader, the manager is, like, having regular check-ins with your new employee, like, at the start, especially. Those would probably be, you know, weekly, that sort of thing at least. And then, of course, like, a big lunch meeting to introduce everybody at a restaurant, or something like that, celebrate the onboarding, but also, you know, weekly check-ins. And then, what are you guys' thoughts on a 30, 60, 90-day plan, like creating one for your new employees? Is that something that works for engineers, or is that something that is mainly something that exists outside the engineering world?

DAVE: I've only ever seen it done once, and when it was done, it was every bit as awkward as we all thought it would be. And it was profoundly effective, and I have missed it ever since.

[laughter]

JUSTIN: Wow. That did not go in the direction I thought it would go. I was like, [vocalization].

DAVE: Yap. It was with the first time I ever saw a manager who felt that being a manager was a full-time job and required him to study and learn management. And so, he was literally learning things like, when you have your one-on-one meeting, you have to quickly triage the meeting into, are they looking for training? Are they looking to just check in? Or are they melting down, and they need to vent? Because you have to handle those very, very differently. And if you try to go and fix and train when they're venting, it's going to go horribly. I'm like, I never had thought about that. And this particular person would come and say, "Hey, we need to go do our one-on-one."

And it was just...I'm like, this is awkward. This is dorky. "Okay, Carl, fine." And they'd get up...and we'd go to lunch, and it would start awkward. And then, five minutes in, we're just chatting, shooting the breeze. And we're talking architecture and where does the team go. And I grew a lot. And this was 10-12 years ago. It was kind of middle of my career. And my skills leapfrogged every single month.

That was the first place where I put on, as my personal goals, to lose weight or to be able to run a mile under a certain time. And Carl was very, very proud of me when I put that on because he basically...I said, "If I do good cardio, I'm going to think more clearly, and I'm going to be more effective as a programmer." And he's like, "I think that's a fantastic goal. Put it down. Let me know what your time is in 30 days."

MATT: Interesting. I like that.

MIKE: I think we have rightly done so, talked about things that work well. One final question I want to ask is, is there anything that poisons the process? Is there anything that makes it just work terribly? To maybe illustrate the opposite, right? You know, as to what you can do well.

MATT: Yes, not following through. That's key, right? As people get busy, people get sidetracked, things like that, I think it's extremely important to follow through on the things that you promise to deliver, and not only is that relevant for onboarding. I think it's relevant just in business, in life, et cetera. But, you know, let's say I'm onboarding someone, and I say, "Okay, next week, I'm going to spend two days with you going through x," and I get sidetracked. I don't follow through. That's a really big failure on my part and, letting them down and not giving them the room to succeed. So, I --

DAVE: And it's critical. It's critical at that time, too, right?

MATT: Yeah.

DAVE: Because that one failure is 100% of their interactions with you to date and that set the direction that they're going to expect everything from them. You could never miss again, and they'll always wonder, is this the time that he's not going to follow through?

MATT: Yep. You paint that picture.

JUSTIN: One thing I've seen is when they've chosen the wrong person to be the onboarding buddy. And if they choose someone who is bitter about their current position or is not optimistic, it kind of can make that new person follow that persuasion, which is probably not what you want. So, --

DAVE: It's important to distinguish between bitterness and pessimistic, I think, maybe. Because I've worked with a programmer who just he was very much, like, leave me alone, don't, you know, da da da da. And the onboarding was basically...Our boss, I remember on day one, the very first thing she said was, "That's Terrence's office. Stay off his radar. He likes to fire new guys." I'm like, "Whoa, okay!"

JUSTIN: [laughs]

DAVE: And it was a government contractor. You could play that kind of politics and kind of stuff. And that was really, really good advice. And I didn't follow it, and exactly what she told me would happen to me happened to me at the next layoff [laughter]. It didn't make me bitter. It didn't affect my attitude. It wasn't office politics. It was office sociology. It was just, you know, be aware. That's a live wire. If you plug it into the power system, it'll run the whole city, but if you lick it, it's going to ruin your day. And so...

JUSTIN: [laughs]

DAVE: And she wasn't bitter. She loved the company. She loved her job. But she was very, very realistic about, like, that's awesome, and that's garbage and, you know --

MATT: Pragmatist.

DAVE: Pragmatic. Pragmatism. Yeah, that's exactly it. If it works, it's true. That's pragmatism. If it works, it's true.

[laughter]

MATT: But yeah, we definitely want to avoid people poisoning the well, right? Because it does, it carries over, and it kind of just...it spreads as well.

MIKE: We've talked before about psychological safety. You're bringing somebody in. It's not just code. And this is a critical thing. Software engineering, even somebody who's just starting and they're going to be writing a lot of code, they likely will not spend most of their day writing code, at least not every day. Software engineering is a lot of things that aren't code. It is a group endeavor. And you cannot build large projects effectively in isolation. It doesn't work that way. So, you're not just getting somebody to understand a codebase. You're helping somebody understand and build a culture. And you can't forget that it's more than one thing you're building.

DAVE: And I think I've mentioned this before: my favorite epiphany that I took away from skills clinic is that we think of development as a process where the developer is in the box, and we take a problem, and we put a problem in the box, and the solution comes out of the box. And that's not a programming career at all. The box is where the effort takes place, and we take a programmer at a problem and we put them in the box. And what you get out is a solution and a smarter programmer.

And if you focus on investing in that programmer, they're going to grow, and they're going to get smarter. And if you focus only on just product, product, product, product, product, some people will thrive, and some people they just miss...and it's just bad luck. It's not that they're not, you know, good or hardworking. It's that they got so focused on optimizing the process that they end up...it's like not stopping to sharpen the saw, right? You get so busy creating the output that you never increase your ability to create output.

MATT: I wanted to make a Beyond Thunderdome reference there, Dave. Two men enter, one man leaves.

[laughter]

DAVE: Right. That's right. Two problems enter, one solution leaves. I like it. I like it.

MIKE: To come back to a story, several years ago, I was riding bikes with my kids. And my oldest son was pulling somebody [laughs], one of my other kids. And he was back behind. And he was really having a hard time keeping up, which is unusual for him because of who he is [laughs]; I'll say that. He usually does not have a hard time keeping up. I usually have a hard time keeping up with him.

So [laughs], he was just dragging, and he said, "Man, maybe I'm tired today." But he was just trying to muscle through it, and muscle through it, and muscle through it, and muscle through it." Finally, he said, "Can we take a break?" And he looked, and the bike he was pulling, you know, the trailer he was pulling, the tire was completely flat. So, he'd been, like, dragging a plow [laughs] through and was riding [inaudible 44:31] for miles [laughs].

DAVE: Just pulling [inaudible 44:33] the whole time.

MIKE: Yeah. If you don't take that time to look and make sure your tools are in good order and keep things going, you maybe can muscle through it for a while, but it's not going to work the long term.

DAVE: Tying it back to earlier, that's all about the hunting badgers, right? It's like the great thing that can come out of that is not, oh, I needed to fix the tire. The takeaway is sometimes I should stop and check my equipment. And that's the [laughter] aha moment of it. And that's the difference between having a good job and making a whole career, right? It's like that. You'll come back to that.

MATT: Instead of fixing the flat, prevent the flat.

DAVE: Or just be aware that --

MIKE: Be aware.

DAVE: You know, check your equipment, yeah.

MIKE: Sometimes flats happen [laughs], and you got to be aware of that.

JUSTIN: I think from a security point, the guy that thwarted the recent attack that was going to be on one of the base systems in Linux was checking his equipment because, all of a sudden, part of it was, like, going slower, and it was just a fraction of a second slower or something like that. But he went in and, like, discovered the bad code because he was checking his equipment.

DAVE: I've never believed so strongly in the a billion eyes makes all bugs shallow [laughter] from Linus Torvalds from the '90s. I'm like, okay, I believe you now, Linus, so...

JUSTIN: [laughs]

MIKE: Any other final words?

DAVE: Be good to each other. Do code reviews.

MATT: Yeah, right. [laughter] Our interactions are always a pleasure.

JUSTIN: Gentlemen, it has been a pleasure. Yeah, thank you.

MATT: Yes.

On this episodes, the panelists dive into common myths surrounding software development. Kicking off with a reference to Fred Brooks' "The Mythical Man-Month," the conversation challenges the notion that adding more developers to a lagging project will speed up its completion. The discussion quickly expands to explore contexts where adding skilled, senior developers might indeed be beneficial, drawing analogies with emergency services during crises where more hands on deck can mean saving more lives.

As the episode progresses, the group delves deeper into the complexities of software development projects compared to other types of engineering and construction tasks. They discuss the unique challenges of software development, such as the dependencies and the need for sequential progress in certain areas, which can create bottlenecks regardless of the number of developers. This leads to a broader conversation about the necessity of strategic planning, the value of experience, and the pitfalls of assuming that increased manpower translates directly to increased productivity.

The dialogue culminates in a reflective discussion on the philosophical and practical aspects of software development beyond mere coding. The group touches upon the ongoing nature of software projects, the critical role of stress management, and the adaptability required in the tech industry. They highlight that, unlike more tangible engineering projects, software development is often about continual iteration and improvement rather than reaching a definitive endpoint. This conversation underscores the blend of technical skill, strategic thinking, and emotional resilience necessary to navigate the evolving landscapes of modern software development.

Transcript:

TAD: Hello. Welcome to another episode of the Acima Development Podcast. I'm Tad, and I'm going to be hosting this episode today. I'm joined by Will, Justin, Mike, and Ramses. The topic we're going to be discussing today is Common Myths About Software Development. And when I heard this, I thought of one of the classic myths.

There's actually a book called The Mythical Man-Month by Fred Brooks, and he talks about how adding extra developers to a late project isn't going to make that project go any faster. Or one of the interesting examples he gives, I think it's from that book, too, is you can't have nine women produce a baby in a month. You know, human development process, nine months, there are no shortcuts. Have you ever worked on a project where they thought, oh, we'll add more developers to this project, and it's actually worked out?

MIKE: There's a very specific context where I've seen adding developers to a project actually helps, and that's when they are developers who are already very experienced in the industry and deeply familiar with the code at play.

WILL: Yeah, we write it small, right? Like, your team lead or your engineering manager, you know, your senior engineers they kind of fill that role, right? And they sort of, like, parachute in firefighter. If things aren't going well, you need to deploy special forces to, like, help get this thing over the hump. Yeah, I think that's kind of standard, right? And, like, you know, you could deploy as many senior resources as you have available and familiar, you know, which is usually not as many as you'd like.

MIKE: You had one phrase in there: as many as you have. Those three words are doing a lot of work [laughs]. Because if you've got an army, you have very few, like, Navy SEALs, right [laughs]? The elite forces are a small subset of your total workforce.

JUSTIN: It's also worth noting that it's very expensive, both in time and treasure.

MIKE: Ah, yes.

JUSTIN: But going back to your original point, Tad, I think it's like, it is something that is not the norm. Like, you just can't hire contractors to come and solve your problem.

TAD: It's a little counterintuitive, though, because in a lot of cases, adding more resources to something does increase the productivity, right?

JUSTIN: I don't know. Like, can you cite some examples?

WILL: Well, I mean, if I had an emergency room, right? I had an emergency room, holy cow, I'm so backed up. There was a train crash, and I have a lot of, like, patients coming in. I could bring in doctors, you know, I could bring them in or on call. And if I had to the degree that the facility could accommodate them with operating rooms and stuff like that, you know, I could bring in ER doctors, you know.

TAD: More nurses, right?

WILL: Fairly --

TAD: And you should probably increase the throughput in your ER if you did that.

WILL: Yeah, fairly linearly. And you could treat more people, right? And that's a high-skill profession, right?

MIKE: Exactly. Exception proves the rule because you're talking about one of the most notoriously highly paid professions out there [laughs] because it does take that deep expertise.

JUSTIN: So, another example that I'm familiar with is I worked in construction for a while, and, you know, this was back when I was in college and such. And you're working on a big project, and you're building a large building. Like, in this specific example, we were building a library, like a college library. And you can paralyze some things, but other things they have to be done before you can work on other things. And so, there's these natural choke points that you literally can't throw more people at or resources at because, like, the concrete has to set, or the dirt has to be, you know, prepared, and packed, and everything.

WILL: Exactly, yeah, yeah.

JUSTIN: And so, it's the same sort of thing when you're developing software. You have to decide on architecture. You have to decide on, you know, the third-party APIs you're calling. You got these things that you have to do, and you all have to follow the same pattern, hopefully, as you do the various things that you need the new thing to do. And if you have everybody doing different things, you might finish it, and it may be functional, but it's not going to be fun to maintain.

WILL: Yeah, and, I mean, I think it goes back to like, sort of, like, running projects. I mean, I think there's really deep parallels between, like, sort of, like, you know, sort of sequential and parallel programming. It's like, you know, multi-cores, right? We've all got eight cores on our desktop. If you want to keep those cores hot, all of them hot, like, that is ferociously difficult programming, right? So, like, to create, I take a process and parallelize it. That's a fantastically difficult programming challenge.

And I think analogous to that is okay; I've got eight people on my team. We all have to get this thing done. You know, how do I keep all these keyboards hot at the same time to, like, not choke point out, right? Because it's exactly as you say, there are natural choke points. And, like, you know, do you set up scaffolding? Do you set up, like, a fake API server so people can do the front end while the back end is still cooking? You know what I mean? All of these things. And that's a very difficult engineering challenge to, like, maximize the productivity of a squad with these natural in-built choke points. It's tough to do, really tough.

MIKE: Well, and that's a really good...thinking about getting parallelized work on your processor, some of that algorithm is to guess, right? To make a best decision and have it start doing some work that you might have to throw away. To get that parallel, that level of parallel work, you're probably doing a lot of throwaway work.

That is part of the price of trying to get that parallelism is that sometimes you have to go down, you know, rabbit holes that you just have to completely drop. And, you know, that may be true even when it's serial load, but even more so when it's parallel. If you don't know which way you're going to go, you have to do both. And, again, there's cost that comes from that. And if you want to massively parallelize, you're probably going to be doing a lot of throwaway work.

WILL: Absolutely, absolutely. I mean, it's a fact, an absolute fact that parallelizing, multi-threading a process is always less efficient, you know what I mean? If for no other reason than the job-switching infrastructure, you know what I mean? If you're real dorky, you could spend a lot of time on that. But, like, you got to do stuff every time that thread switches context. Every single time. You know, say nothing of deadlocks and all that stuff, so, you know, yeah. Absolutely.

But I do actually think developers are too afraid of scaffolding that's intended. Going back to construction again, right? You're building infrastructure with the explicit end goal of tearing it all out. You know, it gives somebody a platform to work on while the stairs don't exist. I think embracing that with both hands winds up being a pretty big net win, in my opinion.

MIKE: That's an interesting observation there. You say you deliberately build something you know you're going to throw out because it expedites the other work getting done.

TAD: Because it's pretty disheartening to throw code away. As a developer, you take an emotional hit when you've been working on something for two weeks, and they say, "Ah, we're going to go in a different direction." You're like, "Oh, sad [laughs]. Well, into the trash goes all my efforts."

WILL: Yeah, absolutely.

EDDY: And it hurts more when you have it tested. Like, when [laughter] you have all your tests [inaudible 07:55] out, and it turns out you don't need it anymore, and it's like, ah.

MIKE: There's a mindset thing here, though, of what value we're trying to provide to the business. We're not trying to provide code. We're trying to solve problems. And [laughs] if we approach it from that perspective, we say, "Yeah, this was a step in working towards solving that problem." You know, I think we have to be kind of relentless at ourselves that way. The code is not the end product here. We're talking about myths. Our job is to code? Boy, that is not our job.

TAD: Well, it's a component of our job, right?

MIKE: It is.

TAD: Well, maybe we should flesh that out a little bit. What are the other aspects of software development other than code?

MATT: Number one, communication.

TAD: Right, because you're talking about what you're going to build before you ever write a single line of code, usually.

EDDY: Stress management.

MIKE: Think about big construction like a bridge. You would not consider hiring an engineering firm that wouldn't take a year planning before they ever put a shovel into the ground, right [laughs]? Because it's recognized that for big projects, planning is a big part of the work. You've got to coordinate. You've got to figure out what you're doing. You've got to figure, you know, you've got to do surveying [laughs] to figure out what's going on.

You may not have that if you're doing small-scale construction, but absolutely for the big stuff. And even for small scale, I mean, you're going to have a blueprint, hopefully [laughs]. Even if you're building a shed, you probably have a drawing, right? That is work. There is a great deal of work that goes into that construction project that is not the hands-on laying down the materials.

MATT: Then it becomes like putting Legos together, right? If you plan everything well, you know the pieces that need to be put together, and then you just need to put them together.

WILL: Yeah, but, I mean, that's the standard. That's the standard. That's your waterfall model, right? That's waterfall development, which, I don't know, I mean, like, I'm pretty partisan, you know, in terms of I like smaller pieces of work, even if the overall strategy is like, I want to get this big thing done, you know what I mean? Like, I like small pieces of work because I just, you know, and we touched on it, I think, the last session.

Like, I am deeply...I'm not skeptical or cautious of overestimating how smart I am and how good my strategy is and stuff like that. With construction, if I'm building a dam, that's where it is, you know? Like, there's a lot of stuff where it's like, you can't go back, or it's prohibitively difficult to go back. But with software, you do have great flexibility for most projects.

MATT: Yeah. You've called out a good point. I think remaining agile is super important in today's software development world, right? It's necessary. Preloading the planning, yeah, sounds very waterfall-ish. If you can do that, maybe it's just the time before it hits engineering that happens, you know, and by the time it gets to engineering, then it's broken down to smaller problems. And you break it down into simple stories, and you can work from there. That's, I mean, to me, that's agile programming.

TAD: Is there a myth, though, that programming is predictable? Because what's that quote? You know, everyone has a plan until they get punched in the mouth [laughter]. I've never worked on a software project that has gone exactly according to plan.

JUSTIN: If it's a very simple well-known use case, you can do it very quickly. So, like, you know, I want to set up a blog, you know, have comments and everything, and that's, like, every single tech demo ever. So, you got every single language that you could do it in, in datastore and such. I was thinking of a really good example of something that is well-defined, and that can be put together in the real world.

And there was, like, a news story about the U.S. Navy putting together a port in Gaza to unload supplies for the refugees or the people who live there. And they basically could put together this pre-assembled port that the Navy uses on any combat situation, and they could put it together very rapidly and people who know exactly what to do. And all of a sudden, you have this port where huge ships can unload supplies on.

And it's just an example of, hey, if you have a well-defined problem and you have the tools to solve it, you can do very big things very fast. But if you go off that path, then you start having to, like, call in the experts and decide, okay, now that we're going off this path, what should I do in such a way that I'm building something that will last and that is durable? So...

WILL: Right. Well, I mean, and that's the nature of software sort of consuming itself in that, like, when you have that sort of, like, pre-built port roadmap, that's a software product, you know what I mean? Like, when it comes to software, like, when you have things defined to that degree, then it's just, like, push a button, you know what I mean? Like Shopify, right? If I want an e-commerce store, Shopify can get me up in an hour, you know, and they do a pretty good job at it. And so, it becomes, well, what else do you want to do? And fortunately enough for us, they always want more, just a little more.

[laughter]

MATT: And if you want to integrate with Shopify, that's another story.

WILL: Sure. Absolutely. But, like, if you wanted to integrate with Shopify, well, that's something, you know, new that Shopify doesn't do. And then, we're into custom software land, and I'm making my [inaudible 13:37] payment this month.

TAD: That does touch on one of the myths I was thinking about earlier today that a lot of times we parallel software development with things like building a dam, or construction, or things like that, and those things get finished, right? They complete the project, and then they walk away and move on to another project. And software, just because it got released, doesn't mean that you're done.

WILL: Oh yeah.

MATT: Never done.

WILL: Yeah, yeah. I think that's a myth.

EDDY: But when you --

WILL: So, the software is released, right? It's done [laughter]. Like, it's –-

MATT: [crosstalk 14:16] that you're finished, right?

EDDY: Sure, but you got to have people to go back and maintain whatever it is you built, whether it's software, whether it's a building, a car; it doesn't matter. There is a level of maintenance that still is required.

TAD: Even saying built makes it sound completed or finished.

MIKE: Yeah, and to take that further, some software products that we use were created for some utterly different purpose. Didn't Slack come out of, like, an internal messaging video game or something?

WILL: Some, like, in-game chat or something. Or was it just inside the company?

MIKE: I think it was just inside the company. So, they went to build a video game, and they ended up building a business messaging platform. Like, when we say build and release, like, yeah, I went to build a video game, and I built business software. And if you were going to build a bridge, you say, "I went to build a bridge, but I actually built a skyscraper with an airport on top." It doesn't work that way, but we have to do that all the time.

Think about that planning that was brought up before. Yeah, that planning, that cost still has to happen. But recognizing that we have to do these iterations, we still do the planning, but we do it on a smaller loop. We don't spend two years doing it upfront because we know that work is going to be thrown away, right? But we still have to do the work. It's just done in a smaller loop. So, you're spreading it out over the two years rather than doing it upfront.

You still have to have that feedback, and there's still the work involved. There's still the job of communicating, figuring out where we are now and where we go next, and that never goes away. Whether it's upfront or whether it's later, it's still there. And if you try to pretend it's not there, you'll get in trouble quick, I don't know.

And that brings me to another myth. I pointed...it's xkcd number 2030, two zero, three zero. I think about this all the time. It's about voting software, which is kind of irrelevant. There's a line from it where the character says, "I don't know quite how to put this, but our entire field is bad at what we do. And if you rely on us [laughter], everyone will die." And if there's one myth I see about software, you expect that people just know what they're doing. No, we don't [chuckles]. We don't know what we're doing. We don't even know what we're building.

We're building a video game that's going to end up business software, right? And that's not even necessarily a bad thing. We're building something out, and then we'll have something of value and see if we can use it. You know, it's this weird product that's kind of, like, other things and kind of unlike, I say, other things in that it's flexible, and sometimes it meets different needs. And it means that it's kind of hard to talk about, kind of hard to deal with. And we make a lot of foolish mistakes.

I've seen this with new developers, particularly people who change career later in life. They come in and they think that we all know exactly what we're doing, and it takes a while to be disillusioned. You got to kind of break that early and say, "It may look like we know what we're doing, but a lot of our job, maybe not most, but a good portion of our job is trying to figure out what we're doing. We're researching it as we go. We don't know how to use the tools. We don't even know what tools we're using, and we don't understand the requirements. So, a big portion of our job is figuring all that stuff out so that we can get something that works out in the end."

TAD: And a lot of times, you can't know what is needed, and what you need to do, and what the requirements are until you actually jump in and start poking around and doing the coding, doing the research, doing the things, getting both hands into the project. And you're like, oh, now I'm starting to see what shape this needs to take.

WILL: I mean, that ties into something that Eddy mentioned in passing, and it's a myth that I think...let me see how to put it. So, what Eddy was saying was, like, stress management is, like, most of what we do. And I think the myth of...and, I mean, I agree with that, and I think I'd maybe, like, broaden it out to say, like, I think people think of software development as this intellectual exercise. And there are sort of fairly specialized slivers of it that can be that. But by and large, you have to be smart enough and willing to sort of, like, bulldoze your way through thinking about the problems, thinking about the requirements, thinking about the legacy source code and how it works, thinking about, like, you know, how you do the stuff.

I mean, if you even look at, like, sort of, like, these programming puzzles that some people like for interviews, right? LeetCode medium, right? It's far in excess of anything that we actually have to do on an intellectual problem-solving level. What we need to do is maintain, like, a sort of, like, emotional equilibrium and focus on this fairly mundane, ditch digging, systems integration, team collaboration, requirement, negotiation, source of problems. Like, that's what will make you...

Like, if you're, you know, slightly above average intelligence but you have exceptional drive and emotional equilibrium, you know what I mean? Then you're going to be a very successful programmer. But, like, if you have exceptional intelligence and, like, you're a little bit squirrely in terms of, like, just grinding through, like, the necessarily frustrating tedium of, like, professional software development within a business organization, you're going to have a very difficult time, you know what I mean?

TAD: Yeah, and I think there's a myth that all software is kind of the same and that the skills needed are kind of the same. For example, I hear a lot of people ask, "Oh, are you really good at math? You really need to be good at math for software development". And then, I have to say something like, "Well, yes, if you're writing software for a nuclear collider, then you're going to need a lot of math. But if you're a front-end developer wanting to make some really cool interactive thing, maybe you need to know a little bit more design, and you don't really need that much math." Or the example earlier, set up a blog. I don't think you need any math to set up a blog probably, right? But those are all software development.

MATT: Yeah. And maybe this is just me, but I think the people who are just naturally good at math make good software engineers, just that mindset and the way they do things. They may not make the best front-end people, but there's a place for them, right? I think a myth also could be that, hey, I should just switch over to software engineering because the pay is great. I've seen a lot of people make that mistake. It's not for everybody.

WILL: They're having their day right now. I think there's a lot of people who came in for the bag, you know what I mean? And now we're in tech winter and, like, those doctor money FAANG jobs are a little short on the ground. And so, it's just like, "Would you only do this for a paltry $120,000 a year like a peasant?"

MATT: [laughs].

WILL: Like, "Whoa, whoa, whoa, whoa. I'm going to go back to finance," or whatever, right? And those kind of people. And then, you know, you have to wake up every day and want to do it, you know what I mean? I mean, just because, like...I'm pretty good at programming, but every project, you know, like, I got to wrestle the pig every time. It's a brand-new pig, and you're going to have to get in the mud every time. Every time. It doesn't get easier. You just get, I don't know, stronger, you know?

I was having a conversation with somebody who was talking about like, you know, like, needing to hire somebody and the difficulties associated with hiring people and how you'll have these people who, they'll go in, and they'll crush their interview. Like, they'll do really good on these algorithmic problems, you know. And then, you'll take them into the actual workplace where they have to go out and, like, you know, do the actual nitty-gritty nuts and bolts jobs. And they'll struggle or, like, they'll do good for a while. And then, sort of like the morale will fall off, and they'll struggle. And, you know, and, like, you'll have smart people who just can't maintain the focus of, like, staying after it, you know. And then, they will be bad because it's a nonstop grind and, like, yeah, it's tough.

TAD: I've heard it compared to a treadmill.

MATT: That's one of the reasons, Will --

TAD: If you enjoy the exercise, then a treadmill is not bad. But if you're constantly learning and you're constantly working on things, maybe you're frustrated some of the days, right? Not everyone enjoys exercising on a treadmill.

MATT: That's one of the reasons I don't ask the mapping of matrices and things like that in interviews. I just don't. I don't ask really computer sciencey questions because that isn't the real world most of the time.

EDDY: Okay, but the real question is if someone came up to you from your family and said, "Can you fix my computer?" Can you? Can you fix that computer?

MATT: Ugh. I hate that. That is the worst question ever, Eddy.

EDDY: [laughs]

TAD: Yes, that's why I have a master's degree.

EDDY: [laughs]

MATT: Sometimes. I mean, you know, sure, we know that stuff, but it's not what we do.

TAD: Yeah. I mean, I can Google as well as the next person.

WILL: I can Google much, much, much better than the next person.

TAD: Actually, that's true.

MATT: Which is what we spend a lot of our time doing, right?

MIKE: That's one of the most critical development skills is Googling.

TAD: Finding the right information, right?

MIKE: If you think that managing hardware and setting up something with, like, the TV is what we're good at, then you should see any engineering meeting trying to figure out the presentation, like, the hardware in the meeting room [laughs], just trying to get a meeting going. It never works.

MATT: Let's not even get started on spreadsheets.

WILL: I'm terrible at spreadsheets. I don't know how to do them. Like, as soon as...whenever I've had a spreadsheet, like, challenge, like, I Google it. I learn everything I need to know for the next, like, 15 minutes of work, and then immediately cash dump it. I'm just like, pfft, flush. That's out. I'm out.

MATT: I do the same thing.

WILL: Pivot table, whatever. I know the word pivot table is usually the answer to my problem.

TAD: [laughs]

WILL: And then, if you put a gun to my head and ask me to write up a pivot table in Excel, you better have a bucket and a mop.

MIKE: You know, you said something about we're focusing on the, you know, the grind and the treadmill. I read a quote from a bikepacking racer. I don't remember which guy it was, but I read it recently. And he said that that kind of racing, like endurance racing, and it could also apply to marathons or ultra runners, that kind of thing, he said that it's mostly about fatigue management. And you think, oh, you know, it's about building strength. And he's like, "No, it's, like, about eating the right food and managing your sleep and not pushing yourself too hard."

None of those are, like, things that you might necessarily think about have to do with racing. Like, eating? Racing? What? No, it's about managing your body and your equipment so that it can keep going. And that sustainability, you know, trying to think about that metaphor, that sustainability really is, like, a central aspect of our craft. And you can get off, right? You can get unsustainable, and that puts you in a bad spot.

MATT: You end up in burnout. Be good at compartmentalizing, right? I know you and I have talked about that before, Mike.

WILL: It's legitimately tough to do. I think there's a lot of places that are in...I will call them...I mean, honestly, I'd say, like, go at doom loop, you know what I mean? And I was talking about this with Mike a little bit, like, before we got on. It's one of the things that I've been seeing because, like, you know, we're in tech winter now, right? So, things are rough. You know, layoffs are happening, you know what I mean? Like, teams are squeezed pretty tight. I think it's across the industry. Even if you made it, even if you're still working, like, you still have, like, the reaper's hand on your shoulder being like, "How are the tickets going?"

[laughter]

And so, you have these situations where, like, you know, you have a business downturn. You have business pressure. And then, they'll have layoffs, you know what I mean? So, like, that impacts morale. You lose head count. Maybe you don't have nine women to make the baby in a month. Maybe you only have eight. And then, that impacts productivity, and then...you know what I mean?

And a lot of it really is sort of, like, it impacts people's emotional equilibrium, their feelings about their work, you know what I mean? Like, I think it's very understated, you know because it all exists in your brain. And without that focus, that willingness to focus, that willingness to, like, really concentrate, things become extremely tenuous. And you get into the burnout mode, you know.

But you can't stop peddling because, you know, you got business pressure. And, like, you got to make your deliverables. You got to make...you have to make money so that the business can give it to you. And you find yourself with these doom loops, these ugly, ugly, ugly spirals. And I see a lot of that in a lot of places when I talk to folks in the business right now.

MIKE: Which comes back to the same thing, that tenacity [laughs] and fatigue management is going to be the thing that gets you through that winter. My grandfather lived through World War II in Norway. And his family survived through the winter on a barrel of pickled herring [laughter]. [inaudible 27:54] through the winter on a barrel of pickled herring [laughs] is how you get through the winter. Recognizing that it may not always be very nice. There probably will be spring on the other end.

WILL: Oh yeah, and it always will be. Nobody's throwing their computers at the dumpster. I don't see it. It's like people would just be like, "Ah, the internet. We're over it. We're done [laughter]," right?

MATT: At least in my experience, there's always a light at the end of the tunnel. You're going to experience burnout. It happens to everyone. Like Mike's grandfather, you just power through it, and you do what you have to do to survive. And, eventually, you know, this can be the greatest job in the world. It can also be the hardest job in the world. And you're going to experience both of those.

WILL: It depends on the day [laughs]. It depends on the day. Sometimes, it depends on the hour. I mean, I will say that, like, that grind...I don't remember where I heard this, but, like, tenure of a software developer is they usually make it, on average, to about 36. That's not a lot better than playing in the NBA.

[laughter]

MIKE: Well, that's interesting.

MATT: That is very interesting.

WILL: And I believe that number could be out of date, at the very least. I remember it very clearly that I definitely read that, and it could be, you know, maybe a shaky study. But, I mean, I remember the number because it struck me so viscerally. I'm like, oh my God. Like, if you look at somebody like, I don't know, like me, I'm 44, right? If I was a physician, an MD, I'd be, like, just entering into my best years, entering it. It's like, well, you know, he's still a little green, but he's starting to get his feet under him. You know, I think he's ready to go.

But, like, as a software developer, it's just like, you know, people looking at you like Old Yeller, you know what I mean? [laughter] Where it's like, you're walking into the office with a limp, you know what I mean? Like, let's take you back behind the shed.

MATT: Well, just with this group here, we're doing pretty good with those odds, right?

MIKE: Yeah [laughs]. I'm thinking people I've known who have gotten past that or maybe the people who did have the ability to keep going. I've worked with some experienced people. I do work with some experienced people that I find their experience to be indispensable. I don't know how they could do their job, or I could do my job without being able to rely on those years of experience that they wouldn't have had at 36.

WILL: I've found that older developers tend to be pretty bimodal, you know what I mean? Like, there's not a lot of, like, developers, you know, like, north of 40 that are in the middle. Like, it's not really a bell curve. It's kind of like you're either --

MIKE: [laughs]

WILL: Awful or you're fantastic. I'm trying really hard to head towards the light because I have met some awful ones, some truly, truly bad ones where it's just like, you gave up several years ago.

MIKE: So, that can happen when you step off the treadmill for a while [laughs].

TAD: It's something that you got to keep working on all the time because technology keeps advancing, and you got to keep up with the advancements, right? Not necessarily all of it everywhere, but at least the stuff that's relevant to what you're working on.

MATT: We're all going to be prompt engineers before you know it.

WILL: All ready. All day.

MIKE: [laughs]

WILL: I think that's where things are going. I think that AI tools I think they are going to be a fantastic force multiplier, but they're not taking anybody's...well, I wanted to say they're not going to take anybody's jobs, right? But, like, if they double your productivity, you know, the head count's going to go down by 20% probably.

MIKE: There was an agricultural revolution back, what, in the 1800s? And that doesn't mean we don't have farmers. We still have farmers. But they generally operate at a much larger scale, and they're a much smaller percentage of the population. And interestingly, new fields of labor opened up because everybody wasn't farming anymore. I don't think our jobs are going away. Like, there's still people developing software.

And it took a while, you know, but agriculture still exists. It's an important industry. And, you know, there are farmers. Farmers exist, but the way they operate is different. And there's new industries that didn't exist, like I was saying. You know, they've enabled different ways of working. And so, a lot of people who would have been farmers are now doing stuff that's adjacent to that or maybe totally different to that because it's enabled by the farmers being more effective.

I think we'll have similar kinds of transitions that are a little hard to think about. You couldn't have explained a software developer to a farmer back in 1830. Conceptually, that work didn't really exist, and we couldn't even conceive of it. But I think that we'll have that similar sort of transition where, yeah, there'll still be people developing software, but it'll be a different format. And the jobs people have it might not be prompt engineer. It's going to be something that is going to be kind of incomprehensible, I imagine, to us right now. But it would make sense if you saw the evolution over time.

TAD: But I think it's kind of a different beast, right? Because I don't know of any organization that's like, "Oh, you guys have doubled your productivity? Oh, well, we're going to keep the same number of feature requirements that we had," right? They're like, "Oh, you've doubled your productivity? Sweet. We can, like, have twice as many features as we wanted before," right?

Well, I haven't experienced it yet where they've run out of ideas or new software projects or things like that, where they're just like, "Oh, well, we're kind of running low on places to spread our software or new places..." you know, it's like...because I imagine, like, even if we suddenly, like, 10x our productivity overnight, they'd be like, "Oh, sweet, we're going to move into, like, more countries than we were planning," or "We're going to have more features per year than we were planning."

WILL: And I'd also say, like, even more so that because of the nonlinear increase in complexity as software scales, right?

TAD: Yeah.

WILL: Let's say everybody doubles developer productivity, which is, like, a pretty bold statement, but, like, attainable, attainable. But that's probably the top edge of attainable, in my view, with, like, the AI, like, stuff. That's, like, 20% more features because, like, it isn't linear, you know what I mean? 20% more features. Because if you have 20% more stuff, like, I mean, so, within Acima, I mean, think about, like, it would be maybe one other big initiative, you know what I mean? You know what I mean? Like, you'd add one more team feature thing, right? I don't know how many teams you guys have now.

MIKE: I think you're onto something really important there. We talked about the planning and the non-coding aspects of the job. The AI stuff really helps mostly with the coding aspect, right? Does it help with getting requirements from stakeholders? Probably not. If 50% of the job is coding and you completely eliminate that [inaudible 34:59]

WILL: And the style, right?

MIKE: 100% productivity will increase in coding, right? Like, it all goes away. You only get to work on twice as many things --

WILL: Absolutely, absolutely. So, you know, honestly, like, I've been really having a tremendous amount of fun with it just because, like, generally speaking, I know what I want done, and I can express it in a reasonably concise, understandable way. I just might not know, like, all its syntax, you know. And so, like, that has just been, like, cleared from my past. It is an absolute delight, as far as I'm concerned, because a lot of the, like, I just [inaudible 35:37] BS ticki tack jobs are just, like, I have an assistant, you know, with a broom, you know what I mean? Like, I could build a house, and I have a helper handing me boards and, like, doing the sweep up for me. I'm like, ah, this is...yes, all day. I love it.

TAD: Yeah. I think that's a great place to kind of stop. So, thank you, everybody, for participating. Thank you for all of your comments.

The panelists discuss innovation and collaboration, particularly in engineering and beyond. Mike kicks off with a personal anecdote about his musical education, highlighting the limitations of conventional music lessons that focus heavily on theory and reading music, rather than fostering creativity and improvisation. This experience ties into his broader theme of innovation, suggesting that traditional educational methods might stifle creativity, which is essential for genuine innovation.

The conversation shifts to practical applications of fostering innovation within organizations. Mike uses the analogy of his own gardening experience, where allowing a garden to overgrow led to unexpected and beautiful variations in his plants. This metaphor extends to the workplace, where Mike argues that allowing some processes to be unstructured can lead to innovative outcomes. He stresses the importance of making space for collaboration and letting ideas "go to seed," which can spur development and bring forth new, valuable changes that wouldn't occur in overly controlled environments.

As the discussion unfolds, Will and other guests explore different aspects of innovation in the workplace, including the potential pitfalls of too much freedom or too much control. They debate the merits and drawbacks of Google's famed 20% time, where employees are allowed to spend one-fifth of their work hours on projects they're passionate about. This leads to a broader reflection on how companies can balance structured and unstructured time to foster an environment where innovation thrives without sacrificing operational efficiency. The episode concludes with a consensus that while not all innovative efforts will succeed, the potential for groundbreaking ideas makes the endeavor worthwhile.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm going to be hosting again today. With us, we've got Justin, and Ramses, Eddy, and Will. Will, I think, is joining us for the first time. Welcome, Will.

WILL: Happy to be here.

MIKE: Happy to have you here. As usual, I'm going to give a lead here and kick off our meeting. I'm going to start with actually two stories today, and both of these are outside engineering. And I've given several music references. I'm going to do another one today [laughs]. It's a deep well to draw on.

When I was a kid, my mom signed me up for piano lessons [chuckles], and I was nervous about it. I thought...like, everybody else talked about it being dreadfully boring, and I thought it would be as well. Actually, I enjoyed it. I loved it. I took lessons for a number of years, nine years, I think.

And, in those nine years of lessons, I think I had zero lessons on improvisation or creativity. And I think I'm nowhere near unique in that respect. I think most people who have music lessons learn how to read music. And honestly, I think that's kind of a travesty. I've read from a mathematician, so there's something worth looking up. It's A Mathematician's Lament. And he actually gives music as an example. He said, "Imagine that music was taught by everybody learning to read music, but you never listened to it," because he took that idea to its logical extreme.

You learned how to write notes, do all this notation, study chord structure, and maybe when you got to, you know, like, your teens or the really advanced students would be able to start listening to music. You know, in a world like that, of course, everybody would hate music. But music is often taught, not quite so badly, but kind of because we learn to read it, but not to write it. It's not given equal weight, and I think that's really unfortunate. Because if you don't give time for somebody to learn to create something on their own, they'll never learn. How could they? If there's not time, there's just no way.

When I was in high school, I took a guitar class [chuckles] and learned to play guitar. And I loved it. And suddenly, for the first time, music theory made sense. I looked at the guitar. I was like, "Oh, there's this circle of fifths thing, and, you know, let's see how this all works."

JUSTIN: [laughs]

MIKE: Then I started, you know, playing chords and chord progressions, and it was great. Later, I got a bass, and it was kind of the opposite of piano. Piano, you have all this polyphony and the bass; you just got one note you're plucking away at, and I loved it. You're thinking of, you know, I was able to improvise, think about just kind of the pure baseline there.

And I've had many experiences that I've really enjoyed, like laying down a bass line. Somebody's playing, or somebody played something, and I try to come up with a bass line to it, or, you know, I've kind of come back to piano in the last few years as well. I enjoy making music with other people, and I like creating music. And I probably do more creative stuff than reading music, to be honest. I flipped the tables because I had time to do it, and I was given a structure that allowed me to do it. And it made all the difference, and it allowed me to enjoy music in a way that I wasn't able to before.

One more story. So, I want to do something different from music. We've done lots of music references, so I'm going to do something very outside of music. I've got a garden behind my house. I like greens. I'm a fan of greens, but I'm particularly fond of red Russian kale. But I'm actually not going to talk about red Russian kale. I'm going to talk about a different green, which is mustard greens. I've got a variety called Ruby Giant. It grows big, beautiful leaves that are red on one side and green on the other side.

Last year, I let some of those go to seed, and when it gets hot, they do that. They go to seed, and they leave seed all over. And then I let the seeds grow, which looked kind of weedy, to be honest [laughs]. It was kind of messy in that area of my garden, but I was curious what would happen. And in the fall, I got a lot of mustard that grew up. And one of the mustard plants that came up was pure red. Usually, like I said, they're red, kind of red mottled on one side, and the other side, it was green. But this one was red, just deep burgundy all the way through. It was an amazingly beautiful plant, tasty, too [laughs]. That's some of it. I actually pickled some mustard greens. And I hope that it comes up again this year.

The point that I'm getting to is, if I had not let the garden get a little weedy, you know, get a little overgrown there and go to seed, I never would have had that second generation. I never would have had that innovation in my garden where I had something that I never had before. If I just kept everything nice and tidy, that would have been it. I would have had that first generation. I never would have had something new. I gave it space. If I had not given the space for something to be a little messy and go its own way for a while, then I never would have had something that was genuinely different from what I had before.

There's my two-lead story. Today, we are talking, and I don't know if I've mentioned it before, fostering innovation and collaboration among engineers. But I haven't talked about engineering yet, but these stories speak to me about what it takes to promote innovation or collaboration. I guess they're somewhat related. If you want collaboration, I think you have to make space for it. And you can't enforce it. You have to let it go to seed a little bit. So, there's my take. Thoughts? What do y'all think?

WILL: I'm kind of curious, like, what do you mean by, like, sort of innovation? In that, for most of the work that we do, innovation is really more customer-facing than technical-facing. You know, like, there are companies and organizations and stuff like that that are actually sort of cutting new trails technically. But by and large, I think the overwhelming majority of innovation is, like, delivering things to customers. And we're not necessarily, like, writing our own database, or writing our own operating system, if we're doing a new LLM based on sort of, like, existing open-source work.

And so, when I think of innovation within engineering teams, generally speaking, it comes in one or two flavors, one of which is a little bit of an anti-pattern and one of which I think is really, really healthy and good. Like, one of them is honestly, I mean, it's sort of, like, engineers gone wild because you've got a bunch of smart people who are maybe underutilized, you know, they're kind of chewing the walls a little bit.

And so, you'll find just shiny object syndrome, you know, the sort of, like, JavaScript flavor of the moment, you know, just because people are bored. People are bored at work. They want to play with shiny, new toys. And, you know, you'll find yourself in a situation where, like, they're reinventing things that aren't really generating a problem. And that's a little bit of an anti-pattern, but that's the evil twin.

The really good part about it is when you have had a nagging, constant pain in an engineering organization and figuring out a solution to it. And so, you need to be, I think, disciplined, you know what I mean? Like, are you solving a real problem? Is this thing better, or is it just new? There's actually technical innovation that is sort of, like, customer-focused and facing, which, you know, I love but isn't necessarily an engineering question. I hate to buy into the grossly incorrect notion that only product could ever come up with technical customer-facing innovation.

MIKE: [laughs]

WILL: I think that's incorrect, borderline offensive. But, like, within the context of a sort of engineering organization, that's where I'm coming at it from.

MIKE: I've got a couple of examples I can think of just off the bat. We have taken times in the past that I think we still got on the calendar, what we've called 5% time. It's just every other week in the afternoon, we have this space to collaborate on something, a time to scratch that itch to work on something together.

And a couple of years ago, maybe three years ago [chuckles], sometime in the last few years, some of us chose to take that time and extract an important service out of the main application we were working in. And that service became actually kind of critical to the company because it's our merchant API, and it allows them to get information. I don't want to go too much into technical details about the company. We were able to extract this application. It ended up being a really valuable extraction and would never have happened. It wouldn't have had any legs at all unless we had just taken that time to pull that out.

More recently, we've done something similar. We had a DevDay project, worked on something that we wanted to get done. We took a, basically, it was a message queue [laughs] that some agents were using, and we pulled it out of the database, put it into Redis to make it perform better. And it was really volatile data anyway. And we put it over there, made it into production. And it's how it's running now. And it's been working [laughs], right?

And, again, it wouldn't have gone forward unless we'd taken that opportunity to schedule some time and say, "Hey, go do an experiment." We didn't finish the experiment on that DevDay, but it gave us something. We actually ended up scheduling [inaudible 08:35] engineering organization because it solved several problems, removed some old code that we needed to get rid of, and an aging library, you know, solved a problem.

And I think to your point, Will, if you just set a bunch of people free and said, "Hey, work from wherever you want to for as long as you want to," and there are some organizations that do that, I think, you end up with a lot of ivory towers that don't accomplish much [laughs]. You know, people build this amazing, perfect software that doesn't solve any problems. But if you box it in a little bit, and I've seen this a lot of times, you box in creativity a little bit, it actually tends to grow. If you restrict the number of options, then people have something to work with. You know, if you've got a riff to play against, well, now you've limited the options, so you can make some music on top of that. It works a lot better.

EDDY: You know, Mike, you mentioned 5% time, and I'm pretty sure that's based off of Google's 20% time. It's just fractioned off a little bit. I want to go on record in saying if I remember correctly, I think some of their best products that Google uses right now on their subscription base was because of their 20% time that they fostered. I want to say some of those were, like, Gmail, for example, and, like, I don't know.

MIKE: I think Gmail did come from that. Yeah.

EDDY: Yeah, and, like, Google Maps, I think. It's been a while since I read that list, so I don't have an exhaustive list. But that's just a prime example of if you foster innovation in your team, you know, you get some really solid projects.

JUSTIN: I think part of that is people who are in charge have the ability to recognize when something has legs as well, you know, a little bit of guidance there from management where they can recognize, hey, this project has a potential to solve a problem. Let's gently encourage people to continue working on it.

And then, also, the flip side of that is maybe those same people look at stuff that other people are working on and say, "Hey, you know, maybe you should be working on something else," or helping them recognize, "Oh, is this a dead end," or "Is this something that you want to keep on going on?" and giving useful feedback that way. You don't want to, like, nip things in the bud necessarily, but at the same time, I think giving useful feedback and being able to receive useful feedback I think is key to being successful with these side projects that become big things, big successes.

WILL: I think if you're doing 5% time, then you sort of, I mean, I don't know, maybe Mike could, like, maybe this isn't how it went down, but I feel like you guys found something good in 5% time and exactly like that, like, management, like, saw it. And they're like, "Oh, this is kind of cool. Okay. You know, like, I'm going to feed this a little bit. I'm going to get this onto some sprints so that it could actually, like, go."

I don't know, I personally favor, like, this 20% time because I don't think 5% is enough to ship a product, you know what I mean? If you actually want to ship a thing that somebody could use, 5% time ain't going to get it. And, like, the thing about, like, sort of, like, management is management has their own priorities, and management gets their say. They get their time. It's not like management is doing a bad thing, but I mean, the idea of unstructured creative time is stuff that isn't on management's radar.

And I think, I mean, I'll say it directly, like, I think management is not as good at innovation as they think they are. It's not that they're bad necessarily, but, like, they don't have more good ideas than the whole of the organization, you know what I mean? Like, really tying people loose will come up with stuff. I mean, and, like, hear me, you have to be, I suppose, like, maybe more specifically what I'd say is that sort of, like, wide open 20% time, you're going to have some duds. You're going to have a lot of duds. You're going to have a lot of stupid, boneheaded, dumb go nowhere ideas. There's going to be some dumb stuff that comes out of that unstructured time.

And management is a little bit...I think they're naturally resistant to just transparently stupid ideas. But you're also going to take stuff that's a good idea, but it's, you know what I mean, a little wild for, like, you know what I mean? Like, you'll never get it into a sprint plan unless it's done, and it's like, oh, wow, this is cool, like, you know what I mean? The Google projects are a classic example of that. They did a lot of stuff that was really, really big, but they also stopped doing it because nothing makes as much money as ads in search. The manager was like, "This is cool."

Everybody loves Google Reader, right? Beloved dead product. Or another one, I mean, like, and this is maybe the other side of the coin, Google Voice, right? Google Voice, right? What a stupid idea. And it never made any money, and it never did anything. Oh, wait, here it is, 2023. We have this absolutely gargantuan corpus of voice data that we have been snarfing up for literally a decade. Man, our LLM would love that, wouldn't it? [laughter] You know, I mean [inaudible 13:37],

EDDY: You just suddenly stumbled upon having a full database of the stuff that you would kill to have, and you have it by accident.

[laughter]

WILL: Yeah. Oops! You know what I mean? And that's kind of the thing, you know, I mean, it's, you know, can you afford it? You know, I mean, there's good and bad points. But, I mean, like, the management control I think it's just a flavor, you know, like, somebody likes chocolate, somebody likes vanilla, and it is all heavily directed by business constraints. Google prints money. They print more money than anyone has ever seen, ever. And it just keeps on coming. So, they had opportunities to do that that people with tighter budgets maybe don't.

EDDY: You know, what I feel like companies typically don't do is they don't reach out to people who are out in the field, you know, doing a bunch of that customer interaction. Let me give you an example. When I go into a restaurant that I haven't been to before, my general go-to rule of thumb is talk to the individual who is in the front-facing. And I'll ask, "Hey, what is your favorite dish from your menu?" They're probably going to have a more concise answer than someone who's up in management because they're consuming the product at a different level.

But not only that, let me take it a bit further. There are individuals who I ask, "What do you do differently to the products that you sell that make it taste better?" Do that sometime. I promise you, nine times out of 10, they're going to give you a different menu item that's not publicly available. But let me tell you what, it tastes so much better because that's not something that they typically offer. If you're trying to promote innovation, reaching out to the people who have the ability to play around with your product is probably the best thing.

To add to that, when I started at Acima a couple of years ago, part of the onboarding process was speaking to people and agents who are talking on the phone, who actively use the product that we develop on a daily basis, and I got to speak with them. And I'm like, "Hey, what would you like the app to do, something that we haven't considered?" The amount of input that they have is insane. So, I feel like there's some lost potential there that we could harness or any company can harness.

MIKE: Absolutely. You know, I mentioned you have to have an untidy corner in your garden if you want something new, and not everybody's going to like that. It's going to make the neighbors uncomfortable [laughs]. It might make some people in your own home uncomfortable. They're like, "Well, no. Why don't you just level that out?" I just want to see some bare soil." But that's kind of the price you have to pay. You have to have some of those wild ideas, some of the dumb ideas, some of the things that aren't going to work because that's what innovation looks like.

You know, to Will's point, management is not uniquely qualified to have smart ideas. What good management does, and this also builds on what Eddy was saying, is that they look for those ideas. It's not, hey, the idea was mine; rather, I found a good idea that somebody had, and I'm going to promote that because it's a great idea.

JUSTIN: I think that's one of the best talents that good leaders have is the ability to recognize good ideas. If you look at Steve Jobs, I mean, he's acknowledged for his designs and everything, but he also has the ability to recognize, you know, hey, what is a good idea? What all fits together? What could fit our company? Or what do people potentially want? And it's not all just him, of course, but it's his ability to recognize that.

And Apple has a way of doing things that makes them one of the top companies in the world. And so, if you look at that, it's like, obviously, as a company, they're generating great ideas. They probably have a dozen misses for every hit they have. And so, being able to have a place where you can have a dozen misses and not be fired, and being able to do what you want to do to encourage that, I think, is key. Having a wild garden or a place where growth is encouraged.

MIKE: Yeah. You mentioned Steve Jobs. One thing I've always been struck by is, as a kid in the United States, a lot of times we hear about Thomas Edison, this great inventor. And you picture some guy in a lab, just some guy in a lab. But if you actually read about his lab, he had, like, a hundred employees [laughs]. His job was not sitting there at a bench inventing stuff. His job was to lead a team of engineers.

And what made them successful wasn't one guy being brilliant. It was a team of engineers being given the space to work on good things and then filtering down the ideas and channeling them. It's a department effort with good leadership from the top that makes a difference. That guy's name's on it, but it really wasn't him who came up with most of the ideas. He just led a team.

WILL: But, I mean, like, if I'm being honest with you, I mean, like, most of the corporate managers I know were not selected for their ability to innovate, you know what I mean? Like, how do you become a manager, right? Well, you know what I mean? You have, like, sort of technical expertise, and you have the ability to, like, sort of, like, organize a team. And a line-level manager is not really an innovator. That's not really the job, you know what I mean? Like, the job is to sort of, like, organize and implement somebody else's ideas.

And how do you become, like, a mid-level manager? A mid-level manager is a line-level manager, and a senior manager is a good mid-level manager. And as you go through, like, successive generations, I mean, like, the original person, okay, that's like a Steve Jobs type, right? Which is why Apple went down the tubes pretty quick after the innovative person he left. And so, I mean, like, most corporate managers, being good at innovation is not how you get that job.

So, I'm [inaudible 19:21] for a very large retailer who shall remain nameless. But they have succession, successive generations of these sort of managers, you know what I mean? Who are implementers, you know what I mean? Generation after generation, after generation after generation of these guys, and they're not dumb guys. But, like, that job is not their job, and sort of, like, they've been evolutionarily selected, for, like, particular traits, and then they get to the VP level, and it's just like, okay, innovate. And it's just like, okay. [laughs]

JUSTIN: So, what you're saying is we are selecting for the wrong thing at the manager level, or many companies are selecting for the wrong thing at the manager level. They should be selecting for innovation when they are instead selecting for the ability to manage a Jira project.

WILL: Well, I mean, like, you need the ability to manage a Jira project and sort of, like, what that 20% time was, you know what I mean? Maybe the argument for it is an antidote for Jira itis implementation type stuff in that, like, okay, let a bunch of wild prototypes out and not sort of, like, curating them because you are a tastemaker, which you're not, you know what I mean? But, like, sort of, like, being humble and saying like, "You know what? I don't know what's going to work. Let's just let a thousand flowers bloom.

And when something goes, when we have a winner, when something hooks, even though it wasn't my idea, then I'll take it, and I'll do what I do, which is, like, get this thing over the line in a polished professional setting." Like, that's the counter, right? But it takes a certain level of, like, egoness and, like, a release of control, which is not necessarily how corporate hierarchical management works.

EDDY: You also need a butt load of money to innovate and take that risk, too, right? So, it's not like anyone can really do that.

JUSTIN: That's an interesting constraint. If you are in an environment where you are trying to squeeze every developer efficiency out, allocating 20% time for other stuff that you don't have a guarantee of return on, that's hard.

MIKE: Two things, Will, you said. The 20% is the antidote for everything else. I think that's exactly right. It's the part you set aside that you don't control [laughs], and controlling the rest is actually really important. That 80% is critical to your business. You don't want that 20% to grow to 80% because then you go down the tubes pretty quick. You know, you need to keep that constraint, but you have to have that innovation.

There's a huge risk in being stagnant. In some ways, that's maybe the most risky thing. If you look at the most valuable companies from 20 years ago, it was like Exxon [laughs]; IBM may have been on there. You know, the companies that you don't really think about that much anymore. You know, there's all these tech companies today. And 20 years from now, it'll probably be another set. If you look a hundred years ago, you had train companies or the big oil companies, and they were huge. They looked like they were unstoppable. They were massive transportation company. But they quit thinking about themselves as a transportation company, started thinking of themselves as a train company because they've got all these trains.

And once you start getting that line of thinking where you don't innovate against yourself, you become stagnant, and then an upstart comes and out-innovates you, and you lose your momentum. You have to have that innovation because that's also risky. It's risky to put all of your eggs into that, but it's also hugely risky not to innovate because you will eventually stagnate and die out.

WILL: And I'd also say, like, I mean, a little bit of a false premise in that, like, you think, like, oh, well, we're going to invest 20% time in people coming up with their wild, crazy ideas. Okay. Well, I mean, that's a big risk. But it's also a risk, like, I'm working for a very large company, where bluntly, the innovation stuff is the decisions are made based on VP-level people who did not get their position through being an innovator. And they have vibes, and they have some vibes.

And then, all the chips go to the middle of the table based on one individual who is...and I want to be very clear; they're smart people and hardworking, you know what I mean? Like, they're not bad people. Everybody that I've dealt with has been. Like, they've all been, like, solid folks. But, like, that's not their thing. They've never invented a product, ever, until they get to do it, and then all the chips go to the center of the table.

And if their vibes are wrong, you know, then you're cooked, whereas, I mean, bluntly, if it's 20% time and everybody's just coming up with some wild idea, I will call a spade a spade. You have a much higher standard that that idea has to meet for somebody in management to pick it up and run with it. It's got to be a lot stronger than somebody just having a vibe.

JUSTIN: Yeah, I got an interesting story about this. My friend works for Ford pretty high up in the finance department. Ford, a couple of years ago, decided that they were 100% all in on the electric. Then they came out with a couple of electric vehicles, most notably, the Ford F-150 Lightning, which came out to great fanfare, but turns out that the battery is just not there for what people use a truck for. So, all of a sudden, his decision to do this very innovative thing...because Ford is, like, 120 years old or 100 years old, right?

And as CEO, he put all his chips in on this new electric drive train, which I think is awesome. I love electric cars. But, for Ford, their number one seller is the Ford F-150 truck, which is a standard truck that people love, and they love the ability that it gives them to, you know, a variety of things. But one of the main things is like, you know, you can go out and gas it up and keep on working. Whereas with the Ford F-150 electric truck, you can charge it overnight and try to find a charger somewhere else. But if you can't find a charger, you're high and dry, and you're in for an expensive tow.

So, he went in on innovation that perhaps was not ready yet, and he's probably going to lose his job over it, which he probably should, in some ways because that's the CEO. But it's just interesting because I think, yes, you need to recognize innovation and go for it, but maybe not bet the company on it [laughs].

EDDY: Well, fear of innovation is also what's going to put you in serious risk...

JUSTIN: Yeah.

EDDY: Right? Of bankruptcy. As an example, off the top of my head, and it's not a jab at this company, but it is a prime example. Blockbuster was amazing, right? Like, they were a striving company for years. Staple name, staple IP. Everyone knows the name. Even now, you could talk to someone, "Hey, do you remember Blockbuster?" "Oh yeah. They were awesome, you know." They had the opportunity to buy Netflix at the time. And then, they passed up on that because they didn't believe that that was the future. And being scared of innovation it was a prime example of why they went out of business. So, there's definitely [chuckles] a balance with fostering innovation in your company.

MIKE: Yeah. 80% of your company going into innovation is bad, and 0% of your [chuckles] company going into innovation is bad. That suggests that maybe there is somewhere in between that is good.

WILL: Well, I sort of like 20%, and I like releasing the illusion of control. I like releasing the illusion of, like...the reason I like sort of, like, unrestricted 20% time is because you're not predicating your ability to continually successfully innovate based on the smart guy that I have designated. Me, the big boss with my big brain, have said, "These four or five guys, these are the smart guys. I know best, and these guys are the people who are allowed to have ideas. And they will be smart guys and give me the correct ideas. Versus just sort of like, I don't know, we're going to try some stuff, and we're going to pick the winners, and we're going to ditch the losers, and we're going to see how things go.

It's the ego control, like, only I know best thing, which I think is endemic to top-down corporate hierarchy, and they will kill you deader than fried chicken. And I would also like to tip of the hat to Netflix because, like, Netflix, they're doing better than ever. They torched their business, remember?

MIKE: Yeah, they did.

WILL: Remember the DVDs?

MIKE: They did [chuckles].

WILL: [inaudible 28:08] dollar company, and they're just like, "Nah, we're done with that. This is stupid. This mail stuff is dumb, you know?" They are sort of, like, this sort of counterexample to, like, you know, they had a really smart guy who really got it, and he had the right idea. He's just like, "Nah, this is it." I mean, it's great as long as you have a Steve Jobs, and you can continue to have successive Steve Jobs. If you could guarantee that, you're going to do great.

JUSTIN: It sounds like we have a great conclusion here that, you know, yes, there needs to be innovation and, you know, enable our engineers to kind of run with their ideas a certain percentage of time. But is there a way that you can make them more successful? Like, granted, you know, X number are going to be unsuccessful, right? You're going to have a lot of duds. But is there a way that you can help structure their projects that could make them more successful than not? What would you do? Would you encourage project planning, or maybe a deadline, or something, some sort of guardrails, or guides that could help them on those?

WILL: I think you got to go the other way. I think people's natural inherent bias is fear of failure. I think that's baked into our DNA, our bones, who we are as human beings. Honestly, like, I think you have to encourage, like, wild, stupid failures. People don't want to...even if it's 20% time, it's like, it's cool. Fridays just do weird stuff. Nobody wants to go to their boss and say, like, "You know, hey, what'd you do for the past year's worth of Fridays?" It's like, "I just wasted it all. It was all just stupid. It was all a complete loss." Like, think about even if it's like, don't worry, it's not going to affect your performance if you fail more, not succeed more, fail more. It's just the bias that we have to correct for as human beings is a fear of failure.

MIKE: I think you're dead on, by the way [laughs]. I actually have...I'm sitting up where my kids do school. I've got a sign that says...I'm not sure I got it quite right, but I'm going to get close. "Mistakes are proof you are trying." I want my kids to hear that every day. It's so important that they know mistakes are part of this. Mistakes are great. Please make a lot of mistakes because that means that you're doing the right thing.

Now, I want you to pay attention enough that maybe you're not making mistakes that you knew better than. But I want you to be trying new things that you make mistakes on. If you're not trying things that are hard enough to make mistakes, that means that you're not really learning anything.

JUSTIN: Well, maybe that's just the answer then. It's just, like, you go in and set the, I hate saying, set expectations, but you set the expectation of, hey, go out and try this. Go make a bunch of mistakes, and recognize that you aren't going to be perfect. And recognize that, hey, go out and collaborate with your teammates and share ideas maybe with each other. Go out and ask for help. Don't be afraid to share what you're working on. And even if it's a stupid idea, you know, chatting with other people maybe...actually, this may be a separate, you know, encouragement of, like, I don't know, some people may want to go off on their own and do stuff all by themselves, but maybe encourage people to collaborate at least.

MIKE: So, encourage pairing, for example? You're going to be pairing with somebody, you know, X number of hours a week, N number of hours a week, whatever N is, three, two, I don't know. Give your [inaudible 31:27]

JUSTIN: I'm just saying in terms of your 20% time.

MIKE: Got it. Got it.

JUSTIN: Because a lot of times, if you go out and you go try to solo something, that's good. Sometimes you just got to, like, research the heck out of something. But in order to get it to the next stage, you recognize that, hey, I need some help. Let's see if I can convince some other people to join my 20% time project.

WILL: Exactly. I mean, we've been talking about a higher bar, right? The higher bar, like, your manager doesn't have to persuade you. They didn't even have to persuade you that they were your manager, somebody said.

MIKE: [laughs].

WILL: And I'm like, "Well, I don't want to work for Mike." Like, "Do you like getting your paycheck?" "Yeah." "There you go."

JUSTIN: [laughs].

WILL: I mean, like, yeah, the fact that you have to persuade, I mean, that's extremely uncharitable of me, but it's incorrect. It's wholly incorrect to say that somebody just, like, made something up. Like, there's even in management meetings, like, you got to make your case. You do have to persuade people. It is necessary.

MIKE: So, to summarize what we've come up with here, if you want to have innovation, you have to set aside some time for it. You have to make a space, and it has to be uncomfortable. That's not to say you put all your time and space for it. You budget a portion because you got to keep the lights on. You got to keep the business running. And doing most of your business should probably be the biggest part of your priority. But you better put a portion of that dedicated to doing these new things, most of which will not work out.

JUSTIN: It's also worth noting that if you aren't encouraging the innovation inside your company, the smart people are going to go out and start their own company with whatever ideas they had, so...

MIKE: I think that's exactly right.

WILL: And I think, like, as I think about it a little bit more, like, there's a level of, like, it's like venture capital, right? There's a level of scale that you need in terms of, like, sort of, like, 20% time. Like, a company as big as Google, 20% time, it's like, okay, that's a lot of different ideas, a lot of smart people. They can do stuff.

But, I mean, like, if you say, like, "Okay, 10% is going to win," and you've got, like, ten people, you're going to have to bet on having a smart guy. You can't afford a lot of losses. You need a certain scale to just sort of like, let's go! If it's let a thousand flowers bloom, okay, that's one thing, but if it's, like, let one flower bloom or five flowers bloom, you know what I mean? Like, it's going to be more difficult to be assured of a win, you know what I mean? Like, then you might have to put your eggs in one basket and, like, be more judicious about it.

MIKE: That probably just comes down though to management [laughs], and there's a flip side. Like, managers actually can do a good job if they're good. I think that's a good stopping point. And this has been a great session. And I look forward to getting together again in the next episode of the Acima Development Podcast.

The panelists explore the tension between conformity and autonomy within software development teams and the balance of standardization and flexibility in coding practices. The conversation revolves around their efforts at Acima to standardize pull request assignments and the dual roles in code reviews, highlighting their team's unique approach of using a chatbot compared with other teams using GitHub Actions. They also discuss the challenges of integrating new developers into standardized practices, especially for teams with a lot of junior developers.

Eddy emphasizes the importance of standardization across teams to reduce the onboarding time and confusion when developers shift between projects or teams and advocates for company-wide code standardization to facilitate easier transitions and maintain consistency. Dave discusses the implications of standardizing tools versus processes and how each choice affects team operations and risk management strategies.

The discussion turns philosophical, pondering the broader impacts of autonomy in development practices. Dave and Tad debate whether autonomy should extend to individual tool preferences or be constrained by team and organizational standards to ensure cohesive and efficient operations.

Transcript:

TAD: Hello. Welcome to another episode of the Acima Development Podcast. My name is Tad, and I'm the one who's going to be hosting today. Today we're going to be talking about conformity versus autonomy, or another way to put it is, how much do you need your devs to standardize versus how much flexibility are you going to allow them to have? Today I have with me, Eddy.

EDDY: Hey, guys.

TAD: And Ramses.

RAMSES: Hey, guys.

TAD: Sergio.

SERGIO: Hey.

TAD: Dave.

DAVE: Howdy, howdy.

TAD: One of the reasons why this is kind of timely for us is because we have several different services, several different teams here at Acima, and we've been making a push to standardize things between the teams and to become a bit more uniform between the teams. So, I have some examples, but I thought I'd throw that out. What are your thoughts? What are some of the things that we're trying to standardize on right now, and how do you feel about it? Or maybe I'll just go first.

So, one thing that we're trying to do is standardize how we do assignments of pull requests. And the team that most of us are on, actually all of us on this podcast today are on, we have two different roles that we have when we're doing code reviews, where one of them is a code reviewer, and that person just reads through the code, looks for anything weird or unusual, maybe asks some questions, and then they give the thumbs up: yes or no.

The 2nd role is a full reviewer, and that person goes through, reads all the code, but then they also go through a bunch of steps to actually exercise the code and assure themselves that that code is working. And that works great for us because then we have two people who are reading the code and two people who are then QA-ing the code because we also have a third QA person. But the other teams don't do it that way.

What we've done is we actually wrote a chatbot that would manage our assignments for us so that we get a mix of people doing the reviews. And not everyone gets back-to-back full reviews because, you know, you can skim through code, maybe on a PR, and that takes you 5 or 6 minutes. But it might take you 30 minutes to actually go through and exercise all that code. And so, because full reviews are a little more time-consuming, we like to make that a little more fair.

And having our chatbot and having coded up a way for us to do our code reviews works great for our particular team. But our team is the only team that does it that way. So, the other teams are all doing GitHub Actions. So, we're trying to reconcile the fact that we have a chatbot, and the other teams have GitHub Actions. I think chatbot is better, but we're outvoted in this. It's not quite a democracy. So, we're currently trying to figure out how to get GitHub Actions to work for our needs.

EDDY: I think that standardization for code reviews and testing, et cetera, is super crucial, right? Because what happens if you pick up...and I don't know if it's common practice in other companies to allow a developer to just clone the repo that the other team works on, and then just do the code change, and then just assign it [inaudible 03:35]. I don't know how common that is. But I do like the idea that Acima facilitates that. And having a standardization, I think, is important company-wide, right? So that you're not daunted with different team dynamics.

DAVE: I think there's a thing to look at here as well that the way our team works and the other teams work we're talking about standardization of, like, generic process, right? It's like there's this implicit mapping from, like, GitHub teams to their team that isn't present on ours. So, it's a struggle for us because we do things a little bit differently. And we sometimes wonder, okay, do we want to standardize on a tool, or do we want to standardize on process? And, in this case, the processes map to tools that are different, right? That don't cooperate with each other. Yeah, it's a sticky issue. Like, either one of the tools has to be upgraded so that it can do both, or one team has to change process.

EDDY: I think it's important to also distinct this, you know, that the Atlas team mostly comprises of junior developers, I do want to say, like, a good chunk of that, maybe half of the pie. Maybe that's deviated since, but historically speaking, the team that we are representing for most of us today are from the Atlas team, which is the name, right? And we adopted the idea of a full review and code review in a way to sort of make sure, you know, that the code changes are solid. And if you have a junior developer pushing out code, I would argue that the probability of a bug, you know, being released is greater since they haven't had the full exposure as a senior developer.

So, the reason why our team deviates a little bit more than others is because of how many developers, one, we have and, two, how many of those are actually junior developers. So, I think that is an important distinguishing factor to give context on why we have different procedures.

DAVE: Yeah. It's a weird thing to draw it back to, but in my mind, it all comes down to how risk-averse or how you mitigate risk on your team. I've worked in shops where everything had to be reviewed, double-checked, super QA'd, jots and tittles because there's finance in this code, like, SOC 2 compliance, where whoever wrote the code cannot deploy the code. And I've also worked at a company where we had, you know, PHI and really sensitive data. But we had the agile mentality of if you want to make a feature happen, you own it. And if that means going into another team's project and into their repo across the building, that's fine because they're doing it to us.

And the mentality at the company was if you need to change something, you go to that team and say, "Hey, I need to add this feature." And, usually, somebody on that team will say, "I can help you with that. I own that piece, and I'll be happy to walk you through it." But then they walk you through it, and then in that code, you did your own deploys. Like, you walk it all the way to production. And the risk mitigation there was if something goes wrong, you own it. Nobody else can handle this for you.

There are other places, like when you've got a team full of junior developers, where you can't own this. If this blows up, you're going to need, for lack of a better word, an adult to come help you. You're going to need a responsible adult, to put it that way. That way, we can exclude me from that group. You need a responsible adult to come, you know, fix the server for you because you took it down. And this happens anywhere you have people that have, like, principle of least access, where developers can't manage stuff, or SOC 2 where you can't do the deploy. And so, all the processes kind of vary.

Tad, you mentioned at the top that we were talking about, like, when do we allow autonomy, and when do we want conformity? And, in my mind, I think it comes to...and this is, again, more risk mitigation, but it comes down to how much of your autonomy is going to affect the people around you, right? It's like your autonomy stops at my IDE, that your feelings stop at my rights, or your rights stop at my nose, or something like that. And so, we'll see people that it's like, if you want to develop on Emacs, or Vim, or RubyMine, it doesn't matter. If you want to ship this straight to production without QA, that matters. That's a process that, you know, that there will be people at your door saying, "This is unacceptable risk."

EDDY: Well, it also depends on how robust your QA integration really is and how big of a change that really...or the impact of that change really is, right? Like, for example, I'd argue that if you're pushing out, like, a simple text change, does that really warrant a QA assignment? Probably not. They can probably allot...their allotment for them would be better elsewhere, right?

TAD: I was going to ask, I think another part of it is can you pull a developer off one service and put them on another? Because if other developers have been pulled off of our team and need to do a PR against our codebase and they don't know how to do it, right? Because it's not the same process to do a PR on their team, like, we have certain standards. Like, it's small, and we have a full reviewer, and a code reviewer, and a QA person. And those aren't the same standards on another team.

And so, you know, it's like Eli Whitney in the interchangeable parts, like, can you move devs between different pieces of your codebase? Or can you move devs between other teams? And if you don't have some standard there, then there's friction. There's an impedance, right?

DAVE: Yeah, that's fair. The shop that I was at where we had...you would...in fact, we literally had developer hostage exchanges where you would literally go work on another team for, like, two weeks, you know, for a sprint and then come back, and they would send somebody in your place.

But, like, if you went to, you know, this other team and said, "Hey, I need to add this feature," you would have somebody that would be your Sherpa and would be your subject matter expert and say, "Okay. Let me show you the part of the project you're going to be working on. Let me show you what specs are covering that code. It's going to come in here. Data's going to go here, here, here, and here. You're following that? Okay, cool. When you hit here, you're going to have to...that's your feature. You have to change that. Change this. Don't touch that. Anything else is fair game. Go nuts."

And then, you'd go off and write it and come back and [inaudible 09:28]. So, basically, you would get that basic grasp of, like, the lay of the land. At that particular shop, we would not just throw you cold into 100,000 lines of code and say, "Good luck. Have fun." That's not fun.

EDDY: I actually want to advocate for company-wide standardization for code for that same reason, right? When you get dumped into another codebase. And I think at Acima, and Ramses, you can kind of correct me if I'm wrong here, but it seems as though each team adopts their own philosophy. It's always been that way from what it seems. And the code is a great example of that, right?

RAMSES: Yeah, totally different between different teams.

EDDY: To me, that's kind of difficult, right? Because I'm a junior developer who's going over a year, right? Just pushing out code changes and stuff. And it was a bit of a [laughs]...so the codebase that I currently work on it's kind of grown a soft spot for me because I know where everything is, you know, like if I need to add a new client, I know where that is. Cool. If I need to do this, oh, I know where the service object lives, whatever. And I can just reference it, and I can be very productive.

I did a change, like, two weeks ago, give or take, on another repository, another service, and it took me, like, God, I don't know, like, half the day to really understand, like, what they were doing before I could do anything. So, it's like, for me, it's like, I felt super, super, like, I was wasting a lot of time just trying to figure out, like, how that codebase works before I even went ahead and wrote anything or implemented any changes.

So, I can't help but wonder, if we standardize conventions for code company-wide, it will just be, like, a seamless transition to go and make another change, right? But at the same time, you run into the idea of, like, standardizing something that you are not a big fan of, right? And so, I guess it's kind of, like, a give and take.

DAVE: Yeah, wanting to have everybody work on the same standard that's a lovely idea. That and five kilos of cocaine will have you creating RuboCop.

SERGIO: But it has the bad side. It's like, if we standardize something, five years later, we figure out that this is not the right solution. It's like, this happened to us. We have this tool that we were kind of removing right now. I guess Tad is talking about that. I don't know if I can talk freely about that here, but it's like, we have this tool we built. I don't know who built in the team this tool. And we have a bunch of services using this tool, and we need to remove this, like, five years later. And it's like, whatever you make right now is going to be outdated in five years. So [laughs], standardizing is good, but sometimes it's [inaudible 12:09], you know, yeah.

DAVE: I think it's important to be careful not to set the standard for all time. We spend so much time as developers trying to tie the hands of future us, which is dumb because future us is always, maybe not smarter, but always more experienced, always older than us. And, honestly, legitimately, my biggest frustration as a developer is when I have tied my own hands. I'm like, I need to change the way this works.

I had a knockdown drag-out with a developer once over whether or not you could order the keys in a hash because a hash is a mathematical function, and the set of keys of a hash is a set. And sets, by mathematical definition, must not have an order. And I'm like, yeah, but the customer wants stuff to come out in the same order that they put it in. Customer gets what they want. Don't try to teach the customer set theory, right? It's tomatoes might be a fruit, but you don't put it in fruit salad. Don't do that.

EDDY: Yeah. Can't you organize a hash, though? I'm pretty sure you can, and it's --

DAVE: By default, Ruby and Python they keep a list, an ordered list as things get put in. Yeah, they have a hash for quick access, and then they have a list for ordering.

EDDY: So, it seems like a pretty trivial change, right?

DAVE: But ten years ago, you could just about have an honest-to-gosh fistfight with the math majors over whether or not you should be allowed to do that. If you can't guarantee something, there are programmers who will spend a lot of time locking things down so that they can guarantee that you can't, which is not the same thing as saying, "You can't guarantee." They want to guarantee that you can't. Now it's locked down. It's perfect. It's airtight. There's no way anybody can do anything wrong with the system. The problem is that every future change is something wrong with the system, right? It's something that hadn't been thought when that first mathematical formula was locked down.

TAD: I was just going to go back to ask Eddy some follow-up questions. So, first of all, when you say having the same conventions, is that RuboCop, or is that style guides, or is that training? Or like some of the people mentioned already, what if you hate those things that the other team does and you're glad you're not on that team?

EDDY: [laughs].

TAD: And you can do it your own way on your team. And maybe a related question that I'm going to throw out to the group is, is there some value to having competing ideas? If everything is genetically the same in your population, then that means some plague can come through and kill everything, right? And so, diversity does have some value where you have competing ideas, and the ones with merit kind of float to the top over time, and then they fall out of disfavor, and new ones kind of clobber those.

EDDY: Well, I think, okay, so it's more of a selfish ask for me than anything else, and I'll preface by saying that. Because, again, like, that was my first exposure contributing to another repository to such a degree, testing against a different codebase. And, again, like, the fact that I had to spend, like, half the day, you know, figuring out, like, their placement, you know, like where they locate a bunch of their code and stuff, and then how they write it; their style is a little different, too.

And so, selfish me would just be like, cool, if we can all just agree on, like, being explicit with our return nils [laughs], you know, like, and that's just adopted across the board, you know, you'll just look at it, and you're like, "Okay, yeah, cool, that makes sense." And I wouldn't have to stay there longer than I need to trying to figure something out, right? Or, like, using optional params, you know, like, in our team, we prefer it. We love it, you know. But other teams are like, eh.

Or, like, a really hot topic is, like, factories versus mocking, you know, it's sort of like...and so, for me, it's like, if we can standardize that, you just know what to expect when you go into the other team, you know. And you don't have to worry about, like, understanding their concepts, especially if you're just there to make a simple change. Like, if you're just there to do, like, a simple change that you need to work and they don't have the resources on their team to do it, you know, suddenly, your task is two times or three times longer than it would have taken the other person to do it, right? And it's just kind of hard to do that, you know, if everyone has their own lick.

TAD: I'm curious: are there some things that we maybe should standardize on and some things that are just trivial but we should standardize on? Meaning, we're always going to paint our bike sheds red, and that's just our standard color, right? So, for example, I know that we've had, in our codebase, like, four or five different HTTP libraries. And really, all the HTTP libraries do the same thing. They're a little bit different flavored, but they all make POSTs; they all make GETs. They all take a URL, right? Like, there's not a whole lot of difference.   And so, would something like that make sense to standardize across all the teams? Where it doesn't really matter, but we should pick something? Are there things like that that we should be considering?

EDDY: I've learned that if Rails can do it natively, use the Rails convention and don't try to get fancy [laughs].

DAVE: Basically, it's the same thing. It's like, bike sheds are red until further notice, or unless specified otherwise for a good reason, right? So, it's like, you can go to the water cooler and talk about how you should do it all day, what color do you want to paint it. That's fine. It's going to be red, unless there's a really good reason, and that's fine.

And for, like, HTTP, my out-of-pocket answer is, yeah, you should standardize because, you know, convention over configuration. Where we run into trouble is when we have convention and screw your configuration, which is kind of where Rails is kind of headed. It used to be, like, Rails 2, Rails 3, man, you could change anything. You could back that thing onto a wacky data store and just, da, da, da, all kind of...and everything was super configurable. And most of that's still there, but it's getting more and more complicated and more ceremonial to change those things.

The thing, specifically with HTTP, I love the HTTParty gem. It's the most simple, straightforward, like, httparty.getURL and response. And you're just done. There's no setting stuff up. There's no socket handling, da da da. Just do it and go. And most like Faraday and the other ones will, Excon, they'll let you do that, too, but it's a little wordier. But HTTParty had, at least as of, like, 2017, had a threading bug. I want to say it used green threads, which is Ruby's internal thread manager, which runs inside one system thread, so it doesn't thread well. You stack up ten threads, and it's exactly as fast as running one thing ten times in a row. There's just no speed-up whatsoever.

And so we switched to Faraday, which at the time did not have just Faraday GET. Like, Faraday 1.0 didn't have that. I could be wrong on that, but I didn't see it in the code. It was always Faraday dot connection do, and then inside the block, do your thing. It was ugly, but we needed it for performance. And so, that's an example of no, this bike shed has to be blue. It can't be red. It has to be blue.

EDDY: You know, I'm starting to think...as we've been talking, I'm trying to think of all the things that we can typically, like, standardize. And, like, some of that is enforced by RuboCop. Is it not? I mean, most of that should be enforced by RuboCop, assuming that your repository uses it, right? So, it really is just a matter of all adopting the same rules.

TAD: The same linter and the same rules for your linters.

DAVE: And to be clear, I love Rubocop. It's just I want to kill some of the cops that we favor as a team. I'm like, I hate that we made this choice.

EDDY: [laughs] I agree. You know, what's crazy is I used to be a huge fan of comments, right? And, like, anytime I wrote a function, you know, I found myself writing a comment about it, even though it was pretty self-explanatory. Until someone basically told me, like, "Your code should be explaining it to you, you know, from the moment you read it." Like, you shouldn't have to dig into the nitty gritty to be like, "Oh, that's what you're doing. You're doing something really fancy and stuff."

So, someone told me, and I can't remember who it was, and I kind of want to put them on the spot because, [laughs] like, that's rang true for me forever. It's pretty much what they said. It's like, if you find yourself making comments to a code, it probably means nine times out of 10 that it's more complicated than it should be, and you should consider refactoring. And that's rang to me true, you know, and for a while.

TAD: I've had an exact opposite experience where people tell me, "Oh, my code is so readable. It doesn't need documentation," and I beg to differ.

EDDY: [laughs]

TAD: I go in...and, like, "I don't need to document my code. It's self-documenting." And when I hear a phrase like that, I cringe.

DAVE: To steal a joke from [SP] Louis C.K., readability, readable, you know, writing readable code, it's like being called a jerk. If you say, "I'm not a jerk," you don't get to decide that. Other people get to decide that for you, right? So, if somebody comes to you and says, "Your code is really readable. I don't think it needs comments," then, yeah, especially if your comment is literally, do the thing, and then the line of code is very clearly do the thing.

It's especially from, like, the old C and C++ days, where you had to sneak up on your solution. You had to trick the compiler into giving you what you wanted as a side effect of the math that you were throwing at it. So, we had a piece of code go by this week where we have a certain number of products that can be in a list, and there was just one thing. There was only one kind of product. Well, now there's two things that can be in it.

And so, somebody stuffed in a bit of code that said, "Hey, if the list of products is longer than one, it has to be this combination." And I'm looking way down the road map, like, six months, nine months, ten months when we add a third product or a fourth product to this thing, and this is going to be wrong. Greater than one that's sneaking up on the side effect of, well, if you have two products, you've got more than one. Therefore, it's this comment.

And Sergio and I changed it so that it was like, "Hey, if you have this product and you have that product, then it's that combination of those two products." It's the same code. It does the same thing, but now I think it kind of doesn't need the comment on it, def combo item has item 1 and and has item 2. That's literally the code. I'm changing the names of the products to be a little vague here, but other than that, that's literally what the code is.

TAD: I should maybe clarify what I like in a comment. If you give me the mindset you were in when you wrote that code, the reason you made the decision, why you took the approach that you did, I love those type of comments. If it's, this adds A and B together and returns the total, then, to me, that's a useless comment because I can see what it does, but I rarely know why you chose to do it that way, what the trade-offs were, what your mindset was, things like that, when you wrote the code. And those types of comments are rare but invaluable.

DAVE: Yeah. Like, you have a thing that's like, A add B using this special adding method. So, you've got this, you know, conglomerate A and B, and the comment is, "Level four figuring compliance." And you're like, oh, that's why we have to use the custom thing to do it, right? And otherwise, it would be just this weird addition.

EDDY: You know, Ramses, you wrote something that's ringing true for me right now, and I kind of want you to elaborate on that, if you don't mind. Because I, too, agree with your sentiment, but I want to hear your opinion behind it.

RAMSES: Regarding the class-level comments?

EDDY: Yep.

RAMSES: Yeah. Well, there's just a lot of times, like, you'll see in a model, or a class, or whatever it's basically just a repeat of what the class name is, like, "Representation of user." Yeah, that's what your model is.

DAVE: It's just repetition at that point, and you're only putting it there because RuboCop demands the documentation. When I document a class, I try to say why. Why does this class have to exist, or what purpose does it serve? And you can...yeah, you can have...I can't think of an example off the top of my head, but you get the idea that the comment should not just be if you've got, you know, class particle accelerator and a comment that says, "A class for particle accelerator."

EDDY: Yes [laughs].

DAVE: It's completely useless, right? But if you've got a comment that basically says, "General class interface for all accelerometrox products," then you're like, oh, okay, now I know why we have a particle accelerator class.

TAD: So, going along with this, would you guys advocate for a standard commenting style? Because there's people that use YARD. There's people that use RDoC. There's people that say, "You must state your inputs, your outputs, and give a why for this class, or this method, or whatever you're commenting."

DAVE: I would say, "Yes," if it's high ceremony, if the interface needs to be...like, if it's a Conway's...[chuckles] Conway's wall, not Conway's Law, Conway's wall, that's like, another team is going to come use that piece of code. It should be documented like the old MSDN libraries where you could hit F1 anywhere in the code in Developer Studio and get, you know, a page of documentation and examples, and gazintas and gazotas, and every possible thing that this thing can do. Yeah, document the snot out of it. If it's a private method buried down deep in some sidebar class that only affects your specific domain in that module of that sub-piece of the code, I don't know, maybe not.

TAD: So, are you saying for things like our shared gems you would advocate for some kind of commenting standard?

DAVE: I feel like you might be baiting me into a trap here, but at the risk of sounding like a complete hypocrite, yes. What I will say about us hypocrites, [crosstalk 25:56] criticize hypocrites how you want, but we do have higher standards than you.

EDDY: So, you know, it kind of begs the question, right? So, like, standardization and comments, I feel like we don't have that, and I'm not very spoiled or privy to that, right? Because, again, I'm a junior developer, not much of experience in the industry. But I do love hitting, you know, certain logic, you know, and they basically say, "Hey. Make sure you keep this in mind when you make a change here or whatever," right? I love that.

I love that because they're being super, like, they're being super nice to me when they're saying, "Hey, dude, if you're going to make a change here, like, just make sure you're cognizant about this, this, and this, you know." And comments do that, and they do accomplish that. And so, I feel like we don't have a standard on that, but I feel like we should, right? I think it is nice.

TAD: So, I'm curious to ask about standardizing on some other aspects and how you feel about it. How would you feel about doing standardization of all of our setup stuff for all of our different services?

EDDY: Oh, that would be nice.

TAD: So, would you be okay with forcing everyone to use Docker or some other system?

RAMSES: Sure, why not? Make the process much easier.

EDDY: Okay, so I have a love-hate relationship with what you just said, right? Because there are people who utterly refuse to use Docker, right? Because their machine just isn't strong enough. So, like, why not provide an alternative for individuals who just don't have a beefed-up computer, for example? Or not everyone uses the same operating system. And what if your documentation is only catered to a certain OS? Then, at that point, you're kind of like, "Ugh," you know, so I don't think you can truly standardize --

TAD: Well, you can standardize hardware. You could say, "Everyone must have an Apple laptop."

EDDY: Wow. Yeah. I didn't realize that that was possible.

RAMSES: Yeah. Tell everyone to start using Electron products, in-line, editors only.

EDDY: [laughs]

TAD: Yeah, like, would you guys be okay with saying, "Everyone uses Linux"? Because that's something that pretty much everyone on our development team would be able to use and that has maybe the best Docker support.

EDDY: I haven't had an issue with Mac. Basically, anything that I needed to accomplish, like, Mac has been able to provide for me. So, like, there hasn't really been a need for me to be like, oh man, I really wish I was on a Linux [laughs] distro. There hasn't really been a necessity for that. I feel like that's because macOS has just gained so much credibility for development. Am I wrong on that assessment?

TAD: Well, we have a lot of contractors that have to bring their own laptop. And so, you have the luxury of getting a company-issued laptop. So, you're indirectly advocating that our contractors would go out and buy Apple products.

EDDY: It's a good point.

TAD: Standardization has trade-offs, right? It's like, it would be great if we all ran on the same hardware, but then everyone who doesn't have the standard has to go out and somehow now fit to the standard.

EDDY: I feel like it's easier to enforce standardization on code than anything else. Like, if you're standardizing something on, like, hardware, you know, I feel like that's super trivial, and you may lose potential talent because you're telling them, "Hey, you're forced to use this specific machine, this specific operating system." So, like, for example, I've paired with our contractors who were from India and some of them, yeah, and like you said, they use Linux. They use different IDEs. Like one of them uses Sublime for God knows what reason, you know.

TAD: Because it's fast and awesome.

EDDY: [laughs] And the thing is, is, like, you would see him just kind of, like, flow through that seamlessly because he understands that super well. And I'm just like, cool, we shouldn't be forcing you to be using something if your innate flow caters to a specific software, so...

TAD: But would it make it easier for you to pair with him if you had standard tools and standard hardware and maybe even, like, a standard IDE that you both were using?

EDDY: Well, I want to confess something on record...Ramses because he's on the call. When we first paired, when I was learning how to code, he would pair with Vim. And for most of the time, like, early on in our pairing, I was like, oh man, I was like, what are you doing? Where are you at? I don't even know what I'm looking at. So, there is a level to that complexity, but I feel like eventually, you kind of just grow accustomed to looking at someone else's IDE, and you just know what happens. You know, I feel like, as wisdom prevails, it's easier to adapt oneself.

TAD: What about you, Ramses? Side note, Ramses is a Vim user, and he loves Vim. So, I'd like to hear what his thoughts are.

RAMSES: It'd certainly be easier if everyone was using the same tool in terms of collaboration. If I want to do a live sharing with someone in Vim, it's technically possible. It's not great, you know. It's not a great experience. It's certainly not as seamless as, like, Live Share on VS Code or live coding on RubyMine or, you know, any other tool. If I'm planning on doing a pairing with someone, I'll just open up VS Code. It's easy enough. It doesn't impact my workflow that much.

EDDY: And to your credit, Ramses, I think you tried using it entirely for a month, right?

RAMSES: Yeah, I've used it, like, I mean, I've used a wide variety of IDEs and other editors. For the most part, I don't have a problem with VS Code. It works just fine, for the most part --

TAD: I think the trade-off here is individual productivity, right? If you let developers choose what they want, then they'll choose the most productive, most familiar environment that they have. But if you standardize, then you might be increasing, like, pairing productivity or some other form of efficiencies, right? Like, if our servers are on Linux, they're all running in Docker. And so, if everyone ran Linux and everyone ran Docker, it would make it a lot easier for our DevOps team. So, are you willing to choose the approach that makes our DevOps team sad, Eddy?

[laughter]

EDDY: I feel like there's not a right answer [laughs].

TAD: I'm kidding. No, it's just hard because there's trade-offs with all these things, right?

EDDY: Yeah, I think, to a degree, there has to be a balance between standardization and not. And I feel like I've been spoiled [chuckles] here at Acima because everyone gets to customize their machine, you know, to a certain degree. You know, it helps them with their flow. So, like, everyone can select whatever IDE they want to use. As long as they get the code out, it doesn't really matter. Or, like, some of us prefer to use different Git commands, you know, when rebasing or pushing stuff up. Like, that's fine.

Like, ultimately, it accomplishes the same thing at the end of the day. So, I like the fact that I'm not micromanaged that I have to choose certain hardware and, like, certain software. Because at the end of the day, the one who's pushing up the code changes is the developer, right?

TAD: I'm going to hit you guys with another one. How do you feel about standardizing on time? Meaning you do things at a certain time, and other people do things at a certain time.

EDDY: What do you mean?

TAD: Meaning, everyone starts their day at the same time. Everyone has their meetings at the same time. Everyone pairs at the same time. Everyone takes lunch at the same time. Everyone codes at the same time. Everyone does PRs at the same time. You could have a pretty regimented schedule and make sure that everyone's pairing at the same time, which means that you never have to coordinate with people to say, "What's your schedule? What's my schedule? Oh man, we're not going to be able to get together on this." Because it would always be free. Like, you could really standardize when certain things happen and how things work for your team.

EDDY: I don't really like that. And I'm just going off of my first reaction here. But I get hungry in different times than other people. So, like, what happens, like, if suddenly we're all forced to have lunch at 12:00 p.m.? But I'm like, "Oh, but I usually don't eat till, like, 2:00 p.m." Suddenly, it's just I'm not being productive because now I'm having to force myself to go against whatever my habits are.

Or, like, what if I'm like, I'm not very productive waking up at 7:00 in the morning and working at 8:00? What if I work better working from 9:00 p.m. or 9:00 a.m. to, like, 7:00 p.m., and that's the schedule that I prefer? And, like, as long as you get the work done, it really shouldn't matter, I feel like. And, like, time management should really be left off to the developer, in my opinion. Just, like, from the first reaction, I feel like the developer should be the one who pretty much decides the time management.

TAD: Okay. Ramses?

RAMSES: I think I am in the opposite camp, at least for the most part. I don't really care when other people start their day. If they want to start it later, end it later, it doesn't really faze me. But I think the parts of standardization that's good for time management is where...and this is something that we follow as an organization, where we have, like, a cadence, and we have structures like, okay, all teams do stand-ups in the morning, you know, between, like, 9:00 and 10:00, or whatever, you know.

And then, in the afternoon, okay, that's developer coding time, so no one else can schedule meetings during that time. Like, I mean, you could probably do pairings with other people during that time because that's when you should be productive, but it's not time for engineering-wide meetings or team meetings. Those are kind of blocked out at very specific times. I think that's helpful for team collaboration.

TAD: Yeah, I mean, I've kind of been playing devil's advocate or being a little bit provocative here, but the reason it did come to mind is because we did, a few months ago, agree on a team cadence where certain things would happen at certain times just because they found that they couldn't schedule things. They couldn't plan things because they never knew when people would be available or not. Or, on the other side of that is, you didn't know when you would have free time and when you'd get pulled into a spontaneous meeting, right?

EDDY: Right.

TAD: So, are there times that you think should be standardized, like meeting times, things like that?

EDDY: [inaudible 36:52]

TAD: Maybe not lunch. Maybe you don't dictate lunchtime.

EDDY: I want to go on record and say I don't really like meetings, personally.

TAD: [laughs]

EDDY: I have limited attention span.

RAMSES: I don't think anyone [inaudible 37:03] meetings.

EDDY: Dude, like, if I get put...and this is just me, like, if I get put in a meeting that's an hour long, after the 30 minutes, you know, I'm kind of wandering, doing something else just because it's kind of hard to focus [laughs]. So, maybe standardizing meetings so that it's short and concise sounds kind of nice.

TAD: How would you feel about something like meetings are standardized at 50 minutes long, and you always have to allow for a 10-minute break? You know, meetings always start on the hour or on the half-hour, and they are always, at most, 50 minutes long. And you always have to give a 10-minute gap before the next meeting.

EDDY: That sounds awesome.

TAD: Personally, I've been in back-to-back to back-to-back meetings where I've had five hours straight of meetings. And my brain is burned out by, like, hour three or hour four. So, would that kind of standardization be beneficial where you standardize the length and the style of the meeting, or maybe even the content of the meeting? Do you say, "If you're going to have a meeting, you have to have an itinerary for that meeting. You have to give everyone the bullet points of the meeting beforehand and say, 'This is only going to be 50 minutes'"?

EDDY: I would agree.

RAMSES: All meetings need to come prepared with a slide presentation.

EDDY: [laughs].

RAMSES: And it has to fit on three slides only.

EDDY: You know, I'm trying to think because, like, I've looked at other code, right? And I'm thinking, huh, the other things I've noticed that they have conventions on is, like, camel case versus snake case and, like, naming for your functions and methods and variables and stuff, you know. I think we can all agree that we share naming conventions for that. Some of the really silly ones are, like, oh, your characters can't...your line can't be longer than X amount of characters. That's where I'm sort of like, ah, you're forced to rewrite the code differently and probably run the risk of it looking worse because you're trying to fit [inaudible 39:13]

TAD: Well, arguably, if your line of code is 120 characters long, it's maybe a little too complicated. It could be broken into maybe a few lines of code.

EDDY: Yeah, maybe, maybe. Unlike silly conventions that, you know, that we have in our codebase, where I'm just like, oh, man, but it looks better with the unless [laughter].

TAD: Yeah. It is nice to read it all as one complete thought rather than three partial thoughts, right? I mean, there's kind of something going on sometimes in the code, where you want to break it into, like, it almost compares to, like, sentences and paragraphs. And you're like, ugh, like, I just want to express it in a certain way.

EDDY: Yeah, I agree. You know what's crazy? Is because sometimes those standardizations aren't present, I've learned to look at a code, and I'm like, oh man, this reads just like Ramses.

TAD: [laughs]

EDDY: And so, like, I'll go to the code. Like, I'll do a git blame. You know, I'm like, oh, it was Ramses. You know, I can just pick out, like, people's styles, right?

TAD: Do you think it would be better if you removed that fingerprint, where you can't quite tell the style?

EDDY: Yeah, right? So, like, if you're standardizing stuff, then it also removes, like, ambiguity because everyone's following suit, you know. And, to me, it's kind of fun to just kind of look at a code block and be like, oh yeah, this totally reads like Tad, you know [laughter]. And you're like, 9 times out of 10, you know, like, you're right.

TAD: I'm curious: how do you guys feel about standardizing on policies? For example, what if you had a policy, "No deployments on Fridays"?

EDDY: You know, a wise person once told me a little while back, and they said, "If you're afraid to deploy your code on a Friday, then your code is probably not ready to be deployed." Like, you got to be confident with your changes, and if deploying on a Friday raises eyebrows, then is it really ready for production? Probably not. I mean, would you both agree with that sentiment?

TAD: Well, I guess, just to clarify, so your argument is that perhaps standardizing policies is a symptom for a different problem.

EDDY: Maybe.

TAD: So, in your argument, the problem is we don't have confidence in our processes and our code quality, so, therefore, we make a policy to protect ourselves rather than fix the confidence problem or the quality problem.

EDDY: I feel like that's not what I said [laughs].

TAD: Okay. Well, then restate. I'm sorry if I'm misrepresenting you. Go ahead.

EDDY: [laughs] I was just saying, like, it's not the symptom of bad code or anything. It's just a matter of I feel like the reason why some certain teams prefer to not deploy on Fridays is because you do...regardless of how robust that code has been tested, you always do run a risk, you know, of it breaking something, like --

RAMSES: Yeah. No one wants to work on the weekends.

EDDY: Exactly. So, it's not necessarily because it's bad code. It's just you do run the risk of it breaking something, and if you do, you're making people work more time than they have to. So, it's more or less just being nice.

TAD: Are there other policies that you wish that we would implement that we don't have right now? We just barely talked about today in architecture meeting; for example, we floated the idea of...so, just for some context, we have a staging environment where we can deploy things, and kind of test things out, and try things in an environment that feels a lot like production but doesn't break things like production would, and it's a bit of a bottleneck. And so, now we're saying if you're going to take a lot of testing time, save that for the afternoon. And if you're just doing a quick check, do that in the morning because that alleviates a bit of a bottleneck that we've had.

EDDY: Yeah, I think that's a good problem to solve, right? How do you both feel about standardization on, like, how big a development team can be? If your development team comprises of over 20x, you know, developers, like, how do you standardize, you know, and keep all your ducks in a line, you know, when you have 20x people pushing [inaudible 43:34] on these, right?

TAD: Yeah.

RAMSES: Like, Amazon says, like, the team should only be the size that it takes to feed, like, two pizzas, so, like, four or five people generally.

EDDY: Really? That's their comparison [laughs]?

RAMSES: Yeah [laughs].

EDDY: Well, what happens —-

TAD: Yeah, they say if your team is too big to be fed by a pizza or two, then you need to split it.

EDDY: But what happens if I can eat a whole pizza? Does that mean that I don't need other people?

TAD: [laughs] They're theoretical pizza-eating people [laughter]. Yeah, it's a way to not necessarily have a fixed number but more just, like, a sense of, yeah, this team is too big, right? It's kind of more rule of thumb rather than saying exactly five. They're like, "Ooh, we're not going to be able to feed this team anymore." We are maybe too big. It is analogous, to some degree, to, you know, feeding a team, meaning providing work and coordinating that work with the other people, right?

EDDY: Yeah. And you sort of lose that as the development team grows, right?

RAMSES: Yeah. Yeah, it seems to be harder to collaborate and for decisions to be made on a larger team.

TAD: Yeah. And I think just because standardization is kind of the topic, the bigger the team, the more of these protocols, the more of these policies, the more of these standardizations, the more things that you need to make that team work.

EDDY: You know, I want to do a comparison, and I've never done the research on this. But, for example, if I were to look at, like, a Fortune 500 company, like, I don't know, like Tesla, Google, Amazon, et cetera, how they structure their development teams and how they standardize certain things, I think it'd be kind of cool to contrast. Like, off the top of your head, do both of you kind of have an idea of, like, how they differentiate themselves from startups?

TAD: I've never worked for a company that big, so I don't have any comparisons.

RAMSES: Yeah, same.

TAD: But I think Amazon still follows that rule of thumb that if you can't feed your team with a pizza, then time to split the team.

EDDY: But yeah, to kind of just come full circle, I do believe, you know, that there should be a balance, you know, on standardization, and there's an argument to be said for both.

TAD: Do you agree with that assertion that I just made that the bigger the team, the more standardization that you'll need to have in order to have that team work together?

EDDY: Well, the thing is, it's easier to enforce, right? So, if you have a standardization on code, then that just becomes an expectation, you know, regardless of if you're new, or you're a senior on the team, et cetera. So, being able to enforce certain conventions, you know, pretty much guarantees that that philosophy is being followed, you know, and it's easier to review and test.

TAD: Yeah. Well, I guess what I'm asking is, if I'm a team of one, how much standardization do I need? Do I need any?

RAMSES: Zero.

TAD: If I'm a team of two, how much do I need? If I'm in a team of 30 [laughs], do I need to add a lot to my linting rules if I'm on a team of 30 than if I'm on a team of 2? Or we can just say, like, "Oh, we kind of like it this way," and then we go with it?

EDDY: I feel like --

TAD: I'm kind of just throwing that out there because --

EDDY: I feel like Ramses has an awesome...you have an interesting opinion, right? Because for the listeners out there, Ramses is part of a smaller team than what me and Tad are in. And his team they finally have been able to claim their own service. And so, they've been able to sort of have a say on the way the structure of that service is written, you know, and architected. So, I feel like you may have some opinion on that, Ramses.

RAMSES: Yeah, I think the standard should be the same, regardless of the team size. It would be slightly more difficult to communicate the standard and probably to enforce it on a larger team. It's just when you have to communicate to, you know, 30 individuals instead of just 2 or 3, it's a little bit more difficult, but it should have the same standard, I would imagine.

TAD: The same standards because we're all part of the same company? Same standard just because that makes it easier for you to work with your, you know, coworkers? What's the reason behind that?

RAMSES: Probably for easier to work with your coworkers. And it could be the same company, but in our organization or in our department, we have different teams that work on different tech stacks, so creating that standard between, you know, Ruby and Java or whatever, it's a little bit different. But all the teams that are using the same stack, I mean, they should generally have the same standards.

TAD: Just because it reduces friction or communication overhead, or what's the...?

RAMSES: I'd say because it...yeah, it does reduce friction, and it increases cross-team collaboration. Like, if I wanted to move to a different team, if we had the same standard, it'd be really easy, or easier at least, if we were using the same stack.

EDDY: I agree, because it's like, cool, I'm used to using this certain gem to do a certain logic, and, suddenly, oh yeah, this service doesn't. They'll use a different gem. And you're like, oh, okay, cool, now I have to do some reading on their documentation [inaudible 49:09] how this works, right? Or, like, cool, we do our schema validation differently, you know.

And so, suddenly, now you're like, okay, now I have to read that documentation in order for me to come up to speed. And it's sort of like, if we can all agree on standardizing on what gems are being used, what libraries are being used, et cetera, it will make it loads easier to jump across teams.

TAD: What about everyone puts their documentation in the same place?

EDDY: That would be in an ideal world.

TAD: Because I know some teams put it in the docs directory, some people put it in the README, but some people put it in the Wiki, some people put it in Confluence, right?

EDDY: Right. So, yeah, in essence, I can agree on both [chuckles], on non-standardization and on standardization.

TAD: Okay. Well, I think we're at the end of time. So, final thoughts: how much standardization do you favor, and what areas do you think you favor standardization in?

EDDY: I think, as far as software is concerned, it should be pretty much free game.

TAD: Software like the tools that you're using?

EDDY: Yeah. For example, like, if you're trying to do some changes on, like, messaging for, like, Kafka, there's, like, a gazillion [laughs] software clients that do that. And you're like, okay, cool, we're all forced to use Lenses, or, like, Conduktor, or something. And you're just like, ah, but, like, I've never used it before. This sucks, you know? And, like, now you have to learn, like, a new...so, like, that just causes a little bit of friction.

So, like, I feel like I'm in the camp I'm like, cool, choose your own IDE. Yeah, choose your own Kafka consumer. Oh yeah, choose your own whatever, as long as you pretty much get the same result. At least that's my current opinion, and it could change tomorrow [laughs].

TAD: All opinions subject to change with new information and evidence. Ramses, what do you think? How much standardization do you like, and in what areas do you like standardization?

RAMSES: 100% conformity. Everyone uses Vim.

TAD: [laughs] Everyone standardizes to what your preferences are.

RAMSES: Or to Nano. I guess if I'm going to make everyone's life hell, I might as well make it as worse as possible.

TAD: Yeah, the least common denominator thing that everyone equally hates is what we'll choose.

EDDY: And if they don't like Vim, they're wrong.

RAMSES: No. Like Eddy said, there's a balance to standardization. I don't think it's necessarily an easy answer, depending on what you're trying to standardize. But I think, you know, at least having the conversation and trying to make a decision is probably the right thing.

EDDY: Agreed.

TAD: Yeah, I think after this discussion, and probably related to what Dave was saying earlier about convention versus configuration, is, I think I do like the hybrid where you have a recommended way, and you standardize around that recommended way, but you don't necessarily enforce the recommended way.

It's like we recommend Docker, but if you want to run Kafka natively, then knock yourself out, but you're not going to get the support. Or we recommend that you, you know, do X, and if you're not going to do X, then knock yourself out but realize that you're going to have a harder time with it. Cool. I think that's it.

EDDY: Thanks, everyone, for listening to us rant.

This podcast episode from The Acima Developer Podcast crew dives into the concept of developer habits, focusing on how habits are formed and their importance in software development. The discussion begins with a recap of the previous episode, emphasizing the "four laws" of habit formation—making it obvious, attractive, easy, and satisfying—as presented by Justin. These principles are highlighted as foundational to acquiring and maintaining good habits that can significantly impact one's personal and professional life.

The conversation shifts to personal experiences and practices that contribute to productivity and good habit formation in a software development context. Matt shares his perspective on taking ownership and how it has been pivotal in his career. He references the book "Extreme Ownership" by Jocko Willink, stressing the importance of accountability in software development. David expands on this, discussing how owning the success of a project often involves being open to feedback and adjustments, even if it means stepping back from one's own work.

The latter part of the discussion delves into Test-Driven Development (TDD) and its role in productive software engineering practices. Eddy brings up the challenge of spending too much time refining projects instead of coding, questioning the productivity of such tasks. This leads to a row on the efficacy of TDD, with several participants, including Matt and Justin, defending its importance not just in writing quality code but also in understanding problems more deeply and fostering productivity. They argue that TDD, when properly applied, not only helps in reducing bugs but also in enhancing the developer's understanding of the codebase, ultimately leading to more productive coding practices.

Transciption

DAVID: Hello and welcome to the Acima Developer podcast. We are doing a part two on developer habits. Justin, you kicked us off last week. I'm going to pick on you again. Give us a quick summary on, like, what are the elements of a habit so that we can kind of just jump straight in. Last week, we spent most of the time talking about, like, why and how of habits, and we didn't get into too many specific engineering habits.

JUSTIN: Right. One of the main things is, and here I'm just going off a table of contents here that perhaps is a good, quick summary, there were four laws that kind of make up a good habit or how you can acquire a habit. And the four laws were: make it obvious, make it attractive, make it easy, and make it satisfying. All of those require perhaps a little bit more work than just the desire to do something. If you actually go through and do what's needed to follow those four laws, you are much more likely to actually acquire that habit and have it make a change in your life. That's, like, the two-minute summary.

DAVID: It's awesome.

JUSTIN: [laughs]

EDDY: I kind of want to, like, punt this to individuals who weren't present in our last just to kind of get, like, a fresh perspective (if you're okay with that, Dave) just to see, like, what other people's opinions are on what makes them more productive.

MATT: I'm happy to talk about it. For those of you who do not know who I am out in GV land, my name is Matt. I'm one of the engineering leads over here at Acima. For me, one of the most important things that has helped me form really good habits over my career is taking ownership. I think it's extremely important in this industry and, you know, just in life in general.

There's a book, maybe some of you may or may not have read, called Extreme Ownership, written by Jocko Willink, and he was a Navy SEAL, I believe, platoon leader. And he's written a couple of books, but that one in particular really helped me. And the preface of it is that we just need to take accountability and ownership over all of the things that we're doing. And in software, part of that is not getting married to your code but having ownership of your code, have some pride in your code, take ownership of bugs, anything that arises with it. And I feel like that's going to help you write better software and think through your problems up front.

DAVID: I like to think of that by saying I want to take ownership of the success of the project. And sometimes, that means letting somebody rewrite my code or somebody push back on my design because cash flow doesn't care about your pretty, little design. Cash flow cares about whether or not a customer can get through the system. And so, when somebody pushes back on me, even if it chafes, I'm often very quick...and this might be an internalized habit, but I try to be very quick to say, "What do I really want here? Oh, this hurts, but it's getting me what I want." So yeah, I'm on board with this.

EDDY: So, I wanted to kind of throw something out there, especially for people who have more experience than I do. The past few weeks the project that I'm currently working on and one I'm involved in has required a bunch, a bunch, a bunch of requirements in order to finally get some work done and finally start writing some code.

Here's the thing, and here's kind of where my perspective lies is: refinement, at least for me at a high level, isn't productive. For me, I've always defined productivity as how much code you're pumping out, right? Because those are the metrics. Those are what really define you as you doing your work. So, if most of your time is consumed on refining a project, for example, can you really, at a high level, consider that a productivity effort, and does that really help? Can you constitute as that? Is it kind of, like, a gray area between you...it's a given. In order for you to be productive, you need to have something defined in order for you to work, right? So, I feel like I know the answer, but I want to get people's opinions when it comes to productivity.

JUSTIN: Like, a classic bad metric is lines of code. I could be super productive. I could just sit with my hands on my keyboard and just churn out code, but that doesn't necessarily mean anything, right? Like, it has to be quality code that meets demand or does something, right? Like, you can't just say lines of code or how much code I've written. I think if you measure that way, you're going to end up going the wrong direction.

DAVID: Goodhart's law says, "When a good measure becomes a goal, it ceases to be a good measure."

MATT: Yeah, I think refinements are every bit as important as the actual code you're writing. When we get requirements, one of the key things about refining this work with the engineers is making sure they have a good understanding of the problem. Without a good understanding of a problem, you can't solve a problem. And if you front-load your work with a good refinement, make sure that your requirements are defined, and clear, and concise, you're going to write good code.

It's just like test-driven development, right? If we're writing our tests first with a good understanding of the problem and a good understanding of the outcome, that almost writes our code for us. Versus if we're doing that after we're writing the code, then we're writing tests to pass against the code we just wrote, which is kind of contrary to what we really want to accomplish. Because we don't know if that code we just wrote is meeting the requirements exactly as defined, whereas a good written test will.

DAVID: Mm-hmm. You end up writing the test to follow the code, and if the code is sort of a sideways implementation of what you thought the card was going to do, the test will then become a sideways implementation of the following of it, right? But if you start with the TDD, you're like, no, this is what it needs to do. You also ran into the thing of like, oh, the code is hard to test, and that's because you wrote the code first. If you had started with the test, the code would be easy to test because it would have to follow the test.

EDDY: I have a caveat to that, Dave, and I think...I spoke with Tad about TDD a few weeks ago. And I told him, I'm like, "How is it possible for you to write a unit test for something, you know, that you don't even know how you're going to write the code for?" So, it makes it extremely difficult to, like, set up all the instance doubles and all of the mock data and all of the stub data that you need. You don't even know where you're reaching, where you're looking, right?

And so, to me, I feel like it's more like a skeleton TDD, where it's like, cool, I can probably get my get, or my it blocks and my context set up the way I think they should be, but that's as far as I've gotten, as far as TDD. I've never actually implemented anything and then wrote the code to pass it if that makes sense.

DAVID: You said one word at the start of that paragraph, and I knew...I'm like, yep, you're headed down the wrong trouser leg of these pants: unit test. A unit test is how you assert that this tiny, little piece satisfies what the whole should do. That's where TDD starts. You write a test to say, "Does the whole thing behave correctly?" Then you start breaking down into the pieces, and then you write unit tests against those. You don't write a unit test to say, "Does this have a callback on the database before save?" until you have something higher up that says, "Hey, when I do this, it updates this other thing." And so, you come out of...it's a multi-layer approach.

EDDY: Right. So, I had a misconception of what [inaudible 07:58] was, right? And so, like [laughs], when I spoke to Tad, I was like, "Oh, gotcha, gotcha. So, you're not writing all the branches," right?

DAVID: Right.

EDDY: Like, making those branches pass. You're essentially doing, like, this is the happy path.

DAVID: Eddy, I want to sidebar with you afterwards and bang out this idea because you just sparked an amazing thought in my head. And poor Justin is just waiting for us to talk about habits [laughter]. But you just sparked an amazing thought in my head, which is, I think a lot of people push back on TDD because they're like, well, I wrote the code, and then the testing was dumb and stupid, and hard, and after the fact. And they look at TDD as write the dumb, stupid, after-the-fact test before writing the code. It's not going to work.

It's like, you have to write something different first, and then you're like, oh, if we were just doing that, right? But yeah, I just realized that, yeah, if you see all this stuff, all the baggage that testing and awkward tests are, if you just move that ahead and try to do the awkward testing first, it's much harder to do that. Why would you do that?

ENRIQUE: Basically, TDD is it's like a good practice when you have a domain knowledge, what you're doing. Because sometimes, for example, here at Acima, it's been the first time where I work with mocks instead of intensively writing the database when you write specs and start with specs. For someone like me in this context, when I don't know anything about the domain, and I don't know anything about how things are mocked, it is really, really hard. I really need to go into the UI, understand what is going on, go into the controllers, and see that everything is working correctly then I write specs.

MATT: Yeah. Back to your refinement comment, Eddy, if you have a really good understanding of the problem and we're writing stories properly and breaking them down to a level at which I feel a story should be broken down, you don't have to have a whole lot of domain knowledge. You just need to know what's coming in and what you expect out, and if you have that understanding, you can write a test.

And you don't have to write the entire test for it to be TDD, right? You know what's going to happen first, so you write that piece of your test. And you should run it, and it should fail. So, you implement that piece, and then you go to the next step. Let's write the next step in the test for it. And you can just step through it that way. And that actually, in my opinion, makes it so you don't require as much domain knowledge to be able to write software in a big application.

DAVID: For me, the secret is to go up a level and identify the behavior that the system should follow. That you can test in a very generic way. It's like, oh, when these things go in, they need to come out sorted. Now you go write a unit test that says, "Oh, I need you to call the sort method," or "I need you to sort it in the database," or "I need to use a B-tree," right? But the behavior is: it should be sorted when we're done, that if I pass in A, C, B, I should get A, B, C out. And how we got from A, C, B to...right? Who knows? We could split the text and pass it through...who knows, right? That's the thing.

And I think if you start your TDD with, okay, I need to mock out the database SQL query with an expect an order clause so that it will sort, oof, that's not test first. That's [chuckles]...Tad will call me on this one. That's [inaudible 11:27], which is test after I thought up the implementation, but before I wrote it [laughter].

MATT: So, and I think that's a good habit to form, right? TDD is going to help us write better code and more accurate to the actual requirement.

TAD: I think when you start out, you're right. You get something, and you're like, I don't really know how to do this. I don't know what this code is going to be, right? And I know, for me, a lot of times, step one is kind of poke around and brainstorm and kind of look [laughs] at where I'm putting the code and what's going on there. But that could be tests, too, right? Like, if I'm exploring the code and I'm testing, I'm like, what does this gem do? What could this code look like? What's the environment that this feature is going to be written in? I could easily write a test that kind of sets that up for me to poke around in.

JUSTIN: One of the things I'm picking up here from TDD, and I've done limited TDD, I'm not as good at it as I would hope, but just if you generalize it to kind of, like, a habit, I think it's worth recognizing that any or most good habits that you think are good habits when you start out at them, you're going to suck.

And it is something that you have to realize you may need to push yourself, or do other incentives, or try a different strategy to do the thing to be able to actually make that habit stick. If one thing isn't working for you, recognize that, and then try another thing, or try a different strategy, or talk to people, all of those things. You know, seek out help that helps you, you know, get you closer to where you're going to try to get to the habit.

MATT: Yeah, good habits are hard to form, right? There's a reason that millions of people make a New Year's resolution that they're going to exercise this year. And then, the gyms are totally packed for about three weeks or so. And then they just fall off because good habits are hard to form. And they're worth it though. The effort is worth it.

EDDY: Since we've been talking a little about TDD, and I kind of want to stay on track a little bit, would you go as far as to say that TDD will make you more productive? Assuming you've honed the skill, and it's just, like, second nature. And you're just like, cool, I'm going to do TDD, and that's it. Like, would you compare someone who is more senior, who's already acclimated doing TDD, and categorize that essentially as, yeah, that makes me more productive than just running the...or banging out of the code first?

JUSTIN: TDD itself is like a combination of a bunch of different things, like breaking down problems, knowing what your target is, you know, all those things kind of add into like, hey, I'm good at TDD now.

MATT: Yeah, so look at it from this perspective: if I'm writing accurate tests on a problem that I understand, the likelihood of me introducing a bug into the codebase is much, much lower. So, from a productivity perspective, I would say if I'm introducing bugs into the code because I'm not testing and I don't understand the problems that I'm trying to solve, that is significantly dropping my productivity because I'm going to have to go back to it or someone is and fix it, which is going to take even more time. It's going to take debug time, tracking things down, and then, on top of that, coding time. So, if you can avoid all of those things by writing good tests upfront and solving that problem, I would 100% say that yes, it is going to increase your productivity.

DAVID: I think a lot of people push back on TDD because they're like, I'm not sure how I'm going to solve this problem, so how am I going to test it? This is really percolating; this idea of TDD is not test after but backwards in time. If you go up a level and figure out what is it that I'm trying to solve, that's the thing that you test and that you can do in advance, and then you go down slow. And then, when you go down a level, you have to write a unit test that say, "Does this code do the...?" Yeah.

You guys have fried my brain here. I'm, like, smoking neurons as well as we speak and, like, trying to get my head wrapped around this idea. What I will say is, if I don't know how to solve something, TDD lets me come at it from above and sketch in and say, "You must satisfy this to make this happen." Where, if you are expecting to come up from underneath of, like, well, I'm going to build a thing that combines this, and then recurses into that, and then calls this query over here, and then; eventually, I have a thing that does the thing in the end, TDD sucks when you're trying to do that. That much more suits the write it, test it, write it, test it, write it, test it.

And there your unit tests are, like, regression specs. They make sure that the next three things that you do don't break the first thing that you did, where, for me, TDD gets out in front. I was talking with Tad about this earlier in the week. I wrote a spec, and then I wrote the code. And the spec was just there to help me keep my thinking straight and make sure that it would do the thing. And when I was done, I had a spec that did not explain what the code was trying to do. And it duplicated a lot of the, like, external stuff that really wasn't useful, and it forced a database hit, which made the thing very, very slow.

And after we had the whole thing written out, the spec was superfluous. It was useless. It was not a really good regression test. It was just going to be a maintenance nightmare. So, I wrote the test. I got the code working, and then I deleted the test and I pushed up the PR. And Tad called me on it because I did it on two PRs. And the second one, he's like, "Where's the test for this?" And I'm like, "Oh, that actually was a good test. I shouldn't have gotten rid..."

The first one was just a database scope, and, like, to test that in Rails, you're going to duplicate Rails, and it was kind of useless. And there's not really a clean way to just say, "This scope should exist." So, anything outside of that was just duplicating. Oh, I kind of start to receive a message chain, but yeah, the arguments could be all over the map.

MATT: To be able to do this, right, you need to know what is your expected input, and what's your expected output. It doesn't need to dictate your implementation. So, it's not going to write good code for you, but it is going to solve your problem, and it's going to prevent you from breaking things when you make choices.

DAVID: Yeah. You just touched on a really key distinction there, Matt. You have to know what your inputs and your outputs are. When we come at these tests, and we're like, "This testing is hard," it's because we're coming at from the other side, and we're saying, okay, I should expect this model to receive the message chain of dot where, dot order, dot...you try constructing the ActiveRecord query, which is all implementation, right? It's all pure implementation, and we're trying to mock and stub that out. And you're standing at the wrong end of the code for TDD when you do that. Man, I feel a blog post coming on. Remember blogs? Three people on the call are like, "No," [laughter].

EDDY: Dave, [inaudible 18:27] might actually have something that's really interesting. And you said test-driven development allows you to come at it from a different perspective, right?

DAVID: Mm-hmm.

EDDY: Like, at a higher level, then we're acclimated [inaudible 18:36]. I want to say something about that. And there's a concept, you know, that is pretty notorious around the tech industry, especially for programmers, and it's called rubber ducking debugging. And I've associated writing tests...I used to not like it, especially doing RSpec because the DSL is crazy, right? But, like, once you, like, really narrow it down, it actually caught...I've caught myself actually approaching the problem differently. For me, it's sort of like a rubber duck, where I'm able to express it in a different way that I wasn't looking at it before, so a new perspective.

Now, we're talking about productivity. That is a prime example. Now, I need to test my code, but my code isn't working. How am I going to set it up? And then, suddenly, the setup is like, oh, that's why it's not working [laughs], right? And you're able to go through it line by line in a different angle, right? So, when we're talking about productivity, I think TDD, in a sense, at least I have some experience with it getting me unstuck, right? Because it's able to provide a different perspective.

So, I do kind of want to go on record and say hey, I don't hate TDD. I've seen the benefits on why we do it, right? And it's keeping ourselves in check. And it's also a way to solidify the code that we wrote is on purpose [laughs] and that we truly understand it.

MATT: Well, right. And it's up to the engineers to figure out the middle, right? And, again, it can't dictate good implementation, but it prevents you from breaking your inputs and outputs. And as long as you know those things, test-driven development is absolutely relevant.

DAVID: Yeah. Matt, you're feeding my brain hamster. It is not an input or an output to assert that ActiveRecord receives this convoluted message chain, right? That's implementation support to make your test go a certain way. And this is the kind of spec that we don't do in our code a lot because we have a very large test base, and we're always trying to speed it up, and we've had productivity hits from trying to talk to the database.

But, like, when you have a thing that says, "Hey, I'm going to write this kind of contract and this other kind of contract. And I'm going to assert that when I call this thing, I get the first contract and not the other one," and that's all handled by database stuff. That's stuff that we would be tempted to stub out the ActiveRecord query and da da da. And that's very implementation-specific. And those are not inputs and outputs, right? The inputs were I had these two contracts, one was even, and one was odd, or one was red and one was blue. And when I ask it, "Give me blue contracts," I need to make sure I got the blue one, and not the red one. And that's input-output. And now it doesn't matter.

Eddy is frantically waving, "Mike's on the call. Mike's on the call."

MATT: [laughs]

DAVID: Welcome, Mike. We were just talking about...well, today on the Acima Developers podcast; we are baiting and switching Justin on his habits stuff. We let him give a two-minute introduction on habits, and then we went down the rabbit hole of TDD and testing. Next week, we'll talk about ADHD and bicycles, which reminds me of a story about bicycles.

EDDY: To be fair, Mike, we also have circled back, and we were able to kind of deduce that TDD really does help with productivity.

MATT: I think it would be hard to argue that it isn't a good habit if you're doing it properly.

MIKE: You touched on something there: if you're doing it properly. There is actually some debate in the community about how religiously you should follow TDD. I think there are circumstances, particularly when I'm doing more exploration and generally within existing code, where I might not write TDD first. But then, again, sometimes it's for sure great to have a test, right? And you can use the test to see if something works. You can actually use the test to, you know, test that interface to see if it works. There may be some personal application there, but in general, TDD, while it may take a little overhead up front, leads to a reasonably performant process and generally leads to stability, right? Predictably, which is always a good thing.

EDDY: Can I just go ahead and also go on record in saying having TDD, specifically in RSpec where you have a context, and it block, it dumbs it down for me to better understand what that method is doing. So, when we're talking about productivity, like, my biggest complaint really is if this method is really complicated, I hate that it doesn't have comments. If you think it's complicated, either leave comments on there or have some gnarly, gnarly test that really breaks it down and tells you this is the context. This is the it block. This is the behavior. This is the [inaudible 23:30].

And having really good tests has actually helped me better understand what the purpose of that method really was and the intention behind it. So, again, circling back [laughs] to productivity, I was better able to have, like, a better-defined picture that attuned on, like, the changes. And I felt safer doing it as a result.

MATT: And writing good, clean code is much more productive than writing code that introduces bugs that you have to go back and fix.

DAVID: Ooh, but in the first five minutes, it's slower, right?

MATT: Oh, 100%

DAVID: It's so much quicker, right? I can jam in a fix that solves the thing that, you know, it's like, oh, hey, look, even-numbered contracts are the ones that are violating the TCPA standard. So, I'm going to reject texting messages for even-numbered contracts. Fixed it! It took me 10 seconds.

MIKE: [laughs]

DAVID: There might be some maintenance later. Future me will hate me.

MATT: Never [laughs].

EDDY: Justin, circling back [laughs] to the topic at hand, do you want to continue?

JUSTIN: Well --

DAVID: Actually, the reason we went...sorry. Sorry to jump on this again. I am actually steering towards you, Justin. The reason I brought up testing, in my particular bit, is that somebody reminded me on Twitter this week if they haven't seen the test fail, they assume that the test is wrong or hasn't worked. And so, this person has a habit that when they write a test, they make it fail every time. If they haven't seen it fail, they assume that the test is either not executing or not touching the code that's being exercised.

And I don't always follow that habit, and I will get bitten by a test that seems to be passing, and then I will break the code. And the test continues passing, and I get very alarmed. And then, I get very good about following that habit for a while.

MATT: Yeah. When I first learned how to do test-driven development, that was a requirement of the team.

DAVID: It's a [inaudible 25:30]

MATT: You know, we were programming, and we said, "Okay, it's passing. Now let's make it fail."

EDDY: If you make it fail, suddenly you have your unhappy path, right?

DAVID: The old Red-Green-Refactor mantra was, you get to red first, then you get to green fast, then you refactor until the code is, you know, not a maintenance nightmare.

MIKE: And I think that there's something there. I think that a lot of times we think, I need to test the thing, and so we're not very systematic about it. We say, okay, we'll start with a happy path, and then I'll start doing edge cases. And after that test passes, we think, okay, I've got pretty good coverage. Now, let me start covering the other things. And I think that that way of testing in this way that I've done it in the past, that I've changed over time, has some flaws. It really isn't structured in order to document your code. And we've talked about this in other podcasts. It's structured in order to help you maximize your efficiency in that moment to cover that specific part in the flaws.

And instead, I like to approach my code and say, well, what are all the branches? And I usually like to test all the broken branches first. And if you do that, you have verified that, you know, all of your validation failures are actually failing out, and then your happy path is kind of the last piece you get to. It takes a little bit longer, but then you get full coverage. You get better documentation out of it and avoid a lot of that risk of like, oh yeah, the happy path test works when maybe that wasn't even testing anything.

MATT: You're preventing downtime in the future.

EDDY: I don't know if it's crazy. For me, I like to see green dots first. Like, it super, super, makes me a lot more enthused to write code. I joked about this on a previous podcast, like, green dots have become my favorite-colored dot.

MATT: On that note, if you are doing test-driven development, it's going to fail first because your failure is what's driving your development.

DAVID: And I don't think those are incompatible statements, by the way, or even what Mike was saying. Like, I still see it. In my head, it's all green dots, right? That green dot means that this is exercising the code and that it's passing. So, we get to red first so that I know it's executing the code then I make it pass. Now I get my green dot, and I get cookie. It's satisfy. It make it work. And testing the unhappy paths, those aren't red dots, right? Like, the unhappy path is you tried to save a username that was, you know, 4K long, and the database only holds 50 characters.

There must be a validation error that says, "Name must be less than 50 characters." And if I save a 4k thing and it writes it and it doesn't raise a validation error, but it throws away [chuckles] 3.9k of the name because it only said, "That's a fail," because it failed to validate, so you make the validation pass. And you get the green dot, and so it is; it's all green dots. At least in my head, it's all green dots.

MIKE: Yeah. It's all green dots all the way, but you're testing all the edge cases either way.

DAVID: Yes.

MIKE: So, you don't have to write a test that doesn't cover that case and then have to go through the pain of seeing the red dot. What you do instead is you write a test that captures all the bad cases and make sure that those all show green. Like, make sure those failure cases actually happen.

MATT: Let's take an integration test with MVC. First thing we may do is load up a model, right? But because we're doing test-driven development, that model doesn't exist. So, first run it's going to fail. Then, we go create our model. Next line, hey, we hit this endpoint in this controller. It then loads the model. We run it. It fails on that endpoint because it doesn't exist. It drives you step through step. And that's what I was trying to say, Eddy, is, sometimes, when we don't understand a domain, going about it in this manner helps us understand the domain but also makes it so we can work inside of that code without that full understanding.

DAVID: The other thing that gets me thinking is, like, when we get a ticket that we have to work in the code, it doesn't say, "Use a red-green or a red-black tree to sort the data in the database." You get tickets that say things like, "Stop sending text messages to customers who have opted out via TCPA because that's against the law." There's nothing in the database that says, "Don't break the law," right? And yet we will pull up code and, like, ultimately, the fix is going to be to add a where clause in the thing. When we do this, we need to join on the TCPA, you know, opt-ins or whatever, or however you can implement it.

And we might be tempted to write a test to say, "Are we checking the database for this one thing?" But if you're starting with TDD, you start with a customer who has opted out and another customer who's opted in. And you try to send them both messages, and one of them should receive the message and one of them should not. And that's pure io. Now, it doesn't matter how it gets developed, right? It literally could be, "Hey, there's this queue that we send to human operators who punch things in on a phone to call people, and that queue should not get the customer that opted out," or it could be a database query, or it could be there's a million ways to select data, right?

MATT: Yeah, because you know your inputs and your outputs.

DAVID: Right. When you get this thing, don't try to write a TDD of changing the way we filter from a list. You know, it's like, how am I going to reject on the MapReduce? No [laughs]. Start with your input. Start higher. Go up a level.

EDDY: I kind of want to punt this to, like, individuals who weren't here in the last podcast. And I kind of want to get their perspectives so more tailored towards Matt, Tad, and Enrique. What are some of the habits that you typically gravitate to when you're stuck on something, and you're trying to get more productivity? Like, what are some of the things that you do, like, instinctively that get you more productive?

TAD: Talk to someone else. A lot of times, if I just talk through it...like, you were talking about rubber duck earlier, Eddy, right? If I stop, talk through it, then, usually, I'll get some clues. I'm like oh, there's some variables I need to check, or, oh, there is where I might have gone wrong, right? Just talking it out, talking it through, even if they don't give me any feedback, is often helpful.

MATT: For myself, I would say my first steps usually aren't reaching out unless I know someone has a context on the problem I'm trying to solve. But I do have things that I almost always do; one would be Prys. I love Pry, you know, ruby-specific, but debuggers, breakpoints. And then, I step through because, ultimately, if you're trying to solve a problem, you need to know where that problem is happening, and I will always time-box a problem, always.

One of the mistakes people make early on in their career is they don't want people to think they don't know something or they're inadequate, so it's a pride thing. Overcoming that and being able to reach out and ask questions will significantly help your process. Every day, I'm reaching out to people. It doesn't matter who it is. It doesn't matter what level someone is. Everyone has something to offer that we don't know. So, I think a good habit is asking questions and just being able to communicate those questions to your peers.

JUSTIN: One other thing that I think is helpful is to...it's been mentioned several times before, but breaking down the problem such that if you are stuck on one particular thing or kind of a general thing, you're like, oh, okay, how do I break this down? And keep on breaking it down until you get to something you can solve.

And then, by the time that you have perhaps identified, you know, okay, I can solve these five sub-steps, but I can't solve six, you've identified enough to bring it to somebody else who may have context. Because they'll probably ask you, "Oh, did you try A, B, and C? "And if you were able to break it down, you'll be able to go to them and say, "Okay, here's my work so far. You know, I don't have any idea how to do this," But rather is, "Oh, okay, here's the original problem. Here's what I've tried to solve it. Here's the progress I've made, but I'm stuck on this particular one," and just explain to them everything that you've done.

And then, they'll be able to answer, "Hey, did you try this?" Or they might be able to answer, "Hey, you're barking up a wrong tree." But it's also, you know, you combine that with all the other suggestions of time boxing and things like...you don't want to go too far down the wrong rabbit hole. So, there's a balance to be had there. But, you know, do your work and do your homework and try something.

EDDY: Oh my gosh, I've never thought about timeboxing a problem [laughs]. That's so cool. Just to bring everyone kind of up to speed, Tad and I were pairing earlier today, right? And we were trying to figure something out. And he said, and it's kind of stuck with me even now, he's like, "Okay, let's try to figure out this problem. But if we can't figure it out, we'll timebox it. We know how to fix it another way, but we don't like this other way. We're going to try to fix it this way instead, but we'll timebox it for, like, 10 minutes." If we can't figure it out in 10 minutes, suddenly, your productivity kind of starts spiraling down because you're still stuck on the same problem.

So, just saying, "Okay, if in 10 minutes we can't figure this out, we still know the solution. We'll do the other solution as a fallback." But you're still moving forward on your problem, right? Oh my gosh, you know, timeboxing is so cool. I didn't think of that.

MATT: [laughs]

TAD: Because I was like, there's an elegant solution here if I could just find it, right? And there's a way that works, but it's kind of ugly. I'm like, "Eddy, if I can't find the elegant solution, we're just going to...in five minutes, we'll stop and do the ugly thing instead."

MATT: That's kind of a good segue into what else I was going to say, and that is stepping away. Oftentimes, just stepping away from a problem is a good idea. So, if you have an elegant solution in your head and you think this other one's sloppy and you implement it, step away. The elegant solution may come to you. You know, sometimes just kind of walking away from something, giving yourself a breather because we tend to get tunnel vision. And I think getting outside of that tunnel is extremely helpful.

DAVID: I used to rock a standing desk, and I legit believe that the number one value of the standing desk...nothing to do with health or with any of that. The number one value of the standing desk was that I was much more likely to step away from the computer once every 30 to 60 minutes just to think about the problem. And it stopped me from fixating and tunnel visioning on a problem for two hours, three hours, five hours so many times.

MATT: We want to solve problems. That's why we do what we do, right? And I think all of us tend to get tunnel vision and get stuck in these problems because we're so stubborn that we just want to solve them. And a lot of times, walking away for a few minutes can help do that.

DAVID: I just realized that dovetails perfectly into Justin's habit thing that if you're standing up at a standing desk, it makes stepping away to switch from fast thinking to slow thinking, to tight focused to, like, spreading out to wider thinking, it makes it so that that's easy. It's attractive [laughs]. There's no start-up cost. You just take a step back, stretch your arm, and then just walk over to the window, right? Because there's no pushback, change state, stand up, deep breath. There's none of that big state change. You're already standing up. Just walk over the window and breathe.

ENRIQUE: I'm trying to Google what time box a problem means.

DAVID: Oh. It means to decide in advance how much time you're going to spend on this. You draw a box on it on the calendar and basically say, "We're going to work on this for half an hour." The value of this is you can say that, "We've got a bad way to do it. We don't want to do it this way. But it's not worth spending 10 hours on to avoid this problem. If we can't solve this in 30 minutes, it's not worth solving."

So, you just draw a box around it on the calendar and say, "We're going to work on this for 30 minutes. And if we're done, we're done, and if we're not, we're not." Or at the end of 30 minutes, you can say, "We're really close. It's worth another 30 minutes." You can extend your time box. But, again, you stay boxed. You don't let your time become open.

MATT: If I have a problem I don't know how to solve, because a lot of us are self-learners and we like to just figure things out ourselves, we say, "Okay, I'm going to spend an hour trying to figure this out on my own. And if I don't have a solution, then I'm going to go talk to somebody else and reach out."

EDDY: I love the fulfillment, though, that I get if I resolve the problem myself. Like, there is some sort of joy in myself, like, self-pleasure, where I'm just like, oh, I did it, right? Like, I was able to overcome that, and I found the solution that I was looking for. However, there's times where, like, I've been stuck in a problem for, like, an hour, reach out to someone who I know will have the answer and be like, "Hey, I have this failing spec, and I can't figure this out, you know" And they're like, "Oh yeah, it's because you need to do this." And suddenly, I'm like, oh man, why didn't I see that before? But there's value to that because, like, that solution gets cemented in your brain much better because you struggled to find the solution much longer.

MATT: I think a good habit, too, since we're talking about habits, is when you do finally reach out for answers and information, make it a point to understand why. Instead of just getting the solution, understand why it is the solution and how it is a solution. That way, the next time you run into a similar problem, you have the tools you need to be able to solve that problem.

EDDY: Yeah, for me, I found that I need to struggle on that problem for a little bit longer so that I can better understand the issue so that when I do get the answer, I understand it more in-depth.

MATT: Yeah, and --

DAVID: I think it's useful to know in advance, do I want to use this solution to solve this thing in the world, or do I want to work on creating the solution? Sometimes I don't want to carve wood. Sometimes I want to build wood carving tools. And sometimes I would rather work on my software tooling than build software with my tooling. And so, knowing in advance, what do I want to do?

Like, if I want to build the tooling, I don't want somebody else's help. I want to discover it myself. I'm in it for that joy. And if I go ask someone for the answer, I give up that joy. But if I just want to solve the problem so I can do another thing, every second I spend trying to find the solution is time not spent using the solution to get what I want. And knowing to switch hands and say, "Oh, just give me the thing," right? "Google, give me the answer. ChatGPT, give me the answer," because I just want to get on down the road to solving the thing.

MATT: Yeah, I think sometimes, I mean, and we're in a business environment, so we do have to limit the time we spend on these problems. But if we're getting the answer, getting the solution, if we also get an understanding of how that solution was created and why that is the solution, then the next time we come around, that tool is in our toolbox.

DAVID: We're not just in the business of producing software. We're in the business of producing programmers. That is one of the outputs of our system. And so, yeah, investing in figuring out the solution, definitely worth it.

MATT: Yep, and everyone benefits from that: we as engineers, the company from the ability we create to be able to develop software and mentor others as they come on board and make them more efficient. It's a win-win.

EDDY: I actually want to say something, and you all might laugh, but I stumbled upon this solution that helps me become more productive, and that is eating [laughs].

DAVID: No joke. Yes. How's your blood sugar?

EDDY: I'm not kidding. I'm not kidding. Like, there's times where I'm just like, I've gotten up. I've gotten to my kitchen, gotten a snack, sit down for a little bit, come back up fed, you know, and I'm like, oh, yeah, that's the problem [laughs]. Eating, eating has helped me tremendously on my productivity.

MATT: There's a reason they call it hangry. I'm the same way. If I go without eating, I can get snappy and --

DAVID: I worked with a guy years ago. The question "Have you eaten?" is on my list of questions to ask when nothing else is working. I actually have a list of things to ask when you're out of questions, and "Have you eaten?" is on there. I had a co-worker who was just grumbling, and he's wicked smart. He got a PhD in geophysics, and he's trying to solve a seismology vibrational problem in software. And I can hear him muttering and cursing, and he'd been doing this all afternoon. So, I was just like, "Don, have you eaten?" And he pauses, and his eyes get real big, and he's like, "No," and I'm like, "Get your stuff. We're going to get a cheeseburger."

And part of it might have been the walk, part of it might have been stepping away from the computer, but by the time he had come back and his blood sugar was back up, he knew where to start the next piece of the software. So, you are bang on. Keep that question, Eddy. It will save your career.

[chuckles]

MATT: Rest is right there with it, too.

DAVID: Rest. Yeah.

MATT: You know, yesterday, Travis, one of our other engineers...you all know him, but our listeners may or may not. We were working on a problem together, and I was running on...I think it had approached about 36 hours of no sleep, and, you know, I started my day at about 5:00 a.m. yesterday, and it was going on the 13th hour of work for the day and 36 hours of no sleep. The simplest of problems become difficult. If you aren't rested, then that's a real thing.

DAVID: Sometimes, the most productive thing you can do is get your head down for six hours. Yeah.

MATT: Yeah, and I just finally I was talking to him, and I said, "Travis, how about I hand this over to you? You drive because I just am not functioning at the moment."

TAD: You've reminded me, you know, like a few weeks ago when we decided we're going to go for periodic walks around the office, and we fell out of that habit.

EDDY: It's a great habit to have, I think, right? Like, it takes your mind away from, like, just being stuck at your computer for so long. And I hope, like, higher people don't hear this, but that is part of why I like going into the office. It's much easier to reach out to someone to the side and be like, "Hey, Dave, question for you. I'm stuck in this problem. What would be your approach?" And kind of, like, bang off ideas out of each other. Like, when you can do that, there is solutions to that. Or sometimes just going on a walk for a little bit, like we said earlier, right? So yeah, just like what Tad said.

DAVID: I think the answer might be written on the mountain. Let's go look. Let's go out to the parking lot and look.

MATT: Yeah, that encompasses the stepping away. And, obviously, exercise is healthy. It gets endorphins going that may not be present just from sitting at your desk. I don't think anyone can argue against taking the occasional break and doing that. It makes us perform better.

EDDY: Enrique actually wrote something, and I kind of want to say it on record. He said going to the bathroom really helps getting yourself unstuck and being productive [laughs], take that as you will.

MATT: And being tracked down by your CTO.

EDDY: [laughs]

DAVID: A little burst of adrenaline. Whoa, what? Getting up and going to the bathroom...I read a neurology study. They found that when you walk through a doorway, there is something neurologically that happens that causes, like, a refresh stripe to go through your brain. Like, it just passes through the neurons.

And they think this is why you can stand up and walk into the room to get a thing, and when you get there, you can't remember what you were going to get. And it's because the part of your brain that's holding on to that attention, go get the thing, gets reset by that refresh wipe as you go through. So, yeah, weaponizing that for good is get up and go to the restroom room, you know, go down the hall. Go talk to a coworker. I need to wipe all of the wrong solutions that I'm holding on to in my head. I need to just power off.

EDDY: Dave, I thought that was always a me problem. I thought, for me, I was just like, man, I think this is, like, early symptoms of, like, Alzheimer's or something. Like, how am I forgetting something so simple? And then, like, as other people, like, affirmed that it happens to them, too, suddenly, I'm like, oh, thank God I'm not the only one [laughs]. Like, other people have short attention spans also.

MATT: We've been talking about all these habits. Just another quick one just for record is, taking notes and recording meetings. Giving yourself something that you can reference, I think is a good habit as well. Historically, I was never a note-taker. Just because of the things I'm responsible for and have to keep track of, I have become a very consistent note-taker, and it's been a really good habit and helped me out a lot in my day-to-day duties.

EDDY: Matt, actually, this is really interesting because, like, I spoke with someone the other day, and they told me, "Oh yeah, I'm a huge note-taker." And they're like, "I can be on my computer, but I find myself getting a notebook and writing, like, with a physical piece of paper and a pen on my hand." And I'm like, "Really?" I'm like, "Doesn't that take longer to jot stuff down? Because I'm sure you can type faster than you do write stuff down, right?" And they're like, "Oh, no, no, it's just, like, I've been groomed, you know, it's instilled in my childhood to take notes with a notebook. So, like, it's ingrained in my head, and so I'll do that, like, natively." And I'm like, "That's interesting."

MATT: I think for a lot of people, just on top of that, is, when you write something down, while you're writing it, you're also recalling that information, right? So, right there, you've heard the thing. You've written the thing and recalled the thing, and you're ingraining it into your memory.

EDDY: Yeah, but that also happens when you're typing. I'd argue that it's –

MATT: Yeah, yeah, sure, sure, but I just mean note-taking in general. I'm of the age where when I went to college, you took notes by hand. And I got really good at it for a while, but I don't write by hand often these days. Just having something to remind you and to recall and go back and reference is a really good habit because we all get so much information thrown at us so often that we can forget and lose things over time.

DAVID: I took a speed writing class in high school back when typewriters existed. You'd take dictation very, very quickly, and your notes are sketchy. And it's trying to, like, what the heck did I write there? And I skipped words. But the very first thing you do after the dictation is you sit down and you transcribe your own notes to better notes. And then, you take your transcribed notes and you type them up, and that process of reviewing your own notes twice is hugely beneficial for memorizing.

MATT: Yeah, and there you go. Just by the time you get to the end of that process, you have recalled that information, like, four times. You start to remember it all.

EDDY: I was surprised when I first started, right? Like, my late high school years going into college, I'm like, how the heck are people able to write all this in a piece of paper and, like, recall everything that was said? And then, one of my teachers sat me down. He's like, "Well, because they're not writing everything. They're just doing highlights of what was said," and highlight. I was like, holy cow, that just opened up, like [laughs], another realm. I'm like, yeah, you don't necessarily need to write the does, the ifs, and whens. You just do only the highlights of what the conversation was had but enough where you can recall. And suddenly, you become super efficient.

DAVID: You start thinking about what is the objective of this? The biggest superpower for me in school was somebody said, "When you're taking notes, if you were writing the test for this class, what questions would you write from what was given?" And I turned in the sheet of my...I had the thing, and I had the questions that I wrote them down next to my notes. And the teacher actually pulled me aside and asked if I had...like, he didn't quite accuse me of cheating, but he wanted to know if I had been going through his desk because I had picked, like, 80% of the questions off of the test as a result.

EDDY: [laughs] But suddenly, that's a good thing when you do TDD, right?

DAVID: Mm-Hmm. Yeah.

EDDY: [laughs]

DAVID: Test-driven class note-taking. I like it. I like it.

EDDY: Suddenly, your coverage is much higher because you practiced.

DAVID: Yes. Oh, this has been a good round table chat. Thank you, guys, for coming. The one advantage of going so far afield when we invited Justin back for a part two is that now we can invite him back next week for a part three. Well, for a redo of part two.

EDDY: [laughs]

MATT: They always get a little off-topic, but I think that's the beauty of it, right? We segue, and we end up talking about useful information that I think everybody should have.

EDDY: I've actually made a habit on going half to what we were initially talking about. Like, I don't mind going on tangents, you know, every once in a while because that's how our brain works, right? But, like, being cognitive, right? Or conscious and, like, going back to what the original topic of the conversation is. Because otherwise, you'll spiral out, and you'll talk about --

MATT: I would argue today we did never really get away from the topic.

DAVID: No, No. You can't predict what you're going to discover when you start synthesizing new ideas. And, like, Matt, you kept just, like, lightning striking my brain. And I'm like, I did not expect that to fall out of my head and in the podcast. And I'm like --

MATT: [inaudible 50:56] [laughs]

DAVID: Well, no. It's like you were saying things, and it was causing just things to just precipitate like I'd had all the pieces of an idea in my head for weeks. And then, you said that thing, and I'm like, oh, man. Or, like, Eddy, when you said unit tests, and I'm like, aha, it's behavioral, not implementation. Yeah, right on.

EDDY: Well, David, this has been awesome, guys. Thank you so much.

DAVID: This has been great.

In this episode of the Acima Developers podcast, the panel talks about the intricacies of habit formation within the context of software engineering, sparked by Justin's interest in James Clear's Atomic Habits. Justin brings up how Clear's work resonated with his desire to become a better engineer and begins by identifying Clear's four laws of habit formation: making it obvious, attractive, easy, and satisfying.

The conversation unravels how these laws intersect with software development practices. For instance, making a habit obvious could involve placing reminders or tools in clear view to prompt action. The notion of attractiveness is discussed in terms of reframing mindsets towards habits, such as viewing challenges as opportunities rather than chores. Making habits easy is touched upon with examples like simplifying tasks to remove barriers to starting, while the aspect of making them satisfying involves setting up rewards and tracking progress to reinforce positive behavior.

Personal anecdotes and professional experiences shared by the panel highlight the application of these principles in their work as engineers, such as leveraging gamification to make tedious tasks more engaging or adopting meditative approaches to problem-solving. The significance of environmental and social factors, like changing one's surroundings or engaging with communities of practice, is also discussed as a means of supporting habit formation.

Transcript:

DAVID: Good afternoon and welcome to the Acima developers podcast. We have got a really fun panel and topic. It's kind of exciting today to talk about habits. We've got Eddy, and Mike, and Ramses, and we also have Justin on. Justin was the one who asked about this. I'm just going to hand this over to you. You've picked up something, and it prompted a question. I'm just going to hand you the mic.

JUSTIN: All right. Well, I don't know if I really deserve it for that much, but we'll go over what I was thinking of. So, I was listening to a podcast, The Peter Attia podcast. He's a great guy who does some very involved podcasts that are often, you know, several hours long. But one of the ones that he has was with an author by the name of James Clear who wrote Atomic Habits. This is a book that came out a couple of years ago, like, in 2018, I think. 2017?

In it, he talks about how to basically make habits that help you and break habits that hinder you. He doesn't necessarily talk about the exact habits that people want to do other than kind of, like, the obvious ones. But it was kind of an interesting conversation around how we make habits and how those habits help define who we are. And then, he goes into, like, how we define ourselves or how we want to define ourselves, how important that is in our lives if we want to lead a fulfilling life.

And he goes in and talks about if you want to be a better person, you have to define that better person. And then, you have to think about the things that that better person does. And by doing those things, you know, that makes a new definition of who you are. And so, a good example was like, a habit that you're trying to do is eat better. And so, rather than saying, "I'm on a diet. I'm trying to stop eating these certain foods," you say, "I'm a healthy person, and I'm going to eat healthy food." And there's kind of an interesting distinction there. But this is kind of a long-winded answer to, like, a developer podcast.

DAVID: No, it's good. We do that here.

JUSTIN: [laughs] But tying it back to this, I was thinking about, I want to be a better engineer. I want to be a great engineer. What are the things that I can do? What habits can I make that define me or that can define me as a great engineer? The question I want to bring to the group is like, what are the habits that people have that they've found help make them great engineers?

DAVID: That is fantastic. I had told you in the pre-call that I hadn't read Atomic Habits, that I had come at this from the other one. But James Clear he's the guy that played baseball, right? And he had, like, a career-ending injury in high school?

JUSTIN: Yeah.

DAVID: And then he picked every behavior performance, behavioral habit that was holding him back or that he needed to work on. And he ended up captaining the college baseball team?

JUSTIN: Yeah, basically, he kicked butt. He actually has a couple of laws in his book, the four laws that help define how to make a habit, and we can go into those a little later. I don't want to necessarily make the podcast about that, but we can go into that later.

DAVID: We certainly can, like, maybe, like, survey them a little bit. I have started Atomic Habits. I think I got to the part where he was talking about his bio, and I'm like, yeah, get to the meat. So yeah, so I came at this from reading The Power of Habit by Charles Duhigg and Tiny Habits by BJ Fogg. It's like an email list.

JUSTIN: [laughs]

DAVID: This will date me a little bit. This is, like, 2009 kind of thing. But do we want to take just a real quick minute and touch on these things? I don't want to spend, like, a thesis defense on all of these things. But if you want, I can have you throw out the four things that you're thinking of because I want to make sure we cover the topic the way you're looking at it. I don't want to just turn it into the Dave Brady show and say, "Oh, let's talk about dopamine, and motivation, and da da da," unless that's where you wanted to go with this.

JUSTIN: Those all come into it. So, why don't I talk about these four laws, and then we can kind of discuss those items? And then, we can go into, like, some specific software engineering habits.

DAVID: Fantastic.

JUSTIN: Law number one is make it obvious. So, make whatever you're trying to do make it very very obvious about what you're trying to do. He goes into different strategies for identifying and creating cues to make and break habits, so things that are very, very obvious.

EDDY: Can you elaborate on number one on what you mean by make it obvious? So, for example, if you're trying to achieve something, it should be easily explainable to someone. Is that sort of what you mean by obvious, or?

JUSTIN: Yeah. So, what I mean is, like, if you have a goal of reading books, of reading specific books, put them right in front of you. Put them on your nightstand. Put them on your pillow. Don't hide them. If you're reading them on your Kindle, you know, put them on your home screen, or something like that, or put goals. Put things that will remind you to do the thing.

Another good example is if you want to eat fruit every day, put fruit around in front of you, hopefully, in front of the potato chips [laughs], such that you see the thing that you want to do. And another one, if we were to tie it back to programming, well, actually, let's wait, [laughs]. Let's wait on that. But first, law number one, make it obvious.

DAVID: My 10-second take on that is that I want to get better at guitar, and I have a couple of guitars. And I hung my guitar on the wall, and it is about four feet away, and it never gets played, which is why my other guitar is literally on the floor in arm's reach, where I literally don't have to even lean over to pick it up.

EDDY: I do want to say that maybe making it obvious, putting it in front of you, does attribute to using it, right? And this might be part of the other topics you want to talk about, like, with your points. But I have found that putting it in front of you, and giving you arm's reach, at least for me, it didn't really help. What really helped I was exposing myself with individuals who already did what I wanted to achieve.

DAVID: Ooh. Modeling behavior.

EDDY: So, for example, if I wanted to read more often, then I'd join a club with individuals who would read on a daily habit. That sort of motivates me to kind of follow the crowd. And it is not just applicable for reading books, like, in general, you know.

DAVID: I think that actually dovetails into...I'm not sure how Clear...Fogg does motivation and trigger are two of his things, and the motivation is that. I bet that's the make it attractive that you were about to say.

JUSTIN: Yeah, exactly. The make it attractive, however you do it, you need to make it something that you want to do. And you can incentivize yourself various ways into doing it. You can bundle it together within a reward. You can make it a habit, some sort of ritual that you do every day. You can put social incentives around it. So, do it with a buddy, or do it with a group.

The other thing he mentions is, like, reframe your mindset, so rather than saying, "I have to do this every day," say, "I get to do this every day." And getting to do it is, you know, making it more attractive in your mind. So, those are the kind of examples on the attractive side.

EDDY: I'm curious if maybe I'm getting ahead of the topics here, but, like, how do you get over the hurdle of wanting to read when you don't have the motivation to read, right? Like, that's really the key, right?

JUSTIN: Yeah. So, all four of these laws are designed to help you get over that hurdle because they go into, like, the theory a little bit behind habits. Like, what makes a person get bad habits, and what makes a person get good habits? They brought up a lot about smoking because, obviously, smoking is not the best thing to do. But you have to recognize that smoking is a very in-the-moment reward. Like, you get that nicotine hit. You might get the habit of doing it. You get the social if you're, like, going out and smoking with somebody.

So, there's a lot of immediate rewards to smoking that makes it easy to fall into that habit. If you think about it, the long-term consequences of smoking are really pretty bad. That's why these laws are designed to help you overcome the short-term attractiveness of bad habits and help you realize the long-term benefits of habits.

Another example kind of going the other direction is exercising. When you start an exercise program, you know, for the first couple of weeks, you're, like, feeling like crap. You're like, oh, my body is not designed to move that way anymore, or you're always sleepy, or you have aches and pains that you never had. And so, all of those things prevent you, like, short-term prevents you from wanting to do that exercising. But the long-term things really help you out a lot. And so, these four laws help you do the long-term.

EDDY: I was actually going to interject a little bit here and say maybe I'm just wired differently than most people because I actually have the opposite effect when I start working out. I'm like, oh yeah, two-three weeks in, I'm like, I have good rhythm. I feel amazing. I'm super motivated, you know. And then comes the big fall of, like, I don't want to get out of bed. I hate getting up at 5:00 in the morning. I hate eating healthy, you know, and, like, just, I sort of, like, spiral then.

DAVID: There's a novelty effect that is especially exacerbated if you have ADHD, where doing something new is ten times more rewarding than doing what you've been doing.

EDDY: Exactly. Change of habits.

MIKE: There's something related to this. You mentioned smoking. I wanted to share, but I'd have to go look up the source. But there was some research done by a woman, and this is relevant because this was done in, like, the 60s. And her research was ignored because she was a woman until it was very clearly verified, and she was vindicated. And she was right, way ahead of her time. But I can't remember her name, so I'd have to go look it up.

She did research about soldiers returning home from Vietnam, and a lot of them came back with a heroin addiction. And she was researching, and what she found was that soldiers who quit before they came back from Vietnam to the United States were usually able to stay clean, which is almost unheard of in addiction. Because if you're addicted, you're addicted. But the soldiers who got clean in Vietnam and then came to the United States were able to stay clean. But those soldiers who came back and used while in the United States were not able to stay clean.

DAVID: Oh wow.

MIKE: And this is, I think, really connected to this idea of habits because if you can change your environment...and, usually, we can't go from being a soldier in wartime to, you know, a civilian life, and that's such a dramatic life change that most of us can't pull that off. But there are things that we can often do to change our environment, to fundamentally reshape the way that we do things. Put the fruit in front, for example. And she found that that was critical, that something even as extreme as heroin addiction, which is, you know, known to be very problematic, could be overcome if you are able to change your environment.

So, not just give up the bad habit but couple that with a lifestyle change, you know, dramatic. When I say lifestyle change, a life change, right? You know, going from one set of habits to another, and I think that some of these fundamental shifts really need that kind of change in mindset. You say, "Well, I'm going to do this different." And the way that you make it happen is by living that different life, and if that's the life that you're living, it becomes this new routine. You know, you've established a new identity almost.

JUSTIN: Absolutely.

DAVID: Those were the first two, Justin. What were the other two?

JUSTIN: So, law number three is make it easy. And this talks a little bit about the two-minute rule, and now we're getting to the parts that I haven't read yet [laughs]. But what I believe this is, is, like, make it so that you can start doing your new habit within two minutes, so whether that means having a gym in your home or whatever new habit you have is not, like, a whole bunch of prep work or friction that you have to work your way through before you can start doing the new thing.

DAVID: I had heard the two-minute rule expressed in another way, which was, it was part of, like, making it tiny and accessible. And I may have misunderstood this. The version of the two-minute rule that I had heard was make it something that you can commit to just do it for two minutes, and then you give yourself an out. And so, it's part of the, like, how do I make this less painful, right? It's like, nobody wants to get an exercise program. That's too big.

JUSTIN: Oh no, yeah.

DAVID: That's too much work.

JUSTIN: Yeah, I think you're right, actually. I think you're right.

DAVID: But, like, I can walk for two minutes and then turn around and come back. And then, I've just walked, and that's all I've done.

EDDY: You know, a really easy way to achieve that is to just get a dog [laughs].

DAVID: You'd think that, but I also have a wife.

EDDY: [laughs].

DAVID: And so, she's the responsible adult.

JUSTIN: Yeah, I think you're right as well. Now that I looked a little farther into it, I was like, oh yeah, you're right. So yeah, doing it within two minutes. It mentions that it, like, overcomes procrastination and laziness that you could do it within that time period. I kind of question about that. But like I said, I haven't gotten that far in the book yet, so...

EDDY: I guess I have a question. Is [inaudible 13:53] the moment that it crosses your mind that you should do it or the moment that you established when you want to do it and do it within those two minutes? Because I'll go on record in saying that I have a really short attention span. So, I can have, like, a really good idea and be like, oh yeah, I should really go get some food [laughs]. If I don't do it exactly within, like, seconds that I think about it, I won't do it.

JUSTIN: So, in here, I just saw another example. It talks about a famous writer called Anthony Trollope, who I've never heard of before, but he published quite a bit. Instead of measuring his progress based on the completion of chapters of books, Trollope measured his progress in 15-minute increments. He set a goal of 250 words every 15 minutes. And he continued that pattern for three hours each day. So, it was, you know, rather than writing a chapter a day or something like that, he shrunk his goal from writing 3 hours a day to writing 250 words in 15 minutes. And that very measurable small goal allowed him to feel a sense of accomplishment every 15 minutes. And it helped move him throughout the day.

EDDY: That's interesting, right? Because I want to say anytime I've always wanted to read a book, I felt awkward leaving it in the middle of the chapter. Instinctively, we want to get to the end of the chapter, so it's, like, a really neat way the way the author intended.

I do want to add something, though. My wife and I tried something different the other day. It's kind of interesting. We'll, instead of finishing a whole episode, we'll watch half the episode and then stop, pick up on that half episode, and leech over to the next episode, and that way, you know, you're stopping abruptly halfway in the episode every day. And then, it doesn't leave you with the sensation of, ah, what happens next? You know --

DAVID: Cliffhanger.

EDDY: So, it's easier to stop, right?

DAVID: Wow. That is the most psychopathic thing I think I've ever heard you say, stopping in the middle [laughter]. That's amazing. That's amazing.

EDDY: But it works, right?

DAVID: Yeah, yeah, no, it's, yeah.

EDDY: Because instinctively, what we want to do is finish. Like, I think we just want to finish whatever we started, and leaving it like that or leaving things unfinished is kind of uncomfortable. But you get over the habit of binge-watching, you know, and you're able to control your habits much easier because it's much easier to tune off, you know, when it's something boring.

DAVID: Related to the two-minute rule, BJ Fogg's Tiny Habits, he dials that up to 11. When he says tiny, he's like, think two-second rule. For him, the definition of a good habit is I want to get better at guitar. When I walk into my office and I see my guitar, I'm going to pick it up, and I'm going to play one note. That is literally the thing that you write up. This is what I will commit to do.

And now it's so small that as a commitment, it's like, oh, I don't have to grapple with my fear of failure as a musician. I just have to play the one note, and whether the note sounds good or bad, I can do it. You can just take things smaller. And this is like what you're saying, Eddy, where, like, sometimes it gets too big and too painful. One way to move the teeter-totter is to just make it smaller and smaller and smaller and smaller without grinding too much into it.

But there's a bunch of stuff that I was reading up on dopamine, and it talks about, like, the writing 250 words in 15 minutes. I've had times where it's been I'm going to intentionally focus, and I'm going to write one sentence. And then, I'm going to step back, and I'm going to look at the sentence I wrote and go, "Yes, I wrote a sentence." And then, I'm going to step in and say, "I'm going to write one sentence," and it's this winch.

And you look at it, and you think that's not going to be satisfying. One sentence doesn't mean anything. And that's how you poison yourself. That's how you talk yourself into this; oh, it's my whole writing career that's at stake because I've discounted that. Because guess what? That whole writing career is based on one sentence, and then another sentence, and then another sentence.

EDDY: You know, early, early, early on in my career when I wanted to penetrate the industry, I read something, or someone told me, I don't remember, but it resonated with me forever. And they basically said, "Starve yourself from dopening." Let me elaborate a little bit on that. And you probably have heard this. But instinctively, what we want to do is we want to reach out to a phone, you know, and browse, whatever, play a game, read something, go through the social media, whatever, you know, and, like, you're inducing your brain with dopamine. You're rewarding it, essentially.

DAVID: Rewarding yourself for doing nothing.

EDDY: For nothing, exactly. And there is a limit to how much your brain, and I'm paraphrasing here, obviously, but there's a limit to how much dopamine your brain gets. And if you starve yourself for those rewards and instead go cold turkey and just don't do it, you'll prioritize the things that you want to achieve first. And then, you reward yourself, you know, with getting on your phone, as an example. And it's just an easier way to kind of manipulate your habits.

DAVID: Yeah. The small slicing on the dopamine, the trick that you're trying to get into, is to trick yourself into—you write the sentence. And this is going to sound stupid, but I've literally done this this month—where I hate exercising. I'm just not built for it. And I was walking down the street, and I literally looked at my neighbor's tree, and I said, "I'm going to walk that far on the sidewalk." And as soon as I got there, I'm like, okay, yay, I did it. Good. Next tree, literally 15 feet. I'm going to walk to that one.

It's tiny, and it's stupid, but you're conditioning yourself to get that little, tiny hit of dopamine from chasing the task rather than completing the task. You want to move. If you focus on getting the reward, you'll actually drive yourself into a negative cycle, where it'll actually taper out. And if you focus on the pursuit of the task, you drive it into where all of a sudden, it's the chasing that gets after it that you enjoy. You don't enjoy winning the marathon. You enjoy grinding through, you know, pounding through mile 17, like I know anything about this. I'm literally talking a minute ago about walking from one tree to the next.

JUSTIN: [laughs].

DAVID: You know, but finishing the program is less exciting to me than it is to just bashing out one more module, one more function, one more refactoring. It's very satisfying to me. Okay, Justin, bring us home. What was the fourth one?

JUSTIN: The fourth one is make it satisfying, and by that, he has a couple of examples of using reinforcement rewards. So, once you've completed your habit, have, you know, some sort of reward.

Next one is habit tracking, you know, writing down in your journal or, you know, recording your exercise by a Fitbit or some other exercise tracker.

Number three is accountability partnerships, so doing your reward and working with somebody and letting them know that you did it. And it gives you a sense of accountability that you are working with somebody to accomplish something together.

And then, finally, gamify the process, which is, you know, doing some sort of competition that incentivizes you to complete, you know, the new habit that you're trying to do.

EDDY: Justin, I can't stress enough the tracking part of it. I'm sure all of you have heard, to a degree, P90X. One of the quotes that's said in one of those episodes is track, track, track. And, like, the way he kind of related to that is, if you don't track your progress on what you're doing, then how do you know you're getting better? Write it in a notebook. Write it in a whiteboard. It doesn't matter.

Because let's say, for example, your first week, you can only do one push-up. So, you go on the whiteboard, and you go, cool, this week, one push-up, right? The following week, you can look at that, and you're like, cool, I started with only being able to do one push-up. Now I did two, right? So, now you're able to contrast, and you're able to see easily the progress, your achievement. And so, being able to track your progress really helps to keep motivated, and I can't stress that enough.

There are times where we want to achieve something and, like, we feel unmotivated because we don't realize how far we've actually come because we're too busy looking at the actual ambition, you know, that we wanted to achieve, but we're completely lacking all the steps that we took already.

DAVID: Tracking is key. I had a boss years and years ago who said...he was like six foot six, and he loved playing basketball, and I was not a very competitive person; I'm still not. But he was trying to explain to me the notion of competition, and he basically said, "If you're not keeping score, why play?" He didn't want to see other people lose, but he wanted to rack up the highest score possible. And he loved it when he had somebody who forced him to score well in order to play. So, it was a very positive style of competition, but he was very enthusiastically competitive about a lot of things.

Dovetailing on that, Goodhart's Law, I think, is what it's called, which is when a metric becomes a goal, it ceases to be a good metric. But the reverse of this is when you gamify something, some things we try to reach an objective, and the goal is to get the objective and win and finish. And other times the objective is to keep the game going, keep it playing, keep it moving for as far as you can. And when you gamify something, you're kind of weaponizing the opposite of Goodhart's Law. You're basically saying, "Oh, I got two points. I got three points. I did this."

Circling around to Justin, because we spent half our time talking about what this is, and I feel like it's important.

JUSTIN: [laughs].

DAVID: But, like, segueing into, like, engineering habits, like, we have, you know, multiple websites that keep track of how many pull requests we've opened, and how many reviews we've done, and how many lines of code we've turned in. All of these we know they are bunk metrics for code quality and how good your project is and all that stuff. But as a scoreboard for gamifying something, it's very satisfying.

JUSTIN: [laughs]

DAVID: And that keeps you playing the game, and that makes you get better. And that indirectly causes the code quality to go up and the satisfaction to go up. Very, very interesting.

EDDY: I don't think that kind of works for me, though, Dave. I'm not competitive by nature --

DAVID: Neither am I.

EDDY: I'm okay with losing, and I'm comfortable with it.

DAVID: Yes.

EDDY: But some people utterly are distraught by the idea of, like, I cannot lose, and if I lose, I hate myself [laughs].

DAVID: Yeah. Yeah. And I am not that at all. But I am not immune to the pleasure of seeing the line go up when I measure something.

JUSTIN: I think it's interesting how you don't have this one-size-fits-all solution. I think the emphasis here is do what works for you. If gamify works for you, gamify it. If, you know, small rewards work for you...we're trying to follow these four laws, but at the same time, there's lots of ways that we can accomplish them.

DAVID: There's developers that I've worked with who you can set your watch to them. They show up at 8:00 o'clock. They pound out a thousand lines of code. They go home at 5:00, day in day out. And they're just machines. And that is not me. I am very much not somebody to mark time on a clock. And I felt really bad about it until somebody pointed out. They were like, "Dude, when you see a rabbit run across the field, you are off like a shot, and we don't see you for three days."

JUSTIN: [laughs].

DAVID: And I'm like, yeah, I will tear off after that field, and my wife will be like, "I guess I'll see you Tuesday." And then, Tuesday, she's like, "Will I see you Tuesday?" I need more fingers than the fingers of one hand to count the number of places I have worked at where I took a sleeping bag to work because I was in the habit of sleeping under my desk to do, like, a 36-hour rotation multiple times. That's my privilege and the way my family grew up. I had that time and a wife that was willing to support that and no kids at home. But it was an opportunity to do some very, very hard work, and I did it, and that was me chasing after rabbits.

So, if there's blood in the water, I go full shark mode. I try to bait the hook with something bloody and fresh and like, ooh, let's go do this thing. And I try to be compassionate with myself when 8:00 o'clock rolls around in the morning, and I'm like, I can't. I can't. It's just, ah, you know. I try to get the steadiness back in. And meanwhile, I lean on the guy on the team who, at 5:01, he's not around to answer the phone because he did what works for him, right? He marked his time. You need both kinds of people. You need dragon slayers, and you need bear skinners.

There is something that you touched on, Justin, and it circles back on the dopamine thing. Charles Duhigg wrote The Power of Habit. There's two books by that name, by the way, so Charles Duhigg is the one that I really liked. They were studying housewives. They put cameras in the bedroom to film them cleaning the room. And there were some wives who...this study was from the 60s, so it's necessarily very sexist.

JUSTIN: [laughs].

DAVID: But there were these housewives who kept an immaculately spotless bedroom, and there were housewives who struggled. And they asked them to just go in and clean, and make the bed, and da da da da. And one of the researchers, he's like...I want to give this guy just the biggest hug in the world because one of the researchers noticed a very subtle thing. It was like, if you blink, you miss it, but once you see it, you can't unsee it.

The wives who were very, very organized they would come in. They would make the bed, and after they make the bed, they would step back and they would just look at it for just a second. And then they would walk out. What they're doing is they're charging up the dopamine. They're basically doing the I did a thing. I made a clean spot. Go me. This is awesome. This is what I want. That's part of, like, the make it satisfying. So, jumping finally into engineering habits.

JUSTIN: [laughs].

DAVID: I don't know if this is [inaudible 27:24] be a graphic. I'll throw one out, and then I'll ask you guys what yours are. But I have a hook that when I want to push up a pull request, literally, there's a script called git cram, and it crams the PR into the server and tracks it and says, "I want to open a PR on this." And the last thing that git cram does is it puts a message in pretty colored font that says, "Did you run RuboCop?"

JUSTIN: [laughs].

DAVID: I always forget to run RuboCop. And I hate RuboCop because it's very much of, you know, organized. We're going to do this organization, whether it makes the code readable or not. I don't know if I can ever be friends with B Batsov. That's the guy who maintains RuboCop. But when I see that sign, I crack open RuboCop, and I just look for the stuff that's low-hanging fruit, and I just do one or two. Maybe I do the dash a and do all the low-hanging fruit.

Then I step back and go, oh, that is actually nicer. I do like this. I made a clean spot. And I might stop there. Half the time, I stop there and say, "You know what? The rest of this is legacy code, and changing that's really tricky." And I don't want to fight that dragon right now." But other times, I'm like, "Ooh, I'm on a roll. I'm going to finish this." And then, you get to that magical time when you run RuboCop, and you get a green dot.

I don't know if the code is truly more readable, but I've got a, and, again, that's, like, the Goodhart's Law. As a metric, it doesn't mean the code is perfect, but I do have something that I've gamified. And I've got a green dot on the board, and that makes me very, very happy.

JUSTIN: And I think it's interesting because you mentioned the green dot, and I don't think that's a mistake at all that green dots are all over the place where it says that you are doing something good. You got green dots when your unit tests pass. You got green dots when you push something that is all clean and everything. And so, that green is just like, hey, I just did something, and it feeds a little jolt of dopamine that you're happy, and that makes you happy.

DAVID: Yeah. There was a phrase going around in the noughties: test infected. And people in the Agile community, especially, were banding this, "Oh, I'm test-infected. I want to write tests. I want to do unit testing." And the hallmark of this is you get addicted to the green bar.

JUSTIN: [laughs].

DAVID: And it's real. I have, like, physically felt uncomfortable because I had not seen a green bar in a long time. People who have paired with me know that I have colored lights, a light strip over on the wall. And when I run my specs, they turn blue, and when it finishes, that whole light strip on the wall over my desk turns red or green.

JUSTIN: Wow.

DAVID: And it is very, very satisfying. People that I work with they don't pay attention to that. They don't track it and do the, "Ooh, that's good," or "Oh, that's bad." You get blind to it. You don't focus on what the output of your specs are. And so, you don't notice you've got pending specs, and you've got warnings, and you've got errors. You just ignore them. You're like, yeah, whatever, specs, whatever. And I'm over here going, that turns the spec output yellow! Yellow is not green! It's...no! Don't do that!

JUSTIN: [laughs].

DAVID: I feel bad! You know, and everyone's like, "Dave, calm down. What are you freaking out about?" I'm like, "I need my green specs! I need green dots."

JUSTIN: All the methodologies that are out there to help us be more productive and things like that, how much that feeds back into these habit-forming things. We talked about the habits of breaking the thing that we want to do down into smaller and smaller parts. That kind of rolls right into breaking our stories down into more manageable stories where we can get that shot of dopamine that we completed a thing within a day or something like that, rather than trying to work on this new feature that's one story for a week or two weeks, or something like that. Having that tighter cycle benefits us in so many ways. And it kind of rolls right into this habit-forming that we have talked about.

DAVID: Absolutely. Absolutely. There are a whole bunch of habits that I have that somehow got filed into the banks of my brain as Engineering Discipline: Capital E, capital D. And it was before I knew what habits were. It was just like, if I'm a good engineer, I have to do these things. If there is a holy first commandment of engineering discipline is, do not commit code that you have not run the tests on. And it is very rare, not impossible, that it happens. It's very rare that when I push a pull request up to GitHub, Jenkins pulls it down and runs it. It's very rare that CI fails on me.

And when it does fail, half the time, it's because of a flaky test or something in the system broke rather than what I did, and it's because I'm religious about it. And when I find myself in a spot where I can't run my tests the way I want, or I'm working on something that I can't lock down, and I get the computer to tell me, "Yes, this does what we think it does," first of all, I get very, very anxious. And secondly, my compromise, my fallback engineering discipline is I commit myself to chiseling away at whatever is stopping me from being able to run my tests.

So, on the team that I'm on, if you run any test suite with database-seeded data, it takes 90 seconds to start up the test, and if you've got ADHD, that's awful. I'm in the kitchen making a sandwich. I'm talking to Liz. I'm at the grocery store picking up a soda pop. And, like, wait a minute, how did I literally end up across town? I'm supposed to be running my test.

JUSTIN: [laughs].

DAVID: So, I figured out how to chisel away the tests so that they don't depend on the seeds. And I will run my tests without the seed thing on. And I was pairing with a programmer earlier this week who shall remain nameless to protect me from the innocent. They did bin/rspec.

And he leaned back and he said, "Now we wait for a minute and a half." And I'm like, "Do you have spec seed turned on in your bash profile?" And he said, "Yep." And I said, "You don't run your tests unless you have to. And you run them at the very end, and you run big, long steps before you..." And he's like, "How do you know all this?" And I'm like, "Because your tests hurt you. And it's like, do they have to hurt this much?" So, that might be an engineering habit of if something hurts, try to fix it. If you can't fix it, chip at it. Just knock a piece of it.

Power of Habit from Duhigg was recommended to me by Amy Hoy, and she did an online entrepreneurial class called 30x500 that I took and I really, really enjoyed. And one of the concepts she talks about is she calls it the winch. And the idea is you can succeed with just an inch of progress if you can make an inch and not lose it—if you can winch yourself forward just a little teeny tiny bit.

So, that makes you want to step back and say, if I work on this, am I slapping at it, and then I turn away, and it just comes back and goes back to rest? Because if that's the problem, I've got to pick it up. I've got to carry it across the lawn and set it down, or I can't make any progress because it's going to blow back and forth.

But if I can find something and lock it down, like git cram, for example, that little thing, that is an actual script on my computer in my path. And it just does git push and git set origin and track and then, you know, display the message. And when I first did it, all it did was push and do a track. There wasn't anything else to it, and it just kind of winched forward from there. I have a bin repo up on GitHub that's just all my public stuff that I put on all my machines. There's, like, 200 scripts in there now that I've built over 15 years.

JUSTIN: [laughs].

DAVID: I'm really proud of Git RuboCop because I type Git space Rubocop, and it runs RuboCop on the files that have changed since I last diverged from master, which, okay, that's not that hard to do. That's just RuboCop and a list of files—getting that list of files. I wrote another script called Git files changed that will tell you what changed, little, tiny thing. It makes the job easier, and then you come back to it.

You do have to be careful not to spend 40 hours gaining that one inch on something that your product owner doesn't need because that can cost you your job, and eventually, it will cost you your career. You have to learn how do I take five minutes on this and gain two hours down the road? There's an old xkcd comic about how long it takes you to do something versus how much time it saves you, versus how many times a day or week that you run this thing. It's just a chart, and it tells you this is how long before it will pay off.

And at the end of the chart, it's like, this is going to take ten years to pay off because you run this once a year, and you spent 40 hours saving 2 minutes on it. You spent 2 hours on it, but you run it 30 times a day, and you only save 2 seconds, but that's going to pay off in, like, three weeks. And you're like, hmm, interesting. What are some things that you guys do that's down at the habit level that you don't have to think about; you don't have to summon this force of energy to tackle? You're just like, oh yeah, I got to do the thing.

MIKE: One thing that's been on my mind as we've been talking about this, you know, there's all these little tricks we can do, and there's some utility to them. And sometimes, maybe you're not going to feel like it [laughs]. There's maybe two related things that I'm thinking of connected to this. So, I'm going to follow this thread first. I ran cross country in high school. For everybody's who's done that, it just hurts.

JUSTIN: [laughs].

MIKE: I'm just going to [inaudible 36:11]. And, actually, I had somebody tell me, a fellow cross country team member, that long-distance running is mostly about pain management. You know, the people who are really good at it get really good at tolerating pain and, you know, learning to kind of segment that off in their brain. And I thought, oh, that's interesting. I don't know how I feel about that [laughter]. I'm not vouching for the truth of that. It's actually not quite my point. My point is that if you think about what's happening to you physically, it's not necessarily fun. There are some, you know, endorphins that come from it that are nice.

But I loved it, not because I necessarily loved the pain that I was going through [chuckles]. And I wasn't particularly good either, by the way. I started the team late. I remember my first race I got last place [laughter]. Don't think of me as a great runner. I got better, but, you know, never great, but I just loved running around outside. I got to run around outside. The sense of capturing the beauty and awe of the world around me still sticks with me. I loved doing that.

I noticed that Justin's profile image looked like he was hiking somewhere [laughter]. You know, hiking is not necessarily different. You go up there because you want to experience that beauty, you know, have some awe, some sense of being in something bigger than yourself. And it's related to the idea of a flow state. You know, people tend to become really successful when they are not immediately aware of what they're doing, but instead, they feel, like, one with the work.

And similar to the hiking, to the running, doing something deliberately to be meditative, and developing that sense of meditative immersion, you might think, well, I can't develop that. But I think we can. It's something that you practice. It's like other things that you can work on. You don't necessarily go into the day looking for that hit of dopamine, or endorphins, or whatever it is. Instead, if you go in just as kind of an open slate, like, what am I going to experience [laughs]? It kind of washes over you. Rather than value judgments to your experience, instead, treat that experience as an adventure.

And that subtle mindset shift it takes some work. It doesn't just happen, but it can change the way that you approach your work. That is one habit, I think, that has benefited me a lot is developing that kind of mindset that I can be there. I can have...the world is on fire [laughs]. Things are broken [laughs]. This is...I want to be cautious here. I'm not saying that we should be in an abusive or in a deliberately bad situation.

However, developing the ability to consciously put yourself in that situation and say, "Well, you know, this does matter to me. I want to provide for myself and my family. This is something I want to be doing." Then you can make a conscious choice to put yourself in that situation and go through all of the fiery arrows coming at you and find it fascinating, you know, and see it as, like I say, an adventure.

And that particular approach works for me far better than being competitive. I don't really care if I win, but that sense of adventure, that sense of awe at the world around me is satisfying or is motivating in a way that nothing else can be. When everything else is bad, which sometimes it will be, if I can do it anyway because I've chosen to have this experience, it changes my mindset. And that's something that has, I think, helped me a lot.

And a minor point, honestly: getting exercise when I can helps me to be a developer. It just helps your mind work better. All the tricks we talked about help with exercise. You know, help [inaudible 39:38] with somebody, go out with my kids, helps a lot. But even more so, developing that different mindset about the work, I think, is a big deal.

DAVID: There's a thing that resonates so strong with me with what you just said. I can't remember where I heard it. I want to say it might have been Jordan Peterson. The comment that they made was bearing your cross is only bearable if it's meaningful. Otherwise, it's just a hell that you have to go through. The world's on fire, and people are upset, and there's money being lost, and people are screaming, and dah, dah, dah, dah, dah. And you're just in there going, "I'm providing for my family." I don't know if I said this on the podcast. I know I've said this to you before, Mike, that sometimes the good money is to be found by shoveling crap out of a latrine.

JUSTIN: [laughs].

DAVID: And I have reached a point in my career where I can get down in that stinky, smelly, awful, terrible, and I don't think too much about what I'm up to my hips in. I just start shoveling because I know I'm making a clean spot. I'm making a clean hole that we can get some work done and get some, you know, some stuff done. So, yeah, you're not down there going, "Oh, yay. Look at all this smelly poop on my shovel." No, it's, I'm making a clean spot. Every bit I lift is one tiny, little bit of poo out of this hole. That's what we go after.

The competition thing, like I said, I'm not competitive against other humans. Like, I can't stand to see other people be sad or be frustrated. But, for me, watching the number go up, like, the gamifying, so I will compete with myself. And so, I think it's important to find what is it that makes you tick?

I've worked with you, Mike, enough to know that you have, like, a very meditative presence to things. And I can picture you just standing in the garden watching the azaleas and thinking, I need to prune the grapes back. I hate being outdoors. I hate plants. I'm allergic to anything if it's got chlorophyll in it. And I was like, what an awful way to do it. And I can see you in that environment going, the sun is shining. There's a beautiful breeze. There's a ton of work to do. And that's not what I'm here about. I'm here about this moment, this meditative moment that I can be present in. So, we've gone around the horn quite a bit. Are there any other tricks and hacks?

JUSTIN: I mean, we spent a lot of time kind of defining what makes a good habit or how to have a good habit. I almost, like, want to have a part two of this podcast that, like, goes more into the good habits.

DAVID: I'm realizing, like, one of the...I talked about my bin folder, and I thought, well, that's stupid. It's so small that nobody will care about it and value it, and I realized I've got a thousand of these little things. I have alarms on my phone. I have an alarm on my phone that tells me to check my alarms. I'm not kidding when I say that.

JUSTIN: [laughs]

DAVID: And what it is, like, I have time blindness. I can sit down at the computer. There's Slack, and there's email. There's a problem; there's a bug, da da da da da. [inaudible 42:20] is going off. And all of a sudden, it's 1:30 in the afternoon, and I haven't checked email, and there's a hot message on Slack that somebody's been trying to get a hold...right? So, I've got a checklist. It's like a pilot's checklist for flying a plane of: check this, sign in. You know, we have four different single sign-in thing, which is, you know, increasingly erroneously named single. We have four SO, I guess.

Sign into all four systems, then, you know, go do this. And that means you have to hit this one system that you don't normally do, which, oh, yes, I need to log in there. I need to check to see if I've got any trainings outstanding in our online training stuff. And it's on a checklist. The training thing is a good example. If I don't have an alarm on my clock to remind me to check, the only way I get my trainings done is when I literally start getting emails from the legal department saying, "We're about to be out of compliance because not all our people are trained," and you are that people.

JUSTIN: [laughs].

DAVID: Oh, okay yeah. So yeah, I probably got a dozen of these that I realize are now probably too small. On the larger side of things, there's a couple of people that I've talked to in the past like Dan [SP] Cub is somebody who comes to mind. He's kind of a minor luminary, moderate luminary, not minor. He's moderately famous in the Ruby community.

He mindfully and intentfully writes experiments, you know, hypothesis. I'm going to try this, and I'm going to document and capture. And he will have, like, five code experiments running every week, and I've stolen that from him. And so, like, this week, I'm trying a thing where I write a test that's useful to me, and then I delete it. And I push it up.

And I just got feedback from a co-worker going, "Where are the tests?" And I'm like, "Yeah, they weren't any good." Maybe I should write some good ones. But it's an experiment, right? Try and...and the habit is try and experiment. Justin, I think you're right. I think a follow-up on this would be really, really good. I feel like we're getting to a good close for this one. Any final thoughts out the door?

EDDY: I will say the thing that attributes to success, at least for me, was getting compensated for it. Like, that made loads of difference [laughter]. Circling back to development, right? Like, you can tell someone, "Yeah, if you have the discipline, you can teach yourself." But what they don't really understand is it's a full-time gig, you know, doing that without any sort of compensation. And so, you're doing two full-time jobs or a full-time job and a part-time job. And it's really hard to continue to keep stabbing and stabbing and stabbing when you're not getting paid for it, right?

DAVID: Yeah.

EDDY: So, I will go on record to say if someone, you know, was trying to get good habits to penetrate the career, you know, and, like, establish some sort of status in development, getting into something, you know, that's remotely close to what you're trying to do really, really, really, really, really aids, you know, in success.

DAVID: There's a whole thing we could dive into about career-adjacent activities, where you can expose yourself to an environment where somebody taps you on the shoulder and say, "Hey, do you want to come play with us?" And it's just blind luck. It's not blind luck. You sat down in the lunchroom next to the people doing the thing that you wanted to do.

EDDY: Exactly. And, like, that can be applied to anything in life. Like, you want to get really good at basketball. You want to join the NBA. You know, you have to be comfortable doing that for hours and hours, for months and months without getting paid. Same thing, like; I feel like the surge of YouTube is a huge example. Like, if you're a YouTuber and you're like, "Oh yeah, I want to monopolize, and I want to make money." [laughs] But what they don't understand, you know, is that people who found success in that platform did it and grinded for years.

DAVID: For two years before they broke a thousand viewers, yeah.

EDDY: Exactly.

DAVID: Chris Williamson and Evan and Katelyn are two people that are really, really big that I like watching right now. And both of them talked about: we did this for years and no payoff. And it's interesting that they said they didn't do it investing in their future. They did it because they're like, this is the grind. We're going to get on this. This is going to be the grind that we're going to be on, and we're going to do this. And they focused on just do the grind.

Don't focus on the reward because the reward is going to hit, and it's going to be pale. You're going to be like, I did all that for this? No, you did all that in order to spend two years of your life on a thing that you can now look back and say, "I'm two years older, and I have all these things behind me." Cultivate that itch to learn something, and that'll keep you up at night reading programming books.

EDDY: [inaudible 46:43] compensation, like I'm saying. I can't stress enough how much value that helps [laughs] that provides, you know, the success.

DAVID: I'll throw this out as my closing thought on it, regarding compensation. I've always had an itch to learn. My wife was very frustrated. We were not making much money, and I was buying very expensive books, like, $100, $150-books in 1990 money, so, like, $50 books back then was a lot of money. And my wife just shook her head. She's like, "I would be mad, except that you read them, and so I can't complain."

JUSTIN: [laughs]

DAVID: And I got into Perl programming, like, in 1999 or so. And in 2002, my Perl number was, like, three, I think? I'd never met Larry Wall, and I had never been asked to maintain the Perl tarball. Like, nobody knew who I was, but if you had a Perl question, I knew the answer.

And I had a co-worker that came over to me, and he says, "Hey, how do you get the length of an array in Perl?" And I'm like, "Oh, it's..." and I gave him the line noise string of like, you know, pounds, dollars, you know, whatever it is to do that in Perl. "And hey, how do you do a file X?" "Oh, it's this." "Hey, how do I check for a range of things in Perl?" "Oh, it's this operator." "Well, how..."

And about the 10th time he asked me a question, I'm literally writing code, and I don't even...I'm just like, "Oh, it's this," and I keep typing. And I wasn't aware that I was doing it until he stopped and he goes, "Do you just know everything about Perl?" And I'm like, "What? What?" I have been glowing and coasting on that compliment for 20 years [laughter]. That happened in 2000, 23 years. That happened in Y2K, and I'm still happy. I was so startled. I'm like, oh my gosh, I'm not any smarter. You just asked questions that I knew the answers to, which turned out to be all of them.

EDDY: I do want to say, though, that the outcome could have been not drastically different but, like, you could have had a different outcome had you been given those books to you versus you having skin in the game and investing hundreds of dollars, right?

DAVID: Yeah. For me, that is the key difference between being invested in the payoff, in the reward, versus being about the grind. It's like, I'm going to do this. I'm going to put this effort in. You say compensation, and I'm 100% on board for compensation in the sense of I'm going to get myself on a payroll because then it's part of the grind. It's like early salary negotiations for me where it's like, well, I can tell that this number is bigger than that number, and I like the line to go up. That is a motivating thing.

EDDY: I typically when someone...because I've had people who have said, "Hey, well, how do you attribute your success?" And I say, quote, unquote, "Success," because, like, that's very subjective. But, like, there's people who do aspire, like, doing something that Ramsey and I have done, where we have been able to penetrate something without prior experience. And typically, my go-to advice has always been grind, discipline, and spending money on resources. And they're like, "Oh, I don't know if I want to spend money on it." Like, if you spend money on something, suddenly, you're invested [laughs].

DAVID: And I would call that investment. I had a co-worker years ago who had 13 children. And we were talking about something, and I had stayed up all night reading about it. And I'm like, "You're not reading about this in the office. How do you have time to have read about this?" And he's like, "I get up at 4:00 o'clock in the morning, and I have about an hour where it's quiet." And so, I'm like, "Okay, so you're a lot more focused because you've only got an hour before all hell breaks loose and pandemonium breaks loose in your house." And he's like, "Yeah, so I use that hour very intentionally."

And we were comparing notes and I'm like, "Yeah, you're getting about three times the efficiency because you're investing a resource that, for you, is so much more scarce." It'd be like, you and I go into the bookstore, and I buy a book for $50, and you buy the same book. But you have to pay $750 for it.

EDDY: Huge.

DAVID: One of us is going to read that book cover to cover and then eat the cover.

EDDY: [laughs]

DAVID: Ah, this was a good talk. I do want to carry this on again. Everyone listening, thank you for coming. This was awesome. I'm looking forward to talking about this again.

EDDY: Let's do a part two.

In this episode of The Acima Development Podcast, host Mike Challis introduces a discussion on code reviews by drawing an analogy between creating music and coding while sharing personal stories about feedback in musical endeavors to illustrate the importance of constructive criticism. These anecdotes highlight the parallels between receiving feedback on creative projects, like music, and technical work, such as code, emphasizing how both require openness to feedback for improvement.

The conversation then goes into the specifics of code reviews, highlighting their value in the software development process and how code reviews are akin to collaborations in music, where input from multiple contributors often leads to a superior output. The panelists explore the nuances of effective code reviews, including the balance between too many and too few contributors, the importance of automation in enforcing style guidelines, and the value of comments within the review process to provide context and facilitate understanding.

The episode concludes with a deep dive into the best practices for conducting code reviews, addressing the challenges of pushback, the significance of understanding the reviewer's perspective, and strategies for ensuring constructive and efficient feedback. The conversation also touches on the importance of security considerations during the review process and the potential pitfalls of approving changes prematurely.

Transcript:

MIKE:  Hello, and welcome to another episode of the Acima Development podcast. I'm Mike, and I'll be hosting today. Here we have with us Eddy, Justin, and Kyle who've been with us before. And we are going to be talking about code reviews. I like to start with a story, and I'm going to do that.

Sometime recently, we talked about music and about how there's often overlap between people who make music and people who write code. I'm going to build on that a little bit. I like to write music. I sometimes just write music for fun. I do stuff my kids like [laughs]. Their favorite song is something I did; it was just kind of a personal challenge. You know the video game Plants vs. Zombies?

JUSTIN: Mm-hmm.

MIKE: I [laughs] listened to the soundtrack of that, and I thought, if I were to write the soundtrack, what would I do? And I wrote a song that kind of captured my take on it. And my kids love it. That's the one they ask me to play most [laughter]. So, I feel pretty good about that. They are honest judges. And, you know, they do, you know, they give me commentary on it. And sometimes there's a situation like that, and sometimes there's something a little bit more formal.

A serious example: I had a friend who passed away a few years ago, and they asked me to do an arrangement for her funeral. And I got together with the woman who was going to be singing. I did an arrangement, and I brought it to the singer, and she gave me feedback. And she said, "Well, you know, I think that there's too much build in the second verse, and I think we should move that to the third verse." And, honestly, she was right [laughs] in kind of every respect. She had the feedback that was necessary in order to improve that arrangement, you know, for something that we both cared about. That's really relevant, and that's not the only time that's happened.

I've got some friends that got a high schooler who's in...he does a lot of singing and drama. He's involved in a lot of those things. And I've done something with him before, too. Once again, I gave it to the singer. He came back with feedback and said, "Well, you know, I think we should change this. I think the melody is a little too repetitive. And I think if it went up here instead of down, that'd be better." And he was right. It made a genuine improvement to the structure of the music I put together.

I think that this story is really instructive. It's a review of something creative that is not in code, but it has many of the same characteristics as what we do in code, sometimes anybody who creates something. There's an intro to a song by a musician, like, a soul singer. She says [chuckles], "You know, remember, I'm an artist. I'm sensitive [laughs]. So, we feel that way, you know." Like, I'm an artist. I make this beautiful, perfect thing. And we think, well, you know, if anybody is giving me any feedback about this, they're criticizing my personal character and saying that I'm just not good enough. It's a really dangerous thing, both for ourselves and our art, I think, to start thinking that way.

It doesn't matter really how brilliant we think we are [laughs], and maybe we are. You know, maybe you have beautiful art that you create. Getting feedback on what you've created allows you to see it from a different perspective. We are all limited by the scope of what we can see with our own perspective. And seeing something from somebody else's perspective can help open up new views into something that may be wonderful.

You may have built something that's truly good. And when an eager collaborator is coming to work with you, they're not interested in tearing you down. They're interested in helping you make it better and in helping you achieve your vision. And they can say, "You know, I think that you were trying to do this. I don't think you quite got there," and help you better accomplish your purpose that you had initially. That framing, for me, for reviews is extremely helpful.

Now, we're going to be talking about code reviews, you know, that's perhaps more technical, maybe [laughs], than some of the stuff that comes from other creative endeavors, but it has many of the same characteristics. We're trying to solve a problem. And getting that second pair of eyes can help us solve the problem better in many cases. Also, the collaborator, in general, and if you're not doing this, you're doing it wrong [laughs], is trying to help you accomplish your purpose, not to tear you down, but to just help you do it better. And that second pair of eyes, that sounding board, is extremely helpful. So, that's the framing I wanted to start with. Does anybody have any thoughts about that?

JUSTIN: It's interesting because you see a lot of people that do great work, like, artists or whatever, all by themselves. And then they get together with somebody, and they do a collaboration. And some of the best music comes out of collaborations. And it's like, a couple of my favorite songs have, like, a lead artist, and then they have some lesser-known artist that is coming in and doing a riff or something like that. But you can tell that they meshed together their ideas and just made something so much better.

If you think about music these days, it's not just, you know, people don't just, like, or most of the time, they don't just slam something down, and it's done. It is a work of months and months and months, and they're tweaking every little thing. If you get somebody else in on that, it makes that tweaking move much faster because you have somebody else that you're brainstorming with to solve the problem. And I personally like pair programming a lot because we are talking about code reviews in this. But the pair programming part it's like an ongoing code review as you go in and solve the issue that you're doing together. We can talk about pair programming later, but those are my thoughts on that.

EDDY: As a band, you know, the more people that collaborate, it's an interesting dynamic because I associate that the more people you have in the pot, the slower it moves because you have more opinions. And so, because music is pretty subjective, the productivity does go down slightly, you know, the more people have input, kind of similar to the code, right? Like, if you're a single code developer [inaudible 05:52] to reviewers, the amount of feedback you get is much lower than if you have five or six people looking into these code changes, right?

JUSTIN: Yeah, I think there's definitely a point of diminishing returns as you add more people to it. That's a pretty steep curve [laughs], and so...

MIKE: You have bands with a couple of people. You have bands with three people. Four are pretty common. Very rarely do you have, like, 20. You've got like, what, Lynyrd Skynyrd, I don't know [laughter]. There's some crazy bands that have huge numbers of people, sort of a collective. But usually, you know, you just have a few because there's the limits, right? There's limits to what you can collaborate together on successfully.

JUSTIN: Though I do love big band music. Give me a good, big band. But, at that point, it's like you have a conductor that is leading the show. Maybe the comparison breaks down a little bit there, but anyway.

MIKE: Well, it's interesting. I mean, I was thinking similar. If you have a jazz combo, you know, a small jazz group, then there's a lot of room for creativity, and they can work together. When you get to the big band, you have a composer, right [laughs]? You have somebody who writes out the music, writes out the lead sheets, and gives it to people. It just changes the way you work.

You know, kind of going back to this idea that there's value in getting the collaborators–when we write code, it's very much common practice in the industry to have a code review or maybe pair programming, as Justin said, which is kind of a live code review, or you're collaborating in real-time, which is another way of providing value, right [chuckles]? Real-time collaboration provides many of the same benefits. It's normal industry to have a code review.

There are a few different purposes that could be proposed for that code review and that are sometimes used, and some of them, I think, have different value than others. For example, if you're sending some code over to code review and the primary feedback you get is stylistic, then I think that was a wasted opportunity [chuckles]. To say that again, I think that is a wasted opportunity.

There are automated tools in probably the language that they're using; if they're not, it would surprise me [chuckles]. There are automated tools in the language that you are using that can enforce style guidelines. And you can download the style guidelines from the internet and use what some big company uses and have a standard on your team. And you can make some tweaks to that. But the automated tool can enforce, you know, whether you use tabs versus spaces. It can enforce what your quotes look like. If you're having arguments about that, then I think you've got a more fundamental problem.

The place that we should start when you're working with the team is choosing the conventions that you're going to use, and that should just be the baseline, you know, that's the foundation. And if those come up in reviews, that means that somebody has either ignored those rules, and that's a different issue [chuckles], or you really haven't gotten consensus on your convention, and you need to go back to revisit the convention. So, I say this one more time, you know, I don't think that that is a valuable use of a code review.

To go to the music example, if you're a guitarist, you probably don't want somebody to say, "Well, no, I think you should pull out a trumpet." "What? [laughs] I don't know what you're talking about. We're a band that does not have trumpets." That's not what you're there for. But if somebody says, "Hey, you know, I think you could change the rhythm on your riff," that's a different kind of comment. It's commenting about the structure of what you've created in a way that is meaningful. The style should just be a given going in. Anybody have thoughts on that particular idea?

KYLE: I kind of agree, even the system's code that we have, you know, using Terraform, and I know that a lot of other languages have similar formatting tools, right? We've kind of just agreed, you know, we're going to use the Terraform format command. That is a quote, unquote "reviewer" per se in the steps in our PR, right? It's going to go through, and it will decline and say that we've failed if we haven't remembered to run that.

But then that keeps us all in line, especially where we're kind of cross-ecosystem, where some of us are running Mac, some of us are running Windows. And we avoid some of that, you know, line ending white space type issue, so that keeps us in line. And then the actual reviewers themselves they're looking at functionality of what we're actually creating the PR for rather than how does it look.

MIKE: Yeah, yeah. Absolutely. And you're not wasting your time [laughs].

KYLE: No.

MIKE: You're not wasting your time haggling over details that have been decided. If it can be automated, it probably should be. We're developers. We tell the computers what to do for us [laughs]. And they can take away some of our work.

EDDY: Would you agree, Mike, that getting an approval without any comments or "Looks good to me" would be just as bad as a wasted opportunity for stylistic comments?

MIKE: That's a great question, a loaded question [laughter] because I think we've all probably thrown an LGTM or something similar onto a review. I think it opens up a broader question, actually, and something I wanted to talk a little bit about. Have any of you read the book, The Phoenix Project? It is classic in the industry and helped give birth to the DevOps movement. So, well, I'm going to strongly recommend it. I've read it multiple times. It consolidates a lot of research into a novel so that you get a really [laughter] enjoyable novel that's noted, right? It gives you references as to where a lot of this material came from.

But one of the key themes of that book is that batch size matters. Let me explain that. There's a surprising amount of parallel between software development and physical goods manufacturing, whether it be cars or, you know, I'll say cars [laughs], the manufacturer of automobiles. And a lot of times, people say, "Well, no, I'm an artist. I read through this creative work." There's really nothing in line because, you know, I have to invent stuff all the time.

And certainly, there are differences, but in aggregate, you take a step far back enough, you're looking at people who are building things. At a high level, that is largely predictable. You can measure it, and you can streamline it, and you can do things to improve the process. And one of the things that is really bad for physical world manufacturing is large batch size.

For example, if you've got a, like, a powder coating station, and you have to have a thousand parts to go through that powder coating station, because otherwise you're wasting the resources of a big booth, then you have to wait for all of those thousand parts to be made every time you pass it through. So, you have an enforced stop in your process in order to wait for the material to pile up before you send it through. So, if you were to stand up on a catwalk up above the floor, you'd see this big pile of stuff in front of a powder coating. You could see that just at a glance that, wow, there is a bottleneck in my process right there. We can't let anything through until it's all ready.

And one thing that you can do to improve that on the assembly line is to rework your machinery so that you could have a continuous flow. So, if you could build a powder coating system that could take one part at a time, you know, you'd be required to rebuild the machining, but it would get rid of that dead spot. It'd get rid of that bottleneck in your process and reveal where the next bottleneck possibly was. But it would allow you to have continuous flow.

So, if your car, if you think about cars, if you have a bunch of auto bodies stacked up, well, that's hard to stack them up, right? How do you stack a bunch of auto bodies? It uses up space. It uses up mental energy. There's a lot of labor that goes into a large batch. And if you can reduce that batch size to a very small one, then a lot of that labor goes away. Sometimes, we think, well, I want to make sure my reviewer doesn't waste their time. So, I'm going to make sure I put a lot of code together, and I'm going to give it to them all at once.

And I think that that way of thinking is usually in error because it ignores those other costs: the cognitive costs of trying to understand a great, big change, the time costs of waiting for weeks, maybe, to have this change go through, where you could have had incremental changes, where you're putting up pieces of code during all that time and getting value out of it.

This is kind of a long answer to your question, Eddy. But this batch size, I think, is critical because if you rework your process so that small changes are going through regularly, it may be that your code changes are so obvious that somebody's going to look at them and say, "Oh yeah, well, of course, that works [laughter]. I really don't have anything to say because there's nothing left to talk about."

But that doesn't mean the conversation didn't happen because it took a lot of refinement ahead of time, earlier up that assembly line, right? Earlier in the process when you were planning out the work you were going to do when you broke it out into stories. That means that you already had those conversations as to what should be happening.

So, the conversation that might have come up in a code review did come up, whether you're pairing or whether you were refining together as a team earlier on. And so, when the reviewer finally got to it, they already knew what was going to be there. And when they looked at it, they say, "Oh, this is exactly what I expected." And in that circumstance, I think that that's actually a win. But the communication happened. If there was no communication all the way along, I would say that there's definitely a problem.

And, if you have a large batch, you've spent two weeks working on this PR, a pull request, you know, those who don't use the lingo. You're giving this to a reviewer, and they look at it for five minutes and say, "Oh yeah, that's good." I think that's a problem [chuckles]. Because you've done a big body of work there, and it needs a lot of time. Really, you probably have asked for a day's worth of that person's time to really fully understand what's going on.

JUSTIN: It's interesting, when people think of coding or when you're a coder yourself, and you're thinking, oh, my job is coding, and I'm going to put out a bunch of code. That's how I know that I'm doing a good job is: I put in a thousand lines of code today or something like that. Whereas, in reality, coding is certainly a significant part of your job, but even more important, is the planning about what you're doing and making sure that the whole process is given equal importance.

You're not just throwing your code over to be installed into production. You work with your team in planning what you're going to be working on. You work with your team on, you know, you do the actual work and then you do the code reviews, and you do the testing. Well, the testing should happen throughout and everything. And then there's the monitoring. And it's truly the software development life cycle, as they say. It's something that encompasses every part.

If you look at a typical software development life cycle graph, each of the parts is actually pretty much the same size. It kind of emphasizes that you can't just drop something. Or if you pay too much attention to a certain part, other parts may lack. But, I don't know; that's a little bit of a tangent with regards to code review, but I thought it was interesting.

KYLE: That's kind of a tangent, but, I mean, to some extent, you maybe think with a PR, in that life cycle, as you're saying, so if I'm a dev and I throw up a PR, I have the experience from developing and whatever my team might have. But I also have the option of throwing somebody from QA, or security, or DevOps onto that, you know, that are down in the life cycle or somewhere in the life cycle as well. They can review it. They can throw their experience in there that I may not have thought about, you know, get those details going, kind of like you're saying, the monitoring and all that jazz before it even hits prod.

EDDY: And, Justin, I have a question for you. So, you're in security, right?

JUSTIN: Right.

EDDY: Are you involved in any of the PRs that are going up in production at all or any?

JUSTIN: Occasionally. And it hasn't happened that often yet. It happens probably about once every other week, maybe once every three weeks, where somebody asks us to take a look at something. You know, usually, it leads to the discussion about what's your intent here? You know, kind of walk me through your solution. A lot of it is, you know, people just need reassurance that what they did is the right way.

But occasionally, we come back and say, "Hey, you know, you really should do some more validation, you know, just consider all your inputs as untrusted," or, you know, something to that effect where we talk through a particular solution. Oftentimes, it's talking about a third party and, you know, what we're sending them, or what they're sending us, and how do we trust it, or that sort of thing.

Long story short, yeah, every couple of weeks, I get pulled in to take a look at something, and it is really cool. I love doing it because, you know, having that discussion and getting into the code is one of the funnest parts of my application security job. One of the least fun parts is the policy writing, which is what I'm doing today [laughs]. But yeah, if you guys ever want me to look at something, please feel free to interrupt me. Whatever I'm doing, I'm going to drop it and come help you [laughter].

MIKE: Can I ask a follow-up question to that one?

JUSTIN: Sure.

MIKE: Would you rather talk to somebody once the code is already written or before they write the code?

JUSTIN: That's a good question. Probably both, but there are certain questions that need to be answered before the code is written. We actually are working with Anna and the product team to make sure those questions are asked at the epic level and answer to the epic level, things like, you know, are there changes in authentication or authorization? Are there changes with PII data? Are there changes with third-party interaction? Those are the main three questions that we want to ask on each epic.

A lot of times it's, like, not applicable or something like that because it's not, you know, they aren't doing that particular work or know or that sort of thing. But the times when we get brought into, when they put yes on those questions, we go in and have a conversation about exactly what's changed, and then, you know, just to make sure that those delicate subjects are treated with the proper respect that they need.

MIKE: Let me take that a little further. Would you like a software project to be written, and then you come in and say, "Okay, now it's time to add security to it"?

JUSTIN: Oh. Well, we've had that happen all the time, unfortunately [laughs].

MIKE: How well does that work?

JUSTIN: It does not work well because it messes up people's timelines. You know, oftentimes, a project is headed towards production, and then, you know, expectations are set. And then somebody asks the question, "Hey, we should have security look at this." And then, you know, we look at it, and we raise some questions.

A good example is we had a question about something the mobile team was implementing, and we had to go back and say, "Hey, what do you guys think about this?" You know, a conversation went in about a particular API they had and how they were accessing it and how it was designed. We all agreed that, hey, changes needed to be made. And so, they had to go back and add another, I think, it was another couple of days at least of development work to fix the concerns.

We really don't want to go in and dictate stuff because the key there is helping people understand why we have concerns. And once we've had that conversation, oftentimes, people are like, yeah, that makes a lot of sense. And that's what we're shooting for is like, hey, we want you to understand why we have concerns. And what results from that is sometimes we change, but also, sometimes they understand why we have concerns, and then we can go in and explain the risks and have a discussion about timelines and things like that.

MIKE: And obviously, I asked kind of a pointed question [laughter] that reveals that it is costly to make changes to a design late in the process.

JUSTIN: Yes.

MIKE: And security isn't just some dust you sprinkle on a project [laughter]. It's an integral part of the way a structure is built, you know, whether that's in physical world or in software [laughs]. You can't say, "Oh, I'm going to build a building with no thoughts as to structural integrity [laughs] initially and then add that later." It's not something you can add later. It doesn't work that way. The security is built in from the beginning. And it's not just security, right? We bring up security as an example.

If the review is the first time that important conversations are happening, I think it's a real problem. Building a little bit more on Eddy's question, you know, "It looks good to me," I think the ideal software development process, software development lifecycle, has communication early so that the review is there to have some dialogue, but it is a continuation of a conversation that was begun much earlier in the process.

JUSTIN: I think we're moving towards a, you know, what should you be looking for in a code review? And as a code reviewer, I mean, we have a couple of things. What should you do to prepare your code for code reviews? And what should you do? As a code reviewer, what should you be doing? If you guys haven't seen it yet, there is a really good example that Google gives. They have a paper that they wrote, a white paper, that has those two sections. It's like, what should I be doing to have a good code review? And then the other section is what should I do as a good code reviewer?

And if you haven't had the chance to look at that, go through it because I liked how it enumerated everything. It let me know what I was missing from both ends, you know, included things like, hey, is this performance? You know, performance is something you often don't think about that goes along with security; sometimes, you don't think about. But performance is like, you know, hey, it works for me in my development environment. And then you're like, somebody else who's doing a code review they should be aware of that. And, hopefully, their experience comes in and says, "Hey, you know, accessing this table this many times is going to lead to some issues," or something like that, so yeah.

MIKE: I fixed a performance issue during the winter, you know, Christmas time break. So [laughs], it's fresh on my mind and very much so. It's like security. You can't just bolt it on at the end. It's like, hey, we'll add performance now [laughs].

EDDY: Well, and, you know, I actually looked into that, too, when I came back, and it was really interesting because you could only replicate that in higher environments with a higher traffic count. And that's not as obvious, you know, when you're testing in a local environment, when you're truly in control of the data, right?

MIKE: Yeah, leaning on expertise and talking about that early is really helpful.

KYLE: I'm sitting here listening to this, too, and thinking through, and we're kind of saying it's hard or almost impossible to add it later, which for, like, performance and both security to the original product, yes. But I think the part that hasn't really been brought up is, so I'm thinking, performance. Yeah, if your app isn't performant, can you fix that? Sometimes you can't. How do you do that? You throw hardware at it. You throw money at it [laughter]. You get it fixed that way. [laughter] Same as security, right? You have to put another layer in front of it in order to fix your application.

So yeah, I mean, I'm agreeing with you guys, but I'm like, it becomes exponentially more costly because how do you end up fixing it? If you're fixing it late in the game, you're throwing extra layers at it, which costs more and more money.

MIKE: There's a lot of these things that are pointing back to having these conversations early. And coming back to reviews, reviews are a conversation. It's intended as a venue for communication. And we may enforce it and say you have to do that [chuckles], but only because we've seen that it's so valuable. You know, somebody's making some work of art, and they want to bring it to somebody. They're trying to open a dialogue. They're trying to have a conversation. And that's what code reviews can really provide us is that conversation so that people are thinking and interacting together.

If there's any antagonism or ego involved in there, it's very destructive to that process. It's just an opportunity to collaborate. And when embraced as such, I think it can provide a lot of value. But also, it opens up to this bigger idea that communication and bringing communication cycles earlier in the process is vital to quality software.

There's another thing I've been thinking a lot about lately. Some years back, I don't even know how many years back, 20 probably, there was this document, the Agile Manifesto that was written. Some developers got together and said, "You know, some things in process tend to be bad, and some things tend to be good." And they wrote a document that said, "Well, this is better than this." And one of the better things, the things that's most important, is having those feedback loops where you have a quick cycle of information. You build something; you very quickly get it to somebody who can look at it and give you feedback. And then you make a little change then you quickly bring it back.

There's an alternative way, which is you spend, like, a year writing documentation, and then you hand it to somebody and say, "Okay, go build it," [chuckles] And that doesn't tend to work very well. Maybe writing policies or [laughs] [inaudible 26:14] with documentation is hard. It doesn't fit very well. It's the way humans work. We work a lot better when we have something more tangible. Something we can look at, something we can talk about.

And encouraging that communication cycle early is really what code reviews are an extension of. It's an opportunity to have built-in feedback into the system so that you can get some feedback there. But it's not the only one, and it's important to not lean on that one piece. Sometimes, people hear, "Oh yeah, I want to be agile." And they think, I got to go get a book that says how somebody else did it and set up a process that follows that, you know, the diagrams they've got in the book.

That misses the whole point of what the document was about. Maybe not all of the point, but a lot of it, which is that some aspects of process are better than others. And central to the process, a good process, is recognition of human frailties and human strengths. And those work best under a tight feedback loop with a lot of communication.

JUSTIN: And so, something that comes up often is, like, I've got a new thing, and it's hard to break down, and there's a lot of features on the new thing. How do I do incremental releases on the new thing and make it smaller? But I don't want to change the old thing until the new thing is all done or the new thing is ready to be released.

MIKE: I get it. I've got some answers I can throw at that [laughs].

JUSTIN: Sure.

MIKE: First of all, to your point about the old thing, I don't want to get rid of the old thing until the new thing is all ready. There's the temptation to think, okay, that means I need to go and build the whole thing, and then I'll have it tested. That's a really bad idea [laughs]. That's a really bad idea. To use a famous piece of hardware like the iPhone, I've never worked at Apple, but I extremely strongly doubt that they put together a completed iPhone, handed it to Steve Jobs, and said, "Okay, this is ready. Go present on it."

What they would have done is they would have had different departments building different components within that device, collaborating together, you know, figuring out the interfaces between them, and iteratively building that thing. Even though it wasn't released, they were internally testing. And, again, I'm guessing, but just knowing how software works, it's the only way it's possible. They would have been internally testing it all along in pieces and in gradually larger chains of components until the whole thing was put together.

So, by the time the whole thing was put together, it had been heavily tested in end-to-end tests all along. Likewise, when we're building new things in software, you know, it's just one example. You know, we think it's physical, so it's a little more tangible. We should never think, well, this isn't going to be released, so I'm not going to bother testing it or getting it reviewed until it's all done. And this is the whole point of that communication loop.

There's no reason you can't have a communication loop with your stakeholders, you know, people who've asked for it, whether that be product people in your company and, hopefully, even some beta testers out in the wild, some users who could be working with you and testing it. So, even though it's not going up to general release, you follow exactly the same process as you would otherwise. And then, when you're ready, you flip a flag and maybe move people over. At that point, you've gone through the process, just that we've talked about. You just happened to have done it with a smaller group.

JUSTIN: And you can achieve that smaller group in a variety of ways, like keep it internal. Keep it behind a feature flag. You know, have alpha testers, have beta testers. All of these things can help you. You don't have to have general release, but you still gain the advantages of iterative releasing.

MIKE: There's a longstanding practice in a lot of companies where they have feature releases, companies that are not yet doing continuous delivery, and sometimes that's imposed on you. If you've got, like, a mobile app, if it's going out through app stores, continuous delivery is hard [laughter] because you've got to get every change approved. So, there's an external constraint. The batch size is forced to be larger. And there could be a temptation in that environment to say, "Okay, well, we'll build everything for the next release, and then we'll get it tested."

But best practice involves having a feature branch and be testing that. Treat that like your finished product and get your testing, get your feedback on that throughout the process so that when you finally get to that endpoint and ready for release, it's just a tag [laughs] almost, right? You're at a point where you know that it's already good.

EDDY: We've talked a lot about, like, why do code reviews, but I'm still kind of curious on how you would do a code review. And one thing that we actively practice, at least in the team that I'm on, is keeping the code file changes concise and below a minimum file change. And I think we've determined that that's easier to be more productive and find bugs if the code changes are small. I like that idea. I've seen other teams that do larger file changes, and, to me, it's sort of like, how the heck are you able to give a good review when you have 40 or 50 file changes in a single [inaudible 31:15], right?

MIKE: I would suggest you probably can't.

JUSTIN: Sometimes, you just have to hit approve. Is that what you're saying?

MIKE: At some point, you're going to have to press that button [laughter]. The clock is ticking. That's why you don't put yourself in that situation in the first place.

I think, you know, it's something Justin is saying. Preparing for reviewers, one thing you can do is not put a huge change; give them a huge change. Well, one thing I like to do is I like to put comments, not necessarily in the code, but on the pull review itself, you know, so kind of meta information. I'll go and I'll leave comments on my own code, you know, on my own pull request, and say, "Well, this is why I did this here. Maybe give this area some scrutiny. This is where some, you know, sensitive changes are." Or "This test was added," you know, whatever it is. I feel like that's really valuable, in my opinion.

JUSTIN: Should those be in the code review, or should those be actual comments in the code?

MIKE: I phrased that specifically because I think, I mean, arguably, they could be comments in the code, but some things I don't think are necessarily appropriate as comments in the code, that is, they don't make sense in the code. If I came back a year later, that comment wouldn't make any sense to me because it's talking about something that's not applicable in that context.

But in the context of that change, it's very important, and that's where I think the value in having that kind of meta commenting, where, you know, you're commenting on the review, you know, on the change, the request, rather than on the code itself means that there's no artifact that's going into the permanent code from your changes. But there is, you know, that artifact that somebody can look at as they're reviewing to get some additional context. That's one thing that I think is helpful.

EDDY: Typically, when we add comments to code, it's because it's supposed to make it easier to understand. If you're adding comments to make it easier, does that imply that the logic for that code that you're making is difficult? And if that's the case, maybe we shouldn't be commenting, but rather, we should be refactoring it or breaking it down.

MIKE: I would agree with that. I don't know that I'd go as far as to say that comments are a smell, but I maybe come close. That code that requires comments is complex. Anything complex is hard to work with.

KYLE: I actually just started doing that. I started putting my own comments on my PRs, like, mostly on stuff that I'm not 100% sure of, like, stylistic and, like, team convention on how we like to do that. Like, I was just writing a validation for just a model the other day. And there was two options I could have gone with, a custom validator, or they have, like, built-in syntax you can do just, like, right at the top, whatever. And I was like, which would we prefer? What's the team convention on this? And I thought it was helpful. And also, explaining my thought process on how I thought this should be done. Is there a better way? That kind of stuff.

MIKE: And did you get some feedback on it?

KYLE: I did. Yeah. Very helpful.

MIKE: Exactly. You opened up a dialogue. You held up that olive branch and said, "Please." [laughs]

JUSTIN: So, one of the things that I've found really helpful is you are your own first code reviewer. And I think you were kind of alluding to that when you're writing comments in your own code review. When you create your code review, you need to go through and review it yourself. Look at all the file changes and everything. And I found that for my code, I catch all sorts of stuff [laughs].

So, it turns out that that code review is really helpful for me to get it ready before I send it out to other people. So, don't be afraid to create a code review and label it as draft or something like that. And then, you'll be making more changes to it, but it really is helpful to see it in that code review mindset. It's almost as if you're wearing a different hat. You know, getting that different perspective just from yourself is really helpful.

KYLE: Yeah, like, I treat it like almost as if you're, like, proofreading an essay. Like, when you're first writing it, it's like, it might work. It might still be correct. But is there a better way to go? You know, you read back through your sentence, and you're like, oh, I'm going to put this verb at the end of the sentence to, you know, add additional emphasis to it, or anything like that. I think it's a good mindset to stay in.

EDDY: There have been times where it's been easier to express my concern with a pull request by just hopping on the call with that developer and be like, "Hey, instead of going back and forth, you know, on this pull request [laughs] that can span 20 or 30 threads long, you know, it's easier to hop on a call and elaborate what those changes are [inaudible 35:54].

The problem with that, and I don't know if you all agree, is if I have that question, probably someone else who stumbles upon the code will have the same question. So, at what point do you move the conversation audibly versus having it traceable, you know, in a pull request so that it becomes easier for someone else? Because, at some point, it becomes documented.

MIKE: You know, I will do the same thing we've been talking about, and I'll leave a comment on the pull request and say, "Okay, we had this conversation." Or "Can we have this conversation?" So, another reviewer looks in and sees, oh, there's another conversation here that I don't know about. I can go ask about that. And I like to call that out for that very reason. If they don't know about it, then they can't take advantage of it.

JUSTIN: Having multiple different ways of communication that's just something that we're used to. Personally, like, I'll leave small comments on PRs when I know that people are working on other things. And they'll get around to it, and then they could just leave some insight. Obviously like, if it's a very long, heavy, big concept, then taking it onto a call or, like, a Slack chat, whatever, definitely would make it easier, for sure. So, I think it depends on the situation.

MIKE: The idea we've been talking about of communication being vital suggests that the channel is not as important as the conversation. But making people aware...it sounds like part of your question was saying, well, there's no artifact to this conversation, so nobody knows it happened. That signaling, putting some sort of signal, that's more communication. You're saying a conversation happened, and things we can do to signal to others that something happened, I think, are really valuable. It's being proactive, maybe not quite noisy, but providing lots of hooks for people to grab onto to know what's going on. It enables that communication to happen more effectively.

JUSTIN: So, that kind of leads into a question of what kind of comments should you be putting in there? There's different levels of comments. There's, like, a nit level of comment. Like, this is a minor thing, you know, consider putting this on one line or do this on, you know, maybe use a slightly different function like a math function or something like that. You have optional stuff. Clearly label it optional if something is optional. And then there's, like, the ones that are like, "Hey, I don't think this will work. Let's have a conversation offline," that sort of thing. What are the different levels that you guys put on there?

MIKE: I think you left out another one that's actually as important, if not more important. Like, "Hey, this is great."

JUSTIN: Yes. Yes, I did [laughter].

MIKE: That sets an atmosphere. It sets a tone. You know, going back to music, you know, I like that riff. What if we change the rhythm? It lets people know that you understand the algorithms. You understand the big picture, and you really like what's going on. So, they have that framework, you know, that scaffolding to hold on to. It doesn't sound like I'm coming here to say, "Your code sucks." That's a miserable experience for everybody. So, I would add that to your list, although I think your list is valuable.

You know, there's different degrees, and I think that it'd be nice if there was, like, color coding or something so you could say. But, you know, there's verbal cues you can give like, "Oh, this is not a big deal. This is a style thing. This is important. Please let's talk about it. But also, you know, I like what you did here. This line is beautiful. I really like how concisely you wrote this."

I think those comments...and you might think, well, I don't need to put those. That doesn't change the code. It isn't going to change anything. Well, it is changing something [chuckles]. It's changing the perspective of the person who wrote the code so they understand where you're coming from.

EDDY: I don't even shy away from leaving a comment being like, "Can you explain what this does? This is pretty ambiguous. Can you elaborate what your intention is?" Because, at the end of the day, the one that's stamping the approval is me. So, like, if I feel unconfident or unsure about that change, I don't shy away from that. And I typically tend to be, like, the straggler.

JUSTIN: So, what do you think about, you know, suppose you do have, like, a comment, like, you don't think this will work. Have you considered, like, do you add additional stuff, like, have you considered X, or have you considered Y?

MIKE: And code examples are really nice in that context, where X has some actual code written out, you know, maybe it's pseudocode, but, you know, it lays out an idea. It's a whole lot easier to talk about something that is concrete.

EDDY: Okay, can I just preface and go on record and saying, hey, I really love it when someone says, "Hey, I think it would be better this way," and they provide a code block. And like, "This is how I would think this reads better." Oh my God, like, how helpful.

MIKE: [laughs]

EDDY: Because then not only do you understand what they're trying to get the point across, but you can see an example of what they meant. I love that. And it takes, what, like, an extra two minutes or so to pseudocode, like Mike was saying.

JUSTIN: I kind of want to go back to the format of code reviews. I mean, the ideal situation is that you're a developer, and you want to go sit down with somebody else, and you are sitting there with them. And they review your code face-to-face. In the ideal world, that'd be awesome, and you can have a conversation face to face. But we can't often do that because of, you know, time zones or, you know, we all have different things going on that we're working on.

And so, code reviews are there to make that experience as close as possible to that experience. And whatever you can do as a code reviewer to make that experience happen, I think, is a good thing. And so, being very clear and being verbose, maybe not too verbose, but being very cognizant of how the other person looks and feels, and what their situation is, and things like that. All that comes into play while you're doing this code review, and they're not sitting there right next to you.

EDDY: What's everyone's opinion on paragraphs being left as code reviews throughout the whole code?

JUSTIN: Wait, a whole paragraph? Like -- [laughs]

EDDY: It happens. Like, what's your opinion? Because you have, like, ten file changes, and, like, on each file, you have, like, a paragraph that's being written on, like, the gravity of that change [inaudible 42:06]. It doesn't happen often, but it does happen. Like, at that point, do you find that disruptive, or do you find that a bit overwhelming? Like, I'm kind of curious.

JUSTIN: I don't know. I've never really had that much [laughs]. My initial thought would be like, oh, that's a lot. But I don't know, have other people had that experience?

MIKE: I'm thinking of one reviewer who tends to leave long comments like that.

EDDY: [laughs]

MIKE: And my default answer is going to be, wow, you cared enough to give me all this information. Thank you. It might take a little bit of time, but that means that they respected this interaction enough to give it that much attention. Somebody doesn't do that unless they care.

KYLE: At times, I'll run into situations where I want to leave a paragraph.

MIKE: [laughs]

KYLE: And in those boats, I don't want to make the individual feel uncomfortable because I feel like when you do that, it's a bit of a, like, hey, change your world, change everything you're doing type of feeling when somebody comes at you like that. So, I prefer to take those offline and, like, do a direct message with them, kind of work through it a bit, and then come back with a smaller comment. That's my approach. I have had people leave those larger paragraphs, and I just know I personally don't feel great about them most of the time. So, yeah, that's what I've done.

MIKE: We've been talking a lot about keeping your audience in mind. It matters. Yeah, I think having a mental model of who you're working with and being respectful of them can make the process go a lot smoother.

KYLE: I do feel like the quicker channels like Slack or Teams or whatever you do, something with a communication turnaround, it's a little bit easier to avoid miscommunications as well than back-and-forth communication on a PR. So, that's what I mean when taking it offline, too.

MIKE: It's a lot easier to have a conversation, a verbal conversation, than to have somebody write ten paragraphs [laughter].

EDDY: And maybe you can take it a step further and do it in person.

JUSTIN: Yeah. And actually, one comment and then one kind of moving to the next level. Yeah, like I said earlier, it's always, like, whatever you can do to convey your intents in a clear manner. Sometimes that is just a single line on the comments, or sometimes it is, like, you having to sit down because it might take you a little while longer to get your intent across. And then, like you said, Mike, it's really important that you understand your audience.

And then, the next question I have, if we were to take this kind of where we're leading, is how do we handle pushback in code reviews? Suppose somebody comes in and says, "Hey, you need to do it this way [laughs]. This is the way." So, how do you handle that sort of situation?

EDDY: I've gotten pushback so far as people will request code changes without even, like, approaching me directly regarding their question. [inaudible 45:08] just sits there [chuckles]. You're just like, ah, like, I could have explained it had you just messaged me directly versus, like, abrupt request code changes. But I'm typically pretty open.

I was told pretty early on, actually, by a lot of people who mentored me saying, "Hey, don't be married to your code." And I live by that. I really do. I'm super early in my career, so if someone suggests something, I'm like, it's for a reason. Like, they've probably been through the same pain point, and they're trying to save you the heartache, you know, having to deal with the same thing they did. So, typically, I'm pretty open, and I'm pretty flexible. Very seldomly do I push back and be like, "But this works, and it does the same thing." Very rare. However, nine times out of 10, I typically say, "Oh yeah, that does look better. That does read nicer."

MIKE: Sometimes there are team conventions that you can point to, and if, as a reviewer, I say, "Well, this is our team convention," and there's justification there. You can point to an example and say, "Well, this is why we should do it." That sounds different than somebody coming and saying, "Well, no, I hate your code. You should do it my way." One of them is conscientious. It refers back to the standards that you've developed as a team. It ties back into the consensus. And the other one is very egocentric. Like, well, this is just all about me and my perspective.

And I think that both as a reviewer and as a reviewee, keeping those ideas in mind matters. The code is a group product. And if we can build the team, then we should try to do so. And it's not really about us personally, and if it is, then you're ejecting yourself, right? Then that is actually not valuable for the company probably. You've brought something that's not valuable to the company, and you're just bringing your own baggage in. You know, and we all have our own things that we're working through, but trying to avoid that where possible, both as a reviewer and a reviewee, I think, smooths out the process, not just in code reviews but in life [laughs].

EDDY: Do you ever find yourself approving code before it's ready, trusting the person that they're going to go in and actually update what you recommended? Let me take that a step further. Let's say you say, "Oh, this looks good. I recommend these changes, but are non-blockers." Like, do you find yourself approving PRs like that, or do you sit back and wait for them to make a change?

JUSTIN: You just said that it was a non-blocker, so why don't you approve it?

EDDY: All right. Because, like, I've seen, like, feedback where it's just like, oh yeah, like, I'd recommend doing it this way, but I get it, non-blocking, blah, approved. Like, do we feel good when it comes to doing approvals like that, or do we prefer to [inaudible 47:47] upon that and only approve it only after you feel really good about the change?

MIKE: There's some people who I know if I left that comment they're going to go back and make the change. I made a comment like that on a review probably a couple of months ago. You know, the coder went back and said, "Oh, you know, that was a really good idea. [laughs] That could have really simplified this." They went back and made the change. And I saw that in my email a couple of days...I was like, oh. [laughs] I didn't get a chance to get back to it, and they totally made the change. And I know that that, you know, that person would do it.

If I'm working with somebody who is just getting started and they're kind of overwhelmed [laughs] and they don't know what to do, then I'm probably not going to approve it, not because I don't approve of them as a person or, you know, of their ability generally but just because you've improved the process to give them a chance to make those changes. It depends some on the audience.

KYLE: It made me kind of think whenever I'm doing a review, I try to keep the audience in mind in the sense that do I know this individual? And if I don't know this individual, what is their background? There will be times when I'm either recommending a change, you know, and how much detail I will go into sometimes I will take into consideration their level in so much that if I'm not aware, I've gone into our directory and seeing, like, what level of the chain they are, you know, like, are they a software 1, 2, 3 type thing, you know.

And if it's a more junior individual, I will take the time to elaborate more or reach out to them. If they're more of a senior individual, I will take more of those, I guess, shorter routes where it's just kind of a quick explanation, kind of expecting that they'll take it and do what they will with it. That's just me, but I tend to do that route.

MIKE: You know, and this recurring theme of knowing your audience maybe is a good point to kind of conclude this session. We started talking about music and art and getting feedback. They say when you're making YouTube videos, don't read the comments [laughs] because we know the internet is mean. There are people that you actually care about and trust that you want their feedback. And code review is an opportunity to get that, you know, it's an opportunity to collaborate with people that you want to collaborate with generally. You can get advice on how to make a better product.

It's part of a broader culture of communication. In isolation, it's probably not going to help you very much [laughs]. You can't stick that security on at the end. But as part of a broader culture of tight communication loops, it can expand the communication in a really healthy, productive way and result in better code.

Thank you, everybody, for participating today. Any final words?

JUSTIN: Let me put my security hat on. Check your security issues while you do a code review. All right, now I'll take it off.

[laughter]

MIKE: Thank you. [inaudible 50:36] for us next time on the Acima Development Podcast.

The podcast begins with Mike sharing his fondness for smoothies and drawing an analogy to Git, a distributed database. He compares the irreversible process of making a smoothie to the irreversible nature of operations in Git. Mike explains that Git, emerging from the same project that gave us Linux, is a distributed database that allows information sharing and verification across geographic regions. He highlights Git's significance in facilitating collaborative development in large-scale projects like Linux.

The discussion then shifts to the technical aspects of Git, such as dealing with conflicts, the concept of a distributed repository, and the uniqueness of Git in managing changes and time. Mike elaborates on the cryptographic operation called hashing used in Git, emphasizing that changes in Git represent patches or differences rather than versions of code. The panel discusses Git's ability to handle vast numbers of contributors and the significance of timestamps and commit hashes in tracking changes.

Finally, the conversation delves into the practicalities and best practices of using Git. The team discusses branching strategies, the importance of not rebasing shared branches, and the impact of different deployment strategies on Git usage. They also touch on the challenges of merge conflicts and the importance of understanding Git's limitations and strengths. The episode concludes with an acknowledgment of Git's ubiquity in various fields beyond software development and encourages listeners to familiarize themselves with Git for collaborative projects.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development podcast. I'm Mike. I'm going to be hosting today. With us, we've got Eddy, JP, and Kyle. Great to have you all here today.

And, as usual, I'd like to start with a bit of a story. And I'm actually going to talk about my breakfast this morning. I have become a great lover of smoothies [laughs]. Years ago...I enjoy them. I've always liked a smoothie, but they never really were enough. I have a lot of appetite. I was a huge eater as a teen. I was late getting my height, so I grew a foot in a year [laughs]. That left me skinny and hungry [laughs], and I also ran long distance. So, huge appetite in my teens. You know, I still don't have the same appetite as I used to have, but, you know, I still like to eat enough. And something like a smoothie for breakfast was never enough.

Well, I kind of by accident stumbled on a combination that scratches the itch [laughs]. I put a lot of seeds in it, and that gives the fat and protein it needs to keep things good, and then lots of whole fruit. I don't even throw juice in anymore. So, I just use water to add enough liquid. That means that I am getting all kinds of fiber, and I get the fat and protein from the seeds. It means that I stay really stable with my blood sugar. It's fantastic. I love it.

So, I throw together all these ingredients in a blender, you know, I add the water and blend it. One thing that I have never tried to do is put the fruit back together afterward. And there's a reason for that, and it is that it's effectively impossible. I mean, technically, you could pull apart all the molecules, right? And reconstitute them. But there is no way in a meaningful amount of human time. And I think it would all have rotted by that point anyway, right [laughs]?

There are some things that only go in one direction, and a smoothie is one of those [chuckles]. You press that button to get it going, and that's the end of that fruit. It still tastes good [chuckles]. It's still valuable. It's still valuable. But it only goes in one direction. That actually applies to several aspects of what we're going to be talking about today regarding Git.

Git is a distributed database, and if you're not familiar with, you're thinking, well, why would I need a distributed database? It's like listening to your friend, who is a huge Bitcoin enthusiast. We all need distributed databases. Unless you're an enthusiast, probably your eyes glaze over [laughs], and you forget about it until the next time your friend comes over. But it's a similar sort of thing. Most of these cryptocurrencies are built as a distributed database so that you can have information shared across geographic regions and be able to verify that data is not corrupted. That's what Git is. There's a few reasons that it is what it is.

Now, Git was spawned by the same project that gave us Linux, the operating system that powers, like, most of the internet [chuckles]. If you've got an Android phone, it's running Linux. If you've got some cars, I think Tesla runs Linux, you know, Linux runs kind of all the things—most of the servers, not all of them. You have some Windows servers out there, some old Unix servers, but a lot of the [inaudible 03:19] are on Linux.

But a long time ago, it was just some guy, Linus Torvalds, who wrote a project where he had an operating system in college, and he released it open source, released his code to the internet. And people actually started using it, and they started contributing to it. And pretty soon, it was this worldwide project where you have thousands, potentially, of people who are all contributing to this project. And sometimes they're going to make changes in the same files. And there were systems for managing that sort of thing in the past, but they all had a few problems.

First, dealing with conflicts was a pain [laughs]. And there's a few ways to deal with that, but you need to recognize that conflicts are a pain. And they are especially a pain, and to go back to my smoothie, they are especially a pain if you have to have some sort of shared state. If you say this is the perfect state of the repository, this is how my code is and that everybody has to work with that version of the code, well, if you have 1,000 people working on the same thing at once, it's pretty much impossible to actually have a shared vision of what that thing is. So, that is a serious problem.

If you have a whole bunch of people trying to make the smoothie at once, it's even worse than it happening in one place. Well, Linus Torvalds had an idea to change from the way things were currently working, where he said, you know what? We're not going to have a central version. There is no such thing as a, you know, single, canonical version of what our code is. You can all code separately, and it's fine. And we're just going to let time move forward. And what the pure, true version is, is just whoever the person in charge says it is, who has their copy of the code.

And here's the other place where the smoothie comes in. This idea of time only moving forward matters. There's an operation, a cryptographic operation, called a hash where you take some data, and you can do some mangling on it, and you get a number out the other end. And you can't go back the other way. There's a few different ways of doing it. There's a popular algorithm, the SHA set of algorithms, that is used to do this hashing. And it's really all it says, you know, hash means to scramble something up. It's basically what I'm doing with my smoothie. You can do it with vegetables [chuckles]. Once you've shredded something up, you can't go back the other direction.

So, this is the other key thing that Git does is that anytime somebody makes a change, it makes a hash of that change. That is, it runs this mathematical operation, takes all your changes, and comes out with just a number on the other end, a long number. And that number is pretty much going to be unique. The chances of it being the same as somebody else's numbers that they hashed is essentially zero.

So, you can have people all over the world making their changes and then creating these hashes, and the hashes are all going to be different. Together with that hash, you have a timestamp. And each of those hashes is associated with a change, and the change is not a version of the code. And here's another key thing to note: the changes represent a change, that is, a difference. They represent a patch or a this is what we're going to do to change the code.

So, what you end up with is this database distributed across any machines. You can have thousands of people. They could be working for months separately, come back, and they have this list, you know, this set of changes. Each of these changes has a commit hash associated with a unique number you can identify with and a timestamp. And each of these also represents a set of changes that's going to be applied. As long as you can apply those changes, it really doesn't matter if you don't have a central version, just [inaudible 06:47] the changes.

And you can imagine sometimes it doesn't work. And that's what we're going to get to a little bit later today is dealing with some of those times when it doesn't work. But the fact that the time only moves forward, that is, you always have to apply changes; you're not going back to some central vision of what the code is. It means that everybody can work separately, and it's just fine.

What you decide to deploy here with Linux is whoever is in charge of the project says, "Hey, this is the version that we're going to go with." And you think, well, how do you know? Well, you know, you probably have a human problem there [chuckles]. That is, we have human structures to deal with governance, so use those human structures. Let the people decide and designate somebody who is going to say, "Okay, this is the version that we're going to go with." And if you have that person, she or he can say, "You know what? This is good. I give it my stamp. We can deploy this."

And then you can have people working separately, and they can contribute to it on their own time by removing that need for a central figure and by giving things, you know, these unique handles so that, you know, everybody can verify that the change is unique. And by allowing time to only go forward—you can only make changes—then you can solve a lot of the problems that we had because there were systems before. Did any of you use some of the systems we used before Git, and how did that go for you?

EDDY: Well, honestly, for myself, I'm pretty new [laughs] to development. So, the only version control that I know is Git. I'm no stranger to previous systems, you know, that were used in the past. Like, out of curiosity, I did look up on, like, what Git was, and I looked at the timeline and what other features they were. I read about, like, RCS and SVN, et cetera, which pale in comparison to what Git is.

MIKE: Kyle, I think you mentioned you used CVS back in the day.

KYLE: Yeah, I was quite junior. I don't remember a lot about it. The one that I would know the most about, which was just that it was always down, was, like, our SVN when I was at a C# shop specifically. It was great when it was working, but it seemed to stop the ship anytime there was a major error. It seemed to stop the ship pretty dead center when there was a conflict, which was always just a hassle to get through.

MIKE: And you hit on something. It was really hard back in the day to try to manage this using the tools that were there. But those tools were a big step forward from what you had before because if you don't have some sort of tool for [laughs] managing, then you just got your code on your machine, and you email it to somebody, right? And then, they try to figure out the differences and hope they get it right. That's not a great way to do things. Really, that's what you had before we had some of these tools.

So, while it may sound like we're casting some shade on Subversion or CVS, that doesn't mean that those tools were not incredibly useful and a huge step forward from what we had before. But once you have to have a central, a central repository to maintain, then you have a single point of failure. And a single point of failure is not so fun to deal with.

JP: I guess also want to call out as well that Git is not the last iteration as well. A lot of people will talk about Git. We have all these beautiful tools that have emerged in the technology industry to work on top of Git. I mean, the foundation is still Git and a lot of these other tools like Subversion and CVS, but really, Git just laid a foundation, like you said, with distributing that data and using, you know, the mathematical computation of hashes to be able to empower this extremely large distributed network of code and make it publicly accessible, which was something that really wasn't there before.

EDDY: Wait, Jackson, you said that it's not the end all be all. So, are you saying that there's other version controls currently that are newer --

JP: Not other version controls, but systems leveraging Git. So, I know at Acima and at a lot of [laughs] organizations, the world depends on GitHub, for example. GitHub itself is a version control system that relies on Git but adds a ton of additional features on top of that system.

MIKE: Right. It's another layer on top of it. When you think about a database, releasing a database to the world is awesome, but not very many people are going to say, "Hey, I'm going to go hit PostgreSQL today." But they might say [laughs], "I'm going to go visit Facebook," right? [laughs] And that layer on top matters with GitHub, or GitLab, or Bitbucket, or, you know, there's any number of different versions out there. I'm not trying to say that one is the best. They add another layer on top, so you actually have an application that runs on top of that distributed database down beneath.

JP: The big call out kind of, Eddy, to respond to that, is, like, what does GitHub give that Git doesn't by itself? Right off the bat, the biggest feature that I guarantee that most people would agree on is pull requests or merge requests, the concept of that. That is not something that comes with Git by itself.

EDDY: But that is available with other version controls, right? So, I'm sure GitLab does something similar to that.

JP: Well, absolutely. GitHub, GitLab all of those more just systems leveraging Git provide that functionality. But Git, by itself, if you're just hosting your own Git server, it doesn't come with that functionality.

MIKE: And Git is the database.

JP: Yes.

MIKE: Git is the database, and the database is not going to give you all the social interaction tools that these applications provide, this idea of a pull request or a merge request with reviewers, what all that you have in there, your descriptions, all of your workflows.

JP: Status checks, yep.

MIKE: Yeah, that isn't provided by the database. The database stores the changes to the code, but it doesn't manage any of that social interaction, and that's by design. You know, it's a tool that's designed to be the database, not to manage that social interaction. That's an important thing to recognize.

EDDY: That's interesting. I never thought of Git as a database. But putting into perspective, I think it actually makes sense in hindsight.

JP: And I guess kind of to, like, layer on top of that, Mike, if you don't mind, we have Git as the database. But there are a lot of other...like, we talked about GitHub, GitLab that leverage that database. But the day-to-day, again, even when developers are working with Git on their systems, they're not working with the database directly. They work with the Git CLI. And there's other alternatives to that as well. So, a lot of people think that Git CLI is...that's it. That's the thing doing my version control. No, it's that combined with the database technology underneath that really gives you that full functionality of Git.

MIKE: That's a really good point. We interact a lot with that command line interface, right? You know, all of the command line tool that Git provides for us or maybe use a graphical tool to interact with that.

JP: Yeah [laughs]. Yes.

MIKE: And a lot of people do. But that itself isn't the database [chuckles]. Having those tools for writing the database are important. You [chuckles] don't want to corrupt your database, and then you've got an even bigger problem [laughs]. It's probably been done, but I strongly doubt that very many people have ever written their own data to that database by hand. They use an automated tool to do it.

EDDY: I actually have a question. And, again, seven months full-time development, so I'm still pretty new to the concept. Git, since it's a database, does it store all the repositories for a vast majority of the companies that use --

JP: Oh, good question, though. Good question.

MIKE: Let me answer that question with a question. Do the databases you use store data from Facebook, and why or why not? So, for your application, say you're using PostgreSQL, do you and your PostgreSQL database store data from another company?

EDDY: Probably not.

MIKE: Probably not. But they may also be using PostgreSQL, right?

EDDY: Right.

MIKE: So, PostgreSQL is a technology that can be used to create a repository for storing data, but it isn't itself of that repository. It's a technology for creating a repository. So, there are, you know, millions of entities out there that have a repository, but that doesn't mean they all have the same one in Git or in any other database. There are many repositories out there, many of which have nothing to do with each other at all.

And thinking about this being distributed, we can all have separate repositories, and that's fine. But many of us have repositories that we connect to each other so that we can have some sense of shared direction and code and data, maybe, I should say. Because it doesn't necessarily have to be code. I've used Git for things other than pure code.

JP: [laughs] And I guess to kind of follow up on that as well to kind of answer it, just using GitHub as an example, and I guess most major version control hosted solutions, each project or set of projects that you're working on usually gets written in its own Git database, its own...what they call a Git repository. That is the terminology for it. So, a lot of organizations, or bigger companies, or even just large projects will have many Git repositories that are all tracked in separate Git repository databases, not in the same one together.

EDDY: So, where does the code get saved then when you push up? --

JP: Oooh, another good question. In that case and, Mike, please, yeah, please bounce off me. Using a provider like GitHub, they're actually hosting a Git server. That's another piece of the Git technology stack, Git repository server that will receive the requests of the commit hashes that you're changing and the associated data, and they will store it on their side.

MIKE: Remember when I said earlier that you designate somebody to be the person to say, "Hey, my repository is the one that matters?" We've elected to let GitHub...I say we, you know, anybody who chooses to use GitHub generally elects to use GitHub as that gatekeeper. And we've chosen their copy of the repository to be the one that's in charge, but everybody else has a full copy. Anybody who works with that data has the full set of data.

GitHub is not special or GitLab, you know, you name it. We'll use GitHub today. They're not special. They could go down, and the world would be okay [chuckles]. You wouldn't because [laughter] [inaudible 16:48]. But we wouldn't lose everything because everybody else who's ever used that code would also have all of that data. You're pushing data. You're just syncing your data to that other repository, but you still have everything else. You're just pushing up a few little changes. Really, you're just pushing up those few commits, those few little chunks of changes with their hashes, and not everything. You're not pushing up the whole repository. You're just pushing up the changes that you made to your own branch.

We didn't mention branches before, but it does allow you to have...it has this idea of a branch in Git. So, you can say, "Oh, you know what? I'm going to make a copy over here." And it's really lightweight. You're not actually copying over all the code because the beauty of this distributed database is you're just making a set of changes. So, naming a set of changes is no big deal. It's just, what, one name [chuckles]? The overhead for that is extremely small.

So, it's really easy to create a new set of changes [inaudible 17:43] go off in your own direction, and you can throw it away. Or you can bring it back here and say, "Hey, you know, I'm going to bring these changes in," which actually brings us to a point that we said we'd talk about is what does it mean to bring changes in? What does it mean to merge branches? And there's a couple of different schools of thought on how to do this.

Let's say you've got a branch of code. It's just the history, right? It's just a series of changes. You can call them patches, commits. Here's the change that's going to be applied. Here's the change that's going to be applied next. Here's the change that's going to be applied next, from zero up to infinity, right? It's just an endless set of changes that are applied. And if you have two people who've been working separately for a while, you can merge those. And as long as you haven't been changing the same files, then it'll work, right? They'll be interleaved.

You'll have one person's changes, and that person made another change. Then the other person made a couple, and then the first person made a couple, and the other person made one, and the other person made one. You'll see it coming in from one person and the other person and one person and the other person. If you look at the history, you'll just see that history of when they made those changes. Each of those changes represents a change to what happened before.

And if you're thinking about reading the history, if you want to just know when they made those changes, well, that's a reasonable way to think about it. But I will say that that's not usually what you care about. What we usually think about is in terms of something more like a feature. And the way that people usually use Git is they have a branch where they build out a feature, and then they want that feature to come in. And if you wanted to go through the history and see how that feature was built, you'd have to go through and see all the little changes interleaved with everybody else's changes.

So, there's another school of thought that says when you're merging, you shouldn't actually do this [inaudible 19:18] what we call a merge, where you just throw all the changes in where they were made. Instead, you go through a process, first, of rebasing, and rebasing actually means more than one thing. It's a bigger tool, and, hopefully, we have time to talk a little bit more about it. Think about that word means base and rebasing means you're redoing the base [chuckles].

Since our changes are just a series of patches, I'll make this change, and this change, and this change. Usually, you're making those changes thinking about starting back whenever you branched off of the code originally. But you can change that. Git will allow you to do so and say, you know what? Instead, I'm going to branch off of where the code is now on my local version, you know, grab the changes or, you know, or even off a remote version.

I'm going to branch off of this specific commit. Remember, they're named. They're named with this big number, this hash. Say, "I'm going to branch off of this hash instead." Then you can do a couple of things. You could either squash all your changes down to one big change, and then you've only got one big change [inaudible 20:11] for your feature. Or you can just put them all in together, but they're all sequential. Then, if you go back through your history, you see a series of the sequential changes for one feature, and then a series of sequential changes for another feature, and then a series of sequential changes for another feature. And it makes it much easier to read.

So, here's the two schools of thought: the hey, just merge it in. It's the easier way. I say the easier way; it's the more simplistic way. It's kind of the default. "Hey, I'm just going to throw my stuff in there. I hope it works," way of doing things. And then there's the rebasing approach, which says, you know what? I'm going to try to be very polite and put my changes all in one nice, little tight bundle so that people can read it later. It does make the history a lot easier to read. But it also means you got to get everybody else on board. And it also means that you're rewriting the history because you're changing your local branch to be starting from somewhere else.

And if you have anybody else collaborating on that local branch, you're going to do them a world of hurt when you do that if they try to pull those changes down because it's going to say, "This doesn't match my local version. The history is rewritten." What happened in the past? I don't know anymore. The world makes no sense. And [chuckles] they're going to have to decide what the true history is. There's some downside to collaborating within a single branch when you follow that approach if you're ever going to make changes after you've done a rebase.

JP: I guess kind of to, like, to step backwards as well, like, simplify what all that really comes down to in Git terminology because these words get thrown around a lot, is you either have a merge commit, which that happens during the Git Merge process, and that's that default mode we talked about a second ago, just merging in the history. And your changes get globbed on top of each other, not maintaining the actual linear history of changes.

Or there's rewriting the history, which is actually moving your commit hashes around and potentially resolving conflicts, which is maybe something we can talk about as well, in order to create a new linear history that more matches the actual timestamp of when those changes happened, even if they were on separate branches and time.

There are definitely, like Mike mentioned, benefits and drawbacks to both approaches. And, you know, it can be a very opinionated thing. But it really depends on, do you care about how your code was put together? And do you want to see the changes in a clear, fashionable way that can be audited according to certain standards? So, hopefully, that's another good way to reference that.

EDDY: Let me ask you this, then, when you roll back, where do you reach out to? Do you use Git as a tool to roll back, or do you use the [inaudible 22:37] to roll back?

JP: That's a really complicated question because that really depends on what rollback means. Yes, that can be part of what they call the GitOps process, you know, doing events based on changes to a Git database. But that's going to depend on organization from organization [chuckles]. It's not necessarily related to Git. It can be, and a lot of organizations choose to do that. But yeah, that's a large discussion [laughs].

MIKE: Well, you hit on something really key that I tried to talk about with my smoothie [laughter] [inaudible 23:08]. Git doesn't have a rollback. It has no such concept, and that is by design. Time can only go forward. Once the smoothie is made, you can't put it back together. Git only has the idea of a revert, and a revert does not undo anything. And this is critical. This is, like, a really important thing to understand: A revert does not undo anything. It does the changes in reverse because time might have gone on, right?

Let's say that ten people have put new code into the repository that you've decided to centralize on after you made your changes. What would rolling back mean? Getting rid of all those changes? Getting rid of some of them? Doing some sort of rebase where you put other people's stuff on top of yours? It's ill-defined. It's not a well-defined problem. And Git has decided that there is no answer to that problem.

Instead, when you're undoing your changes in Git, it just applies your changes in reverse. And then, when you go to try to merge them in, they might not work. You might not be able to go backwards anymore because things have changed. And you're going to have to deal with the conflicts because, again, time only goes forward.

JP: This even occurs, let's say, if you wanted to delete history. The same concept happens in Git if it only goes forward. Yes, you removed, or you rebased, is really what the terminology is, your history of commits. There's a mark in time that a rebase occurred there, and it's by new commits. The new commits come in, and that is only going forward. You can never really return back to that location that you were at before.

EDDY: Wait, but if you're going back, right? And it does it in reverse, like, how does it have context of all the code that's been merged since?

MIKE: It doesn't. It doesn't at all. It has zero.

JP: It only has context of what files changed and what lines and where. So, it just tries to repeat the previous commit or two or how many you've chose in reverse.

MIKE: And it might conflict with those changes that come afterward. It lets that set of changes be no different than any other set of changes. It's just a set of changes going forward. That's a key distinction that makes Git special. It actually makes it easier to use. It's weird to think about because we like to think about, oh no, there's this pure, true version, and I need to go back to that version.

JP: [laughs]

MIKE: But no, there is no such thing in Git. There's only time moving forward.

JP: And it's also kind of on that same vein; a lot of people usually take a moment in time in Git and turn that into some type of artifact that's this code in this point of time, but it is not Git where you're making that decision. It's where you package it up and put it somewhere else.

KYLE: So, basically, you can...the idea of rollbacks, or whatever, could only be possible with an actual binary.

JP: A binary, package, artifact, whatever you call that, but that comes after exactly the Git process.

MIKE: But you can copy out a version, right?

JP: [laughs]

MIKE: And then later, you can go back to it. That's not Git doing so. That's a choice outside of your database, you know, outside of your version control.

EDDY: I feel like having an analogy of what Git is would really help. While we were talking, I looked up a more simplistic way to think about what Git is. The repository is like a storybook, and the pages in that book are the files. The writers are the developers. Changes and additions are the commits. Book's memory is the history. Going back in time is the version control that we were talking about. Working together is collaboration. And then sharing the book is the remote repository. Would we agree, in essence, that that's a fair way to do a comparison?

JP: Yes and no.

MIKE: Digital book? Yeah.

JP: [laughs]

KYLE: I was sitting there thinking with Eddy's comparison there; I'm thinking back to all of my college books, right? And it almost seemed like it does work pretty well there. The revisions of your college books they don't change a lot. But they might fix some wording, and they might fix a few quote, unquote, "bugs" here and there. But they don't revert back to anything. They have to release a new version, regardless of what they're doing.

MIKE: Yeah, that's true. You can only do revisions forward. Once it's out there, you can't get it back.

JP: Great point.

MIKE: One thing we haven't talked about...we've talked about this idea of rebasing. We've talked about merging. We haven't talked about what happens when it doesn't work.

JP: [laughs]

MIKE: And yes, that will happen to you today if you [laughs] are just starting using Git because it does happen. Although if you use a very responsible and different ways to go about this, you know, if you have a very careful process...and I would strongly advocate for continuous delivery. Thank you, DevOps. We've got some DevOps people here. Thank you so much for making our lives better [laughter].

If you make lots of small changes, you're less likely to have a problem. If you make a few huge changes, you're almost certainly going to conflict, and it's going to be painful. Sometimes, you're going to make a change in the same place somebody else did, and there's no way to get them in alignment automatically. That's what we call a merge conflict, and Git doesn't solve those for you. It instead gives you both versions side by side and lets you decide.

And sometimes, the answer is to pick one. Sometimes, the answer is you pick the other, and sometimes, the answer is you have to somehow make them work together [laughs] and combine the two. And this is inescapable because, yeah, you both change the same place. You're going to have to deal with that. And no, there's no magical system that can address that. So, instead, it just makes it easy. It says, here are your two versions. Go ahead.

EDDY: It's harder to resolve, or it's more common to run into when you have a larger team working on the same codebase.

MIKE: The number of people working in the same files is really the same thing. You could have two people working in the codebase working on the same file and have it happen a lot. It's the people working in the same part of the code that really gets you.

JP: I guess to call out that distinction as well, what is Git actually recording in its database? Now, without going too deep, it's usually recording line changes within a file, and that's why merge conflicts come up. It's not because multiple files change. It's going to be multiple lines changed by different people in the same file, and that's why we have to resolve that.

Take a JSON file, for example. A lot of people are familiar with that data structure. If I'm adding a new item in a list in a JSON file while you add a whole new object in that same location, you can't just merge those automatically. Those are two different data types that will break the JSON format. That won't be a valid source code file anymore to use within my code. That's why we check up on the lines that you're changing and why you have to make the decision and the distinction of which one makes sense. And sometimes, like Mike said, it's neither [laughs], and you have to fix it yourself.

EDDY: 100%. Well, and it's kind of cool that Git kind of just sits back and be like, you tell me what the right version is, and then I'll go [laughs] ahead and push it up. So, it gives the responsibility to the writer.

MIKE: And, interestingly, you're probably doing this in a branch. And you're probably using some of these tools on top of Git that will then let you do a review process. And so, other people can look at it and say, "No, that's not how it's supposed to go," or "Oh, yeah."

EDDY: So, I've heard that before, we used local databases as a way of version control. Like, is that true prior to, like, cloud-basing version control?

JP: I mean, and that's still what we use databases for [laughs], just not for source code, if you want to think about it that way. We've chosen to add an additional set of tools and technology in the Git database to manage source code and files. But we have customer data, potential [inaudible 30:47], mathematical computations, whatever you're storing in a database. It's still version there and sometimes has its own scheming.

EDDY: I got to imagine, right? Like, if you were storing that in a local database somewhere, what happens if that data gets corrupted, right?

JP: [laughs]

EDDY: So, it's like one point of failure, essentially.

JP: Absolutely.

KYLE: You work later.

EDDY: [laughs] You work later.

KYLE: You work a lot later.

JP: [laughs] And there's the whole concept as well of what Git is, right? I mean, Git is a simple database technology that runs off files on your system. You're storing something in a large database technology like we've been using Postgres as an example. You have to run a whole set of technology as well. So, let's call out Git as well. You don't have to run anything. You just have to use tools that apply the Git technology, Git database technology in order to work with it at runtime. You don't have to keep something running 24/7. That's another big benefit.

MIKE: It relies on the file system you already got, you know, and there are distributed databases that people use for customer data. Couchbase is one I've [inaudible 31:53], which is designed for cases where sometimes somebody's not going to be connected for a while. It might be good for a mobile app where, you know, somebody's going to be offline a lot [chuckles]. And they're going to be making some changes, and you want to merge those back in later. So, these kinds of ideas are not exclusive to Git and can be used for other kinds of data. But we're talking about Git, which is designed around files.

JP: I guess to also talk about–Eddy, because I really like what you mentioned; there are other stores, and one extremely popular, like, data store that isn't doing it for, like, source code but for in general for files as a whole is, like, S3 or similar blob storages. They can also version your files. They don't do it as a conglomerate like Git does, where everything that's changed in this specific directory structure is recorded. S3 can take specific files and manage their versions over time.

So, just like Mike said, there are other version control technology out there. You could store your code in S3 and version it that way. It's not going to probably be very beneficial. But some of those ideas that have come from Git and the previous VCs technologies are definitely used in a lot of other places.

KYLE: Slap Athena on top of it, and you're good to go.

[laughter]

MIKE: We've talked a lot about how Git works, what it does, this idea of merge conflicts. We haven't necessarily talked much about best practices.

EDDY: Don't force push if you're collaborating.

JP: [laughs]

MIKE: That goes back to the rebasing thing. You change history now the other person has got a divergent history. And what do they do? That's the gotcha for using a rebase approach. I actually am partial to the rebase approach, but that doesn't mean that it solves all problems. I like to have a Git history that's easy to follow [chuckles]. I think it's a wonderful thing. [chuckles] I mean, when you have to go back and try to figure out what happened in the codebase, it's great.

By the way, this is just database. You can look at the history. If you've got the Git command line, you can do Git log, and it will show you all of those commit hashes plus the description of the change all the way back through history, and you can search it. And you can pass the dash p for the patch flag, and it'll show you the differences. And you go back and just search, like, uh, this is the name of the function that I'm looking for. And you can go back and search through your history. It's just like any other file. You can go back and search through it, which is fantastic [chuckles]. It's a searchable database.

But, you know, so, I really like to be able to search for things, to be able to understand things. But yeah, if you ever are in the same branch as somebody else, if you change your history using this rebase, if you change the history and then push it up to a shared repository—you have this force push that allows you to do that—then anybody else who pulls that is going to have to deal with the consequences, which is now their local version is not in sync with the remote version. They have a different history. And they're going to have to figure out what to do.

They can throw out their local changes. Hopefully, they haven't done anything important, right? That rebase type flow really only works if you have individual contributors working in a branch, and then a shared branch you treat as something that you don't change, right? You don't ever rewrite the history on anything that's shared.

JP: You know, and you mentioned best practices. I'd also, like, include as, like, a subcategory and kind of really [inaudible 35:03] on that is also branching strategy as part of best practices. A lot of popular projects out there, especially ones that are, like, libraries for simple projects, use something called trunk-based development, which is usually a branching strategy of where you have a default branch typically named, like, master or main.

And your code, when you branch off that, and you return back to that original branch at some point in time, usually, through a pull request or a merge request against standards that you want to follow, when you're using Git, usually, you choose to go back to that branch. But there are many other strategies out there, and your team or project should help decide what branching strategy you should use.

A lot of mobile projects for a long time have used Git Flow as a branching strategy, which I won't even attempt to try to explain right now. But it is a whole nother branching strategy with a whole nother set of standards. So, you really have a lot of flexibility to be able to define these on your own, but each one comes with their own caveats.

If you're using trunk-based development where you're merging back to that master or a main branch, just as Mike mentioned, let's say I'm working directly on the main branch. You should never do this, but let's say that you are. Let's say I force-push a rebase that I had just done on the main branch up to main, and Eddy comes to put up a pull request with his changes that he had branched off main. Now he's going to have even more issues because the branch he's trying to go back to, to merge back into, he has diverged his branch's history from that branch he's going back into, which can be a nightmare.

What I'm trying to say, in a nutshell, is know what branching strategy you want and have certain rules for the certain branches that you're working with of what they are allowed and not allowed to have happen within their history.

MIKE: A simple rule, based on what you said, Eddy, is never rebase shared branches. It's pretty straightforward, but you have to come up with a way to do that. How are you going to do this? You can work on your own stuff, and you can rewrite your own history because it hasn't been shared with anybody else yet. But once you put it out into the world with other people, then you have to only go forward, right? It's past the time when it's responsible to go back and try to rewrite history and change things.

EDDY: I wanted to mention, just so I can get some more context, what's the branching strategy that we use in our team?

MIKE: We used to use something closer to Git Flow, but we have moved back to a trunk-based strategy because, again, thank you, DevOps [laughter]. We've got continuous delivery. And if you're deploying 20 times a day, you're making lots of little changes, lots of little low-risk changes. That doesn't mean no risk, right? You need to be responsible in your changes.

JP: [laughs]

MIKE: But if you're making many small, concise changes, they're unlikely to step on each other, and you can just go back into your default branch. One reason the mobile teams use something like Git Flow is that you maintain a parallel branch. You maintain a parallel branch for development and periodically come back to your default branch, to your primary branch. The reason for that is something like mobile; you can't deploy many times a day because you're working with an app store, you know, you're working with a third party that doesn't allow that kind of continuous deployment that you can get in a web application type of environment.

So, everybody has a different shared branch that you periodically merge into the one that ends up going to your app store, and that way, you still have your shared branch. You still have this idea of a shared branch similar to, you know, that main or master, but it's different. You have this development branch, and that's what you are treating as the shared special branch. And then the main one you only use for those builds.

JP: So, in a nutshell, you can say, when you're defining your standards and your branching strategy that you're using with Git, it's really influenced on how you're deploying something. Git doesn't do the deploying, but how you use Git and the database will be influenced from that deployment type. So, a mobile app, like Mike said, is your source code packaged up in some formation. But a web application or a web server that's something that's ingested and can be constantly updated on every single network call that your client is making. So, those just have very different deployment strategies.

If you even see, like, talking about, like, Slack or another tool that's built on Electron, that's even more complicated because it is a web application running as a desktop application. So, you have to do both types of deployment strategies. So, it's really cool. And you have to definitely pay attention to both sides of the coin. How do I want to develop, and how's it going out after I'm done?

MIKE: There's no one right answer.

JP: No, no [laughs].

EDDY: Yeah, the takeaway, for me, so far, has been Git is much more vast than I initially thought it was, which is really cool.

JP: Git's simplistic in nature of what it's trying to record. So, it makes it a powerful technology.

MIKE: It follows the long history of Unix tools...

JP: [laughs]

MIKE: That are really good at one very specific thing so that they can be chained and used together. Even the fact that it uses the filesystem as its storage means that it lives within an ecosystem. You can swap out the file system; no big deal, right? In fact, you would likely have different developers on different file systems, and it's all okay [chuckles]. That is not a problem at all. It's intended to do a good job within its own little niche and never step outside that. If you need to step outside that, use another tool.

KYLE: I mean, that's ignoring the carriage return, right?

JP: [laughs] Hey, they fixed that too, Kyle [laughs].

MIKE: If any of our listeners don't know, different operating systems have historically created files that used different line endings. And trying to collaborate across different operating systems where [chuckles] they have different endings of your lines means that some people are going to have a carriage return, some people are going to have a line break, some people are going to have both. And having to merge those is potentially painful. And they've had to improve that cross-operating system handling overtime, partially by using some heuristics, say, you know what? I think you're on a Mac. I think I'm going to go with, let's say, carriage returns only, [chuckles], you know, and they just make some guesses.

EDDY: If you were to make a new utility to replace Git, what would be your approach to making it more predominant?

JP: I mean, all jokes aside, my first thing would be, what am I trying to fix? Because that's how you find not only your target audience but how you build the right technology. What does Git not solve for you right now? And I guess people have actually asked that question. And there are other tools out there. That's why GitHub exists. That's why GitLab exists and adds those additional feature sets.

MIKE: There is a modern version control system called Mercurial that has a niche following, that I think has a lot of these same features. But it's partially proprietary, and people are cheap. And one thing people like about open source, not always, but often, you can get it for free, as in beer, as well as free as in speech. And they like that free stuff.

EDDY: Is Mercurial, the one you are mentioning, open source?

MIKE: My understanding is that it was not, at least times in the past. Is it today?

JP: I don't know. I'm going to find out.

MIKE: It looks like it is today [inaudible 42:12]

JP: I keep finding people saying Mercurial is free. It does not say open source but free.

KYLE: GPL V2.

JP: That would be open-source

EDDY: For the listeners, open source and free are completely different.

JP: [laughs]

EDDY: And it did take me a bit to [laughter] [inaudible 42:30] when I first started to find that distinction.

MIKE: According to Wikipedia, which could be wrong [laughs], it says that Facebook uses Mercurial, Mozilla, the W3C.

EDDY: What is Mercurial trying to solve here? Like, what's its selling point?

MIKE: Another thing on the Wikipedia page says that Bitbucket dropped Mercurial support back in 2020 because less than 1% of new projects use it. So, it's one of those things that it may be solving lots of problems, but I don't have enough familiarity because I don't use it.

JP: Eddy, I'll ask you the same question in a different way: of why do people still write in COBOL?

KYLE: Because of our banking software.

[laughter]

EDDY: Because if you have a repository that is written on COBOL and trying to [inaudible 43:18] would be a [inaudible 43:21].

JP: Yes. I would say not only is, like, there's the cheap factor of software, but there's also the familiarity in the habits of software.

EDDY: Well, familiarity isn't really always a good thing because with technology always changing every single day --

JP: Then we end up with COBOL.

[laughter]

EDDY: It took me a few months to really get familiarized with Git. And to have to adopt a new technology just because someone has an itch [laughs] is not always fun.

MIKE: You know, Git it is a tool that has met a lot of people's needs, and I think it became popular because it worked [chuckles]. That doesn't mean that there are not other tools that work and may even be a little better. But it does a really good job. And any differences are just not enough for people to care about. And momentum is hard to beat. If you've got a tool that does the job for almost everybody, then it's going to get the attention. It's going to get the fixes. Eventually, it's going to get the features.

EDDY: So, I'm assuming Git is something that's widely used predominantly in the software industry, right?

MIKE: I honestly can't think of the last time I talked to somebody who wasn't using Git [laughs].

EDDY: So yes, listeners, I would say if you're wanting to become a developer, I would think familiarizing yourself with Git [chuckles] is crucial.

MIKE: My 11-year-old was asking me about Git the other day. It's that ubiquitous [laughs]. He ran into it independently, looking into Minecraft modding, I think. But the first answer he got is, "The first thing you need to do you need to set up a Git repository." [chuckles] And like, "Oh, can you tell me how to get set up with GitHub? What's GitHub?" [laughs]

JP: And then I would also say, not even just as a developer, if you're even a data engineer and you're looking to leverage tooling and other things out there, it can be very useful to know at least some of the fundamentals of Git in order to get your way around those tools and see how they work. Documentation has also made its way, like you were kind of hinting at, Mike, into the Git world. So, learning about that if you're doing any kind of technical documentation is also a really good place to be.

KYLE: Even more, like, layman tasks. My wife ran into it a while back. She has her website blog that she uses, and she was looking up how to modify one of her themes, and she needed to get something off of a Git repo. You know, it's kind of trending towards a common tool that everybody's going to have to know about, you know, even if you're just a writer, or in social work, or something, right? You don't have to be an engineer. It's out there. There's some obligations that you're going to find in Git.

MIKE: And I wouldn't say just a writer or just a social worker.

KYLE: Well, those -- [laughter]

MIKE: We provide value in different ways [laughter]. That does not --

KYLE: Yes. Yes. You're right. You're right. My bad.

MIKE: [laughs] Git is an amazing tool. Those of us who've been around for a while use some other tools. Most of us have deeply fallen in love with Git and how well it works for us. It's one of those things that just works. Anything that just works is wonderful. Whether you're maintaining your social work documentation, or whether you're writing code, or whether you're doing something exotic, I can't think of [laughs] that it's amazing using Git.

It's a tool that generally just works. It helps me to understand what's going on, to understand what it can do and what it can't do, and to go back in time. That actually is a feature, not a bug. And it allows you to work with other people well. If you haven't used it yet, give it a try. Collaborate [laughs]. We'll talk to you next time on the Acima Development podcast.