Acima Development

In this episode of the Acima Development Podcast, the panel dives into the complexities and consequences of choosing the right tools for software development and engineering. Justin opens with a story from his first job, where his team implemented an Oracle Customer Management System that ultimately led to disaster. The system, despite promises of extensive customizability and robust functionality, failed spectacularly due to performance issues, particularly with search speed, which took minutes per query. This failure, which ended in a costly rollback to the previous system, highlighted significant lessons about the importance of realistic testing, validating vendor claims independently, and having fallback plans for major tooling changes.

As the discussion progresses, the team shares additional insights on the risks associated with both large, established tech tools and smaller, emerging ones. They debate the pros and cons of "buy vs. build," generally advocating for buying established solutions rather than building in-house, particularly when these tools fall outside the company’s core value proposition. However, they recognize that established tools, while more reliable, often become stagnant, and smaller, innovative companies may offer better solutions that come with risks, such as sustainability and long-term support. A balance is often difficult to strike, and each choice comes with potential downsides, whether it's cost, dependency, or eventual obsolescence.

The episode closes on a broader note about the inherent "bitter lesson" of technology: that simple algorithms powered by ample resources often outperform intricate, specialized solutions. This leads to a candid reflection on open-source contributions, corporate influence, and the evolving tech landscape, where many advancements are the result of unsustainable VC-backed ventures. The hosts humorously conclude with the advice that any tooling decision likely involves trade-offs and inevitable challenges, encouraging listeners to accept the imperfect nature of these choices.

Transcript:

EDDY: Hello and welcome to Acimas Dev Podcast. Today, we have an awesome crowd, right? We have Mike; we have Kyle; we have Justin, and Melina, right? And today we decided upon popular opinion, right? That we'd be talking about tooling, right? Should you be married to a tool, pros and cons of switching to a new tool, when to do it, why to do it. And Justin actually was talking off-air on having a horror story behind something that sort of segues into this. So, Justin, by all means.

JUSTIN: All right. So, what I have is a story from my first real programming job, and it was the decision to move to Oracle. Boy, I don't even know what it was called, but it was an Oracle customer management system, the CMS. And, if you're a programmer, sometime in your career, you will either write or rewrite a customer management system. That is, like, a given. I was lucky enough to do it or unlucky enough to do it on my first job. And the decision was made at levels far above me that we were going to do Oracle’s customer management system, which was, you know, Oracle a huge company, huge corporation, and it had a tool for everything and, especially, it had a tool for managing customers that we had never used before.

And the thing about it was is, like, they advertised it as, hey, you can use this tool, and you can customize it any way you like it because we have a field for everything, and you can add your own custom fields. And every field could be validated, and every field could be a search, and every customer can have any number of objects associated with the customer. And so, it was everything to everybody. And the VP in charge was like, oh, this is going to be awesome.

And we were a medical company that did medical tests. And we had customers in the numbers of tens of thousands, if not pushing a hundred thousand. So, the order of magnitude was actually not that big. But as we went along, we got given a deadline saying, “Hey, we are releasing this day, do or die.” And by gosh, we did it. We released that day. And come to find out the next day that nobody could use it. It wasn't a login issue; rather, it was a performance issue. Every single search took upwards of five minutes.

And it was just a horrible, horrible experience. And it slowed our customer relationship team down to a crawl. Phone calls couldn't be completed. Doctors couldn't update their customer fields and things like that. And for the next, I think it was week and a half, we tried to fix it. And then, the CEO came in and said, “Screw it. You guys are moving back to the old way.”

And, ultimately, we ended up throwing that away. And it was a, I don't know, I think it was on the order of a 5 million project, which some people think that's huge. Some people think that's not that huge. But to us, it was months and months of work and licensing fees and data. And it's actually probably over a year of work, come to think of it.

But the key there was that it was an environment that nobody really knew how to do, an infrastructure that was new to folks. I mean, we'd done Oracle databases before, but, all of a sudden, we were asked to develop within the Oracle framework and using their tools, and it just sucked. And not only that, but our test environment was certainly not populated with a large enough data set to validate everything. And I believe that the fallout from that was the...I think that VP got fired. The rest of us just, you know, kept on going. And so, that's, like, the tale as old as time is, you know, some middle manager or upper middle manager gets canned because their project was a horrible implementation.

EDDY: So, what's the underlying problem here? Was it lack of training? Is it the fact that you're developing in Java, or is it just Oracle in general [laughs]?

JUSTIN: I don't know. There's probably plenty of blame to go around, right? But the lessons that I learned was the fact that you cannot depend on salespeople for the numbers that they give you. You have to prove it out yourself. And then, you have to test with as real production data as possible, and that includes complexity. That includes number of rows. That includes number of transactions, all of that.

And you actually should have a rollout period where you have a beta with production data for X number of time because it basically shut the company down for a week and a half while we tried to make it work because they didn't have a fallback. And, luckily, they were able to fall back. They were able to roll back to the old way after a week and a half, but it was just...it just sucked.

And, also, from the developer point of view, using the Oracle tools...because we had to use their packages. We had to use their architecture. Previously, we were just using their database as a data store. But now we were using their libraries. We were using their SDK a little bit. We were using their, you know, all these things, and it just sucked. Long story short, Oracle sucks, and use Postgres [laughs]. Even back then, Postgres was a thing, so...

EDDY: I'm just surprised the company was willing to go a whole week with production being down until they finally decided, oh, you know what, cut the losses [laughs]. Let's go back to...

JUSTIN: It's funny because different companies have different tolerances. This was a medical company. And so, when I say the whole company was shut down, I mean just, like, the customer side. We were still doing tests and things like that. And so, there was a certain amount of tolerance there for a couple of days. But the CEO was just like, you know, after a little while, he just couldn't take it anymore.

KYLE: So, with that, were you privy to why they concluded on Oracle at all? And was there any competition why that tooling was selected?

JUSTIN: That was all so far above me.

KYLE: Far above you.

JUSTIN: I was fresh out of college, but, man, it was a trial by fire. And just to follow up that a little bit, we were a development team of about 20. Over the course of the following year, I think they lost 15 of the engineers just because, you know, there were other problems with that organization, technology-wise. That was kind of the same time I left as well.

MIKE: Yeah, I can say it's not always the big players. In my first job [laughs], we switched to a new customer management system, and it was built by, like, some guy in his garage. And it really was just somebody [laughs], I think, the CEO knew. And the entire thing, the entire system, was built around stored procedures in SQL Server because [laughs] they provide functionality to call out to externally compiled binaries, which, of course, you don't have the source code for that he had built. And there's no source code control, right? Because it's all just implemented straight inside the database. And there's no searching. You just have to know which store procedure to look at or look through them one by one until you find what you're looking for. And that was the entire system.

And we bought it, and then, of course, the person who we bought it from wasn't available for assistance. So, it was on us to manage it ourselves. And we kept that tool [laughs] for as long as I was at that company, meaning I learned how [laughs] to navigate through every bit of that code in order to make it work.

There's a lesson here about how you provision tools at a company, I think. As a company, whoever you are, you're probably not a customer management system company. And if you're putting all of your efforts into building that in-house, you're diluting your expertise in what you’re probably supposed to specialize in, whether it be medical products, or some sort of medical services, or financial services, or whatever the case may be. You are focused on somewhere outside your area of core expertise, and that often goes badly. I am, in general, in favor of buying the tools that do the right thing, that do the right job. But there's some caveats in there, right? It's the right tools. You go through some processes that involve people actually using it.

WILL: Yeah. I mean, you should never, I don’t know, you should never invest in a tool outside, in my opinion, I don't think, without an overwhelming reason, you should never invest in any kind of tooling that's outside your sort of core value proposition past a certain scale. Because there's people who've done stuff where Facebook basically rewrote PHP, which is ludicrous for anybody but Facebook.

EDDY: They built their own framework, too, right? Like, it's insane.

WILL: Yeah, yeah, absolutely. And building their own framework, I think that one's, ah, you know, like, how do I put it? I think that proposition is dubious. I think React might be stunting a little bit, which is because if they invested that amount of resources into one of the existing things and just fixed the problems that they had with it, would they really get a worse result? I bet not. But, eh, whatever, you know, probably [inaudible 10:31], you know, like, they have their thing, and it's really deep, and they did good, and they’re like, React Native is actually really dope these days, you know what I mean? For, like, cross-platform, mobile, like, that's it. That's the game. And they actually did it, and, like, it kind of sucks for web now, but that's fine. Anyway, that's a tangent. Yeah, but don't invest in tooling outside of your core value proposition.

MIKE: You're saying buy, not build

WILL: Always. Always. Always. Always. Always. Always buy, not build. Never. As an engineer, just...engineering's the worst. Just don't do it.

JUSTIN: [laughs]

EDDY: So, you're basically saying don't build my own AWS system, right?

WILL: Is that a sub-tweet? Like, if you're discussing it, like [laughter], entirely, like, I'll be the hatchet man. They can't fire me. Literally can't. So, like, no [laughter].

MIKE: One nice thing about not being employed at Acima is you can speak your mind [laughter].

WILL: You know what, actually, you know what I mean? Like, an engineer, one of the core engineering mindsets is everything is a trade-off, right? Everything's a trade-off. And I'll say this, man, because another...it's not a requirement to be an engineer, but they [inaudible 11:52]. I'm a miserable cheap bastard, and AWS is expensive. These people have been tightening the screws pretty hard. I've been seeing some bills, and I've been like, how much does a rack cost these days [laughter], you know? Like, racks aren’t that bad. It's not so bad. I could plug cables.

KYLE: Well, and a lot of the other cloud providers are up and coming, right? And they're kind of cheaper at the moment. So, yes, you get a lot of the large support from the Azures, the Googles, the Amazons. But there's others that, depending on what you're doing, they're completely viable.

WILL: I mean, large support. I've never been large enough to get large support, so I don't give a ****.

KYLE: Exactly.

WILL: But I also, like, my experience with AWS is twisting their arm to fix something they broke in the first place. I was like, oh, man, us-east-1 is down again. I better email Amazon so they can tell me to go stuff it. I'm like, all right, cool. There it is, us-east-1 down again. What are we going to do about it? We'll get back to you. Cool. Thanks for the support.

KYLE: And that kind of brings up a good point. You're relying on a tool to provide you a service, and you're running into a situation where you're like, hey, your product’s down in a certain zone. What are they recommending? Well, become Multi-AZ, then. Become multi-region. Buy more of our product. Use more of our services. And it's like, where do you define the line? And it's, okay, do you go into more regions? Do you invest more in their product? Or do you go multi-cloud, or do you go hybrid, or, you know, like, how do you solve that? Because is the right answer going multi-region and putting more money in their pockets at that point?

EDDY: I think that --

WILL: Is the answer, like, is the answer, like, how many cancellations am I going to get really? Because, like, okay, we have to always be up. We have to always be up. And I'm like, actually, no. If you can't get a TV today, you'll be okay. You're going to be all right. It’s okay.

EDDY: So, can you ever consider --

WILL: But can you check that cap spreadsheet, you know what I mean, before I go on vacation? It's like, well, maybe you hit it from the plane. Like, a lot of this stuff isn't actually a life-or-death situation. And to the extent that it doesn't cause churn or cancellations or bottom-line stuff, you know, like, I know we're not allowed to say it out loud, and it's not the Will's boss podcast. So, again, I could just say what I want, but I mean, like, eh, [inaudible 14:52].

JUSTIN: It totally depends on what company you're working for. Because I worked at a FinTech firm that, the downtime was measured in millions per minute, and nobody was going to die, but it was literally millions per minute.

WILL: I don't know, man. FinTech and downtime goes together like peanut butter and jelly [laughs].

JUSTIN: No, this --

WILL: I have worked with a financial system before. Y'all people [laughs].

JUSTIN: Yeah, no, this...I’m trying to [crosstalk 15:28]

WILL: ...Trading type stuff?

JUSTIN: Yes. Yes.

EDDY: I mean, I'd argue that part of the conversation that happens when switching tooling is cost, right?

KYLE: Yeah, big time.

EDDY: So, if at one point the tooling that you're using becomes way too expensive, either to maintain or to license, I think conversations will be had about switching to a different system, right? I always advocate to having a fallback, right? So, if you're using Amazon exclusively for your hosting service, if one of them goes down, like, one of the regions goes down, like, you're done. And so, conversations need to be had about what do you do in that scenario? Do you go to a different provider that's not Amazon? And if you do, what's the cost? What's the risks? I think that's something that needs to...

KYLE: It's a conversation of expertise, too, because a lot of the time the company will invest not only in a cloud, but it will invest in engineers that are specialized in that cloud. And so, then, if you're going from Amazon to Google, well, now you got a bunch of Amazon cloud engineers. And what does it cost then to switch to Google Cloud? And that's sometimes a lot more costly than just eating the increased billing cost.

WILL: And also, man, in all honesty, it's like a parallel algorithm. You might have to rewrite your whole thing to be like, oh yeah, that data center is down. It's like, oh, I'll just use the other one. Okay. Whoa! Whoa! Whoa! This database went out. What happened to all the transactions? And is that cool? That was...whoa! Whoa! Me and Mike spent, what, a year reading a book on that, and it was heavy duty, man [laughter].

JUSTIN: So, I think there are a couple of things that you need to look at when you are deciding which architecture you want to be. And I'm glad you brought up that Google Cloud versus AWS. If you can write your infrastructure as code, which, you know, you guys are, and we are, and most people are these days, you can be provider agnostic to a certain extent and be able to, you know, the migration from AWS to Google Cloud or to Azure would hopefully be less painful because you've got everything defined in your Terraform or, hopefully, using Terraform, maybe not CloudFront or whatever.

But as you go along, I think that is the key. You design your systems to be implementation agnostic. And thus you can, if something better comes along, you can make the switch as painlessly as possible. That's not to say that's easy, but I think that's the ideal.

WILL: The eternal lie of cross-platform development [laughter], hmm, [inaudible 18:50] so much to do that.

JUSTIN: Hey, I was a Flutter developer for four years.

WILL: Oh, I'm so sorry --

JUSTIN: And it actually worked out. It worked out. iOS and Android, it can happen [laughs].

WILL: Sure.

EDDY: Just use Kotlin.

KYLE: Just use Kotlin [laughs]?

JUSTIN: Kotlin multi-platform isn't there yet, but it's getting better. Anyway...

WILL: Nor will it be because iOS is actually, like, in the back end is actually...iOS is really stingy on the RAM. They don't tell you how much RAM an iPhone has, and that's not because the numbers are too good.

JUSTIN: No, no [laughs], they aren't. iPhones suck that way [laughter].

WILL: They really do. Well, I mean, you could say the iPhone sucks, or you could say the JVM. JVM, come on, buddy.

JUSTIN: Oh, wow.

WILL: Do you really need that much?

EDDY: Which is why having open-source software is really important.

WILL: Yes, absolutely. The operating system should be open source. No operating system in the world ever, ever, ever should be closed source. It shouldn't be allowed. It should not be allowed. But I don't think that is podcast for the day [laughter].

JUSTIN: No, but you look at the couple of other things that you have going on there in terms of what is running your actual system. And if you have everything dockerized, you have everything set up as infrastructure as code, you can have it all be pretty provider agnostic because you could run your Kubernetes on AWS. You can run that on Google Cloud. You can run that on Microsoft Azure. And the principles are mostly the same theoretically on this specific thing. I don't know.

WILL: No, you're right. You're right. I was just busting. I was just busting balls. I mean, in the end, right? You need to have...in the end, it's different flavors of containerizing Linux, right? Except for Azure and those people. Whatever, man. There's a lot of .NET developers in the world, and they technically are people, too. Legally, they're humans.

JUSTIN: They have high therapist bills.

WILL: Yeah, yeah, Kubernetes and Docker and all that stuff, they can flavorize with different flavors of Linux pretty well. We can do cross-platform as long as we heavily water down what the notion of platform is, where it's like, hey, you can run both kinds of Linux, Alpine, no problem. You know what? You can even have Ubuntu. Suit yourself [laughter].

MIKE: You're hitting on a key point there. If you are using a pretty...and it goes back to what you said before about the operating system. If you use something that's standard underneath, and you're doing something that's really standard, that is, spinning up virtual machines within a cloud that provides compute, right? Then that is pretty commoditized. That's straightforward. But what if you want to do functions like AWS Lambda?

JUSTIN: Lambdas?

MIKE: Exactly. Is that going to exactly map from platform to platform? And the answer --

JUSTIN: Nah, you shouldn't do...I've been burned too much by Lambdas [laughs]. I don't recommend them anymore.

MIKE: I'm not speaking for or against. My argument is [laughs] that once you diverge from doing something that is extremely well-proven and straightforward on a standard platform, then it starts to get platform-specific, and then the costs start to skyrocket because then you have to...then you have to split, right? Now you're not really doing the same thing anymore, and it goes back to the cross-platform. Yeah, you can do the basics pretty straightforward, and then, all of the cost comes into everything that goes beyond the basics. And if you have to make it work across both sides, it's probably not going to.

JUSTIN: So, is that the lesson you should take? Like, if you are straying from the path, that should be a gigantic code smell or architecture smell.  And you should really analyze your requirements to see, are they really forcing me to go that way?

WILL: My takeaway personally is the eternal lie of cross-platform development because, inevitably, developers are going to...it's like a game to them. They’re like animals. They can't help themselves.

JUSTIN: [laughs]

WILL: It’s a compulsion. It’s a sickness, a disease. They're going to find the edge of the map, every time. They're driven towards it like conquistadors. I don't know what pushes them, but they're always going to do it. And they're always going to find the edge of the map. And they're going to write some ****. And it can't go down. And then, you start again over and over.

JUSTIN: Haskell. Haskell.

WILL: You know what I mean? Some kind of thing, man.

MIKE: Let me throw a metaphor out here. You're going to get married [laughter].

WILL: Oh, man. Oh, it’s getting hot. You’ve been trying to [inaudible 24:32].

MIKE: You keep your finances separate. You carefully schedule times so that neither of you have any disruption to your schedule with your old friends.

JUSTIN: [laughs] Why are you getting together again?

MIKE: That's exactly where I'm going. Were you ever married? There's a tricky line here that sometimes if you're going to go to a provider, you have to accept that you're going with that provider. If you're going to go AWS, you're going to go Google Cloud. You're going to go Azure. There are some things you can probably do cross-platform, and there's other things that you probably can't. You've got to make a choice. Go back to the marriage [chuckles]. You've got to make a choice. If you're hedging and you haven't really committed yourself, it's probably not going to work out very well. I don't know. Thoughts? It's a metaphor. I don't know.

WILL: Where does Larry Ellison as a divorced lawyer, like, fit into this equation? I'm trying to make the joke, and it's just not coming to me.

KYLE: So, I'm thinking about this because I'm more of an Android fan and to the ecosystem talk here and, you know, considering that tooling. I have always looked at Apple and said, I don't want to be tied to an ecosystem. I don't want to be in that. I would prefer Android. Android allowed me that, like, what, 5, 10 years ago. Now you're either in the Google system. You're in the Samsung.

Like, at some point, you do have to pick a marriage. It's not really, like, with all of our tooling, it feels like you're being pigeonholed into some type of a marriage, and you have to kind of accept it, at least I have, I feel like. Because if I want my watch to work with my phone, I got to pick a brand. I got to pick something. And what do you go with, and where do you settle? And yeah.

WILL: I think that's maybe more of an interesting question than the eternal lie of cross-platform development, right? Like, how do you pick who you're going with? Because you are right. I'll say this straight up. Me, personally, in terms of back-end infrastructure, I'm a Linux ride-or-die. Whatever kind of server crap they've got, forget it. I'm not doing it. I won't. I won't do it. I also jumped on Apple, which I have misgivings about because I jumped on Apple, right?

Maybe this is an illustrative example. I jumped on Apple because I was an indie app developer. And Android people don't pay anything, right? So, okay, so my choice was clear, right? I wanted to eat. And so, to have money to purchase food, I was going to have to do Apple instead of Android. And so, then I was stuck, and, philosophically, yeah, they rub me the wrong way. I believe in, like I said, I was rather militant in saying that all operating systems should be open source, without exception. I don't think anybody should have a closed-source operating system, and Apple ain't that way. And so, what’s a good way of choosing? Like, how do you pick the pony, you know, that you're going to ride?

JUSTIN: So, I'd like to give you guys an example of what we're facing right now, and I'll pick your brains a little bit. Three advisors. So, we are trying to decide between JFrog Artifactory and CodeArtifact. JFrog, the big industry name, CodeArtifact, not so big, but still, they get the job done. We need some place to park our builds, and be available, and we can validate them and everything, right? And we can have them part of our build process, and we can do dependent libraries and all this and that. And so, there are a lot of questions about which way we should go. JFrog, obviously, is the big industry one, but they want 150,000 a year.

WILL: Woooo.

JUSTIN: Right? But all the tools work with it.

WILL: I want to say two things, like, one, as a permanent JFrog user, JFrog sucks, and I hate them.

JUSTIN: [laughs] I'm going to put that down on my con list.

WILL: Yes. Two, this is really, like, meta, and it doesn't have a lot to do with engineering. It has, I think, more to do with finance and the rise and fall of empires. So, for me, I believe that every company kind of follows the same arc. Some people have a longer arc with better management. Some people have a shorter arc for various things, like, not going well, but every company eventually calcifies, you know what I mean? And then, like, the sort of the MBA managers take over, or a hedge fund buys them, or they're trying to make their quarterly results, and things like engineering get squeezed to death. And then, their feature set, and their agility, and stuff like that locks in place.

Now, what's the upside to that? The upside to that is they're the standard, and everybody has to meet the standard. And they have to be the thing. They suck. They're going to continue to suck. They're going to get worse every year, forever, until they collapse because they've locked it, so the concrete is set. This is what you get. And every year, it will get worse.

So, what's the other side? You got these new, up-and-coming companies. And they're hungry, and they're modern, and they're building things better because the second time you build anything, you do a better job, right? What are the dangers there? The danger there is they don't make it, right? The danger there isn't support and tooling around them. The danger there is they, in turn, do not get to make their numbers for their IPO, and they get bought by private equity, and they fire everybody. And then, they are in turn setting concrete, right? And then, they start to, you know, their inevitable slow degradation. And so, you're sort of, like, jumping on a wave, right?

The new up and comer they're going to do a better job because they can avoid all the landmines that JFrog hit, right? They exist because they didn't screw up the same way that Artifactory did. If Artifactory did it right, these guys they'd have no sunlight. Like, they wouldn't have anything. So, the question becomes, well, all right, are they big enough to be viable at the scale you're at? And that is a question that I can't answer because I got to sign JFrog. And I think about this stuff the absolute bare minimum that I can, other than cursing JFrog's name when they do me, which happens quarterly.

KYLE: That is something that we did go through, very similar story. Nexus is what we were using, and CodeArtifact was something that was on the table as well as JFrog. And we ultimately landed on the tooling that had the best support for our packages and had probably the best support for infrastructure as code, which happened to be JFrog. So, that's kind of what it came down to for us. What it comes down to for you guys, I don't know. I mean, we are a very vast, what, language base here at Acima, so we had to support a lot.

MIKE: Will got down deep there, talking about the innovator's dilemma. Big companies get stagnant and then they get...an innovator comes in, and the big founding company laughs at them like, well, they don't offer any of the features we do [laughs]. But they're cheap, and they work well for the people who want to use them. And, surprisingly, there's a tipping point, and, eventually, the old company becomes obsolete. You know, there’s Standard Oil [laughs], you know, a lot of the railroad companies, you know, you think about it, they were huge. Enormous. And --

WILL: Standard Oil still exists, baby. They changed their nameplate, but they're still around. I forget which one they are. Like, are they Exxon? Are they...I think they're Exxon.

JUSTIN: Going back to the tech world, you look at the rise of Wiz, like, specifically in the security space, Wiz was nothing until...I think the company was founded in, like, 2018, something like that. And it was a company that had to rapidly and cheaply connect to and identify all the processes that could exist on a cloud infrastructure, you know, multiple different kinds of services, multiple different versions, multiple different data stores, and everything else, and be able to present that in a great format in a cheap way. And they had to compete with the other scene companies or the existing security monitoring companies.

But, basically, this, what, six-year-old company turned down a 20-billion-dollar buyout offer from Google because they thought that they were worth more than that, so 6 years, 20-plus billion dollars. But it's a classic example of somebody, you know, coming in and just wiping the floor with existing competitors because turns out that those existing companies, they had a large feature set. But they didn't, you know, probably most of the people didn't use half their features that they had. The industry moved on. And Wiz was able to, you know, rather than try to maintain these old feature sets, Wiz was able to come in and build exactly what people wanted and do it in a cheap manner that attracted all of the industry folks to switch security monitoring systems.

And, you know, it’s nuts in some ways. But I think it's a perfect example of somebody coming in and creating exactly what people need at a price point they need, and people recognizing for them, and yeah, so good on Wiz. And I feel bad for all the other companies in the space because they took a haircut, massive haircut.

WILL: Yeah. And when you’re betting on sort of, like, startups and stuff like that, like, you're rolling the dice, right? I mean, you are rolling the dice to a large degree, right? I read a very interesting article about OpenAI and how they actually might be screwed. I don't know, you know, like, generative AI is not my deal, really. Like, I've used a bunch of it, right? I’m playing with all the toys as they come off the assembly line.

But, basically, what they were saying is the model stuff has achieved an asymptotic level of goodness, right? And Facebook is just giving their stuff away. They're pulling a page out of the standard big company playbook and saying, like, we'll just give it away for free because we make money on ads, but we could starve you out, and you'll just die, and then we win, which, okay, right?

And so, generating the large language models that's asymptotically valueless, and everything is going to go around tooling and shaping the data, in such a way that it can be consumed by these models. It was an interesting concept but more broadly to what we're talking about, that you're betting on this startup, this new idea, right? And so, what are you going to bet on? I know I have some clients who are operating on OpenAI models. And I'll be like, “Hey [laughter], how much are you paying OpenAI? Is Llama 3 good enough? Because if Llama 3 is good enough, I bet you'd save some money paying me to swap you over because OpenAI ain't cheap, you know?”

JUSTIN: So, I have a great example of that. Last month...it’s a Utah Java users’ group. They invite people to come present every month. A company CEO founder from a company called...what was it? Qualiti.ai. They do automated QA for companies. And, basically, they automatically create tests that will test your product end to end, and they will maintain themselves basically. And they spent...prior to, what was it, 2 years ago, they spent tons and tons of resources developing their own models. And all their resources went in to train their own models and everything, and that was the number one expense.

And then, and it was like the thing that made them valuable for these other companies, their clients were these models that could go and create these things. And then, OpenAI comes in and, specifically, ChatGPT. And they came in with this general model, and with the general model, it wiped their...all the stuff that they made was useless because it was, like, next generation or two or three generations beyond what they were doing.

And this guy, the CEO, looked at it, and he was like, “I've just spent the last six years of my life trying to make this work, and I made a mediocre product compared to what OpenAI was.” So, he had to make the really, really painful decision to examine what his company actually did. And what his company did was not train models; it was provide a way for companies to create automatic QA.

So, once he recognized that his value add wasn't the models, rather it was giving people these automatic tests, then he made the decision to throw away all the training that he did. And he basically contracted with OpenAI, and all the automated QA things are now just API calls into OpenAI. And he's doing his best to make that third-party agnostic. But it's just like, OpenAI was, like, head and shoulders above all the other competitors for his use case. So, it was just like...it goes back to, you know, what are you trying to do as an engineering org? And recognizing the value that you add and getting the best tool for the job when you're doing buy versus build.

WILL: Right? Never build anything that isn't directly related to your core value add. Are you a computer scientist, you know what I mean, convolutional neural networks research company, and you're the best in the world at that? Or are you a QA company that uses these neural networks? Like, the neural network stuff, like, that's heavy-duty, man. Like, we're still in, you know, very, very early days with all that tooling and building all that stuff and stuff like that.

Like, I was trying to get...and I'll tell you how I know it worked in early days because I was trying to get one of those image generation models running on my local computer. It was a real pain in the ass, and all the documentation sucked. When I find myself on a forum, like, if I find myself on a forum page looking for something, I'm like, oof, all right, look at me, Oregon trailing this thing.

JUSTIN: [laughs] You're going to die of dysentery, man.

WILL: Wouldn't be the first time.

MIKE: Have you all read the...It's by Richard Sutton. He published something years ago called The Bitter Lesson. He had learned that a number of companies will spend years building this custom machine learning model to match their use case. And they'll come up with all these tricks and do everything just perfectly to eek out just a little bit more performance. And another company comes along that uses just a dumb algorithm. And they throw a bunch of compute and data at it, and it works better. Then [laughs] they were able to accomplish all these years of work. That's the bitter lesson. Then, in the end, a simple algorithm and a ton of compute and data win every time.

WILL: Yeah, yeah. No doubt. No doubt. No arguments there. I think it's another topical tech in the news today. It's amazing how many amazing open-source projects, and products, and platforms, and such are really just the result of a VC-funded company blowing their budget and imploding in an altruistic manner [laughter]. It’s WordPress. WordPress, right?

JUSTIN: WordPress.

WILL: Automatic, right? Automatic. Invented WordPress. No. No. WordPress is a VC failure. They built this beautiful code temple, and then they just abandoned it. And good, old Matt he rolls in, inherits the whole thing, plants a flag. That's WordPress, baby, open source. But if you don't want to host your own thing, you can pay me, you know. WordPress was not...he didn't invent WordPress. Like, Ruby on Rails is actually one of very few, you know what I mean? Look at good, old Java. How's Sun doing? Not so good.

JUSTIN: Oracle's doing fine.

MIKE: [laughs]

WILL: Oracle's doing...yeah, Oracle's doing fine because, well, yeah, anyway. Yeah, Oracle's lawyers are doing fine.

JUSTIN: True story.

WILL: I actually think Oracle, the technology company, is not so great, to be perfectly honest with you. I know there are Oracle databases in the world, and I know your bill goes up every year. But I don't know how many new Oracle databases are being spawned in the wild anymore.

JUSTIN: No, everybody --

WILL: I don't know how many people have paid for that Java license to be perfectly blunt.

JUSTIN: Yeah, OpenJDK.

KYLE: Something you kind of glazed over there, in the open-source community, too; it's kind of amazing how many of the end products by the larger companies are just wrappers of open-source products. Just for instance, today, I was looking at upgrading Packer only to find out that the Azure Image Builder is just a wrapper of the Packer software. That's like, so [laughs], I mean, everybody uses these smaller pieces, you know, in the open-source community, whether or not it's them hosting it themselves and having you pay for it, or just a tool that they provide you as, you know, a paid service.

WILL: There you go.

MIKE: We've talked a lot about tooling, and there really hasn't been one conclusion but a lot of different threads [chuckles]. There's not easy answers here. But sometimes you got to make a choice, and you might get it wrong. But you got to consider those trade-offs.

WILL: Yeah, I feel like this is our least conclusive one to date. Honestly, you know what I mean, like, so, remember, kids, any choice you make is probably wrong and sucks, no matter what [laughter].

KYLE: If it doesn't suck now, it will later?

WILL: Happy Friday!

[laughter]

MIKE: It's a bitter lesson, but an important one [laughs].

WILL: Thanks, comrades. All right, well, I'll go get started on the weekend drinking [laughter].

MIKE: Until next time.

In this episode of the Acima Development Podcast, Mike humorously shares his experiences with his three-year-old son’s unconventional love for books, recounting how his son physically devours them, forcing Mike to create an elaborate security system to protect the family’s book collection. This story transitions into a discussion about problem-solving and determination, linking it to testing and validation in software development. The team laughs about how motivated problem-solving, much like a child's persistence, can reveal gaps in security or design, paralleling the way developers must rigorously test software to uncover flaws.

The podcast delves into the importance of validating features beyond just passing unit tests, emphasizing that even bug-free code may fail if it doesn’t meet the right requirements. The conversation touches on the limitations of unit tests, pointing out that they are only as effective as the person who writes them and that they cannot catch misunderstandings in requirements. The hosts also discuss tools like Qualit.AI, which use AI to generate tests, but caution that even with advanced tools, ensuring correct specifications and validation is crucial to avoid costly errors in software.

The conversation expands into a broader discussion of testing methodologies, including dynamic application security testing (DAST), the value of cross-checking systems, and the challenges of relying solely on automated tools. The hosts highlight how important it is to approach validation from multiple angles, much like using "belt and suspenders" for security, to ensure robust software that can withstand different types of user interactions and vulnerabilities. The episode concludes by reaffirming the importance of rigorous validation processes, whether through unit tests, security scans, or AI-driven solutions, to catch errors and ensure the overall quality of the code.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I have here with me Eddy, Justin, Dave, and Kyle.

I have a three-year-old, and he loves books, just absolutely loves books. He devours them. I can't tell you how many books he's been through. The problem is, I mean that literally [laughs]. He has eaten more books than I can count. He just loves the cardboard, chews it right up, removes the binding, tears into pieces. It is a real challenge [laughs]. [inaudible 00:41] the opportunity to read, but, you know, he definitely gets his fill of books, just not in the way that we intended [laughs]. It's the point where we have to take some security measures.

In my living room, I have some stairs. At the top of the stairs, there's a bathroom, but next to the bathroom, there is not a full room; it's just a loft. It's an area that's enclosed on three sides by the boundaries of the house and then a wall with the bathroom, but it's not enclosed along the railing. So, you can look out over the railing. And it's open. It's not a full room. And there's no door. You just turn left and walk into the space, and that's where we have our books. And, you know, we have a lot of books. We like books [laughs]. We're book lovers. We have lots of children's books, lots of shelves of books. And he'd go in there and just have a meal. So [laughter], that's a problem.

So, we started coming up with tactics to dissuade him from doing so. You know, we can always go in there, and grab a book, sit with him one on one. But we're trying to encourage him to appreciate books in a different way. We tried to child-gate. That didn't last long. He learned how to climb over it within weeks. So, we got rid of that. I put up a sheet of plywood and thought, well, this will dissuade him. That did dissuade him for a short amount of time. He figured out how to move that out of the way. I literally put an anvil in front of the sheet of plywood. And you know that if you're very determined, even if you're three years old, you can slide an anvil across the carpet, and he got in.

And, eventually, the solution I had to do was to mount. I cut the plywood down a little lower. It's over a meter [laughter]. It's about...it's a little under five feet high probably, and, you know, I think, like a meter and a half. And I've got it mounted on hinges, hinges for a gate, so gate hinges. So, it means they can rock both ways. So, you can turn it, you know, 180 degrees. And I've got that mounted into the stud in that part of the railing. It's an enclosed railing, So, I've got it mounted into the stud, so it’s solid. And I’ve put a gate latch on the other side that it mounts into.

At first, we got this, and it worked. But pretty soon, he figured out how to undo the latch. So, we had to get a wire that had a screw class that held it together. He got through that, too. So, we have a padlock with a combination on it [laughter]. I hope that you get some sense of how impressive my son is [laughs].

DAVE: This is motivated problem-solving, absolutely.

MIKE: Absolutely. Extremely motivated problem-solving.

EDDY: Having a three-year-old being able to slide an anvil across the carpet is pretty impressive, just saying.

MIKE: The truth is, he did that about a year ago. So [laughs], you know, he, yeah, he is strong and dedicated. I'm not going to go into all of the stories [laughs]. The other day, he saw a package on the front doorstep. So, he opened the window, pushed out the screen, jumped out the window, ran over, got the package, and came inside because he couldn't reach the latch on the door.

EDDY: He'd make a really good QA member.

MIKE: He would. And he'd make a great, well, he'd make a fantastic QA team member, which is the perfect segue into our topic today [laughs].

We're going to talk about validation of our features. And there are lots of things that people talk about to make, you know, your code good. And, you know, we talk a lot about unit tests. We focus a lot on unit tests. We've had multiple podcasts talking about RSpec [laughs] and best practices in that particular library for testing, which is great. I love unit tests. They give me a ton of confidence.

I've also had experiences in the past where I have written some code, written the unit tests, gone through testing, everything worked perfectly, got it deployed, and then realized that it didn't meet the requirements [laughs]. Regardless of how well it was written, it missed the point, right? It missed the mark. And that is completely outside of what RSpec can save you from. Unless you have...you just can't. Like, if your requirements are wrong, if you misunderstand the requirements, then your specifications are not going to catch you.

EDDY: Well, RSpec is only as good, or a unit test is only as good as the author, right? It can only be as good as the person who writes it.

DAVE: The problem behind the problem is you built the wrong thing. There's no amount of tooling that can save you from successfully building the wrong thing.

JUSTIN: Well, I went to a presentation by a company called Qualit.AI. They have AI tests or their tests are generated by an AI. And it's actually pretty interesting. You can feed it the specs that are written by the product owner rather than engineer. And, theoretically, it will solve some of that problem.

MIKE: But, again, if the specifications are written wrong, we have trouble, and there's a gap there. Now, you can only take that so far. You know, at some point, people have got to get it right. There's this really important component of creating the validation instructions to make sure that we get out the feature that we really want early on. And you can have bug-free code, technically bug-free, because it does exactly what you told it to do that is not the feature you wanted.

Further, unit tests live as a unit, right? They test a unit of code in isolation. By design, it makes them fast. It means you can run your test suite and validate your code quickly. They do not run an integrated test to make sure all the components run well together. And that's another prong of a testing strategy. There are also things like security tests, you know, penetration tests or probing for libraries with security holes. I mean, there are a whole range, a whole range of ways that things can go wrong.

And the way that you validate your code is the way that my son does. He works on it until he gets to that book. And unless it's essentially impossible for him to get in, he's going to be eating that book. And sometimes, we don't have quite that attitude. Sometimes we think, well, you know, I think it passes the unit test. It must be good. You know, I've written good code here. And once we start to get a little lax there, we get ourselves in a lot of trouble.

JUSTIN: This kind of goes into what type of testing you want to do. And the reason I bring this up is I'm in charge of the DAST testing at our company, and we use a tool called Rapid7. And one of the things that it does is it does, basically, you know, it scans the application, and it just shoots everything under the sun at the application, and it's not looking for functionality. It's not looking for usability. What it's looking for is vulnerability. And so, it's like, a specific type of thing that you're looking for, I mean, that kind of drives your testing, you know, what type of testing you're doing. And the test suite that Rapid7 uses takes basically two days to run. So, we always run it Friday at like, you know, 4:00 p.m.

But it's just kind of nuts because it throws everything under the sun at every single form parameter API that it can detect. And, you know, it gives you results, and you've got to parse through them and everything and, you know, kind of see if they were useful or not. But I think it goes back to, you know, you've got to test with intent and, you know, what you're looking for as a result of it depends on what type of test you're running.

DAVE: You said DAS testing, that's dynamic application something, something testing?

JUSTIN: Yeah, security testing?

DAVE: Security testing, okay.

JUSTIN: Yeah. So, it basically scans your application. It pulls all the endpoints, all the submits, all the things like that, all the gits. And then it just puts trash into every single parameter that I can figure out, and then it just looks at the results. And it's not like it's...like I said, it's not looking for UI results. It's not looking for usability. It's just looking at vulnerability results, which it has a limited, I mean, semi-limited set of expectations that reveals a vulnerability. It is something that can be automated, but it's like something that is, you know, thousands or tens of thousands of results that fit within, you know, that could reveal a vulnerability.

MIKE: And that kind of captures an aspect of testing that we don't always think about. Sometimes we think that our attackers are really savvy, like, nation states [chuckles] that are going to come and break into your system. You know, honestly, I've read, you know, if a nation-state wants to scan your system, they probably will. And they have reason to keep it private, so [laughs] they may get in there and just lay low. What really is the kind of attack, you know, is not a sophisticated mastermind. It's like a horde of zombies.

[laughter]

JUSTIN: Are you calling our users zombies? [laughter]

MIKE: I am not disparaging our users because it's not just our delivery users, you know, there's scripts out there. There are bots. But, you know, it's a way of thinking about the users, right? There are a lot of things that they do by accident. You give enough time; somebody's going to click that button [chuckles], whatever that button is.

I had an experience somewhat recently where there was a button that had worked. Nobody had used it in a while. Somebody clicked the button, and the data set had gotten large, and so it put a lot more load on the system than it had historically. They probably didn't even mean to click the button [laughter]. But, you know, and they're curious. What happens if I click this one [laughs]? And, you know, you will have intelligent users who will come up with clever workarounds, absolutely. But you also have the accidents, the bumps, the shambling horde bouncing around on their keyboard [chuckles] through your application, knocking stuff down. And you have enough of that; if something can be broken, it will.

And we get in a lot of trouble because we build this feature, and we think I’ve built this perfect feature, and this is exactly how it will be used. And the interface is perfect, and people are going to come and use it. And the first person comes, and they use it entirely differently and the thing blows up because you misunderstand the way that people use things. All of us like to explore the boundaries. We want to get and eat that book.

DAVE: Delicious, delicious literature, yeah.

JUSTIN: You know, when you went with that metaphor, I thought you were going to go developers wanting to get their code to production, and they'll figure out all the different paths that they can do [laughs] to get it without having to pass through the gates that you set up, including QA [laughs] and, like, automated scans and things like that.

MIKE: Give a junior developer a path to deployment, and they will use it [chuckles] or a senior --

DAVE: Developers interpret IT as damage and route around it, like, 100%.

JUSTIN: [laughs]

DAVE: And it's not because IT is bad. It's because developers are...we optimize horrifically. There's a related thing. Richard Feynman was on the Manhattan Project, worked on the atom bomb in the 1940s. And he went out to Los Alamos, top-secret middle of White Sands, New Mexico. And, like, there was a scramble project. Like, we got to go invent the atom bomb. So, they had built this little shanty town. It was all super top secret. You couldn't go in and out. They had to put a fence around it.

And they were building this camp while the scientists were trying to work in it, which meant they had a great big construction crew, and the construction crew wasn't allowed to be in front of the camp where the gate was. So, they put a great, big chain fence around it, and then they cut a hole in the back of the fence so the construction workers could get in and out.

And Dr. Feynman's like, what? Guards, gates, guns, top secret, and there's this hundred-foot section of fence that's just missing on the base. And everyone he asked was like, “No, leave it alone, leave it alone, leave it alone. Don't ask questions.” And so, he's like, “Okay, here's the thing, if I walk in the front gate, you have to sign in.” And, like they take an inquiry of what you have with you on your way in and out. “If I sign in and then go out that hole in the gate and then sign in again, they will immediately arrest me for stealing stuff, for, like, exporting things off of the base. So, I did it the other way around.”

He walked in the base through the hole in the fence and then signed out, and then walked in through the base. And every time he's signing out, they're patting him down. He's got nothing. He says, “The fourth time through, they arrested me [chuckles], which is what they should have done.” And the next day, that hole in the fence was fixed. And it took him validating the security flaw before he could get anybody to take the security flaw seriously. And I want to say the construction camp also was allowed to move forward because they realized, oh yeah, we're strangling you guys. I don't think Dr. Feynman was popular with the construction crew after that.

[laughter]

So, a real quick anecdote. This happened to me in 2010, which is a long time ago, but also way too recently for this noise to be happening. I had a developer, senior developer look me dead in the eye and say, “Every line of code has a chance of introducing a bug.” So, if you write as many lines of test as you have lines of code, you are doubling your chances of defects.

And he was dead serious that you should not write test code because it doubles the chance of writing a defect. And like, yeah, there are facepalms going on in the cameras right now, yeah, 100%. And I was so stunned. I was so flabbergasted by this that I had no response, and he declared victory and moved the conversation on. I'm like, no, no, no, no, no, no, this is not the victory face that I'm making.

I finally was able to come back and say, “Okay, here's where you went wrong. You are saying that every time you work a math problem, you might introduce an arithmetical error. You might forget to copy a zero or copy a five as, you know, as a seven, or something like this. What you are telling me is that if you take the test and then you check your work, you are twice as likely to write an arithmetical error, and that is true.

You had 40 problems. You're now working 80 problems because you're working every problem twice. But the error in one is unlikely to be reproduced in the other, and the chance of you getting a higher grade on your test is significantly higher because attentional errors don't happen on cue. Attentional errors happen randomly, and that's what checking your work will catch. And I keep that in my back pocket when I start thinking about good tests versus bad tests.

Goodhart's Law says that as soon as a measure becomes the goal, it ceases to be a good measure, and code coverage is the same way. Like, when you turn in a PR, you must write a test. And there are times when the developer they wrote the code they just want. They want just enough tests to get their PR approved so they can get out the door. And you can tell, right? The RSpec context barely read like English. They don't string together coherently. They took the matrix of all things and just blotted it out into a big, old file. And, more importantly, if you change the source code, the test doesn't break. And if you change the test code, the source code may or may not relate to it, right?

And the other side of this is that if you want to actually upgrade the code and change the architecture of it, the test breaks, like, tests outside of that code start to break because they've got their fingers reached in. This is that, this is the you took 40 problems, turned them into 80. And if any of those 80 has a problem, you get an F on the test. That's what that kind of testing is like.

What makes a good test is when that test works as a cross-check. And, for me, and this speaks to validation, and I promise I won't spend much time on my soapbox, but I'm going to get on it. Write your tests first. You will write better code. You legitimately will write different code if you write your test first because the test will force you to write code that’s testable.

Next time you're trying to test some code, you go, man, this is really hard to test; just look yourself in the mirror and say, “I did this to me,” because you didn't write your test first, right? One drives the other. The test must conform to the code versus the other way around. And if your tests declared their intent, it's very hard to copy the architecture and details of the how you did it if all the test says is what has to be done. But if you write the code first, oftentimes, you just want to get your code coverage up.

So, you write a test that just duplicates the how, and now you've got twice as many lines of how, and a bug in either one is going to slow you down. The maintainer can't maintain the code because the tests are going to fight because, yeah, it's terrible.

And this gets me thinking about things where when any one of these things can fail. If the whole system goes down, you have a very delicate system. And if any one of these things can validate the system, you have a very strong validation system. Not to dive randomly down a weird crossed metaphor, but I'm going to say firearms because that's a topic that it's a little political but more importantly, it gets everyone going, “Wait, what?” And it gets everyone paying attention.

There's four rules to firearm safety. Always assume the gun is loaded. Never point your gun at something you don't intend to destroy. Keep your finger out of the trigger guard until you're ready to shoot. And always be sure of your target and what is beyond. Those four rules. Here's the thing: if you break any one of those rules, the other three will save you. If you break any three of those rules, the other one will save you. In order to injure someone with an accidental discharge or a negligent discharge, you have to break all four rules! Versus –

EDDY: Did you mention the safety? Sorry, I was listening [inaudible 19:48]

DAVE: Oh yeah, yeah. Sorry, I did not, and that's because I stated keep your finger off the trigger. Keep the gun on safe and your finger off the trigger until you're ready to shoot. Yep. You're right. Thank you.

EDDY: That's kind of a huge gap.

DAVE: Thank you. You phrased that...Yes. Yep. Yep.

EDDY: [laughs]

DAVE: Yeah, no, it's a good one and absolutely valid. And the thing is, you have to break all four of those rules before you can run into this problem. On the other hand, you'll see tests where if you don't do this thing, the whole system blows up. If you don't do this thing, the whole system blows up. If you don't write...all four of those things have to be true in order to keep people safe. That’s a very risky operation.

And if you write your test after the code and they duplicate the how, it becomes every single line of test code must duplicate the implementation of the code, or the tests won't work. And if you want to modify the code, your test will fight you. But if they cross-check each other, then you can change the implementation and the tests are still happy because it still works, right?

And you can update your tests without having to rewrite the code. Yeah, it's nice. It's not easy. I will freely grant people that are annoyed and frustrated by this the reason we don't do it is because you don't get taught how to do it, and learning how to do it after the fact is very hard to do. And it is so worth it. All right, I’m off my soap box. Thank you.

MIKE: You got me thinking about some mathematical evidence of this. Has anybody here ever used a Kalman filter?

DAVE: Sounds familiar.

MIKE: Okay, so let me give some explanation. The math can get hairy, but the intuition is actually really simple. They use this in self-driving cars and vehicles of all kinds. Let's say you want to figure out where you're at, and you've got a few different sensors, and each of them is kind of iffy, right? It kind of can tell where you are. None of them are all that great. And if you think about how probability distributions work, they're often modeled with the bell curve, right? Gaussian curve. And if you have a really bad measurement, then that curve is going to be really flat and long, right? Because it can be kind of over a wide range.

Here's the magic. If you have two of those two really wide, flat, Gaussian curves, and you think, what happens if you combine them? What happens if you multiply them together to get the combined distribution? It is a narrower distribution and with a higher spike every single time. And if you have four different sensors, they're giving you four different distributions. You're tightening that up until you have a really good idea of where you're at. Mathematically, if you combine those probability distributions, you end up with a much better awareness, you know, from most sensors as to what they're trying to sense. Here, we're talking about tests.

DAVE: It’s genius.

MIKE: You’ve said that you want to validate that you haven't broken something. And all of your validations have their own flaws. But if you combine them together, you have much greater assurance than if you use them independently.

JUSTIN: So, how would you combine them together?

DAVE: There's a thing that I heard about from the ‘80s. And the way you combine them, right? You don't want to have one dependent on the previous; depend on the previous because that extrapolates. Your error makes it worse. I want to say it's called Bayes’ Law? Don't quote me on that, but the story behind it is if you Google how many piano tuners are there in Chicago, that's the story that everyone tells with this. When you try to guess, I have no idea what it is, the more variables you can guess at, the more accurate...and you're completely guessing, right?

What is the population of Chicago? I don't know, but let's guess. Let's say it's 10 million. Maybe it's 1 million. I could be completely wrong. It's probably not 450 million. That's more than there are people in the country. But some people are going to guess that. Most people are bad at estimation. If I had to guess, like, 8 or 9 billion of people are bad at it [laughter]. So, anyway. But if you guess too high on the population, your next guess is what percentage of those people have pianos, 10%? 1%? 50%? I don't know. I might be right. I might be wrong. But there's a 50% chance that my wrong is going to be in the other direction and that's column filtering.

The more things you can add, how many of those people, how long does it take to tune a piano, versus how much does it cost to tune a piano, versus how much money does a piano tuner need to make, you start throwing enough variables out at this, you start coming down with, well, you need to make this much money, which you’re going to have this much work, which you’re going to have this much lead time. That means the pool must be this big, which means there must be this many piano tuners in Chicago.

And if you worked out with enough variables, you start getting spookily accurate, assuming that all your efforts are our best guesses and that you never make the mistake of saying, “Well, if there's a hundred piano tuners, they will have a thousand tools.” That's chaining. That's going to multiply your error. It's when your errors cross-check each other that it gets spooky and amazing.

MIKE: Just so you know, it's kind of hard to find a piano tuner in Chicago [laughter] from personal experience.

DAVE: Okay, that's another point of data, and we'll just put that in the [laughter] -- Is it easy, or is it hard?

EDDY: So, are you telling me, Dave, that I should or should not buy a piano?

DAVE: Yes, absolutely. Actually, no, the correct answer is no.

JUSTIN: I know how to play the piano. I know how to, and I have the piano tools, but I've never been paid for it, unless I would pay myself.

DAVE: There you go. Oh, that's a good point, yeah. That becomes another set of things. Is there a shade tree market for this, right? If piano tuners are really rare and really expensive, there's going to be a shade tree market for people doing it for free. You can go down that complete crazy raffle. It's fun.

There's a Numberphiles. Anybody here look at Numberphfile on YouTube, number P-H-I-L-E? It's a British channel. They get into math and science and stuff like this. There's one on estimating the number of tanks in Germany in World War II. There was one part of the transmission that had a serial number, and they knew the serial number started at one and that they incremented linearly, and they said, every time you blow up a tank, get us that number.

And it turns out that if you have serial numbers selected at random, you can estimate how many total there are. With five or six guesses, you're within 1% of the actual total number. It is spooky. They had, like, 15 number samples, and they came back, and they said, “Germany's making 251 tanks per month,” and the actual number was, like, 249. They nailed it off of what looks like random numbers. And it's because of the column filtering. Every random number starts to exclude the extreme.

MIKE: So, how do we use this to our benefit?

DAVE: For me, it comes down to cross-checks versus that...depending extrapolation. What is a question that I can ask that will exclude the greatest number of things, and will that thing exclude valid data, right? I need to go look up a thing. Eddy and I had a conversation in private the other day that talked about...oh, it was about a comment. Okay, I realize this may not sound like validation, but Eddy had a comment about a comment that I wrote in the code because RuboCop demands that we comment our classes, right?

So, we get class transfer group, and RuboCop will say, “You must comment this class.” And the poor maintainer who's never seen this and they're just trying to document the code and satisfy RuboCop they'll write, “This is the transfer group class,” which is a completely useless comment, complete waste of time.

What is the thing that you can say about this class that will communicate the most information that will be valid? I have to start saying things about, well, what is a transfer group? What does it represent? Well, it's this thing. I start talking about this business need. Well, that makes me nervous because if the business need changes, that transfer group it's now mislabeled.

But you can kind of look at what you wrote and start to see if that definition changes, like, somebody comes along later and says, “We're going to use transfer group to identify merchants that are related on this other access.” Well, that's not a transfer group anymore. That's a related merchant thing. And that comment, while now inaccurate, is going to immediately tell a maintainer, “I am now mislabeled.” And that's because this class is now incorrectly named because some future maintainer changed the meaning of what this class was. And that's a great time to re-document the class, or rename the class or both.

EDDY: That’s assuming that the person --

DAVE: And that's a long way of saying --

EDDY: I was going to say that's just assuming that the person who happens to touch that class and alters it also reads the comment and remembers to do it.

DAVE: That's true. Okay so --

EDDY: So, documentation is only good if you still keep it up to date; that's all I'm saying.

DAVE: So, I have two unfortunate facts for you. The first one is documentation is the first thing to go out of date every single time because you'll get a maintainer who's in a hurry, and they want to make it work. They'll come in, and they'll fix the bug. They won’t even read the comments. The other unfortunate fact is that when you are not familiar with the code, and you're reading through it, you will believe a comment over the code. I have literally; I kid you not; this was in Assembly language, so it was slightly encoded. But if the line of code was mov eax1, which means move a one into the AX register, the A register on the CPU, the comment was put zero in AX, which is the exact opposite of what that statement did.

Language primary used to be 40 years ago; if you wanted to put a zero in a register, you don't move a zero. That takes two clock instructions. You can XOR a register against itself, and any number XOR itself is zero. So, you write XOR EAX comma EAX, one clock cycle you've got a zero in AX. Great. You're done. Somebody came along later and said...so, somebody had written, mov eax, zero, put a zero in it. They came along later and said XOR EAX with itself, and they commented it to say, “Put a zero in EAX, so that you'll know...” why are you XORing...oh, we're putting a zero in it. Great.

A maintainer came along later. That needs to be a one, not a zero. They changed it back to mov, and they left the put a one or put a zero in AX. And that comment stood for 15 years because nobody dared remove the comment. People believe your comments. They are important. Please write good comments. And don't document what the code is doing. Document why the code does what it does; that way, that comment will last.

JUSTIN: No, I ran into that exact situation earlier this week. There was a comment in the content of the security policy file. And I was like, is that still valid? And it took me, like, three or four hours to track down. And the comment was not valid anymore. It was giving people bad information. I was like, okay, now I got to go in and change this, remove this comment.

DAVE: And when you have a comment in there that says, “Vendor X refuses to whitelist our server,” and then you've got this egregious violation of, like, hard-coded IP address or something like that, that's actually a good comment because it doesn't say “Hard code the IP.” If I write hard code the IP, I can see that. That's in the line of code where [inaudible 30:51] follows it.

But seven years from now, when vendor X doesn't even work with us anymore, and we don't need this whitelisted anymore, that comment of Vendor X doesn't whitelist us, you can then go, oh, we documented the intent. We documented the discussion. I can now validate without any extra help that this is garbage. This code is now suspect. But ten years from now, we have all kinds of code in our app, and it's not just us. Every place I've ever worked, you'll see code that's five years old that says, “Put a one in EAX, and the line of code below it puts a one in EAX,” and you don't need that one in EAX anymore. You haven't needed it in three years because you don't work with that vendor anymore. But the comment just documents what the code does, and that remains true, and it's useless.

So, the cross-check is, why do we want this? Some of you guys have heard me say, when you write up a Jira ticket, don't document the decision; document the discussion because, a year from now, that decision is going to be wrong. And you've thrown away the discussion that would help a future person say, “Oh, that discussion is still valid, and the decision is now different because of a dependent fact.”

EDDY: Which is why I always push for, like, a more public open discussion.

DAVE: Absolutely.

EDDY: Any time any [inaudible 32:07] decision is being made, whether that's in a pull request, whether that's in a Jira ticket. It does not matter, right? Have it in a more open forum so that other people who stumble upon that same thing can be like, ah, that's why we decided to do it this way, instead of [inaudible 32:23]

DAVE: 100%. 100%

MIKE: I'd like to latch on to that a little bit. You just said the discussion in a public forum is an important tool for generating quality code and avoiding defects. There's no clear, like, linear relationship between those two that leads to this. But I think you're absolutely right. In fact, I think that may be the most important thing we can do to lead to quality code. Because something that's...what do you say? Sunshine is the best disinfectant?

DAVE: Mm-hmm. Mm-hmm. So, I think it was Will was on with us. Will joined the call; by the way, partway through, people listening at home. It was Will that said he hates when people DM him a question that should be in the knowledge base. I don't want to put words in your mouth, Will. I think what you said was you want people to ask you in public so that the conversation is documented in the channel.

WILL: Always. Always, always, always, always, always, always. DM me for personal stuff, for sensitive stuff, or whatever, but I don't like any technical discussion in DMs ever, ever. I can't think of an exception.

EDDY: Can you dissect that a little bit? I think that's really important. Because I feel like, in practice, that's not really the case, and I'm wondering if that's something that can be implemented across the board.

WILL: There's a couple of things going on. So, one thing is I work in a distributed team, right? I don't like the word remote; it's distributed, right? And so, some of that is particular to that sort of, like, that sort of workflow, in that, like, I've never been in a room with any of these people, and I probably never will be. So, a lot of implicit communication and scuttlebutt and, like, what's everybody, like, doing over here, you know what I mean? Like, if you just see a cluster of people freaking out on something, right, then you'll naturally gravitate to it, and there's an implicit communication there. So, like, you want to do that. And it's just visibility. You’re like, what's going on, and why, right? And I think that's really valuable.

And the other one is like, as we said, sort of, like, documentation, it is, to a degree, self-documenting code. One habit that I've gotten into working in a big organization, you know, with a lot of different functional groups and a lot of different workflows, and everything's changing all the time, right? The documentation isn't the documentation. We have, like, a big Confluence Wiki, and that's fraudulent. It's hilariously bad.

MIKE: [laughs]

WILL: Like comically, a comic misrepresentation of someone's hopes, dreams, broken promises, et cetera. It's all garbage. With Slack, if you're a good Slack --

DAVE: Confluence is where organizational knowledge goes to die.

WILL: And/or Jira archaeologist, you could find the tea, you know what I mean? GitHub, Slack, and Jira. Because those things are all, you know what I mean, like, Jira's maybe the least accurate. But, like, everything that gets through had a ticket, right? So, somebody touched it. And then, Github is the word of God, right? Like, that's it.

And then, Slack is just sort of useful for the things that didn't make it into a PR, where it's just like, oh yeah, the app broke because this symlink was wrong to this version of Python. Then you had to go in and hand-edit this thing. Well, yeah, I see you laughing, Justin. But you know just as well as I do that that problem right there, you've seen it, or variants of it, and that's going to stop your whole development team cold. You're dead in the water. Stone cold stopped.

JUSTIN: Those data science team.

[laughter]

DAVE: Jerks.

WILL: Yeah. And there's other stuff where it's just like, something will screw up. There'll be some stupid workflow that I need to know once a year. And, like, once a year, I'm frickin **** if I can't...oh, what was that? And it's not worth documenting. I mean, it is, but I can't justify it to myself. Anyway, so, I mean, that's my general philosophy on that. And I would say, like, you know, both, like, I encourage everyone, across the board, think really hard about doing this, and it will bear fruit over time. Even if it's stupid, man, because if you keep –-

EDDY: I've had someone tell me that the code is the documentation.

WILL: I mean, it is.

EDDY: I mean, yes and no. Yes and no.

DAVE: I'm just going to say no, not even yes and no, just no.

EDDY: Yes and no.

DAVE: That's like saying the product documents the workmanship. No. No, it doesn't.

EDDY: [laughs]

WILL: I broadly agree with the code as a documentation. Like, just get good, guys. Get good. Just read the freaking code. Just read the code.

DAVE: Better is better, yeah. Better is better. That's tautological. Yeah.

EDDY: The thing is, it would take me longer to look at a code to see what it does than if you just had something to tell me what it does.

JUSTIN: You need a ChatGPT or something [laughs] to tell you what it does [laughter].

EDDY: Copy this whole class and ask it, “What does this class do?”

JUSTIN: It's freaky. It works a lot of the time. But, anyway, that's kind of here nor there.

DAVE: We're doing a Copilot demo. In the lesson, the presenter was saying, “If you ask Copilot, ‘What's your favorite flavor of ice cream?’ It will say, ‘I only want to talk about code.’” So, I immediately started trying to get Copilot to talk about ice cream with me. Like, well, okay, but if you did what would...and about the third time through, Copilot said. “You have SQL embedded in your Ruby code.” I'm like, okay, to be fair, that's more important than what I’m [inaudible 38:15] [laughter]. I thought that was hilarious. Like, okay, yeah, yeah, you're right. You're right.

KYLE: I have found that it will work if you tell it what you're talking about is code; it'll talk with you.

EDDY: Oh, I've never had to try to have a human conversation with AI. It's still too weird to me. I refuse to accept it.

DAVE: I've been playing a lot of [crosstalk 38:38] lately, and it goes weird. It goes weird in some really interesting ways. Yeah.

MIKE: [inaudible 38:44] that will tell me what your favorite kind of ice cream is.

JUSTIN: So, my AI anecdote of the week was I was at a choir this week, and I wanted to know if I was singing off-key because we're in a choir, and sometimes it's hard to tell. And I asked Claude.ai, which is my favorite AI tool, “Write me a Flutter app that would use my phone's microphone and tell me if I was off-key or what note I was on.”

And within a five-minute conversation, it gave me working code, and, you know, I could have run that at my house and had it running on my phone. And so, it's just like, that was my, you know, oh my goodness, holy smokes AI of the week, you know, using libraries I wasn't familiar with using, you know, other things, but they were the right...I mean, it was working. I don't know if it was like, I mean, it was okay from an architecture point of view, and it was, like, an easy solution for the tool that I was asking for.

EDDY: Here's my problem with AI. It's like you have to speak to it like it's a toddler. Let me elaborate.

[laughter]

DAVE: They're getting better currently. I know what you mean. I know what you mean, yeah.

EDDY: You can't just say, “Hey, go up the stairs.” You have to be like, “Hey, get up. Use your limbs. Use one after the other. Grab the railing on the side. Start with your right foot, then your left, then your right, then your left. Make sure you don't fall. Keep your balance. And then, when you make it on top, stop [chuckles],” You know what I mean? Like, you've got to be very, very specific for it to be good, right? Like, if you ask it very vague questions...it's getting a little better, but it still hallucinates.

MIKE: Just like a toddler, also they ingest lots of books.

[laughter]

DAVE: Exactly. So, if you want to spook yourself, ask the hard question, but then start a conversation with the AI about the hard question. For example, I just asked ChatGPT because it's saving out old chats now, and I asked it just now while we were chatting, based on my conversational history, what do you think my favorite flavor of ice cream is? It got the right answer.

EDDY: Vanilla.

DAVE: Salted caramel.

EDDY: Oh.

DAVE: Like, legit.

JUSTIN: Whoa, oddly specific.

DAVE: Yeah, oddly specific. And it explained like, the AI it said, "Based on our chat and your quirky sense of humor, it's going to be something oddly specific and nontraditional." So, it was like, espresso, salted caramel, and one other, like, bubble gum or something like that. And I'm like, second one, salted caramel. So, there's stuff in there. Have ChatGPT guess your age based on your chat style, on your writing style. It will decline at first because it doesn't want to, like, trigger you, or, you know, like [laughter], like, it's programmed [crosstalk 41:34]. Well, it's programmed to not say things that would get it canceled, right?

EDDY: It's like, Dave, based on our history, I think that you're ten years old [laughs].

DAVE: Yeah, emotionally, you are 12.

WILL: [inaudible 41:48]

DAVE: But your writing style suggests you're 12 from the 1970s. And I'm like, yep, got me. Yep.

EDDY: All I'm going to say is that have we all just tried eating books? Because he could be onto something, and books could just be really good. Just eating.

MIKE: A wad of old, soggy cardboard in your mouth?

EDDY: There could be something there. Have we all tried --

JUSTIN: I've been to restaurants that, you know, have been worse than that.

[laughter]

DAVE: I almost said the name of a restaurant because there's a place here in American Fork in Utah. They cater to the blue hair brigade, to people over the age of 60 and 70. So, the food has no flavor whatsoever. And yeah, it's a joke in our family. So, you all know a place like that, right?

KYLE: My immediate thought was to call [inaudible 42:37] pizzas, frozen pizzas

EDDY: They're actually pretty good. They're pretty good. Don't [inaudible 42:44] on them.

KYLE: Back in the day, they were.

[laughter]

EDDY: Back before I was married, they made good food. So, I have a question --

DAVE: And then, there's other places that it's ketchup on a saltine.

EDDY: We did touch a little bit in the AI. I'm wondering if anyone has had any experience to use AI as far as running validations.

JUSTIN: Oh, I'm glad you brought that up. There is that company that I'd mentioned before at the beginning of this conversation, the Qualit.AI. And they have a lot of good clients. I mean, they're a fully functioning tech company that is taking in money and providing value by using AI to create tests for clients. And it's kind of freaky in some ways because it's like I've written tons of tests, unit tests. I've written tons of integration tests. I've written, you know, end-to-end tests. We've all done the, you know, hey, let's use Selenium to write all our web tests or whatever.

And all of our tests are always brittle and, you know, and they're outdated in the next release. But it's like, you go in there with good intentions, and it turns out to require a lot of work to maintain that test suite. But, all of a sudden, you can take your application and your specs and throw it at this AI, and you may not have it perfect, but it gives enough value that people are willing to pay a lot of money for it. And it's interesting and it'll be even more interesting in a couple of years where the AI becomes even better at maintaining those. And you just, like, throw your documentation at it, and then your application endpoint, and, you know, you say, "Hey, tell me if anything's wrong with this." and it comes back five minutes later.

MIKE: I think there's something really pertinent here. We've talked about how AI makes mistakes. And so, now we're back in the exact same boat that we are in for ourselves in that I'm writing some code, and it might have some problems. So, what are all of the things I can do to validate this code to make sure that it works? And as we automate some of our code writing responsibilities, the job doesn't change [laughter]. Because our attitude is still, you know, our approach is how do I validate this code?

DAVE: I just hit on a crazy synthesis of everything we've talked about up till now with this AI thing. One of the things about AI is that the neural networks are formed differently to the way humans build them, like, organically over time, which is why when you get an AI and teach it chess, it will do moves that no human has ever played, and then still win the game. Chess players have trained against existing patterns, and it's doing random things, right?

Something that I have noticed over the past month playing around with Copilot is that when the AI hallucinates, it's glaringly obvious to me. But when my co-worker hallucinates, I just accept it as a point of fact. And when you take those two networks and run them against each other, it's a cross-check instead of an extrapolated error. Oh, my head! Oh, I'm so glad I listened to the podcast today. Wow!

EDDY: [laughs]

DAVE: There's a blog post or two in that. Wow. Hey, you guys remember blogs?

MIKE: [laughs]

WILL: Yeah, sure. Copilot's great. [inaudible 46:07] the way down.

EDDY: Copilot is particularly accurate, like, 95% of the time. It's really nice when it knows what you want, but when it doesn't, it's kind of annoying.

[crosstalk 46:18]

DAVE: But it's a useful annoying. That's what I'm saying. It's a useful annoying.

EDDY: Yeah. It's like, good idea. I see what you did there, but no, that's not what I want. This is what I want. Be smarter next time.

DAVE: That's my life goal. Honestly, like if I had a personal mission statement, it would be to be usefully annoying.

EDDY: [laughs]

WILL: You'll get there.

DAVE: Just climbing up that useful tree [laughter]. Got the annoying nailed, so...

MIKE: So, the recurring theme today, we make stuff that isn't perfect [laughs], and the way to find that is to take a multi-pronged, multi...

JUSTIN: So, I'm glad you mentioned this. My co-worker, Dan, he says we take a belt and suspenders approach to security, and we do the same thing with testing. It's like, they do the same thing; they keep your pants up, but they support each other. They back each other up. A great example of this on the security side is, we have a number of gates in our automated CI, you know, which include a static code analysis, a dependency checker, a secrets analysis, among a couple of other things.

And so, all those run, and then we also do, you know, we use Wiz to scan our containers, and then we use it to scan our deployed containers. And then, we also do...we pay an external company for a penetration test, and then we also do a DAS scan once a week or, you know. And so, all of those things are theoretically finding the same bugs. But the cost of having a security issue is so high that it's worth it to us to have all these tools checking the same code at multiple points of the SDLC that it's worth it to us to pay that, you know, to pay that thing. And it's kind of nuts, but that is the, you know, one of the costs of doing business in some cases, so...

MIKE: Where you want to tighten that curve, right?

JUSTIN: Yeah. But, like you said, David, it's like this and this, and then another dimension coming in on the intersection of all those things gives you a very high sense of confidence that the thing you are doing is correct.

EDDY: It's so funny that you mentioned the suspenders also help [laughs] keep your pants up. I don't remember the last time I've seen anyone wear suspenders. So, is that really a reliable thing [laughs]?

DAVE: Get off my lawn, you kids. I've seen them.

[laughter]

JUSTIN: I was going to see you pull out your suspenders, David.

DAVE: What cracks me up is I've actually seen somebody hook their suspenders to their belt, and it's a perfect example of extrapolating your errors because if your belt fails, your suspenders are going to go with them. They're giving you nothing except for something to put on your shoulders, right?

EDDY: [laughs]

JUSTIN: And I'm glad you mentioned that, too, because if you are depending or you're testing the wrong thing, you create a false sense of security or a false assurance. So, writing tests for tests, you know, just to have a test there is not very useful, and that’s --

DAVE: It's almost negative.

JUSTIN: Yeah, it's almost negative because, all of a sudden, you have to support this extra code. And so that is a, I mean, that's a really important thing is to, like, test with deliberation, with intent. And be aware of everything that's in your pipeline and know why you're doing it.

WILL: See, I came late, and I think I kind of almost want to offer, like, a little bit of a different perspective on the whole testing thing; I mean, in that, like, I don't really see testing as a tool for finding bugs. I've found very few bugs as a result of testing, writing tests, right? Because I write the tests around my understanding of how the thing is going to work. And if I thought of a bug that I would catch with a test, then I would like, well, you know what I mean? Just sort of like a developer verification or validation of, like what I've written, you know what I mean? Very few things, you know what I mean? Because I just push the button and make sure, you know, pops up with something right before I send it off to QA. And I'm just like, good luck suckers, you know, because it's just like, it's just a waste of everybody's time, and I hate redoing work.

And so, where I get value out of testing and where I think an organization gets value out of testing is in a really formal specification of how this thing is supposed to work. And so, when assumptions around how this thing is supposed to work change, then you have your test suite, God willing, just raising their hand and saying, “Excuse me, what?” So, I was dealing with a team that manages pricing for products across this thing. So, like, pricing is kind of like the last line of defense, right? Like, we have old, old pages, decades old, you know, decades-old pages, right?

But they're still getting money. They're still getting traffic. People are still, like, going there and being like, oh, what about a TV? What was a cool TV to buy in 1995? Well, I don't know. I want to know. All right, fine, whatever. That will be a thousand dollars if I can even find one, whatever. And so, they have this ongoing layer upon layer upon layer upon layer like the rings of a tree. Every iteration of our sort of pricing platform that they need to support, right?

And so, they have, like, easily a half a dozen different ways of slapping a price label on a product. And they all have to work because it's literally millions of dollars, right? Some of these long tail links they still convert, you know. And if we screw them up, we can hear about it.

And so, the test suite for that team, and I think a lot of teams, it's just like, if I changed, if I have this new, great idea, that test suite is like a formal specification of, like, these are the wickets that you have to hit, you know, these are the hoops you have to jump through. It's got to work here and here and here and here and here, or else it's no good. And that's where your test suite is going to save you because nobody could think of all these oddball, you know what I mean, oddball corner cases and situations.

You build up the test suite to make sure that, like, when I make this change, and I tested it, and I verified it, and, like, QA verified it, and everything is good to go, you're still, you know, you haven't forgotten anything. And so, I think of it more of like a...I think of like a ...talk about like documentation, returning to that concept. I think of the test suite as the for really real documentation of, like, what the requirements for this piece of code are, not telling you whether you're right or wrong. But, like, this is what it has to look like in the end.

And it is a sort of a self-enforcing, self-updating documentation in that, if it fails, well, then you're going to have to get right with it one way or the other and say, like, "Okay, well, okay, it’s not like that anymore. Not really.” Or, you know, oh, oh my, you know. But it's neither necessary nor sufficient to ensure that your code is right today, right? It's this long-term institutional memory that's formally specified, pretty formally specified, right? But it isn't, like, did I ship a bug? Maybe, you know. Maybe.

MIKE: We did touch on this earlier in the call before you joined, but we went into a lot more depth on the weakness.

WILL: Well, then, yeah, ignore my child-like interruptions.

DAVE: No, I think this was valid. I think so. I had a great discussion with somebody about unit testing addition. Like, why would you do this? Everybody knows how addition works. And I actually pulled out two examples from RSpec, and I said, if you use lets and you put them up and you say, expect x plus y to equal z, if you can't find x, y, and z, you have no idea what that test does. If it passes, you have no idea. It has told you nothing about what the application actually does.

But if I have a test that says, expect three plus two to equal five, okay, it does integers great, but in the very next line, if you see expect Bob plus car to equal Bob car, I just taught you that this thing will concatenate strings using the integer operator, and that is useful knowledge that's outside of what you might expect a system to do, especially if it's a currency adder. Like, why would the cash machine add strings, right? And that would start a good conversation. And that's a cross-check at that point, instead of just a dependent extrapolation.

EDDY: Will, you hit on something that kind of resonated with me, where you said there's very little times where your test will keep you honest and find a bug in your code. But I'd argue that that's what made writing the test worth it. It's those small moments that catch your error that make your unit testing worth it.

DAVE: There's a whole conversation about that, but I've had a conversation of, like, when do you delete tests, and there's stuff where I will write, expect three plus two to equal five ratchet, ratchet, ratchet. And those tests are useless to anybody, except what it is; I've got so many things in play that I just want to lock down every piece until I get to the end. And when I'm done, all those tests document how the system does what it does; I will delete those because they're not useful to any...they're useful to me as an incremental developer to make sure I'm not going to get blindsided by this thing, you know, flapping loose. But it doesn't document what the system should do. So, when I'm done, I delete it, and I'm left with the original spec, which was more of a context integration-level spec.

EDDY: So long as you run your tests with a format and the context and it blocks are really descriptive, that documents itself. That's all I'm going to say.

DAVE: I had a very happy day when Tad said that when you say, rspec dash fd, we all just assume that means format for Dave because I always want people to run with your format turned on. And you can tell which developers just get green dots because they do not write tests that read like sentences. And the people who do, their tests read like sentences.

MIKE: We've talked a lot about, you know, multi-modal sort of testing. Do your unit test, but that is not going to save you. You've got to have an attitude that you're going to try this any way you can. You're going to drag the anvil out of the way [laughs] and get in there. And if we approach our tests that way, you know, we could just get so much more.

And when I say test, it's validation, right? And this applies to a much broader thing than just unit tests. Making sure that things work is more than just one thing. I think that's the key takeaway here is that it's more than just one thing. Hopefully, you can put some of this knowledge to use.

And till next time [chuckles] on the Acima Development Podcast.

In this episode of the Acima Developer Podcast, Dave Brady, along with panelists Eddy, Kyle, Ramses, and Will, discuss the various ways people learn, sharing personal anecdotes and insights. Dave kicks off with a humorous story about learning to swim by being thrown into a lake, which sparks a broader conversation about how some people thrive when thrown into challenging situations, while others prefer a more structured approach. Will identifies with the former, explaining how he learns by setting impossible goals and pushing himself to figure things out, often attributing his approach to an undiagnosed ADHD coping mechanism. Eddy contrasts this with his preference for having safety nets in place when learning, underscoring the diversity in learning styles.

The discussion also touches on the importance of incremental learning, with Will emphasizing the strategy of "debugging from the known to the unknown." He shares advice from his engineer father, explaining how building knowledge step by step can help navigate new problems. They also dive into how experience in multiple programming languages helps make learning new ones easier, but caution against assuming that all programming languages follow the same rules or paradigms. The group explores how learning through experimentation, much like playing music, can lead to deeper understanding, yet each new tool or language requires its own dedicated study.

The conversation wraps up with a reflection on the importance of continuing education for software developers. Will argues that the industry often neglects training, which leads to burnout and inefficiency, as engineers are left to constantly catch up on their own time. Dave adds that knowledge is crucial to a developer's career longevity, stressing the need for ongoing learning even if employers don't invest in it. The episode highlights the parallels between music, programming, and learning in general, emphasizing the importance of both structured and self-driven approaches.

Transcript:

DAVE: Hello and welcome to the Acima Developer Podcast. I'm your host, Dave Brady. Today, we've got Eddy, and Kyle, and Ramses, and Will. We've got a really great panel today. We're talking about learning. And Mike usually likes to start with a story, and I don't have a good...I've got too many stories about learning, and I don't want this to be the David Brady talks about his life story again show.

I’ll just start with one of the stories from my childhood, which is that a lot of times, I learned the hard way. My dad taught me to swim by throwing me in the lake. And the hard part wasn't learning how to dog paddle. The hard part was getting the knot untied from inside the sack.

WILL: [laughs]

DAVE: So --

EDDY: The joke is that you were destined to fail because you weren't given the proper...gotcha.

DAVE: Yeah. Except, a lot of times...for the people listening at home, about 30 seconds before we started this call, I told Eddy that he should host, and he said, “No, no, no, no.” And I was like; I'm this close to just making him do it because the best way somebody [inaudible 01:07] that they can swim is just throw him in the lake. And it feels like you're tied up in a sack because you don't feel like you're maximally competent. You are minimally competent. That's what beginning is. And so, yeah, all right, so we took the joke. We overexplained it, and then we turned it into a metaphor. Why not?

EDDY: I will say, David, it's funny because I've met some people in my life who learn best by being thrown in a bag and tied in a knot, and some of them prefer that. They like to just be tossed in the deep and figure it out, you know? I know that there's very little people in the world that learn that way, but I do know that there's a lot of people who actually excel being in a [inaudible 01:46].

DAVE: And, for those people, skydiving and working with cobras is probably not a good career.

EDDY: Will, I saw you raise your hand. Are you one of those people?

WILL: [laughs] It's me. Like, I am that person. Like, I have tried, like, so many times in my life, you know, so many times in my life, to be like, you don't have to do it that way, do you? Not that way. Not again. Again? But how I learn is I get myself in over my head. I assign myself crazy goals, impossible deadlines, crazy projects. I have no idea what I'm doing, and I don't know; I’ve figured it out. I pull it off [laughs].

EDDY: Is it because you want to tear things open? Like, if they assign you something and they're like, “Hey, Will, get this done,” you prefer to be proactive and undig and, you know, be...

WILL: I think it's, I mean, as I learn more about sort of mental health and, you know, work styles and stuff like that, I think it's just a...I think it's an undiagnosed ADHD coping mechanism that that's the root of it. But, man, I don't know, I mean, like, I think there's a certain level of, like, if I may pat myself on the back a little bit, the breadth of things that I've done pretty successfully over the course of my career is significant.

And I don't know a lot of people that have covered as much ground, and I don't know...it's hard to summon up the requisite psychic energy to really push yourself in a completely non-toxic fashion. Jet fuel is deadly poison, but it'll...you put a big enough pile of it under your butt and set the fuse, and you're going somewhere, you know [laughs]?

DAVE: There's a YouTube guitarist named Rhett Shull, and his slogan is, there is no plan B. Like, he will actively destroy plan B. If he wants Plan A to work, he cannot have a Plan B in his pocket, or he'll fall back on it every time. And so, that's literally his channel slogan is, remember, there is no Plan B.

EDDY: I will say I learn way different than you, Will. Like, if you throw me in the deep end and I don't know how to swim, I expect to have a lifesaver somewhere, like a jacket, or floaties, or whatever. It's like, cool, yeah, like, I'll put you in there, but you at least have a safe haven where you can fall back on in case you can't figure out how to swim. That's my personality. However, if you don't give me floaties and you don't give me a life jacket or a vest or anything, I can at least learn how to float [laughs], you know, stay afloat, you know, like, I won't drown entirely. It's just not the ideal spot you want to be in.

WILL: I mean, it's not like I won't ask for help. It's not like I won't ask for clarification or anything like that. I just, I don't know; I mean, that's just how I do it. I need a certain amount of fire under my butt to do my best work.

EDDY: That brings up a really interesting point, though. Like, what are some of those techniques that you do then? If you get put in that situation to learn, what are some of the things that you gravitate to that sort of help you?

WILL: When I'm in it? Yeah, it's one of the best pieces of advice I ever got was from my dad, a capital E engineer, and he always said, “Debug from the known to the unknown,” right? And so, what you always want to do is you want to sort of expand what you're doing, expand your knowledge from a functional state, from a working state, and it doesn't matter how petty, and trivial, and stupid. So, if I was like, let's say I'm going to do a brand-new thing, right? I'm going to write a thing in a brand-new language, right?

Like, I've said this on this podcast multiple times, and you haven't heard the last of it after I get this one out, is start as small as you need to get a functional kernel, a nucleus of something that you can do and then expand from there, which is the easy thing to say, but it's a hard thing to do because the space of a new problem, a new technique, a new territory is big. You don't even know where the edge of the map is. You don't know where you are.

And so, you need to collapse down to something functional and then sort of build out from there. And that's, you know, that's the best way that I know to do it. And when you approach it like that, I mean, it really can be a very simple mechanical kind of a thing.

EDDY: I like that.

WILL: I was actually listening to a podcast. It’s the Andrew Huberman Podcast. He was talking about, like, literally, like, a process of learning, right? If I could paraphrase, like, a podcast that I only finished half of, the learning technique, the science that he was sort of describing was saying...he’s like, the key to learning is not so much learning; it's preventing the natural process of forgetting. Your brain is ---

DAVE: Neural pruning.

WILL: Well, your brain's taking in information all the time, right? You're constantly bombarded, flooded with data, and most of it gets thrown out because it's not useful. Your brain is actually really efficient at flushing that cache out. And so, what you want to do is try as hard as you can to establish it, and you build those hooks, tie it in with other things that you already know.

Like, I've gotten really good at picking up languages because I've picked up a lot of languages. And it's just like, you know, somebody who learns seven languages, the eighth one will be easy. Well, if you know seven programming languages, then the eighth one isn't going to be so bad. And so, we're getting new techniques. We’re sort of attaching techniques onto things that you already know how to do.

If you wanted to write...let's say you didn't know Java, but you knew Ruby pretty well, running a web server in Java, you would have a theoretical construct, a matrix to think about things. It's like, oh, well, I know about, you know, routes in Rails. You know, does my Apache Server have routes the same way? It sure does. Or, I mean, Apache is not the right one, but anyway, I forget, Spring Boot server. And so, you can connect things. So, it's like, oh, Ruby has an array. Java's got an array. It's got a vector, right?

So, I mean, like, okay, so we've got this and this. Do we both have a hash? Okay, we both have hashes, right? So, we have maps going back and forth each way. And then, hey, you're actually going, right? Like, the wheels are turning. Things are moving for you. And there's not that many new ideas under the sun. You know, like, once you've gotten your hands deep into something, picking up the next thing's not so bad, you know?

DAVE: You sparked a turning thing to what you just said, which is really interesting. I ran into an interesting thing where I started in Visual Basic, and then I moved into C++, and then into C, and then Assembly. I moved backwards, like, down the tech tree, down to where I was like...literally to the point where I was soldering transistors on a board, and then came back up to a high level and very high-level languages. And as I was coming...like, I got a job moving from PHP into Java, and I'd done C++ and da da da da. And I was just like, just tell me where the braces and semicolons go. All languages are the same.

And I was talking with a grizzled, old hacker, and he said, “Go learn Lisp.” And I'm like, okay, whatever. And I really struggled with it, and I couldn't figure out why. And, honestly, I bounced. I went and looked at Scheme, and I went, okay, yeah, this is like really, really tight versions of, you know, semicolons and whatnot, but okay, fine.

It wasn't until I tried to learn Elixir, which is Ruby on the Erlang BEAM, right? On the Erlang virtual machine, and I struggled and struggled and struggled. And this language just kicked my butt, like, nothing would work. And, finally, somebody sat down and said, “You're thinking bols,” B-O-L-S, block-oriented, lexically scoped languages. C++, Java, Python, Perl, PHP, all of these are block-oriented lexically scoped languages, and Lisp isn't, Closure, Scheme aren't. I mean, those are all Lisps. Erlang and Elixir aren't, Smalltalk as well. They live in completely different paradigms, and block-oriented lexically scoped does not translate.

So, when people say, “What language should I learn?” I'll tell 'em, “You know, go learn...” You know, actually, JavaScript is the perfect example of this because JavaScript is not a block-oriented lexically scoped languages. It looks like people look at it, and they think, oh, this is Java, or this is, you know, C++. It’s not. It’s not even an object-oriented language. It's a prototype-based language, and it will behave like an OO language for, like, 98%. And then, you get this weird bug. Where the heck is this coming from? Well, it's because there's not actually any inheritance in this language. You're just layering a prototype. And there's a key point where the language goes into [inaudible 10:51].

So, knowing...I 100% agree that seven languages makes the 8th language easy, but you have to be careful when you step into that 8th language to go, is this going to be easy because I know 95% of it, or is this going to be easy because there are some huge differences in [inaudible 11:10]? And follow them very, very carefully. With Smalltalk, it's like, send a message to an object. You don't, you know, don't try to...it's rigidly OO. And if you try to write imperatively like you can in C, it will kick your butt. And it's fantastic [crosstalk 11:23]

EDDY: I will [inaudible 11:23], Dave, because you mentioned something that I think is actually kind of interesting. You said that there was a hindrance because you already had prior knowledge to a language or different languages, and it was a whole different dichotomy. So, do you think that had you gone in without any prior knowledge to any language and had to learn, would it have been easier for you to pick up? Because then you weren't translating any bad habits that didn’t --

DAVE: Let me qualify the term easy because I bounced off of Lisp so hard. I would categorize my first two years of Lisp as failing completely, utterly completely, because I was trying to make the language act like Perl, Ruby, Python, Java, C++. And because I refused to see the languages behaving differently than...I mean, I’d come from Assembly language. Everything was ones and zeros. So, obviously, I know how it's going to act at the high level, right? I didn't. I had no concept of this. And it's because I refused...and this will be a thing that we'll probably touch on a lot, but the map is not the terrain.

Sometimes, there's a difference between the map and the terrain. And you can watch people get locked up when the map and the terrain don't match, and they will hammer that terrain until it matches the fricking map. Or they will just circle that part of the map and say, “Here be dragons; don't go there.” And, no, just update the map. That's all you need to do.

And so, random 10-second one-sentence thing. The best parenting advice I ever heard was, take the time to get to know your kid because your kid is a different person than you are. And the idea of treating a child...like, I grew up in a generation where children were meant to be seen and not heard. Children were meant to be input only. You're not a transmit. We don't want to hear anything out of you. Taking the time to actually interact with another human being validates them and turns them into a very different person than...right?

I realize that's a really weird analogy, but it's the same architectural pattern. Listening to something that you think you can just transmit nonstop, to learning that you can listen to that is a huge power-up. Sometimes, pay attention to the terrain rather than the map. Did that make sense? Did any of that make sense?

EDDY: [laughs] I like the analogy where you said all your kids are different, right? And you can't treat them the same. And then, you can probably force them to act a way, but then it's not going to be as fluid. It's not as natural because that's not the way the kid is designed.

WILL: Well, let me ask you this, though, because I think you'll probably agree with me, but we'll see. I still think you would pick it up faster, even, like, having some sort of, like, you know, a little bit of blocks, right? Like, mental blocks around stuff.

DAVE: Yes. Oh, 100%.

WILL: I still think you got there faster than if you started from zero.

DAVE: Yes. Oh, yeah, yeah, because once I figured out that I was bouncing off this language, once I realized, oh, it's the map, and the terrain don't match, I was able to go, oh, and then, all of a sudden, that 95% match was able to apply again. And, yeah, and then other things, like, one of the best things I heard in the twenty-teens was, if you want to be really good at Ruby, go learn Lisp because it will make you a better Ruby programmer. And I've meditated on that a lot, right? It's like, why would you learn this archaic language that nobody uses? Because it'll make you a better programmer.

Because there are things that you will learn that we don't ever use over here, unless you've come from Lisp, if that makes any sense at all. It's kind of like taking a data structures class, you know, or an algorithms class. The first time you learn about Big O Notation, and, all of a sudden, you realize why that database query is taking so long. You don't cover databases in the Big O Notation class, right? You usually are covering algorithms and processing time. And then, all of a sudden, everything around you, like, people lining up at the DMV, you realize this is a Big O Notation problem. And being able to apply the structures to it, yeah, hugely, hugely beneficial.

EDDY: Dave, actually, the reason to my previous question was because I wanted to lead to eventually that final hypothesis, where it's like, yeah, even though a language that you are avid in is completely different on the way it behaves than a new language, you have still trained your brain, you know, to be proactive and resourceful and think like a programmer, to think like a computer, right? And that sort of thing is kind of hard to learn if it's your first time, right?

WILL: I have a only mildly unfair amount of skepticism around programmers that really only know one language well, you know? Like, if you're like, “I'm a Java guy.” I'm like, “Well, do you know, like, Ruby or Python?” “Nah, nah.” “Do you know JavaScript?” “Nah.” “Lisp?” “Nah. Nah, I just do Java or Kotlin, you know, both of them.” And I’m just like, you know, you're not necessarily dumb, but, you know, it's like you open up a toolbox, and it's just hammers, you know what I mean?

DAVE: Yeah. There's two types of seniors that I've seen to do that. First of all, a junior who only knows one language, I'm kind of okay with that, right?

WILL: Yeah, it’s fine.

DAVE: And if you've spent five years without learning another language, that's when I'm going to tap you on the shoulder and say, “You need to go learn another language. You need to get out. You're starting to rut in, right?

But I've seen a programmer with 30 years under his belt who was just C, not even C++, just C. And he was really, really, really good at programming firmware. Like, this man knew his hardware inside and out. So, he was basically using this language to do his career. His career was not programming, if that makes sense.

This guy, man, he knew graphics cards, and he knew things about like, “Oh, yeah, the 3D chip is probably going to be in a bad mood when the bit blit comes in off the 2D pipe.” And you're like, “How do you know? Like, they don't have moods.” And he’s like, “If I tell you it's a mood, it's easier than spending a week with you explaining the histories to states on da da da da.” You’re like, “How do you know this?” “Just keep doing the same thing for 30 years.”

But, for him, it was a career, and it was a tool. And if you are a programmer, first and foremost, especially if you're in the tech economy where you're changing jobs every two years, you better be fluent in seven languages because that is your career.

EDDY: So, Dave, let me ask you this then. If someone is saying, “Hey, I'm a senior developer,” what is a fair rule of thumb for that person as far as how much diverse they are in other languages and how fluent they are? Like, what would you think is a proper and fair expectation for someone who considers a senior?

DAVE: I am going to give you an answer that's going to sound like a mystic senior kind of answer, which is no.

WILL: [laughs]

DAVE: When I interview somebody for a programming job, I don't want to know how many languages. It's good to know. If you tell me you know 17 languages, or you know seven languages and you know 3 of them really, really well, that gives me a one-sentence idea of the shape of your mind and your career. If you've gone 15 years and you've only learned one language, that tells me something about who you are.

When I interview a senior developer, I don't care. I expect you to be fluent in some languages, but I don't need you to be fluent in Ruby. I don't even necessarily need...and if you're a junior, I don't even need you to be fluent as...I'm saying this badly. If you come work for me, I need to know if you can think; that's what I'm trying to say. I want to know if you can think. I cannot teach you to think.

But if you've been doing this job for 15 years and you've only learned one language and you don't have a really good reason for only learning one language, you've just told me how you think, which is kind of crusty and, you know, resistant to learning. And for my personality and the way I like to work, that's going to be a problem. For other people, that might be great, I don't know. So, I use language fluency and capacity and facility with language as a proxy for your ability to think and learn over time.

WILL: You know, I don't think I've ever asked or particularly cared about someone's language background. I think it's never really come up for me, past, like, you know what I mean, like, if I have a specific job, I need to know that you could do, you know, that specific job with some facility. But, I mean, generally speaking, I'll talk. I'll ask about projects you've done and things you've done. And, honestly, if you have a sort of a diversity of like, oh, I worked on this for a while, and then they need some help over here, so I did that, and then, they need some over here, so I did that, and I just picked up this thing, then it's almost a natural kind of thing.

And so, if you have kind of, like, a diversity of like, oh, I've worked in a couple of different jobs, a couple of different projects, a couple of different teams, you know, then you'll just get it. I mean, it'll just...you would have to be working pretty hard not to do it. Like, the only way I think you could really reasonably work on a good diversity of projects and not pick up just various languages would be if you were working maybe a long term, like, a long-term employee of a company that was fairly rigid in terms of, like, how they do things, you know.

Like, I could see somebody being good, who's doing good work and taking on diversity of work and is, like, thinking, well, you know, but they just happen to be in this environment. Although I will not lie to you, like, it would factor into my evaluation in that I would need to make a judgment call as to whether this person could, you know, spread their wings and fly outside of this, you know, particular corporate ecosystem.

DAVE: Yeah. And if somebody's had...if they've been in five different places over the last eight years and we ask them, you know, “So, tell me about this...” and you very quickly you start to...they all like, “Well, at this place, the team worked on this, and at this place...” and you start realizing it sounds like you were in the backseat there, and okay, so you're a backseat person who waits for things to happen around you.

And then, you get other people that are just like, “Oh, I did here. Man, we did this, and we went over here, and we did this.” And even if they didn't sound like they were in the front seat, you can tell that they're leaning forward. They're like, I want to know what's coming down the pipe next, and I want to engage. That's a pretty interesting...we've been talking about learning, but we've now kind of diverted into, like, how do you measure somebody's personality? But you get a feel for this. And, honestly, I think your willingness to learn...I'm worried that this is going to –-

WILL: [laughs]

DAVE: Please, please don't let this turn into a political thing. I'm going to say this.

WILL: Uh-oh.

DAVE: I used to operate off of the old-school definitions of conservatives and liberals. Old school conservative is somebody who does not like progress because there are good things in the existing system. And they understand that if you change something, we might lose some of the good things.

Old-school liberals are people who say there are flaws in the existing system, and if we don't change something, we'll never get to the even better things. So, that's the old school of this, whether you're averse to loss or whether you're averse to failure to grow, and they're both really valid, and they do well in this.

And I think a lot of learning falls into that, right? The jump in, get thrown into the lake, you know, light a fire under your butt. That's a very kind of, I mean, lowercase l. These are not political terms. These are attitudinal terms, right? Lowercase l. Progressive-minded people are going to want to just jump in and flail and get to it.

But if you're programming pacemakers, we want somebody that's a little bit conservative-minded. If you're doing SecOps or, you know, other security stuff or stuff where failure can be catastrophic...I don't know if I've told this story here. I've told this story on Ruby Rogues a bunch of times. I once got a bug report that started with the sentence, “Fortunately, no one was killed, but... And that will stop the conversation in a room, right?

We had things that could shake big, heavy things. Toyota was using it to shake cars to try and rattle the welds to see where the resonances were to find out if the car was going to rattle apart on the road. And we had a bug, and we punched one of the thrusters maximum, from zero to maximum instantly. And we launched a Camry across their fab. And everyone wore their brown pants that day. Yeah, so when you send the frame of a car, it’s scary.

EDDY: I will say that you got my attention immediately when you started your sentence [laughs] [inaudible 24:04].

DAVE: Yeah, yeah, fortunately, no one was killed.

EDDY: [laughs]

DAVE: But, yeah, yeah, that's...right? So, if you're programming a pacemaker, that's an issue. My neighbor just bought a Tesla, and I am boggled by the fact that the cars that don't have the auto drive still have all of the servos and linkages and stuff like this, to, you know...And oh, it’s so that you can purchase this upgrade, you know, right from the dash of your car. And I'm like, yeah, but they put 20,000 worth of servos in the motors that if there was an SE Model that never was going to have auto drive, they could have saved half the price on manufacturing the vehicle. Why would you do that? And he said something that scared me. He says, “I'm not actually certain there's a linkage mechanically from the steering wheel to the steering column,” And I’m like --

WILL: There isn’t.

DAVE: What happens if the computer crashes on the road? Like, safety engineering is like, if you brick this car at 70 miles an hour, you have to be able to get stopped without dying. And I'm sure the smarter people than me have an answer for this that there's probably some way that the car will keep you from, you know, dying in a fireball, but you have to think of that. Pacemakers have the same problem.

WILL: Listen, I don't want to take the podcast for a turn, but all I will say is Tesla has a different risk management envelope than the large automakers. And if you want to see the level of operational risk that they're comfortable with, you could type in “Autopilot fails” in YouTube.

DAVE: [chuckles]

WILL: And you're going to see some things that Toyota would not ship, and that is a decision that they make for themselves and also me, who is driving next to them with my children.

DAVE: Yeah. And there's some issues with that, yeah. It's like when the Ford Pinto...just Google, “Ford Pinto class action lawsuit,” or whatever it was, $4 part. They had a memo. Somebody said, “We can save 10 million by not upgrading this part with a recall. We'll just pay out the class action suits because if they get rear-ended, they will burst into flames, and people will burn to death. But we'll just pay. We'll just settle them.” They figured it would be cheaper. And that's a poor...I would have rather had a conservative or a pessimist, there we go, optimist pessimist that's probably a better word than liberal conservative.

So, we get somebody who is risk-loss averse because there's times when you want that, right? Marty Seligman's book, “Authentic Happiness,” he talks about there's times when you want a pessimist because a pilot who is an optimist will try to take off when the wings might be still icy. A pessimist will stop. Your plane is going to be late. Everyone's going to be...it's going to cost all kinds of extra operational costs, but they won't crash and burn and kill everyone aboard. And that's kind of a feature that we actually kind of want. So, sometimes you want a pessimist in those places. If you're programming pacemakers, you want somebody who's like, okay, number one, we have to not kill you, and then anything we do has to build on that. We can't violate that constraint.

EDDY: Gotcha. And that's how we learn, right, Dave [laughs]?

DAVE: Well, okay, so, how does this apply to learning? The optimist learns by getting thrown in the lake, and, you know, as long as they're not tied up in a sack, then they're going to learn to swim. Music. Pick up a guitar and just start banging on it. Find the music that's within you. That's a fantastic, optimistic way to do it.

I was, as a kid, trained by a conservative/pessimist who basically said, “That's not the way Bach wrote it, and you must [vocalization]. You must do it exactly.” And they sucked all the joy out of music. But anybody who's going to go to Carnegie Hall needs that teacher because somebody who learns optimistically they might be the most soulful blues player in the world, but they're never going to have the articulation and the perfectionism necessary for a specific type of performance recital.

So, how do you want to learn? Some people want that way to learn. Some people want to be taught, you know, sit down, do this, do this, do this, lead me by the nose, and don't do this. Don't do that. If you deviate from perfection, right? You want both, right? In the end, the way you learn to play music is to have 30,000 hours of just banging on the instrument, and you –-

EDDY: Well, Dave, you kind of hit on music, and I think that's kind of interesting because I wanted to add to that and say, I'm the kind of person that learns but based on just grabbing a guitar, right? Strum a couple of things until something sounds all right, and I'm like, oh, that sounds pretty good. Let me see if I can replicate that, right? And I'll try it again, and I'll try to add to that. And I'm like, cool, I think this is awesome. I think this sounds great. You know, I have no basis on it, on why it sounds great. It just sounds good, right? But it's fun. It's fun to learn that way because you get to experiment, right? Get your hands dirty.

I've also played with musicians where they learn by learning the craft, like, the roots, like, the internals of like, oh, I've got to know what notes are. I got to know how composition works because I have to dissect it before I even begin to touch it. And I'm like, God, that's so boring. Like, why are you doing all this research? Just play. That's, like, the fun. The harmony of it is just the experiment, not the research and development, right?

WILL: That was the problem. That was the problem with me and Ruby, right? I struggled with Ruby for a couple of years because I had a mental block around not the functional parts of it, right? Because I came from ML, like, you know what I mean? This was no big deal. I could do functional programming, which is fine.

The problem was the loosey-goosey, funky syntax and all the magic pixie dust sprinkled throughout it because I came from C, you know what I mean? Where like, if you wanted to know how that sausage got made, it would tell you in excruciating detail every single thing. And ML, ML had a nasty type system, but your types were right, or you weren't doing nothing. You know, there was no ambiguity.

There was no like, ah, you know, it just kind of works. It's like, if it looks like a duck, quack, quack, baby, you know, like, uh-uh, none of that, none of that. Like, is it a duck? What kind of duck? What is the species? What is the gender? What's its age? These were all...and so, I struggled with that. Like, I needed to know how things were going.

DAVE: I struggled coming into Ruby and Java from C++ because destructors fire the instant the object goes out of scope. Like, the way the compiler does it when that object is going to be...when we're done with it, the compiler executes the finalized code. And so, I had written a class called, you know, clogger that would say, “I've entered this scope, and I'm going to do this thing.” And its destructor would log the exit message.

And I ported that to Ruby, and it didn't work. I had things that were exiting, and then nothing. And then, my program exited, and I got 15 out of my 20, and then 5 of them just never happened. And it's because finalize happens on garbage collection, not when the object exits scope. And, like, yeah, that sucked.

WILL: Yeah, but your [inaudible 31:44] can happen anytime, you know what I mean? Like, your access after free, that can happen anytime, today, tomorrow, in a week, a year, like, oops, you're [inaudible 31:58]

DAVE: Yeah. So, in C++, the destructor is something that the system does for you at a specific point in time that you can then leverage for your operational details. In Java and garbage-collected languages, the destructor happens for the resource. It's going to free that thing up so that somebody else can use it, or it's going to, you know, if you've got a handle that you have to dispose of correctly, it's for that resource. It's not for you. It's for you in that we don't let your hard drive fill up or that we don't let you run out of memory. But it's not for you. It's not synchronous, where in C++, it was very deterministic, yeah.

WILL: Yeah, destructors, C++. Lots of fancy lies up there with your destructors.

EDDY: I want to add to something just because I want to take advantage that we have Mike on this call; by the way, Mike joined us.

DAVE: Yeah, Mike just joined the call. Welcome.

EDDY: I kind of want to dive in a little bit. I want to circle back a little bit to music because Mike is a musician, right Mike? Like, you compose music, right?

MIKE: Sure.

EDDY: So, if someone were to ask you, “Hey, Mike, you're a musician. We need someone to write music for us. Here's an instrument that you have no knowledge or context about how to write.” How do you learn? Knowing that you do understand how composition works, right? Just not the instrument. Like, what is your take?

MIKE: Well, as talking before about how you've been with musicians who want to understand structure and that [inaudible 33:26]. I think it goes the other way. You give a kid...you have a toolbox, you know, they get the hammer, and they'll hammer everything. They'll just be joyful hammering on everything. And there's nothing wrong with that, right? I mean, there's a lot you can do with a hammer. It's a useful tool.

But after a while, sometimes, you want something that is more nuanced, more specific, a special-purpose tool. Maybe you need a chisel, for example. It's hard to do with a hammer what you can do with a chisel. In fact, you need a hammer to use a chisel. You kind of need them both. It's a collaborative sort of thing.

WILL: Hammer's all the way down.

MIKE: Having more tools in your toolbox is really useful. And I think the people who pursue music longer when they discover the broader toolbox, I’m thinking about Dave saying, oh, yeah, it's just a hammer. That's not a box sounds like. And he plays guitar, like, hey, I can do this differently. But, man, having a bigger toolbox, whether that's being able to do more improvisation or being able to think, well, maybe a, you know, five, seven chords, is going to resolve really well down to my root, and that sounds really nice, you know, is really useful to know because it allows you to kind of visualize the structure, have some bigger, more...I say bigger, more specialized tools so you know what you might use in that spot.

I would say that if I had to do something for a new instrument, and I've thought about this before, you know, what would I do with some random instrument? I would have to go and study what the range of that music is, or the instrument, right? What's possible. Because you probably still have a sense of what's...you certainly know what's possible with harmony, right? With how pitches are going to sound together. But you might not know what's possible to do with that instrument.

So, you're going to have to study and from a very technical perspective, you know, all the dull parts [laughs]. What is the range of this instrument? But you still have the joy of the music creating because you're like, well, I know how it sounds, that, I get, and I can put these two together. And you actually can, I think, get some pleasure out of discovering what this new tool can do.

Hey, I just gave you a new, I think, plumb bob. If anybody's ever used a plumb bob...I think it's my favorite name [laughs] of a useful tool. But, you know, you have this very specific tool that maybe most people haven't even heard of. And what does it do? What could I use this for? And you start envisioning, well, what could I do with this? And so, just the way of thinking about it changes. It's not that you're not still enjoying that process of creation, whether that be music or whatever it is, but you're thinking about it from a different part, you know, from a different area, coming at it from a different direction and, you know, working on the other side.

WILL: So, I mean, like, I like the music analogy in terms of learning because it's a little bit decoupled from the day-to-day nuts and bolts of what we do. And so, if I was going to sit down and I was going to teach somebody an instrument, because, first, I've got kids, and I also am a musician, and I would love it if they were musicians, too. And I'm not going to, like, you know what I mean? Because they’re my kids, right? My oldest is just barely getting old enough, so whether it's a possibility for him.

So, what would I do? Number one, I would find something that he had some interest in, like a musical instrument that he gravitated to on some level, whether it's the flute, or the saxophone, or the drums, or the guitar, or the bass, whatever. Find something that he likes the sound of, like, he's drawn to on some way. Two, I would get one for him, right? I mean, which, you know, okay, it sounds obvious, but it can't be overstated, some platform with which he could practice on. Three, I would find them some instruction, some in-person instruction.

I know, Eddy, you mentioned, like, you know, playing guitar and learning guitar and stuff like that. And, man, I’ll tell you, like, the biggest piece of advice that I give to beginning musicians, that is almost universally ignored in this day and age, it's like, it takes some lessons, man. It takes some lessons. You'll get so much better, so much faster.

And then, the last one is I would put him in an ensemble, right? An ensemble so that they can play music in a group to get the fun, the joy of, like –-

DAVE: Together, to jam.

WILL: Hanging out and not just playing music yourself, playing music within the context of other people to get that joy. It's much like, hmm, I'm not going to make that dirty joke, but [chuckles] think about stuff that’s fun you can do by yourself and how much funner it could be --

DAVE: With people, yes.

WILL: And so, like, how do you analogize that to, you know, learning engineering, software development, stuff like that, you know? You could get all those things in together, and any sort of formalized method of training, be it a bootcamp or a bachelor's degree or, you know what I mean, anything, is going to incorporate all of that in, right? But it's a lot harder to do by yourself.

EDDY: I'm actually kind of curious; if someone were to ask you like, they give you the directions, says, “Hey, I know you know how to play guitar, but I need you to learn to play bass.” And the response is, “I can make my guitar sound like a bass. I don't need to learn bass [laughs].”

WILL: You can't, and that's only spoken. Only somebody who doesn't know what they're talking about would say that.

EDDY: [laughs]

DAVE: There's more [inaudible 39:18] to that, right? There’s --

EDDY: So, it’s like, we need to build this feature, and someone says, “Oh yeah, I'll do this in Ruby,” and you're like, “No, no, no, no, but Ruby isn't the equipped language to do this, what we need to do.” And he's like, “Oh, but I can make it do that. Like, I can make it bend the way it's not meant or designed to,” right?

DAVE: There's a beautiful distinction between Turing completeness and the idea of a Turing tar-pit. A Turing tar-pit is where everything is possible, but everything is a pain, right? You can play bass on a guitar. The White Stripes, the big song, Seven Nation Army, everyone thinks it's on a...no, he's playing on the guitar, on the low E string with an octave drop pedal, right? Okay, yeah, you can do it.

There's two sides to this; one is, music comes from the musician, not from the instrument. So, if you take somebody who's an excellent bassist and you hand them a three-string guitar, they're going to be okay. It's not going to sound like a bass guitar. And if you tell them, “Sit in on this ensemble as the bassist,” and you hand them, you know, a soprano ukulele, it's not going to work.

So, there's the idea of, like, I can make anything fit here because that's who I am. And then, there's the converse idea, which is, can you make anything fit into this round hole? Not necessarily. Sometimes, you’re going to have to find the right thing. Mike was talking about the tools. And something I wrote on a blog years and years ago is always use the right tool for the job. I was standing there, hammering screws into a board with a wrench, and my dad came over and smacked me and said, “Use the right tool for the job. Use a hammer to pound screws into a board.” And it's like, wait, it's still the wrong tool, right? It's the right tool.

We keep talking about music, and I love that because the conservatory nerds are going to say, you know, it's like, there's only 12 notes and, you know, on a heptatonic scale, there's only 7. And you have to be perfectly inbound this. And then, you could talk to the jazz musicians, and they will tell you there are no wrong notes. You're just not confident enough. And I'm like, wait, but surely not everybody can think that way.

And then, you go back, and you listen to country music, and you realize the twang of, you know, [vocalization], that [vocalization], that they're literally bending the string into a half sharp minor note, or a half flat, whatever it is. But they're literally bending the string to a note way halfway between. It is not on any out of key, out of tune, out of everything. And it's on purpose, and it sounds perfect because it belongs there. It's this ugly, sour note. And you need it to make the rest of the consonant notes sound great.

WILL: Sure. Sure. I mean, like, non-western music uses a lot of microtones that are not...they're not even in the 12-note scale, you know, and, like, they're not accidental, and they don't sound bad either. I mean, it's just a different way of thinking about music. But I do want to pull it back to engineering. And what I would say is I talked about this sort of like, you know, this baseline, get an instrument. Get a teacher. Get an ensemble. And then, make sure you're making it fun. Make sure it's structured. Like, all that stuff that's great. And, like I said, like, anybody who's given any thought to teaching how to do a thing is going to have all of those aspects.

You're going to have a theory. You're going to have personal projects, and then you're going to have group projects, you know, have the whole thing. However, once you get thrown out into the industry, you get nothing. It's just like, “Hey, man, like, it'd be cool if this...hey, our Go servers crashed, and the guy quit, so you know Go now.” “I don't know, Go.” “Like, yeah, you do. Get to work,” you know? And that's the gig.

Or, you know, even worse than that, you're scraping together hours in your free time where it's just like, yeah, man, I don't know if, like, I don't know if, what is it, .NET? I don't know if .NET programming is for me. Maybe I'd like to learn Ruby. And now you're sort of trying to claw together nights and weekends and stuff like that. And I think that is a substantial driver of burnout and sort of like, I don't know, it uses engineers up. And so, how do you engineer that stuff for yourself when there's nobody who's going to do it for you? But it doesn't become less, you know, less relevant to maintaining a career.

DAVE: Yeah. There's a thing that...we keep bouncing off these contrasts between overprescription and under-leading insufficiently. Like, sometimes you pick up the guitar, and you're like...or you stand on the panel, and you don't know what to do. You can't jam in a group if you don't know the key that we're going to play in, right?

And I wanted to circle back to this a little bit. One of the things that was super powerful to me early on my career was the discovery of a concept of mentoring, where a mentor is somebody who knows many, many, many, many, many things. They’ve forgotten seven ways to solve any problem that you're going to deal with, but they won't overprescribe. They will show you one way to do it, or they'll say...ah, my guitar teacher, I could not get this chord change, and he said, “Maybe try switching to this finger.” And all of a sudden, I could go from this chord to that chord, and I would just pivot. And it was easy and ergonomic.

And finding a mentor can help you really accelerate both because the mentor can say, “I'm not going to tell you the 73 ways you can do this and force you to learn all 73 of them before I let you do any of them. I'm just going to give you one to try, and I'm going to debug...oh, you're struggling here because of this. And there's this problem that you don't know that, you know, do this and, all of a sudden, your program just works.” And the first few times, you don't know why, right? Because your mentor knows [inaudible 45:25]

WILL: I wish we had time to get into the continuing education aspect of this thing. Because I think this is where, like, in my opinion, this is one of the great sins of the industry, in that, like, technology is moving so fast, all the time, so fast all the time. I do feel like it's slowing down a little bit, which is nice for me. But --

DAVE: Is it slowing down, or has the frenzied, fresh, cutting-edge moved away from where you're focusing? I'm running into that. There's as many forest fires to fight every single day. I just don't see them. And I walked into, like, the AI stuff, and I got to run so hard just to stay at the back of the pack.

WILL: I'm not sure. I'm not sure. I'm not sure. I don't know. You know, I just don't know enough about AI to really know, like, what all is going on. Although, I mean, I will say that, like, it doesn't seem like, I don't know, who knows? That's a different, separate tangent. But --

DAVE: Yeah, the idea is the sigmoid pattern, right? Things that are on fire tend to taper off over time. Yeah, yeah.

WILL: Well, exactly. So, the question, I don't know, I mean, the one thing that I wish we could have gotten into a little bit deeper is continuing education for engineers and how to keep things going, you know, because, like, it isn't...the problem with, you know, engineering education isn't so much that we don't know how to do it, you know, when we can be bothered to.

The issue we run into is, can we be bothered, you know, can we be troubled to keep these things going? And I think it's shockingly wasteful. We burn a lot of engineers, they're perfectly good engineers, just because we can't bother to be training them on the new stuff. And, I mean, if you compare that to other industries, other professions, other career development stuff, nobody does it like this. It's really crazy the way things are going. But it's still incumbent on us as practitioners, as craftsmen to keep that stuff going. And if your boss won't do it, if it isn't part of the industry, then you just have to work it out yourself.

DAVE: Yeah, your knowledge portfolio [inaudible 47:45] career, and if your boss isn't investing in it, you're dying. You need to invest in it.

WILL: Yeah. Yeah, precisely. I mean, you know, I mean, they'd be happy to get rid of all of us and replace us with new grads for half the money.

DAVE: Yeah, until they find out that what they needed for that server fire last week was an engineer who has forgotten 19 different ways to solve that problem. All right. It's great to have a team full of people that can solve any problem first principles, but it's kind of a pain when your entire team has to solve everything from first principles.

EDDY: I think basically just coming back and having a [inaudible 48:25] is basically is there's different ways to learn, and it just depends on the personality of the individual. And it's funny how we keep coming back to music, unironically, from other sessions. Like, there's a parallel between music and programming. It's great.

DAVE: Music, linguistics, and programming has one of the most creepily off-the-charts, oddly specific overlaps. Get a musician, a linguist [inaudible 48:58], a programmer, just every time. Yep.

WILL: Interesting.

EDDY: Yeah, it's awesome. I'd like to see that study. It'd be great.

WILL: All right, well.

DAVE: It was just anecdotal, so... But so, now we've got our next...this is turning into the podcast where we tell people, let's do a part two of this, and then we never do the part two of this. Someday, we'll get around to part two, so...we hope.

WILL: Well, thanks, y'all.

DAVE: I think this has been a really good chat. Yeah, thanks for coming. Thanks for listening if you're at home. And if you're not at home, go home, and then, thanks for listening at home.

In this episode of the Acima Development Podcast, the panel, led by Mike, explores the concept of team sizes and how scaling impacts management and functionality. Using an analogy from the film Honey, I Shrunk the Kids, Mike highlights how physics, like management strategies, change with scale. The group discusses how larger teams can't be managed in the same way as smaller teams and why losing a team member in a small team is more critical than in a large one. The discussion then shifts to defining the role and purpose of a team within an organization, and how the size of the team depends on its function, specialization, and the applications they support.

As the conversation evolves, the participants debate the ideal team size, suggesting that teams larger than 10 people become difficult to manage effectively. Justin and Matt argue that team dynamics, including the experience levels of team members and the distribution of leadership responsibilities, play a crucial role in determining the optimal team size. They emphasize the importance of having a blend of senior and junior members for long-term sustainability and organizational growth. The idea of "seed corn" is introduced, highlighting the importance of nurturing junior members to eventually fill senior roles.

The panel also touches on the downsides of excessively moving team members around, which can harm team cohesion, yet acknowledges the benefits of cross-training and preventing silos from forming. Will and Dave provide examples of how large teams can succeed with strong leadership and effective management, but caution that poor management can lead to inefficiency. The episode concludes with a discussion on how cross-team communication is one of the biggest challenges in scaling and maintaining effectiveness, with the panel agreeing that the sweet spot for team size lies between 6 to 10 members.

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. We have a great panel here today, a large group. We've got Matt, Eddy, Will, Ramses, Justin, Dave, and Kyle all here with us. We're going to talk about team sizes. There’s kind of a carryover from a couple of weeks ago. We talked about other items of scale. But we're going to talk about team sizes.

I'm going to start by talking about the movie, “Honey, I Shrunk the Kids.” People of a certain generation watched that movie [laughs], and it's not the only one like it. You know, there's a history of movies where people shrink. There's Marvel movies, you know, people go really small and really big. And, in those movies, generally, all the laws of physics are basically exactly the same as they are when you're larger and people are smaller, and that's it [laughs]. Everything works exactly the same.

One thing that kind of blew my mind when I ran into it is not even remotely true. And you can see it by looking at ants. Like, in that movie, the ants are big, and everything's fine. All the physics work totally normal. If you look at an actual ant, when they get hit by water, it is fundamentally different than what we experience water as, as humans. Because even just at that difference of scale, water is like glue. If you look at an ant when it gets in water, it is like, you know, it touches that drop of water, that ant is stuck, and it is a huge effort to get extricated, you know, it's like running into a giant pile of tar. Because you go down in scale, it fundamentally changes the nature of physics.

You know, you can see an ant. It's very visible that, even at that scale, our physics are different. Water does not behave in the same way at all as it behaves for something larger scale. It's just different. And, you know, and you look at a microscope. It gets even crazier. I mean, you can see in a microscope with a Brownian motion, where you get small enough, and you see things just kind of randomly moving around because of quantum-type effects. And you have molecules bouncing around; stuff is just going to generally float in a way that we just don't experience at our scale.

And it breaks our intuition because our intuition is built on the scale that we normally work with. But even something you can see, and you can look at an ant, it's different. The physics are totally different for something as simple as water, you know, droplets of water, when you go down to that size. I say this is relevant because there's a lot of things like that, where scale changes things.

We're going to talk about team size today. And I'd argue pretty strongly that you can't just take a team of two people and manage it the same way when you've got 100 people [laughter], and maybe you can try. I hear laughter [chuckles]. It's different when you scale up in size. I had this conversation recently, thinking about different company sizes. If you've got an engineering team of 1000, the way that you can run that is totally different than even at 100, you know, you go a factor of 10, and that's very different. Then you take 100 versus 10 it's totally different. If you've got 10 people in your department and you lose somebody, that is a critical loss [laughs].

If you've got a team of 1000, well, you're probably losing somebody every week, and that's probably fine because you've got your systems built up to handle that. Now, on individual teams, we have a variety of team sizes, you know, beyond just department sizes, and those scales matter. This [inaudible 03:47] in a lot of ways. So, I've been talking for a bit talking about how scale matters to kind of launch the conversation. Based on that, I'm going to ask the pointed question that prompts the debate. I mean, how big should a team size be, knowing that different scales matter?

WILL: I would first ask the question of like...And I don't think this is a singular answer, but what is the role of a team in your organization? Why a team? Like, what's the role that the team is organized around doing stuff? It's a functional group, right? Sometimes people have sort of like, okay, this is a functional group, like, this is my DevOps team, and it serves this function. Sometimes it's sort of like a...I call it an administrative unit, where it's like we have interfaces with the bureaucracy, either through product, you know, leadership, or just sort of administrative management stuff. Sometimes they're areas of specialization within the codebase. They could all be different things. So, how big it is is maybe directly dependent on that.

So, let's say you have...DevOps, I think, is a good example. So, if you have a small DevOps team, maybe you only have two, you know, DevOps engineers. Then the right size is two, you know? If you have maybe six DevOps engineers, where, like, does two teams of three really make sense? Maybe it doesn't. And so, then you'd have a team of six, which could be a little heavy for a software development team, right? A team of six engineers is starting to get kind of big. And so, the size of it is definitely going to be informed by the functions of the team.

MATT: Yeah, I think there's some questions that you have to ask yourself. How big is the application they're going to be supporting? Are they supporting multiple applications or services? Can those things be broken into domains? Do we have teams that support specific domains, or do they just umbrella and support everything? I think if you can answer some of those questions, it makes it a lot easier to determine what that team size should look like.

JUSTIN: I would suggest that there's a maximum team size that exists that you don't want to go larger than X, no matter what application you're supporting. Because it suddenly becomes a case of people don't know what other people are working on. It becomes hard to coordinate. There's not interconnectedness. All the manager is doing is managing people and losing technical insights. So, yeah, I think there may be an upper limit there that you can't cross.

MATT: Totally agree. Another thing that you need to think about is, are we talking team size from an HR management perspective so for, like, an engineering manager, or are we talking about teams from a team lead perspective? Because you may have 10 engineers that are technically part of the same team but broken into, say, sub-teams because we have a lead to support those engineers, right? So, they can work on different things, have a different person leading projects instead of the manager trying to lead all of them at the same time on multiple projects.

WILL: Justin, I saw your face, Justin. You're like, no, no, no. No, that's two different teams. You just got two different teams in the same bucket, you know what I mean? It's like a duplex. We don't live together. We just live in the same building, you know [laughter].

MATT: So, semantics are important.

WILL: But I am really interested in sort of what Justin brought up, which is like, okay, you're right; there's a ceiling for team size. Is it determined by a bunch of really specific things, right? You can't say, like, “No more than 10,” although that's a pretty good limit, you know [laughter]?

JUSTIN: I agree.

WILL: But how do I know, like, okay, like, you got to split it up; you got to split the team up; this is not working anymore? And then, like, more of, like, things that you'll see happening rather than, you know, a number.

MATT: Yeah, what if...let's say six people is this example team size. What if all six of those people are all subject matter experts and senior-level people versus a team of six people that has one subject matter expert and senior-level person and five entry-level or junior-level engineers? I think then that changes things. That dynamic is different.

WILL: Well, you may actually just want to spread them around [laughs].

JUSTIN: Yeah, exactly. That's not using your resources correctly.

MATT: Well, it may be the case that those are the only resources we have. So, do you keep them together, or do you split it into two teams of three and have a junior engineer as your lead for one of those teams? I mean, those are the kind of questions we need to think about.

JUSTIN: I think maybe a better way to look at this is like, what ingredients do you need in a team? You need probably at least one senior person or a tech lead. What else do you need?

EDDY: You need a junior on your team [laughter].

WILL: Maybe. Maybe.

MIKE: Yes, you do. I made this argument emphatically sometime in the last week or two. If you don't have a junior developer, what I said is that you're eating your seed corn. Because if you don't have somebody growing in experience to become the next subject matter expert, then your senior engineers, when they move on, leave a hole that you are not going to be able to fill. I think you absolutely need, if not a junior, you know, somebody who is learning the ropes and growing into that position.

DAVE: It's almost like a long-term, short-term sustainability, right? Because I mean, you can have a small, tightly focused tiger team that goes in, takes care of the business, and gets out, but that's not a long-term sustainable...right? That's like a task function. And you don't need a junior for that. That's fine. You can have an operating theater full of brain surgeons, but if you want long-term sustainability, you need to view...

I think we talked about this a long time ago, Mike, that when you give a programmer a problem, we don't take problem in and solution out. You actually take a problem and a programmer in, and what you get out is a solution and a smarter programmer. And if you start viewing it that way, as, you know, you are part of the output of this unit, you do, you need new people to step up because someday you're going to be gone, and that team will be gone if you weren't eating your seed corn, yeah.

MATT: Teach a man to fish, give a man a fish, right?

DAVE: Yeah, light a man on fire, you keep him warm for the rest of his life, yeah [laughter].

WILL: You know, I don't know; I mean, on one hand, I get what you're saying. But, I mean, one of my persistently more controversial software management beliefs is that we should be moving people around a lot more. So, this sort of like, you know what I mean, this thing you said where it's like, it's bad to have task-oriented teams. Like, I love task-oriented teams. I love getting together a team to do the thing, and then they go away. Dude, I like it because by enforcing that sort of procedural process discipline, you don't let people put together the little fiefdoms, and you don't let people evade accountability for corners that they may have cut, you know what I mean?

Like, you make sure that people are cross-trained around stuff, and you can have little domain expertise areas of stuff that's going on there. But I find that one of the real sort of long-term rots that gets into a software engineering organization is people get into their little...they carve out a comfortable little niche, and then they just sort of sit there. And they don't necessarily, I don't know, I hate to use the word rot, but there's definitely a lower level of operational intensity once you really have your little flow within the organization.

MIKE: They talk a lot about silos. They talk a lot about silos, right? And you know what a silo is for? Sometimes people think, who are unfamiliar with farming, that a silo is for storing grain, and that's not true. A silo is for storing wet grain so that it will rot. The purpose of it is to ferment it because that's a different way of preserving it.

DAVE: Oh wow.

MIKE: So, you take, like, green corn, and you throw the whole thing in, or you maybe throw your corn in there. But yeah, silo is different from a grain elevator. A grain elevator to keep dry. A silo is for keeping wet so that it will ferment so you can feed that fermented food to the animals.

DAVE: I didn’t know that.

MIKE: And [laughs] it's a great image because once you build a silo, it's going to ferment [chuckles]. And we talk about silos a lot. I don't know that we always think about the implications of that, that when you silo something off, that's going to start to go rotten by design.

[laughter]

DAVE: Can I offer a counterargument to both of those ideas, though? I agree with both of what you said, 100%, but it's not something to be taken completely without a qualification. There's a...oh, I'm blanking on the author. There's a book called “The One Thing You Need to Know.” And it's a book on basically what makes people happy at work. And, basically, they interviewed a bunch of people and say, “How happy are you at work and how are...?” And then, they started looking for clusters of things that were true, you know, random things that happened to be true about these people.

And the number one non-trackable business, there's no KPR spreadsheet for this, the happiest people all had a best friend at work, which I thought was really, really interesting. And I'm a naturally gregarious, very loud person. And when I read that, I'm like, I need to double down and start making actual friends at work, and I did that. When I was at CoverMyMeds, I did this. And they had a transition. They got acquired, and they brought in some senior management whose job was just to fluff the grain and just screw everybody up just to prove that we'd been acquired.

And we had a manager who the very first thing he wanted to do was separate everyone, so he just took all the teams and just sliced them up. So, you were on teams with complete strangers. And I think he had the idea of cross-pollinating, right? But if you take, and I'm going to mix my metaphors badly, if you take your sourdough starter and you spread it so thin that you kill the culture, it's not sourdough anymore. You haven't spread the culture or homogenized it. You've just wrecked it and ruined it.

And it broke me because I went to the next team we were on. I started forming friendships. We got really, really close. And they did it again. And the third time through, I'm like, I think you guys are actively trying to demoralize us. Because they were cutting us off, like, every three months. There was no chance to form any kind of a personal relationship with people. So, I suggest that as a counterpoint. I 100% agree that the professional objectives, absolutely, they should be changed, or you get bored.

MIKE: So, I talked about the silos because that's what silos do, but I actually was going to push back some as well. Have any of you read “Team Topologies?” I'm seeing a couple of nods here. It's a book worth reading. One of the things it says is that teams become more effective over time because it takes a while to learn to work with each other.

So, there's two kind of competing principles here. One of them is that if you’re isolated from everybody else, you tend to become very insular, and stale, stagnant, right? The other is that teams that have worked together know how each other work, and they're just going to work better. In general, a long-running team is a more effective team. And so, an effective approach needs to somehow combine those.

The idea of sub-teams got brought up. I don't know that sub-teams...they may just be separate teams. But one thing that can happen is, on a team, you can have rotating assignments, such that you still have a...you have a persistent team that takes on different kinds of work, and you may have different groupings within that team also. But you still have strong cohesion because you have your group of six, or whatever it is. You know each other, [inaudible 16:49] each other really well.

You may be grouped together on different projects, and you may be bringing in projects that are very different than the ones you did. Having a team work on different projects can help break the silos without necessarily disrupting the team for no reason. And I think that disrupting a team just to reorganize is a really bad idea.

WILL: You know what? Any virtue taken in excess is vice, right? You can screw it up in both directions, right? But, like, persistently, what I would say, like, the cancer eating away at software organizations, there's two things, cross-collaboration among engineering teams and organizations is just a bag of hurt. It's sucked everywhere, and some people suck more; some people suck less. But it is the number one bogeyman of everybody, everywhere, all the time, forever. Like, I've never seen an exception to this, and second only to crappy managers.

And those are two things that must be hunted down and eliminated constantly and mercilessly. And it's just sort of like, if you're not stirring the pot, those two things are going to creep in. It's a constant bailing out of water to keep those two things from sinking you. It's inevitable, you know; you have to have an answer to it.

And I think moving people around, you know what I mean, like, I agree, you can do it too much. But moving people around has a lot of really strong positive effects in letting your organization thrive, you know what I mean? It's like, oh man, this manager is always getting it done. But like, oh man, everybody hates working with this guy. Like, oh man, this junior engineer is really good. We don't have a senior seat for him, but like, man, he's really ready to go, ready to move up.

Or, like, oh man, this senior engineer has kind of lost the plot. And they're smart, but they're just not working well with people. Like, all of these things are stories that I'm sure a name popped up into each and every one of you guys’ heads as I was saying each and every one of these things. And if you're stirring the pot, if you're rotating through, if you're giving people opportunities to do good or to do bad, then that kind of stuff is going to surface itself, and you could do something about it.

MIKE: Do you know all of these people, that is --

WILL: Every last one of them. Every last one of them.

MIKE: My question is, do you personally know these people within the broader organization today? And if so, that means that you're not talking about moving somebody somewhere where they don't know anybody. You're talking about a subunit of the company that's cohesive enough that you actually know all these people.

WILL: It would be stupid to just drop somebody, just airdrop somebody in by a parachute in some situation where they could do no...you know what I mean? I mean, I'm not dumb. I understand that you can't just be like, “Hey, you're doing Haskell today.” Like, yeah, yeah [laughter], you're DevOps this quarter. Get to work.” You know [chuckles], that's not how it works.

DAVE: I've seen it work, and I've also seen it fail spectacularly, where we had a system recently where we swapped an engineer out to help another team. And that other team's process was very, very different, and it was very rigid. And they were under the gun. They could not stop and say, “We need you to do this, do this. Everything you're doing, we need you to do it sideways because we will do layer cake instead of,” whatever, that kind of thing. At the same time --

WILL: Oh, you have a bad manager. What a wonderful thing to know about.

DAVE: Right? And I understood, like, why. Like, all the intentions were good, right? It's like we had a problem. We need to throw more bodies at it, right? Conversely, when I was at CMM, and they were still a unicorn, we came up with a thing that...I called it hostage exchange program, where we would go to your team and say, “Give us a developer. We'll give you one. You give us one.”

And so, we would swap with another team, like, all the way across the building, completely different project, different vertical of the company, different profit centers. And we would swap out one person for two weeks. And very importantly, it was seed corn. It was, we're here to cross-pollinate you, to make friends, and show you how we work. We are not here to sweat you and try to get you to scream everything over the line. Because when you go to a new thing, you're going to be useless, right? But after the exchange, because we had one of yours, you had one of ours.

And then when you swap back, this amazing thing happens because it's like, “Oh, man, we need to cycle the pods on the Jenkins thing. And how are we going to do that?” I don't know. That team doesn't like to talk to us. And the one guy who was in hostage exchange goes, “Oh, hang on. I'll just ask Justin,” right? And all of a sudden, there's a personal relationship formed. And that was like, if you forgot everything you learned while you were on that team and you just remembered who you met, it's a victory, hands down.

WILL: Exactly. No, absolutely.

EDDY: Or better yet, Dave, you can just be like, “Oh, I've done that before. Let me go [laughs] and do it.”

DAVE: Yeah, yeah, hands down. Mike, we were talking about this on the team the day we had a deploy issue. And we have this really convenient, for those people listening out there, we have a really convenient way of deploying. There's a little thing that we click, and we get a notification that says, “If you want to deploy this, click here to deploy it,” and it's great. And you don't have to go into Jenkins and fiddle with the drop boxes and find your image and all that stuff. And I kind of forgot how to do it manually.

And so, we had an incident last week where we were like, you need to roll this back. And it wasn't me, thank goodness. But I'm just kind of on the sideline eating popcorn going, you know, if that had been me, I would have been stuck because I have forgotten how to do it the hard way. And I’m like, we need to get the team together, and we need to review how to do it the hard way. So, like you said, so if something happens, you don't have to call somebody who knows how to do it. Just, like, everyone is just like, “Oh, wait a minute, I know this.” And if you've got a process to document, you're like, I know where to look for it in the wiki, and then all my notes will be there, and yeah.

WILL: Absolutely. At the large retailer that I work for, you know, they have an unofficial tradition, 52 [inaudible 23:14] pickup, Q1 of every year, where they just, like, completely demolish the org chart, and it's anarchy. And I don't think they do it on any kind of, like, high-minded management, you know, theory principles. They just do it.

But one of the nice things, like, having been here for a couple of years and gone through several purges, is I've got people all over the company. Anything I need done, like, I've got a person. I’m like, oh, I'll take Dave, or I'll take Reed, or I'll take Max, or I'll take whoever. And I have somebody to ask to get a thing done, and it makes me really effective.

DAVE: There are people in your organization who will try to shut that down, thinking they're doing a good thing because once a team starts...like, when IT starts getting overloaded, they start saying, “We need everyone to fill out a ticket on Jira because we can't manage our workload,” right? And if you know somebody and you pick up and you...I'm just going to call Corey. He'll take care of this for me. And, like, Corey has to say, “I need you to fill out a ticket, man.” But you don't have to kill either of those, right? You can still do the, “Hey, I need this,” and your friend in IT can go, “Oh yeah, just check the PRAM, just reboot it, do the thing, you know.” “Oh yeah, I remember that.”

And then if it's going to be more, it's like, you know what? You're right. That's serious. Fill out a ticket, and I'll jump on it. So, you can still get the best of both worlds, but you can have people that are simple-minded of, like, just do it this way. And like you said, a virtue taken to an extreme can turn into a vice.

MIKE: So, we've talked a lot about swapping people around and getting them into different experiences. And I heard a couple of numbers thrown out. Somebody said 10. Oh, you shouldn't have more than 10. Somebody said 6. There may not be a clear line there. But what is that number X? And maybe, more importantly, why is there a maximum team size? Why does it stop working?

DAVE: In the pre-call, Mike, you posted a link to Dunbar's number on the wiki. Real quickly, for people listening, the size of the frontal cortex of a primate's brain determines how many meaningful relationships that primate can have at any one time in their life, and, for humans, it's about 150. I wanted to throw that one out there because I recently learned of another one called optimal tribe or optimal primal group...I don't know if it has a clear name, but the way they calculate it is fantastic. Anthropologists have determined that when you have this tribe of, like, 150 people, you're spread out over the valley, and you actually have subgroups within it.

You might be in a group of, like, 500 people, but you're spread out over about...150 of those are your relationships. But the people that you rub shoulders with every single day of your life is more like 30 to 35. And this number self-balances because among humans, somewhere between 30 and 35, the murder rate balances the birth rate. I love that number, that it's, literally, I will kill you if you add one more person to this team is how we dealt with this 10,000 years ago, and I think that's fantastic. I think it's brilliant.

[laughter]

JUSTIN: So, I don't want to be on David's team [laughter].

DAVE: Yeah, because we've got 34 right now.

WILL: I don't think we're going to see deaths necessarily.

DAVE: No, no, no, not where you can see ‘em.

MIKE: [laughs]

WILL: I think what you do see is you run into the bureaucracy associated with creating and processing and assigning work, you know what I mean, like, breaks down. That's what happens. You can't effectively utilize your people because you have this sort of, like, feast or famine, these wild swings and the amount of work you've got to do. Like, I'm getting ready. I'm warming up for a giant work hangover next week because we've got a big thing we're shipping out.

And all of our leadership management capacity has been dedicated to getting this thing over the line. And it's a big fella, so I forgive them. But we're going to have a giant night hangover, where there's just nothing in the pipeline, and then they're like, “Oh, uh, do this.” So, that's where you break down, like, the ability to assign and manage and, like, hey, provide feedback. All that stuff it breaks down. You just can't [inaudible 27:44]

JUSTIN: So, can you have a larger team if you have people helping lead it? And by this I mean you have project managers who are tracking the projects that are out there. You have your tech lead. You have your engineering manager. So, how does that all fit together? And then, you actually have the people doing the work of which, hopefully, your tech lead is one. And then, you got the other, you know, senior, regular, and junior engineers.

WILL: I think there's a finite amount of leadership and, like, I don't know, man. I mean, sometimes that's the project managers. Sometimes that's them but not all that often.

MIKE: If you have really good leadership of the kind you describe in sufficient quantity, I think you can push above maybe that 10 limit. Maybe you can get close to 15, pushing 20, but it is continuous work. And those leaders are going to be just incredibly busy. I mean, that's going to take a finely oiled machine, and it's really hard to replicate. They talk about this in “Team Topologies,” yeah, you can maybe go over 10. And you might be able to pull it off for a while, but if anything goes wrong, the whole thing falls apart.

JUSTIN: Wasn't it the two-pizza rule? You can only have a team that you can use two pizzas to feed. That’s your team limit.

DAVE: That's right. So, 200 people if we really sweat them and give them the [inaudible 29:11]

EDDY: What if I eat a whole pizza, Justin? Like that’s -- [laughs]

DAVE: Then the other nine people split the other one, yeah [laughter].

MIKE: When I was in high school, I was running cross country and growing like a weed. I could down two extra-large pizzas myself [laughs].

DAVE: I grew up in Ranch Country, and there was a phrase that people would say that, “I have teenage boys. I'd rather clothe them than feed them [laughter] because it's cheaper.” So, the number with the group size, the thing that they found out of this was...because humans, we've obviously gone past the point where we can be around 35 people without committing a murder. And they see anthropologically, in the records from about 15 to 10,000 years ago, three things started happening. We started dancing. We started singing, and we started worshiping together as groups.

And what this does is it gives you the ability to walk into a room and meet a stranger, you know, stranger danger, we're afraid. Like, you know, primates, we're just like, [vocalization], you know, do I need to worry about you? But we've got this thing that we're all looking at, and I’m like, oh, wait a minute, we're all here to worship at the same time. Or here, we're all here to sing and dance and experience this pleasant experience in a safe environment. I can call you my tribe. I can call you my friend. And from there, we then get scaled all the way up to, you know, patriotism for a country.

I used to hate, hate with a passion, town hall meetings where the CEO comes down and tells you about the corporate vision. And now I realize it's absolutely essential. You can have 25 people on a team if you all are looking...like, the interpersonal team communication is going to go exponential, right? It's a triangle number like n times n plus 1 divided by 2, something like that.

The links between the number of nodes goes up exponentially, and that will fry and become exponentially difficult to manage. But for a short time, you can exceed that number if all of you are looking in the same direction, if you're trying to pursue, you know, common gods, common goals, common enemies, that sort of thing. You're like, okay, yeah, I can pick up and pull in the traces next to you without knowing your life story.

WILL: You must be going to different town hall meetings than I am.

MIKE: [laughs]

DAVE: I am a different person in town hall meetings than I was two years ago. And I would say I'm going to different town hall meetings than I was two years ago. And, yes, I've been at Acima for more than two years. So, what I'm saying is I have changed, and the experience of the meetings is different now, yeah.

WILL: I actually see most of the corporate town hall meetings that I have witnessed are actually like a case study in, like, this isn't working anymore. This is no longer anything. Because my experience with the meetings has been, typically, that we're not on the same team, like, anymore, like, the C-level executive that I'm meeting and me as like a, you know, a skilled worker, certainly, and somebody who's like, you know, I punch above my weight in terms of impact. But we can't have an honest conversation. They're not doing that with me anymore.

I am a resource to be managed and not really dealt with forthrightly. It's too big. We're too far away. Our interests have also rather sharply diverged. And so, there's a lot of, like, I don't know, man. They're lying, and they know they're lying. I know they're lying. We're all sort of like going through this theatrical display, you know what I mean?

DAVE: 100%. 100%.

WILL: I'm just like, all right, guys. We're not that. We're not that. We're not on the same tribe anymore. Don't put my face in it, okay? Like, I could listen to the investor call if I want to know what you're doing to justify your share price. If we need to do that, that's okay.

DAVE: The corollary that I take from that is that there's times when I sit in church, and I'm like, we are all here to do the same thing. And then, there's times when I remember a...my dad had nothing to do with church or religion for most of his life. And it was because he would sit, and his experience was, you'll shake my hand on Sunday and stab me in the back on Monday.

And if you've ever had that experience of sitting in church-going, you SOBs are all miserable jerks, and not one of you believes in what we're doing, that's a horrible town hall to be in, right? It's like, oh, yeah, yeah, you're not trying to incentivize us. You're trying to sucker us into doing work now in exchange for a promise for something you're never going to deliver, right? When there's no faith in that, it's brutal, right? It's the opposite of it.

WILL: Well, I mean, I don't have a problem with working for Corporate America. It's fine. I'm not some guy who's [inaudible 34:02] is communist, you know what I mean? Let's all get paid. Let's all make really cool stuff for our customers and delight them. And then, they will give, you know, us some of our money, and then I will get my piece of that. You'll get your piece of that. We're all going to be good. But that's not a forthright conversation that we're having.

DAVE: No, it's not.

WILL: It's like big meetings, right? Big meetings, big companies. Like, it just gets so big that you're not really...the gap's too wide.

MIKE: Well, we're talking, I mean, this is germane to the conversation because we're talking about ways to maintain group cohesion. And you're saying, as you get large, you get to town hall size, it is really hard, that getting everybody on the same page and making them happy about it may just utterly fail. So, if those tactics are challenging with a large group, it says something about the team size, right? And if you look at hierarchy of organizations, there usually is a hierarchy.

WILL: Sure.

MIKE: And at any level of the hierarchy, you have, you know, probably that six to, you know, 12 or something people at that level so that everybody is within a similarly sized team because it's really hard. It's really, really hard when you try to scale that up.

JUSTIN: So, going back to your talk, I mean, your chat about, you know, getting everybody aligned in the same direction, I can't help but think about the example of Twitter, X. And say what you will about Elon Musk. He forced his way to everybody be in the same direction, and I am very curious to know what impact that had on their team sizes. Because I'm like [laughs], did they...

EDDY: Correct me if I'm wrong; it might be inflated, but I think over half of the team left when he forced his way.

JUSTIN: I think so.

WILL: 70%, yeah.

EDDY: It was quite significant.

JUSTIN: And so, do they have smaller teams, or do they have larger teams with fewer managers? I bet that's what happened. My theory is that Musk got rid of most of the middle management, and they have the engineers left who are interested in working for Musk and are probably well-compensated as well.

WILL: [laughs]

JUSTIN: And so, they're all aligned somehow. But, I don't know, that is going to be an interesting case study in some future, you know, business school or engineering school.

WILL: I’d imagine you're right. I’d bet you any sum of money the team sizes are larger. And I think probably you got a lot of people on a visa who aren't empowered to just go anywhere. And I think they're probably making less money and working harder, and maybe just didn't have the option to hop over to Amazon, or Microsoft, or Google, or whatever. We'll see. We'll see.

I think there are a lot of people who are leveraging tech winter right now. And whether that remains to be a sustainable management practice or not is going to be told, you know, when tech starts to boom again because nobody got off their phones, man.

MIKE: [laughs]

WILL: And then, the market for talent becomes a little more competitive. I think there's a lot of people who, you know, it's your way in this, like, rather sharp and prolonged economic downturn for our slice of the economy but long term, I don't know, man. I don't know. Is it good management? We'll see, you know. Sorry, that's a little bit of a tangent, I think.

MIKE: Yeah, no, it still comes down to, you know, how do you work with the group? How do you keep the group cohesive, and the challenges with that? So, is 10 a reasonable cap? Like, if you got a team bigger than ten, you better look at cutting it?

JUSTIN: I don't think I've ever been on a team that large. I think the most I've been on is...well, I guess it depends on how you define it because I've been on front-end teams where we have the product owner and then the designer. And then, I was a full-stack engineer, so I helped the frontend and the backend, and the database [laughs], but there were also people on each of those. And so, I guess we were, you know, if you include the designer and maybe a database guy, I don't know, it's like, you know, hey, we're together for a project. But I don't think it ever got up to 10 in our standups. I think the max number of people in our standup was, like, seven or eight.

DAVE: I'm hesitant to say a number. I worked at Evans & Sutherland around the turn of the millennium, and I was on a team of 50, 25 software engineers, 25 hardware engineers. Basically, we'd come up with the same idea that NVIDIA had, that you could do 3D in hardware and do it very, very fast. You could do stuff in real time that couldn't be done at that point, and, it worked really, really well.

The 25 people, the status meetings got [inaudible 39:00], but those went on and on and on and on, right? It took forever to get through the status stuff. And, yeah, subdivided, and, like, I was literally...my sub-team was in the middle of a cube farm, but, like, my job was 2D-bit blit block image transfer. My job was to move rectangles from one place in memory to another. And there was somebody else whose job was to handle 3D and somebody's for curves. Literally, we had a triangle team, like, literally. But at 25 people, it functioned.

But what I will also say is I was on version 3. They had built the entire graphics card for real image 1, real image 2. I was on the Real Image 3,000 team for making version 3. And we also did largely waterfall because there was hardware coming through the pipe that had to be anticipated 18 months in advance. And the rule for waterfall is don't do it unless you've built the exact same thing twice before, which they had. And so, there were a lot of stuff that, you know, that had happened in that.

WILL: Right now, like, my daily standup has 18 people in it.

DAVE: How long are your stand-up meetings?

WILL: 30 minutes.

DAVE: 30 minutes? That's a status meeting, yeah.

WILL: Eh, it's okay. Thirty minutes is fine. They're pretty efficient. They tend to go over fairly regularly, but not because of the status meeting being bad, just because we're just getting ready for a big release, so there's a lot of nuts and bolts to make sure everything goes right.

And I would say that the team is functional. And it is really kind of one team, but there's a manager, and then there's sort of an assistant manager. And there are some, you know, pretty senior, you know, sort of shadow management. Like, we have at least three people that could and do act as managers, like de facto managers. And so, it's big, and it works. And it works as, like, a single team, but there's a lot. It isn't just a guy. There's a lot of support.

MIKE: So, proof. There's an example case of all the support you can push up close to 20 and make it work.

DAVE: For a while.

WILL: But there's like, well, yeah, but there's also at least three people who are acting, like, two named and one unnamed people who are doing that job. And if you took any one of them out, the whole system would collapse. And so, you think like, oh, we've got 18 people on one team. But it's like, yeah, but you've got three managers, so that's, like, three six-person teams kind of in one thing. It would be hard to make that a sustainable thing other than just sort of a special case where it's like, if you have the right mix of people, you can make it go.

DAVE: There's something going on right now where I live and work. And, Eddy, I think, Eddy, you're in Michael's team, right? I think I see you in the standups. Maybe not. I'm on a team of 28. Well, I'm on 4 teams that total 28 people. And shout out to Michael McMullin. I don't know what he's done behind the scenes, but our standups are three minutes long. Like, I'm not kidding. Like, we walk in --

WILL: Three minutes long?

DAVE: Yeah, literally just long enough to...is everybody here? Does anybody have any blockers? And, like, we don't even do the here's what I'm working on, and here's what I need to. And part of that is because we have a channel for that. Mike, you'll be happy to hear this. I have made some impassioned pleas against using a Slack channel for standups. And there's times when that has been completely opaque to me. And when I first started, it was very transparent what everybody was working on, and it's back to that.

I'm back to being able to see what everybody is doing, and because of that, I walk into the meeting prepared to leave the meeting. It's like I kind of know where everybody's at, and we walk in, and then one person will go, “Actually, I’ve got...” and they’ll raise a blocker, and then we'll talk about the blocker for five minutes. I think the longest a meeting has gone is, like, 12 minutes.

I don't know if Michael has a gypsy woman prisoner in his basement, and she's just forced to hex everybody else and not him. I don't know how he's doing it, but it's amazing. It's very exciting to watch. I love it when something miraculous can happen and I'm like, how does that work? Like, I genuinely don't know. But it works --

WILL: I mean, I think it’s working by, like, not doing. I haven't been to the meeting. I can't speak to it. I won't do it. But like you said, like, oh, it's a three-minute meeting. I'm like, then why have the meeting?

DAVE: Well, because sometimes if there's...the reason it was three minutes was there were no blockers that day. Usually, the meeting is like 5, 10, 15, but not half an hour. Like, it’s scheduled for --

WILL: And how many people are in this meeting? How many people are on it?

DAVE: 28.

WILL: 28? Forget it. Like, get rid of the meeting. I’m sorry, like, you're not doing anything. And, I mean, it's great, like, I don't know, if you don't [inaudible 44:21]

DAVE: Hmm, I disagree.

WILL: With the daily standup, that’s valid. But why even have it? Why even have it? If you're not going to do it, then don't do it. Figure out another way, and that’s okay.

DAVE: We could stop having that meeting and see what breaks, and I kind of don't want to because, like I said, it's magic right now. And I genuinely don't understand it. There's a book that I highly recommend called “Read This Before Our Next Meeting.”

WILL: I have to say the only thing I’ve heard is good is, like, it’s very short. That's the only benefit that I've heard described. And I don’t know how the team runs --

DAVE: Okay. Okay. Hang on. Cool. Yeah, let me answer the question that you're really asking then, which is, what's going on? There's a book called “Read This Before Our Next Meeting,” where you basically say, “Everyone, this is the objective of the meeting. Come prepared to reach that objective,” and that's the difference. We're not wandering in going, who's doing what, and where are we? We're not looking for a status on anybody. Everybody knows everybody else's status. So, that part of the meeting doesn't need to happen. You're right, that part of the meeting doesn't happen.

But we do need to know if anybody's stuck, and we need to dogpile on something to save somebody from something. And if there's an issue, we dogpile on it, and we take care of it. And that's the point of that part of the stand up for us, yeah. So, the status part is being routed through another channel, and that's what saves us 27 minutes. Does that make sense?

WILL: I like to see people, man, like, I don't know. Like, that’s just me. I like to see people, and I like to look at their faces, and I like to look 'em in the eye and hear their voices and how things are going. And I get a lot out of that. And I don't think software engineering as a discipline is served by having more people typing into screens in isolation, generally speaking.

JUSTIN: So, Will, I think you bring up a good point, which is part of having a team is creating the culture that, you know, you want to engender within your company. And the act of meeting together and of interacting with each other is how you create that culture. Well, it's one of the aspects of creating that culture, right? Obviously, working with each other, time outside the meeting. But bringing it all in and kind of sharing a joke and doing, you know, interacting in some way, I think, is a great benefit to engendering a sense of teamwork, and so on. And so, that can be hard to do with a larger team. It's a lot easier to do with a smaller team; let me just put it that way.

WILL: I mean, if you have 28 people in a daily standup, like, I don't know how you could do it differently than Michael McMullin's doing it just because you just can't.

DAVE: You can't. It would turn into an hour-long status meeting. And to your point, though, we don't socialize. Well, we do. We all show up a couple of minutes early and then we get the water cooler stuff going on. And I don't count that as part of the official meeting, but I probably should because, yeah, there's some socialization that's essential. And it's good because it's cross-team because, like I said, there's 4 teams actually in there. And if you're going into this meeting expressly to socialize, and communicate status between people, and where are your projects, and that sort of thing, you're not going to get 28 people through in 3 minutes. Yeah, absolutely.

WILL: So, it's like 4 people in sort of a meeting. It's a four-person meeting. Well, I feel like 24 of those people don't need to be there. Like, 24 of those people are not supposed to be in that meeting.

DAVE: I'm hesitant to change the formula because it's working, and I don't know how [laughter].

WILL: You’re right. No, no, you're right. You’re right. Like, we agree, right? Like, don't do it, but at the same time, like, you've got four teams. You've got four managers who are supposed to be explicitly doing this job. And you have all these other people, what, holding their hand while they, you know what I mean?

DAVE: Yeah. See, we just came off of that. We just came off of a representational republic, and it was satan. It was the worst. The representative served as an information killer, as a communication deaddrop. And we had the team not knowing what the rest of the company was doing or why we were working on what we were doing. And our issue is we’re going up to this person and not going out...we finally started routing around them.

WILL: Oh, so you have a bad manager. What a wonderful thing to know about.

DAVE: That was said, not I, yeah. So...

WILL: Like, I don't even know who I'm dragging here, and I have no accountability whatsoever. But it sounds like somebody has a really important job, and they're blowing it.

DAVE: I won't say how recently before this team. I won't say how recently. I’ll just say that before I commit career suicide.

MIKE: Well [chuckles], you know, there's some aspects of this. We talked about some things to make a group meeting work, but still, you're talking about four teams, around seven, you know, six or seven. It just keeps on coming to that number. Yeah, you can get up to about six, and then that's it. So, you know, empirically [chuckles], this number keeps on returning [inaudible 49:18]

WILL: Okay, what's the smallest number? Let's go in the other direction. Wait, actually, no, hang on. We might not have time to get into this. But I am curious, like, where does the team no longer make sense? One person, you're not really a team, right?

DAVE: I mean, I have arguments with myself, but –-

WILL: Two people, why are you on your own team? Three people, have we seen a team, like, an effective functional unit with three people? It really becomes like, honestly, more, like, on an org chart level, like, does it make sense to give these people a manager, right? Are there enough workers for me to rationally advocate these people that are on manager? It's almost like can I justify the accounting line item, you know, more than anything else?

DAVE: I think it comes down to how much of your objective is explicit and static, where, an organization, you need a lot of that. You could have a team of two, just a partnership. And what direction do we want to go in? We don't need a lot of external, right? And now you can go after a goal, and that goal can change and adapt very, very quickly. If it's a much [inaudible 50:23] thing, then changing the goal is a lot more complicated. You've got to synchronize a lot more people.

WILL: I'm kind of curious to hear from Kyle. As DevOps, you guys are not necessarily always sort of as robustly staffed as, like, software engineers. What's the smallest DevOps team you've been on, like, one or two, two or three, one, just you by yourself? I feel like you feel that more than we necessarily would.

KYLE: Three was the smallest, and I thought that worked really well. Scaling up from that, there's quite a few DevOps tooling that you run into locking situations. So, I feel like you kind of start to develop sub-teams, which we kind of have as we've grown to 7, 10 people.

So, while I'm sitting here listening to you guys talk about all this, I'm just kind of thinking of team size, to me, I was thinking, well, the minimum and the maximum are kind of dependent on the pipe size, and it will depend on what pipe is coming in or what pipe’s coming out, so your work coming in, the channels that you have to get the work out. So, if you've got a build pipeline that backs up, maybe it's only six people, it makes sense. That 7th person is always just twiddling their thumbs because they're never ever actually able to deploy something, right?

Or, you know, maybe the pipe that's the smallest is the management pipe. The manager over here he can't manage 10 people. He can't manage he, she, it, can't manage eight people, right? Like, so, where's the pipe the smallest is kind of what I'm thinking of, just because I think of everything as systems and, you know, flow like that. I don't know if there is a magical number kind of like we brought up, and I'm wondering if it's more of that pipe situation, because, yeah, kind of like we've said, well, does it make sense to have a team of 1?

Well, if you've only got enough work for 1 person, it's a team of 1. That's your pipe. And, you know, if you don't have a manager, do you go past 3 people? Do you have, you know, the ability? Do you have the pipe to actually function as a team if you don't have somebody managing that fourth individual, or whatever you want to call it, to kind of scale that up [inaudible 52:44]?

DAVE: I think also it has to do with your organizational, like, you can have a team of 2 that aren't really a team because you're all just doing the same amount of work, right? The same kind of work. This is a weird quote out of nowhere, but General Schwarzkopf from Operation Desert Storm in 1991 said in the battles in Kuwait, they very quickly learned that the Kuwaitis were not capable of, how did he put it, they were not capable of putting together an attack above the brigade level. And that's very much a team organizational statement, right? And they were able to capitalize on that in that particular venue.

And I think about that in terms of team organization, right? Like, if you've just got 2 people, you don't have a brigade, right? And if you've got silos of 5-person teams in a 50-person team, you can't put together an attack above the brigade level, right? You're going to have that same problem.

MIKE: We could probably dig into cross-team communication, which is one of the most challenging things, if not the most, that we deal with. I think that came up...it was either in the pre-call or earlier because it is a huge challenge to deal with cross-team communication.

But I think that we've covered our topic pretty thoroughly. We've talked about some of these physics of scale, right? That once you get things bigger, the rules quit working, right? It quits applying. And that there's kind of a sweet spot, you know, the Goldilocks zone of somewhere around half of 10 [chuckles] that works really well and allows teams to collaborate well together.

DAVE: I like that our podcast is kind of a, well, here's the thing, and it's like, we're going talk about this, but there's all these adjacencies to it that affect and almost sometimes even govern, right? Because they become a dominant term.

MIKE: Well, we have to put maybe a pin in that one and come back to it later and talk about cross-team communication because, again, that is a challenge in [chuckles] our industry, probably not just in our industry, but universally crossing those boundaries without maybe literally killing each other [laughs] and making that work. But for here, we've kind of talked about the team size. Is there any final words people want to get out there before we end this?

WILL: I just want to apologize to all the managers that David threw under the bus.

[laughter]

DAVE: Yeah. They've got plenty of company under there. It's, yeah, it's...My mouth tastes like feet all the time, so... But --

EDDY: I would just make sure that you add a comment out loud to whoever edits and says, “Please leave my apology.” [laughter]

DAVE: Yes, please...so, I heard a great definition. The definition of a friend is somebody who listens to your BS, and then calls you on your BS, and then keeps listening to your BS. So, any time that you hear me dragging somebody, I swear to God, I'm not dragging them. Like, you'll know if I'm angry. And this is just me being happy, and sometimes stuff goes weird.

And yeah, I apologize if I'm dragging somebody inappropriately. I hope I dragged them appropriately; that's a very different thing. Because I'm going to go right back to listening to their BS because there's nobody that I want to get rid of, you know, so...some people I'd like to adjust. But I'm sure they'd like to see me pipe down and adjust as well. So, it's fair.

MIKE: [chuckles] Well, thank you, it's been a great discussion.

DAVE: Thank you…

MIKE: Until next time on the Acima Development Podcast.

In this episode of the Acima Developer Podcast, host Dave Brady and the panel, including Mike Challis, Kyle Archer, Eddy Lopez, and Will Archer, delve into a satirical discussion on how to ensure time estimates for software projects go wrong. They explore various humorous yet insightful approaches, such as using aggressive negotiation with engineers, ignoring edge cases, and focusing on points as extrinsic values rather than consistency. The conversation is full of playful examples, with stories from past experiences, all illustrating how poor communication, over-optimistic estimates, and leadership pressure can derail projects.

The team highlights how over-reliance on tools like Pivotal Tracker or Jira without considering the actual complexity of tasks can lead to skewed results. They also mock the practice of placing value on productivity metrics, like story points, while ignoring the consistency and accuracy of those numbers. Throughout the discussion, the panel pokes fun at the missteps that can occur when management overrides technical teams' decisions, fails to understand project requirements, or implements unrealistic planning methods, using examples from both personal and industry-wide experiences.

Towards the end, the group turns their satire to broader topics like testing practices, infrastructure management, and project launches. They humorously suggest that skipping QA, bolting on security post-launch, and involving high-level executives in day-to-day decisions can lead to even more dysfunction. Ultimately, the conversation serves as a playful critique of common issues in project management and development, with a clear undertone of how these behaviors should be avoided for the sake of successful project outcomes.

Transcript:

DAVE: Hello and welcome to the Acima Developer Podcast. I'm Dave Brady, and I'm hosting today. We've got a good panel today. We've got Mike Challis, Kyle Archer. We've got Eddy Lopez and Will Unverified. I don't think that's your...

WILL: It's Will Archer.

DAVE: It's Archer. Archer? Yeah. Welcome, Will. So, last week, we talked about some structure, and we thought we would carry over, and I think we're going to table that for a week. Today we came up with, actually, it was Will. You came up with a really interesting question phrased in kind of a devil's advocacy. Do you want to phrase that up for us and tee us up?

WILL: Well, so, if I wanted the worst possible time estimate for my project, how could I do it? How could I really screw up my time estimates?

DAVE: I like it. I may or may not have some stories related to this. I'm seeing knowing smiles around the board. Okay, so I'll give two stories really, really quick. And the second one is actually the counter to the first one.

Negotiate aggressively with your subject matter experts in a threatening manner. Specifically, negotiate story points down because you don't want to spend extra time dealing with the problems that your people are thinking of that might be coming down the pipe. And a great way to do it and sound like you're not doing it is to begin every sentence with, "Why don't you just...?"

And so, you can take something really super-duper complicated...I mean, throwing the ring into Mount Doom that's a one-point story. How long does it take you to drop a piece of jewelry into lava [laughter]? It's super easy, right? Super-duper easy. Why don't you just do that? It's fine. And yeah, it's that kind of thing, right? Where the actual thing, you know, getting to the site to deliver it, you know, OSHA injuries where he gets his finger bitten off, right? That whole thing.

WILL: Take the Eagles, guys! It's [crosstalk 02:00] a one-hour flight. Take the Eagles, duh.

DAVE: Yes.

EDDY: I was going to say, don't you like the idea of having an extra cushion to your points?

DAVE: [inaudible 02:08]

EDDY: Yeah, because if you over-deliver and you get it done, like, two or three weeks ahead of time, the only thing that you get is just applause, right? Like, you got it done earlier.

DAVE: Well, so, I think this is kind of the other way around, right, where it's like dropping this in. We're going to budget two hours for this, which is plenty of time to drop a piece of jewelry into lava. And the project, as we know, is going to end up taking three months from, you know, hindsight. And that's the kind of stuff where...like, engineers often get paid to think about edge cases, and weird things, and, you know, what about this thing, and da da da da da.

And very, very smart people who are low in trait openness...openness is one of the big five psychological traits for gauging a personality. People who are low on this are very certain they're the J type on the Myers Briggs. They're judgy, not perceptive. What's in their head, they will press that down on the world around them.

And this leader that I worked with would negotiate aggressively, and because he was the CTO, every discussion was an uphill push, so, okay. And we used Pivotal Tracker, and Pivotal didn't care. Tracker doesn't care what you think your velocity is. It will tell you what your velocity is. And because the stories were getting smaller and smaller numbers on them, all of Lord of the Rings became a one-point story, which means it took us three months to deliver one point of value.

MIKE: Wow.

DAVE: And he did not like that. We literally watched Tracker go sub-integer. We had a velocity of 0.2 at one point. And that did not make the CTO happy either because he wanted the points to be big, which might be another evil thing is, like, place extrinsic value on points.

The counter to it is the second story, which I'll just tell in just one quick sentence. I had a manager who looked at me, and he said, "I don't care what your velocity is. I don't care what the number is. If it's consistent, I can bank on it." If the team velocity is 180, that's great. If it's accurate, I'm going to ship on time, and I'm happy. If the team velocity is 1.8, that's fine. Points don't matter, remember? And if I can bank on you shipping on this day, and I can queue up the next thing, I'm happy." So, if you want to mess up your schedule, focus on the points themselves as extrinsic things, and ignore consistency in your estimates.

EDDY: So, Dave, for the people that may not be aware, what's Tracker? Can you give an overview?

DAVE: Pivotal is a company that makes software, and Tracker is their time-tracking thing. It's Jira, or VersionOne, or Rally. These are all competitors. Jira is the really big gorilla kind of taking over the scene. Basically, you do this, deliver it, and then it goes into the backlog, and you put points on it. And then, Tracker gives you your to-do list against a calendar. It's like, no, no, this is your...I've been measuring your velocity. This is how many stories you have. This is when you're going to be done. And after you've been using it for about three or four months, it starts getting spookily accurate. And if you don't like what it says, you can change the numbers to be wrong.

MIKE: I had a couple of stories I was thinking of as you were sharing them [chuckles], but I'm going to go a different way. I mean, we were talking in the pre-call about the first line in Anna Karenina, which is all happy families resemble one another. Every unhappy family is unhappy in its own way. Like, there's many ways to ruin your estimate [laughs].

And I saw two kind of opposing examples. In one, product came to our team, and they had a big, international expansion project, and they had been burned by some optimistic estimates in the past. I said, "Okay, what's the worst case that can happen?" And so, the team talked about all of the worst cases and all of the unknowns we had [chuckles] and everything that could go wrong.

And we got an estimate that it was going to be this huge project that was going to take, I don't know, like, eight months or something. And then, the estimate was so big that management decided to...they said, "Oh, that's just...that's crazy." And they quit trusting our software. And this is, like, new management at the time. And said, "We're just going to outsource this." And when they looked at what was going to be involved in outsourcing and how expensive it was, we said, "Well, no, we can actually do it a lot faster than that." Ended up actually getting the project done in 2 or 3 months, way below what the previous estimates were.

But, you know [chuckles], there was, like, a year that that project didn't get worked on because nobody wanted to invest that huge amount. And engineers actually didn't even know that that was going on. We just did what we were asked to do. It was the worst case, so we gave it to you. And, of course, the worst case is not the average case. It's going to be some of the cases. So, you want to probably aim high, but that one failed.

And then, on the flip side, we did some annual planning and didn't have a whole lot of information. So, we planned out a lot of stuff quickly [chuckles], and then somebody went through it and said, "No, I don't think it actually will take that long." And they just chopped the number of our estimates in half [laughs] without any input whatsoever. And not surprisingly, a project that had that happened to has gone about double budget because when that project finally finished, it was about double budget because the earlier estimates were actually pretty good because we compared it with previous work and matched. It can go either way. If you don't trust the past experience, a similar future experience, then you're probably going to have some bad, bad, bad numbers.

DAVE: There's another symptom in what you said that I think is really interesting, which was management sent people off to go make a decision. They didn't have a discussion. They said, "Go away, discuss and decide, and then give us the decision." And then, out of context with no dependencies, or corollaries, or caveats, they're just like, "Here's the number." And because they have completely different assumptions, they made a completely different decision than they might have had they been in the room saying, "Well, what about this? Well, what if we cut this feature, or what if we brought in contractors, or what if..." you know, what if. There's no what if. It's just, this is how it will be, and it's unqualified. This is how it will always be in every universe.

MIKE: So, first key to give a really bad estimate is give numbers with no context whatsoever [laughs]?

DAVE: Mm-hmm. Mm-hmm.

EDDY: Well, I do want to say, just because the idea sounds easy doesn't mean the implementation will be.

MIKE: Well, that's an interesting one as well. Have you all ever been in a situation where product did the estimate and then told you what the estimate was?

WILL: I tell you, man, like, I think a really good idea to get these estimates really messed up...so, what you got to do, right? Engineering is so expensive, right? It's really expensive. And so, really, you got to have a business case for it. You got to have a business case for it, and this case is dependent on impact and timelines, how much money.

And so, really what you got to do is you got to go to the very highest level, like, the business folks with the spreadsheets. And you got to present your case for your engineering idea, your engineering idea to them, right? And you really got to get them, like, you have to get them aligned with sort of an outcome before you even talk to engineering because, like, you know what I mean, it's such a...you got to justify this line item expense, right?

So, you really got to go from the business case, top, top, top-level approval of budgets, and then you go down to engineering. Because, as we know, if something went wrong, it's always easier to piss off the C-level executive because they're really, I mean, they're disciplined, emotionally intelligent. They know all of that stuff. And it's easier to piss off your CEO, who maybe made some promises, even to the board, to the public markets, to all that stuff, rather than line-level engineers that might know things that you don't know. Like, as high as you can go, that is just absolutely critical.

DAVE: There's a phrase that weirdly has come up, like, three times in conversations for me this week. And so, this is number four, which is people are sometimes blind to the fact that if you have a problem and you start percolating it up the chain, your team lead can't help you with it. So, you go to the, you know, the department head, and they can't help you with it. So, you go to the engineering.

Once you're about three levels up, it is very, very hard for that person to solve your problem. It is much easier for them to solve you. And it's the same kind of thing, right? It's like, if I go to the CEO of Rent-A-Center and say, "I need this," he's going to say, "Stop needing that. Go away," right? Because that's the fastest way to solve that, especially if he's got five layers of people that he trusts, which you're going to do if you're in the C-suite. So, whether they're trustworthy or not, you've built them to trust. So, yeah.

MIKE: Will touched on something, the person on the ground who's familiar with the system. A great way to have a really out-of-whack estimate is to completely ignore what they say [laughs]. It's very easy if you're in a leadership position to say, "Well, I know what I'm talking about." You get inflated ego. It's a malady that has been present throughout history [laughs] that people will think, if I'm in charge, that means I know what I'm talking about. And a great way to get to get the estimate that you want to hear is to ignore what people are telling you and just don't even ask, right? Like, well, you know, this should be [chuckles], take the eagles, right? There's an easy way to get to Mordor.

DAVE: The thing that I'm hearing a lot is, like, committing to the map instead of to the terrain. We see this a lot. There's a specific case that I will tell people. Sorry, I need to phrase this in the evil way, right? You should make your decisions as early as possible and stick to them because, later on, you're going to get more information, and that's just going to annoy you. So, make your decisions at the point of minimum information. Never, ever, ever defer a decision, even if making it now would be hard. That's a good one.

WILL: Ooh, I got one, just to pile on. Hard launch everything. It's so cool. You have this grand unveiling, right? I mean, it's like you're showing somebody a finished product. They can get excited rather than a bunch of Tiki-tac incremental stuff you can't hardly do a press release on it. Hard launch, always hard, hard, hard launch, big unveiling. Think Apple, you know what I mean? Like, just the biggest possible spectacle. That's the way to do it.

MIKE: Nailed it. Nailed it. And let me take that a step further. You know you're going to have that big launch. You better plan everything you're going to launch on day one. Make sure you have all of your plans up front.

DAVE: Waterfall. Yep.

MIKE: Plan that --

DAVE: Don't start designing anything until you have planned everything.

MIKE: Everything. Yeah. Absolutely. And follow that plan to the T. Bonus points if you can get out ahead, you know, that means that you're really good at planning. So, if you've got, like, a two-year plan, make sure to do all that planning upfront.

WILL: And the people who did the plan should really be in charge of the estimates, too. And when you give those estimates, right? So, you lay out, this is the product; this is the timeline; this is our hard launch. Get all that stuff done. And when you present it, you should present it with a timeline, right? And this is really important, right? It's like the intuitive mind, right? When you present the thing, you want people's gut instinctual reaction, preferably in a public meeting with everybody there.

So, you just want to get people right around the table. And you want to be, like, you just want to shoot from the hip, boom. First thought, best thought. You want that creativity. And you want people...like, the accountability is critical, right? So, you got to make sure that you say like, "I think this is two months. What do you think?" so that people feel empowered to speak up in front of their peers and say, "I'm going to disappoint you and your boss. I'm not going to be pressured. I'm going to give you the real estimate because this is an art, not a science." You got to let people just go with their feelings. Let it flow.

MIKE: The more public, the better, right?

WILL: Absolutely. Absolutely. It's their time to shine, right? So, the bigger the hats you can get in the room, you know, get a CTO in there. Get some VPs in there, really, like, you know, elevate your team so that they can show what they're capable of because everybody wants to move up. And you don't get to see these guys all the time, right? So, they can really get a feeling for your ability to say, "Yes [laughs]."

DAVE: 100%. I had a scrum master that I worked for 15 years ago who loved that step 11 is the commitment. The team must commit. And we had all these blank checks. We had tickets that had been pointed that literally were, "Click to add description." Stop me when you've heard me bang on about that one, right? And it was like, no, we've got this many points in the sprint. Does the team commit to this? And I said, "No. The team commits to work as hard as we can and give it our best level effort, but this is the plan, and it's not going to go that way."

And I got a lot of pushback. So, make sure you push back on the people like that. Call them disloyal. Make sure that they do not feel psychologically safe pushing back on that [laughter], and whatever you do, never, ever, ever consider the possibility that you have not refined tickets well enough to commit to making it work.

EDDY: I do want to say that some of the things that I've noticed in my two years or so is that it's really easy to overestimate how fast a project is going to get completed based on the number of cooks you have cooking, right? Because you could add more talent, but you spend more time bringing that talent up to speed than it would take to have people who already know the context to just do it themselves. So, in a sense, sometimes it slows down the process by adding more people who don't have the context to do it. So, from experience, anyways, it's like, help is great, but you got to be smart about it.

MIKE: Well, no, throw more people at it. It'll always speed it up, right? Like, a good example is pregnancy.

DAVE: Brook's law.

MIKE: Nine women, you can have a baby in one month.

DAVE: One month. There was...I worked at...

EDDY: I think someone told me, as a response, sorry, Dave. But I think someone told me --

DAVE: It's okay.

EDDY: They're like, "Oh yeah, you can't have nine women cook one baby quicker." And I'm like, okay, yeah, but the retort was, "Yeah, but what if you impregnate all nine women? Suddenly, now you have [laughs]" –-

DAVE: Parallel development, yeah [laughter]. I worked at Acclaim Entertainment, which was not my proudest moment. They're not a great company. But we were working on a video game, and they always get into crunch time as the Christmas season approaches because you got to get your games manufactured to be on the stores before Black Friday and that sort of thing.

And they had brought in a whole bunch of contractors from our London studio, flew them over to Salt Lake, and literally just throwing bodies at the problem. And this was when I read Brooks. And so, I wrote, "How long does it take to make a baby if you assign nine women to the problem?" And I came back the next day, and somebody had written, "Two weeks, because we're going to hire nine more contractors."

[laughter]

WILL: I'll tell you what's another real needle mover in terms of team efficiency is, when the deadline starts to slip, the more people you can get on a project asking for status reports, and timelines, and when that's supposed to be [laughter], it motivates. It allows team members to have that sort of clarity of purpose because they know how important it is. Otherwise, I mean, developers, I got to be honest with you; sometimes, when we're behind on a project, we can kind of drift. We start taking long lunches. We go for walks around the office, playing a lot of ping pong.

Without that focus, you could kind of...honestly, sometimes we just forget that we're behind on these commitments. And sort of if you can have somebody every hour, every 30 minutes or so really getting the status reports, it's such a weight off everybody's mind. I forgot. I was like, "What? Oh my God. Stand-up was two hours ago." I already forgot that this project is way behind, like; oh, thank goodness. I could have wasted the entire day not updating you on where the thing was just because, like, oopsie [chuckles], what was I doing?

KYLE: I think having as many of those status updates as meetings that really speeds things up, too. So, make sure that all of those are meetings and scheduled throughout the day.

MIKE: And you want everybody to be aware of it. So, make sure to bring the whole staff in, right?

KYLE: Yeah. The entire staff.

EDDY: I was going to say, make sure everyone who's able to attend to attend; that way, it's more efficient.

WILL: The really critical people, the people you've got to have even more so than product managers who...I know they're coordinating the tickets, but I'd say the people you really, really need to get in those meetings, as many meetings as possible, is the senior technical leaders like your senior engineers, your team leads, the people who are the people who really know the critical bottleneck issues that need to get plugged to enable maybe the junior level team to do work. It's like, you got to know what's really, really going on, and you got to have technical expertise to get those real serious moment-by-moment updates.

DAVE: Yes, if you're going to clean the supply closet in a hospital, make certain that every surgeon in the building is required to be in that meeting, every single one of them.

WILL: Are we going too dark? I feel like we might be going too dark [laughs].

DAVE: I'm here for it, but yeah.

[laughter]

EDDY: I do want to also just say, someone who didn't pick up on it, just for our listener out there, all that was satire, by the way [laughs].

DAVE: Oh yeah, this is a sarcastic episode.

EDDY: [laughs] Please don't take this to heart.

DAVE: Are you being sarcastic? No [laughs]! Yeah [laughs], good times. I think we touched on this a little bit, but make sure your businesspeople are making technical decisions. Make sure your technical people are making business decisions. This is important. If you're at the keyboard typing away, writing some software and that last function just won't work because of this problematic workaround to it, just cut the feature. It's too hard to do; just change the feature to the way you want. If it doesn't comply with some stupid law, whatever, that's somebody else's problem. Businesspeople, at the same time, don't tell people how much stuff is worth. Insist on that you know the price. And we had talked about that. Make sure you're telling people what their estimates are, so...

WILL: I will say, I think, it's really important to make sure you go up the entire chain to product if you want to do something like what color should this button be, or the copy should it be, like, select, or okay, or see all. I'm going to need to check in with marketing, sales. I can't make that decision on my own. I'm just an engineer. I got to have sign-off from VP of marketing on what the okay button text should be. What is the design thing? Should it be white or off-white? I don't know. These are critical questions. Completely above my pay grade. I can't use common sense. I'm an engineer. I'm half autistic, you know what I mean [laughs]?

DAVE: I have worked at a shop where the art team would hand us comps, and they had to be pixel-perfect, or they would just stop the whole team. It's like, "No, no, you rounded this button with a three-point fillet, and I specifically specified a 3.5." Yeah, I can slice pixels. I don't like it, but I can do it because I had to do it.

MIKE: Well, if you really want to make sure your project gets done on time, you don't want to ever have your developers idle. So, make sure you get them started working on the project immediately while you're gathering the requirements.

DAVE: Maximize utilization. Absolutely.

MIKE: Absolutely.

DAVE: Absolutely.

MIKE: It might take you several months to get the requirements for the project. So, you need to make sure that they're really busy working so that when you get those requirements, they already have most of the work done.

EDDY: You know you're being efficient when you're reverting a bunch of the work you're doing because you're releasing a lot of code.

DAVE: I may have put that on a pull request this week. I love it when I can add features by deleting code, yep.

KYLE: [laughs]

DAVE: I think it was yours, Eddy. I think it was your PR that I did that on, so...yep.

WILL: I'm not going to lie. I actually...

DAVE: It wasn't your code that we were reverting. For anybody listening, it was some legacy code that was no longer relevant, so...

WILL: Anyway, I don't know, like, unironically, not devil's advocating, I like just sort of getting to work, you know what I mean?

DAVE: It's very satisfying.

WILL: I like the aspect of, like, well, just agile sprinty. I mean, unironically, I'm not being the devil's advocate. Let's just throw something up and let people look at it, not release it. But it's like, "Here. Do you like this?"

So much ambiguity can be resolved by giving somebody something that they can touch and feel and put their hands on. I'm no longer being ironic. I like giving people prototypes, and they can click the buttons and be like, "Oh yeah, it clarifies so much." I don't know; I've found there's, generally speaking, a difficulty among people who haven't sort of managed a lot of software projects to think about things in a fully fleshed-out way in the abstract. If I showed you a form, and a button, and a widget, and a web page, even if it's connected to nothing, I can get much better feedback, much faster, you know what I mean? I like that.

EDDY: I would argue that's part of your UX design person. It doesn't really fall on the developer.

WILL: You go to war with the designers you have, man. They're art school kids. Like, that's not their gig. It's okay. They're going to make it look good. That's their job. Like, I don't know, if I can make it easier, if I can work harder technically to get a better product, then I'm going to do that, and I'm not going to complain too much about it. Like, that's my job.

MIKE: And, unironically [chuckles], right? I agree with you. I was thinking of an example of, you know, go spend three months working on backend without requirements, without providing a single prototype. And yes, I've seen that kind of thing happen [chuckles] in the past. And it goes about as well as you'd imagine. The [chuckles] services you're building don't match what's actually required because you need to do...to your point, Will, again, unironically, that quick feedback is so critical that if you can't loop on it, it's terrible, which is a good reason to build out one small domain out of your project first so that you can have something that actually works.

WILL: Yeah, yeah, that is an important caveat. I would say that this is kind of frontend only. You can't show your VP of sales a slick API. I don't know who the VP of sales is, but I have a pretty clear idea of, like, that is some slick JSON, you know what I mean? [inaudible 28:00]

MIKE: [laughs]

WILL: But you can show people interfaces. And if you show people a frontend, you know, the backend exists to serve the frontend. Like, the backend is your problem. Get the dorks on it. Anyway.

DAVE: There's a web mockup tool that was designed...it's called Balsamiq. And I don't know if it's still like this. I haven't touched it in years. But when it first came out, it looked like napkin sketches. Like, on your computer screen, it would draw somebody with a ballpoint pen on a napkin, and they did it very deliberately.

And I have had this happen to me where you mockup a quick prototype, business looks at it, and they go, "That's great. Ship it." You're like, "No, this is literally a prototype. There's nothing wired behind this. This is 4% of the system." And they're like, "That looks like 90%. Ship it. You've got a week," and you're like, "No."

And so, Balsamiq literally shipped sketch stuff to stop that, to let the businesspeople understand that this is nowhere near complete. And it's related to what you were just saying, Will. I think knowing your trade-offs is super important. There's a time when you want to spike, and there's a time when you want to just knuckle down and trudge. And we've got good estimates, and we think we're right, and there's a lot of work to do. Let's sled along on this, or let's...versus jumping in ahead and working on the wrong things, but, spikes, sure. And they do happen frontend, backend.

I have literally spiked a mockup of an entire website, full-stack, database to frontend with just Rails, generate, generate, generate, generate, in a car on the way to an investor meeting, because my boss had said, "No, we're not switching to Rails. We're not switching to Rails". And, in the car, he's like, "How long would it take?" And I'm like, "Probably mock it up in the car." And he's like, "No, you can't." I'm like, "Bet." And I did. And so, we switched to Rails.

But that spike there was nothing shippable in that spike. I built it to throw away. It served its purpose, which was to say, "This is possible. This legacy codebase that we spent five years building, here's the prototype. And based on the prototype and our existing codebase, I'm thinking 18 months." So, I was able to give him a realistic estimate off of it. So, knowing where your trade-offs are, I think, and --

EDDY: If that doesn't tell you how powerful Rails is for a start-up company [laughs], I don't know what does [laughs].

DAVE: To be fair, this was Rails 3.0 when you could still write a blog in 15 minutes on it without having to know Stimulus and a front-end packer, and a pixel shipper, and the, you know, the asset poop line, and all that messy stuff, so...

MIKE: So, if I'm going to go...I'm going back into the snark.

DAVE: Yes.

MIKE: Advanced warning [chuckles]. One great way to make sure that a project gets done on time is to bring in as many teams as possible. But make sure they work independently and never coordinate with each other because you wouldn't want them to have any inefficiencies of communication. Also, let them design the interfaces independently, and then we can just fix that communication layer at the end because that's not the important part. The important part is that we all make sure that they're all working, and they're all heads down. We can handle any and many communication issues later. Make sure the APIs are in line. That's not a big problem, right?

DAVE: Yes.

EDDY: Well, if you want a way to speed up your service, just don't write tests.

DAVE: Oh yeah.

EDDY: Just ship your code.

DAVE: Fair point. Yeah. And there's a generic form of that, which is never check your work, right? Never cross-check. In fact, we should put that as part of the waterfall. Make sure coding is 100% done before you start debugging. That's literally part of the original SDLC when they were saying, "Stop doing this."

EDDY: Yeah, also, let your developers be the sole QA. That's good, too.

DAVE: Make sure your creators are editing at the same time that they're trying to create.

EDDY: 100%.

WILL: I mean, software QA exists. The entire career exists because I cannot be trusted.

MIKE: [laughs]

WILL: That's it. That's it. There's nothing a QA can do that I can't and don't. You cannot trust me. I cannot be relied upon. Like, I can't do it. You can't, like, jump. Oh, wait, wait. Hang on. No, no, hang on. I said that wrong.

DAVE: You got it backwards, yeah. How dare you be candid?

WILL: Bro, why do you even have a QA? Devs know how it works. Like, they understand it better than anybody. QA is just a waste of time. Like, you have an extra person? They don't even know how to code, bro. Like, What? Why even? It's a nonsensical position to have. Like, the devs could do it. They did it. Just shut your work, guys, duh.

DAVE: If they can't code, we could have them doing tech estimates for us, though. There's that option, so...

MIKE: [laughs]

EDDY: And you can have your product guys go ahead and help you commit a bunch of stuff. When you're short-staffed, you know, just rely on other departments.

DAVE: Yep. We talked a little bit earlier about Fred Brooks, about how many women and da da da da. Fred Brooks wrote a book called "The Mythical Man-Month" that's all about this. And Brooks' actual law that he quotes is, "Adding people to a late project makes it later."

EDDY: It's true.

DAVE: That's where it comes from, yeah.

WILL: And I'd say another note on QA. I'd say the best people to do the QA are the high-level executives when you're doing your demo, your final presentation, preferably even after it's already been out to production and you got your VP out there.

DAVE: [laughs]

WILL: They're pretty loose people. Like, they're not detail-oriented. They don't have real visions. And even if they did, they found something; it's not like you're going to look like an idiot when your VP of product is checking stuff in prod, and your links are broken. And, honestly, those guys, they don't even notice half the time. They're just like [inaudible 34:30], whatever. They're just so chill. They don't even care.

MIKE: They don't have any relationships outside of your business either, right? And they're too busy working with people in the company. They don't have the CEO of your partner's company on their speed dial because, yeah, they don't have time for that. There's no way anybody would ever, you know, important business partner would ever reach out to them at 3:00 in the morning wondering why the service is down. That's just not their job.

EDDY: You know, one way to really ramp up the way wave to meet the deadline is to just ship it, even if it's broken. You met the deadline. Just merge --

DAVE: Yeah. We can ship DLC later.

KYLE: I've got a controversial one, I think.

DAVE: Oh, please.

KYLE: DevOps is a culture, not a team. I've seen where, you know, that's on the developer. They can do their own DevOps and manage their own infrastructure. And --

DAVE: It's called DevOps for a reason, Kyle. We're the devs. We do the ops.

KYLE: Yeah. So, we don't need a specialized team for any of that.

DAVE: Yeah. And all you need to know as DevOps is just chmod A plus X minus R. Just go nuts, just 777 for the entire hard drive. Yeah, it's fine.

WILL: It's called CI/CD, bruh, continuous deployment. I'm just going to throw that thing up. New endpoint? I don't know. There's probably no scaling requirements. The server was fine before. I'll just throw this new API up there. Let's roll, YOLO.

DAVE: Ooh, related to this, Mike's going to wince a little bit, but don't do any tech debt. If your CI machine catches on fire, get some marshmallows. That's the best thing you can do for it, yeah. So... [laughs].

EDDY: Well, you don't have tech debt if you just delete [laughter] [inaudible 36:16] you've broken.

DAVE: Mike is covering his mouth. There may have been an incident at work this week, so...

EDDY: You can't have [inaudible 36:36] that's broken, Dave. You just delete the code.

DAVE: That's right.

EDDY: Just delete it. It's not broken no more.

DAVE: Yep. Actually, touching on what Kyle said about DevOps, if you really want to mess everybody up, divide your company horizontally. Make sure that the database team are in their own spot, servicing all the database clients, because then you've got all your experts in one place. And anybody that needs anything from the database team now has to go up through their boss and their boss, and then come down through the data management people, and they've got objectives that they need done. Same thing with your sysadmin, same things with your DevOps, same things, you know, if you've got anybody that's writing backend and frontend, get them on separate teams. Whatever you do, do not under any circumstances have a full cross-functional product team.

KYLE: I think another good one is launch your service or whatever deliverable you have before you consider security.

DAVE: Yeah. Yeah. Yeah, you can just bolt that on later, come on. There's no security. That mockup that I showed you that, you said, "Ship it," and gave me a week; we can just do security later. It's fine. 100%.

EDDY: Mockups don't have permissions. Dave, are you crazy? You don't need to scope permissions to your users. Just whatever --

DAVE: If you can see the mockup, you can access the thing, yeah.

MIKE: You'll never have a user who will try changing the URL to see, you know, another user's data. Our users are never clever like that.

DAVE: Ooh, slightly related to that, I'm not sure how to phrase this, but as you're developing, if you trip over something, don't flag that. Just step over it every single time. Definitely don't put rules in your engine. Don't write an RSpec thing to check to make sure that there's no implicit scopes or default scopes on your Rails models. And you want default scopes everywhere. You want to tenant your customers at the default scope. That's really, really important --

EDDY: Make sure you unscope, Dave. Like, that's –-

DAVE: And then, what you'll also do is you'll have your scope that says, "Valid," which is going to select for things that have not been deleted so that then when you say "Unscoped," we throw away the deleted at so that you can pull all the records, and we also throw away the tenanting. And now you're looking at every single customer in the system. I had that happen at a company where showing somebody else somebody else's data involved federal oversight. Like, you messed up. There was a department that you had to report to and say, "We did that."

EDDY: I was going to say, Dave, that's kind of oddly specific [laughs].

DAVE: And if you press me further, I will deny it, yep.

EDDY: [laughs] [inaudible 39:13]

DAVE: I've never worked with an entire company full of idiots. I self-select out of that environment. I've seen all of these design...most of us have, right? All these design hell problems, and they usually come about based on really good partial information.

EDDY: I was going to say, something that can really speed up is just always, always, always going against your framework's convention. Like, it works 100% of the time.

MIKE: Well, I can one-up you there. You want to get out quickly. You don't have to deal with the constraints of a framework. So, make sure you implement from scratch because, that way, you can...

DAVE: Roll your own.

MIKE: Yeah, roll your own. That way, you won't have to memorize the conventions. You won't have to deal with, like, libraries that will fight back against you when you're trying to do direct database manipulation. Just build everything yourself. It's a much faster way to work.

DAVE: This week, I am maintaining some code that has some custom snippets of SQL wrapped up in a custom DSL that are wrapped up in the source code. And it's just a joy to work with. It's absolutely fun. I have not ripped out any of my fingernails. I have not shaved my head defensively to save tufts of hair from being ripped out. It's, yeah, it's just been a dream, absolutely a dream. So...

EDDY: You know, another thing that's helped me, Dave, and I want to say, like, [inaudible 40:41] loading has been wonders. Like, you want me to ship something? Yeah, copy and paste someone else's work, you know, that's done before. It doesn't matter if it's being used in that context or not. Just [inaudible 40:53] is always --

DAVE: Yes, going back to the good intentions, end up in hellish places; drying things up to the point that they become chaffy is fantastic. You should compress everything and especially, you know, that SQL DSL that I'm working on, get into RSpec, and there's an include, not, like, include shared context. It's an include module, and the module was in the support directory. And I opened that up. And when you include the module, it creates a shared context and includes it. And the shared context creates lets. I have told that nightmare story here at work multiple times, thinking certainly that I would never see it. We saw it this week, so it's pleasant. It's kind of fun.

EDDY: Mine is a controversial one.

DAVE: I will say that I am sincerely saying that I'm enjoying working on this because I feel a little bit vindicated when I work on it. Also, in defense, because I did actually say something about a real company that is doing real work, and I don't want to get yelled at for this, I do want to point out that the developer who did it that way did it in 2015 when it was a best practice. I've written a lot of code that had shared context and riffling the deck to shuffle all the pieces so that they, like, laminates on an overhead projector with multiple layers to make them line up and that kind of thing.

It's very satisfying to dry that up in your text editor because you've got it all in your head, and you're like, oh, yes, why should I duplicate this? Let's put this over here. It's not until you come back later that you realize that you have just burned off every entry point for intuition and for someone else to be able to follow your path.

MIKE: I was thinking about a real-world story as well about avoiding the framework. I'm going to go back a lot farther, I think, like, 2003 [laughs]-ish.

DAVE: Ooh, okay. So, probably not Rails. It might have been Rails, but probably not Rails.

MIKE: Not Rails. I don't think Rails was out there. Anyway, this was actually some sort of Visual Basic. I don't remember the exact flavor of it. And there was no database library used whatsoever. It was all just raw rights to the database. And it only takes one unsanitized database input. It only takes one to lose a million dollars [laughs] of your customer's money.

EDDY: Yeah. Just don’t --

DAVE: What's the .NET... .NET has a SQL thing, like linker, SkLink. There's a name for it. But yeah...that would've been coming into existence about two years after 2003. So, yeah, if you're going to write SQL, it's going to be in the code. It's going to have to be in the code.

MIKE: This is back...I remember Hibernate was just getting started in the Java world and, like, wow, this is great. You can use some library and convention to write your SQL rather than hand-coding it all [laughs], but not so in the application that we took over [laughs].

DAVE: Oh, never switch. There's some people on this call that are going to feel seen and attacked at the same time, but this is actually not. But if you're going to add a new technology, add it. Don't change the existing stuff. You want people to have to maintain both. It's even better if they don't know which one they're working on until it's too late. That's a good one.

MIKE: As many text stacks as possible?

DAVE: Mm-hmm. Mm-hmm.

EDDY: Ship --

DAVE: For real, I used to walk into stand-ups...if I wanted to terrify everyone, I would give my stand-up report as, "So I did a quick spike, and I rewrote this in Rust." And, if you can sell that with a straight face, you can get, like, the blood to drain out of other people's...they're like, "You're doing what?"

MIKE: [laughs]

EDDY: Also, make sure you support multiple versions. I think that's --

DAVE: Oh.

MIKE: Simultaneous.

DAVE: Arbitrarily, arbitrarily. Don't just do it for the interface stability reasons. Do it arbitrarily.

MIKE: Well, if you don't have three versions of bootstrapping in your codebase, you're not a real project.

DAVE: What are you doing right? Yeah, and Angular 1 to Angular 2, they redesigned everything. So, you're going to need both, or you're going to be missing stuff out.

EDDY: It's a good thing that you can't support multiple versions of Ruby, though, like, satire aside, right? Like, you can only have one true Ruby version for the project. You can't support multiple. Is that correct? I see you smirk, Dave. Like, I wonder [chuckles], is that possible?

DAVE: It's absolutely possible. Like, Ruby 1.0 shipped with a library, dRuby? Distributed Ruby. And it made remote procedure calls very, very easy. You could write your program in two places. And if they knew how to find each other, you could call the function here, and execute it over here, and send it back, and nobody uses it. It was an interesting experiment, and nobody uses it. But 100%.

And, honestly, we do this here, where we've got service-oriented architecture. So, this team is on one version of Ruby, and this team is struggling because they've got this really important science library or finance library that isn't up on the latest. So, they have to stay back on an old version. Yeah, you do see that. So, what you have to do is you have to get out of the processor. You can't share memory with each other. I'm not going to say that super confidently. Somebody will say, "Here's a proof of concept."

MIKE: [inaudible 46:34]

DAVE: But, to my knowledge, if you're going to have multiple versions, they're going to have to drop to a protocol and communicate with each other like separate programs.

EDDY: Not ideal.

MIKE: We've come up with a lot of great ideas. I think that we could come up with an estimate that's pretty much whatever we want.

DAVE: We might be able to bring down the whole system.

[laughter]

EDDY: I was going to say something controversial, but some people may or may not agree, really just depends on your perspective. But I was going to say, as a satire, make sure you always do integration tests, and use factories to make sure you're testing the real data so that it gets committed to your database, but --

DAVE: And committed without any trade-offs.

EDDY: But I feel like some people do agree with that, and that's an interesting [laughs] topic.

DAVE: As long as you're ignoring your trade-offs so that you've got your test where you need to go really, really fast, as long as they're writing to disk and doing everything slow, and exercising the full-stack, and the stuff where you just need to test an accessor method that doesn't even talk to anything else that should be talking to the database, too, then, yeah, you'll be fine. You'll be fine.

MIKE: Well, you need to make sure that everything works, right? So, make sure --

EDDY: Make sure you test Ruby and Rails, too, like --

MIKE: Well, yeah, you can't trust the framework, and you definitely can't trust your third parties, so make sure that you make live API calls to their sandbox in your integration, you know, in your CI environment.

DAVE: Oh, yeah. If you come up with a workaround to a problem, solve it as far up the tech stack as possible. Like, I worked with an engineer who would recompile Ruby to add features to it, and we set him straight pretty quickly because [laughter] we're like, "We're not shipping your copy of Ruby to all of our servers. It's not happening. Stop asking." To be fair, he was a genius. He could shatter a human mind at 50 paces just by thinking about it, and he was very, very, very clever, and sometimes he was too clever for all of our good, so...

KYLE: I'd say –-

DAVE: I had a funny idea for how to end this podcast, which is to say, "Okay, now we estimated this much time," and then cut off right in the middle of the sentence, just no outro.

[chuckles]

EDDY: Kyle, you were going to say something?

DAVE: Sorry, yeah.

KYLE: I was just going to say, and don't worry about the performance of your code. We can always throw more hardware at it.

DAVE: Right [laughs].

EDDY: It's your computer that's slow, dude, [inaudible 49:11] [laughs].

KYLE: And we can ship that into the cloud if we need to.

DAVE: Yes.

EDDY: Develop to the cloud; it has unlimited resources.

DAVE: And I will see you in raise. You should care about maximizing performance at the start.

MIKE: Yes [laughs].

DAVE: You want to make the code as hard to read and as difficult to maintain as early as possible. And unrolling anything that's readable into something that's just hyper-efficient, go for it. That way, you're guaranteed to optimize things that aren't important. So, we will still need to scale up into the cloud to get around all the bottlenecks that we didn't even look at.

MIKE: And make sure that you do lots of planning around that optimization during the beginning of your waterfall process.

DAVE: Yes.

MIKE: Because that's what's going to matter the most is that optimization.

EDDY: I was going to say one thing, too. I was going to say, don't add comments and be super brilliant to what you're doing, so it's complicated for everyone else.

DAVE: Yes.

EDDY: So that you --

DAVE: Never ever pave a path through the code. You need to vanish into the jungle and be lost without a trace. Anybody who needs to follow you needs to solve it from first principles. That's important. It builds character.

EDDY: 100%. Like --

DAVE: And you need to prove your worth.

MIKE: So, if you don't have tricky code to read, well, then you're probably doing it wrong, right? I mean, how smart are you if everybody can read what you're writing?

DAVE: Yeah. We can go into a whole tangent about, like, your ego is more important than right and wrong, and that should be protected at any cost.

KYLE: I think one thing that goes really good on readability is the largest PRs as possible.

MIKE: Oh yeah.

DAVE: Definitely.

MIKE: You wouldn't want to waste time with QA having to review a bunch of small PRs. So, make sure you release the whole feature in one, that way, they can just test it once –-

KYLE: Save testing time.

EDDY: And test in production. That's the best way --

KYLE: Oh, that's the best one. Yeah, we can go to production first.

DAVE: Yes --

EDDY: Who better to tell you that your feature sucks than actual users, you know what I mean?

DAVE: I have a suggestion for a good final one, which is, if you're listening to this podcast and you think we're making fun of you, ignore it. It's fine. Don't take any lessons away from this. Remember to just stick by your guns, no matter what. Nobody has done any of these things in real. We're not talking about you; it's fine. Yeah, definitely don't take a hint from this to maybe try to improve steadily and stably. If you are going to react, make sure you overreact. I think that's the important takeaway here.

MIKE: We'll take the same advice because, of course, we've never run into any of these things ourselves.

DAVE: Mm-mm. Mm-mm. That's fantastic. Are we at a good stopping point, gentlemen?

EDDY: I think so.

DAVE: This has been a lot of fun. Thank you, guys, for coming out: Mike, Kyle, Eddy. Will had to drop off for a family issue. But so glad you guys came out. This was a lot of fun. And thank you all for listening and watching The Acima Developer Podcast. We'll see you again in a couple of weeks.

In this episode of the Acima Development Podcast, Mike kicks off the discussion with a few personal stories before diving into a technical debate. He recounts his recent epic hike in the Rockies with his son, which turned into a grueling trek due to altitude sickness. He also shares a nostalgic memory of visiting the massive Edmonton Mall in Canada as a teenager, noting the stark contrast between the pre-internet shopping experience and today's online marketplace like Amazon. The conversation then shifts to a comparison of governance structures in various countries, which sets the stage for the episode's main topic: software architecture and the debate between monoliths and microservices.

The panel explores the evolution of software architecture, with a focus on the trade-offs between large monolithic systems and distributed microservices. Mike and the others reflect on how early approaches to software development often favored building large, cohesive systems, but over time, the trend shifted toward breaking these systems into smaller, independent services. They touch on the challenges of communication and synchronization among microservices, and some express a sense of regret over the movement toward extreme modularity, suggesting that, while microservices have their advantages, they can introduce significant complexity and overhead when overused.

The conversation also delves into the practicality of maintaining and managing distributed systems, particularly from a DevOps perspective. The panel discusses the fine balance between creating boundaries in code and allowing flexibility for different teams to work on separate components. There is consensus that, while microservices can improve scalability and modularity, they can also lead to what one panelist dubs "micromonoliths" if not properly managed. Ultimately, the panel agrees that the choice between monoliths and microservices depends on the specific needs of the system and the organization, underscoring the importance of choosing the right tool for the job.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting today. We've got a great panel with us here today. We've got Will, Eddy, Adam, Dave, and Kyle. And today...well, I'm going to hold off on what we're talking about. I'm going to give three stories; three things to start us off.

WILL: [laughs]

MIKE: But the first one is a story. I'll start there. So, last week, I was on vacation, drove out west, saw the Rockies, went on a, I'd say, an epic hike, which is true, but it was also just, like [laughs], in some ways, a brutal death mark hike [laughs]. Went on a hike with my oldest son, and we did, like, 8,000 feet of elevation gain, and, like, 20 miles, like, 16-hour hike, just a big, brutal hike.

Loved it, but it was tough, particularly because we got up to the high elevation, and my son got altitude sickness, and he was just sick. I'll spare you the details, but there were some digestive issues going on that were serious [laughs], and that cost us quite a bit of time [laughter]. And he was not having a good time. Everything's fine. We got down to a lower elevation, and he felt a lot better.

And it was great, I mean, the view was amazing. Didn't see very many people. It was a gorgeous hike, but [laughs] it was rough. We've been back home this week, and he's been doing some online classes, but, in between, he's been running up and down the stairs for exercise. So, like, you know, go one flight of steps up, and I think he's been doing 60 times up and down the stairs, seeing how low of time he can get it, and no altitude sickness [laughs]. It's working out better for him. So, there's my first story.

Second story, when I was a kid, and this was a long time ago, I went up to the Edmonton Mall up in Canada. I think it's the largest mall in North America. And I don't know how old I was, like, 14? So, I'd say it was a long time ago. So, I really can't speak to them now, but I can when I was 14. It was this huge mall, and there were several stores. There was, like, three of that store within this mall. It was so big that there was, like, a region where you go to a store and then go to another region of the mall and go to the store again. And you just get lost in this place [laughs].

I don't know how many, you know, fast food places they had. I remember there's some places that had at least three stores. It was crazy. And they have, like, an indoor amusement park, and water park, big mall. And, you know, thinking about the era that I'm talking about, that was, like, the pinnacle of shopping experiences because you could get all the places, you know, all the stores in one place. Since that time, times have changed. That was pre-internet era.

Well, now, if you want to go to something similar, you just pull up your phone, and you go to Amazon, right [chuckles]? Not that people don't still go to the big mall in Canada; I'm sure they do. And they probably enjoy that experience, but they probably lean more into the experience part of it, your amusement park and water park. Because you go to Amazon, instead of having everything in one place, they have their little stores all over the world, right?

And by distributing that work, and just making it a central, like, bazaar, you know, where you come and see all the stores in one place, you get way more selection. And the world's kind of fallen for it. We're not going to debate the pros and cons of Amazon here [laughs] [inaudible 03:54] that they have been widely embraced, that they have been widely embraced with that different model.

Third story. And this one is not so much a story as a comparison. Let me talk a little bit about governments. Now, in most large...I say most, I'm going to mention the United States and the European Union. You do have a central federal government, but you also have a contentious group of states within that central government that don't always agree and regulate themselves internally. And sometimes it works, sometimes it doesn't, but [laughs] it does allow for some local rule and for those independent states to run things, you know, to their own liking. And then, they federate for the big decisions that need to be made together.

There are nations that don't work that way. I was thinking about Saudi Arabia. Very much, we've got a monarchy [laughs]. We say what you do, and that's it. The interesting thing about that, though, is it's not like those fractious internal debates don't happen in Saudi Arabia. If you read anything about the internals of the ruling family there, it's scary [laughs], murderous sometimes, and I don't want to mess with that. So, it's not like those fractious internal debates go away. They just get handled very differently when you've got kind of a monolithic rule, which brings us to the topic of our discussion today.

We're going to talk about software architecture and about monoliths versus microservices. This has been an ongoing discussion for a long time now in the community. It used to be kind of everybody went the [inaudible 05:33] think [laughs]. You'd build a big thing and then deal with it. And people start saying, "Well, no, big is bad. We need to have lots of small things." And then, some people pull back and say, "Well, no, maybe big wasn't so bad after all because there's a lot of issues with communication among lots of small services and keeping those all in sync."

I brought up a few examples outside of the software world to give us some fodder to chew on, and I'm going to start off by just asking you all in this spectrum, saying, hey, you know, putting everything in one big application versus breaking everything up into small apps. Where do you stand? Where do you think that right balance should be?

TAD: Can I get a definition of what you would say a monolith is and a microservice? Because I remember way back when we were just...everything was services, and then this huge trend came along of, oh, what if we made, like, really, really small, very, very narrow services and did everything with Sinatra and never touch Rails again because it's too big and bloated? So, what makes something a microservice, versus a service, versus a monolith, versus anything?

MIKE: There you touched it, right [chuckles]? I used the word spectrum deliberately [laughs] because I don't think that anybody has a perfect definition there. And you could certainly, say, "Hey, we're going to have an architecture where there's no service that has more than one end point." Sure, right? I don't think most people would like that. Just going out on a limb --

DAVID: It's arbitrary. That feels like an arbitrary decision, right?

TAD: Well, I've seen that. I've seen places that they're like, oh, we're going to have a routing number validation service. We're going to have a credit card validation service. We're going to have, for any particular thing that you need to do, you've got a service that you hit. And it's divided up amongst...it's almost like functional level [laughs] services.

MIKE: But how did it work? [inaudible 07:43]

TAD: Not great. Not great. You can probably tell that I'm like...every time I hear the word microservice, I just [vocalization] shudder and just, I don't know.

WILL: So, one, I feel like a monolith versus microservices architecture it's like pornography, you know what I mean, it's hard to define, but you know when you see it. And, two, and what I've seen, I have some direct experience, and then I have received...maybe a year and a half, a year, year and a half ago, there was this top-down mandate at the large electronics retailer that I work for that all of the teams are going to be cross-functional teams, right? So, they said like, "Nah, we're not doing monoliths. Everybody's doing everything," right?

And almost immediately, all of the cross-functional teams immediately spread out into their, you know what I mean, on a micro level, so instead of having a team of mobile developers, or a team of backend developers, or a team of frontend developers, everybody sort of split out and said, "All right, I know how to do the mobile app stuff, so I'll do mobile app stuff. You know how to do back-end stuff, so you'll do back-end stuff. And you know front-end stuff, so you'll do front-end stuff." And everybody just sort of self-organized  into their silos of choice.

And even if it was one giant repo with one giant Windows desktop application, right? Like, if you had a team of developers...and they will naturally, instinctively, like honeybees, self-organize into functional groups. It's like how even if you have an open seating plan, everybody will kind of pick a chair, and they will sit in their chair automatically. And they will get familiar with this one siloed piece of the codebase because it takes time to boot up. And so, it's an emergent sort of organizational thing. Microservices will happen instinctually. And they're like, oh, maybe we don't have separate GitHub repos. Even if it's all in just one big pile, humans will instinctively do that, you know what I mean?

MIKE: I think that there's a lot of truth to that. Well, we have limits as to what we can fit in our brains [chuckles]. There's finite limits to human capacity to think through a problem, to think through some domain within a problem. And once something gets outside of that space, you know, once it outgrows that, it becomes really hard to think about. And, you know, you probably have to map something out and think about sub-pieces of it. So, you just naturally look at sub-pieces of it.

And there's a few ways to approach that. You can have it within the same repo, right? Not the same repo but the same application. You deploy it all together. And then, you have, you know, lots of units, functional units within a single codebase, or you can just blow it up and have separate, you know, fully independent. If you're out on the web, if you have the advantage of doing so, you can have all these separate units, you know, deployed separately, managed separately. You don't even have to think about the other ones, except via API. And really kind of harden those boundaries to where, you know, you only talk to each other via API.

And by enforcing that, now you don't even have to think about those other things, maybe [laughs]. Maybe you don't have to think about them. And I think that's kind of the microservices or the, you know, the small services advocates would say, "Well, yeah, no, you should definitely make a hard boundary. That way, you're not tempted to cross those boundaries." Where people who've seen that go too far tend to say, "Well, you know, maybe we can maintain boundaries in code, and we don't have to put it, you know, deploy separately." You're saying, "Well, this happens automatically." And I think that there's a lot of truth to that. Do you think there's value in hardening those boundaries to enforce an API?

WILL: You know, if I'm being totally honest with you, what I see, like, the difference between sort of a monolith versus a microservice thing is, I see it as analogous to sort of serial versus parallel programming, right? So, with serial programming, you do it right, and then if you do everything right in a row, procedural, like, tick, tick, tick, tick, tick, tick, you know? Add it all up in a line, and then there's my result. I return my result, and I'm good. But it's not as fast. It's not well, you know, broadly.

Like, if I want to write some slick parallel algorithm thing, that's a harder job to do. But you can get that 16 core, you know, server. You can keep them hot, and that's great. But it takes higher skill. And so, similarly, the spectrum you're choosing your path on in terms of microservice versus monolith, if you have a monolith, you're leaning really, really hard as things get bigger. As the monolith becomes larger, you're leaning really, really hard on super high-skill, super domain knowledge engineers that can wrap their heads around everything and understand how to deal with the monolith without knocking it over. Because we think of it like a pyramid, but really, it's like a pillar just swaying in the breeze.

And so, when you go over to the microservices thing, then you're leaning on sort of managerial organizational capacity to manage this sort of 16 core. Instead of 16 core CPU, you've got 16 dev engineering team. And you need to make sure that everybody is coordinated and pulling in the same direction and all those things. And so, that's sort of the spectrum. Like everything in engineering, it's not good versus evil, angel versus devil things. It's just sort of where are you on that spectrum, and what's the right tool for the job for where you are right now?

DAVID: And, Mike, you mentioned at the top of the call you were talking about monoliths versus services and whatnot. And do we sit down and write a big thing? And I think, no, when you start out, you always write a small thing that, you know, every monolith started out small somewhere. And, for me, it's always down to a trade-off, right? If you've got services, you now have very expensive communication between those services. But the things that are in those services are now decoupled, which is nice.

In a monolith, the communication between them is very fast and very easy. I just want to look at that module, just go look at that module. The drawback is that if that other module wants to look into your stuff, they can just do it because they're inside the firewall, and so there's that coupling back and forth.

So, I made a laundry list as you guys were talking. Tad, you mentioned definition of microservices. What I saw in the teens, the last decade, was a lot of people doing like, oh, monolith's bad; SOA good. And what they went out and wrote was every service was its own monolith. It's like, oh, instead of having one, you know, 100,000-line code base, you now have 500,000-line codebase. This is not moving in the right direction.

TAD: Now you've got two problems.

DAVID: Now you've got two problems, exactly. And so, yeah, every problem in computer science can be solved with another server in the microservice cluster, except for the problem of too many services in the cluster. And I heard a really good definition of what a microservice is, and that is something that you could rip out and rewrite in two weeks. And I've never [laughter] seen somebody write...yeah, exactly. That is the correct reaction. And if you're saying, "We're doing microservices," and your services are all 30,000 lines and have 73 models, those are not micro. You couldn't rip them out and rewrite them.

The interfaces between things are very expensive. Well, they're much more expensive because you have to negotiate with both sides, and, often, there's, like, three or five different deploys that you have to do to make it possible, and then use it, and then require it, and da da da. And you have to work out that dance. And if you need that secured, and hardened, and guaranteed, then you want that to be formal. You want it to be in a service. So, you want it to be locked down on a contract. But if it's fluid and it's flowing, you're just paying through the nose for something that you could have spiked out with an agile team overnight. But now it's a six-month initiative because you made it as expensive as possible.

EDDY: I'm actually curious to hear this from a DevOps perspective. Do they prefer to maintain microservices over a monolith?

DAVID: Do we have DevOps on...Kyle?

KYLE: Yeah, I was going to say, at least from my perspective, I'm all about orchestration tools, and something like Kubernetes is extremely powerful. So, I would much prefer a microservice, but I do feel like we've kind of talked about it. A lot of microservices end up being micromonoliths, is what I've called them in the past. And it's still one of those things, smaller units that can be divided out. And then, it's, at what point do you continue to divide? Because you can have quote, unquote, "microservices," or you can go [inaudible 16:54] the functions because there's stuff like AWS Lambda or serverless, right? So, at what point do you stop...each rendition complicates the next one. With functions, at what point do you keep them hot? Because that's something with Lambdas.

Originally, they weren't kept hot, so you had to wait for them to spin up and be able to function; now, they have hot functionality, and the same thing with even running microservices on servers. You have to have hot servers. Otherwise, you're waiting to deploy. And then, you go to the smaller units, at least in my experience, and you've always got that one server that you can deploy to because it's always up. But you start going out. It is nice in the sense that you can divide your workload amongst smaller machines and cut costs and stuff that way at times. But that's a long way of me saying it really depends on the implementation, I guess. I prefer microservices, but not to an exaggeration.

EDDY: So, micromolothiths.

WILL: Well, I always felt like, from a DevOps perspective, you've got really heterogeneous workloads, in that you've got boxes that are like, okay, I'm doing this thing. I need this amount of stuff. And I have other things that, like...and so, the more heterogeneous jobs you have running on the same machine, I feel like, correct me if I'm wrong, because this isn't really my trade, right? It's like, it makes it hard to correctly provision a machine. Because it could be doing anything under the sun, then you have to create a larger machine, larger buffer because the workload could widely vary.

KYLE: Yeah, yeah. And even within inside of our company, we've got services that are doing different things that are different sizes. And we will make it so that they won't land on the same nodes as themselves or the other large services, and sometimes we don't even want some of the small services to land on there. And kind of what you're talking about, we do have more general-type workload servers, along with we have more scheduled workload like our data science teams, where they've got crons that sit there and run more so than anything, maybe serverless is what it would be closer to, where it's just very spiky workloads. It's just, throughout the day, just bam, bam, bam. And, like you're saying, those need very different machines to support them.

WILL: Right. Right, yeah.

DAVID: One of the things about Kubernetes and Docker—and its predecessor Vagrant, and Chef, and Puppet—this idea of, like, oh, you say you're doing microservices, but you need an entire operating system now? Your microservice needs...you got to configure how many CPUs it has? That's not a microservice anymore. If you need to install Ubuntu, all of Ubuntu [laughs], to run your service, it's not a microservice anymore.

WILL: I don't think I track.

DAVID: Oh, just something you can rip out and rewrite in two weeks. Oh, but you're going to do a Docker container. So, you have to provision an entire operating system drive space. That used to be called full-stack dev. And it's like [crosstalk 20:17]

WILL: And maybe I don't understand because, like, I don't know, I'm stupid, you know what I mean? I thought Docker was cool. I was like, oh, I could just do a Docker. Like, I'll fill up a pod, and I don't have to do a full-on EC2 install. This is so great. What should the microservice, I mean, if it's a proper microservice, how should we be doing it if not [inaudible 20:38] a Docker image? [inaudible 20:40]

DAVID: I want to be careful to not walk into the trap of proper because I think the way we do it now is kind of proper. But 15 years ago, all the hardware in the colo was the same. Like, everyone had the same processor. Everyone had the same hard drive. And everyone would say, "Can I please have more RAM on the server?" And ops would say, "No, you can't have another gig of RAM because another gig of RAM on your box means we have to provision every machine in the colo. It's a 20 million upgrade." And Docker gets us away from that. And I'm 100% on board that that's a feature.

I'm just saying when I see Docker scripts where it's like, okay, yeah, well, let's go get this thing, oh, and now let's download Python and patch it, and da da da da, and it all becomes part of the entrails that are actually part of what you're working on, suddenly, it's very, very complicated.

Mike, you mentioned story time at the top of the hour, and there was a specific story that I think is actually relevant to this. When I was at a company that shall remain nameless because I might still have friends there and I want to keep it that way, they wanted to do a big push for SOA. And we had over 700 separate services at one point, and it was trending up. So, when I parted ways with the company, it was 700 different services, and there was incredibly painful fatigue around adding a new service. Like, if you wanted...oh, I want to do a new service to this thing. You would get...like, everyone would just automatically just...

Sorry, Eddy just asked in the chat, "What is SOA?" service-oriented architecture. So, take your monolith. Instead of a single architecture, it's spread it out. And then, most people, when they do SOA, they'll take...instead of doing micro, they start tiny, but they get bigger and bigger and bigger. And then, you've got complexity inside each of these distributed services. So, that's where that came from. We had 3 different services that were putting records into the God record, right? Every startup has that one table, right? For us, it's like the contracts that we do. And in a medical company, it'll be your prior authorization request, that kind of thing.

And we had three separate services that were creating new items, new documents, which they needed to create, and I can't remember...they could not just stick it in the database and read it back to find out what the primary key was. They were, like, physically disconnected from each other, which means they had to come up with primary keys without colliding with each other. And for whatever reason, it wasn't GUIDs. So, these services had this...I want to point out this was in the last 5 years of my life or 10 years of my life, and I've been here for 4, last 10 years of my life.

Every time you created one of these documents, it would try to insert, and then we'd go, did it work? Uh, no [laughter]? That primary key was taken? Okay. Here's a new one. Does it...no? Okay. And, yeah, and it was fun because we were running out of key space. We had meetings about exhausting the key space. We had used all of the keys randomly through the entire search space. We had over 50% of them, and so we had this system that would retry 5 times, and it was failing 0.01% of the time, then 0.02%. And we're like, you know what? We're going to retry six times. We're like, this is not solving the problem. We are running out of places to randomly roll dice and try to hit a blank spot somewhere in the key space.

WILL: [laughs]

DAVE: It was a nightmare. And so, I came in and I said, "Guys, this is the textbook example for a microservice," a random number generator that can look at the database and see, is it colliding? No, no, it's not. Great. Oh, and, by the way, we could make the random number generator deterministic. They couldn't have ordinal because they were worried about ID prediction attacks. So, they had to be random.

And so, yeah, little microservice. It was five lines of code, baby. Fantastic. And everybody would talk to this to get their primary key and insert it in the database, and it was guaranteed to be available to them. Life was good. And everyone dug their heels in and said, "No. No more new services. No more." And I'm like, primary key on your God object table, if there was a reason for a microservice, this is it. And because we get away from why are we doing this and into this is the right way to do it, we stuck to the doctrine rather than to the principle. And, hmm, yeah, good times, good times.

MIKE: So, they'd done so many microservices that people refused to do another one, even when --

DAVID: Yeah, there were over 700. Somebody came back from DEF CON...I won't name names, but his initials are David Brady. He came back from DEF CON...I told my team, "You got to see some of this stuff." And I started talking about war gaming and red team versus blue team, and, you know, what's our disaster recovery. And the company was in the process of getting a second data center. So, all of a sudden, we went from 700 servers to 1,400 servers because we had to duplicate everything. And the war game plan was, if this colo gets nuked from orbit, we have to be able to fail over to this one. It was a nightmare.

And it took over a year to get to a point where we could get everyone together, and we had a war game night. And it was literally when traffic died down on Saturday night. We all lined up, and we started...we basically simulated an outage as politely as possible. We let everybody shut their services down politely so that you didn't have to do data recovery on the restart. But we took everything down out of the Atlanta data center and started bringing everything up in the Chicago data center, and those war games took 9 hours. We started at 6:00 p.m. And we figured this will probably go till midnight. No, it took till 4:00 a.m. It was a nightmare.

And the plan was to bring it down, move it over, bring it up, test it, bring it down, come [inaudible 26:17] back. And once we got up on the new data center, we were, like, "This is our new primary data center. Nobody ever move this again." And, a year later, they said we needed to war game again. So, every year, that company swaps data center. I don't know if they're still doing it that way, but that's...yeah. So, too many...literally, just the act of saying, "Where is the next service going to be?" It gets expensive. And it's a low exponent, but it is exponential. And if you get enough, that line will start to curve up on you.

MIKE: Well, I think that your story captures something, and it's been touched on a few times in this call, and it goes to the idea of coupling. If your services are tightly coupled, you have not separated out your system. You've just made the same system with a more challenging interface. I've heard it put that a tightly coupled system that's distributed is just a distributed monolith.

DAVID: I like it. I like it.

MIKE: You've got all of the problems of both monolith and a distributed system. And the coupling there is, I think, absolutely the key. If you can't have one of those services go down, then you haven't really extracted it. Anything that's a single point of failure is not really meaningfully extracted. It might as well be inside your system because you depend on it just as much as you did internally. And you have all of the cost of network overhead, and managing that service, and everything else that is involved in the cognitive overhead of having that separate. You really haven't separated it out.

There are tools, though, I mean, you can have...if you build your system right, you can, many times, have a service that can go down and nobody notices [laughs]. I mean, hopefully, you're monitoring it, and you notice, but the other services can keep running. And there's challenges there. You have to have some data redundancy. You have to have, you know, whatever. It's very situational-dependent, right? So, I'm not even going to drill into that.

DAVID: So, kind of a weird...I was reading on some stuff on how Amazon did their architecture way, way back when. They did things in Lisp because they had a bunch of crazy gearhead PhD doctorate candidates that loved to do stuff in Lisp, which is wildly inefficient, but they were writing this highly distributed, super scalable system, right? And it was because they had everything super, super tiny. The weird thing that they said that I have taken away and I just keep it in my back pocket...and I've forgotten the why. So, I'm going to throw this out there undefended and let you guys just pick it apart. And anybody listening to this can pick this apart but just think about it.

When you're building two Rails models, if you do a belongs to in one direction, it's just instinctive, right? You automatically go to the other one and create a...if it belongs to, then you go to the other one and do a has many, right?

TAD: Has many. Sure.

DAVID: Yeah, and the guy that was writing the...I want to say this was the microservices book that came out of Amazon from this guy. He said, "Resist that temptation." Bidirectional communication in CS is often ten times harder than single direction. And he gave a very specific example that if your customer has a shopping cart, okay, if customer, you know, has shopping cart, that's great. If I want to pull up your order page, I have to loop in the customer server, and I have to loop in the shopping cart page. But if it's bidirectional, then if you turn around and you look at the leaf node, as long as you know the customer name or stuff that you can go look up later, that's fine.

But if it's indexed and it requires this thing to be in the other present, like you said, you now have just duplicated monoliths. You have to have...that's what it was. He was specifically talking about having a database where you might have shopping carts and no customers. Like, you literally don't have customers in that database. You have to go to the service to get it.

But in the other system, the customer is there, and the shopping carts are there because they're linked, and they have to be efficient. So, that was the rule that I tried. It's like, I try not to just immediately do...and I want to say we actually have a RuboCop rule that says, if you put the one side in, always put the other side back in. And that's not always a bad idea, but it's not always a great idea.

TAD: I think it's pretty surprising to not have bidirectional stuff. Like, if you encounter something, you assume that that relationship goes both ways. So, --

DAVID: I could give you a better example. If you want to...so, we do leasing here, for those listening online. You need the customer record in order to service the lease. If you're in the call center, you need the lease, and you need the customer. But if you are at the contract origination system, you don't need any of the servicing stuff on the account because the account has...but you do need the customer information. And that was where the argument was.

And I agree with you that if you're down over in the servicing side, it should probably be bidirectional because you always need both. But if you're at the other end, single point, and that was the guy's push, was, like, by default, go single directional until you need bidirectional. And I've been caught in the neck with, you know, it's like, I just need this item, and it's over here, but there's no way to propagate. And so, you have to do a refactor PR to add that, so...

TAD: Well, I could understand you don't want to load everything up, but it seems surprising to me that you wouldn't want both things to know about their relationship with each other.

WILL: I'm with David. I'm with David. I'm strongly with David because, like, I don't know whether you guys saw the lightbulb turn on, right? Because one of the easy patterns that I've seen in this sort of database-centric things, I mean, we talked about a Rails application, but there are many where the lines of ownership get really muddy because you can get to anywhere from anything. And you'll find these situations where you'll get sort of, like, hop ons, right?

A better thing is I do have this sort of aggregator object. I've got, let's say, a lease, and I have things attached to a lease. Or I have a customer, and I have things attached to a customer. And it promotes very clear lines of ownership, where if I've got a customer that, I can get what I need from the customer. Or I have a lease, I can get what I need from the lease.

A way to visualize the thing that I'm thinking about, at least, is, like, think about an unstructured graph and traversing this unstructured graph where things can just go topsy turvy, any old place, versus a tree where I go from the node, and I could go to the leaves, but I don't go up, right? There are no cycles in this graph. And think about sort of traversing and just writing sort of traversal algorithms on an extremely structured tree, where it has rules, and the rules must be followed, like a bee tree or whatever.

TAD: But doesn't a tree have a node? Like, the parent node knows about its children, and the children know which node they belong to.

WILL: It depends on the tree, [inaudible 33:13]

TAD: It seems super weird to me that a leaf, you can say like, "Who's your parent?" And it says, "Oh, that guy." But I can ask that guy, and he's like, "I don't have any kids. I don't know of any relationship to my children." That just seems really surprising to me.

WILL: I'm thinking about really sticky, complicated, little data graphs, data associations because I've seen, I don't know, I've seen them go off the rails, where there are all these things snarled up together because there isn't clear ownership. There's not a clear, I am the container node that everything sort of spins around. And what I'm thinking about specifically is these situations where you need to put in...so let's say I want...I want to make a service call, and I have to attach a bunch of different extraneous stuff to my thing because it's like, oh, well, I have the shopping cart.

The shopping cart has orders, but I have to have the customer because the customer has the rewards account index, right? So, I can give them their rewards points. And then, I have to have the associated store because the order is going to be fulfilled from their preferred store. And you get all these things that are just sort of, like, there isn't a reason that they all have to go together, but because these things have not been organized...and, I mean, I don't want to be, like, I'm not a zealot on this. But, as a rule of thumb, making things a one-way hop, unless there's an overriding need for it not to be, that's really compelling as a guiding principle.

DAVE: I may have explained it poorly. Actually, I may have achieved exactly what I said when I started this story, which is I'm defending this poorly, and so it'll start discussion on it. But yeah, it's interesting to kind of push and think about, like, if you've got literally two services, one for orders and one for customers, one of them needs to know about the other one, but the other one might not always need to know, and that's, yeah, even though they do always have that relationship, yeah.

MIKE: Well, I think that's actually kind of important. Let me take your idea a little further. Let's say you have customers, and you've got leases, and you've got locations, and you've got merchants, and you've got shopping carts, and you've got, you know, any number of things. I think that if the customer has to know about all of those things, then it's just a big ball of mud. You've got all of that complexity tied up in the customer. But if you have a customer and it's just customers and your shopping cart is in some other service that handles shopping carts, and they have a reference back to the customer, so they know who the customer is.

But the customer knows nothing about shopping carts. Doesn't have to know a thing about shopping carts and doesn't even have the ability to look it up because he doesn't know about shopping carts and doesn't have to. But your customer stays simple, and you don't have to deal with that complexity. Shopping cart knows what the customer record is and could look it up if it needed to, right? But that's a one-directional relationship, and it doesn't need to be otherwise.

And where those shopping carts are very ephemeral, right? I mean, they come into existence; they pop out of existence; you have many, many of them, and they really have nothing to do with the core idea of the customer, that actually disentangles, I think, your architecture dramatically. And when you have all kinds of things, you know, and I could probably come up with 100 things that need to reference those customers, if the customer has to know about all of those, then your customer gets extremely complex, where I think that you haven't really decoupled, unless the customer doesn't have to know, right?

You've got something that maintains, here's your customers. And somewhere in a data warehouse, you can map everything together, right? Or maybe even in the frontend, you have some ability to tie things together because you can make the, you know, you can [inaudible 36:56] backend and put things together. But I don't think that whatever is holding, you know, managing your customers should know about what payment you made, you know, what your payment method was that was used three days ago. It gets entangled with something that has nothing to do with that. And that's the idea, I think, in the best sense of what Dave is trying to say.

DAVID: I think I might see a distinction here as well that if you talk to the entire system, you're going to have to know about both and the system output. But if you want to have a conversation about shopping carts, you can't just go talk to the customer service. You have to go talk to the shopping cart service. It's allowed to talk to customer, so there's that direction. But the other direction, there's no link. There's no way to link customers down to their shopping carts because that service is whittled down and kept very, very small. That's the piece that, at the system level, you absolutely can go both directions. I'm not saying sever the link in one way. I'm just saying this particular tiny service we sever it just for that service.

WILL: Well, I like this shopping cart customer kind of thing, right? Because there's a clear ownership thing, right? Customer has a shopping cart, right? But maybe case study, like, what I'm imagining here. It's like, okay, I've got, let's say, I'm trying to do some new feature from this shopping cart, and I want to go out, and I want to say, "Based on the stuff you've bought, I'm going to machine learning up some things so that when you're in your shopping cart, I can show you some other stuff because you've got cash in hand." And I'm like, okay, I got a hot buyer. Let me see if I can upsell them on some stuff based on the things that I know about them.

And so, then as developers start developing out the feature, then you start to tunnel through the shopping cart, then through to the customer, and then through to their purchase history. And now you're going from this shopping cart thing, and you're tunneling through, and you're getting into all this other stuff, which is like, whoa, whoa, whoa, whoa, whoa this is a little bit, you know, this is a little funky, and now you're sort of injecting these dependencies by going through the path of least resistance. Like, I have a shopping cart. I'm checking out, right?

So, rather than pulling in the stuff maybe kind of a top-level way, you're side-loading this relationship, and you're using that to tunnel your way through the system to avoid doing it on a straight-up way, where it's like, hey, I want you to, like, you know, like, you're bringing in a shopping cart, and I want you to bring in some recommendations for me, you know what I mean? Is this analogy coming off the rails? I'm freestyling a little.

TAD: I think what I'm kind of thinking about now is, I think that you need to know the relationship, but you also need to respect the relationship, meaning Eddy and I are coworkers. And he's like, "Hey, let's go for pizza," or something like that, and I'm like, "Hey, yeah. Can I borrow some money? [laughter]" He'd be like, "Sure," right?

But if I reach into his pants and pull out his wallet [laughter] and then grab that money out of his wallet, I've violated that relationship, right? I think it's important that I know like, oh, Eddy and I...Eddy is someone I could borrow money from, right? But if I'm going past that interface of that relationship, I'm violating the boundaries of that relationship. So, I think, that's kind of how I'm seeing it right now.

WILL: Well, yeah. I think we're all seeing it. I think we actually are vigorously agreeing because, like --

DAVID: Yeah, violent agreement, yeah.

WILL: We all understand that this is maybe a guideline, but not a hard and fast rule, which is why it would be totally fine. And you were like, "Hey, I need to borrow a couple of bucks for lunch." And Eddy was like, "Yeah, yeah, yeah, just go grab my wallet out of my jacket. It's over there," right? Like, obviously, that's not a blank check to just get into Eddy's pockets anytime you'd like, but, like, reasonable people can make exceptions.

And the thing that I like about not doing these bidirectional relationships by default is it sort of puts guardrails on the relationship where you need to do it on purpose. Like, I'm doing this on purpose because these things are so coupled, and I actually think shopping cart to customer might be. In the end, it might be one of those such tight relationships, where it's like, yeah, we should probably go both ways.

MIKE: Well, it depends on your business.

WILL: Yeah, exactly. But, on the other hand, I don't trust you guys with nothing. Like, a big piece of my life is just hiding the sharp objects from the junior developers.

EDDY: Jeez, well, I don't know, man, if someone's reaching into my pocket, I want to be able to reach into theirs, too, you know?

[laughter]

DAVID: That's a bidirectional relationship, yeah [laughter].

TAD: That's a violation.

WILL: [laughs]

DAVE: I just want to say I'm so happy to be on this call with y'all and to not be the person who launched the sentence about reaching into somebody else's pants. That's normally Dave Brady brand but thank you. You did give me a...I'm sorry, go ahead.

MIKE: If HR might need to get involved, then it's a violation of best practices.

DAVE: Yes. Yes.

WILL: That's why I work remote, man.

[laughter]

DAVID: I had a light bulb moment when we were talking about this, that if customers and shopping carts are coupled...and stop me when you've had this...we've had this pain. Everyone in this room, I'll wager...I know, maybe not Will, and maybe not Kyle. I don't know how much coding you're doing lately. But everyone else on this call, I know you've had this problem where it's like, oh, I need to, Will talked about the shopping cart, I need to calculate the shipping and, in order to calculate shipping, I need to know how many boxes and how big the items are. So, what boxes so that I've got to do the packing on it, and then I need the weight on this.

We can go to a shipping calculator. It needs to know what state we're going to send it to, and that's going to come from the customer. But other than that, the shipping service doesn't need to know anything about the customer. It just needs...and, honestly, the only thing it needs to know from the shopping cart is what items are in it. It needs an inventory list.

And if you can say, "Here's the state, and here's the list of items," that service now doesn't need to know anything about the shopping cart or the customer. But we've all written specs, where you're like, okay, I want to do this. Oh, I need...this is a shopping cart. Oh, I need a merchant. Okay, in order to have a merchant, I need a location. And to get a location, I need a merchant, to get a merchant...oh, I need the legal paperwork that says, "The merchant has been onboarded and has signed the paperwork." Because they're a valid merchant, they're supposed to go through the system, so they're going to be checked at every point.

EDDY: You don't have to worry about that if you use factories, Dave.

DAVID: Right? Exactly. Exactly. So, just, you know, we'll just do everything with factories, put it all in the database. And do you want half-hour spec suites? Because this is how you get half-hour spec suites [chuckles].

WILL: And also, the other thing about shipping is like, maybe it's not for me. And I also said it, like, because I could speak as a large electronics retailer. We do a lot of shipping, and let me assure you, shipping is much more complicated at the state. There's lots of places [inaudible 44:17] you can get stuff the same day, and there's a lot more places in Utah you cannot [laughs]. Like, my dad lives in Michigan, but he lives in the part of Michigan where Prime is not a thing. You'll get it next week, period [laughs].

DAVID: The last mile problem we were all talking...like, in 2008, we were talking about the...whoever solves the last mile problem...this is before Uber existed, right? I might have that year wrong, but, you know, before Uber existed, solving the last mile problem was the reason why there were only three big, you know, FedEx, UPS, and DHL were it for shipping stuff, and now it's all super federated, and everybody's distributed.

I tracked a package on Amazon yesterday, and I saw a new status update, which said, "Delivery appointment scheduled," And I'm like, "What?" I'm used to carrier picked up package, package in transit, received at facility, left facility, you know, out for delivery. That's all you need. And, all of a sudden, we've got a delivery appointment. And I'm like, somebody's working on that last mile problem, and it's not well...what is it? The future is here; it's just not well-distributed, right?

There's still places...there's a place in Nevada that does not have cell phone services. There's only four pairs of copper going in and out of the town. Great Basin National Park, that's what it is. There's a little town there and a little national park. And yeah, I remember pulling in there, and I'm like, oh, my phone doesn't work. Let's have a modern horror movie now. We've now solved that problem.

WILL: Yeah. I mean, Great Basin, I want to write that down. Like, I want to go camping. I only want to go camping places if there are no bars.

DAVID: It's a...not to turn it into an ad for this, Great Basin National Park is a glacier in Southern Nevada, in the middle of the desert, right adjacent to the Mojave Desert. It's a glacier because it's on a great, big, volcanic talus cone. And you're just driving across the desert, and then, all of a sudden, there's this mountain sticking up out of nowhere, and there's a glacier on top of it. It's crazy. And Lehman caves, which is gorgeous, so...I may have vacationed there recently.

WILL: I'm underlining it [laughs] --

EDDY: So, I kind of want to...Sorry, Will. I just kind of want to go back to where we initially started. So, I guess, essentially, there is no rule of thumb to follow, right?

DAVID: Maybe not a rule of thumb. Well, there's lots of rules of thumb. You just have to remember that they're all based on principles, right? You can distribute something. And if you distribute it in such a way that the distributed nodes have very slow communication between them, and they don't do very much communication, you're great. You can do all your high performance in your silos now, and everything's great. And if you do it wrong, you're going to take the high-performance stuff and then stretch it over a boundary, and now everything is over HTTP that used to be in the CPU cache. Ugh, been there, done that.

WILL: I don't know, I've got a rule of thumb. I've got a rule of thumb in terms of, like, monoliths versus distributed services. And this is going to be on the record hot take right now: You should keep it as monolith for as long as possible.

DAVID: I agree.

WILL: Because, as I see it, the fundamental trade-off that you've got between monoliths and distributed services is you're levering engineering output versus sort of management and the ability to communicate, you know what I mean, to coordinate and keep everybody pulling in the same direction and keep the children from fighting with each other because it's just natural human nature. And, in the end, this is probably my engineering bias, but I've run into a lot more effective engineers than I have effective engineering leadership and [inaudible 48:12], you know what I mean? I see you, Kyle. You can hide behind your hand all you want, but I saw that [laughs].

EDDY: Isn't there a cost, though, to having a monolith, though? Like, if you say, like, "Hey, no, like, I have a monolith that was the first service that we developed for their startup. It's been developed actively for 10-plus years," right?

DAVID: Sure.

EDDY: And you're like, oh, go and develop for this feature, right? Suddenly, isn't there, like, a higher ceiling to come in and get acquainted with, versus if it had a microservice, a true definition, to Dave's point, to a microservice, right? Wouldn't it be easier for you to hit the ground running because you know how to scope or comb through all this [inaudible 48:57]?

TAD: Well, the company I worked for before, and it's out of business, but I won't say their name anyway, they had the entire company was an entire single Go codebase. And you're talking 60-70 developers all working on a codebase that has to compile. And so, every subdirectory had a README file, and it's like if you want to touch the code in this folder, you got to get sign-off from these three people [laughs].

DAVE: The stakeholders.

TAD: And that's how they --

WILL: Stakeholders, David, yeah.

TAD: Like, took care of it. And it's like, oh my gosh, like...

WILL: So, what you run into doing that stuff is you'll have these sort of rather than management fiefdoms, you'll have these engineering fiefdoms, where you have this increasing important hardcore set of engineers that know everything and everything has to run through them. It's just like parallel programming.

TAD: Well, I'm just saying, like --

WILL: People will max out, and you'll feel it, you know?

TAD: It felt like the worst parts of a monolith, where I want to change these three files, therefore, I have to get, like, eight people to sign off on it. Oh, two of them are on vacation right now, well...Whereas if it were a single service with five people, any other four people on my team could probably sign it off, and we could roll it out that day, right?

DAVE: Yeah. To Will's point, there's an agile principle, which is to try to defer your decisions as far as possible. The idea is to defer the decision to the point of maximum information because the information increases monotonically over time. So, if you make a decision early before you've got all the data in, and that decision commits you to a path, and then you find out you were wrong, it can get very, very expensive.

In SOA, once you split, if there's a decision that has to be made on both sides, that affects both sides, that's twice as...well, more than twice as expensive, probably four or eight times more expensive because they both have to change, and they both have to communicate. We've all had that thing where you have to do a tandem deploy because my service wants to change this, and it'll crash you if I do it before, and da da da, and that whole game.

So, to Will's point, yeah, I think you should start small and start monolith, until there is pressure to do. At Acima, we have a great, big monster service that has been calving icebergs off of it. It's not even a monolith now. It's a continent on it, and it's just, you know, calving little subcontinents across. And that's a great way to do it. I don't know if it's a great way to do it. It's exhausting, and it's painful. And we have architecture meetings every week where we scream about it. But it worked. It started small. It got big. It got too big, and then we broke it down.

And if we had started out eight years ago breaking things into, okay, you're going to be this service; you're going to be that service...the first time we tried to change a low-level thing, like, the first time you try to move outside the United States to do business, and, all of a sudden, you need postcodes instead of zip codes, right? If you have to change that in every single service and there's validations that are hard that will crash your service if that's wrong, in every single service, it's going to be a nightmare. If it's a monolith, just update the zip code table, right? Just, right? Less expensive if it's all in one place. Yeah.

TAD: How do you know when to start slicing off baby services from your service? Is it complexity? Is it team size? Like, what's your rule of thumb?

DAVE: I let pain be my guide. Like, when something hurts long enough, when something hurts, I kind of make a mental note, and if it hurts twice or three times, I start going, maybe we should fix this, right? I personally am constantly driven to just polish out the tracks where I'm frequently going. So, if I'm doing something, like, pairing with somebody and I opened up a PR, and then I went back to my terminal, and I typed git set PR number, like, why did you do that?

And I switched over on another branch, and I did git open PR. And it opened a browser to the pull request for that thing. And I'm like, well, it's because I wrote a little thing that ties branches to PR numbers. And I'm like, yeah, that's hard work to do that. I'm like, yeah, but it's really painful to not have this. And, for me, it was less painful to stay up over a weekend writing a little PR management, you know, a little number associator.

WILL: I think if you talk to your senior engineers, you'd be like, "What sucks?" Like, everybody –-

DAVE: You'll get answers [laughs].

WILL: Yeah. And they're going to be...I'm saying there's a whole, like, how do you chip away your icebergs? That's a [inaudible 53:56] question that we could spend a series of podcasts on. Honestly, I'm more interested in, how do you maintain the communication and collaboration among teams? Because I have seen the distributed services architecture is a tremendous leadership challenge.

And I've seen so much just...we could just call it deadlocks, just like parallel programming. I figured it exactly like parallel programming and sort of deadlocks, and race conditions, and stuff like that with these sort of distributed things. It becomes a leadership problem. I mean, everybody knows the reasons why you should start branching out services, why you spawn a new threat. But how do you keep these things together? How do you keep the teams working with each other, right?

DAVID: Let's do that next week. I think that could be a whole call. How do you keep parallel teams productive? Yeah.

EDDY: I guess to kind of –-

WILL: Absolutely. To keep people from fighting, you know, keep people accountable.

EDDY: I guess the answer really is...it was a long-winded conversation to just say, "It depends."

DAVE: That's the other name of this podcast, a long-winded conversation to say, "It depends," yeah.

WILL: Well, yeah, I mean, it always does. There's no right or wrong in engineering. It's just the right tool for the right job. And, believe me, there can be a wrong tool for the job, and you're going to feel that [laughs].

DAVID: Mm-Hmm. Cool. Well, we have lost our host. Mike got called out to a production issue. Thank you for listening and watching our podcast. We appreciate it.

In this episode of the Acima Development Podcast, Mike shares a personal story about his father, a custom staircase builder, to illustrate the importance of balancing speed and thoroughness in work. Mike’s father, known for his meticulous craftsmanship, often took extra time to perfect even the smallest details of his projects. Mike contrasts this with a quick-fix job he once helped with, where speed was prioritized over perfection to meet a deadline. Both situations were appropriate for their respective contexts, highlighting the need to adjust the level of care based on the type of project at hand.

The hosts dive deeper into this theme, exploring how the right balance between velocity and thoroughness can differ depending on a company’s stage of growth. Will points out that startups often need to prioritize speed to survive, while established companies must focus more on quality and performance. They discuss the challenges of maintaining speed when stakes are high, particularly in large organizations where even small inefficiencies can have significant financial impacts. The conversation emphasizes the importance of adapting processes and expectations based on the specific goals and constraints of the work.

Finally, the episode touches on the crucial role of communication and knowledge sharing within development teams. Pair programming, mentoring, and asking questions openly are all identified as valuable practices for ensuring that team members can move quickly while maintaining quality. The hosts stress that leadership should stay involved in the coding process to maintain a connection with the work, encourage a culture of learning, and foster an environment where mistakes are seen as opportunities for growth, ultimately helping teams strike the right balance between speed and quality.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I am hosting again today. With me, I have Dave and Will.

DAVE: Howdy, howdy.

MIKE: I'm going to start with a story, actually, well, two kind of connected stories. I'm going to talk about my dad. He's a builder. Well, he's retired, but [laughs] he spent most of his career as a builder. And most of what he did was custom stairs. Most people don't want custom stairs [chuckles]. It's the people who can afford the custom stairs who ask for the custom stairs. So, he did a lot of work for, you know, people who can afford to pay to have custom stairs done. And the people who pay to have custom stairs done are looking for a certain kind of work, right?

WILL: Gone with the wind, grand entrance, duh. Yeah, I didn't get the Scarlett O'Hara dress for nothing.

MIKE: [laughs] So, circular stairs, you know, he did the spiral stairs, and that was kind of his specialty. And it's not just my dad; he did it with his brothers. He's got two brothers he did with. And they, you know, built stairs for years. And in my...well, I have personal experience. So, my dad built a house to sell, and this was in the late '70s. Everybody knows a big economic issue happened then, and people weren't buying. So...

DAVE: That was like the gas crunch, right?

MIKE: Yeah, exactly. So, the house didn't sell. He ended up moving in, and he's still there. [laughter] So, he probably over-housed, but he's been there. But, you know, he just can't...he spent his career doing this custom work, this custom, you know, the best quality you can get, right? And he's continued to work on the house. Can't help himself [laughs]. And so, everything he does, the floor and the kitchen, it's amazing. But you wouldn't pay for it because you couldn't pay for it.

He found an apple orchard where they were getting rid of their trees, and he cut the trees into layers and dried the wood, and then sanded it down, and cut it into the right shape. Well, cut it into shape, then sand it down because it warps after you've cut it. And so, it had to be cut thick, and then process every one of those pieces to get it to the right shape. And it's like every one is, like, at the heart of the tree. Once it was sanded, laid it all down, put epoxy on it instead of, like, your standard polyurethane, so it will last forever. It is just absolutely stunning.

But it's the thing you can only get by doing custom work, right? And when I've helped him...and I've helped him a lot. I've done some work there, and I, of course, worked with him when he was working outside the house, and [laughs] his standards are exacting. He's not a jerk, but [laughs], you know, he does quality work. And people work for him; he wants that work done.

And so [laughs], the work is great. Everything he does is done, exacting perfection. A lot of the work he does himself because he knows he's the only one who will give it that kind of attention [laughs] to actually get it done the way he wants it done. The one time when I've been doing a lot of work with my dad, I went and helped a cousin of mine who was moving out of the trailer he'd been living in. Though it was kind of a dump, and they wanted to make it look nicer before they moved out.

And I remember I was putting in some molding along the floor, just some cheap, little, thin stuff, and they asked me if I could do the corners. I was like, "Yeah, sure." And the speed at which we did the work in that trailer...I was getting together some, you know, some other family folks. And we got together and quickly did a whole bunch of work, you know, got together in three hours and redid all kinds of stuff in this trailer.

The floor was totally out of true, right [laughs]? So, trying to get that corner right was nearly impossible. I think, eventually, I put some sand under the linoleum to get it [laughs] leveled so [laughs] [inaudible 04:11]. And we got it all together. It looked a lot nicer, so we could sell it. And we were done. And here's the thing: in both situations, the right work was done. If you're going build a custom stair for somebody who wants a custom stair, well, you're going to give that every little bit of attention to detail that it deserves. If you were getting that trailer ready to go so you can quickly sell it, you're going to do that quick because that's what it deserves.

And that's not to say that the trailer doesn't deserve care, but it would be incongruous [chuckles] with the rest of it. It wouldn't be worth the value of that whole structure. You know, there's people who actually make a small home look really nice. I've seen that before. This one was not. This was not one of those homes [laughs]. This was not [laughs] one of them. But both of them were right.

There's a balance that you have to strike based on the work at hand, and sometimes you got to get that done. And, in fact, most of the time, the work we do is probably more like that trailer, right [chuckles]? We need to get the work done. And I'd say this: I say more like, I don't like to do shoddy work. I want to do quality work, but some things need to get done.

But then there are other things, you know, if I'm building something that has anything to do with money [chuckles], I want that to be done right. And I want somebody inspecting that [laughs], going over with the white glove to make sure there's no dust on it, you know, because you do not want to mess that up. And, you know, there's a time and a place for both.

What we're going to talk...I'm leading with a story. I haven't even talked about our topic today. We're going to talk about the trade-off between velocity, you know, between working fast, getting things done, and being thorough. And I thought about, you know, the stories because it is a trade-off. If there was a clean answer, then we'd all be doing it.

And the problem is we often tend to err on one side or the other, right? We tend to maybe go the wrong way for the work at hand, and we get in trouble. And sometimes, you maybe get...and we've probably all worked with somebody who had the work to do, and then they just didn't do it at all, you know, they're off [inaudible 06:37] [laughs].

I did some construction work in New Orleans area. I used to work for a construction company in New Orleans. And there was one job we'd worked on it for months, and, basically, it was done, right? It was done, but we weren't ready to move on to the other contract. And there was a couple of days where there was nothing to do. So, they said, you know, "Go around and look busy." I spent two days polishing the outlet covers [laughs]. So, when somebody came in, it looked like we knew what we were doing.

And there were some people who probably spent more time polishing those outlet covers on other days [laughs] than they should have. There's that problem, too. I think we may delve a little bit into that, but starting with the topic of balancing this work of, you know, this balancing between being thorough and being fast. I've been talking a lot. I heard that Will had a story. I'm interested in what y'all's thoughts are and how you'd link it up.

WILL: So, more I have a thought to share because I think a big piece of the thoroughness is sort of the phase of the company that you're in, right? But when you're in a startup and you're throwing features out the door as fast as you possibly can, and maybe you're going to take down the app; it'll come back up, there's not that much...you have different set of risk, right? Like, the risk you have now is you're fundamentally insolvent. You're going to run out of money. You're going to run out of time. And you have to figure out how to make a product that is worth buying. And that is your existential risk.

However, if you're working for a large, multi-billion dollar company and you screw something up, and your page loads 5% slower, you know, there are direct correlations to sort of website performance and click-through rate, and conversion, sales conversion, right? Like, Amazon did studies on this stuff, and if your stuff is too slow, people don't buy. If your site is not fast to navigate around, people will leave. And so, that stuff is really, really important.

And so, I'm talking about sort of a thoroughness-fitted finish kind of a thing. I'm not even talking about your stuff crashing. I'm talking about your stuff not being fast enough. It's like, oh, well, what if I lose five basis points worth of sales because I just didn't do as good a job as maybe I could have? Five basis points. A basis point is a percent of a percent, right? I just burned my salary for the year, just like that.

This is the phase of the company, and there are indisputable properties of just having a going concern that makes a lot of money. The impacts to the bottom line are substantial. You do need to be thorough, and I would...I always advocate for my developers. Like, if you've got a billion-dollar business, you should pay more for the people working on your site because it makes a difference, whether they're good or bad.

And one of the things...and maybe I'm going to answer your question with a question, or maybe I'll just add on another one. How do you maintain velocity in a situation where you have a big enterprise; you have a going concern; you have stakes? If you screw something up, how do you keep that velocity? Because it's easy to analyze yourself into a state of paralysis. You still need to innovate. You still need to develop products. You need to be, you know, delivering results on a quarterly basis. Because, on every level of the company, they demand that, and that is what you're getting paid for. How do you balance that, right? Because both things are true. You can't screw up, but you got to get things out.

MIKE: That's a good question.

DAVE: I keep coming back to this idea of, what are you really trying to do? What are you trying to accomplish? If you're in a startup, you're not going to be taking on IBM or Amazon. You're not competing at that level. And there's a phrase that you'll hear a lot which is, "The ideas that got us here won't get us there," when you're trying to scale. And the ideas that got you out of startup mode into productivity those ideas will not get you to scaled enterprise-level stuff. But the reverse is true.

 When you transcend the old ideas, don't discard the old ideas because what gets us there won't get the next cohort here. You end up transcending, and then pulling the ladder up behind you. I don't know if I'm making sense with that, but it's like, you start saying, "Hey, we need to have SOX compliance. We need to have peer review. The person who writes the code should not be deploying the code."

And then, you bring in a scrap team that are going to do a skunk work thing of like, "Hey, we want to stand up a quick, little server that will just check this thing on this rating service." And you're like, "Okay, cool. Where's my SOX team? Where's..." no, no, no. You have 30 days to ship this, and the right solution is to just ship it. Cut features until you ship. When you have to play at that level, like, if that little service is going to connect to the accounting database, okay, no, you cannot ship this in 30 days. That is not a working solution. But if you're just trying to do something like, oh, I just want to check your membership loyalty points, that's easy. It doesn't need auditing. It doesn't need review. Just ship what they need.

I'm sure you've all heard this. Growing up, I grew up in a kind of rural, you know, redneck part of the country. And I always heard all the time, "If you don't do it right the first time, when will you have time to do it over?" And then, I got into the software business, and I realized, no, the mantra in startups is, if you don't do it wrong fast, when are you going to get the cash flow to fix it and do it right?

And so, yeah, the ideas that get us to here are not the same ideas that will get us to there. And you need both, but not at the same time. You have to stay alert to, which mode am I in? Am I in enterprise, secure mode? Am I trying not to lose the eggs that we've collected into our basket? Or am I desperately trying to forage quickly and get as many eggs into the basket as I can, and then we'll sort out if I accidentally stole your eggs, or whatever; we'll sort that out later? But at the enterprise level, I don't need that level of security. I have to get the eggs collected. That's 17 metaphors rolled up in a weird ball, but there you go.

MIKE: You know, you're talking, Will, yes, you know, how do you keep the velocity going when you have those constraints? Another story [laughs]. I ride bikes a lot with my kids, bicycles, not motorbikes, but, you know, pedal-powered. And I discovered something great [laughs], and it took me years to find this, and I'm glad I did.

You can attach a tow...it's not a rope. You can attach a tow cable. Basically, it's a bungee cord that's wrapped in ripstop nylon that's been scrunched, so it's tough. You got the bungee cord inside. So, you don't have all the problems with a bungee cord, where it's going to have that hook, and it's going to tear your eye out [laughs]. But it, you know, it expands and contracts.

So, I attach it to the back of my bike, and then I attach it to the bike of one of my kids, right? And then, I've actually got two of them, so then I attach it again in a daisy chain [laughs]. And I've got my three-year-old that I'll put on a seat on my rack. So, I've got three kids, you know, all in a line behind me, and I'll go for a ride. And I've been doing that a lot over the last year.

And I'll tell you, the first time was hard [laughs] because that adjustment from riding, you know, riding with no constraints on you to riding with three people pulling behind you is a shift [chuckles]. And I even had to make a change. The bike that I was riding a lot...well, I've got a fat bike. So, I've got fat tires that are, like, four and a half inches wide, so I can ride in winter right on the snow, packed snow. It still doesn't work in too deep snow [chuckles]. And I've done a lot of riding on that and some mountain biking stuff. Even though it was made to go up some steep stuff, it wasn't geared low enough. So, I've got a gravel bike that's got a...lots of stories about bike, but we need it for context here [chuckles].

DAVE: It's okay.

MIKE: It's got a huge gear ratio. It's got a gearbox. It's got a huge gear ratio. And I can go really slow. And I can go pretty fast, too. But I can drop that thing down into an incredibly low gear. I have never gotten to the bottom gear yet, even pulling three kids, even going up a steep hill. I've never gotten to the bottom gear. I rarely get below...like, the lowest I ever get is, like, fourth [chuckles] out of 12. So, I'll give you a sense that, you know, this thing goes way low. It's awesome.

But [laughs] I had to change the way I rode my bike so I could keep moving forward, right? I had to change. And then, the reverse is also true. On the occasions I go out alone, I, like, almost fall down [laughs] because I'm used to having all of this mass behind me. And I have to rethink my riding style. We've got the same kind of thing, you know, thinking about this velocity. One time, I went on a longer ride with my kids, and I went up this, like, mile-long hill. And there was a headwind, a really strong headwind. And I was going mostly up that hill at, like, between three and five miles an hour, so, basically, at a walking pace.

DAVE: Just fast enough to keep the bike from tipping over.

MIKE: Exactly. Fast enough to keep the bike from tipping over. And if you were trying to do that, like, most people on a bicycle would just agonize over that because you're just not getting...that spot in the distance is not getting there very quickly. But the important part was not to get there quickly. I had a headwind. That was just the reality, right? I'm going up a hill. That's the reality. I've got all this weight, and this is the reality.

The important thing is that I keep pedaling. And as long as I'm keeping on moving, you know, your brain, you can adjust to it, right? You can adjust to these new constraints, and you just keep on going, right? You're not going to go as far. You're not going to go as fast, but it doesn't matter. The important thing is that you don't stop. Because you could think, well, I'm not going very fast. What difference does it make if I just stop? Well, it makes a huge difference [chuckles]. Stopped is very different from moving forward.

We're going to be in different situations, right? Sometimes we're going to have the headwind. Sometimes we're going to have a bunch of constraints we have to deal with, a bunch of weight we pull along with SOX compliance, you name it. But you can still set up the processes and run basically the same, right? If you're moving forward, you're moving forward.

The key thing is that you never get in that situation like, well, we're not going as fast as we were over there. So, what we're doing doesn't work. We've got to break everything, either by stopping because, like, well, this doesn't even matter. Very bad choice. Or saying, "Well, we need to completely reorganize our process. We need to get our velocity up to exactly this." Well, that's not realistic either. Because if you've got all these constraints, I think they're there, and you got to just deal with that. And you can adjust to it. Like, you can adjust to a lot, as long as you're keeping on moving. And it's really critical not to try to compare different kinds of work because they really don't correlate.

DAVE: You just connected two epiphanies that I've been carrying around for 20 years. This is exciting. When you mentioned that the important thing is to keep peddling, to keep moving, I had a job very early on in my career. I was moving out of, like, minimum wage jobs and into, like, my career. I was actually writing software for a living. I was on the networking side, so I was pulling cable as well. I was literally, like, crawling around behind people's desks and dragging cable. And if there was a hardware problem, I had to deal with the hardware. I ran a soldering iron at one point, right? I was all over the map.

And my job was to be the network guy. My job was not to break down boxes in the shipping room. My job was not to, you know, manage the sales cycle. My job was the networking stuff. And I came in and the stockroom in the back was getting piled and piled up and piled up with supplies of stuff. And I was in this mentality of, hey, I got to get this network stuff. It's not my job to do this.

And I walked in, and my boss was in the back room cutting down boxes and breaking everything down. And I'm like, "Wait a minute, if it's not my job, it's really not your job. What are you doing?" And he just looked at me, and he's like, "We need the space in order to get our job done. I'm an expensive stock boy, but I'm going to get it done."

And that was the epiphany for me of; sometimes the objective is, am I doing the job that I'm supposed to be doing? And sometimes, the focus is what do we need to accomplish? The how doesn't matter, but the what is what matters. So, I've literally carried that around in my head: I'm an expensive stock boy. I carry that around, and it has availed me very good over the years.

There's been times when it's like, there's this big, messy part of the project that nobody wants to touch, and I don't want to touch it either. But then I look, and I'm like, well, that is between us and done. So, I'm going to go be an expensive stock boy and just take care of it. And, usually, somebody higher up on the chain will go, "Ooh, that's an expensive stock boy. Let's get a stock boy." Or they go, "Yeah, we just needed that room cleaned out once, and it needed to be done. So, that was the right price to pay for it once."

WILL: I think that's a big...I've seen it a lot as sort of like, you know, teams specialize, right? And they sort of fragment into this archipelago of little islands, you know, that make up this country. And then, you have, you know, directors and VPs and stuff that they have their little region, their little fiefdom.

And I think that is one of the hardest things in terms of scalability, in that you still need people who take ownership of the entire organization, the entire app, the entire end-to-end experience. And, honestly, like, one of the things...I was just talking about this today. Like, where I think that breaks down really badly is sort of in the corporate hierarchy, like, the people you'll have are, like, VP or a director that isn't in the code every day, doesn't know where the bodies are buried, doesn't know how all these things interoperate. They have the visibility, but because they're not in it every day, they're very detached.

And I think there's a little bit of a black spot in the industry for somebody who is VP level or whatever your broad organizational level leadership. But they're an engineer. They're doing work. They will cut down boxes. Oh, this ticket was screwed up? I'm going to clear the ticket, and if you can't...I mean, bluntly, I know a lot of smart people, and I know a lot of smart people who've moved into management, you know, leadership positions. But you have to pay your dues. You have to pay the toll. If you want to ship code, you got to ship code. You got to put in your [inaudible 22:46]. Nobody gets an out.

If you are rusting, you know, then you're rusting. I mean, just like after a year on the bench as the manager of the team, like, you're going to have to ramp up just like a new hire would, maybe not that bad. I've seen it fixed places, you know, with things like principal engineers and stuff like that, but it's a systemic problem. And I haven't really seen a systemic solution other than senior, senior leadership recognizing the problem and being proactive and making sure those people are there and empowered.

Because a lot of the stuff you'll see is, they'll go around peeing in some Cheerios. There are a lot of people who have pet projects. They have things that they're doing. They have deliverables and stuff. And somebody comes in and says, "Hey, like, the interop between this department and this other department is all screwed up. Our codebase is all screwed up. We have technical debt. We've got performance and security considerations. And I'm going to have to like, you know, I'm going to screw up your quarterly results because this broad thing happens at like..." you know, there are a lot of smart people with interests in kneecapping those kinds of systemic engineering initiatives and accountabilities and stuff like that, you know, it's a ferociously difficult problem.

And I was contending with that earlier today, like, wanting that person in that role to come save me like Superman. I need heads cracked, and I don't have the juice to call these people, you know, behind the [inaudible 24:44] I can't do it. I know it's screwed up. They know it's screwed up. Anybody who's looking at it knows it's screwed up, but, like, the people that can can't [laughs], you know what I mean? I suppose a tangent, but a thought that was on my mind.

MIKE: Well, you said something. It's something I've thought about a lot. I've heard a lot of times, people who do hands-on, you know, typically blue-collar work, working with their hands, talk about how much the leadership doesn't get it. Like, "They just don't understand the work we do." And I can't count the number of times I've heard that.

And I think it's generally, like, I almost take it as a truism because I've heard it so many times that if you have somebody who hasn't done the work or is not connected to the work, they're going to miss things that are important. They're not actually going to understand what's going on. And, to your point, it doesn't matter how much you're looking at it if you don't know what to see. And it seems like there is a critical need for leadership to actually have either gotten their hands dirty or be willing to listen to the people who have and actually respect what they have to say.

WILL: Well, I don't want to be a jerk about it, but I will say this directly. You got to keep on paying the toll. Rust never sleeps. Every...I don't want to say every but, virtually, every senior engineering, leadership, management person that I have worked with I know in my bones started out as an extremely competent individual contributor. At one point, they were all good. They were all really good. I mean, once you get to the director and above level, present company accepted; once you get to that level and above, it starts to catch up with you because you know what your schedule looks like. Mike's shown me his calendar.

MIKE: [laughs]

WILL: When are you going to get your hands on the iron and do it? It's a luxury. That's like a day off. And if you don't do it, it catches up with you because everybody has to pay the same toll. If you want to be good, you have to continue sharpening the saw. Rust never sleeps. And that's when you run into these sorts of difficulties, the very difficult line to walk.

DAVE: It works backwards as well. I remember the first time I worked for a manager who had a string of managers in the early noughties and mid-noughties. They were, like, senior developers. And cowboy coders didn't get along with anybody, but they were very, very good. And they ended up in, like, CTO positions, and nobody wanted to work for them, right? Because they had no people skills. And the ones that had people skills weren't necessarily applying themselves to what type of battles do I need to fight at this level? They were still trying to focus down here.

And the first time I worked with a manager who came in and said, "Yeah, I used to be a developer, but now I'm a manager, and this is my job," and, like, the first three or four months working for him I hated because he would actually make me do one on ones.

WILL: [laughs]

DAVE: And I'm like, no, everyone knows that one-on-ones are uncomfortable, and nobody likes doing them. And after three or four of them, I'm like, this is really productive. I actually feel like I'm driving my career in a great direction because I'm actually doing this.

And that was the time that I finally realized that...and it goes back to what I said earlier. The ideas that will get us there are not the ideas that got us here, right? You need that reference. You need that empathy for the people working under you. And you need to be able to look beyond and say, "What..." It's the same thing. If you're not focused on what it costs to write code, you're going to have some struggles.

But if you are the manager and your job is to be aware of like, oh, AI is coming, and we need to be ready, or we're going to wake up, and our industry is going to be gone, you know. And do we want to have gone with it wherever it goes? And, I mean, that's one example. Somebody will react too strong to the mention of AI, but you get what I mean, right? There'll be something that, like, we're going to do this initiative.

We talked about this in a meeting earlier today about a manager who said, "We're going to pursue this initiative." And it was the wrong initiative. Turned out we got there, and there was no opportunity there. And it cost some people their jobs as a result, and that was no fun. And it was failure in both directions. Like, this person didn't understand the engineering, and they also didn't understand...they had a view of what ecosystem they wanted to put in place. And they managed to solve the wrong problem in both directions.

WILL: [laughs]

DAVE: Somebody said, "We're leveraging the inefficiencies of both approaches. [laughter]" That was, like, anti-synergy.

WILL: That's almost impressive.

DAVE: And I've been there. I've been on a team where I had a cowboy coder working with me, and I was, like, the meticulous, make sure your tests pass; make sure your code is expressive; make sure to actually dah, dah, dah. And there were times when I'm like, "You're going so fast, and you're breaking things." And he was like, "You're dragging an anchor when we need to get stuff done." And there were times when we got so much at loggerheads that we ended up shipping too little, too low quality and too late because we leveraged both of us. We both dug in our heels, and we ended up with nothing.

And I think I've told this story here, but I worked with a guy named TJ at CoverMyMeds who was astonishingly good at picking his battles. And he and I were really good friends, and he got really, really good at seeing when I was about to turn and go down a rabbit hole. And he would just kind of put his hand on the back of my belt, and say, "Whoa, whoa, whoa, whoa, wait, we don't need to go down that rabbit hole." And we fought a few times over [inaudible 30:55]. And I finally said, "You know what? You're right. We don't need to go down that rabbit hole."

He was one of the first people I've ever worked with where pair programming delivered on every promise pair programming has ever promised anyone. And we were also linearly faster. Like, the two of us could complete a sprint's worth of work for two people in, you know, a week and a half. We literally...every way we went faster because the synergy worked so, so well because we were solving the right problems, and we were leaving the wrong problems alone. And that...oh, so good.

WILL: I'm a pair programming seller, and I always evangelize pair programming.

DAVE: Amen.

WILL: I think it's good. It's not a silver bullet. It shouldn't always be everythings all the time, but I think it should be...I think we should do a lot more of it. You know, pair programming now, pair programming, you know, forever. Like, that's all. That's all I'll say on that. That's all I'll say right now.

[laughter]

MIKE: I got to say, it's one of my favorite things to do because the amount that you learn on both sides of a pair [inaudible 32:11]... a lot of times, you have a more senior, a more junior person, but it tends to flow both ways. The amount of knowledge transfer that happens there is unlike anything else that I've experienced in software development.

Like, you can read documentation, and a majority of that is just going to pass right through your brain [laughs], not that it's not technically possible to go read some dry technical documentation and get lots of clarity from it. But when you're watching somebody work, or they're watching you work and seeing how you make each of those decisions, it just changes that equation typically. You see all that weren't captured in the documentation that you just wouldn't see otherwise.

WILL: Yeah, well, I mean, and if it's your stuff, you know you didn't write everything down. You know you didn't do it. But, anyway, a lot of stuff you're talking about, I really do feel like a lot of those that, as line-level managers, I'm going to say flat out, I'll take my hot take; this is your hot take of the session, I think all line-level engineering managers should spend 25% of their week, maybe 20%, right? Eight hours a week, a full workday, every single week should be pairing with the people on their team so that they have both an understanding of like, okay, here's what's going on.

If you're a line-level manager, you should be a very proficient coder. You should be a senior-level engineer. You should be good at that. You can't be like, "I'm a VP. I don't care anymore." No, no, no, you don't get that out anymore. You should be providing that kind of judgment, that kind of insight, that kind of understanding. You should be working with your team and figuring out where everybody is. Don't [inaudible 34:15] like a one-on-one. Get in there and feel it.

All right. Are you writing tests good? Are you thinking about parallelism? Are you thinking about error handling? Are you thinking about design? You know, all this stuff. You get paid for that. That's why people pay you to be around. And it keeps your hands, you know, in the iron, and you have a feeling for how things are going.

I'm actually on a Slack thread right now with three, four staff engineers who are like, "Why are my builds so slow on these laptops?" And I have been fighting the good fight. There is a security team that has installed real-time virus checking on all engineering workstations. And it is virus scanning intermediate build products in real-time. It's a substantial CPU hit. I have been wrestling with these guys for a long time, many months. And these guys didn't even know. They're just like, "Wait. Wait, what? It's doing what?"

And I'm like, "Yeah, man." I've been arm-wrestling these guys to try and find a more economical accommodation to satisfy their security concerns while I can still ship these features. And if you never code, okay, you know what I mean? You're a director. You're a VP. You're up the levels. That's not actually your job anymore. You probably are wasting your time doing that.

But, like, line-level managers, I want your hands...I want you in the muck with the rest of us, at least a little bit, so you know what the hell's going on. Because even when that front line, you know, your NCOs or whatever you want to call them, you know, the people who are on the front line, if they've checked out, how's the information going to filter up? You know what I mean? Anyway, my two cents.

DAVE: It's how do you get feedback? Yeah.

MIKE: I've got one pushback against your hot take. I don't think 20% is enough [chuckles].

WILL: Yeah, sure. I mean, you're right. I agree. I think 25%...I think 50% is probably where I would cap it, but it's somewhere in between there. But I don't think...how do I put it? Like, I'm not asking for 30% and expecting to get 20%. This isn't like a negotiation where I know I'm not going to get everything I want. Like, I'm just saying what I need. This is the real number. I want a real 20%, and getting 8 hours consistently every week out of a manager's schedule to pair program is a substantial ask. That's hard to do. You need support. That is a rare bird to make even that happen.

MIKE: I stepped into leading an internship once, where the person who was leading it...I say an internship, it was an apprenticeship. There were some people that we wanted to bring in to be full-time engineers. They weren't coming straight out of college. These were generally people who'd maybe been in another profession, and they were ready to come and learn what they're doing. And I'm not the one who started it. The one who did left, and I stepped into it. This was a number of years ago.

My manager at the time said, "I think you should be spending 50% of your time writing code." And I laughed at him [laughs] because I was spending, you know, like, 80% of my time pairing with those apprentices. Because how else were they going to learn the ropes unless they were working with somebody? And --

WILL: Well, but that counts. Pairing time is coding time.

MIKE: That's true. Fair. Fair.

DAVE: I remember pairing with Eddy Lopez here at Acima when he was fresh out of it. He was fresh on the engineering team from QA. He might have still been on QA. And he and I were pairing, and I asked him something, you know, "Where's this in the source control?" And he wasn't sure how to...and he's like...And I'm like, "Okay, hang on." And I cracked open a terminal, and we spent an hour playing with Git. "And I'm like, here's where to put it. Here's where your sandbox is. Here's the distributed nature. I can be the server. You can connect to me, and I've got all the changes because da da da da da." And I was explaining where the deltas go and all that stuff.

And we were on the podcast a few months ago and Eddy pulled this story from his perspective, which is that he has paid that forward so many times. And I'm like, that was probably one of the most effective hours of my entire career because I invested in someone who then has now paid in, you know, bazillions of dividends. And I'm not trying to blow my own horn. I'm really happy that that happened. When Eddy was like, "Oh, yeah, you did this, and I love it," I'm like [vocalization], I'm going to go for a long time on that compliment. So, it's awesome.

MIKE: We landed into the value of sharing information. We've rerouted a little bit because [laughs] it was just such an important topic. Have we really hit how you keep that velocity going, how you keep moving? Because you asked, I think, a vital question, Will. And I gave kind of a metaphor, but I don't know if I got really into the technical details, how you do it on a development team. How do you keep that motion going?

WILL: And, you know, actually, like, so there's another issue. Like, I have a lot of meetings on Friday. I'm trying to put my meetings on Fridays, because, listen, man, I'm going to show up to work on Friday because they're paying me, and I will do it, but they're not spending their best [laughter]. It's a fairly linear degradation of my engineering effectiveness. I'm still not bad, okay? You got your money's worth out of me today, but yeah, you know.

And so, I was talking with my boss, so my boss long-time consultant. And he sort of, like, you know, went full-time and moved into engineering management. But he spent a lot of time as a consultant freelance, you know, developer, you know, that's me. Like I'm, you know, technically unemployed. Like, I've got a long-term contract with my large electronics retailer.

And it's sort of like my team, you know, that I run sort of runs things my way, which is sort of very like, what have you done for me lately? Like, because I'm just used to the mentality of, like, people are writing me a substantial check for my services every week, and I want them to know what they're getting all the time, and that gets my contract renewed. I got another year. And it's a mentality that sort of the boss shares, and I share. And he's working with another team, and they don't necessarily think about things the same way because they were not brought up in an environment of existential fear [laughs] the way we were.

So, I have a story, right? And it's my mom. She had dogs for a long time. She had greyhounds. Greyhounds were her favorite dogs. She had greyhounds. And she liked to get, like, rescue greyhounds, like, retired dog track racers. I don't know if that's such a thing anymore. I think we've kind of phased out dog racing, which is probably good. But, you know, they would retire. They couldn't run anymore. They tore their ACL. Whatever. And they'd just be like, well, we're just going to dump them in a landfill. But I guess if you want them, you can have them, right?

So, she would get these dogs. And they were the laziest animals that have ever lived. You'll roll home, and you'll have a golden retriever, and it'll just be ecstatic beside itself, jumping on you, giving you kisses in the face. Almost, if not literally, like, wetting itself because it's so excited that its people are home. It's so happy. The greyhound version of that was...they would lift their head off the ground as they were lounging [laughter], look at you, and their tail would wag three times [laughter], wag, wag, wag. Boom.

DAVE: Head back down.

WILL: Head back down. But every time you would take them out in the backyard to feed them, you would get the most magnificent, beautiful, joyous, absolutely astounding run. They would tear around the backyard in a figure 8 at 30 miles an hour. I swear to God.

MIKE: [laughs]

WILL: It was magnificent. Magnificent. The most beautiful thing you've ever seen in the world would be just to see the beautiful athletic power and grace of a greyhound, wholesale. Because they had grown up in this environment [laughs] of existential fear, where literally, if they wanted to eat, they needed to run like their lives depended on it because they kind of did.

And sort of like, you know, so my boss and I were talking about how can we...it's not that this other team doesn't have...it's not that they're not smart. It's not that they're not hardworking. It's nothing of the kind, but they don't necessarily have that mentality. And so, the question was like, we've got these sort of deadlines, you know what I mean? Economy is tough. Macro is tough. The business is like...it's not bad, but it's certainly challenging. Everybody is pushing really hard to keep their head above water. Everybody's working hard. And how do we get this team to do the thing?

And what he settled on, and it's been on my mind since we had this meeting, and now that we have this conversation, it kind of brings it full circle, what he settled on was he just sat down with them, and he's like, "Okay, you know, this is our thing." And he just started coding it up.

And he exhibited the mindset of like, okay, we've got to move fast, so let's move fast. Let's do this then. Let's just put it together. Let's get the scaffold up. Let's get the skeleton, the framework. We're going to make some...we're going to build a jig. We're going to make some operating assumptions that allow us to move forward. And we're going to sort of start the rough sketch, and then we start filling in the outline, filling in the details. We're going to paint the color, and the shading, and all those things.

And the only way he could really get this mindset onto these people, you know, which comes from decades of high pressure, high production consulting is to just get in there and show it to them, show them how it feels. And then, as he did it, the clouds parted. And they were like, "Oh, okay, I see what you need. I see what you want to do." And that kind of knowledge transfer, endemic to the work we do. That's why I'm a paring zealot because I don't know of a better way to do it. I don't know of any other way to do it. And I am fully aware of how fantastically indulgent it seems on the surface, and the sort of visceral, like, oh my God, I'm going to pay two of you to just sit there on a single keyboard all day?

MIKE: [laughs]

DAVE: That's twice as expensive.

WILL: Do you have any idea how much this is costing me, you know, to do this? But I just don't have a better --

DAVE: And then, you let that stop you out, and you're like, okay, well, I'm going to work alone. And you just kind of sit there and go, do you have any idea how much it's costing you to have me work on this without a second brain? It's, you know, yeah.

MIKE: Well, that's it, right? That what works is what works. It doesn't matter how much you don't like seeing 2 people sitting next to each other on 1 keyboard. It's what works [laughs].

DAVE: And if you're banging out 30 controllers with just RESTful actions, then one or two for pairing to make sure everybody understands this is how we roll this out. And then, yeah, go to your office and just I'll do these 15. You do those 15. Let's meet back up at 5:00, and we'll review each other's stuff because you're just banging out boilerplate at that point. But you do the first ones together because there's knowledge transfer going on, and yeah.

WILL: Yeah, absolutely. The question that we had was sort of, like, how do you get that mentality? Because it wasn't even a technical thing. These guys don't know how to do the work. They understand the work. They're very competent programmers. They're really good. But that sort of, you know, that piece wasn't something that they really understood until they saw it. And then, once they saw it, they're like, "Oh, oh, okay, I get it. I understand now. All right, let's go." And then, you know, boom, that's how I [inaudible 48:00]

MIKE: You know, the kind of meta stuff around software development is so vital. It ends being most of what we talk about on the podcast because [chuckles] it's critically important. But you're probably not going to learn it in school because it's...I don't know if it's...it's probably hard to communicate about but it also sometimes, I think, seems trivial.

You think, well, we need to teach you the hard stuff. We need to teach you about writing code, and writing code is critical, right? Learning to express yourself in that way is a change in mindset that does not generally come naturally. And you have to bang on that for years to get good at it. But if you just focus on that, you're missing half the picture.

Like, Dave, you talked about spending an hour teaching somebody Git. I mean, that is important, learning how to use those command line tools, you know, whatever tools that you use, you know, that are on the command line. Like, learning the tools for your job —that's important. And without that, you're not going to be able to work very effectively. And there are so many things like that.

WILL: I thought so many people like the debugging, you know what I mean? Like debugging. Like, hey, how to use a debugger. Like, some people don't like using debuggers. Like, you should know how because a well-configured debugger where you could just stop the code, it's like, whoa, whoa, whoa, hang on. What are we doing? Why is that X instead of Y? That's critical. Just the idea, the mentality of debugging.

I taught some kid just today, you know, another one of my golden bedrock principles, which Mike has heard out of my mouth many times, and I'll repeat it a thousand times more with no shame at all: Always debug from the known to the unknown. I had a guy, he was like, "Oh, my analytics, they're not working. I thought I turned this analytics thing off, but I'm still seeing [inaudible 50:17]. This is all screwed up." I'm like, "Known to the unknown," you know what I mean? And I'm like, "Okay, I wish you'd check out the whole component."

And he's like, "Okay." But the analytics are still fine, but known to the unknown, take out more, delete more code. You got version control. Rampage. Don't do the thing. And he's like, "Oh my God, you know, 15 minutes of just taking a [inaudible 50:40] at this thing until I finally made the noises stop." And I was like, oh, somebody duplicated it. Like, this thing exists in two places from, like, known to the unknown, always go from the known to the unknown. However, you can make it work, make it work, and then move from there.

I'm like, that's just the kind of thing where it's just like, I think, hopefully, if I got it into his head, that was another one of those, like, one-hour conversations that he'll be like, "Old man Archer said, 'Known to the unknown.'" And I'm going to pop up over his shoulder like Yoda, if I did it right. And I will save him a thousand times [laughs].

DAVE: Amen.

MIKE: It's interesting the arc this conversation has followed because we talked about velocity and balancing that. And, certainly, there's a balancing act. But we didn't really linger much on that idea because maybe it's clear in all these different kinds of work. We've talked about how to communicate with each other, and share information, and keep organizational structures effective.

WILL: Going fast is a feeling. You got to feel it. When you're on a team, and it's clicking, when you're writing code, and it's clicking, when you have the feeling of like, you know, mastering the command of your tools, understanding the organization, the needs, the codebase, it's a feeling. When it all comes together, you feel it in your gut. It feels good. I can't explain it to you, but I can show you how it feels.

DAVE: It feels like flow, right?

WILL: Yeah.

DAVE: You get into it, and you're excited, and then, all of a sudden, it's 6:30, and your wife is like, "Are you coming home?" And I'm like, "Oh, did I miss lunch?" And she's like, "Yeah, you missed lunch. Come home. You're missing dinner."

WILL: Like, you know, the mentality of like, you know, we need this thing out. We are going to make compromises. If we are going to cut some corners, we're going to cut the right corners, not the wrong corners. We're going to give an appropriate amount of spit and polish to this thing. How do you make those judgment calls? I mean, you just got to know it, and you can get it through osmosis by sitting here with me, or with my boss, with whoever.

DAVE: Sure.

WILL: But I can't give you a book and just be like, "Here you go." And that's the weird part of engineering; you got to feel it.

DAVE: I think it's interesting that we're in a phone call talking about velocity versus thoroughness and all three of us are bringing communication. And what I realize is I think at intermediate to senior levels in your career, interpersonal communication becomes the most difficult technical skill that you want to scale. And if I was having this conversation with you 25 years ago, I would be like, you got to know your text editor. You got to know how to do da da da da. You got to know this tool and that tool, and you got to know this other thing. And, oh, you need to learn this other thing.

And it was all the kind of things that a junior to intermediate level developer would be focusing on, mastery of what's right in front of me. And then, you start realizing, oh, I spend all my time tackling problems that are bigger than 12 people together. We need to learn how to communicate. And I'm no longer in a spot...

And the great thing is, we end up hiring juniors who are trying to master their tools and, like, don't bother me about communication. I got to figure out how to use this tool to do this thing. And you're like, yes, you need to keep mastering your basics, but while you're doing that, I need you to come talk to me about...And I think it's interesting that the higher up in the seniority you go, the larger the problems you take down, and then, all of a sudden, the problems that you've experienced are at that level instead of one level down.

WILL: Well, I mean, I also say this. I think one of the real bear traps in the industry, another [inaudible 54:48] difficult problem in the industry is not knowledge transfer from juniors. Juniors are actually pretty comfortable, usually. Usually, they're very comfortable. The really tough ones are the seniors.

MIKE: [laughs]

WILL: That's leadership with a capital L because, like, you know what I mean? You're going to take Ls. Things are going to move. Things are going to change. The needs of the organization are going to change. The needs of the industry are going to change. Like, things are going to move, and there's a natural human reluctance to look foolish to, like, be a beginner again, and again, and again, and again, and again, and again, and again. Knowledge transfer among seniors, and keeping seniors current, and keeping seniors flexible and empathetic, and stuff like that, that's the real challenge.

DAVE: Convincing somebody that this style of code is more readable than this other style when they are utterly familiar with that other style, and they're like, "Well, clearly, it's readable." And I'm like, "Nobody else on the team agrees with you. It's familiar to you. It is unreadable to us. Would you consider doing it this way?"

And you get that spot where, as a junior, you're humble, and you're willing to change anything. And you get to intermediate, and you're like, "What I know works." And then, to get from there to senior, you have to go, "What I know works, and what you know will also work, and we're going to do it your way because we need to move."

MIKE: And that is hard. I read an article the other day by a scholar of the Hebrew scripture. The specific text isn't that important. The important thing is that it was written a long time ago. And what he said was that there's this recurring theme that you have a leader who starts out humble, and they apply that humility, and they become very prosperous, and they get set in their ways, and it causes all kinds of trouble [laughs]. And it's not a new problem. It's not a new...but we still have to fight it, you know [laughs]? And we have to be willing to do some introspection and say, "I'm that guy now. And I need to be willing to humiliate myself."

WILL: Well, I mean...so, go ahead.

DAVE: I'll just say the biggest compliment, I think, I have ever been given in the last year was also from Eddy, ironically. I like Eddy. He says nice things about me.

WILL: [laughs]

DAVE: But we were in a meeting, and I asked, like, a bone stupid...I mean, like, the kind of question that everyone in the room was like, you're asking that? I won't go into the question now because if I don't give you enough context, it's going to sound like an offensively stupid question.

WILL: [laughs]

DAVE: And it was, it was borderline offensively stupid. But as soon as I asked it, all the juniors went, "Yeah, I want to know the answer to that, too." And so, we realized we had zero knowledge share. Like, everyone on the team thought everyone else on the team knew the answer, so they didn't want to ask. And Eddy turns to me, and he says, "You are fearless about asking stupid questions."

[laughter]

And momentarily, I was kind of offended. And I'm like, no, he meant that as a compliment, and he's sincere, and he's right. I'm willing to ask really, really dumb questions. If I feel guilty, if I'm self-conscious, I will say, "I know the answer to this, but just so everyone else knows, you know, it's like, how is this working?" And you got to be willing to do it. I've had people get impatient with a stupid question, but I've never had somebody think I was stupid for asking a stupid question. I've never had somebody think I was a coward for asking a stupid question, to put it that way.

WILL: Do you guys want a life hack for embracing your inner idiot?

DAVE: Oh yeah.

MIKE: Yeah.

WILL: All right, so this is what I do. This is, like, actionable stuff. This is the Slack life hack. I won't discuss ticket work in a DM, and I won't tolerate other people discussing ticket work in DM. If you got a question about how to do your job, it goes in a channel. Maybe the team channel. It doesn't have to be in a public channel. You don't have to put it on general, or whatever, but everything.

If it's like, "How does this work? Hey, do you remember how to do this API?" Anything. Anything at all. I mean, if you want to have a private conversation with me, it's totally fine. Let's talk about our weekend. Let's talk about the dumb thing that thus and such said in thus and such meeting. Like, "I can't believe David had to ask that, Jesus, that guy [laughs]." But, like, none. Ever. None. Zero. No chance.

DAVE: There's a countervailing force that you have to be very vigilant for, which is all it takes is one VP to say, "Just get it done," when somebody asks a question. Those questions will evaporate because, all of a sudden...we've talked about psychological safety before on the podcast, right? Where it's like, if you have someone out there...when somebody asks a question like that, the response from the people who know the answer should never be, "You should know this."

The answer should be, "Yeah, you should know this. Let me tell you. Thank you for asking this because I should have told you this." And that is so, so helpful is to be welcoming of the dumb questions because if somebody's not willing to ask a dumb question, you got a lot of people walking around with a dumb question unasked, which means they don't have the obvious answer, which means they're dumb. It's...yeah.

WILL: And, I mean, the other thing that is like, you know, to be perfectly honest with you, I mean, there's always teams shifts and teams ebb and flow, and people come in. And conventional wisdom that everybody knew six months ago it's like, well, you got three new devs on the team. They don't know that. How could they? They weren't there.

And sometimes I just forget, okay? Like, I'm lazy, okay? I am. I'm as lazy as I can get away with being. And also, also, also, it becomes, like, an unofficial Wiki. I'll search a Slack, you know what I mean? I'll search a Slack Channel. Oh, weren't we talking about that thing the other day? And then, I'll find it, and I'll probably post it again. I'll probably repost that link because that's just how it goes.

I mean, in terms of VPs just being like, "Just get it done," you want to be out of this thread because you got other stuff? The response to that is, "No problem, Jim Bob. We're going to work it out." I'm going to start another thread, and Dave and Mike, and I are going to figure out exactly how this thing gets done, right?

If VP is saying, "I'm delegating this," that's fine. The only thing I want to hear is, "We got you." And then, the team figures out how to do it, and, like, they don't pinged every 30 seconds as we're hashing it out. That's fine. If he wants to go cook, let him cook. It's fine. This is my job. I got it.

MIKE: I think that's a pretty good stopping spot.

DAVE: It is. This was awesome [laughter]. You know it's a good podcast when you want to do two more hours on the tangents that you've dug up part way along the thing, yeah.

WILL: Well, happy Friday, gentlemen.

DAVE: All right.

WILL: You all be well.

DAVE: Have a good one. Cheers.

MIKE: Thank you. Until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, the panelists join forces and discuss the things that scare them in code and how to avoid these pitfalls. Mike begins by recounting his experience with a fast-growing startup that used a poorly managed customer management system. This system lacked version control, searchability, and a test environment, which forced developers to tweak code in production. Additionally, Mike shares another experience with a codebase riddled with security vulnerabilities, illustrating the dangers of inheriting poorly written third-party code.

Will adds to the discussion by highlighting the broader organizational issues that can lead to scary situations in coding. He emphasizes the importance of maintaining a strong chain of trust within a distributed workforce and ensuring diligent code reviews. Will warns that when organizational control breaks down, it becomes a significant concern, especially with large volumes of foreign code entering the system. He argues that fostering a sense of ownership and responsibility among team members is crucial for maintaining code quality and preventing issues.

The conversation then shifts to strategies for cultivating ownership and responsibility in coding teams. Mike and Will discuss the importance of giving developers genuine control over their work to foster investment and care. They touch on the challenges of balancing team freedom with organizational needs, noting that allowing developers to make decisions, even if it means making mistakes, can lead to better long-term outcomes. Eddy also emphasizes the value of training and mentoring junior developers to ensure consistent code quality and adherence to best practices.

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I am hosting again today. With me today, I have Eddy and Will. There's a chance we may have some additional folks coming and going. We're kind of in an influx today [laughs], but we're going to start here. I am excited about what we're talking about today because I think it'll be fun.

And I'll just lead with the topic. We're going to talk about things that scare us in code. And how do we avoid having some of those things happen? Because I'm hosting, I get to start with some stories [laughs], and I'm going to start with stories that are long enough ago that the victims are not obvious [laughs]. And the companies are likely, well, more than likely; the companies are out of business, so we don't have to [laughs] be too sensitive about it.

There's one example I think illustrates kind of everything, maybe not everything, but many of the things that scared me about code. I worked at this place early in my career, a fast-growing startup, really fast-growing startup. They were growing like crazy, and they needed to have a customer management system to manage all of the new customers they had. And somebody knew a guy who [laughs] had built something, and they paid for it, and we got it.

And this system consisted of a Microsoft SQL Server having a whole bunch of custom extensions added to it and a whole bunch of queries so that the entire system ran inside of the Microsoft SQL Server. And some of the components they called out to were compiled code for which we didn't have the source code. And the areas for which we did have the source code, that source code was just stored procedures inside the database, and that's where most of the code was.

Some things about that. There was absolutely no version control [laughs], which means any code there is, the only chance you have to do anything about it is to tweak it in production because there was no test system. There's only the production system, and there's no code repository. There is only the production system. If you want to change something, you go change that stored procedure live and hope for the best. And [laughs] further, because it's just stored in the database, all you have is those stored procedures, and you want to find anything, there's no search system. You just have to try to figure it out. And they referenced each other all over the place. So [laughs], you had to go find them.

And sometimes you drilled all the way down, and you found a compiled component for which we didn't have a source code, and you just hoped it did the right thing [laughs]. Maybe it was sending out emails. Who knows what it was doing? You didn't have access to it anyway. And we used that thing for years. Oh, that was an interesting piece of software. I ended up making a number of changes in there that we needed. But I could go on. I'll mention some of those key characteristics again. No version control. No searchability. No test system. You can only test it in production. And it came from a third party. It came from a third party. The development team didn't have any choice over it.

At that same company, we inherited a codebase. Our company decided to bring it on to try to bring in some customers. And I didn't know about this codebase ahead of time, and none of us did. But apparently, they had some security holes before we acquired it that were actively being exploited. At least there was something like version control in this system. However, this was their SQL injection. They were handcrafting all of their queries. In this case, they hadn't sanitized their SQL. And this was in processing financial data. And they'd already lost, I think, over a million dollars before we found the hole. You imagine that didn't go over very well [laughs], especially this small startup. And probably led to the demise of the company shortly thereafter.

Some things in common between those. Notice the third party. Developers having no access until afterwards, so inheriting a codebase. And a lot of times, you don't acquire code from a third party, but it's handed off, and you don't have the insight, and lack of adherence to anything like best practices and difficulty in searchability. There's a couple of worse stories [laughs], but they also illustrate a lot of the things that I think are worst in code, scariest in the code, which are stuff that I don't understand, and I have to go dig in because I'm inheriting it, and I have to go figure it out.

Poor adherence to good practices [laughs]. Lack of searchability. Lack of test environment. Man, all of those are horrible. And both of those projects kind of checked all of the boxes. And I could go on, but I'm going to give you all a chance. Will, you've been around in this industry for a while. You've probably got your own stories. What have you seen that scares you in the code, and what have you learned from it?

WILL: If I'm talking about things that scare me, it really honestly goes [inaudible 05:11] time. I can remember the time where I deleted the prod database or the various things I screwed up. But really, they all had this one centralized theme, I think, which is when sort of organizational control kind of breaks down. That's really the thing that concerns me the most with sort of bigger organizations, like if you're an organization where they're really pushing sort of a distributed outsourced workforce, right? Like, I might be experiencing some of that right now. And there is...a lot of people have, like [inaudible], is minding the store.

Code reviews are really, really difficult. And it is still very difficult to find people who are diligent in their code reviews and making sure that people are really looking after stuff. And that's when things start to go bad, especially when you get this sort of injections of a large volume of foreign code into your organization, your distributed system. So, that's really what keeps me up at night, where it's like, people are checking things into production. People are checking things into the app all the time. Who is minding the store?

And it becomes, you know, past a certain point, you're doing a trust fall by necessity. You physically cannot manage all of the code. You can't do it. It cannot be done by any single person. And so, you're just sort of...you trust the people and that sort of chain of trust is a very fragile thing and shockingly easy to break down.

MIKE: Yeah, I think my experience is totally --

WILL: It's just sort of like this organizational chain of trust.

MIKE: Yeah, I was just saying my experience is totally consistent with yours. And [laughs] the way you say that organization breaking down, and it doesn't have to be inheriting a bunch of that outside code, like you say. You can have just something as simple as a change in organizational structure so that you don't have the same team cohesion that you had previously. And say the people watching the store were out taking care of something else; stuff starts to fall apart.

WILL: Let's talk about this. Let's talk about maintaining that chain of trust and how do you do it. Because I'd say the best model maybe that I've seen that is scalable, not without faults but scalable and practical is, you know, each sort of section of the code, you know, major subsystems are owned by a team. And they have a watchdog who is responsible and proactive enough to keep this subsystem under their thumb. And I don't know if I have a better working model, really.

MIKE: Well, that watchdog you talk about, I feel like if somebody feels like they own it, that is [laughs] not told that they own it, but genuinely has some opportunity to get invested in something, it makes a huge difference. I know when I build something, I care about it, even if it's something, you know, that's not that important. If you make a pile of rocks, you can get a little fond of that pile of rocks because [laughs] I built that. That act of creation binds you to it.

I think that some of that applies in our jobs as well; if we feel we had say in what we're building, feel because we actually did. I'm trying to make the distinction here. Because you can have kind of faux control where somebody says, "Yeah, you own this. You're on call tonight." But you don't get to make any of the decisions. Well, you don't really have control. You have responsibility without control in that case.

And that's where I think is kind of an anti-pattern because if you have the responsibility without any control, it just burns you out because you're disempowered. On the flip side, if somebody gets to actually make decisions, creative decisions of what they're doing because, you know, they own it, they get to make those decisions, then they become much more invested. You know, it sounds like I'm talking about a trick like, here's how you get your team invested. Rather, I'm saying you've got to surrender some control.

You've got to, as leadership, be willing to say, "I don't get to call all the shots here because I'm not close enough to be able to make the good decisions." And if you're making those decisions, you may make the wrong ones. And there's always cases like this where there's more art than science, where it could go either way. If you don't have enough trust in your team to make their own decision, then you're seizing that ownership for yourself for no gain. It's just hogging the ball, right [laughs], and not letting anybody else play. And that, I think, really poisons things. I think that in order to get the ownership that Will was talking about, to have somebody really claim the code and be willing to care enough about it, they have to actually have it genuinely be theirs.

EDDY: There's a few key things that both of you said that I kind of want to elaborate a little bit on. And losing control of the codebase is a little interesting because if the staff is trained right with the expected conventions and what we have planted for team rules and such, that can be mitigated. But also, there was a joke that was said [chuckles] a few weeks ago. I'm not going to name-drop the person just because he's not here.

But it basically said, I'm paraphrasing, but it basically said, as a senior developer, if you take a junior developer under your wing and cultivate their decision making, you can then, at that point, have a proxy battle from them in your behalf in order to enforce some of those decisions that you want. It's kind of funny. But the more that I've adapted to that ideology, I've found myself also enforcing some of the ideologies from my peers, which is another benefit, I guess, to that fear.

MIKE: You know, a few hundred years ago, for any skilled trade, maybe not any, but a large number of skilled trades, people learned the trade by apprenticing with a master in the field. And, to your point, one of the best ways to learn something is to watch somebody else do it and you apprentice with somebody. Well, and you can see this in the works of, say, artists, where artists who apprenticed with the master end up having work that looks very similar. Of course, they diverge in their own way, but the apprentice does tend to follow the master. And I don't think that we should shy away from that.

Certainly, people are going to diverge and go their own direction. But there's real value in not having to make some of the hard decisions yourself when you're getting started. And you're going to rely on the expertise of the master while you're still learning. That's what kids do with parents. And, of course, parents sometimes get it wrong. And a lot of times, you know, the basics of adulting [laughs], you're going to pick up by watching. And they're probably going to have at least somewhat right, and you're going to pick up on that and learn the things to do. That's how we learn as humans. We learn from more experienced members of the tribe and don't have to make the decisions: hey, should I eat this berry? Is this going to kill me? And look [laughs] to somebody else who might know that answer already.

And so, you know, I think that it's worth leaning in, leaning in there. And if you are a leader, it's okay to tell people, "Hey, that berry, I saw that make somebody really sick the other day," or maybe "20 years ago. I've been doing it for a while. And I don't want that to happen to you all, so don't eat those berries. There's a reason we don't eat those berries." Maybe there's a reason we don't test in production. And follow the best practices of setting up a test environment, setting up a staging environment that you can test in so that you can test things. And put a lot of investment into that so that you can do solid testing before you get to production. You're not going to get poisoned that way, in the way that you would if you go straight out...

EDDY: I want to say that it's kind of funny because you say if you've lived that pain, you're saying, "Don't eat it. It's bad for you." But sometimes, your body acclimates to that substance that you're eating, right? And then, you just don't notice that it just tastes awful. And that's kind of just what happens in code. It's like, you might be doing a bad practice, but you're just so used to doing it. Your body just got so used to doing that that you just don't realize, you know, that it's bad [laughs] until you have someone else who's trying it for the first time, and they're like, "Oh, dude, this is pretty gross, dude."

MIKE: [laughs] Yeah, it's true.

WILL: There's a lot of things that were, you know, they were good until they were bad, or even worse, more commonly, they were relatively benign diseases. They weren't going to kill you until you got to a particular stage. I've done all kinds of things in a small team, small, high velocity, tightly integrated team, but we're fine. They were all fine. But on a more, call it, enterprise scale, you know, organization, they would kill you dead as fried chicken. So, it's not a cut-and-dry thing. The argument would be that for those small, high-velocity teams, if you're not moving fast, if you're not cutting corners, you're not going to make it to enterprise scale where you can go so slow.

MIKE: That's a really interesting point. You can get away with a lot of stuff when it's just yourself. You can get away with a lot of stuff.

WILL: And you have to. Let's talk about the things that are going to kill you when it's just you out there on a wing and a prayer. Like, you need to ship, and you need to ship today. And like, oh, you don't know how to do that? Well then, you're just going to do the best you can. Overwhelmingly, the thing that is going to kill you is running out of time or money, right? Like, 10 [inaudible], if not higher. And so, it's like, oh, security is not so tight here, and I'm like, no, it's not [laughs], you know? It's not.

EDDY: There's an estimation of good time for half a day. So, you need prep time to get there. But sometimes the party is in the next hour, right?

MIKE: [laughs].

EDDY: So, you're like, well, dang [laughs]. Well, I guess I'll just put in an extra cook to turn on the heat a little bit, you know, get it done in an hour, and hope that it tastes the same.

WILL: Exactly. I mean, being worth robbing, you know, by cybercriminals, being considered worthy of being a ransomware target is, you know, that's first world problems, baby. You made it [laughs]. It's like getting sued by a patent control. Like, hey, there you go, like a disturbing truism that, like, a rite of passage to be like, oh, I've got an actual business now. I'm actually a businessman now, is getting sued for the first time. Your first legal letter where somebody tries to shake you down. It's like, oh, you made it, baby. There you go. Look at us now.

MIKE: [laughs]

WILL: I'm worth robbing.

MIKE: So, we're digging into this idea that the consistency that is absolutely critical for managing a larger organization can be less so when you're working with a smaller organization because your circle of trust with a smaller organization is so easy to maintain. You all know where things are bad when you've got the small team. I'll say, "Oh yeah, there's that bad thing. We all know it's terrible. Don't step there."

But as you grow, that shared knowledge becomes harder and harder to maintain, and it becomes more and more important to have standardized practices, standardized ways of sharing information, ways to replace people when they move on or, you know, otherwise removed from the organization. And that's a challenge as you grow, and it's going to chafe a little bit. And you're going to be like, "Hey, I miss my freedom to do whatever I wanted." And to have the maturity [chuckles] to say, "You know what? I need to change. I need to change what I'm doing, or else I'm going to cause harm because being a cowboy is not going to work anymore."

EDDY: There was an off comment, you know, that was kind of relevant to what you're saying but I kind of want to expand on a little bit. It was, you can tell someone the direction and tell them, "Hey, don't step there, right? There's a trap door. You're going to fall." But what do you do in the instance when you tell someone, "Don't step there." And they respond with, "Oh, it's okay. I can fly," right? Like, what do you do [chuckles] in that circumstance? You're like, "No, dude, you can't fly. I've been there." And they're like, "Oh, no, no, it's fine. I'll be fine." Like, what do you do? How do you keep quality when the individual believes that they're not affected?

WILL: It depends on the blast radius of the problem. I am a believer in sort of you break it, you bought it. If somebody is willing to say like, "I want to do this thing. This is my thing. I will take ownership of it," and the blast radii is small enough so that if it goes bad, then...you know what I mean? Like, it is a human-sized error with a human-sized sort of level of consequences, and somebody comes to me and says, like, "No, I'm going to do it my way," I'm like, "Okay, do it your way," you know? I believe in sort of agency like that. You have to be mindful because, a lot of times, things can be slow-growing systemic problems. Make a judgment call whether you can afford to let somebody go and try it their way.

In my estimation, the way I make those decisions is, is it a really big deal for you, and can you shoulder the consequences of it yourself? And so, if it's a manager, it's like, okay, the manager's team can shoulder it. If it's an individual developer, then the individual developer can shoulder the consequences thereof. I don't know. If I can say "Yes," then I will, honestly, because that's how people learn. And if I can, I try to explain why. Or I'll come up with a sort of rubric to be like, "Okay, try it your way, and these are the conditions under which I will say, you were right, and I was wrong. These are the conditions under which I will say, ah, you know, I mean, this is just tomato, tomahto. Both are, you know, more or less equivalent. And these are the conditions under which, like, can we all agree that I was right and just do it my way now, you know?"

EDDY: I agree. You're like, at what point does it start tasting bad? If you're saying add more sugar but without the added sugar it still tastes okay and you're okay with the taste, you're like, "Yeah, yeah, it's fine. It still tastes all right."

MIKE: [laughs] It goes back to --

WILL: And I'll say there's a lot of stuff out there. The overwhelming majority of decisions fall into that, I would say, broadly, you know, that tomato, tomahto kind of a situation. It's very rare that you'll have a black-and-white this is wrong scenario.

MIKE: I was going to say that goes back to the ownership discussion that we were having before. If you don't give the team...so this is opinion of me, but I think it's right [laughs]. If you don't give people ownership, then they won't have any reason to take responsibility, and you get ownership by making those choices. You're committed now. You've made the choice. You see how it works, and now you care how it turns out. And, yeah, maybe you've made the wrong choice, and you learn from it. Maybe you make the right choice, and now you've got something defensible, and you'll care about that the next time.

Giving people that chance, you know, the tomato tomahto chance to do it their own way, it may sound superficially important, but I think that it's actually critical for giving your team the opportunity to actually make consistently good code. Because if you take away that control, well, then nobody cares. I don't want to say it's like slavery because you can leave the job [chuckles]. But if you don't have any agency in it, then you just don't care.

EDDY: Well, and I think what's more important is if you have a new cook in the kitchen and you're constantly holding their hand, they're not really going to learn anything. However, if they end up burning the kitchen, they are going to remember. And they're going to be like, "Dang, I had to clean all this up. And all this mess, I had to clean it up." But then, guess what? The odds of them burning the kitchen again for the same recipe probably isn't going to happen anymore. So, I think there is some benefit to putting trust in your cooks.

MIKE: I was building on what you said before, where you have to have that watchdog that keeps going. And the way I say you develop that, I think you're right that you need to have somebody who cares enough to keep people in line. And I think you only get that, and this is why I've been talking about the ownership: if that watchdog actually feels like they own the code, feels like because they do.

They're only going to care enough to keep the code good if their actions matter, you know, if they actually get to make those choices, which means you got to let go of the hand and let them do it, which is going to be uncomfortable. But then they will own it, and they'll learn, and they'll become that fierce owner, you know, it's their thing, and care enough to keep the code solid. And I think that having a culture of giving people the chance to actually own the code is hand in hand with having quality code, that you can't take away control and expect the best practices to continue because it rubs against human nature. That was my high-level idea.

WILL: Yeah, no, I totally get it. And so, the question that I have for you guys, this is kind of a broad thing, but, like, how do you cultivate that kind of [inaudible]? How do you value people who have been around long enough to take ownership? What are engineers like that worth to you? How do you build that stuff up? How do you keep...I would say, I love institutional knowledge. I love people with ownership. I love all that stuff. I am very fairly stridently against the idea of seniority as if, like, sort of years on the job is a suitable proxy for these desirable qualities. So, how do you make that happen?

I won't bury the lead, but one of my great bugbears of the industry and one of the things that I...I really hate the fact that you've got to move around to move up, and I understand the countervailing forces there. But, like, if you didn't want to do that...because all these things that we're talking about, these watchdogs, this ownership of code, this isn't the kind of thing that you can have with people moving every two years. But if you're not moving every two years, there are consequences for your career, which can be fairly significant. How do you manage all that stuff? How do you create these cultures that we all agree we need but there are so many headwinds against?

MIKE: Well, Netflix, if I remember right, so I'm going from memory here, but my understanding is that for years, they had a policy that if you got a job offer somewhere else, they would match it. No questions asked. They cared enough about that continuity. They put their money where their mouth was. And so, they had some of the best-paid engineers in the industry by some margin, sometimes, like, you know, 30%, 50%, 100% more than some of their peers.

But I tell you, you do that, and you get a committed team who cares enough to stay around. I think they did the things because the company cared enough to actually do what they said and value their team members. The result was they got a team that made them extremely successful. So, I think that there are examples in the industry of people doing right. But there's a lot of examples of being short-sighted because there's a lot of incentives to be short-sighted, to say, "Well, we need to save money. We need to make the stock price go up., And we're going to get that by reducing costs," which is a reasonable thing to say. But then what happens next is dicey, right [laughs]?

You say, "Well, what you do next shows what you value. And if what you do next is continue to value your team and pay them appropriately, well, that shows what you value." If what you do next is say, "Well, let's get rid of all of our senior experts because they cost a lot, and let's hire a bunch of new folks without any context and, you know, it'll work, right?" Well [chuckles], you've shown your values. And there's a hazard, I think, to leadership to make decisions that will probably turn out well in the short term but will lead to a steady decline over the long term.

EDDY: It's like what do you do when the only two people in Coca-Cola suddenly die and they're the only ones that know how to make your recipe, right? You can get pretty close to the genuine taste, but you just know that it's going to be a different taste, and it's not going to be the same.

WILL: Well, like, the great [inaudible] are full of indispensable [inaudible 27:49], right? And I think there's a counterargument to be made that your initial team is the people you could get. But as things go on, as the company gains legitimacy and financial power to compensate and hire and retain good developers, that original team maybe, like [laughs], uncomfortably accurate definition of a technical co-founder, is the best developer you can get to work for free.

MIKE: [laughs]

WILL: So, you could start bringing in more heavy hitters as time goes on. And a lot of the decisions, engineering decisions that were, I think, defensible, if not absolutely correct at the time, are no longer so. There's a flip side to that as well. Everything is a golden [inaudible] [laughs], right? You're always trying to find a balance.

MIKE: Sure. What you probably don't want to do, though, is hire these great, new folks, keep them around for six months, and then send them out the back door. And maybe you're deluding some of the people there originally, or maybe you do ask them to leave. But you're still looking for continuity with people who know what they're doing. There's still the respect for expertise and, again, not seniority per se but expertise.

WILL: Well, yeah, and then it becomes a sort of a leadership challenge in that you need to sort of walk people down where they were, at one point, a law unto themselves, and now, you know, maybe they have a manager. And I think, arguably, the current stock option regime mirrors that, I think, pretty well in that, you'll have your founding teams, and they'll get a big slug of equity. Like, your initial hires they'll get a big slug of equity, and that equity will appreciate and appreciate and appreciate.

And people will come in. You'll get your series A round. And you'll start to bring in some real hitters, and they'll get more compensation. But the people who came there blocked the stock, and it's worth a lot more now. And so, they have, you could argue, golden handcuffs on one hand. But you could argue they're paying for institutional knowledge and continuity, even though they might not be continuous in the same role. Everybody sort of gets a...it's a fair shake there.

MIKE: It's interesting that we started by saying, "What scares us in the code?" And we're talking about stock options [chuckles]. And I don't think that's off. It seems like a tangent, but I think it's inseparable in that we're talking about the practices that lead to ownership and people caring enough to make the code solid.

WILL: Well, exactly. Well, we talked about the code for 5 or 10 minutes, and then we immediately jumped to the real problem, which is sort of maintaining the team, maintaining that culture of engineering excellence, maintaining a sense of ownership and responsibility as things scale, as things get bigger, as the problems get more complicated. I mean, that's it. That's the issue. There's no two issues around it. As long as everybody is coordinated, and focused, and disciplined, you can sleep pretty soundly. I mean, every once in a while, something's going to go bad. But, man, dude, nothing makes me happier than a problem that got successfully opened, closed, and resolved, and I didn't have to lift a finger. I'm just like, ooh, ooh.

MIKE: [laughs]

WILL: Oh yeah, oh yeah. What's up? Oh, that was Saturday.

EDDY: For those who are fans of --

WILL: You guys had a rough Saturday, boys.

MIKE: [laughs]

EDDY: For those who are fans of Emperor's New Groove, there's a meme where he goes like this, and it's like an okay because it's just so perfect. And that's what you made me think of when someone reports a problem, and it just goes fluid, and it just goes perfect. Awesome moments that you sleep like a baby.

I do want to say that there's a couple of situations or circumstances that sort of keep me up at night, and one of those are having a tower, where the foundation of that tower on the right side is held by a twig. It's so freaking scary because that's legacy code. And touching something where you just don't know where it is, you can make the change that's super scary. And I feel like a lot of us sort of try to avoid that and find another alternative because it's like your whole app is rooted off of it [laughs]. So, that's one of the things that keeps me up at night.

MIKE: That goes back to the continuity we've been talking about. Hopefully, you can keep the continuity so that you have somebody who knows how to work with that twig and can share it, sometimes, though, you don't. And in an older codebase, you probably don't have people, and if they do, they're probably spread thin and don't have much time to talk to you about it. And that's a difficult problem that –-

WILL: I don't know, I'll say that just, for me, I don't spend a lot of time thinking about pieces of the codebase that have been running without complaint for many years. Those are not high on my [inaudible]. Nothing like battle-tested code. Although, gentlemen, if you can't hear, I am being paged. I think it's time. I think it's my time. I will talk to you guys. See you guys.

MIKE: As Will was saying, if it's working, you don't have to worry about it. Sometimes, though, it stops working [laughs] or needs to be changed. And then, you got to have somebody who can go in there and fix it. I can bring up one example. She's not here, so I'm not going to name. But I worked with a really good engineer who was obsessive about detail, and she'd see something, and she just couldn't help figuring out why it worked and how it worked.

And it meant that it took her a long time to get things done initially because she would understand what was going on before she pushed up the code. But it also meant that once she'd worked in the area, she understood it, and she could get that job done. And anybody else who needed to work in that code would go to her because [laughs] they knew that she could help them understand what was going on.

Having somebody who can have that level of focus where they will take the time to actually figure something out is so valuable because it solves your twig problem, Eddy [chuckles], because they'll take the time to figure out exactly how it's working. And there's a hazard there because you might think, well, their output is not as high. They're not cranking out a bunch of code because they're going to take longer to get things done generally because they're not going to push out their changes until they're sure they work. But they provide immense value to the team [chuckles] because having somebody who actually gets it is just so valuable.

EDDY: I agree. I agree. Now, there is another thing that keeps me up at night. And this is when you have a long chain, and they're all connected to one another. And in this chain, it calls another chain, which calls another chain, which calls another chain, which calls another chain. And, suddenly, you have 17 chains that are hooked up, right?

MIKE: Link after link [chuckles].

EDDY: Oh, that was so bad because then what you have to do is...and they tell you, "Oh, modify this chain." And you're like, ah, see, I don't know. I don't want to modify the middle. That's too scary. I don't want everything to just split open. So, what I find some people do it's like, oh yeah, we'll modify the tip. We'll modify the end. We'll modify the other end. And that's why you get a long series of chains, and that really keeps me up at night any time I have to touch any of that code.

MIKE: Well, and if you're doing that also without doing any refactoring, you probably end up with 20 parameters to the end of the chain, where the chain began to try to promote reusability. And then, oh yeah, well, there's this component that could be reused. But if you haven't figured out what's going on in that chain, the reason why you have it, then you're not going to reuse one of the components and kind of branch the tree where you need to. You're just going to tap in at the end and add another parameter to make it do what you wanted, rather than going deeper into the chain. The part that does do what you want, you could have [inaudible] in there.

EDDY: Yep. Which is why I sort of adopted the ideology of singleton classes. I really do enjoy them a lot. They're much easier to work with.

MIKE: Singleton can have a few different meanings. Could you clarify exactly what those parameters are that you're --?

EDDY: Sure. So, for me, a singleton class is just a single hand of responsibility for that class, and that's all it does. [crosstalk 37:45], right? Like, don't branch off of that. It's possible that you can go beyond that if you want to, but don't violate that. It's part of...there was a book that we read on Ruby, which was...I don't recall the exact name of it off the top of my head. You might remember it, Mike. But the whole –-

MIKE: On "Design Patterns"?

EDDY: The "Design Patterns," yeah. So, part of the idea of the design pattern from that book was don't go beyond of what the class is intended to do because you risk maintainability and reliability from that class. So, if you say, oh, this class' whole purpose is to structure a hash, and that's all it does, and you're like, oh man, but it would be really nice if it would also do this, and you're like, yeah, see, it works. And you can do that, but don't do it, right?

MIKE: Single responsibility, single responsibility. So, I wasn't sure where exactly you were going because sometimes singleton can mean that you've got something that's self-contained, so it's both...there's only one instance of it in all of your running application because the class will only create one instance of itself. And so, you have that single shared instance across the application, so you avoid issues with duplicates of it.

And so, if you're managing a connection pool, for example, you don't want to have two managers of the connection pool. You want to have one, so you don't...only one person in charge. But what you're saying is something a little different, which is single responsibility principle, your unit of code. And if you're functional programming, you don't have classes, but you should have your function. It should do one thing. It should do that thing well, and it should do nothing else. And if you've got something else to be done, well, some other unit of code should do that.

EDDY: Agreed. Now, you run into issues where you implement modules or concerns that essentially do the same thing, but they deviate slightly. And that's also a problem that's really hard to address.

MIKE: Are you talking about people writing code that's almost the same but not quite?

EDDY: Yes.

MIKE: [laughs] Yeah. And so, they end up diverging.

EDDY: And, suddenly, instead of maintaining just one, you're maintaining two. You're maintaining three [laughs].

MIKE: Well, that gets to the don't repeat yourself. Don't repeat yourself is actually kind of subtle because there's times when you absolutely should repeat yourself. And here's where I take that. In cases where the repetition is trivial and makes things easier to understand, then you should repeat yourself such as in a display component. To take something to its logical extreme, let's say you said, "Well, all markup uses tags, so I want to make sure that every single tag is written by shared code." And so, you have to go through this shared code to render a tag.

And maybe you have some sophisticated component. Maybe you're writing a front-end component where that actually does make sense. But if you're just writing out some markup, then that's probably a really bad idea because you're going to have all kinds of complexity, or you could have just thrown some text out there, and it becomes far harder to manage.

Likewise, unit tests. People sometimes think, man, I'm going to make sure that I have no duplication. So, they take anything that might possibly be shared and put it into a shared example that's in some other file that you have to go digging [laughs] to find. It makes it so hard to see what it is that you're testing, where if [inaudible] the details and repeated them, then it would be a lot more comprehensible. So, I'd say don't be so DRY that it chafes [laughs].

EDDY: I've received code where I've reviewed where they've obscured all the dependencies to top level in the specs, and they just call one subject, right? Boom. And then, like, that's the whole context for that test. And I'm like, no, what? What's going on here? And so, I have to go and I have to dig to figure out what's all the dependencies for that. And I'm like, sometimes I wish, man, can we make this a little bit more DRY? Just a little bit more so that we have...because tests, in general, should document themselves, right?

MIKE: Yes.

EDDY: And if you're obscuring that for your reader, then you lose context and it's really easy to keep all that in your head. So, I have tried to push a bunch of new tests that I've written, right? And I've applied a bit of dryness to it.

MIKE: Well, less DRY [inaudible], like, in this case, less DRY. You are going to repeat yourself. It's important for legibility. But then, on the flip side, you say modules will do the same thing. That's where it actually matters a lot to not repeat yourself because you are going to have to maintain two things, or then three things, or then four things that are slightly different that basically do all the same thing. And now you've got to maintain all of them. And, in that case, for important business logic, yeah, put it in one place and one place only. For unimportant display issues, repeat yourself at will.

EDDY: I have pretty much bent the knee, and I've said, all right, I'll settle with damp [laughter].

MIKE: You like your code moist.

EDDY: Yeah, it's a little moist, you know, best of both worlds. It's fine. You know, not too extreme one way or the other. But you have ideas from both is really where...because you have to be able to differentiate between syntactical opinions, right? That doesn't really, like, it has the same functionality, you know, same everything. But the holdup really just becomes from that individual's opinion on how the code should read versus how it functions, right? So, at that point, you really just...you have to give in. And you're like, eh, it was just syntax, not a big deal.

MIKE: Let people make their own decisions a little bit [laughs].

EDDY: Agreed. And it came back.

MIKE: It did come back. That's maybe a good stopping point, actually.

EDDY: Agreed.

MIKE: So, we looked around a lot and didn't talk a lot about specifics in the code but a whole lot about practices that tend to drive good code. And so, we sleep well at night when we give people the agency or ownership to make their own decisions. It sounds like it might make you sleep worse at night [laughs]. What's my teenager out there doing [laughter]? And if you've given that teenager the chance to make good choices since they were three, then maybe you are sleeping at night because you trust them to make the choices you've already seen them making.

EDDY: I'm not a father personally, but I do attribute my tests to do the parenting for me [laughter]. So, I sleep better at night knowing, you know, that my tests are holding me accountable.

MIKE: That's great. Well, thank you, Eddy, and thank you [inaudible]. It's been great and until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, the panel delves into the topic of changing programming languages in applications. They reflect on their experiences transitioning between languages like COBOL, Perl, PHP, and Ruby, highlighting the complexities and costs associated with such endeavors. They stress the difficulty of converting large, established applications to new languages due to the potential for high costs and disruptions, as exemplified by legacy applications still running on COBOL.

A significant part of the discussion revolves around the value and challenges of rewriting applications in new languages. Will presents a strong argument against rewriting code, citing examples where organizations have suffered due to poorly executed language transitions. The group agrees that the decision to switch should be driven by practical needs, such as performance improvements, scalability, or security, rather than the whims of new leadership or a desire for modernization. They also emphasize the importance of mentoring junior developers and avoiding language specialization that can lead to organizational inflexibility.

The podcast concludes with reflections on the current trends and the future of programming languages, referencing the Stack Overflow Developer Survey. They note the rise of languages like Python and the enduring niche of Ruby. The hosts acknowledge that while modernizing or changing languages can sometimes be necessary, it should be approached with caution and clear rationale to avoid disrupting productive teams and established workflows. They suggest using opportunities like refactoring to evaluate and implement new languages gradually without jeopardizing existing systems.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting today. I have with me Matt, Justin, and Will. We're happy to have Justin with us. So, Justin and Will are not Acima employees. They have been in the past. And it's great to have both of them with us to provide their insight in their current endeavors. Great having you back here, Justin.

JUSTIN: No problem. And I do have to say that I went from one Utah startup that is an A startup with five letters to another Utah startup that was an A startup with five letters.

MIKE: [laughs]

JUSTIN: Both of which were acquired by big corporations and both of which have interesting relationships with the parent corporation.

[laughter]

MIKE: Does it --

WILL: I know, like, one guy at a startup number two. I'm very curious. I'm very curious. Like, you know what I mean, I'm always curious about like, you know, the hot goss which we won't get into.

MIKE: Exactly.

WILL: I went from a Utah retailer to a big box electronics retailer, that one [laughter]. And, you know, it's a different environment.

MIKE: Well, and this is perfect. But today we're going to...I'm just going to jump straight to the topic today. We're going to talk about changing languages in your apps. Now, this applies in, you know, languages that you're doing, but we all have applications that have been around for a while on some languages, and, eventually, that doesn't work very well anymore.

Now, there are still applications out there in COBOL. I don't think that anybody says, you know, "I want to create an application in COBOL," [laughter] because it's a language that hasn't been actually [laughs] developed for, like, what, 50 years or something [laughs]? I don't know. I have to go look at exactly how long it's been since COBOL was, you know, like, a popular language.

But you know what? There's still COBOL developers out there [laughs], and there's still COBOL applications out there. Because it is so hard to take a big working application and so expensive to convert that into a new language, which will itself eventually fall into disuse, and then you have to do it again. And, you know, the bigger the application, the more that costs.

MATT: ...Started my career.

MIKE: Did you? COBOL?

JUSTIN: COBOL?

MATT: Converting COBOL and FORTRAN applications to Perl and PHP web applications.

MIKE: Wow.

JUSTIN: Wow.

MIKE: To Perl. So, I wonder if that's happening. So, I was thinking about leading out...I thought, you know, I'm going to lead out with COBOL. But I thought about talking about my [laughs] first year as a professional developer, where I wrote in Perl, [laughs] right, you know, as well as other languages and, you know, comparing that. I'll say, I worked in Perl and then went away from it. I came back to Perl, and it was, like, starting from scratch [laughter]. It was a challenge.

MATT: I do still have some love for Perl, though.

MIKE: It's a cool language.

JUSTIN: [inaudible 03:24] as well on my first programming job, also.

MIKE: Oh.

JUSTIN: It was great.

MATT: PHP, not so much love, but...

JUSTIN: No, [inaudible 03:32] PHP.

WILL: I don't know, I like...like, Perl's, like, the Vi of, like, programming languages. I mean, like, in that, like, I could do it, but I never really got right with it. But I saw people who were, like, old, grizzled, wizard beards, sys admins just cooking on some, like, Perl scripts. And I was just like, okay, okay, like, this is, like...like, I'm seeing what's happening here, and I could recognize that, like, this is a power tool for, like, serious people to do serious work. This has utility.

MATT: You can accomplish a whole lot in Perl with very little code. It's the king of obfuscated code.

[laughter]

WILL: Ah, it is, yeah, yeah.

MIKE: Perl was the inspiration for the name of the Ruby programming language because they have a lot...

JUSTIN: Ooh.

MIKE: Yeah. Ruby would not be called Ruby if there had not been a Perl beforehand. But I got to say, when I started Ruby, it made sense, and that never went away [laughs]. Like, Ruby is like Perl, and it has still some weird Perlisms in it, if you go looking for them. It's like Perl intended to be easier to read.

WILL: Yeah, with an emphasis on sort of, like, readability, and, like, actually object orientation and stuff like that, yeah, completely. I got a hot take for this one, but I don't know whether anybody wants to get in here.

JUSTIN: So, I've got a hot take. I mean, I'm sure we all have hot takes for this one [laughs]. Why don't you go first? I have a good, lukewarm, and a bad experience, so three different experiences.

WILL: My hot take on, like, when should you rewrite your program in a new programming language is don't. Don't. I have seen...I don't even want to say 10 bad examples to 1 good example, you know what I mean? But I think it's actually even higher than that. And I have seen entire organizations, entire companies ruined with just this sort of thing. It's almost like an organizational anti-pattern, where you'll get some new hotshot CTO. He comes in. He has a tech stack that he favors. I say he; it could be a she, but it never is. It's always a dude with an ego and a tech stack, an area of expertise, a team of, like, people that he trusts.

JUSTIN: Yeah, and he brings --

WILL: And then, they come in like a bull in a China shop. They rewrite everything in their tech stack, bungle the job, screw up the product roadmap, and, like, take the entire organization down, you know what I mean? And then, he sails off in his golden parachute to do the exact same thing again. I recognize that, like, there are no such things as absolutes in engineering, but I am an overwhelming no. Absolutely no. Like, there's almost always ego, programmer laziness, and sloth.

I can't think of a language, a modern production language, that can't be reformed instead of replaced with 10 times less cost and a better outcome. Even PHP, which is a dog, gutter terrible language, has been remediated. It's fine. You can do PHP well now. You can do it, so do it. And, like, if you're a programmer that just doesn't want to learn another language, [inaudible 07:12]. Sorry. Like, I mean, it's laziness, sloth, and arrogance. Like, there's no...you know what I mean? Yeah, that's it. That's my hot take. Don't. If you want to do this, you're wrong. Don't do it.

MATT: So, you said a couple of things there. One, you used the word modern.

WILL: Yes. Important [inaudible 07:34].

MATT: I am wholeheartedly against the idea of modernization for the sake of modernization. If it's not broke, don't fix it. You know, we all face this. Everywhere we go, we face this, right? Some hot, new language comes into town these days. It's Node.

JUSTIN: Java.

MATT: And JavaScript backend.

MIKE: It's some new language that'll be transpiled into JavaScript. That's what it'll be [laughter].

MATT: Yes. Yes. And --

WILL: And has transpilers all the way down [laughs].

MATT: And everyone wants to jump on that bandwagon when they have a stable, expandable, scalable system.

JUSTIN: Okay, I've got one example where it worked, and that was in mobile, in the Android world. It was originally written in Java, and it still is in Java. About 2017, 2018, Kotlin came on the scene, and Google and JetBrains, no, IntelliJ, whatever, IntelliJ.

MIKE: JetBrains.

JUSTIN: They did it right. They were able to convert Java programmers over to Kotlin programmers. And they were able to convert codebases from Java to Kotlin with minimal issues. And it helped. I mean, there was a lot of things there about why it worked, but a couple of the main things that I could think of were that Kotlin, at the time, was just plain better than Java.

And then, two, Kotlin was compiled down into the JVM. And then, three, to convert a Java file to Kotlin, IntelliJ had a button that said, "Convert this Java file to Kotlin," and it converted it into working Kotlin. It may have been a little wordy, but it worked. And you can do that class by class. And it would work alongside the Java files and the Kotlin files. They would just work alongside. They were in the same codebase. And when you hit compile, it all worked.

MIKE: I've also seen a success story that's even different than that. It was a Java application. And this kind of goes back to Will's [laughs] saying before...the incoming people...it was an acquisition. The new people said, "You know what, we're not going to use Java anymore. We need to use a scripting language." And their preference was PHP. And this was a long time ago, but still [laughs].

WILL: Oh, man. Oh, oh.

MIKE: And I had been to a Java users' group where they had talked about Ruby on Rails and showed how, hey, you know, this is a, you know, a structured framework that is responsible and uses scripting language. You got to think about it. And so, I thought, you know what? This is old PHP, right? This is before all of those problems that you talked [laughs] about, Will, had been fixed. They were all still very much there. But I was like, you know, I like my life. I don't want it to go [laughter] in all sorts of bad directions.

So, I went, like, in a weekend, I learned Ruby, wrote a prototype for the new version of this, and said, "How about we try this? It's already starting to work," and got them on board with trying this other, you know, language: Ruby. And we converted over. We ended up with one-tenth of the lines of code and more features.

MATT: I've had similar success. And when I said that, I didn't mean you should never do this, right? Because there is a time and a place. And one of our successes was a big PHP monolith into Ruby, so much better, so much cleaner. But there was reasoning behind why it happened, you know, new lines of business. It was in healthcare. So, we now needed to support claims and imports from health plans, EDI imports, and, you know, all sorts of things that the old system didn't do. And it justified, hey, it's time to version this and roll out a new system.

WILL: But I want to...hang on, I want to get in here because I do take your point, Justin, and I agree. And I have experienced the same thing with sort of, like, legacy JavaScript stuff moving over to TypeScript. However, I don't think those are rewrites. Those aren't rewrites. And I don't actually think they're new languages. I think Kotlin is just JavaScript 2.0, you know what I mean?

JUSTIN: Java 2.0.

WILL: Yeah, Java 2.0. Similarly, like, TypeScript is JavaScript, like, 1.1, right? Where, like, you have...no, well, you have strong interop. You had the same project. You had the same codebase. And you're just sort of like, I know that they're different languages for, I think, mostly legal reasons, you know what I mean? Like, there's two big organizations butting heads, and like, so, like, the Kotlin people just forked it, you know? TypeScript is the de facto standard. Like, TypeScript took over JavaScript. Like, I don't even think they should call it TypeScript anymore because if you're writing JavaScript-JavaScript, like, you should stop [laughs].

But that's it. That's my point. Like, I agree, but I actually think it's more of a...that's more of an upgrade, you know what I mean? Like a major version upgrade of your programming language rather than, like, redoing it, you know what I mean? Subtle point, but yeah.

MATT: Now, there's ways also to approach this, right? You have a monolith. You decide you want to start extracting microservices. Perfect time. You know, maybe there's technologies that will support the job you're trying to accomplish better than what that monolith was written in for this particular task. And I think those things are super important, as well as how hard is it to find engineers for the languages you're supporting. That's a big one, right?

WILL: I think that's a mistake mostly in hiring, if I'm being direct. Being that, like...how do I put it? Like, I think if you're hiring engineers with language expertise, you're probably doing it wrong because I find, like, language expertise to be...for a good engineer, it's not that big of a deal. You're going to spend...it's a 10 to 1 ratio around learning the codebase, and the company, and the systems, and all that stuff, versus it's this much figuring out how things are done in Acima, and, like, this much learning Ruby. And Ruby is probably one of the hardest languages just because there's so much magic, you know what I mean, associated [crosstalk 14:30]

MATT: Yeah, I agree, when we're talking about senior engineers, right? I've always considered myself language agnostic because I don't care. I mean, I do care a little bit, right? I'm not going back to PHP. I'm just not. But when we're trying to mentor junior engineers, you're moving people over from other departments, that becomes a little more difficult, right? Because the learning curve is already pretty steep. When they start to get used to one language, to throw them into a new one, I think, provides more challenges.

WILL: Yeah, but they get more support.

MATT: Once you have some experience, absolutely, I agree, right?

WILL: I think it's more impactful on a junior engineer because it's like learning a different, like, human language, in that, like, you learn a different language, and you think in a different way, you know? You learn Ruby, and you think in a Ruby way. You learn JavaScript; you think in a JavaScript way. You learn nasty, old C, and you think, in, like, a different way, and you have a different perspective. I think for junior engineers, actually moving around in ecosystems is actually even more important because they'll learn better habits and they'll have more informed opinions.

I think when you have a junior engineer, God help them, right, that makes it through all the way through to senior without ever having sampled, you know, from a combo platter, that's when you get this sort of, like, senior people that are not language agnostic. And those are the people that come in because they don't know any better, and they're like, "This is all I'm good at. All I'm good at is .NET." So, you're a .NET shop now. And those people are dangerous.

MATT: Yeah, well, I agree with that. Point being, it becomes more difficult at that point, right? Because, as a company, we still have to focus on delivery, and it makes it harder to deliver. I agree. And we're all ADD, right? Every single engineer is ADD. We love to learn new things.

WILL: That's because engineering is boring [laughs].

MATT: We love to make things easier on us because we're lazy, and we're very inquisitive. So, we like to solve problems. I love learning new languages. It's one of my favorite things to do, just, you know, that exploration and experience of something new is awesome. But there are some challenges behind it.

JUSTIN: So, Will, you mentioned earlier about the refactoring. And, you know, Martin Fowler wrote the book on "Refactoring." One of the things I vaguely recall him talking about, when you are going in to refactor something, and that's basically what you're doing, is, when you're rewriting something in another language, you are deciding that you are going to refactor it, even if it is two totally different languages or as you say, Matt, like, you're changing...you got a monolith or something, and you have to break it apart. You have to identify the parts that you're going to move over to the new language and do that in an organized manner. And you have to be very cognizant and purposeful about what you are doing.

And those types of things mean that, I mean, help you, if you have to do this anyway, help you on your road to success. So, I don't know, big hot shot coming in and mandating stuff, sometimes you just have to go in and do what they say. And so, if you're going to do it, you might as well do it right. And that involves, you know, being an expert at refactoring and being very purposeful about it, so...

MIKE: You suggested something really interesting because if you extract out portions of an application by one, eventually, you know, you may end up with nothing of the original application because all the pieces have been extracted. So, you've refactored by breaking up into different services. And there was replacement, rather than rework, by extracting services. That's a way you can do this without actually rewriting the application, per se, so much as, hey, you know, this needs to be refactored enough. I'm going to rebuild this component. Once you've built enough components, eventually, the old codebase is gone. There's a kind of a strangulation technique, like a strangler fig. Eventually, the old --

JUSTIN: Yeah, yeah, exactly. And that's the example, the classic example, right? So...

WILL: Well, I mean, and I would also say, like, as I was thinking about it, like, I'm in JavaScript land now, right? And, like, one real serious problem with JavaScript land is framework fever. Man, like, that tribe of developers, they love a framework, God help them. And one thing that I encourage people who are, like, sort of, like, thinking about picking up a new framework to do is, if you want to grab a new framework, grab it as a refactor. Don't grab it for, like, the greenfield stuff. Grab it from, like, this old way, right, where, like, oh, I have this old thing. I'm going to refactor it using this new framework, this new idea, right?

Because what people will so frequently try and do is they'll so frequently try and do, like, okay, well, we're going to do a new thing. And I want to play with my new toys. And we'll do both of them together, which I think is an exponential increase in complexity. And it makes it very difficult to discern whether it was a good idea in the first place, right?

MATT: Yep. You understand what the problem is if you're doing an extraction, right?

WILL: Yeah.

MIKE: And you can A/B comparison.

WILL: Yeah. Like, is this better, or is it worse? And you can also, like, how do I put it? Like, a lot of times, especially with new frameworks, like, you'll have, like, you won't have a lot of the grime that real-world long-term requirements sets entail. You know, like, you'll do it, and you'll do the simple version, right? Versus, like, the actual reality of like, okay, this is a complex line of business app, and a lot of stakeholders have a lot of asks for it.

And you'll sort of, like, you'll skip over all that stuff, you know, and you'll do the tutorial, like, the demo version of something, where it's just like, it's not going to be that simple, and especially for, you know, new, fancy frameworks that haven't been, you know, road tested for a long period of time. Like, a lot of that stuff, like, start getting into the details of like, oh no, I need this. I need this. The security team needs this thing.

And like, well, no, I mean, like, the security team, like, often, they will have asks, like, specific asks about like, "Oh no, I need...anytime you read this thing, I need a log. I need a log of who, you know, even read this document," crazy stuff like that, which, like, well, no, for financial stuff, like, that's for real. Like, I need that. I need, you know, I need to encrypt all this stuff on the server, you know, for whatever thing. You know, all kinds of requirements where it's just like, oh, this is the new, hot JavaScript library. But, like, grimy old Java, or, you know, crusty Ruby on Rails, like, they had to fight these battles already.

MATT: Yeah, I don't like change for the sake of change. I always want to know why. Why do you want to switch everything to Node and TypeScript, right? Is it for performance? Is it for concurrency? I don't think so. But you know what I'm saying? I would like to know the reason for these changes because, oftentimes, people are misled by hype. And you're making change simply for the sake of change. And I don't think that's healthy for anyone, especially not a company whose financials are at risk, you know?

WILL: Okay, so let me ask the counter, right? When is it a good idea to change? Let's say I had, you know, like, kind of a core piece of the company that was written in a programming language that was, you know, like, modern. It was current, but it was, like, say it was something like Haskell, right, where it's difficult to write. It's difficult to hire for. It doesn't offer, like, meaningful performance enhancements, and, like, the team is sort of, like, notably unproductive, right? You know what I mean? Like, when is it worth investing in, like, moving away from the thing, right? It's like, oh, I have COBOL written in the 1950s. Okay, that's an easy layout. That's fine.

MATT: Sure.

WILL: But, like, when does it start to make sense? When is it okay?

MATT: I think you just gave the wise with your question, right? Engineers are really hard to find. Team is unproductive in that language. It's hard to support. You can't move engineers from other teams to go support it, those types of things, right?

JUSTIN: Also, it's like, if it's an older language, all of a sudden, the community is not updating it. You know, I'm putting my security hat on here. They're not updating it for security fixes that are on there and, you know, especially, I've seen that with older libraries that are included in whatever language, right? You're depended upon this library, and, all of a sudden, there's a security hole there, and the community has gone on from that library or that language.

MATT: Absolutely. We run into that all the time with gem support, right?

JUSTIN: Yeah.

MATT: You start using these gems, and, all of a sudden, there's no longer support. Nobody's doing security patches. They're not being updated. And now you're stuck. Either you have to take over support of that gem, write your own, build it into your codebase, or find something different.

MIKE: What if you have, like, a super performant Haskell team that's knocking it out of the park? They've got a good connection to the university. You've got, like, a pipeline of people coming in where you don't have a problem. That would change the parameters of the equation, right?

JUSTIN: Yeah, totally.

MATT: Absolutely. Absolutely.

WILL: If I walked into a high-performing team in, I mean, nearly any language, if they were writing COBOL but it was working, everything was clicking, you know what I mean, like, it wasn't just maintenance mode, it's like we are writing modern COBOL and everything is kicking butt, like, I wouldn't change a thing. [crosstalk 24:57] I mean, I don't have a problem with Haskell, you know what I mean? I actually like Haskell. I've never seen it productive in practice, you know?

MATT: Well, you could throw any language, right?

WILL: Like, one of my favorite languages, one of my favorite languages in the world is, I programmed in, like, five years, is OCaml, right? It's object-oriented ML. It's a very...It was like Haskell or like ML if they were, you know, it was designed by people who were trying to get things done. It's [inaudible 25:29] native. It's really fast. It's like, you know what I mean? And it's practical. Like, you can just sort of like...you can downshift and just do some imperative programming and just get it done if you need to. There's one and only one company in the world that uses OCaml.

MIKE: Jane Street Capital [laughs]?

WILL: Jane Street Capital. [inaudible 25:48]. OCaml wouldn't even exist outside of academia if it wasn't for Jane Street. But, like, it works. It works. Should anybody else use OCaml? Probably no. Like, I love it, but I would not advocate anybody to pick it up for anything, ever, you know.

MATT: You said something that I think is really important and that is, you wouldn't disrupt any team that is productive, just meshes, clicks, and gets things done, and meets business needs. However, we see it far too often. Back to your first example, some new hotshot comes in says, "Hey, we're going to start using this language. We're going to change everything. Turn your world upside down."

WILL: I'm a consultant. I will come in, and I will write this platform for you, but I'll only do it if you let me do it in Haskell.

[laughter]

JUSTIN: Well, going back to the situation I'm in right now –-

WILL: I might get emails for this. I know I am.

[laughter]

JUSTIN: So, the company I'm working for right now was acquired by a big bank, which is a Java shop, and they were acquired in 2023. They came in and dictated the Ruby application, which made the company, the startup company, the money, and made it all attractive; they dictated that, hey, this needs to be rewritten in Java.

MIKE: Oh wow.

JUSTIN: And they spent six months trying to do that. You know, there were zero Java engineers. And the big company did not want to give the resources or, you know, provide the expertise and the guidance and everything. Six months later, they said, "We're not doing it. You guys –- "

WILL: Who said it?

JUSTIN: The company, the smaller company that got acquired they, said, "If you want us to keep on making new features and everything and, you know, improving this product, we can't convert this to Java like you want us to. We got to stick on our platform that we want, I mean, that we're on, that we have the expertise on, that we have the, you know, the domain knowledge and everything." And, you know, so, basically, six months of development time went down the drain. Luckily, all that happened before I got there.

MIKE: [laughs].

WILL: That, like, I mean, so I guess the big question is, like, you know, what is the big bank going to do? I mean, because, like, you know what I mean? And so, the dangerous part about it is like, that's such a transparently bad idea, right?

JUSTIN: Yeah.

WILL: You've got technical requirements that, like, a Ruby on Rails application cannot achieve, right? Then, okay. But if you're just sort of, like, saying it just to say it, like, you are already making bad engineering decisions, right? And, like, are these guys going to be able to turn the corner and say, like, "Oh, I'm sorry. We were wrong"? You know what I mean? Like I said, risky thing to do. And if the big bank had the flexibility to say, like, "Okay, okay, okay, just do your thing then," I mean, [inaudible 28:56]. We all make mistakes. It takes a lot to learn from them. I'd say that's bleeding edge, big bank agility, and flexibility.

JUSTIN: It was, and they accepted it. And they said, "Okay, you guys can come in as you are." They have to come into the environment, right? And then, they were like, "Okay, if you come in, though, you do have to use our compliance framework and everything like that." And we have to add support for Ruby, which wasn't there before. And so, all of that is a pain, but it's all doable. And it's all fitting within the, you know, within the existing architecture, so...

WILL: And I'd say, well, I'd say, like, it's interesting to me, right? So, okay. So, let's get you in trouble at work, right? So, here's the question that I've got, right? So, all right. You've got Ruby on Rails app, you know what I mean? It is functioning. However, right, it's the odd man out. It's through an acquisition, and you're doing stuff differently from the rest of the bank, you know what I mean? Like, you know, it's an acquisition. It wasn't a merger. Like, Daddy is using Java, and you're using Ruby. And that means you are wrong, not them, right? Because you're swimming against the current, against this entire big bank, and it puts you on an island, right?

JUSTIN: It does.

WILL: And you're in Salt Lake City. You're not in New York. You're in Ruby. You're not in Java, right? Like, I think it is incumbent on sort of like, you know what I mean, like...and, like, there are advantages to it as well, you know what I mean? Because you want to be able to have this interoperability, this interactivity with the rest of the organization. Like, how do you, as the smaller party, how do you integrate into the larger thing?

JUSTIN: So, I'm glad you mentioned that because there is an initiative, and they have hired a few Java engineers to do microarchitecture, the microservice architecture. And so, they are moving over...I'm a little unclear if it's, like, net new or existing functionality, but they are going to be moving piece by piece things over to Java. But it is down low on the priority queue.

Our number one thing on the priority queue is, you know, continued growth, which is one of the reasons why the bank bought us. And the continued growth, and the new features, and everything there, all that, where we can serve our customers, is all still in Ruby. And so, it's a separate small team that are doing very clearly defined success story, you know, microarchitecture success goalposts that they could focus on. And then, they'll move piece by piece over to that. And, all of a sudden, our timeline actually went from, "Hey, you have to do this in the next six months," to, "Hey, you've got a couple of years."

WILL: Well, I mean, not for nothing. I actually, like, sort of, like, I don't know whether you guys are doing this at all, but, like, there is JRuby. There's JVM-compatible Ruby. Like, you can run in the JVM. The JVM can sort of, like, swallow you whole, like, with sort of, like, Java interop. And you can start, like, shading off routines, shading off business logic, you know what I mean? Like, I have moved away from JRuby because, like, it was kind of a lot of work to make it go.

I had an application that ran on JRuby in olden times because, like, the JVM had, like, kind of...I don't want to say the bad, old days of Ruby but, like, bad-er days of Ruby when, like, there were kind of memory stuff and scalability issues. Ruby's been working on it for a while, but, like, it was a hobbyist thing that kind of blew up. And there were funkiness. There was, you know what I mean, like, memory management, especially Ruby had some real issues, you know what I mean. And so, I was restarting a lot of servers a lot of the time.

MATT: It's funny [laughs] --

WILL: And so, JRuby is in there, you know, and you can have that call an in interop. You're not going to be able to just push a button and [laughs] convert the file. But, like, you can call...if you can get over to it, you can call Java, you know, from Ruby, you know, pretty sleek.

JUSTIN: I'll have to take a look at that.

MATT: It's funny --

WILL: Anyway [inaudible 33:19], sorry.

MATT: It's funny you say memory issues while talking about Ruby and Java is in the same sentence [laughs].

WILL: Well, I'm sorry, but, like, Java's [crosstalk 33:29] in ways that Ruby wasn't for a long time.

JUSTIN: Java has gotten a lot better over the last couple of years. And Java has almost, I mean, the last three years, Java looked at Kotlin and stole the best parts of Kotlin. And so, all of a sudden, Java's, you know, picked up a lot of the nice things that Kotlin had, especially, like, the functional programming idiom. A lot of the null pointer exceptions have gone away, things like that.

WILL: Did it get the coroutines? Did they pull the coroutines in?

JUSTIN: I'd have to take a look, so...But, in any case, it'll be interesting to see how it goes forward, but, you know, they are recognizing, like, even if they were to stay in Ruby, I mean, we're currently on a monolith, and so going in that direction anyway we're breaking apart the microservices because we're growing the teams. We're growing the engineering teams. It just makes a lot more sense. And so, it's a natural breaking point anyway that we can move forward to the parent org desired architecture, desired language.

MATT: Yeah, bottom line is we have to roll with the times, right? Unless we're the ones --

JUSTIN: That's very true. I'm glad you brought that up. It's like –-

MATT: And unless we're the ones making these decisions, then we have to adapt, and that's just the bottom line.

MIKE: If anybody is making the decision, though, don't go and destroy your teams and your precious goose that lays the golden egg. Don't do that [laughs].

WILL: Well, I mean, like, you know, it's the classic, like, you know, the classic, like, joke, right? I mean, the difference between a heart surgeon and a car mechanic is the heart surgeon works on a car while it's running, you know. And similarly, like, you know, if you want to be an engineer working on this enterprise stuff, making them enterprise bucks, you don't get to do it the easy way. You have to do it the hard way, where you can maintain the uptime and you maintain, like, continuity. And you're working on the engine as it's turning. I mean, sorry, you know, like you wanted the big check; go and earn it, you know? Like [laughs], stuff's hard.

And also, I'll say like, you know, I'll throw another bomb because, you know, I like stirring the pot. One of the more disastrous clinging to languages that I have seen has been in the Ruby community, where, like, people are trying to force Ruby onto, like, mobile apps, right? It's just not going to go. And some of that is not a technical thing, right? Like, it isn't that I don't think Ruby could make a perfectly, pleasant mobile application. It just won't be tolerated by the powers that be.

And sort of people are just like, well, we're a Ruby shop. We might be the developers and maintainers of Rails. And we're committed to Ruby in a way that, like, is maybe slightly personal. And so, we're going to try and force our way onto the iPhone. And it ain't going to go like that, you know? Write some Swift; write some Kotlin; write some JavaScript, you know. It's going to be all right, guys. Sorry [laughs], you know.

MIKE: Well, and [inaudible 36:46] example, you can back yourself into a corner. The recurring theme here is that, yeah, you got something new, build it in the language you want to do, the one that fits. Just don't destroy what you already have. But if you don't already have it, well, it's a really good time to reevaluate because those choices change over time [chuckles]. And, well, we're talking Ruby, so, you know, Ruby on Rails was really huge 20 years ago. Almost, not quite, but almost.

WILL: God, yeah, almost. No, no, you're right. No, that math is correct. You really bum me out, Mike. [laughter]

MATT: That really hurts.

MIKE: And it's still got a solid community. Like, I'm not saying that Rails is going away anytime soon, but Python has taken over a lot of that space as a very similar language. And --

WILL: You really don't need both, you know?

MIKE: Exactly. And so, you know, you've got to do some serious soul-searching. Maybe you've got some brilliant developers who are really good at what they're doing and, you know, stick with the language that works. But you better ask that question and think about how that question is going to be answered 5 years from now and ten years from now because you don't get that opportunity very often. Because once it's built, that's when you don't get to change it.

WILL: Yep. Yeah, absolutely.

JUSTIN: So, Stack Overflow, of course, does the Developer Survey every year. And so, there you can see the directionality of language usage. So, not necessarily that should make your decision for you, but it should be a consideration. And I'm looking here at Ruby. Let's see here. I'm trying to figure out.

WILL: Share your screen I want to see.

MIKE: Sorry, listeners, you don't get to see the screen.

WILL: I can Google this. I have the Google. Like, I got the whole internet. Stack Overflow. What was the search? Like, language survey, is that right?

JUSTIN: Developer Survey, yep.

WILL: Yeah, Developer Survey. All right, 2023, let's see it. Oh man, here I go. Oof, yep, I see it.

MATT: Python's, like, 30% right now.

WILL: Yeah, Python's, wow. Oof, goes bigger.

JUSTIN: So, here you have desired and admired. And I'm trying to think of...Rust is the most admired language, with more than 80% of developers want to use it and they want to use it again next year. Compare this to the least admired language, MATLAB [laughs]. I don't even know if MATLAB counts as a language.

WILL: It sadly does.

MIKE: It totally does. I've used...there's an open-source equivalent called Octave. I've used it before. It's actually very similar to the R language. Have you ever used R?

JUSTIN: Yeah, I have. But the thing is, I've only ever used that in college, so...

WILL: There's a lot of work that gets done, like, in MATLAB, like, big E engineering, like, type stuff, where you, like, do sort of, like, you know, anyway, yeah, it's a thing. It's a thing that does real work. It's for real.

JUSTIN: Yeah. So, the way I read this is admired versus desired. So, a good example here is Rust. 85% admire Rust, and they want to continue to use it, versus 30% of people who aren't using Rust want to use it. So, it's like, once you start using Rust, you love it. But if you aren't a Rust person, you don't even know what's going on here, 30%. This one here, JavaScript, this one's a really short bar, you know, 40% desire it, but of those who are using it, only 60% want to continue to use it, so...

WILL: I like the TypeScript one. That's the one I like. I'm looking at the same thing, right, where, like, people are just like, no, we really want to...Wait. So, desired versus admired, I'm curious. So, explain the difference between desire and admiration to me one more time because I think I missed something. I think I misinterpreted.

JUSTIN: Okay, so the question itself was, "Which programming, scripting, and markup languages have you done extensive development work in over the past year, and which do you want to work in over the next year?" So, you can answer twice, I guess.

WILL: Okay. So...

JUSTIN: So, which one are you using right now, and which one do you want to use next year? And then, they could be the same answer.

WILL: Okay, so what do you want...something that you want to try and people who want to keep using it. Is that admirable? Like, I tried it, and I liked it is admired. And I want to try it is desired. Is that right?

JUSTIN: So, desired is the second one. And admired means that you answered the same for both languages, I believe.

WILL: Wait, no, because, like, have used it and want to continue. Admired is have used it and want to continue. And then, desired is, want to use. So, like, I want to try it versus I've tried it, and I want to keep going. So, that makes sense to me for Rust because, like, I came up programming nasty, old C. And, like –-

JUSTIN: [laughs]

WILL: Yeah, you could do better. You could do better, and later they did. And, like, a lot of C developers, you know, really want to keep the good parts of it.

MATT: Look at that gap with Erlang.

WILL: Erlang. I'm looking --

JUSTIN: Erlang. Is it even on here?

MATT: Just scroll down, yeah. [crosstalk 42:14]

WILL: I see Elixir.

JUSTIN: So, the people who use it generally admire it, but the people who have never used it don't list it as something that they want to use.

MATT: And Elixir matches.

WILL: I'm looking at C++ and C, and I just don't understand...those don't make sense to me [laughs]. What? Like, I don't get it. Is it people who are like, I'd like to learn how to program in C, and, like, people who want to keep going? I'm unclear what this is because --

MATT: Those are your game programmers.

WILL: Okay.

JUSTIN: Is Unity in C, C#?

MATT: Unity is C#. C#. That is where I learned to write code in C# is because of Unity because I wanted to start building some games.

WILL: Interesting.

JUSTIN: Nice

WILL: I have never done any game programming. I feel like I'm the...well, not since I was in, like, high school, you know?

MATT: It's a lot of fun.

JUSTIN: So, unfortunately, I need to drop. But I suggest people take a look at this. I love this survey because it does help understand, like, here for web frameworks is, you know, the number one thing is React and then Node.js, Next.js, Vue.js. So, you mentioned earlier the JavaScript frameworks.

MATT: Think about those top three as you use them all three together.

JUSTIN: Yeah, you do use them all three together. And then, there's the folks who use Angular [laughs].

MATT: I felt really burned when I built out a huge application in Angular 1, and then [inaudible 43:56]

JUSTIN: Oh yeah, and then Angular 2 is like a whole new language [laughs].

MATT: Yep. Didn't even work.

JUSTIN: Didn't even work.

MIKE: Well, I think that we've reached a good kind of set of conclusions here, which is don't break it if you don't have to. But every time you refactor, which you should be doing, and you should be extracting things, it's a great opportunity to rethink what you're using today. And that's your opportunity. Use it while you have it. You know, it's don't miss the opportunity of a lifetime because you've missed the lifetime of the opportunity [laughs].

MATT: Yep. [inaudible 44:33]

JUSTIN: That's the first time I've heard that. I really like that.

WILL: I still feel like Ruby has enough of a critical mass to make it viable. I think the people who love Ruby really, really love Ruby. They're really good programmers. Like, almost all really good programmers I know that have tried Ruby are like, "Oh, I love Ruby. And, like, in the end, you know, I mean, like, and this is, like, purely personal bias, so I'm contradicting myself, like, rather severely, I know how to turn products around in Ruby really quickly, and if you can do it real fast, real well, you should not screw with it.

But, at the same time, like, I mean, like, everybody can see the direction the wind is blowing, and I wonder whether there's, like, an analogous, like, you know what I mean? I mean, like, what's the analogous, like, legacy language, you know what I mean, where it's like, it was good, and it had a hardcore community, but, like, we shouldn't have...like, what was it? Flash, you know what I mean? Like, can you compare it to Flash?

MIKE: I think Ruby is more like Perl, where it's really good at its niche, and, you know, 20 years from now, people will still be using it, but it's just going to get smaller and smaller and smaller, and you're just going to have...well, I actually saw, like, a comparison once for cars, comparing programming languages to cars. And they said Perl was like an old Volkswagen bus. And I think it was Ruby or Python, it doesn't matter, is like a minivan [laughs], you know? It's the same basic thing, right? It solves the same purpose, and it fits in the same niche. It's going to be the same thing. I think that there's a lot of equivalence there.

WILL: Yeah, I don't know, maybe I should learn Python, you know. Let's just switch over. I mean, it doesn't really matter. I mean, I don't know, I could [inaudible 46:34] really. I'm like, what's the difference between Python? Like, you've got, like, the significant white space, and, like, it's got...it's a little more typier. Is that right or wrong?

MIKE: It's not really even all that typier, no. It's just...so; Ruby leans heavy into, you know, its block. So, anonymous functions are, like, central to the way the language works. So, you don't have loops in Ruby, you know, people don't do loops. It's all built in the language. So, it's very functional in that respect. But Python has much less of a language. It's just much less language. Python just tries to be pseudocode written down as code. So, it doesn't have all of these cool features that you get in Ruby, but it gets the job done.

WILL: I don't know, maybe I should just do Kotlin, you know, just stinky, old Kotlin. Kotlin is fine. I've never done a web app in Kotlin. I just do, like, mobile apps. Anyway, sorry, a little bit of a tangent. We should wrap it up. You were trying to play us out, and I ruined it.

MIKE: I was, but there we are. And I'm with you. I love Ruby. And if it's working for you, keep using it by all means. I'm not pushing back against that. Think about your environment that you're in. Make your choice when you can.

WILL: But I've already abandoned them for, like, front-end stuff. Like, the way Rails is going with the frontend, I don't agree with it, and I don't have to use it, and I don't [laughs].

MATT: No, I'm not a fan of Rails frontend, personally, either. I love Ruby and Rails for APIs. You can work so quickly. It's nice. But is it the future? I guess we'll see.

WILL: No, it's, I mean --

MATT: I mean, that's not the future, is it?

WILL: But that's the reason I'm so frustrated with it because, like, they're just sort of like, they're sort of like, they're digging in their heels. And they're just saying like, the thing that we were doing in 2005 when we came out...anyway, sorry, tangent. We can fight about this next week, maaaybe.

MIKE: [laughs]

WILL: But nothing needs to be said. Like, everybody knows the score with Rails, you know?

MIKE: [inaudible 48:44] Thank you all, until next time on the Acima Development –

WILL: See you guys.

MATT: Thanks, guys.

In this episode of the Acima Development Podcast, host Mike starts by sharing an anecdote about his three-year-old son, emphasizing the importance of guidance and support in learning. This sets the stage for a discussion on onboarding new employees, highlighting the similarities between guiding a child and mentoring new hires. Mike notes that new employees, like his son, have potential but require proper guidance to become productive members of the team.

The discussion then delves into various strategies for successful onboarding. Matt praises Mike's analogy and underscores the importance of mentorship and guidance for new employees. Eddy adds that providing the right tools and environment allows new hires to realize their potential. The group agrees on the significance of assigning buddies to new employees and promoting a supportive culture where asking questions is encouraged. This not only helps new hires learn faster but also fosters a collaborative environment. They also stress the importance of documentation and how it can be a valuable resource for new employees when mentors are not available.

Towards the end, the conversation shifts to the role of tools and technology in onboarding. The hosts discuss the benefits of standardizing tools to streamline workflows and make the onboarding process more efficient. They also touch on the challenges of remote work and how it can impact the onboarding experience. The episode concludes with a reminder that onboarding is not just about processes and tools but also about building trust and relationships. By treating new hires with humanity and fostering a culture of openness and support, companies can ensure a successful and smooth onboarding experience!

Transcript:

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting today. And with me, we've got Eddy, Ramses, and Matt.

And I'm actually going to start today by talking about my family. I have a three-year-old, and [chuckles] he is very much three years old [laughs]. He is figuring things out, and sometimes, he does the wrong thing, like chasing you around and hitting you with a stick. Like, no, that's the wrong thing to do [laughs]. And you tell him that, and he'll be upset and think that you're trying to hurt his feelings. Like, no, I love you. I just don't want to get hit by a stick [laughs].

And, you know, three years ago, he just laid there and cried if he was hungry, right [laughs]? He had a lot of abilities, but none of them were developed at all. And I honestly don't expect him to be a contributing member of society for another 15 years [laughs]. And that's pretty normal because that's true for all of us. We all get this start, you know, we've got this potential, but not really any knowledge about the situation and got to figure out as we go. And it takes a long time before we're productive.

I mention this because today, we're going to talk about onboarding new employees. And, hopefully, you're not putting your three-year-olds or your infants to work [laughs]. But we all start, to some degree, you know, in ignorance. You know, we come into a new job and there's a lot of stuff we don't know. We have a ton of potential, but there's so much that we don't know and things we don't even know we don't know that make it very hard to get up to speed.

And, you know, depending on how well that onboarding process goes, depending on how much support is given to those new employees, it may take them a fairly short period of time to get up to speed or...hopefully, not 18 years [laughs], but it can take a while. I mentioned the support they get, and that's something we can influence. To some degree, there's also the intrinsic complexity of the systems that we're working with that we don't control. There are some things we can do, you know, we can do refactoring, try to improve our systems, but, you know, some problems are hard, and it's going to take people a while to get up to speed. These are real problems, and they're worth talking about.

And that's our topic for discussion today is, how do we onboard? It's also timely because we've got some interns starting on Monday, and we'd really like to give them the support that they need to quickly get up to speed. Now, I have some thoughts about the things that are most important for helping somebody in their onboarding, but I'd like to open the floor. What do you all think about what's important for helping people do that onboarding?

MATT: I'd first like to say you are always so great with your segues and analogies because this one makes perfect sense, right? You have a three-year-old who is learning as he goes. And would he probably figure out life on his own? Yeah, eventually. But without your guidance and without some leadership, it would take much, much longer than it would instead of you providing it for him, right? And I think that's kind of what we have to look at when onboarding new employees, whether it be new to the company, new to a role, or switching departments, you know, any of those things. I think everyone requires a little bit of instruction and guidance until they get up to speed.

MIKE: Absolutely.

EDDY: You mentioned individuals having potential but not the right guidance, kind of paraphrasing a little bit. And any individual that you hire, you interview, and you say, "Yeah, this person's got the right attitude. They got potential," but if you expect this person to hit the ground running, they're going to fall over, or they're going to drown. If you're patient and you provide the right reinforcements, and the right help, and tools, utility, et cetera, the person will grow into their own, and they will show you what you perceived during the interview.

I'd like to believe that some of the individuals who are engineering now are showing that on a day-by-day basis, right? Individuals, you know, who were ambitious, you know, who had the courage, you know, to push forward and be given the opportunity to do that shift in career have now been able to provide all that throughput, you know, that was initially observed during the interview. So, I think that so long as you do provide the patience and the right atmosphere, people will grow into their own, and they will surprise you.

MIKE: I can say that's true of you, Eddy [laughs]. You've done amazing work. And, honestly, I would say that's true of each of you. I know many of you listening are like, oh, I don't care about that. I do. I'm going to call it out publicly. I'm working with three people that I'm very happy to be working with.

MATT: And I second that. One of the things that I think is really important and maybe sometimes we overlook when we're onboarding people is that in this process, we can also empower them to be mentors and help them be able to onboard the next people that come on. And I've found, at least throughout my career, that one of the important things as the mentee or new person being onboarded is to ask questions and not be afraid to ask those questions. And as mentors, we can help guide that with our new people and encourage that. That can go a long way.

MIKE: I think that culture does make a huge difference. If you shut down any question quickly, there's not going to be very many more questions, and you do a lot of harm. Back to the three-year-old, you know what three-year-olds do? They ask questions. "What's this? What's this? What's that? What's this? What's this [laughs]?" And you can shut that down and not respond. And the result is that curiosity will be lost, and that's an awful outcome. You wouldn't want it to happen to your three-year-old. I don't think you want it to happen to your new employee, either. That curiosity is probably a lot of why you want them there—is that hunger to grow.

MATT: Yeah, and it creates isolation if you do shut it down. As annoying as those "What's this?" questions might be, that's how they're learning, and that's how they're gathering all this new information. And will they retain all of it at once? Probably not. But that's why they need to continue asking these questions.

You know, I've been in the industry forever now, but here with the company, I'm in a new role, and Mike is present in most of the meetings I am. And I am one of those people who's always asking questions. And it may feel a little disruptive at times, but I try to ask the question once and remember those things so I don't have to again, and that way, if someone asks me about those same things, I have the answers for them.

EDDY: Can I say how much talent it takes to understand something that's been directed to you the very first time [laughs]? I will have the individual reiterate three different ways for me to really understand that before they say, "Do you get that?" "Yeah, yeah, I get that [laughs]."

MIKE: Okay, so let's talk about that. So, going, you know, with the analogy we've got here, this morning, my toddler asked me, "What's this?" And he was pointing at a cell phone tower. And I said, "It's a cell phone tower [laughs]." And he now has, like, some label for it that he's probably going to forget [laughs], and that's probably about all he has. He has no concept of what's going on there. He's three. I'm not going to explain to him electromagnetic radiation and, you know, low-frequency waves that can travel through a lot of distance and through even solid materials sometimes, and how those resonate with your antennas.

I can just say, "That's a tower," right? If you don't have something to hold that information onto, to kind of pin it to, it's like trying to remember 20 numbers in a row. You're not going to remember it. Your brain won't do that. You can only hold about seven. But if you have that narrative, if you actually have the story, you have something to pin it to. So, you're going to need somebody to repeat and repeat and repeat with greater depth each time. Because the first time, you might, if you remember it, learn the name. The second time, maybe you have a little more story around it, and so on. And with each iteration, you're going to internalize more information.

MATT: Yeah, and let's face it, when you're starting somewhere, you're drinking from a fire hose, and you definitely can't retain everything that you're being told. You know, you just have to try and pick the things that you think are important and hold on to those, and maybe they aren't the right thing sometimes. But you're right; things do take more than once to really ingrain.

EDDY: In order for me to reassure myself that I've been able to internalize something new, I try to explain it to someone else who has no context, right? I read this article a few weeks ago that basically says if you're having a hard time explaining it or dumbing it down, you probably don't understand it. And so, I've gotten into the habit I'm like, oh yeah, cool, I think I get it. But let me try to explain it to someone else and see if they can at least get the concept of what it is. It has done wonders because if I'm able to be like, oh, huh, wait, I don't know if I'm sure about this analogy; let me just make sure, it really harps that ideology down.

MATT: Yeah, if you want a better understanding of something, teach it. And you're probably going to pick up something along the way while you are.

MIKE: Absolutely, and that connects back to the onboarding. One thing I found, so kind of the first tactic I'm going to throw out here that I have found to be extremely helpful for onboarding, is to pair people together who are starting at the same time or starting near the same time. If somebody started three months ago, they have everything fresh in their head. And they're going to be able to probably more easily help the person who just started than actually somebody who's been there a long time. And it has the added benefit is now they're explaining it. Now, they're finding where the gaps are and deepening their understanding.

And if people are starting at the same time, it's kind of a back-and-forth, right? It's this pairing that allows them to do that, you know, in a very tight loop with each other, and it works out really well. There's another tactic I'm going to get to in a minute. But I think that that's one of the most helpful things I've seen is to pair people together who have started at around the same time. Have you all seen that to be similarly effective?

MATT: Yeah, absolutely. And each of them are going to retain different things. So, if you have them together...in fact, we are just going through this on one of my teams. I have two new hires, and it worked out that I could start them both the same day; much, much easier that way because then they can work off each other, as well as it frees up some of our engineering time, right, to be able to do that with two people at once. But it's a much broader conversation when you have three people involved versus two.

MIKE: So, in short, it works, right [laughs]? Put people together. You know, to bring up another tactic that I've found is extremely valuable, is give somebody a buddy. Give whoever is coming in an assigned buddy. Like, "You, this is your friend [laughs], and you're going to work with them." In the absence of that, people don't know who to ask, right? You're like, well, I should ask somebody, and they'll go and try to ask whoever they saw first because [inaudible 12:22] familiar face. You don't know who to talk to. And you can go talk to the person next to you, like, oh, I don't know, right [laughs]? And then, you drop right into that isolation. You need to have a line of contact, somebody that you can ask.

Now, they may not be the right person to answer your question, but they can point you to the person who can answer that. They can point to the documentation. They can bring you down to IT to get you a new laptop. You know, having somebody have a person is just one of the most important things I've seen.

I think I mentioned in a podcast, a recent podcast that we were on, I remember a job I had over 20 years ago, comfortably over 20 years ago now, well over 20 years ago, where I was doing some construction work. And my first day, they set me up with somebody else, and there were some language challenges. We didn't understand each other very well at first, well, mostly me not understanding [laughs] him very well. This was in Louisiana, and he had a French background, but we still communicated. And, man, having that buddy made such a difference. It made such a difference. And pretty soon, I was able to understand. And we worked well together. I learned the ropes, and we actually got to be really close friends.

RAMSES: Do you speak French now, too, Mike?

MIKE: Not a word [laughs]. Well, probably a word but not meaningfully [laughs].

EDDY: How's your Spanish?

MIKE: Not a paid promotion, but I use Duolingo, and it has been helping. I've been studying Duolingo for almost four years daily.

RAMSES: Oh, nice.

MIKE: It has been very beneficial. I can read Spanish pretty well now.

MATT: I do love that we can practice those languages here at this company. I've been trying because I went so long without speaking any Spanish, and Eddy and a number of the other team members have really helped out with that.

EDDY: To be fair, this is a side [inaudible 14:11], but I've always associated programming terminology in English. And I've never once thought, huh, how do I say array in Spanish, right [laughter]? So, like, I can say everything else except array, and I'm like, oh, man, how do you say that, right? But, like, it's kind of funny, right? You're like, yeah, I understand, but not all the lingo.

MATT: Have you looked? Is there a direct translation?

EDDY: There is, and if you're interested, I can tell you.

MATT: I am.

EDDY: Yeah, array in Spanish is regla like a ruler.

MATT: Great. Interesting.

EDDY: It is, yeah.

MATT: So, one of the other things I think is really important for onboarding is to have a plan instead of just flying by the cuff every time we bring someone on, and I know I'm guilty of that. But having a plan, having a schedule, assigning these buddies, assigning a pair, those things, if you have them lined up up front, will really help improve the process.

MIKE: About seven years ago-ish, more or less, Acima was running their first apprenticeship program. And the one who had been running it actually took a position in a different company after they were about two weeks in. And I was brought in to lead that program and a team two weeks in [laughs]. That was a challenge [laughs], I've got to say, because that continuity was lost, you know, and the preparation was [inaudible 15:45]. And I put a lot of time into getting on top of that. And I really wish that I'd had the preparation ahead of time.

In subsequent years, I learned my lesson [laughs] and tried to get well out ahead of that. You know, for an internship or apprenticeship, you know, we've got a schedule. We've got a project. We've got the person they know to cling to and talk to. We've got the buddies lined up. And having those things lined up means that you can come in and have it go much more smoothly.

MATT: I think another new challenge we're facing this year, in particular, is as we're operating under the corporate umbrella, some of these interns are in an entirely different state now, and we're used to having them in office historically. So, that, I think, presents some new challenges that we need to overcome.

MIKE: One thing we're doing with the interns is dividing. So, in general, the Draper office, we should have local folks primarily. But, you know, the pandemic made things really interesting because we were all remote, no face-to-face contact, and that brings with its own set of challenges. Now, we're getting farther away from that. Most companies have at least a hybrid connection where some people are seeing each other sometimes. But, you know, it was all on camera or voice, and you do lose something. You do lose something, some connection. And you have to...I'm not saying that you shouldn't do it. But it takes additional work to be connected when you're not physically in the same space.

MATT: Absolutely. Even working with some of these people that we've worked with that are in our Plano offices for a few years, you don't really feel like you know them until you actually meet them in person and get a little face time. I mean, it improves those relationships so much. I think it's super valuable.

MIKE: Well, even things like...actually, Eddy and I were talking about this before we started the call. Turning on the camera [laughs] when you're working with remote people (It is a little off-topic from our main subject.), but it makes a difference. It gives you more connection. I've done a lot of remote work, decades, actually, of remote work [chuckle], so I'm very familiar with it. But one thing that I used to years ago, partly out of necessity, not have a camera on, you know, it was all just voice or text chatting. But over the years, I've become much more...well, I'll say I've developed a habit of turning my camera on, even when I don't feel like it [laughs] because it can be exhausting to have people staring at you all day. But I try to do it anyway just to establish that greater human connection.

EDDY: We were pretty lucky, Ramses, myself, and Tad, just to put names out there, sorry, guys. We were given new laptops, and we got to upgrade from Intel to Mac, right? And that was awesome. But I was also a little skeptical at first because I'm like, dang, I haven't onboarded in a while. Like, do I still remember how to do this? And we got out pretty quickly, but it's only because I had a bunch of the context already that I didn't know I had, you know, until I had to reapply that again.

And I'm like, oh crap, it is not working. What was that again? Oh yeah, cool, cool, cool. And I was able to fill stuff in. I cannot imagine doing that with this being the very first thing. Like, if I'm an intern coming in and like, hey, set up Kubes. Set up AWS. Set up your bundles. Like, it could go over your head. And I realized that maybe documentation could be a lot better. But in order to help facilitate new onboarding, I think documentation is critical to how fast an individual gets up to speed.

MATT: Absolutely. And, yes, it's really hard without some assistance and guidance setting up, especially our services. We have a pretty complex ecosystem. And I experienced that when I started with the company. It was right when the pandemic hit, and we all got sent home, and I didn't have all of my services set up yet. So, I just kind of had to fumble my way through it, reaching out via Slack where I could. It was tough. So, you're absolutely right. And I think something probably everyone, at least through my experience, can improve on is documentation, especially with onboarding new employees.

MIKE: Well, let's dig into that. We've talked about having a human connection, but that person can't be there all the time. That's just not possible unless you're some privileged person who can pay for a personal mentor [laughs], but, you know, generally, you're not going to get 100% of somebody's dedicated attention. And you're going to need to figure stuff out on your own sometime. How much do you think we should invest into documentation? Because documentation can be very expensive because it's an artifact, just like code is.

One idea that has really stuck with me is the idea that code is debt. Usually, we think, no, we're building this great thing. Well, yes, you built something, but now you've got to maintain it. It may provide value. But what's providing value is that you solve the problems, not necessarily the code itself. And if you have that code, that artifact requires maintenance. Likewise, documentation is the same way, and it doesn't even provide value. It doesn't pay for itself by, you know, having users use it. It indirectly pays for itself by making things go faster. It doesn't mean it's not valuable. But it's expensive. It's expensive and hard to maintain. So, to what degree should we invest in documentation to make onboarding easier?

MATT: Well, I know that you've said this before. I think that it's worth the upfront investment just for the cost savings you're going to get by, you know, increasing the rate at which people can get onboarded and up to speed. But something you've said a number of times, and it's stuck with me, is while we're onboarding, we have those new employees help maintain that documentation and make sure it gets up to date. That way, you're kind of killing two birds with one stone. You're getting people onboarded. They're getting more familiar because they have to go through and help maintain this documentation. So, it will bring up questions that maybe they haven't thought of before. It'll identify gaps in that documentation that they can help fill. So, I personally think it's invaluable.

MIKE: So, you just said something, I think, important here. A way you can keep your documentation fresh is everybody who uses it, particularly this onboarding documentation, updates it and make that formally part of the, you know, the responsibility. It builds on something we've already talked about of, you know, teaching and explaining [inaudible 22:38] the documentation.

EDDY: Actually, I've experienced that firsthand over the past couple of weeks. Part of the project that I'm working on and part of the team is integrating with a third party, right? And a bunch of that has been through the course of APIs and, like, what kind of response, what the shape looks like, what they take, you know, what attributes are required, which ones are optional. And having a robust documentation on that goes leaps and bounds, right, to answering your own questions, and so instead of waiting for individuals to respond and give you the understanding, not just there; it's just in general, right? I just want to harp on, like, how crucial it is to have a robust onboarding, right, that just grows.

MATT: Yeah, and on that, I know the project you speak of. On that project, you guys have also been creating documentation on the flows, things like that. And I'd be willing to bet that you understand it much, much better because you've had to go in and update and create that workflow and help maintain the documentation on our side.

EDDY: Agreed.

MIKE: So, we've talked about pairing people together, putting the new folks together so they can help each other out. We've talked about assigning an experienced buddy. We've talked about documentation. Are there any other key tactics that are helpful in onboarding? And we're speaking kind of generally, but this kind of applies maybe even across industries.

MATT: Yeah. I think something I try to do is help with also building soft skills. And when you're onboarding people, one of the key things is building trust. It's much easier to work with people that you have trust in. And if you can establish that upfront and help build that trust, not only will it help them trust you, but you're going to help build some trust in them as well because you're opening up those communication channels and really establishing that interpersonal relationship that I think is important to work well with people.

MIKE: That's interesting. And I had a thought about building trust. I mean, what is it you do to develop trust? You know, in dogs, you see a dog that wants to show that it's trustworthy, you know, it will, like, roll over on its back, show its belly. It shows vulnerability. It makes itself vulnerable. And one of the most important things, I think, that's maybe a key thing that we do to establish trust is show that we're willing to be vulnerable, show that we make mistakes, show that we're willing to, you know, listen to feedback. Establishing trust, I think, begins largely with yourself being willing to listen. What are your thoughts about establishing trust?

EDDY: Gosh, just having someone you can reach out to when you're in doubt that does wonders, right, to helping you feel better about your job.

MATT: Yeah, I think also, too, let them know they're doing a good job in picking up on things, you know, reinforce that. Help them build some confidence as well.

RAMSES: Related to the documentation point, I think having a system that promotes searchability. Whenever I'm learning a new thing or a new process or working on a new feature, if I don't have a point person immediately available, I like to be able to have a good tool where I can search and find out how the feature works myself. So, it's related to documentation. And I think if our system is documented well, you know, and if they don't necessarily have that point person immediately available, they can at least be able to search and try to find an answer.

MIKE: I actually got a story about that from my first dev job. That searchability is a big deal. I remember that I was writing some code. It was some Windows code accessing libraries. Was it the MDC? Whatever it was back then, a long time ago. And I was just looking through the documentation. And, you know, I looked in the index, you know, looked through the relevant sections of documentation. It was posted, but I couldn't find in the documentation an API that did what I needed to do. And I was kind of giving up. And the guy I was working for Googled it and found the answer.

And this was a big deal because Google was brand new at the time, and I didn't think to use it because I hadn't used Google before. And he pointed this out to say, "You know, I'm not trying to make you feel bad. Google is this amazing tool." Now, there are other search engines now, but that was the one, at the time, that became available and fundamentally changed the way that the documentation was searchable, fundamentally changed the way software development was done.

Because instead of poring through documentation, you could search and actually get an answer. That discoverability was just...I don't know how I would have done, you know, had a career in software without that searchability. And the productivity gains that you get from adding that searchability are amazing. And documentation tends to be pretty opaque. It is hard to get through there and find what you're looking for. That's a really great point, I think, you're making, Ramses, that doing what you can to try to make it searchable is a big deal.

EDDY: I think it's cool. Google is this new thing in our arsenal, in our utility belt that we can use in order to be more productive. Now we have AI [laughs], and that takes it even a bit further. You're like, cool, now I have a catered, like, bot that I can interact with and get even faster results.

MATT: I'm on board the AI train for sure. But you said something there, Mike, that I think I would at least like to ask everyone's view on, and that is tools, right? Google, this new tool, can get you everything you need. What are you guys' thoughts on tooling, such as standardizing IDEs? You know, we use Vi. We use Visual Studio Code. We use the JetBrains tools in our company. And they all are different, very similar, but different. Do you think there's a benefit in standardizing some tools so workflows can become much more similar? Because then you can help with shortcuts, macros, things like that, code generation. Would that benefit in the onboarding process?

EDDY: If we can have everyone just use Visual Studio Code, the whole department will be really happy [laughs].

MATT: Except for a JetBrains user like myself, and I think Mike's a Vi user.

MIKE: I am [laughs]. And, you know, a lot of that's because I had a mentor who taught me Vi, who was doing a lot of server administration. And it got in my tool belt and became [laughs] really useful because I took the time with it. And that's a tricky question, right, because if you've already got people on different technologies, there's going to be a cost to making that uniform.

MATT: And I asked because the same thing, like you, you know, I started in Vi and Vim, moved over to some other tools. I think, at the time, I was using Sublime. And then, we had a project that we did a lot of paired programming on, and everyone I was working with used RubyMine. And I saw their workflows and all the cool shortcuts they were using that was really speeding things up. So, I thought, hey, I'm going to get on board, and I'm going to switch to this because this is what everyone I'm working with is using. And I've kind of just stuck with it since. But I think that was a big part of the onboarding process for that particular project that really helped me out.

MIKE: It's almost inevitable that you'll probably end up using the tooling of your mentor. And even if you don't choose to standardize, and maybe that's a discussion for another day, getting somebody set up in the tooling that you're using so that you can do something in common is probably going to help them move a lot faster.

MATT: We're all going to go back to Macromedia.

[laughter]

MIKE: Oh boy [laughter]. The olden days of yore.

MATT: Yes. It's interesting things to think about.

MIKE: Well, so, we started scratching on tooling. I mentioned, you know, search engines have changed things. I don't know how much AI is going to change things in the future, but I'm sure that it will. And the next generation of AI that goes beyond, you know, text generation, you know, maybe they add reinforcement learning; maybe they add a new algorithm we haven't thought of yet may fundamentally change the game again.

And our industry is like that. It's quickly moving, and there's new tools. And it's easy to miss that and end up getting behind. I think we need to be careful about that. And going back to the idea of trust, maybe your new hires have some good ideas. And we shouldn't get so ingrained in our flows that we're not open to improving our process, and probably should actively, you know, seek for that, look for ideas from new hires. What do they have that they can offer that might change the way we do things?

We recently hired somebody who has a lot of experience with PostgreSQL and using some of the maybe less-used features like search for doing full-text searching. And he's got some great ideas as to how we might employ those. It might change the way we're doing things. But if we don't consider that, then we might end up stuck and never end up doing it better.

A few months ago, we had a new hire who improved our unit test suite, made it go significantly faster by going and making a variety of changes from experience that he had, tooling that we hadn't used before. It made a big difference because rather than just relying on what we knew, we let the new guy show some of his experience. You know, you give a new person, let her, let him do their thing, you may end up learning a lot more yourself than you expected, as well as just teaching.

MATT: Yeah, well, I mean, we hired all of these people for a reason, right? And everyone has something to offer. We just need to be open to listening and seeing those things that they do have to offer us. Yeah, I mean, could you imagine being able to have a conversational chat and just look for something and have it go through all of our Slack history, all of our Confluence docs, all of our Wikis, all in one go, and just be able to ask it a question? "Hey, how do I patch this particular thing because my system isn't running?" And someone's asked that question before in Slack at some point. And it just responds and says, "Oh, well, here's the solution." Powerful.

MIKE: So, we've talked, you know, having a group to work with, having a buddy, documentation, discoverability in that documentation, and flexibility of the tools, taking time to think about tools and work together on tools. Again, these things aren't even necessarily industry-specific, not even work-specific, because a lot of these things could apply to a three-year-old. You know, without thinking about it, we can easily overlook it and not take the time, not invest the time, to make that onboarding process work well. Are there any other key things that we're missing that you all want to bring up?

MATT: As you are summarizing things, I am also taking notes to make sure that not only are we talking about it, we're putting some things into action and helping to improve our process here.

EDDY: Would you think that the individuals who are local can get some benefit with being desk buddies with the individual who's mentoring them?

MIKE: I do. I remember starting at a job a lot of years ago. It was a startup, and they were in a tiny, little office, and [laughs] me and three other people basically just crammed in a corner of this [laughs] space, kind of almost on top of each other. It was hard to even get out of the little corner we were in. And later, we moved to a bigger, nicer office space where we were separated from each other. And then, we ended up just...we ended up very much more isolated. Because we all just ended up messaging each other, doing some sort of text messaging, rather than talking to each other because, you know, you don't want to talk across the aisle and interrupt everybody else around you.

So, we ended up not talking nearly so much as we did when we were kind of forced to be in a closed space. And I'll tell you, when we were in a closed space, I learned so much. I learned so much, and part of it is that I was new. But my understanding of what I needed to do just leaped forward because I was just so closely exposed all the time as to what I needed to learn. Have you had that experience?

MATT: Yeah. You know, what is the likelihood, even though it's right at our fingertips, of someone who's sitting at home working remotely, spinning on something versus sitting next to one of their peers in conversational range just turning their head and asking the question? I personally think that the likelihood of the question in person is going to happen much, much more quickly than if you're isolated at your home, even though we have Slack right here and everyone available. I'm going to ask that verbal question much sooner than I am going to reach out via some form of messenger and ask the question.

MIKE: It requires work. You have to develop a habit of becoming...it feels annoying when you're not used to it. You have to make a real effort to be much more assertive than you would otherwise be when you're in a remote position. And that takes time, and so that's a gap. That's a barrier that absolutely is there that takes time and effort to overcome. Now, maybe you're a remote-first company. That absolutely happens in a lot of cases. If you're in that situation, you have to make that effort. You have to encourage that culture strongly to encourage that inquisitiveness.

MATT: Yeah, and I think by just general human nature, we don't want to be invasive, and we don't want to bug other people. But I think in order to be successful, we have to ask those questions. And the likelihood that the person on the receiving end is perfectly fine with it is pretty high, you know, I mean, people ask me questions, and they think they're annoying me. I am not annoyed whatsoever. If I am available, I'd love to have those conversations. I'd love to answer questions and help people. And I think most people really feel that way.

MIKE: Absolutely. I'm usually buried in questions [chuckles], to tell you the truth. If I'm not in a meeting, I'm answering questions somewhere else. So, I'm answering questions in a meeting, or I'm answering questions in some sort of messaging tool. I don't find that annoying. That's, like, my job, right [chuckles]? To go and help people. It's what I expect people to do.

And if people are not asking the questions, that's worse because then I feel bad. You know, then I'm thinking, well, now you're unable to be effective, and I'm unable to help you because you didn't ask. So, I love the fact that people be proactive. But, again, you have to establish that culture. That's a critical thing for this onboarding is that...and this is brought up a couple of times now, that you have to establish that culture and encourage the safety.

MATT: Yeah. If you're not asking me questions, then I'm asking questions, you know [chuckle], entirely different types of questions than I would be asking. So, I encourage it. And I think we all want everyone to grow and be successful.

MIKE: You know, and that's probably a good place to finish this conversation is that it comes down to that relationship. You know, we're talking about solving business problems, but, in the end, we're people working together, and we should treat people with that humanity. If you want somebody to be successful, treat them like a person [chuckles] and develop that kind of human relationship. Think about what the human needs are and be responsive to them. And I think you would be far more successful and happier as a result, too.

Any final words from anyone?

MATT: As always, I always enjoy being a part of this. So, thank you all for letting me participate.

MIKE: Of course.

EDDY: It's always a treat when we can get Matt [laughs].

MATT: Thanks, Eddy.

MIKE: Well, thank you, Eddy, Ramses, Matt. Until next time on the Acima Development Podcast.