Software at Scale

Naphat Sanguansin was the former TL of the Server Platform SRE and Application Services teams at Dropbox, where he led efforts to improve Dropbox’s availability SLA and set a long-term vision for server development.

This episode is more conversational than regular episodes since I was on the same team as Naphat and we worked on a few initiatives together. We share the story behind the reliability of a large monolith with hundreds of weekly contributors, and the eventual decision to “componentize” the monolith for both reliability and developer productivity that we’ve written about officially here. This episode serves as a useful contrast to the recent Running in Production episode, where we talk more broadly about the initial serving stack and how that served Dropbox.

Apple Podcasts | Spotify | Google Podcasts

Highlights

1:00 - Why work on reliability?

4:30 - Monoliths vs. Microservices in 2021. The perennial discussion (and false dichotomy)

6:30 - Tackling infrastructural ambiguity

12:00 - Overcoming the fear from legacy systems

22:00 - Balking the traditional red/green (or whatever color) deployments in emergencies. Pushing the entire site at once so that hot-fixes can be checked in quickly. How to think of deployments from first principles. And the benefits of Envoy.

31:00 - What happens when you forget to jitter your distributed system

34:00 - If the monolith was reliable, why move away from the monolith?

41:00 - The approach that other large monoliths like Facebook, Slack, and Shopify have taken (publicly) is to push many times a day. Why not do that at Dropbox?

52:00 - Why zero-cost migrations are important at larger companies.

56:00 - Setting the right organizational incentives so that teams don’t over-correct for reliability or product velocity.

Transcript

Intro: [00:00] Welcome to Software at Scale, a podcast where we discuss the technical stories behind large software applications. I'm your host, Utsav Shah, and thank you for listening.

Utsav: [00:15] Hey, welcome to another episode of the Software at Scale podcast. Joining me here today is Naphat Sanguansin who is an old friend of mine from Dropbox and now a Senior Software Engineer at Vise. At Dropbox, we worked on a bunch of things together like developer productivity, and finally on the application services team, where we were in charge of dropbox.com and the Python monolith behind the main website that powers Dropbox. Thanks for joining me, Naphat. 

Naphat:[00:47] Yeah, happy to be here. Thanks for having me. 

Utsav: [00:49] Yeah, I think this is going to be a really fun episode because we can remember a bunch of things from our previous jobs basically. I want to ask you what got you interested in working on Dropbox, on the main website? So there were a bunch of different things we were doing in the past, and at some point, you transitioned to work on various parts of the site. So what got you interested in that?

Naphat: [01:15] Yeah, that's a good question. There are multiple factors but timing, I think, is probably the most important one here. So that was right when I had just moved to a new country, I moved to Ireland, I had switched teams completely, and I was sort of looking for the next big thing to sink my teeth into. And you’ll remember, at the time, Dropbox was also right at the time that they were trying to move the SLA from three nines to three and a half nines, I believe. And what that actually means is, they can only be down for 20 minutes instead of 40 minutes. And so that triggered a bunch of reliability problems, like some assumptions we had made before about how quick pushers can be, how many downtime we can get, things like that. They all failed hard so we needed to unpack everything and figure out how to do this correctly. For historical context, back then, there wasn't-- Let me just back up. 

So Dropbox is built on a large Python monolith server, we actually have a blog post about this. The way to think about it is imagine how you would build a server and if you had just come out of college. So you would probably start with a Python application, [the language 02:34] at the time. You might use some kind of Mythos framework, let's say Django. Dropbox doesn’t use Django, but let's just say something like that. You would find a bunch of endpoints, you write your implementations, and you would just ship the website into it. Dropbox started like that. It's definitely very, very similar to [Inaudible 02:54] company. And then we sort of grew that codebase. So we added a bunch more engineers, added a bunch more endpoints, and fast forward 10 years later, and you're at a place where we have a couple million lines of code in this monolith, we have thousands of machines filling this same copy of code, we have like 1000 plus endpoints. And at that point, the age of this monolith has-- Throughout the entire history of Dropbox, the monolith sort of went through multiple transitions because the age started to show each time. And then back in 2019, it really just came to a head, we couldn't keep going the way we were anymore. Up to that point, also, there wasn't really a dedicated team that owned this endpoint and so it hasn't received real investment in a long time, [Inaudible 03:54]. And now that they’re unraveling all these assumptions we made about it before, we needed to put the investment. And so I was asked if I wanted to come in and look at this and figure out what to do with it, and like, “Okay,” I have actually never looked at the product codebase at all. Prior to that, as you know, we were [Inaudible 04:17] actually were parallel for a lot of years. So prior to that, I was pretty much working on the development side of things. I was doing a little bit of CICD. And it became a new challenge. Lots of interesting things that I haven't thought about before, I decided to take it and see how it goes. And so that's sort of how I came to start working on this monolith, this Python server. And fast forward two years later, and here we are.

So yeah, a lot of people think about monoliths versus microservice or something. If you had to start Dropbox [05:00], 10 years backward, if you had to start your own company today, with your experience, would you not start with a monolith? Or is it just that, at least to me, the opinion is that we just have to continuously invest in the monolith rather than getting rid of it. What are your thoughts?

Naphat: [05:16] Yeah, I always believe that we should always do what's practical. And the monolith model served Dropbox really well for the past 10 years or something. It's really when Dropbox grew 1000 engineers, and 1000 commits per day, that's when it really started to breakout. If you're at a startup and you ever get to that point, you're already very successful. So I will say do what’s practical. If you start with microservices from the beginning, you're putting a lot of infrastructure investment upfront, whereas you could be spending time working on things like actually getting your product off the ground, making sure that you have product-market fit. So in a way, I would say the journey Dropbox went through is probably going to be very typical for most startups, maybe some startups will invest in it sooner than later, they might do continuous investment over the lifecycle of the startup, it really depends on what the company needs are and what the problems are. So there isn't a one-size-fits-all here. Once you get to a size like Dropbox, where you would probably have a tens to 100 team working on the entire monolith, that's okay. That's when it starts to make sense that okay, splitting this apart into separate entities might make more sense, you're able to push independently, you're able to not be blocking other things. But there's a huge spectrum in between, it’s not one or the other. Yeah.

Utsav: [06:51] So how do you tackle a problem as ambiguous as that? So you want to go from 99.9 to 99.95. And what that means, as you said, was your downtime per month can be no more than 20 minutes. How do you unpack that? And how do you start thinking about what you need to solve?

Naphat: [07:12] Yeah, that's a good question. So we pretty much approached everything from first principles when we started looking into this project. So we put ourselves on-call first of all, and then we started looking into what are the various issues that come up. We look at historical data going back years, and what are some of the past outages that are common in the past, and what the themes are. And we also started talking to people who actually have been looking at this for a while and just figuring out what their perspectives are, and trying to get the lay of the land for ourselves. Once we had all this information, we just started assembling what we believe the biggest hotspots were. We knew what the goal was. In our mind, the goal was always clear that 2019, stabilize this as much as possible, get it to a point where we can buy ourselves easily, another year, year and a half to not even look at this, and then figure out what the long term path is beyond the monolith.

So with that goal in mind, in 2019, we had to stabilize it, get to the newest way that we wanted. That allowed us to identify a bunch of problems. So some of the problems that we encountered, for example, was the push itself. When we had to do an emergency push, the push itself could easily take 40 minutes or longer, just because of how unstructured the push is. I think there were like 10 plus clusters in the same department just because no one has ever invested in it. And so the way we were doing the push, we actually took about, I think close to 20 minutes end to end just to do a round restart of the service. And nobody understood why that number was picked. And there was actually a lot of fear using the number. So that's one problem. We knew we had to go fix it. 

We identified a bunch of other problems, again, the general mindset we had really is we cannot be afraid of the monolith. If you’re going to be on-call for this, if you're going to be investing a lot of energy into this over the next year while we build the next thing, we need to get to a point where we know exactly what it’s going to do, and we know exactly where the limits are. And so I think I spent a good one to two months just poking at the service in various ways, taking away boxes, trying to drag utilization up, and then figuring out how it fails and how I can recover from it, figuring out how long to push takes and what exactly goes into the startup sequence, figuring out how long to build takes, [10:00] figuring out what the other failure modes are and figuring out how do we prevent the failure modes, to begin with?  Once we have all the information, we just started getting to work. We knocked down the problems one after another and eventually got to a stable location, a stable place. I think this is true with engineering problems in general. We just had to approach everything from first principles. We just had to make sure that we don't have any biases going in, we don't have any assumptions about how a problem should be solved, and just start. You know, you have to break some eggs to make an omelet, right? So I just started doing things, started poking it, start seeing what it does. Figure out how to do it safely. First of all, figure out how do you get a safe environment to do that in. 

For us, the way we did it was we redirected a small percentage of traffic to a small cluster, and then we only operated on that cluster. So even if that were to go down, it wouldn't be such a huge outage. And that allowed us to mimic what it would look like on a larger cluster. And that gave us a lot of confidence. So I don't know. What's your opinion on all this? I know you also came into this with a different mindset as well in 2020.

Utsav: [11:24] No, I think that makes total sense. I think the idea of splitting off a little bit of traffic to another independent cluster where you can play around and actually get confidence. Because a lot of decisions when you're so large, and all of that context has been lost, since the engineers working on that have moved on, a lot of decisions are fear-based decisions. They're not really rational, and then you kind of backfill your rationalization on like, “Oh, it's always been this way so it can probably never be better.” But you can always test things out. I think what's more interesting to me is when we, I guess went against some industry-standard things, and I think for the better. One example is, in the industry, people talk about red-green deployments where you have another version of the service running in parallel, and you slowly switch one by one. And that's basically not possible when you have like 2000 machines or just the large cluster. And yeah, if you have a limit of being down for no more than 20 minutes, you can't wait for 20 minutes for a new code push to go out because a bad outage means you've basically blown your SLA, and you can't fix that. So I'm curious to learn or to know how you thought about fixing that, and how you basically validate that approach? Because I know that the solution to this was basically pushing all of Dropbox in parallel, which sounds super scary to me. How did you gain that confidence to be okay with that?

Naphat: [13:06] That's a good question. And let's talk about push in two separate steps, or two separate categories of pushes, let’s say. Let's talk about emergency pushes, which is what we're talking about here about getting a hotfix as quickly out as possible. And this is why you had to mention our SLA. And then we'll get into later on how we do the regular push, and we'll talk about what the nuances are, what the differences between them are. At the end of the day, again, it comes down to doing what’s practical. So what do we know? We know that we have a 40 minutes downtime or 20 minutes downtime SLA. What that means is that for the most part, you should probably be able to push in like five to 10 minutes or 15 at the most. And so how do you do that against 2000 machines? That just completely threw off some pushes. Like you said, we are not going to be able to do any kind of meaningful Canary on all this. That's just not enough time to get it on the beta. So what do we do? 

So we started looking into what the build and push actually does and breaking down what are some of the current timings, which one of them are things that we can never change, and which one of them are things that we can just configure. So what goes into a push? When we tried to sell our hotfix, we track everything in the beginning. We start from people writing code to actually fix it. But let's say that you had to do a one-line fix to something, before we started all this, it took about maybe 10 minutes [15:00] or 15 minutes to actually create the branch and get someone else to do a quick approval and actually commit it in and then make a hotfix branch. So that's usually 10 to 15 minutes lost. We then have another build step that, prior to this, took about five to 10 minutes as well. So that's like, let's say 20, 25 minutes, at this point lost. And then the push itself, in one step, the push took about 20 minutes without any kind of Canary. So we will add about 45, without any kind of Canary, without any kind of safety. So this seems like an impossible problem. So let's break it down. 

Why does it take 10 to 15 minutes to create a commit? It really shouldn’t. The main problem here is that we were using the same workflow for creating a commit for hotfix, as we were for doing regular kind of code review, which they're really, really different. You're doing a hotfix, it should either be a really small change, or it should be something that you already committed somewhere else into the main branch, and you already know with some other validation that it probably works already. Anything else that you’re going to review, you're probably not going to figure it out in 20 minutes anyways. So you're not going to go through the hotfix flow. Oh, and I forgot to mention that when something goes wrong, the first thing you should do is check whether you can roll back, that actually is a lot faster. Now we'll get into that in a bit, and how do we speed it up as well. 

[16:17] So we run a quick tool, just a tool that takes a commit that’s already on the main branch and creates a hotfix branch with the right configuration, and then kicks off the push. This reduces the 10 to 15 minutes initial time to about two minutes. So okay, making good progress here. We then started looking into build. What's going on here? And it turns out that because of how the Build team was structured at Dropbox, a lot of the investment that we made in the build speed, and we had a lot of talk about this externally, they’re using Bazel, we actually cache our builds and all that, a lot of the investment we made was submitted to CI, and not to production build.

And this is not a hard problem, it's just someone has never thought to look into this. And someone has never spent the time to look into this. So coming from a CI team, I knew exactly where the knobs were. And so I just talked to a few of my old teammates, including you, and we figured out how do we actually speed up the production build, and we cut it down to about three to four minutes. So that's pretty good. So between this and creating a commit, we're at about five to six minutes. And that leaves us with another 14 minutes to do the push that we need to. 

[17:42] And now let's talk about the actual push. We need to get the push time down. Twenty minutes is never going to work. So we need to get it down to something that is manageable and something that we believe will be, in the event of a rollback up, will give us more than enough time to make a decision. And so we sort of picked five minutes as our benchmark, as our loss tine, and we wanted to get there. This just came out from intuitively you have 20 minutes, and you need to do a rollback, you kind of want some time to develop and you want to have plenty of wiggle room in case something goes wrong. So let's say five minutes. We took a look at how the push was done and really, the 20 minutes push that was happening was completely artificial. There was a bunch of delays inserted into the push with the fear that if you were to restart a server too quickly, because this is a very large deployment, other services that this server talks to might be overwhelmed when we first create the connection. It is a valid fear. It has never actually been tested. And there are actually ways that we have set up our infrastructure such that it wouldn't have this problem. For example, we actually deploy a sidecar with each of the servers and that's reduced the number of connections to upstream services by about 64x because then you run 64 processes per box. So we have things that we know to be true that make us believe that we should be able to handle much quicker restarts. 

[19:20] So how do we validate it? There really isn't a much better way to do this than to try it out, unfortunately. And so we made the small cluster that I was talking about earlier because at the end of the day, what we care about is-- There are two things we have to validate. We have to validate that the server itself is okay when it restarts that quickly and then we have to validate that the upstream services are okay. With the server itself, you don't need a large deployment. You can validate under small deployment and eliminate one side of the problem. And then with the upstream services, we just had to go slow. [20:00] There isn't really another way to go about this. So we just had to go slow and monitor all the services as we go. So we do 20 minutes to 18 to 16 to 14, eventually to five. And we fixed a bunch of things along the way, of course, because issues did come up. And now we have five minutes push. So if you look at the end to end now, we have about five to six minutes to create a commit, and then five minutes to do the push. That leaves us about nine minutes to actually do extra validation to make sure the push is actually safe. 

[20:35] And so we started thinking, “Okay, maybe we do a very informal Canary,” where if we were to do a hotfix, we probably know exactly what we're pushing, it’s probably only one commit, or rather, we [Inaudible 20:47] so there’s only one commit. And there’s probably only one very, very distinct change. So what if we just very quickly push Canary within one or two minutes, because it's just a subset of the machines, and then just see whether for those new boxes, obviously, you've seen the same errors as before, we have the metrics to tell us that. And this is very different from doing a blue-green deployment, where you would have to create two clusters of machines, one with the old code, one with the new code, and then try to compare metrics between them. But this is just all eyeballing, all looking at exactly what error that we know is causing the site to crash and seeing whether it's coming down. So we built that in. 

[21:37] We also built in another validation step for internal users, where we were to push to a deployment that only failed internal Dropbox users. And this is optional, depending on just how quickly you want to go. And then we codify all this into a pipeline that we can just click through with buttons in the UI. At the end of the day, we got that built and pushed down to about 15 to 18 minutes, depending on the time of day, or depending on how lucky you are with the build, and it worked really well. And then, at that point, it became a question of, “Okay, now that we have this build time and push time down, how do we actually keep this going? And how do we make sure that things don't regress?” Because a lot of the problems that we discovered are things that will regress if you don't keep an eye on it. The build cache could be easily broken again. So we established a monthly DRT that teams that were on-call were supposed to go and try doing the push ourselves and see that it actually completes in time. And then we postmortem every DRT to make sure that, “Okay, if it didn’t complete in time, why didn't it complete in time? If you do the breakdown, where was the increase? Is it in the build? Is it in the push?” and go from there.

Utsav: [22:51] Yeah, and I think one of those things that stands out to listeners, like I know the answer, but I want to hear it again, is that if you push all of Dropbox at the same time, doesn't that mean you drop a lot of requests, like for those five minutes everything is down? And that's what people are worried about, right?

Naphat: [23:11] Right. So we are not exactly pushing everything at the same time. You should think of a push as a capacity problem. So how many machines can you afford to take away at a time while still serving the website? Whatever the number is, take that and restart those machines as quickly as possible. So that's what we did. And so we set the threshold for ourselves at 60%. We never want utilization to go above 60%. And just to be safe, we only took 25% and actually used that for pushing. And then that allowed us to push everything in for budget. And that means that each batch needs to restart as quickly as possible. It's isolated to that one machine and so we will stop traffic to that one machine and just kick it as quickly as possible. It takes about one minute per batch, one and a half minutes per batch, depending on the best.

Utsav: [24:04] But then still, wouldn't you see like a 25% availability here, if you're pushing 25% at the same time?

Naphat: [24:11] No, because we stop traffic to the boxes first and reroute the traffic to the other boxes. So this is why we need to turn it into a capacity problem and make sure that, okay, if you know that any given time the site is never more than 60% utilized, you can afford to take 25% away and then go to 15% overhead.

Utsav: [24:30] Yeah. How do you route traffic to the other boxes? What decides that? Because it seems like this is a complicated orchestration step? 

Naphat: [24:40] It actually isn't that complicated. So the infrastructure at Dropbox is very well structured in this sense. So there is a global proxy. At the time, we actually wrote a few blog posts about it. There is a global proxy called Bandaid. Dropbox [25:00] was in the process of replacing it with Onvoy right when you and I left. But there's a global proxy called Bandaid. This one keeps track of all the monolith boxes that are up. And when I say up, it means passes health check. So when we go and push a box, we make sure it fails health check right away, the global proxy kicks it out of the serving pool within five seconds, we wait for [requested rain 15:16], and then we just kick it as quickly as possible.

Utsav: [25:29] Okay, so it becomes this two-step dance in the sense where the global proxy realizes that you shouldn't be sending traffic to these old boxes anymore. And it can basically reroute and at that time when you can get a new version. 

Naphat: [25:43] Exactly 

Utsav: [25:44] Okay. 

Naphat: [25:45] Exactly, exactly. It's not at all a hard problem. It's just a matter of, “Okay, do we have the architectures to do this or not?” And we do with the global proxy in place.

Utsav: [25:65] Yeah, and with things like Onvoy, I think you get all of this for free. You can configure it in a way that-- And I think by default, I guess once it realizes that things are failing health checks, it can kick it out of its pool. 

Naphat: [26:09] Right.

Utsav: [26:10] Yeah.

Naphat: [26:11] And this is a feature that you need anyway. You need a way to be able to dynamically change the serving size. Machines will go down. Sometimes you add emergency capacity, but I feel like you need the global proxy to have this particular feature. And it comes out of often Onvoy like you said. It's just a matter of how do you actually send the proxy that information? At Dropbox, we did that via Zookeeper. There are other solutions out there.

Utsav: [26:37] And can you maybe talk about any other reliability problems? So this basically helps you reduce the push time significantly, but were there some interesting and really low-hanging fruits that you're comfortable sharing? Small things that nobody had ever looked at but ended up helping with a lot of problems.

Naphat: [26:58] Yeah, let's see. It’s been so long, and I haven't actually thought about this in a while. But one thing that came to mind right away, which is not at all a small one, but I think it's still funny to talk about. So as part of doing filling out the push, we needed to know, what is the capacity constraint on the server? As in how many percent utilization can it go through before it actually starts falling over? The intuitive answer is 100% but severs are like things in life, never that clear or never that easy. And so, among the monolith on-call rotation, there was a common belief that we should never go above 50%. And this bugged the heck out of me when I joined because it seems like we're leaving things unused. But we have seen empirically that when you get to about 60, 70% on some clusters, the utilization often jumps. It jumps from 70 to 100 right away, and it starts dropping requests. We had no idea why. And so that's interesting. How do you actually debug that? Very luckily for us, there were a lot of infrastructure investments that went into Dropbox that allowed this kind of debugging to be easier. 

[28:22] So for example, just before I started working on this, Dropbox replaced its entire monitoring system with a new monitoring system that has 16-second granularity. That allowed us to get a different view into all these problems. And so it turns out that what we thought was 70% utilization was actually something that spiked to 100% every one minute, and then spikes back down to less than around 60% and really averages out to about 70. So that’s the problem. So that turns out [Inaudible 00:29:02] it’s just a matter of profiling. And it turns out that that's just because the old monitoring system that we hadn't completely shut down yet, it would wake up every minute in a synchronized manner because it's the same time. It would wake up on the minute on the clock and do a lot of work. And so the box will be entirely locked up. So one of our engineers on the team said, “Okay, you know what? Until we shut down the monitoring system, what if we just make sure if not synchronized, and we just play it?” That allowed us to go to about 80, 90% utilization. That's pretty good. And that's why we kept it about 60%. 

[29:46] So this is one of those things where we really just had to not be afraid of the things that we are monitoring. And so the way we discovered all this is we just, again, created a small cluster and then started taking [30:00] boxes away to drive the utilization up, and we just observed the graph. And the first time I did this, I was with another engineer, and we were both basically freaking out with each extra box we were taking away because you never know with this particular monolith, how it's going to behave, and we still didn't fully understand the problem. And we actually caused maybe one outage, but it wasn't a huge outage. But it’s fine, this allowed us to actually debunk the actual real cause and now we actually fixed it. And of course, when we shut down the old monitoring system, this problem goes away entirely. So yep, that's one thing. Then you might also ask why we were only at 90% utilization and why couldn’t we go to 100. Because that’s the quirks in how our load balancing works. Our server is impressive and so because of that, our utilization is fixed, unlike most companies. Most companies, when they get to 100%, they can serve a little bit more, they will just slow down everything. For us at 100%, we just start dropping requests. And unless your load balancing is completely perfect and knows about every single machine, every single process, it’s not going to achieve [Inaudible 31:14] 100% and our load balancing isn't perfect.

Utsav: [31:19] That makes total sense. And I think that's why distributed systems engineers should remember the concept of jitter because you never know when you need it, and for how long it will basically waste so much capacity for your company. That’s one expensive mistake. I remember seeing those graphs when we shift that change. It was so gratifying to see. And also, I felt, and I wish that we had added a little bit of jitter and we had never had to deal with this.

Naphat: [31:48] This is the theme of what my 2019 looked like. At the end of the year, we actually sent out a giant email with all the fixes we did. I wish I still had it and I could actually read it to you. But we try and email all the fixes we did. Let's say 60% of them might have been major fixes but 40% of them were all minor fixes like this, all one-liner fixes that we just never invested enough to actually go and look. And so it's funny, but it saved us a lot of energy, it saved us a lot of time. By the end of the year, the on-call rotation was pretty healthy. You joined in 2020, you tell me how healthy it was. But I don't think there were that many hitches by the end of it. We could actually--

Utsav: [32:37] It was surreal to me that the team in charge of pretty much the most important part of the company, in a sense, had lesser alerts than the team that had their own CICD system. I think that just made me realize that a lot of it is about, thinking from first principles, fundamentally, it's just a service. And the amount of time you spend in on-call toil is directly proportional to the amount of investment but inversely proportional to the amount of investment you have into quality and reliability. And things don't have to be that bad, but it just takes a little bit of time to fix and investigate.

Naphat: [33:31] Yep, for sure. And this is just the mindset that I want every infrastructure engineer to have. By the way, so a lot of the things I talk about are things like what you would normally give to a team of SIE to solve. But the people who worked on this with me, only one of them was a full-time SIE. And that's not saying anything against SIEs, I'm just saying that when you are infrastructure engineers, you need to be in a mindset where you don't divide the work completely between SIE and SV. If you have the flexibility, you have the resources, sure. But when you don't, you need to be able to go in-depth. You’re only going in both directions. Infrastructure SIEs should be able to go and do some SV work. SV infrastructure should be able to go and investigate how servers react. And it will just allow us to build better infrastructure in turn. And so yeah, no, this is just-- Looking back, it's kind of funny, it seemed like such an insurmountable problem at the time. But at the end of the day, you know what? We just had to go fix it one by one. And now we just have good stories to tell.

Utsav: [34:33] So then all of this begs the question, now that the monolith is generally reliable in 2020, and the goal for 99.95 is right there, why is the long-term decision to move away from the monolith if it works? What is the reasoning there?

Naphat: [34:55] That's a very good question. [35:00] So what you're talking about is our 2020 project. And it's called the Atlas Project. There's a blog post about it if you Google Dropbox Atlas.

Utsav: [35:07] I can put it in the site notes or in this podcast notes.

Naphat: [35:11 Perfect, perfect. So it is basically a rewrite of our biggest service at Dropbox. Biggest sideline service, I’d say. Dropbox also runs Magic Pocket, that is way bigger than this. That's a staple service. So this is a rewrite of our biggest service at Dropbox. We undertook it for two main reasons. First is developer productivity. And what I mean by that is, at some point, the monolith really starts to restrict what you can do when you have 1000 engineers contributing to it. Well, let’s say five to 600 product engineers contributing to it. What are some restrictions here? For example, you can never really know when your code is going to go out. And this is a huge problem when we do launches because it could be that you're trying to launch a new feature at Dropbox, but someone made a change to the help page and messed that up. And we can’t launch something with a broken help page. And this kind of thing, yes, there should be tests but not everything is tested. It's just the reality of things in software engineering. So what do you do at that point? Well, then you have to roll everything back, fix the help page, then roll back out. 

And actually, we didn't talk about the actual push process, but we actually went to-- We have a semi blue-green deployment similar to that. It takes about two to three hours to run. This is the thing that we used to push out thousands of commits but we have to be more careful because we don’t actually know what we are pushing. So if you had to restart the entire process, you set back any launch you have by two to three hours. So this was a huge problem with Dropbox. There were other problems, of course. You never quite know if someone else is going to be doing something to the memory state that will corrupt your endpoint. You are not allowed to push any more frequently than whatever the cadence that the central team gives you which happens to be daily. When you're doing local development, you can never really start just a special site that you care about, you have to start the entire thing, which takes a long time to build. So the monolith itself really restricts you to what you can do, how productive you can be when you get to a certain size. And again, this is a huge size, I'm not saying every company should do this. We got to a few million lines of code and five to 600 engineers contributing to this, 1000 plus endpoints, so this is an extreme scale. 

On the other note, the other reason we embarked on this project was also this particular server, it was built at the beginning of Dropbox. It used the latest and greatest technology at the time, but it really hasn't caught up to the rest of the company. For example, the server that we were using, there wasn't any support for HTTP/2.0. And so we were, at some point, having to buff where-- It didn't have support for HTTP/1.1 and 2.0. It only supported 1.0. Our global proxy, the Bandaid proxy that we talked about earlier, and also Onvoy, these proxies, they only support HTTP/1.1 and above. And so we were having these two things talk using incompatible protocol for the longest time. For the most part is fine, except for some parts that it's not. And so we would actually have a bug for like an entire week, where, in some cases, we will return a success, 200, with an empty body, just because [Inaudible 39:07] incompatible protocol. We could probably upgrade this Python server but doing so requires a significant amount of work. You already go through with that amount of work, maybe actually we should also think about, “Okay, what else do we want to change here? What else can we do to actually move this in the direction that we all want it to move in? 

And just for context, in all this time that the server existed at Dropbox, every other service at Dropbox had already moved to gRPC. So gRPC was a very well-supported protocol at Dropbox. It was very well tested. There was a team that actually upgraded it regularly and run flow tests on it and all that, but this thing just hasn't kept up. So we needed to find a way to get to something else that is equally supported when we get to gPRC itself. [40:00] So enter Atlas. We decided, “Okay, time to invest in this for real. We bought ourselves time in 2019. Let’s now go do some engineering work, let's figure out how do we build something that we're going to be proud of.” And I take this very seriously because back when I was still doing interviews when I was still based in the US, every time I had to talk about the Dropbox codebase, I sort of talked around the complexity of the monolith because we all know just how bad it was. And when I was actually talking about it, I just had to say, “Yes, it's bad, but also look at all these other things.” I don't want to keep saying that. I just want to say, “You know what? We have a great platform for every engineer to work in, and we should all be proud to work here.” That matters a lot to morale at a company. 

Utsav: [40:51] Yeah.

Naphat: [40:53] And so we embarked on a project. We first started building a team. The team had to be almost completely rebuilt. You joined. That made my day. That made my quarter, let's say. And then we started putting together a plan. It was a collaboration between us and another infrastructure team at Dropbox. We had to do a bunch of research, put together a plan for, “Okay, what do we want the serving staff to look like? What do we want the experience to look like?” We should also get into what our plan actually was and what [Inaudible 41:29] were shipping. Do you have anything that you want to add to the backstory here before we move on?

Utsav: [41:35] No, I want to ask actually a few more probing questions. So if you see other big companies Slack and Shopify, in particular, they seem to have worked around the problem by, first of all, their codebase isn't as old as ours. So I doubt that they’re running into the kind of random bugs that we were running into, but they're still pretty big. And they seem to work around the problem by pushing 12 times a day, 14 times a day, they just push very frequently. And that also gets developers’ code out faster. So that solves one part of the developer productivity problem. I guess why did we not pursue that? And why did we instead decide to give people the opportunity to push on their own, in a sense, or have one part of the monolith’s code being buggy not block the other part rather than just push really frequently?

Naphat: [42:30] Right. I would have loved to get to that model. And that was actually the vision that we were selling as we were selling Atlas to the rest of the company. It's just that the way that the monolith was structured at Dropbox, it wasn't possible to push that many times a day. It wasn't possible to push just a component, you had to push everything. So about pushing many times a day, for example. We couldn't get the push to be reliably automated, just because of how many things we’re pushing and how many endpoints we’re pushing. So we built this blue-green deployment, I'm going to call it Canary Analysis because that's what we called it at Dropbox. So the way it works is that we have three clusters for the monolith. We have the Canary cluster, which receives the newest code first, we have the control cluster that receives the same amount of traffic as Canary, but will be on the same version of code as prod. So it's the same amount of traffic, same traffic pattern, and all that. So during a push, we push Canary and we kick control, so they have the same life cycle. If the Canary code is not worse than control then all the metrics should look better - CPU should look better, memory should look better, all the same, everything like [Inaudible 43:44] and all that should look better. 

We then write a script that goes and look at all these metrics after an hour and just make sure that can you actually proceed forward with a push or not? It turns out that let's say more than half the time, the push wasn't succeeding and so Dropbox actually had a full-time rotation for pushing this monolith. If we were already running into problems every other time, there was just not a feasible way we were going to get to aggressive pushing, not even four times a day, let's say three times a day. We weren’t going to get that. It's not going to fit within the business day. Keep in mind also the Canary Analysis itself takes about two to three hours. And the reason it took that long is also A, we wanted the metric but also, we made each push itself very slow just so that it made the problem we catch it in time, we rollback. So because of how it's built, it's just not feasible. So we would actually have to figure out how to work around all these problems, and Atlas, the project that we were building towards will solve the problem or will at least give us the foundation to [45:00] go forward with that eventually. I remember this because this is one of the first questions you asked me when you joined the team. And I completely agree, and I really hope that Dropbox will get there. 

I left before the project was completely rolled out but having talked to people who are still there, they are very much on target. And going forward, they can easily then start to invest and say, “Okay, you know what? Now that push is reliable, let's push multiple times a day.”

Utsav: [45:29] Yeah, I think philosophically, the reason why people like splitting their stuff up into multiple services, it's basically there's like separation of concerns and all of that. But also, you get to own your push speed, in a sense. You get to push without being blocked on other people. And by breaking up the monolith into chunks, and if not letting people push on their own, at least not blocking their push on somebody else's bad code, I think that basically breaks down the problem and makes things much more sustainable for a longer period of time. Because the monolith is only going to get bigger, we're only going to get more engineers working on multi-products and more features and everything. So that's why I feel like it's actually a very interesting direction that we went into, and I think it's the right one. And yeah, we can always do both in parallel as well. We can also push each part of the monolith like 12 times a day and make sure that that doesn't get blocked. 

Naphat: [46:31] Yeah, but just to make sure that we don't completely give the win to microservices here, there are real problems with microservices. And Dropbox tried really hard to move to the services model a few years earlier, and we couldn't quite get there. There are real problems. Running a service is not easy. And the skillset that you would need to run a service, it's not the same skill set that you will need to write a good product or write a good feature on the website. So we're asking our engineers to spend time doing things that they're not comfortable doing. And we're asking them to be on-call for something that they are not completely comfortable operating. And from a reliability standpoint, we are now tasked with setting the standard across multiple services instead of just dealing with one team. There are real problems with microservices and so this is why we didn't-- 

[47:25] Let's talk about Atlas a little bit. So Atlas, the next generation of Dropbox’s monolith, we sort of tried to take the hybrid approach. First, I said microservices have real problems, but our monolith is not going to scale. So what do we do? So we went with the idea of a managed service. And what that means is, we keep the experience of the monolith, meaning that you still get someone else to be on-call for you. We get our team, our previous team to be on-call, you get someone else pushing for you, in any case, it's going to be automated. It's going to be a regular push cadence, you just have to commit a quarter section-time, and it will show up production the next day. You're not responsible for any of that. But behind the scenes, we are charting everything out into services, and you get most of the benefits of a microservice architecture, but you still get the experience of a monolith. There are some nuances like we do have to put some guardrails into, “Okay, this only works for stateless services, you're not allowed to open a background thread that goes and stuff,” that's going to be very hard for us to manage. You need to use the framework that we are going to be enforcing, gRPC, you need to make sure that your service is well behaved, it cannot just return errors all the time, otherwise, they're going to kick you out, or they're going to yell at your team very loudly, or very politely. So there are rules you have to follow, but if you follow the rules, you're going to get the experience, and it’s going to be easy. So that’s the whole idea with Atlas. And I'm probably not doing it justice, which is probably a good thing to Dropbox but it's really interesting that we got there. With Atlas, it seems like the right balance. It is a huge investment, so it's not going to be right for every company. It took us a year to build it and probably going to be another six months to finish the migration. But it's very interesting, I think it is the right direction for Dropbox.

Utsav: [49:36] Yeah. I think the analogy I like to give is it's like building a Heroku or an App Engine for your internal developers. But rebuilding an App Engine or Heroku, what's the point? You could just use one of those. The idea is that you give them a bunch of features for free. So checking if their service on Heroku is reliable, we do that. We [50:00] automatically track the right metrics and make sure that your route certainly doesn't have only 50% availability. We basically make sure of that. Making sure that your service gets pushed on time, that that push is reliable, making sure for basic issues, operations just happen automatically. So we can even automatically Canary your service. So we push only 10% and we see if that has a problem. All of these infrastructural concerns that people have to think about when developing their own services, we manage all that. And in return, we just ask you to use standard frameworks like gRPC. And the way we can do this behind the scenes is if you're using a gRPC service, we know what kind of metrics we're going to get from each method and everything. And we can use that to automatically do things like Canary Analysis. I think it's a really innovative way to think about things. Because I think from user research, we basically found this very obvious conclusion in retrospect - product engineers, they don't care that much about building out interesting infrastructure. They just want to ship features. So if you think about it from that mindset, everything that you can do to abstract out infrastructural work is okay with people, and in fact, they prefer it that way.

Naphat: [51:29] For sure. And this isn't specific to product engineering, every engineer thinks this way. We have a goal, we want to get to it, everything else is in the way. How do we minimize the pain? And I really liked the way you phrased it that we sort of provide a lot of things under the box. I think one of the directors at Dropbox used the analogy that Atlas will be a server with a battery included in the box. You don't have to think about anything, it just works. It sounds innovative, it probably is. You have to give credit where it’s due. A lot of these other toolings that we have, it already exists at Dropbox, we’re just packaging it together, and we're saying, “Okay, product engineers at Dropbox, here's the interface. Write your code here and you get all of this.” We’re just packaging it, you get automatic profiling, you get automatic monitoring, you get automatic alerting. It really is a pretty good experience, I kind of miss it working at a startup. 

Utsav: [52:28] And I think the best thing is you get an automatic dashboard for your service. You build something out and you get a dashboard of every single metric that's relevant to you, at least at a very high level. You get all of the relevant exceptions. You also get auto-scaling for free, you don't have to tweak any buttons or tweak any configuration to do that. We automatically manage that. And that's the reason why we enforce constraints like stateless because if you have a stateful service, auto-scaling and everything is a little funky. So for most use cases, an experience like that makes sense. And I think it really has to do with the shape of the company. There are some companies where they have a few endpoints, and they have a lot of stuff going on behind the scenes. But with the shape of Dropbox, you have basically like 1000 plus endpoints doing a little bit of slightly different business logic and then there's the core file system and sync flow. So for all of the users and all of the engineers working on these different endpoints, something like Atlas just makes sense for them.

Naphat: [53:40] Right. And there's a real question here about whether we should have fixed this a different way. Like, for example, should we have changed how we provide endpoints at Dropbox? Could we have benefited from a structure like GraphQL, for example, and not have to worry so much about backend? It's a real question. Realistically, at a company this big, with a lot of existing legacy code, we had to make a choice that is practical. And I keep coming back to this, we had to make something that we know we can migrate the rest of the things to, but it still reasonably served the need we need. And this is actually one of the requirements into shipping Atlas. We're not going to build and maintain two systems. The old system has to die. And that constrained a lot of how we were thinking about it, some in good ways, and some not in good ways, but that is just the way things are. 

Someone I respect very deeply said this to me recently, “Don't try to boil the ocean.” We just try to take things one step at a time and move things in the right direction. And just make sure that as long as we can articulate a two to three years, whichever one this to look like, and we are okay with that, that's probably a [55:00] good enough direction. 

Utsav: [55:03] I think, yeah, often what you see in companies, especially larger ones when you have to do migrations is people keep the old system around for a while because it's not possible to migrate everything. But also, you impose a lot of cost to the teams when you're doing that migration. You make them do a lot of work to fit in the new interface. Now, the interesting thing about the Atlas project, which we've written about in the blog post is the fact that it was meant to be a zero-cost migration, that engineers don't have to do some amount of work per existing piece of functionality in order to get stuff done. Of course, it wasn't perfect, but that was the plan, that we can automatically migrate everyone. I think it's a great constraint. I love the fact that we did that but why do you think that was super important for us to do?

Naphat: [55:56] Yeah, again, think we have been burned so many times at Dropbox by tech death and incomplete migration, and so we benefited from all these past experiences. And we knew that, “Okay, if we're going to do this project correctly, we need to complete the migration. And so we need to then define a system that will allow us to easily complete the migration. And we need to assemble a team that will help us complete the migration. So the team that we put together is probably responsible for a good majority of the migrations at Dropbox in the past. And so the experiences that each person brings, like, I've seen one teammate, for example, write some code that passes the Python abstract syntax tree, and then just change some of the API's around for our migration. I was in awe of the solution. I didn't think it would be that easy but he did it in a 50 lines Python file. So I think that it's a very ambitious code you have, and you need to start it in the beginning, you need to know that you're going to be designing for this so that you do everything the right way to accommodate it. But now that you have it, and now that you're doing it, by the end of it you get to reap all the benefits. We actually get to go and kill this legacy server, for example, and delete it from our codebase. We get to assume that everything at Dropbox is gRPC. That is a huge, huge thing to assume. We get to assume that everything at Dropbox will emit the same metrics, and all that, but will not behave the same way. 

But I think it's very satisfying to look back on. I am really glad that we took this time. And we’re patting ourselves on the back a lot, but it's not like this project went completely smoothly. There were real problems that came up. And so we had to go back to the drawing board a few times, we had to make sure that we actually satisfy a real customer need. And as a customer, I mean, Dropbox engineers, our real customers while we're building this platform. But I think we got there in the end. It took a few iterations, but we got there.

Utsav: [58:32] Yeah, and maybe just to close this up, do you have any final thoughts or takeaways for listeners who are thinking about, just infrastructure and developer productivity in general. The interesting part organizationally, to me was, our team was not only responsible for keeping the monolith up, so its reliability, but also for the productivity of engineers. And maybe you can talk about why that is so important? Because if you focus only on one side, you might end up over-correcting.

Naphat: [59:05] For sure. I think one thing I want to focus on is how important it is to have empathy. You really need to be able to put yourself in your customers’ shoes, and if you can't, you need to figure out how to get there, and figure out, “Okay, what exactly do they care about? And what problems are they facing? And are you solving the right problems for them?” You're going to have your own goals. You want to create some infrastructure. Can you marry the two goals? And where you can't, how do you make a trade-off? And how do you explain it? It's going to go a long way into a migration like this because you’re going to need to be able to explain, “Okay, why are you changing my work for? Why are you making me write [Inaudible 59:44]” You can then tell a whole story about “Okay, this is how your life is going to get better, there’s are going to be pushes, you're going to get these faster wheels. But first, you have to do this, and this is your story.” That's a lot better, but it requires [01:00:00] trust to get there. 

So I would say that's probably the most important thing to keep in mind when building an infrastructure or really building anything - make sure you know who your customers are, make sure you have an open line of communication to them. I think there were a few product engineers at Dropbox that I came in weekly and just tried to get their opinions and bring them into the early stages [Inaudible 01:00:25]. And it was great. They sort of became champions of the project themselves and started championing it to their own team. It’s a very amazing thing to be watching, to be of service. But it really comes down to that. Just have empathy for your customers, figure out what they want. And then you really can only do that if you own both sides of the equation. If I only own reliability, I have no incentive to go talk with customers. If I only own workflows, I have no leverage to pull on reliability. So you really need to own both sides of the equation. Once the project matures, and you don't really need to invest in it anymore, you can talk about having a different structure for the team. But given how quickly we were moving, there was really only one way this could have gone.

Utsav: [01:01:16] Sounds good. And yeah, I think this has been a lot of fun reliving our previous work. And it's been really exciting to be part of that project, just to see how it's gone through and seeing what the impact is. And hopefully, in our careers, we can keep doing impactful projects like that, is the way I think about it.

Naphat: [01:01:39] That is the hope, but really just getting to do this is a privilege in its own right. If I never get to do another large project, I mean, I'll be a bit sad, but also not be completely unsatisfied. Yeah, I'm really glad that we got to have this experience. And I'm really glad for all the people, all the things that we got to solve along the way, all the people we got to meet, all the relationships we built, and now all the memories we have. And it just really energizes us for the next thing, I think. 

Utsav: [01:02:15] Yeah. And yeah, thank you for being a guest. I think this was a lot of fun. 

Naphat: [01:02:20] Of course, of course. This is a lot of fun. We should do this again sometime.

Utsav: [01:02:26] We have a lot more stories like this.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Christina Cacioppo and Robbie Ostrow work at Vanta, an automated security and compliance company with a mission to secure the internet. Vanta sets up monitoring via a set of continuous tests to ensure basic security best practices, like mandatory MFA for employees. Each test bubbles up to one or more compliance standards like SOC-2 so that companies can rapidly move their audits and unlock deals.

Apple Podcasts | Spotify | Google Podcasts

This episode is special because of two reasons: I currently work at Vanta, and it’s the first combined interview with both the CEO and the first engineer at the company, which led to an interesting conversation with multiple perspectives.

As usual, the episode focuses on the technology and business of Vanta, and I’ve tried to not go easy on them, even though there’s an obvious bias involved :)

Highlights

My notes are italicized

2:00: “In order to work on a security company, you’d actually best start with compliance company” - compliance is a “hair-on-fire” problem for companies since it helps unlock deals, whereas security is often an afterthought. Solving compliance helps make companies safer since the incentives align better. This idea and the headache of SOX compliance at my previous job convinced me to work at Vanta.

5:00 - Continuous security monitoring vs. snapshots that are double-checked in audits

11:00 - How Vanta was initially built.

17:00 - Should security reports be standardized or extremely customizable per company?

20:00 - How does someone decide on the set of security policies? Do customers ask for advice?

31:00 - How should engineers think of developer productivity for their startups? What has the impact of initial choices like MongoDB and GraphQL been as the company has grown?

40:00 - At what point should a founder decide to hire an engineer? What qualities should the engineer have? At what point should the founder stop interviewing engineering candidates?

52:00 - How to effectively build a brand for a security company? Experiences over the past few years.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Alexander Gallego is the founder and CEO of Vectorized. Vectorized offers a product called RedPanda, an Apache Kafka-compatible event streaming platform that’s significantly faster and easier to operate than Kafka. We talk about the increasing ubiquity of streaming platforms, what they’re used for, why Kafka is slow, and how to safely and effectively build a replacement.

Previously, Alex was a Principal Software Engineer at Akamai systems and the creator of the Concord Framework, a distributed stream processing engine built in C++ on top of Apache Mesos.

Apple Podcasts | Spotify | Google Podcasts

Highlights

7:00 - Who uses streaming platforms, and why? Why would someone use Kafka?

12:30 - What would be the reason to use Kafka over Amazon SQS or Google PubSub?

17:00 - What makes Kafka slow? The story behind RedPanda. We talk about memory efficiency in RedPanda which is better optimized for machines with more cores.

34:00 - Other optimizations in RedPanda

39:00 - WASM programming within the streaming engine, almost as if Kafka was an AWS Lambda processor.

43:00 - How to convince potential customers to switch from Kafka to Redpanda?

48:00 - What is the release process for Redpanda? How do they ensure that a new version isn’t broken?

52:00 - What have we learnt about the state of Kafka and the use of streaming tools?

Transcript

Utsav [00:00]:Welcome Alex, to an episode of the Software at Scale podcast. Alex is the CEO and co-founder of Vectorized, which is a company that provides product called Redpanda and you can correct me if I'm wrong, but Redpanda is basically a Kafka replacement, which is a hundred times faster or significantly faster than Kafka itself. I am super fascinated to learn about why we're building. I understand the motivation behind building Redpanda, but what got you into it and what you learned on process and thank you for being here.

Alexander “Alex” Gallego: Yeah. Thanks for having me though. A pleasure being here. This is always so fun to get a chance to talk about the mechanical implementation. To a large extent to this day, this thing is pretty large now. So I get to do a little bit of code review of the business, and it's always fun to get the details. Yeah. I've been in streaming for a really long time, like just data streaming for like 12 years now and the way I got into it was through startup in New York. I was doing my PhD in crypto. I dropped out, went to work for this guy on a startup called Yieldmo. Doesn't mean anything, but it was an Ad tech company that competed against Google on the mobile market space and that's really how I got introduced to it. The name of the game back then was to use Kafka and Apache Storm and ZooKeeper 32 and honestly it was really hard to debug. So, I think I experienced the entire life cycle of Kafka from the zero point maybe seven release or 0.8 release back in 2011 or something like that. All the way until my previous job at Akamai, where I was a principal engineer and I was just sort of measuring latency and throughput. And so I sort of seeing that I will listen to Kafka and before they required ZooKeeper and all of these things. I guess the history of how we got here is, at first we were optimizing ad then we're using a Storm and this and then Mesos has started to come about. And I was like, oh, Mesos is really cool. The future of streaming it’s going to be. 

You're going to have an Oracle and then something that's going to schedule the container. In retrospect now, that Apache Mesos got archived or it got pushed to the Apache Attic a couple of weeks ago. We chose the wrong technology. I think it was the right choice at the time like Kubernete is barely working on a couple of hundred nodes and Mesos was proven at scale and so it just seemed like the right choice. What we focused on though is streaming. Streaming is a technology that helps you both sort of extract the value of now or in general deal with time sensitive data like a trade or fraud detection, Uber eats are all really good examples of things that need to happen relatively quickly, like in the now and so streaming systems are kind of technology designed to help you both deal with that kind of complexity. So what Concord did was, hey, we really liked the Apache Storm ideas. Back then when it was a [Inaudible 3:17], Storm was really slow and it was really hard to debug this thing called Nimbus and the supervisors and anything, stack traces on poor languages at [Inaudible 3:27] closure and Java. And I was like, I need to figure out three standard libraries to debug this thing. And so we wrote Concord in C++ on top of Mesos, with that squarely on the compute side. And so streaming is really where store and compute came together and then at the end of it, you do something useful. You say, hey, this credit card transaction was fraud claim. That's what I did for a long time. And, you know, long story short I've been using Kafka really as a storage medium. I personally couldn't get enough performance out of it with the right safety mechanics. So in 2017, I did an experiment where I took two optimize edge computers and literally with a wire back to back to each other. So no rack, latency, nothing. It's just an FPF wire connected back to back between these two computers and I measure it. Let me start up at Concord Server I think maybe two, four, something like that, or two, one at the time, a Concord Server and Concord Client and measure what it can drive this artwork to for 10 minutes by both in latency and throughput. Let me turn that down and let me write a C++ program that bypasses the cornel and bypasses the page cache for the storage level too, and see what the hardware is actually capable of. I just wanted to understand what the gap is. Where does this accidental complexity comes from? Like how much funding are we leaving on the table? 

[5:00] The first implementation was 34X tail-latency performance improvement and I was just floored. I took two weeks and I was comparing the bytes of the data on this, just making sure that the experiment actually worked. And so, yeah, honestly, that's, that was the experiment that it got me thinking for a long time, that hardware is so fundamentally different to how hardware was a decade or more ago when Kafka and Pulsar and all these other streaming technologies were invented. If you'd look at it, actually the Linux scheduler, block algorithm and IO algorithms, basically, it's the thing where you send data to a file and the Linux sort of organize it safe for optimal writes and reads is fundamentally different. It was designed for effectively a millisecond level latencies and the new disks are designed for microsecond level latencies. So this is a huge gap in performance improvement. Not to mention that now you can rent on Google 220 cores on a VM. You can rent a terabyte of Ram.

So the question is, what could you do differently with this new [inaudible 6:11]? Like it's so different. It's like a totally different computing paradigm. And I know that [Inaudible6:19] has coined the term, "there's no free lunch." you basically have to architect something from scratch for the new bottleneck and the new bottleneck is the CPU. The delayed [Inaudible 6:30] of this, are so good and same thing with network and all these other peripheral devices that the new bottleneck is actually the coordination of work across the 220 core machine. The future is not cores getting faster; the future is getting more and more cores. And so the bottlenecks are in the coordination of work in the CPU's. And so, we rewrote this thing in C++, and that's kind of maybe a really long-winded way of saying how we got here.

Utsav: That is fascinating. So maybe you can explain a little bit about where would someone deploy Kafka initially? You mentioned like fraud, and that makes sense to me, right. Like, you have a lot of data and you need to stream that. But the example that was, I guess, a little surprising was like Uber eats. So how would you put Kafka in like Uber eats? Like where would that fit in like the pipeline?

Alex: Great Question. Let me actually give you a general sense and then we can talk about that case. Event streaming has been a thing for a really long time. People have been trading systems, but in the modern stack, it's called event streaming. And what is an event? An event is what happens when you contextualize data. So you have some data, let's say a yellow t-shirt. Right. Or like a green sweater, the one I'm wearing today. That's just data doesn't mean anything. But now if I say I bought this green t-shirt with my visa credit card and I bought it from, let's say, just Korean seller that is coming through www.amazon.com. And then I start to all of this context that makes an event. There's a lot of richness there. Implicitly, there's also a lot of time to that transaction. If I buy it today, there's this immutability about this facts. And so event streaming is this new way about thinking on your architecture as this immutable, contextualized data, things that go through your architecture.

And so in the case of Uber eats, for example, when I go into like my Uber eats app and I select my favorite Thai restaurant, and I said, hey, get me number 47, I can't pronounce it, but I know it's the item I always get from the third restaurant across the corner. It’s like Chinese broccoli. And so, it's immutable that I paid $10 for it. It is immutable that the restaurant got it, 30 seconds later after this order. And so you start to produce kind of this chain of events, and you can reason about your business logic as this effectively a function application over a series of immutable events. It's kind of like functional programming at the architectural level. 

And why is that powerful? That's powerful because you can now actually understand how you make decisions. And so to go back to the case of fraud detection its really useful, not in making the decision. Like you can just create a little micro-service in node or in Python. It doesn't matter. And you just say, hey, is whatever is the credit card, both $10,000 is probably a fraudulent for, for buying Thai food. That's not the interesting part. The interesting part is that it's been recorded in an order fashion so that you can always make sense of that data and you can always retrieve it.

[10:01]: So there are these properties about Kafka that Kafka brought to the architect, which were durability. I mean, this data actually lives on disk, and by the way, it's highly available. If you crashed one computer, it's going to live on the other two computers. And so you can always get back this data and 3 it’s replayable. If the computing crash, then you can resume the compute from the previous iterator. And so I think those were the properties that I think the enterprise architects started to understand and see it. They're like, oh, it's not just for finance in any way of doing trades, but it works almost across any industry. Today, we have customers, even us and we're a relatively a young company, in oil and gas measuring the jitter between oil and gas pipelines, where you have this little raspberry pie looking things. And the point of this Kafka pipeline, which was later replaced with Redpanda was just a matter of how much jitter there is on this pipeline. Should we turn it off or it's really cool?

We've seen it in healthcare where people are actually analyzing patient record and patient data. And they want to use new technologies like spark and mail or TensorFlow and they want to connect to real-time streaming. For COVID, for example, we're talking with the hospital in Texas, they wanted to measure their COVID vaccines in real time and alert things for all sorts of suppliers. We've seen people in the food industry. It's like in sports betting. It's huge outside of the United States. To me, it feels like streaming is at this stage where databases were in the seventies, like before that people are writing to flat files and it works like that's the database. Every customer gets a flat file, you read it. Every time you need to change it, you just rewrite the entire customer data. And that's kind of like a pseudo database, but then database gave users and higher level of abstraction and modeling technique. And to some extent, that's what Kafka has done for the developer. It's like, use this pluggable system that has the three Tico to them as the new way to model your infrastructure as an immutable sequence of events that you can reproduce, you can consume, it's highly available. So I think those were the benefits of kind of switching to an architect like company.

Utsav:

Well, those customers are super interesting to hear. And that makes sense, like IOT and all of that. So maybe one more question that would come to a lot of people's mind is, at what point should you stop using something like SQS? Which seems like it provides like a lot of that similar functionality, just that it'll be much more expensive and you don't get to see like the bare-bone stuff, but like Amazon supports that for you.  So why do customers stop using SQS or something like that and start using the Kafka Redpanda directly?

Alex: Yeah. So here's the history. So Kafka is 11 years old now and the value to developers on Kafka is not Kafka the system, but the millions of lines of code that they didn't have to write to connect to other downstream systems. It's like the value is in the ecosystem, the value is not in the system. And so when you think about that, let's think about Kafka as two parts, Kafka the API and Kafka the system. People have a really challenging time operating Kafka the system with ZooKeeper. Even if I know that there's might be some listeners that are thrilled and they're like, oh, ZooKeeper 500 was released. Then we could talk about that about what KRaft means and the ZooKeeper 2 later. But anyways, so if you look at Kafka, it's two things, the API and the system. The reason is, and why someone would want to use the Kafka API, which by the way, Pulsar are also started supporting, it’s not just Redpanda, really like a trend in the data streaming system is you can take spark ML and TensorFlow and [inaudible 00:14:04] and all of these databases, it just floats right in, and you didn't write a single line of code.

Alex: You start to think about the systems of these Lego pieces. Of course, for your business logic, like you have to write the code, it's what people get paid for but for all of these other databases, all of these downstream systems, whether you're sending data to anything: Datadog for alerting, or a Google BigQuery or Amazon Redshift, or [Inaudible00:14:32] to be, or any of these databases. There's already this huge ecosystem of code that works that you don't have to maintain because there’s people already maintaining it and so you're just flogging into this ecosystem. So I would say the largest motivation for someone to move away from a non-Kafka API system, which you know, is before Google gloves pops up and Azure event hub and there's like a hundred of them is in the ecosystem.

[15:00] And realizing that it is very quickly at the community. I think Redpanda, for example, makes Kafka the community bigger and better. We start to expand the uses of the Kafka API into embedded use cases. For example, for this security appliance company, they actually embed Redpanda because we're in a C++, super well footprint, but they didn't embed as processed to do intrusion detection. So every time they see an intrusion into the network, they just write a bunch of data to disk, but it's through the Kafka API and in the cloud, it's the same code. It's just not one Redpanda local instance is the collector. And so I think for people considering other systems, whether it's SQS, or [Inaudible00:15:46] or pops up or Amazon event hub. First of all, there are specific traders that we need to dig really to the detail, but at the architectural level is plugging into this ecosystem of the Kafka API is so important in getting to leverage the last 10 years of work that you didn't have to do and it takes you have five seconds to connect to these other systems

Utsav:

That is really fascinating. I guess large companies, they have like a million things they want to integrate with like open source things and like all of these new databases, like materialize all of that. So Kafka is kind of like the rest API in a sense.

Alex:

I think it's become the new network to some extent. I mean, people joke about this. Think about this, if you had an appliance that could keep up with the throughput and latency of your network, but give you auditability. It gives you access control. It gives you a replay ability. Why not? That I think some of our more cutting edge users are using Redpanda as the new network, and they needed the performance that Redpanda brought to the Kafka KPI ecosystem to enable that kind of use case which is where every message gets sent to Redpanda. It could keep up. It could saturate hardware, but now that they get this tracing and auditability. They could go back in time. So you're right. It's almost like you have the new rest API for micro services.

Utsav: Yeah. What is it about Kafka that makes it slow? Like from an outsider's perspective, to me, it seems like when a code base gets more and more features continued by like hundreds of people over like a long time span. There's just so many like ifs and elses and checks and this and that, that tend to like blow the API service and also slow things down. And then somebody tries to profile and improve things incrementally. But could you maybe walk me through, like, what have you learned by looking at the code base and why do you think it's low? Like, one thing you mentioned was, you just do cardinal bypass and you skip like all of the overhead there, but is there anything inherent about Kafka itself that makes it really slow?

Alex: Yeah. So it's slow comparatively speaking and we spent 400 hours benchmarking before it comes out because I have a lot of details about this particular investment. Let me step back and think. An expert could probably tune Kafka to get much better performance than most people. Most people don't have 400 hours to benchmark different settings of Kafka. Kafka is multimodal in performance, but I can dig it out a little bit. But assuming that you're an expert and assuming that you're going to spend the time to think of Kafka for your particular workload, which by the way, it changes depending on throughput. The performance characteristics of running sustain workloads and Kafka are actually varying. And so therefore you're threading model of areas, the number  of threads for your network and the number of threads for your disk and the number of threads for your background workloads and the amount of memory, I think is this, [Inaudible00:18:51] of tuning Kafka that is really the most daunting task for an engineer. Because it is impossible, I think in my opinion to ask an engineer who doesn't know any of the internals of Kafka unless they go and they read the code to understand, well, what is the relationship between my IO threads and my disk threads and my background workloads and how much memory should I reserve for this versus how much memory do you serve reserve for that. There's all of this, trade-offs that do matter as soon as you start to hit some form of saturation. 

So let me give the details on the parts that where we improve performance which is specifically in the tail latency and why that matters to the messaging is great and the throughput. So by and large, Kafka can't drive hardware to the similar throughput as Redpanda. With Redpanda there's always at least as much as Kafka and in some cases which we highlight is in the block [Inaudible 00:19:51]. In some cases we're a little better. Let's say like 30% or 40% better. The actual improvement in performance is in the tailgate and the distribution. 

[20:00] Why does that matter? I'm just going to focus on what I think that Redpanda brings to the market rather than the negatives of Kafka because I think we are built on the shoulders of Kafka. If Kafka didn't exist we wouldn't have gotten to learn the improvements or understand the nuances of oh, maybe I should do this a little different. So on the 3 latency performance improvement, latency, and I've said this a few times, is the sum of all your bad decisions. That's just what happened at the user level. When you send the request to the micro-service that you wrote, you're just like, oh right should I have used a different data structure? There's no cache locality, etc. And so what we focused on is how do we give people predictable tail-latency. And it turns out that for all of our users, that predictable tail-latency often results in like 5X hardware reduction. So let me materialize. All of this performance improvement, where we are better and how that materializes for users. We paid a lot of attention to detail. That means we spent a ton of engineering time and effort and money and benchmarking and test suite on making sure that once you get to a particular latency, it doesn't spike around. It's stable because you need that kind of predictability. Let me give you a mental example or mental model, which you could potentially achieve really good average latency and terrible tail-latencies.

Let’s say that you write, and you have a terabyte of heap and you just write to memory and every 10 minutes you flash a terabyte. So every 10 minutes you get one request that is like five minutes long because you have to flash the terabyte with the disk and then otherwise the system looks good. So what happens is that people need to understand that you start to hit those tail-latency spikes that Kafka has the more messages you put in the system. Being that you are a messaging system monitor of your users are therefore going to experience the tail-latency. So we said, how can we improve this for users? And so in March we said, let's rethink this from scratch and that really had a fundamental impact. Now we don't use a lot of the Linux Kernel facilities. So there are global locks that happen in the Linux Kernel when you touch global object. For example, the page cache. And I actually think is the right decision for the page cache to be global because if you look at the code, there's a ton of edge cases and things that we have to optimize for it to make sure that it even just work. Then a lot more to make sure that it worked fast. So it's a lot of engineering effort that we didn't know it was going to pay off, to be honest and then he happened to pay off. So we just believe that we could do better with modern art. And so we don't have this global locks kind of at the low level on the Linux Kernel objects and because we don't use the global resources, we've partitioned the memory across every individual core. So memory allocations are local. You don't have this global massive garbage collection that has to traverse terabytes heaps. You have this like localized little memory arenas. 

It's kind of like taking a 96 core computer and creating a mental model of 96 little computers inside that 96 core computer and then it's structuring the Kafka API on top of that. Because again, remember that the new bottle making computer and the CPU is rethinking the architecture to maximize and really extract the value out of hardware.  My philosophy is the hardware is so capable, the software should be able to drive hardware at saturation at all points. If you're not driving hardware saturation at throughput, then you should be driving hardware basically at the lowest latency that you can. And these things need to be predictable because when you build an application, you don't say, oh, let me think about what is my tail latency for this and that and most of the time I need five computers, but there's other 10% of the time we need 150 computers limit. Let's take an average of 70 or 75 computers. So it's really hard to think about building applications when your underlying infrastructure is not predictable and so that's really a big improvement. And then the last improvement on the Kafka API was that we only expose safe settings. We use rapid as a replication model and I think that was a big improvement on the state of the art of streaming. If you look at the actual implementation of Kafka, ISR replication model, Pulsar, I think it's the primary backup with some optimization replication models versus our rapid implementation. You know, that we didn't invent our own protocol. 

[25:00] So there's a mathematical proof of replication. But also you understand as a programmer, oh, this is what I'm used to have two or three replicas. So this is what I meant to have three or five replicas of the memory. So it's kind of all of this context. So that was a long-winded question, but you ask such a critical thing that I had to be very specific just to make sure I don't give room for ambiguity or try to.

Utsav: Yeah. Can you explain why is it important the partition memory per core? Like what happens when you don't? Like, one thing you mentioned was, does the garbage collection that has to go through every day. What exactly is wrong about that? Can you elaborate on that?

Alex: Yeah. So there's nothing wrong and everything works. It's the traders that we want to optimize for is reduced. Basically make it cost efficient for people to actually use data stream. To me, I feel that streaming is in that weird space [Inaudible00:25:57] a few years ago where there's all this money being put into it, but very few people actually get value out of it. Why is this thing so expensive to run and how do we bring this to the masses, so that is not so massively expensive? Basically anyone that has run other streaming system that is in the [Inaudible00:26:17], they always have to over-provision because they just don't understand the performance characteristics to elicit them. 

So let me talk about the memory of partitioning. So for modern computers, the new trend is going to increase in core count the frequency, the clock frequency, the CPU is not going to improve. Here's the tricky part where it gets very detailed. Even on one CPU or CPUs individually, it still got faster even if the clock frequency didn't improve. You're just like, how is this possible? It improved through the very low level things like instructions, prefetching, basically proud execution, like pipeline execution on there's all of these strikes at the lowest level of instruction execution. Even if the clock frequency of the CPU, wasn't getting faster, it made something like, 2X performance improvement or maybe 3X over the last 10 years. But now the actual larger training computing is getting more core counts. My desktop has 64 physical cords. It's like the Verizon 3900.

In the data center, there's also this weird trend, which actually don't think the industry has settled on where even on a single motherboard, you have two sockets. So now when you have two sockets, you have this thing called NUMA memory axes and NUMA domain, which means every socket has a local memory that it makes "low latency access and allocations, whether it worked like one computer," but it can leverage remote memory from the other sockets memory. And so when you rent a cloud computer, you would want to understand what kind of hardware is it. To some extent you're paying for that virtualization and most people are running in the cloud these days. So why is needing the memory to that particular thread important? It matters because like I mentioned, latency is that sum of all your bad decisions. And so what we did is we said, okay, let's take all of the memory for this particular machine and I want to give you an opinionated view on it, which is if you're running this for really larger scale, I'm going to say the optimal production setting is two gigabytes per call. That's what we recommend for Redpanda. You can run it on like 130 megabytes if you want to for very low volume use cases but if you're really aiming to go ham on that hardware, those are kind of the memory recommendations. So why is that important? When Redpanda starts up, it's that I'm going to start one P thread for every core that gives me the programmer at concurrency and parallelism model.

So within each core, when I'm writing code in C++, I code to it like it is a concurrent destruction, but the parallelism is a free variable that gets executed on the physical hardware. The memory comes in in that we split the memory evenly across every cores. So let's say you have a computer with 10 cores. We take all the memory, we sum it up, we subtract like 10% and then we split it by 10 and then we do something even much more interesting. We go and we ask the hardware, hey hardware, tell me for this core, what is the memory bank that belongs to the Relic in this NUMA domain. In this like memory and in this CPU socket. What is the memory that belongs to this CPU socket? And then the hardware is going to tell you- based on the motherboard configuration, this is the memory that belonged to this particular core. And then we tell the Linux Kernel, hey, allocated this memory. 

[30:00] And pin it on this particular thing and lock it. So don't give it to anybody else. And then this thread reallocates that as a single byte array and so now what you've done is you've eliminated all forms of implicit cross core communication. Because that thread will only allocate memory on that particular core, unless the programmer explicitly programs the computer. You've got to allocate memory on the remote core. And so it's kind of relatively onerous system to you get your hands on, but if you're programming an actor model. So what does that mean for a user? Let me give you a real impact. We were running with the big fortune 1000 company, and they took a 35 node Kafka cluster and we brought it down to seven. 

All of these little improvements matter because at the end of the day, if you get a 5.5 X performance improvement, hardware cost reduction at 1600% performance improvement, all of the things. There's a blog post, we wrote a month ago where we talk about one or the other mechanical sort of sympathy techniques that we do to ensure that we give low latency to a Kafka API. And so that was a long-winded way of explaining at the lowest level of why it matters to allocate memory. It all boils down to the things that we were optimizing for, which is saturated hardware, so streaming is affordable for a lot of people, making it low latency so that you enable new use cases like this oil and gas pipeline, for example. And yeah so that’s kind of one of the really deep [Inaudible 00:31:40]. I'm happy to compare with our pool algorithms and how that's different, but that's how we think about building software.

Utsav: Now I wanted to know, what is the latency difference when you use memory from your own NUMA node, versus when you try to access like the remote memory? Like how much faster is it to just stay in your own like core, I guess.

Alex: It's faster relative to the other. I think the right question to ask is what is the latency of crossing that NUMA boundary in relation to the other things that you have to do in the computer? If you have one thing to do, which is you just need to allocate more memory on that core, it'll be plenty fast. But if you're trying to saturate hardware, when you're trying to do this on Kafka, I think then let me give you orders may be made to comparison. Its a few microseconds to cross the boundary that's separate an allocate memory from another core. Just some experiments I did last year to cross the NUMA boundary and allocate memory.

But let me put that in perspective with writing a single page to disk using the NBME with [Inaudible00:32:55] bypass. You could write a page to a NBME device assuming non 3D Cross point technology, just regular NBME on your laptop in single to double digit microseconds. So when you say now a memory allocation is in the microsecond space, you're just like, well, that's really expensive in comparison with actually doing useful work, like send them this link to this. So I think it's really hard for humans to understand latency and unless we work in the low latency phase or have like an intuition for what the computer can actually do, or the type of work that you're trying to do in that particular case. It's really hard to judge, but that adds up now. If you have contention, then it's in like the human time depending on how contended the resources are. Let's say that trying to allocate from a particular memory bank in a remote NUMA node. If there is no memory, then you have to wait for a page fault and stuff to get written to the disk. These things just add up. And it's really hard to give intuition but I think the better intuition is like, let's compare with the other useful things that you need to be doing. They useful thing of a computer is to be used for doing some useful thing for the business, like detecting fraud protection or on our planning an Uber ride around to your house or doing all of these things. I think really expensive in comparison with the things that you can actually be using the computer for.

Utsav: That makes sense to me. And is there any other fancy stuff you do, like in terms of like networking, because recently I've heard that even NIKSUN and everything are getting extremely fast and there's like a lot of overhead in software. Does the Kernel not get things exposed?

Alex: I think this is kind of such an exciting time to be in computing. Let me tell you a couple of things that we do. 

[35:00] So we actually expose some of the latency that the disk is propagating. So let's say you're writing to disk and over time to start to learn. The device is getting busier. So the latency is where one to eight microseconds to write a page and now they're in like thirty to a hundred microseconds to the write a page, because there's like a little bit of contention. There's a little bit of queuing and you start to learn that. At some point, there are some thresholds that we get to monitor because we don't use the Linux Kernel page cache. So we get to monitor this latency and propagate those latencies to the application level, to the raft replication model. Which is very cool when you co-design a replication model with the mechanical execution model, because it means that you're sort of passing implementation details on purpose through the application level. So you could do application level optimizations. 

One of those optimizations is reducing the number of pluses. So raft in order to participate on this, you write the page, and then you flash the data on disk. But we could do it with this adaptive batching so that we write a bunch and then we issue a single flash with like five flashes. That's one thing. The second thing is what this latency gives you is a new computer model. We added WebAssembly. We actually took the V8 engine and that's two modes of execution for that V8 engine currently. One is as a sidecar process and one is in line in the process. So every core gets a V8 isolate, which is like the no JS engine. Inside of V8 Isolated there is a thing called V8 context. And just go into the terminology for a second because there's a lot of terms.

It means that you connect to the JavaScript file. Inside is a V8 Concept. In fact, a context can execute multiple JavaScript files. So why does this matter? Given that Redpanda becomes programmable storage for the user. Think of like the Transformers, when Optimus Prime unite and then all the robots make a bigger robot kind of thing. It's like the fact that you can ship code to the storage engine and change the characteristics of the storage engine. So now you are streaming data, but because we're so far that were like, oh, now we have a latency budget, so we can do new things with the latency budget. We can introduce a computer model that allows the storage engine to do inline transformations of this data. So let's say you sent the JSON object and you want to remove the social security number for DDPR compliance or HIPAA compliance or whatever it is. Then you could just hit the JavaScript function and it will do an inline transformation of that, or it will just obscure for performance. So you don't reallocate just write xxx and then pass it along, but now you get to program what gets at the storage level. You're not ping-ponging your data between Redpanda and other systems. You're executing to some extent. You're really just sort of raising the level of abstraction of the storage system and the streaming system to do things that you couldn't do before, like inline execution or filtering inline execution of max gain, simple enrichments. Just simple things that are actually really challenging to do outside of this model.

And so you insert the computation model where now you can ship code to the data and it executes in line. And so some of the more interesting things is actually exposing that the WebAssembly engine, which is just V8 to our end users. So as an end user, we've now the Kafka API where you say, RPK this command line, things that we run, wasn't deploying to give it a JavaScript file. You'd tell it the input source and the output source and that's it. The engine is in charge of executing that little JavaScript function for every message that goes in. So I think this is like the kind of impact that being fast gives you. You're now have computational efficiencies that allow you to do different things that you couldn't do before.

Utsav: That's Interesting. I think one thing you mentioned where there was like HIPAA compliance or something to get rid of information, like what are some use cases that you can talk about publicly that you've seen that you just would not expect? And you were like, wow, like I can't believe, that is what this thing is being used for.

Alex: Yeah. let me think. Well, you know, one of them is IP credit score. So why that's interesting is not as a single stuff, but as an aggregate of steps, it's really critical. So, let me just try to frame it in a way that doesn't [Inaudible00:39:50] the customers, but we have a massive customer internet scale customer. 

[40:00] They are trying to give every one of their users. So really like profiling information that is anonymous, which is kind of wild. So every IP gets a credit score information. So let's say in America, you have credit scores from like 250 to 800 or maybe 820. And so you give it the same number or like a credit score to every IP and you monitor that over time, but now they can push that credit to score in [Inaudible00:40:27] inside that IP. And then you can potentially make a trade on that. And so there's all of this weird market dynamics. Let me give you this example. Let's say you watch the Twitter feed and you're just like, oh, what is the metadata that I can associate with this particular period and can I make a trade on that? So it's like a really wild thing to do. And then the last one that we're focusing on is this user who is actually extending the WebAssembly protocol, because it's all in GitHub. So you could literally download Redpanda and then stop our WebAssembly engine or your own web assembly engine. Here's actually spinning out very tall computers that are tall both in terms of CPU, in terms of memory and in terms of GPU. He has a [Inaudible00:41:19] job running on the GPU and then every time a message comes in it would make the local a call to this sidecar process that is running this machine learning thing to say, like, you know, should I proceed or should I not proceed with this particular decision? Those are the things that we just didn't plan for. I was like, wow when you sort of expand the possibilities and give people a new computing platforms, everyone will use it. It’s actually not a competing platform. It enriches how people think about their data infrastructure because a spike and Apache Flink, they all continue to work. Like the Kafka API continues to work, were just simply expanding that with this WebAssembly engine. Yeah.

Utsav: I think it's fascinating. Let's say that you want to build a competitor to Google today. Like just the amount of machines that they have for running search is very high. And not that you'd be easily be able to build like a competitor, but at least using something like this will make so much of your infrastructure costs cheaper, that it's possible to do more things. That's like the way I'm thinking about it.

Alex: Yeah. We're actually are in talks with multiple database companies that we can't name but what's interesting is that there's multiple. We are actually their historical data, both the real-time engines and their historical data. So Kafka API and of course Redpanda give the programmer a way to address by integer every single message that has been written to the log. It gives you a total addressability of the log, which is really powerful. Why? So if you're a database, imagine that each one of these address is just like a page on a file system, like an immutable page. So like this database vendors, they're streaming data through Redpanda, and then, it gets written to the Kafka, a batch, and then we push that data to S3. We can transparently, hydrate those pages and give it to them. So they've actually built an index on top of their campaign index that allows them to reason of a Kafka batch, as you know, it's real, the guiding way to fetch pages. It sounded like this page faulty mechanism and can also ingest real-time traffic. And so that's another like really sort of fascinating thing that we didn't think of until we started building this.

Utsav: Yeah. So then let me ask you, how do you get customers ready? You build this thing, it's amazing. And I'm assuming your customer is like another principal engineer somewhere who's like frustrated at how slow Kafka is or like the operational costs. But my concern would be that how do we know that Redpanda is not going to lose data or something and right now it's much easier because you have many more customers. But how do you bootstrap something like this and how do you get people to trust your software and say, yes, I will replace Kafka with this?

Alex: Yeah. So that's a really hard answer because building trust is really hard as a company. I think that's one of the hardest thing that we had to do in the beginning. So our earlier users were people that knew me as an individual engineer. Like, there are friends of mine. I was like, I'm building this would you be willing to give it a shot or try it. So, that only lasts so long. So what we really had to do is actually test with Jepsen, which is basically a storage system hammer

[45:00] that just verifies that you're not b**********g your customers. Like if you say you have a rapid limitation, then you better have a rapid limitation according to the spec. Pilo is a fantastic engineer too. And so what we did is that we were fortunate enough to hire Dennis who has been focused on correctness for a really long time. And so he came in and actually built an extended and internal Jepsen test suite and we just test it for like eight months. 

So it seems like a lot of progress, but you have to understand that we stayed quiet for two and a half years. We just didn't tell anyone and in the meantime, we're just testing the framework. The first rapid implementation, just to put it in perspective, took two and a half years to build something that actually works. That is scalable and this isn't like that overnight. It's like, well, it took a two and a half year to build a rapid implementation and now we get to build a future of streaming. And so the way we verify is really through running our extended Jepsen suite. We're going to be releasing hopefully sometime later this year, an actual formal evaluation with external consultants. People trust us because their code is on GitHub. And so you're just like, well, this is not just a vendor that is saying they have all the cool stuff and underneath it seems just a proxy for Kafka and the Java. And I was like, no, you could go look at it. A bunch of people have bought like 300,000 lines of C+++ code or maybe 200 something. It’s on GitHub and you can see the things that we do.

I invite all the listeners to go and check it out and try it out, because you could just verify this claims for yourself. You can't hide away from this and it's in the open. So we use the thing [Inaudible 46:51] so everyone can download it. We only have one restriction that says we are the only one that is allowed to have a hosted version of Redpanda. Other than that you can run it. In fact, in four years it becomes Apache 2. So it's really only for the next four years. So it's really, I think, a good tradeoff for us. But you get trust multiple ways. One is, people will know you and they're willing to take a bet on you but that only lasts with like your first customer or two. And then, the other ones is that you build a [Inaudible 47:24] like empirically. So that's an important thing. You prove the stuff that you're claiming.

It is on us to prove to the world that we're safe. The third one is we didn't invent a replication protocol. So ISR is a thing that Kafka invented. We just went with Raft. We say, we don't want to invent a new mechanism for replicating data. We want to simply write a mechanical execution or Raft that was super good. And so, it's relying on existing research like Raft and focusing on the things that we were good at, which was engineering and making things really fast. And then, over time there was social proof, which is you get a couple of customers and they refer, and then you start to push petabytes of traffic. I think a hundred and something terabytes of traffic per day with one of our customers. And we thought, some point the system is always intact. If you have enough users, your system is always intact. And I think we just stepped into that criteria where we just have enough users that every part of the system has always been intact but still, we have to keep testing and make sure that every command runs through safety and we adhere to the claims.

Utsav: Yeah. Maybe you can expand a little bit about like the release process. Like how do you know you're going to ship a new version that's like safe for customers?

Alex: Yeah. That is a lot to cover. So we have five different types of fault injection frameworks. Five frameworks, not five kind of test suites. Five totally independent frameworks. One of them, we call it Punisher and it's just a random poll exploration and that's sort of the first level. Redpanda is always running on this one test cluster and every 130 seconds the fatal flaw that is introduce into the system. Like literally, there's like an estimated time in that logged in, not manually, but programmatically and removes your data director and it has to recover from that. Or [Inaudible 49:32] into the system and that's K-9 or it sends like the incorrect sequence of commands to create a topic in Kafka at the first level and that's always running. And so that tells us that our system just doesn't like [Inaudible49:50] for any reason. The second thing is we run a fuse file system and so what it does is then instead of writing to disk, it writes to this virtual disk interface, and then we get to inject the deterministic. 

[50:08] The thing about Fault Injection is when you combine three or five, like edge criteria is when things go south. I think through unit testing, you can get pretty good coverage of the surface area, but it's when you combine like, oh, let me get a slow disc and a faulty machine and a bad leader. And so we get to inject like surgically for this topic partition, we're going to inject a slow write and then we're going to accelerate the reads. And so then you have to verify that there's a correct transfer of leadership.

There's another one, which is a programmatic fault injection schedule, which we can terminate, we can delay, and we can do all these things. There's ducky, which is actually how Apache Kafka gets tested. There’s this thing in Apache Kafka called ducktape and it injects particular balls at the Kafka KPI level. So it's not just enough that we do that internally for the safety of the system but at the user interface the user are actually getting the things that we say we are. And so we leveraged now with the Kafka tests, because we're Kafka API compatible is to just work to inject particular failures. So we start off with three machines. We take Liberty Kafka and, then write like a gigabyte of data and we crashed the machines. We bring them back up and then we read a gigabyte of data. We start to complicate experiments. And so that's the fourth on and then I'm pretty sure I mentioned another one but I know we have five. And so the release process is actually every single committed to Dev gets run for a really long time and ZrZi is parallel. So I think every merchant to Dev takes like five hours or something like that but we paralyze it. In human time, it takes one hour, we just run like five different petals of the time. So I mean, that's how we do it. That’s really sort of the release process. It takes a long time and of course, if something breaks, then we hold the release. And in addition to that, there's like manual tests too because there things that were starting to codify into chaos.

Utsav: I wonder if the Kafka can use the frameworks that you've built for it. And maybe that will be an interesting day when Kafka start using some of that.

Alex : Some of the things are so companies to the big. To launch tests, we have an internal tool called Veto like vectorized tools and the name of the company. So we say veto, give me a cluster and it just pick the binary of from CI, deploys it into a cluster and start particular to the test. And it's specific to our setup. Otherwise a lot of these tests, I think like three out of five are in the open. The things that are actually general people can just go on and look at so it could help. But the other two, the ones that are more involved are just internal.

Utsav: Okay. What is something surprising you've learned about the state of Kafka? You had an opinion when you started Vectorized - that streaming is the future and it's going to be used everywhere. And I'm sure you've learned a lot through talking to customers and actually deploying like Redpanda down to the wild. So like, what is something surprising that you've learned?

Alex: I feel like I'm surprised almost every day. People are so interesting with the stuff that they're doing. It's very cool. Multiple surprises. There's business surprises, there's customer surprises. So from a business, I'm a principal engineer by trade. I was a CTO before, this is the first time I'm being a CEO. It was really, I think, a lot of self-discovery to feel can I do this and how the devil I could do it. So that's one. That was a lot of sort of self-discovery because it started from a place of really wanting to scratch a personal itch. I wrote this storage engine. If you look at the comments, I think to date I'm still at the largest commuter in the repo and I'm the CEO and it's obviously through history. It's because I wrote this storage engine with kind of a page cache bypass and the first allocator and the first compaction strategy because I wanted this to be from a technical level. Then it turns out that when I started interviewing people and we interviewed tens of companies, maybe less than a hundred, but definitely more than 50 somewhere in between, people were having the same problems that I was having personally. 

[55:02] This varies just from small companies to medium companies, to large companies. Everyone knows we're just struggling with operationalizing Kafka.  It takes a lot of expertise or money or both money and talent, which costs money to just get the thing stable. And I was like, we could do better. And so the fact that that was possible, even though Kafka has been around for 11 years, I was shocked. I was like wow. There's like this huge opportunity. I'll make it simple. And you know, what the interesting part about that is that the JavaScript and Python ecosystem, they love us because they're used to running on JS and engine X and this little processes, I mean like later in terms of footprint, like sophisticated code and they're like, oh, Redpanda does this really simple thing that I can run myself and it's easy. They feel we sort of empower some of these developers that would never come over to the JVM and Kafka ecosystem just because it's so complicated and so just to basically productionize it easy to get started, let me be clear, it’s hard to run into stable in production. And so that was a really surprising thing. 

And then from a customer level the thing we used for the oil and gas was really, I think, revealing. The embedding use case and the edge IOT use case was I was blown because I've been using Kafka as a centralized sort of methods hub where everything goes. I never thought of it as it being able to power IOT, like things. Or like this intrusion detection system, where they wanted this consistent API between their cluster and their local processes and, Redpanda was like the perfect fit for that. I think in the modern world, there's going to be a lot more users of that and I think that sort of people pushing the boundaries. I think there's been a lot of surprising things, but those are good highlights.

Utsav: Have you been just surprised at the number of people who use Kafka more than anything else or were you like kind of expecting it?

Alex: I've been blown. I think there's two parts to that. One streaming is an AdSense market. I think Concord just filed their SCC recently. I think saw that today. I think they're going to have a massively successful life beyond. I wish them luck and success, because the bigger they are, the bigger we are too. We add to the ecosystem. I actually see us expanding the Kafka API users to other things that couldn't be done before. I think it's big in terms of its total size, but it also think is big in that it's growing. And so the number of new users that are coming to us about like, oh, I'm thinking about this business idea, how would you do it in real time? And here's, what's interesting about that.

You and I put pressure on products that scientifically translate to things like Redpanda. Say I want to order food. My wife is pregnant and she wants Korean food tomorrow, but she doesn't eat meat, but when she's pregnant, she wants to eat meat. And so I want to order that and it's like, I want to be notified every five minutes. What's going on with the food? Is it going to get here? Blah, blah, blah. And so end users end up putting pressure on companies. A software doesn't run on category theory, it runs on this real hardware and ultimately it ends up in something looking like red Panda. And so I think that's interesting.

There's a ton of new users where they're getting asked by end users, like you and I, to effectively transform the enterprise into this real time business and start to extract value out of the now, what’s happening literally right now. I think to them, once they learn that they can do this, that Redpanda empowers them to do this, they never want to go back to batch. Streaming is also strict superset of batch; without getting theoretical here. Once you can start to extract value out of what's happening in your business in real time, nobody's ever want to go back. So I think it’s those two things. One, the market is large and two, it is growing really fast and so that was surprising to me.

Utsav: Cool. This is a final question. You don't have to answer this. This is just a standard check. If Confluence CEO came to you tomorrow and said, we just want to buy Redpanda, what would you think of that?

Alex: I don't know. I think from a company perspective, I want to take the company, myself, to be a public company and I think there's plenty of room. Like the pie is actually giant. And I just think there hasn't been a lot of competition in this space. It's my view, at least. 

[1:00:00] Yeah and so I think we're making it better. Let me give you an example. Apache web server dominated the HTTP standards for a long time. It almost didn't matter what the standard said is like Apache web server was like the thing. If anything, they implemented that was the standard implied. Then NGINX came about, and people were like, wait, hold on a second, there's this new thing. And it sort of actually kicked off this new, I think, interest in trying to standardize and develop the protocol farther.

I think it's similar. I think it's only natural to happen to the Kafka API. The Pulsar is also trying to bring in the Kafka API. We say, this is our first class citizen and so, I think that there's room for multiple players who offer different things. We offer extremely low latency safety. So we get to take advantage of the hardware. And so I think people are really attracted to our offering from a technical perspective, especially for new use cases. And yeah, and so I don't know. I think there's a chance for multiple players in that.

Utsav: Yeah. It's exciting. I think like there's open telemetric. There'll be like an open streaming API or something that eventually there'll be like a working group that will be folded into CNCF and all those things.

Alex: Exactly. 

Utsav: Well, thanks Alex again for being a guest. I think this was a lot of fun and it is fascinating to me. I did not realize like how many people need streaming and are using streaming. I guess the IOT use case just like slipped my mind, but it makes so much sense. This was super informative and thank you for being a guest. 

Alex:

Thanks for having me. It was a pleasure.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

John Egan is the CEO and Co-Founder of Kintaba, an incident management platform. He was the co-creator of Workplace by Facebook, and previously built Caffeinated Mind, a file transfer company, which was acquired by Facebook.

In this episode, our focus is on incident management tools and culture. We discuss learnings about incident management through John’s personal experiences at his startup and at Facebook, and his observations through customers of Kintaba. We explore the stage at which a company might be interested in having an incident response tool, the surprising adoption of such tools outside of engineering teams, the benefits of enforcing cultural norms via tools, and whether such internal tools should lean towards being opinionated or flexible. We also discuss postmortem culture and how the software industry moves forward by learning through transparency of failures.

Apple Podcasts | Spotify | Google Podcasts

Video Highlights

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Nipunn Koorapati was a Software Engineer at Dropbox, where he worked on two distinct areas - Developer Productivity and Client Sync. He drove many initiatives like consolidating various repositories into a server-side monorepo (read more here), and was part of a high leverage project to rewrite the sync engine, a core part of Dropbox’s business.

Apple Podcasts | Spotify | Google Podcasts

I worked with Nipunn in 2020, and we discovered interesting but unsurprising similarities between the software challenges facing Git and Dropbox. We explore some of the reasons why a tech organization might want to consolidate repositories, some of the improvements being developed into Git like partial clones and sparse checkouts, the similarities between Git and Dropbox, how to think about and roll out a massive, business-critical rewrite, and more.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Ben Sigelman is the CEO and Co-Founder of Lightstep, a DevOps observability platform. He was the co-creator of Dapper - Google’s distributed tracing system and Monarch - an in-memory time-series database for metrics. Finally, he’s also the co-creator of the OpenTelemetry and OpenTracing standards.

We spent this episode discussing Dapper and Monarch - their design, rollout, and lessons learned in practice.

Apple Podcasts | Spotify | Google Podcasts

Video Highlights

Transcript

[Intro] [00:00]: Welcome to Software At Scale, a podcast where we discuss the technical stories behind large software applications. I'm your host Utsav Shah and thank you for listening.

Utsav Shah: Hey Ben, welcome to another episode of the Software At Scale Podcast. Could you tell our guests just about your story, because there's so much in your background that is interesting to me, so right from, starting off at Google, they're like creating LightStep.

Ben:

Sure. Thanks for having me, I'm excited to be here. I don't know whether my background is interesting or not, but to me it's kind of boring, but yeah. I graduated from college right in the thick of the.com bust and the sort of 2003 era, and I was very fortunate to get an offer to work at Google at the time. And when I went over there, they actually put me on some stuff in the ad system that was incredibly boring, to be honest with you. And also of course, ads make a lot of money at Google, but it was this particular part of the ad system that wasn't making any money, so it was kind of boring, not very lucrative for Google and I didn't like it very much. And the way I got into dapper and distributed tracing was actually incredibly arbitrary, but it's a funny story. They had this one time event where you could opt in to this, I don't know what they called it, but it was this program where they would take everyone who opted in. They look at a bunch of different dimensions, like how long you've been at a school, what office you worked in, what languages you worked in, where you were in the org chart; that kind of stuff, and they think they have 10 dimensions. And then they found the person who also opted into this program, who is literally the furthest from you in this 10 dimensional space and then set up a half an hour meeting with no agenda, and that was it. So I was working on this stuff and ads that, as I was to say, totally pointless. And they paired me up with this woman named Sharon Pearl, who was a very distinguished researcher who had come over from Digital Equipment’s Research Lab when it kind of fizzled out after the merger in the late nineties. And she, and some of the other old guards at Google were doing all the really cool system stuff. And she asked me what I was doing. I don't want to talk about it, what are you doing? And then she went through this list of really interesting systems projects. One of them was kind of like a predecessor to an S3; it was like a blob storage . There was some NLP thing she was working on and then in this list was this prototype of a distributed tracing system called dapper that never really saw the light of day, it was just kind of an idea and she described it to me. I just thought it sounded incredibly useful and really fun , and my manager at the time had 150 direct reports, direct reports. I don't think that is more of a hundred, but he had no idea what I was doing, obviously. How could you, and so I just started working on it, basically switch Utsav Shah: Teams or anything.

Ben: 

Well, Google famously had this 20% program, so it was kind of that type of thing, but I really liked it and I thought it was quite valuable actually, and so I moved to New York for personal reasons and I just started working on dapper , full-time my manager Yorick also had like a hundred direct reports. 

So he had also had no idea what I was doing and I got it to the point where it was in production and it was actually solving problems pretty quickly, just because it was IT. Well, I can get into that if you want, why it was possible to do that ,and I got hooked on that stuff and I really haven't looked back. That was early 2005 and now sixteen years later, I'm still basically working in that same overall space of how do you observe complex distributed systems and what you, what kind of improvements can you make to the software engineering process? If you are able to observe them effectively after working on dapper for awhile, I just wanted to do something different. So I went over, did a couple of systems projects that really didn't work that are not well known because they were failures. I'm happy to talk about those too, if you want, but I eventually found my way over to Monarch. I started to create a multitenant high-availability time series database, basically and it in terms of the open source world, probably the closest parallel would be M3 or something like but ended up working on that for about three or four years and then left, Google started a social media company that was as a product of complete failure, about a year into it. I realized that it was never going to work, abandoned the product, but realized I enjoyed being an entrepreneur and I wouldn't even say Pivot's the wrong word, because pivot implies that you keep one foot in the same place. I just started playing a different sport, but with the same investors and that's actually LightStep. LightStep was founded as a social media company in 2013 and a year and a half in, I was just that I completely changed what I was doing added some co-founders at that point and here we are six years after that, and I'm still working on building stuff and really enjoying it. Utsav Shah: Yeah. I think that is a super interesting background. Next, the [05:00] first question on that is, was that the era when like Larry Page or whatever, decided that there's no need for managers, that's why they just hired all of them.

Ben:

I don't think so much they fired them, but they would just hire a lot of engineers and hire managers to go along with it. Utsav Shah: Yeah.

Ben:

I think there was this idea that management was bad in some capacity, and I understand where they're coming from. I definitely don't agree. I think good management is actually one of the most incredible supportive things you can possibly have in an organization. But I think that they were a lot of the people who had come to believe that we're just coming from really bad management. Certainly bad management is worse than no management, but good management better than all of it.

Ben:

Right. Utsav Shah: The other thing that was interesting about Google at the time was that they were growing so quickly that if you didn't like what you were doing, you only had to wait a couple of months and some new person would take over.

Ben:

So that paper's over a lot of issues. I think there was a belief that Google had solved the management problems through software or something like that. 

That was another thing, there was a belief that by writing internal software systems to do a lot of the blocking and tackling that managers might do, and they certainly had tech leads, which serve a managerial purpose for just dividing workup. And there was a belief that they've solved that issue, and once the company everything, there's a law of large numbers, even though Google has been very successful at some point they had to grow slower. And when that started to happen, the need for managers became much more obvious and sure enough, at this point, I don't know what the ratio is, but I'm sure it's not 151 more so that they realized that they needed to correct that. But it was liberating in the sense that you could do whatever you want, but I think it was pretty disorganized and not very efficient. Utsav Shah: Yeah. Could you talk about the architecture? You said you worked on some part of the ad system that wasn't particularly interesting from my understanding, and I could be completely wrong about this, there was one monolith, Google web server, like DWS and not that many services around, is that like roughly accurate? Because I'm also thinking why did diaper make sense if it's just like one large server, but I guess that's clearly wrong.

Ben:

Yeah, I don't think that's correct. Certainly if you go back far enough into 1999 or something, it was probably true, but by the time I showed up, we didn't call them micro services, but they absolutely were. And I would say again that the micro services at Google were the best , probably the only good reason to adopt micro services is going back to management. It's difficult to get more than 15 or 20 engineers to work on anything efficiently in a single code base that's deployed as a single unit; it's just difficult to do that from earliest engineering standpoint. So micro services serve a purpose from a software development management standpoint, where you can create a unit of deployment that micro services at Google were much more about horizontal scaling. And that was a necessary thing. They add throughput that required that kind of horizontal scaling, but they definitely had. I remember when he turned dapper on, in production, we'd never really been able to visualize it before, but a cache, miss and Google web search. Certainly what two quests GWS, which you're referring to web server at the top of the stack, but by the time it got down to the bottom of the stack between the front end load balancers, the final thing that actually would look through some index on disc, it was 10 or 20 levels of depth to get down, so yeah, it was definitely quite distributed and also huge fan out. Oftentimes a parent would have with paralyzed request to 30 or 50 or a hundred and in some cases, children that had different parts of the index. And so you had a tail latency, things were really scary and stuff like that. So yeah it was quite distributed, especially on the web search side, early on. There were other parts of the system, like the ad system was the front end of that system that merchants would actually use was basically like a database than a Java web server. So there wasn't everything, but that was for the high throughput, low latency stuff. It was pretty distributed from early on. Utsav Shah: Interesting. And just out of curiosity, did Google prioritize the consistency or availability ? Because of that large fan out, I'm assuming availability and it just dropped a data coming from like a few shards that they were too slow or something, but yeah.

Ben:

Yeah. I don't think there's one answer to that, but Jeff Dean did a talk that the slides are online at Berkeley and like 2010 or 2012, it was really good talk where he discussed a lot of the techniques that they would use depending on the situation to deal with, tell and see. And I understand you're referring to the cap there but another trade-off that I think we had to wrestle with a lot was basically just cost or efficiency versus latency. [10:00] And we would often end up with something that was more expensive in order to put us a tighter bound on latencies. So if you had three copies of some service, you'd send the request to two of them in parallel and just take the first one that came back in order to manage high latency, outliers and things like that. But I don't think there's a single answer from a availability to consistency standpoint. It really depends on I guess, the business requirements. Utsav Shah: Yeah. I've seen the tail at scale stuff, setting that might be what you're referring to. That's interesting and, you turned on dapper in 2005 is what you said and what was the immediate engineering impacts from engineers at Google where your customers. So, what was their reaction and did you see like some immediate changes based on releasing it and showing it to people?

Ben:

That's a great question. And one of the most interesting things about dapper is that when we first got it out there in the world, well at Google, it was definitely not something where everyone's like, oh my God, this is incredible, and telling your office mates about it; it was nothing like that. In fact, I would basically go and find a tech lead for, you name it like Gmail, web search and anything that was operating at scale had a lot of services and I would kind of beg them to like meet me and then I would show up in their office, the UI admittedly was terrible, but it was still good enough to be useful. And I would show them some traces and they would always be like, wow, this is actually really interesting, I didn't know this. And would often, explore it with me and we'd find something that was troublesome and novel to them. So know they would get something that was interesting to them. And sometimes they would go in and fix that issue, but it wasn't like we had our own dashboards to track activity and it really didn't get a lot of use. I did generate a lot of value in the sense that we're able to find some, highlight the outliers and understand where that latency was coming from and make some substantial optimizations. But it was very much a special purpose tool used by experts doing performance analysis in the study state. 

That was really what it was primarily used for initially and there are some technical reasons for why that was the case. But if you were to think of it from a product standpoint, the issue is that we weren't integrated into the tools that people were already using. And that is still the number one problem with the sophisticated side of the observability spectrum is that the insights that are generated are genuinely useful and insightful. And even self-explanatory when you put them in front of someone, but they simply are not going to find them themselves unless it's integrated into the tools that they're already using. And it's still, I think the number one barrier to value and observability is just that it's not integrated into the kind of daily habit tools, whatever those may be. At some point we did make a change, Josh McDonald, who actually still works with me at LightStep, who was working at dapper in 2005 as well. He eventually made a change to stubby, which is the internal name for GRPC essentially anyway. And particularly this library called request C, which was used to look at active requests that are going through the process to basically just cordon off the request that had a dapper trace, that set to true. And so you could go to any process where people are already using this request, see thing all the time to see requests going through their service. And it kept a cache of slow requests from an hour or whatever at different latencies. And we had a little table of requests that had the upper traces where you could click on the link and go directly to the trace. And then it was something people are already using and the number of people that used, dapper I don't remember exactly, but it must've been like a 20 X improvement when we released that, and it was a huge change. And the only lesson dapper didn't get any more, it didn't get any more powerful when we did that. It just got a lot easier to access. And so it's all about being in the context of the workflow. That's something where some people it's kind of Jonathan similar, who incredibly smart person, much smarter than I am that's for sure. But he ended up really pressing us to build kind of a bulk data API to run Map Reduce and things like that over the dapper data. And he was in charge of something called Terra Google, which was actually the largest part of Google's index, but also the least frequently accessed. It's a very complicated system, the way that it worked, I won't go into it just because we don't have time. I don't know if I'm allowed to talk about it, but suffice it to say it was really complicated. And he did some fascinating work to understand the critical path of the system using both , it's some really substantial improvements as a result of it. So there are people like that who made these big improvements, but it's a big difference between having, delivering quote unquote business value to Google, usually in the form of latency or reductions and having a lot of daily activity, but daily activity really didn't come until we integrated into these everyday tools. [15:00] And I think that was one of the most important lessons from, the dapper stuff is that the cool technology really is not enough to get retention from engineers who are busy doing other things. Utsav Shah: That's super interesting. And I think I've heard the term Tara Google maybe five years ago when I interned there. And I think I finally learned what it meant. I'm sure I forgot about it in like three months. That's, interesting and request, see it seems like a front end towards like visualizing a context or Google's context in a sense, is that like an accurate way of phrasing it? And why did engineers user requests? That's something I'm curious about now?

Ben:

Well, for different things, but what was particularly nice, but request C also known as well, RPC Z container put requests C but was the part that we're really talking about, what it allowed you to do, I guess it was basically just a table. That's all that you saw the table would have a row for every RPC method that you had in your stubby service, your GRPC service and then, so each row is a different method. Okay, fine, that's simple enough and then the columns were basically different latency buckets. So you'd have requests that took less than 10 microseconds, less than a hundred microseconds, less than one millisecond, less than 10, et cetera and it would go all the way up until I don't know, things that took longer than 10 seconds. And you could examine a very detailed kind of micro log of what took place during that request. So you could think of it as just a little snippet of logs that were pertained to that request and only that requests. And then as I was saying, if the thing was that portrays, you could then link off to the distributed version of it and see the full context. The thing that was particularly powerful though, is that it had one special column for requests that were still in flight that he would be taking a really long time. 

So what would happen is you could have a request that was stuck and you were trying to debug it in an alive incident. And you could inspect the logs just for requests that were stuck usually because of , let's say it was often that there is a new tech slot that was under contention restock, waiting on it. You can go and see that exact thing had happened and there was a lot of really pretty clever stuff. They did an implementation to defer any of the evaluation of any of the strings in the logs until someone was looking at them. So you could afford extremely proposed with the logging on this thing and then you only evaluated the logs when an actual human being was sitting there, hit a refresh in the browser, looking at it. So unlike most logging frameworks for that sort of, that's not generally how it works. So there's a lot of pretty complicated reference counting and things like that, but it was also that if you were having an issue, you could figure out that you were blocked on this big people tablet server, and it was this particular UTEC flock that was contended. And that was being contented by these other transactions, which you can look at and figure out where the contention came from. But to be able to pivot like that in real time was pretty powerful and then having that linked into dapper to understand the context was also pretty powerful. I don't know if that makes sense, but it was a tool that was one of those things that I really haven't seen. I've seen, I think open senses and Z pages have some of that functionality, but it doesn't really make sense unless everything is using that little micro logging framework. And I just haven't seen that outside of Google or Google open source, so I still miss that. It was a really useful piece of technology. Utsav Shah:  No, I think that's amazing given this was like so long ago, and then it makes me think about taking a step back, I think maybe five years ago or ten years ago internal tools at Google are probably better than like the development experience externally. Right? You have so much stuff for free. You talk about blaze and you talk about all of these different tools, but things have evolved a lot recently, it seems like there's so many startups coming up with like different things, and even like Datadog and stuff are like fairly mature, now you get a lot of stuff for free from them. Would you say that the development environment externally is probably better than anything that Google can offer in the sense of you get the holistic experience now? Or do you think there's still things as you said, request like some functionality that's just missing because of the lack of consistency and you have to integrate like a million different things in maintaining like this Rube Goldberg set of integrations to get like a similar development experience?

Ben:

Yeah, that's a good question. It's really hard to compare the inside and outside of Google experience and it's not that Google was all better, and a lot of stuff that Google was actually really annoying. And I was just talking to someone about this yesterday, but the trouble with Google was that everything had to operate at Google scale and there's this idea, [20:00] which is totally false in my mind, that things that operate at higher scale are better and they're usually not, there's a natural trade-off between the scale that something can operate at and the feature set. And so a lot of the stuff we had at Google was actually pretty feature poor, compared to what you can use right now in open source. But the only thing that really had going for it is that it scaled incredibly well. The exceptions were mostly areas where having a monitor with almost no inconsistency is in terms of the way things are built it gives you some leverage. And of course there are a lot of examples of that request, to find example of dapper is actually a fine example, too. 

The instrumentation for dapper to get that thing most of the way there was a couple thousand lines of code for all of Google ovens, but whereas, just look at the scope of puppet telemetry or something, to get a sense of like how much effort is going to be required to get that sort of thing to happen in the broader ecosystem, so that they had this lever around consistency. A lot of the tooling at Google was it's not that it was bad, it was very scalable, but it didn't have a lot of the features that we would expect from a tooling outside. I'd also say that in my seven years of Google Workman infrastructure and observability, I never had a designer on staff and I barely had a PM ever and it really showed it's like having worked now with really talented designers, not just who can make UIs look nice, but who really think about design with a capital D and stuff like that. And just a completely different ball game in terms of how discoverable some of the value and the feature set and things like that. So like the Google technology often lacked that sort of Polish and I think there are many different vendors out in the world right now that I have built things that are much easier for an inexperienced user to consume, even if the technology is equivalent, I think the user experience is not. So that's another area where I think what we had at Google and is actually unfortunately a pretty far cry from what you can get now; it's just by buying SAS. Utsav Shah: Yeah. Well, we thought about if people ran infrastructure teams like product, like you're trying to sell each piece of your infrastructure to potential buyers, you would create a better product because you'd have to think about user experience and you have to think about, make customers actually getting value. So, trying to make that happen, it's not the easiest thing in the world, but that makes sense. When you released dapper, did you have any sampling at all or was it like, I'm assuming yes, but wouldn't have just worked before?

Ben:

Yeah. Sampling is an interesting topic. There's a lot of places you can perform sampling in, a tracing system and dapper performed it almost in every one of them but yeah, dapper had actually a pretty aggressive sampling. We started with one for 1024, so that was the base sampling rate in dapper ,and then we realized that even after that cut of one for a thousand centralizing the data. When we initially wrote the data just to local desk where we wrote log files and we deployed a Damon that ran on every host at Google is actually by the way, if you ever want to like jump through some hoops, try to deploy a new piece of software that runs as root on every machine, tell you that was a real nightmare from a process standpoint. But anyway, so the day this thing would sit there, it would scan the log files and basically do a binary search anytime someone was looking for a trace, and so that's the thing that I started with , that was honestly a terrible way to build that system really bad. So eventually we moved to a model where we would try to centralize that data somewhere for all the reasons you might imagined, but it turned out that the network costs and centralizing that data, even after the one for 1000 cut was substantial and the storage costs were also really substantial. So we did another one for 10 on top of the ones with 1000. So we were doing one for 10,000 sampling randomly before we got to the central store that was used for things like now, producers and stuff like that. And it pretty much means you can never use dapper for all sorts of applications. Like for web search was fine because I don't rememb6er the number, but it was order of like a million grids for a second, so fine but for something like people check out where people are actually buying stuff. 

It's of course intrinsically a much lower throughput service, but the transactions are actually more valuable, so you're getting cut both ways ,and we didn't have a dynamic sampling mechanism on that for when I was working there and people could adjust to the sampling rates themselves, but they usually didn't. So the technology that's really not that useful except for the high throughput services where that sampling, wasn't a complete deal breaker. I think with LightStep and with other systems that have been written in the last couple of years, there's a recognition that sampling really serves a couple of purposes. One is to protect the system from itself. So you don't want to have [25:00] an observer effect and actually create latency through tracing with dapper. We had that issue because we wrote a local disc we're basically entering the Colonel at least on disc flush and for hosts that were doing a lot of disk activity. We could actually create latency with high sampling percentages, but there's no need to do that, you can just flush the stuff over the network and especially that was 2005, that works a lot faster. Now, next to a lot faster, now you can actually get the data out of the process without sampling in almost all situations. They're probably an outlier cases here or there, it's not true, but overall there was no issue with flushing all the data out of the process. And then you just need to decide how much you're willing to spend on network and how much you're willing to spend on storage, and that's a whole set of other constraints. The other thing that I have recognized is that long-term storage is quite cheap. The wiring networking costs in terms of a lifetime, that data end up being almost as expensive or in many cases, more expensive than storing it for a year, so if you can find some way to push the storage closer to the application itself, even if it's just in the same physical building or availability zone, that's a pretty big win as well. So a lot of the work that we've done at LightStep is actually trying to take advantage of some of those, you're just trying to be on the right side of those cost curves in terms of where we actually do the high throughput, and then where we do the sampling stuff like that. Utsav Shah: This reminds me of how Monarch is designed. I was just reading up on the paper before this, it's the same where you're trying to flush something that's in a local data center or the local availability zone. And then finally, when you query, you're getting such less data that you can do that once and ask questions of multiple regions. Is that roughly accurate?

Ben:

Yeah, I think there are definitely some similarities in modern. I have to say, if we could go back in time, I would have pushed back harder on some of the requirements that were put on UIs. I don't think we did the wrong thing, given the requirements that were handed down, but the requirement that we depend on, almost nothing except for, physical DM or physical, DM's kind of a misnomer, but the fact that we weren't allowed to take advantage of Google's other infrastructure beyond just the scheduling system and the kernel and things like that, it really limited what we could do. And then when you also pile on some other requirements around performance and availability and kind of forced to store everything in memory, and we did and then the paper goes and talks about the number of tasks, which are basically virtual machines that moderate consumes and the steady state. And I remember correctly, the paper's number is like 250,000 VM steady state and that is just extraordinarily expensive system right there. A VM of course is not the same size as the physical machine, but it's a lie and that's not even counting the VMs that are being used for durable storage and long-term storage of the data. And wherever they're putting that stuff in Google's longer-term storage systems, I mean just a tremendously expensive system and that's not a good thing and I'm not convinced that's the right approach. We've certainly, with some of the work we've been doing lately LightStep, we basically had to write our own time series database from scratch and rather than trying to re-implement what we did with Monarch, I think a lot of the lessons we've learned is that there are ways to do that are far more efficient without really paying a penalty in terms of performance. And, yeah, I remember that we felt like we had no choice, but to do everything in memory, there are some similar systems that Facebook like the grill system, I think also ends up making the same decision at about the same time, maybe it was because flash wasn't quite commodity at that point and so we felt like it was disc or like physical spinning disc or memory. And now of course there's some interesting things, but that was expensive. I don't know if it's a cool system and it's very powerful, but awfully expensive. Utsav Shah: Yeah. So just for listeners, Monarch is a monitoring system. You can see it provides the same end interface too, as like Promethease, it's designed in a very different way internally, including to the user. I think the configuration system is different, but it provides kind of like the same purpose. A design did replace Boardman, which is the original monitoring system, which like engineers had to deploy for themselves, whereas like Monarch was like a SAS service in the sense that you just had to add your metrics and things would work automatically. Is that like a good summary?

Ben:

I think that's exactly right about what Monarch is. The Promethease thing is a little funny though. Promethease is architecturally much more similar to Boardman than Monarch important. Boardman had a lot of issues that Promethease, I think has improved upon that were, self-inflicted like Boardman to actually use Boardman to monitor your system, [30:00] you had to use not one, but five different, totally unique to Google domain, specific languages, all of. Utsav Shah: PSM.

Ben:

Yeah. All of which were totally arcane if you want my honest opinion had like lots of got you's, like for instance, sorry, this is the ramp, but if you wanted to do at arithmetic, which of course is something you'll want to do when you're writing queries, you could use the minus operator. No surprise, but if you had variables that you were subtracting and you didn't separate it by spaces, it allowed hyphens to be a variable names and it would just like silently failed it. Oh, that's not, it would just substitute a zero for that expression and crazy stuff like that. And of course, since it was a handwritten DSL that wasn't particularly well documented or maintained, there really wasn't staffing to improve that there was definitely a period of Google where it was kind of awkward on anytime you ran into a new problem to write some kind of language, some of these languages in the borderline university were pretty small to be fair. But the point I'm making, if they each have their own grammar and their own rules, and most people basically just copy paste in someone else's portal in order to hit their launch criteria. So there wasn't a lot of thought and care being given to writing maintainable code , and it definitely is code. I think if I remember correctly, Gmail's configurations, which, you know, admittedly were generated programmatically, but those borderline configurations for like 50,000 lines of code and it was totally inscrutable. 

So there's a lot of frustration about that kind of stuff. Whereas problem I think is far more sensible, I could critique this for that, but it generally makes sense , I get it, I think that I don't want to sound overly critical if this is not a good or a bad thing, that's just sort of recognizing that every system is designed for a certain set of problems or whatever. But for me, if you did in here at one of the most problematic from the sweat of mine, which is that it wasn't really designed for distributed pretty evaluation, you can kind of do it, but you have to manually share the thing yourself. And that's a very difficult thing to maintain, to do all the rebalancing and things like that ,and I think that the initial effort at Google was actually, it wasn't Promethease, but it was almost like community is let's fix Boardman and building a new system that has the same scaling characteristics, but has one language, not five better language improvements to this or that; a better internal time series or things like that. But it was still basically the same architecture. And my recollection is this guy, Alan Donovan is another person who's lot smarter than I am; really clever person, but he was working on this stuff at the time. And I think his observation was if we're saying that the system board of mine has tons of issues, how could it be the right thing to architect it and have the kind of block diagram be exactly the same, but how each block just be better? Shouldn't , we be thinking about this a little bit more holistically and to really examine the problems that people are having. And I think when we did that, we realized that the number one problem that was causing a lot of the other weird stipulations people are doing with the fact they had to manually shard and balance this thing, and that distributed credit evaluation was kind of a hack. So the thing that made modern so interesting and also so difficult was that it really was horizontally scalable and that users did not need to worry about where their data was being balanced is also a multitenant from day one, which was allowed a central team to run it for all of Google, instead of trying to repeat that effort with every team in their own little cluster. 

And it made the design much harder, but ultimately I think more robust and I'm not knowledgeable enough about for me, if he has to know how much effort would have to go into making it really do that. I've seen Thanos has added some functionality like this, but I think that the pretty evaluation really pushing that down and making sure that you do as much of the aggregation as you can at the lowest level, and then bring things back up, have a lot of. There's a lot of subtleties that I think we felt like we had to build into the design pretty early. That's the thing that we are really trying to escape from with Boardman was a design that made it difficult to do distributed per evaluation and that's difficult to handle really large datasets that don't fit in a single feed because that's the underlying pain point in Boardman that led to a lot of other pain points. Utsav Shah: That is super interesting. The name Alan Donovan, I think I've seen it with basil get logs. He wrote like Star Luck For Go, and I think I might be in that Google group.

Ben:

So yes, that's right. I think he wrote one of the official Looking Go Programming Language has books. He has some languages background, really nice person, very intelligent guy. But, I credit him with sort of forcing us to step back and really think about what problem you're solving with Monarch. And yeah, that was really fun though, I loved building that system. That was probably my happiest time that Google was the summer that we were prototyping, that it was just like amazing team that went very quickly. It was a lot of fun. Utsav Shah [35:00]: Can you talk more about the district query evaluation? I don't fully get why it's problematic. So let me explain to you and you can tell me where I'm wrong. So what you're saying with Bergman Boardman that query evaluation mostly happened at the higher layers where, I guess if you could just explain to me because I don't fully grasp it. What exactly is the difference in lecture?

Ben:

Yeah. I wasn't being clear, so totally makes sense, so let's take a simple query. You want to understand the ratio of your error rate to your total request rate across your application and you want to group it by RPC method. So let's assume that the amount of data that you have for all of these types of series is large; to put this in context, some of Gmail's metrics were distribution value. So the actual value type was a histogram and a single metric with all the cardinality turned into 250 million times series in the steady state. So very high cardinality surface area that we were trying to aggregate around and the problem that you have that you're trying to do that sort of query that erode ratio query, it's a joint. So you have two different queries, you have a rate query and account query, and then you have to compute, you have to create buckets for each of these RPC methods, just doing a group buy and then within each of those, you have to do a bunch of math. One option is to basically have a credit evaluated at the top of the stack that just talks to all of the sort of like leaf nodes. And in Monarch, we called them leaves each leaf node. And you would say, okay, give me all the data you have for this particular metric , and they would stream the data back to you and you just do that. You do the math, it turns out the data size is large enough that if you do that, you're pretty evaluation times it moves into the tens of seconds or minutes in some cases was kind of a non-starter. So instead, what you'd like to do is say, okay, fine. So we'll compute this at the leaves, but the problem is, and this is the most important point. If you're a grouping by RPC method, there's absolutely no guarantee at all. In fact, it's just not true that all of the data for one RPC method is going to be on one server or another. So each of the services in each of the Monarch leaves is going to have some portion of the data, so what you want to do is compute what we would call a partial aggregation. So everyone confused, they're part of this particular query, so they each make the RPC method buckets, and then they pass those partial results up the stack to the mixer level where now you've done the aggregation so the data size is pretty small over the wire.

And then you complete the aggregation now that you have all the data, get the final numbers for both the error and the account and then at the last step at the top, you join the two things, divide them all and you've finished yourself. So that the most important thing to understand is that it's not possible for the lower level nodes where the horizontal scaling has happened. They cannot compute the final number because they don't have enough data to do it. So they have to have some way of communicating partial results back to the top of the stack, and that example, it's not that difficult, but in terms of the full query plan for an arbitrary query and the language that we are doing, it's a lot of subtlety and complexity to how those different types of praise can and cannot be pushed down to be evaluated at the leaves. And if you ever end up in a situation where you need to pull all of the data up into the mixer level, the whole thing totally falls apart from a performance standpoint. And oftentimes even from a feasibility standpoint, you end up owning that thing if you're not careful. So there's a lot of streaming, evil and pushback on channels, so you don't flood the thing that's getting this huge fan and from all the children and stuff. So this guy, John Benning, who another person much smarter than I am, but he designed that thing and worked on it for years and to kind of optimize it. And yeah, it was a really interesting piece of technology, but it was just quite subtle. And I think if we hadn't designed it for that initially, it's just hard to make the query model that isn't designed to create these like partial aggregates. I think it was hard for me to imagine how you would send that in after the fact, because of the way you approach the computation is you have to be able to kind of truncate, the computation and send it as a partial computation up the stack instead of as a set of query results. And I'm not saying it's impossible, I'm assuming it gets pretty ugly. So that's the thing that I was referring to. Does that make sense? Utsav Shah: Are you saying that the query language itself also needs to be designed with this thing in mind? Or is that mostly just the way you shared out and you make your query plan?

Ben:

We really tried not to put constraints on the query language because of this. There were certain types of joins that were really hard. Like a full relational join, it was deeper than that, this problem, but it wasn't designed to be. And relational [40:00] database and those things are really quite difficult to implement, but the joints are aware. I think we had to kind of draw the line on some of the functionality, but a lot of other stuff, it was kind of a query language has some very powerful capabilities, especially dealing with the time dimension, but also was much more limited than you. It wasn't as powerful as a lot of SQL like languages for doing just general purpose computation, stuff like that. So it was very much designed for time series data, but I would say it seems to have been general enough to represent a wide variety of like operational use cases. I just wouldn't want it, it wouldn't take the place of inquiry or something like that for general purpose computation. 

Utsav Shah: Yeah. And one thing I think listeners should note is that you might think that, users are not running that many queries. How does it matter? But like a lot of people write alerts and like monitors and they're basically super complicated and they have to be evaluated a lot. And I'd imagine that a lot of your load was from these alerts that have to be continuously evaluated to make sure, and it's super critical that they fire quickly. You don't want there to be like a production outage and there's a slow down due to the monitoring system.

Ben:

Certainly the case! Utsav Shah:  So it's a really interesting problem. I think Dropbox tried to build their own and I think they rolled it out successfully and they just put cardinality limits. You can have queries with super high cardinality, but very few other limit, because I think the cardinality exclusion is where a large source of problems is, is that accurate?

Ben:

Yeah, that's definitely accurate. So that's another rant if you don't mind me going in that direction for a minute. This is one of the things I did not realize written on Monarch, but I've come to believe that there really are two types of monitoring data. There was telemetry, I guess you could call it, there is statistical data, which we usually call metrics and there's transactional data, which we usually call traces or sometimes logs or structured events. And those are the two flavors of data and the Achilles heel for the structured events, the traces for the transactional data. 

But the Achilles heel there is that at high throughput just retaining it for a long time. It's just really expensive to so much of it, like you were processing a lot of transactions. It's big data, it's expensive. I don't mean big data in the sense of big data, but it's a lot of data. And then on the metrics side, you can handle the high throughput very naturally because the only thing that changes the value of the counter, it just goes up, but it doesn't actually make the metric data larger; it just changes the numbers if you have high throughput. So the Achilles heel for metrics data is high cardinality. I actually wrote some stuff about this on Twitter a few weeks ago. I'm checking send to you afterwards, if you want to include it in the article, but, the thing that's so frustrating is that cardinality is necessary. You it's totally inevitable that you're going to want to include some tags in order to understand and isolate symptoms of interest, and I think metrics should be used to isolate symptoms of interests that connect the health at some part of your system to the business. at some abstract level, that's really what you're supposed to be doing metrics. And then I think because that's the tool that people use, it just becomes the hammer for every nail that they see. And you just try to use cardinality to address every aspect of observability, which is a complete disaster from a cost standpoint and from a user experience standpoint, and I will try to elaborate on that. So let's go back to this example of RPCs or something like it's totally fair and smart actually, to have a tag on your RPC metrics for the method, because you want to distinguish some reads. Totally fine, because those are different things you might want to independently measure from a health standpoint, but then you might want to say, Traffic spike or latency spikes and I want to understand why. And so if you're using a metric system, the only tool you really have to understand that variance or that change is to do a group buy on some other tag and hope to see that one tag value explains that blip. 

And so you now have a lead to go and follow that tag value where it leaves you, whether it's a host name and I understand the appeal of that, but there's two problems. One like you add a couple more tags and the confident, total explosion is immediate and you're suddenly spending a ton of money, whether it's on from Promethease or a vendor, it's really expensive, and then I think even more pressing issue is that in the code, you only have access to things that are locally available. So you have access to your own host name and things like that, but it's often the case. In fact, I think the numbers I've seen, which I would bring true for me is that 70-75% of incidents in production are caused by an intentional change, like a deployment or a config push elsewhere in the staff that version change or config push is not going to be in your local tags. If you're in service A or if you're in service B [45:00] at the bottom of the stack and service A at the top of the stack pushes a new version, that's flooding you with traffic service A's version is not going to be available for grouping and filtering anyways, so you're paying a lot of money to have all this cardinality and you can't even group by the thing. They have to explain it now in the transaction data and the tracing, you absolutely do have that. The traces flow through both services. These health metrics are linked to the transactions via hosts, by a service names, by method names, all sorts of stuff like that. There are ways to pivot over to the tracing data programmatically in an observability solution, and in the transaction data, the high cardinality is not an issue. You can do an analysis of thousands of traces in real time, and actually understand that the thing that changed is that before, service A above, you was on version five and now it's in version six, and then that explains the difference in the health metric you started with, but using cardinality as a way to do it, sort of ability is a big mistake. Sorry, metric cardinality is the way the observability is a big mistake.

I think that metrics should be used to understand health and nothing more. And then you have to be using observability tooling. That's smart enough to pivot over to the transaction data where it's both cheaper and more effective to understand these sorts of systemic changes that lead to these health changes in the first place. So at Google, I think we were actually way behind where we are now, honestly, with LightStep in terms of how we would pivot from time series data over to transaction data and back again, but that's really the essence of it because you can't do high throughput with one and you can't do high cardinality with the other, so you have to feel that to use tools that pivot from one to the next intrinsically. And I think that's the thing that allows you to kind of get out of that cardinality trap more than anything else, it's not setting a cardinality limit, it's just not needing it to be in your metric data in the first place. I think that's really the solution that we'll find ourselves pursuing over the next couple of years. Utsav Shah: Well, I think that makes total sense. And maybe just super quick to explain the lessons, like why is high cardinality bad? And I think the answer is because like in a time series database, when you have like a data point with a different cardinality, you have to basically store it as like a different drawer or different columns, and that's what causes the explosion.

Ben:

Yeah, that's basically it. And high cardinality isn't bad, high hardly metrics are bad. I think the issue is that in a time series database, you can basically think of it as a huge spreadsheet and each row is a different time series. And it turns out that creating a new row is a lot more expensive than creating a new cell. 

Utsav Shah: So you're just incrementing like a number in that cell?

Ben:

Yeah, or adding another data point to an existing time series as far cheaper than creating a new time series. I think that the issue with high cardinality, it sounds so esoteric, but it ends up being an issue that like your chief financial officers quit and start caring about. Because you can add one piece of code, literally one line of instrumentation that says, I've got this metric, that tracks requests, I'm going to add customer ID and host name that just explodes, and then every single value, every single interaction of that line of code is going to create a new time series and you're probably dealing with, it's not vendors being evil by the way, they have cogs, they have to pay for it. But if you write a piece of code that incurs, 10 million time series in some PSTB somewhere, that's going to be expensive, no matter what he has to be are using. So some are more expensive than others, but it's still just different flavors that are expensive and I think ideally the developer can add that code a platform team can write some kind of control that says, I never want cardinality for any metric to exceed X, and then you can do some kind of top cave thing to retain the high frequency data and aggregate the rest. I think that's the kind of long-term vision I have for how cardinality should work, but really it would be great use event data tracing data for high cardinality where you don't pay a penalty at all. It just literally doesn't matter in terms of the cost of the solutions and then stick to the metrics, high throughput data where you need precise answers about critical symptoms. Utsav Shah: And then the flipside of that is why is cardinality not a problem for tracing? How do you store tracing data that in a way that cardinality is an issue at all?

Ben:

I think it's depends on how you index it. So cardinality it's an issue with the index and the TSB. Utsav Shah: Yes.

Ben:

In the tracing database. I am actually interested, there are many different ways of doing it. Some people have column stores. I think I don't work on any of them, so I don't want to misspeak, but I'm pretty confident that their underlying data database is a column store where, you have different trade-offs in that type of world. But LightStep does is too complicated for me to explain right now, but we have our own way of managing cardinality and the trace data. So it's not that it's free or something, but it's not an issue like it is for a time series database. So I think that different people have addressed it in different ways, [50:00] but I don't feel, it's a very satisfying answer to your question in a time series database, because of locality constraints you have around time series, like adding cardinality just has a fixed cost that's relatively high. I think that's probably the best way I can explain it. Utsav Shah: Okay. Is the index and electricity data, just the trace ID or the hash basically, or a service, which is relatively straightforward index?

Ben:

There's a lot of different ways to do it. That's why I'm kind of hedging on this because I'm trying to be precise, I like being precise and it's actually quite a diversity of ways that it's done right now. So I can't specify how it's done everywhere. In the dapper ecosystem, we did have a few indices that we special case, and then if you want anything that, wasn't one of those indices, you had to write a MapReduce, which is super high latency. But other systems have no limits and cardinality others as well, index up to end values of every key and no more. I mean, there are a lot of different approaches to it. Utsav Shah: That makes sense. And I can read up more on how Jaeger works, but when you search for a trace ID in, you get that fast, everything else is kind of not that fast, so that makes sense to me, and I think there was a lot of good information. I feel I've just learned a lot on how all of this infrastructure works for sure. Do you have anything that you want to add on top of this? Just what you've learned, all of this stuff, you left Google in 2012, what was like the one thing that was just really something that you still use or some information that you still remember from that time that is one design principle perhaps, or just one way of thinking about things?

Ben: 

That's a good question. I kind of referred to it, but not as precisely, but one thing that Jeff Dean said a few times, which really did stick with me, was just this idea that you really can't design a system that's appropriate for more than like three or four orders of magnitude of scale appropriate being the keyword. And this goes back to this idea, that system is Google. We're not better because they're more scalable, and one thing I liked, it's not about Google, I think most companies are like this, but the in-house technology at Google came with pretty accurate advertising for what it was good for and what it wasn't good for. And there's no shame in saying, yeah, this database is good until you hit the scale and this one's terrible if you go below that, because it doesn't have all these features that you'd expect or what have you, and my experience outside of Google, or whether it's open source software or vendor software, it's just that people are understandably reluctant to describe the scale that their system is appropriate for. And I mean, it's a great question to ask, actually, if you're talking to someone about something that they're really excited about and they're trying to pitch you on it, just say, so tell me like, what's too much skill for this? and what's too little skill for it? And if people can't come up with an actual answer to that, I think that's a bit of a red flag in my book. And that was something that really stuck with me after Google and I think it applies to any technology that you're building, and also it's good to be humble about that too. You can remind yourself, you built some really scalable thing, it probably doesn't do something that less scalable thing could do for a site and just to try and think about fitness for purpose, that's maybe the thing I feel that's comes up over and over and over again, anything from engineering ,to product ,almost to marketing. To think about what market segment is this really appropriate for? What, where, would the scale that we're targeting live in the marketplace? So I think it's relevant at that level too. Utsav Shah: Yeah. It reminds me of this meme, MongoDB is like web scale.

Ben:

What does that mean? No comment.Utsav Shah:  The web is so different for so many different things, there is no concept of like web scale, it just sounds fancy. My current company uses MongoDB, it seems to work so far, it's probably fine. And I have so many more questions in terms of like, I want to ask you about open tele metrics and open tracing and all of these things and lifestyle. But I think it would be nice if you do that in a follow-up search. This was great, and I feel like I learned a lot, so thank you so much for being a guest.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Liran Haimovitch is the Co-Founder and CTO of Rookout, a new style debugging tool that enables developers to debug web applications by adding debugger style breakpoints in production (without actually stopping the application).

Apple Podcasts | Spotify | Google Podcasts

Rookout belongs to a new class of developer tools that aim to make application debugging more interactive than the standard “inspect logs” experience that is standard industry practice today. I’d encourage checking out the demo to understand the tool better.

We discuss the state of developer tools today, debuggability, observability, and “understandability” of code, the technical implementation of Rookout, the engineering workflows around debugging tools, the sales process for Rookout, and more.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Emma Tang was the Engineering Manager of the Data Infrastructure team at Stripe. She was also a Lead Software Engineer at Aggregate Knowledge, where she worked on the data platform.

We explore the technological and organizational challenges of maintaining big data platforms. We discuss when a company would require a “Big Data” system, what the properties of a good system look like, how some of these systems look like today, some of the tools/frameworks that work well, hiring the right engineers, and unsolved problems in the field.

Apple Podcasts | Spotify | Google Podcasts

Highlights

0:30 - “Big Data” for software engineers - when does a company need a big data solution

2:30 - The transition from when a company uses a regular database to a big data solution, with a motivating example of Stripe

4:20 - Verification of processed output. Some of the tools involved: Amazon S3, Parquet, Kafka, and MongoDB.

9:00 - The cost of ensuring correctness in the data processing. Using tools like Protobuf to enforce schemas

13:30 - Data Governance as a trend

16:30 - Why should a company have a data platform organization?

21:30 - Hiring for data infrastructure/platform engineers

24:00 - How does a data organization maintain quality? What metrics do they look at?

28:30 - Trends of some problem areas in this space.

33:30 - Emma’s interest in data infrastructure, and advice for those looking to get into the field.

Transcript

Utsav Shah: [00:13] Hey, everyone, welcome to another episode of the Software at Scale podcast. Joining me here today is Emma Tang, who is an ex-Engineering Manager at Stripe on the Data Infrastructure team of the Data Infrastructure group. And she used to be a Lead Software Engineer at Neustar, which is an AdTech company. Thanks for being a guest today. 

Emma Tang: [00:30] Thanks for having me, Utsav.

Utsav Shah: [00:32] Yeah. Could you tell the listeners and me, because a lot of us are mostly just experienced with full-stack software development, we do front end, we do back end, where does data infrastructure come in, or data in general? When would I use something like Spark? And when does that become business-critical?

Emma Tang: [00:51] That's a really good question and I think that's a question on a lot of folks’ minds. It's like, “When does my data become too big that I need to use big data?” Or “Should I consider it right now, given the set of circumstances that I have?” I personally don't think there's a hard and fast rule in terms of volume or size that defines big data. You can't say, “Oh, I have one petabyte of data, therefore, I need to start using Spark.” I think that actually, it's really depending on your situation, like your architecture and your business needs. So, some symptoms of when you need to consider big data include, oh, you're actually hitting your production database for analytics and you're really worried about the load on your production application. That's a time where you need to consider maybe something that's more specialized for dealing with big data. Another example is, let's say you integrate with a bunch of third-party vendors like Salesforce, and Zendesk, and Clearbit, and you want to integrate the data into your system and yield insights from that, you should also consider big data when you're at that point. And then obviously, another case is, you just literally have a business need. For example, you need to produce data-heavy reports for customers or regulators, or other internal purposes, you also should consider spinning up big data custom for that as well.

Utsav Shah: [02:10] Okay, so maybe, walking through this as somebody who has no experience in this field at all so I'm just guessing here, maybe like V1 of how Stripe would be implemented is just a relational database that has transactions, it's like a ledger. But at some point, you want to do a lot of processing on those transactions, like what the tax rate is for a certain transaction. And there's also really complicated transactions like you're paying maybe an Instacart shopper, and then you're also paying Instacart something and that money is coming from some random account somewhere. At that point, you would start considering managing those transactions using something like Spark? Is that a good way to think about it?

Emma Tang: [02:51] Yeah, you're definitely going at it the right angle here. So, at Stripe, for example, some of our earliest use cases for big data was actually in kind of the reporting and reconciliation side. So basically, because we're a FinTech company, we deal with a lot of money. And when money is involved, accuracy is super important. So, we can't be like, “A rough estimate of what happened yesterday.” No, we have to know exactly what happened yesterday, and not only yesterday, for all the days before that as well. If a transaction was somehow reversed a few days later, we need to know about that, and we need to reconcile the systems. So big data is great for that. You can very easily take in the whole universe of all the data you've ever had in your company, and just crunch it in a few minutes. So that's the power that you get. At Stripe, some of our earliest use cases, including our internal billing system, so at the end of the month, we need to charge our customers a fee and say, “Oh, you made these many transactions on Stripe, we need to charge you this much money.” That process is actually a big data process at Stripe, it's literally how we make money. And big data is at the epicenter of that. 

Some other examples are just product examples. So, Stripe has things called Radar fraud detection, or Capital, which is our lending product, and Sigma analytics products. All these products are powered by big data. Without machine learning without big data, none of these are possible.

Utsav Shah: [04:13] Okay, that makes a lot of sense. So, you basically have something like a crone job or something, just bear with me here, which will go through like one day's worth of transactions and figure out, “Okay, these are the final fees that I need to charge people, or charge our customers based on whatever transactions they made,” and that will finally go through.

Emma Tang: [04:33] Yep, sort of like that. And we have to combine different data sources together as well. It's not just one transaction source, you also have to combine it with their custom fee plan, because they might have a very different fee plan from other merchants. And you have to combine maybe currency codes, maybe they operate in different countries, and there's different currency codes and different currency exchanges and all that. So it’s a lot more complicated than I’m making it out to be, but that's where big data shines. You can really do very complex stuff on really [05:00] big data and everything, the system is built for that kind of processing.

Utsav Shah: [05:05] Okay. How does one verify that what the number-crunching did is actually correct when you have such a large pipeline?

Emma Tang: [05:15] That's such a good question, and one of the biggest pain points in big data honestly. Again, I think it really varies by business need. If it's a B2C company that's like Facebook or social media, where you want to know how many clicks happen on this post or whatever, it's not super mission-critical that it's super, super accurate. And therefore, the way you implement that system might be a little bit different. You don't need at least once or only once like semantics. Whereas at Stripe, we really do care about accuracy. So, for us, we actually build in a lot of custom validation frameworks on top of the jobs themselves. So once a job finishes running, we actually have a bunch of separate post hoc processes that we run after the job runs to validate the data, that it is the right format, that it follows some rules, for example, it has to be always growing, or “the max may not exceed this”, or unique key validations. So, we have all of these that run after each job to automatically validate this. And if it doesn't satisfy these, we immediately invalidate the data and stop the entire pipeline from running. So, this is something that we built bespoke at Stripe, but I honestly feel, and we'll talk more about future trends and stuff like that, but I honestly feel that this is one of the things that the industry needs to do better, which is easy ways to build validations on top of big data pipelines.

Utsav Shah: [06:40] That's really interesting. It sounds like it's the inverse of machine learning where you're trying to make sure your training data is clean before it goes into the pipeline. Here, you need to kind of figure out your output data is clean and so you run a bunch of verifiers, and you need to--

Emma Tang: [06:56] Yeah, exactly.

Utsav Shah: [06:56] Okay, so that makes sense. So maybe a system design for something like Stripe would be, you have an API, which pretty much writes information to some kind of big data warehouse like Hive or something like that, and then you have Spark jobs that crunch all of that information that go into Hive and produce this final output, which is like reports and billing, and then you have a bunch of verifies that check other-- 

Emma Tang: [07:21] Yeah.

Utsav Shah: [07:22] Okay, that's really interesting.

Emma Tang: [07:23] I can describe the system at Stripe for you. It's a little bit different from other companies because, again, big data is so core to our business at Stripe, therefore, it does look a little bit different for us. So, there are a couple of different sources of data for us to be used in big data. One is our online database. So, at Stripe, we use a combination of different types of databases, we use some relational, but also some NoSQL databases. So, at Stripe, we have a really huge Mongo presence. And there are actually ETL pipelines that take the data from Mongo, and basically apply yesterday's changes to yesterday’s snapshot to create today's new snapshot. That snapshot will land in S3 for downstreams to use. In that process, we also apply a schema, and we do validations on top of that Mongo data, so the data is actually in Parquet format when it lands in S3. So, another source of data for us is from our Kafka Streams. So, these are more immutable event-based streams but also end up being archived in S3. And we also transform that to a Parquet format. And then there's other smaller kinds of data streams that flow in mostly from third-party integrations, like I mentioned, Salesforce, etc., those also land in S3. So as Stripe, our data lake is S3, the source of truth is S3. However, we do have data warehouses on top of that, but they are not considered the source of truth, even though they have almost all the stuff that S3 contains. 

So at Stripe, we use a combination of Redshift and Presto, but we're in the process of deprecating Redshift. So, I think we're very close. So that basically sits on top of the data, and it gets ingested in every day through another pipeline as well. In terms of batch distributed computation, we use Spark. We use just mostly DataFrames, and we also have quite a bit of Spark SQL. We used to use something called Scalding, which is something based on MapReduce, which is an older technology, but we've mostly rubbed off that at this point.

Utsav Shah: [09:29] Okay. And the benefits of Spark are you get basically a streaming infrastructure rather than a map job, like a map phase and a reduce phase. And each phase taking a long… especially the reduce phase, could potentially take a lot of time to complete.

Emma Tang: [09:42] Yeah, so we actually don't use Spark for streaming purposes. So, Spark actually still has analogous phases. There's mapper phases, but there's shuffles where it's kind of like a reduce phase. So, I think the basic concept is very similar, but I think the main advantage [10:00] to do MapReduce is it's a lot of in-memory processing, which is a lot faster than having to [Inaudible 10:06] onto disk over and over again. So that's why it's a lot faster. It's also just because the people working on it, there's so much attention on it, the optimization they're doing is really, really great, they have a really strong optimization engine. And if you write data frames or SQL, they can actually optimize your query very well, to the point where it will outperform raw Spark, without using DataFrames by like 10x, 20x. It’s quite amazing. So, there's a lot of interesting things that are happening on the Spark SQL side. 

Utsav Shah: [10:39] Yeah, that definitely sounds really interesting. And it seems like you have a lot of different data sources that put stuff into S3, and then that gets processed by the Spark jobs, and you have this business-critical output. And it's a very different shape of a company compared to your regular tech companies that don't have to deal with… or a lot of tech companies don't have to deal with large scale data consistency. As you said, a lot of it is analytics so it's okay if it's slightly wrong. That's fascinating.

Emma Tang: [11:10] Yeah, it's fascinating but it's also very challenging in different ways, I would say. We care a lot more about accuracy, so our pipelines tend to be really robust, but also slightly more computationally intensive. Like for the same job, we’re probably spending three times the computation power than a normal company would to make sure everything's great, accurate,

Utsav Shah: [11:31] Okay. So just in the additional verification, and all of that is where that additional time goes through.

Emma Tang: [11:36] Yeah, there's a lot of that. And also, sometimes we take the more robust route. For example, you can get an estimate by just crunching three days of numbers, but we would go back in history and crunch all 10 years of data.

Utsav Shah: [11:48] Oh, interesting. And it's only going to grow with time. 

Emma Tang: [11:51] It’s only going to grow.

Utsav Shah: [11:51 And even transaction volume is only going to grow.

Emma Tang: [11:53] Exactly. So, there's actually good ways to work around that issue as well but we can go into that another time, as long as-- There's a lot of stuff here. And there's some things I didn't even touch upon that's also in the sphere of big data. For example, scheduling is also a major issue. You have pipelines, you want to make sure that this job runs after job A and job B runs after job D. That kind of what's called a directed acyclic graph is something that is very important to use in making data pipelines more robust. At Stripe, we use Airflow, but there's also other schedulers out there as well. 

Utsav Shah: [12:32] Okay. Yeah, I'm familiar with Airflow, where, as you said, it seems like it allows you to make these DAGs off different jobs so you make sure that the entire pipeline goes through. But what happens then when somebody adds an intermediate job that starts failing? Does the entire company get paged? How do you maintain reliability?

Emma Tang: [12:53] You ask all the best questions. 

Utsav Shah: [12:55] Thank you.

Emma Tang: [12:56] Unfortunately, sometimes that does happen. So different people deal with it differently. At Stripe, we have basically different types of rules for each job to specify what the failure behavior is. So, there are rules like “Stop the entire pipeline”, that's usually the default. But there's also rules like, “If this job fails, and the next job needs to run, the next job is allowed to use yesterday's data.” Or “If this job fails, there's some other default job that you can default to. That job will run instead, and then the downstreams can run.” So, we have a lot of complicated logic around this, and unfortunately, it just really depends on the situation.

Utsav Shah: [13:37] Yeah, that makes sense. So, I guess one example I can think of is if you're trying to detect whether there was a lot of fraud or not, you don't have to use the previous day's data. I don't know, I could be wrong here on what the business criticality of that is in case that job is failing temporarily. But definitely calculating how much you need to bill your customers; the entire pipeline has to stop.

Emma Tang: [13:58] Yeah. So, unfortunately, when that happens, people do get paged. All of us get paged.

Utsav Shah: [14:04] Yeah. So how do you maintain data quality? This is a very persistent problem that I've seen where sometimes data might just not be-- For example, country codes, but there might be an invalid country code over there in that, and then some assumptions that a downstream job makes on valid country codes could just break. How do you make sure that data isn't getting dirty because of some intermediate job? I don't know if that makes sense.

Emma Tang: [14:32] Yeah, no, that does make sense. So, like we talked a little bit before, I think validations are a really important part of it. So, we do spend a lot of again, computational resources and ultimately money on making sure that the data is really correct. Some architectural decisions that we made to make sure things are good is, for the most part, at Stripe, in terms of Spark jobs, it's almost all purely Scala, so it's type safe. [15:00] We also make sure that all data that's important also has a schema associated with it. So, all data is written in Parquet but we also have a schema representation for every data set. So, we use a combination of Thrift and Proto. We basically, used to only use Thrift; we’re moving on to Proto as a schema representation. And they need to match one to one with the data you're reading. So, if you have a mismatched data schema anywhere along the way, it won't work. So, we rather things fail than be false is our approach. It's an extreme way to do things, again, but that's just the world that we operate in.

Utsav Shah: [15:39] I think that's fascinating. So, every single schema is basically specified via a Protobuf message. 

Emma Tang: [15:45] Yeah.

Utsav Shah: [15:46] Or at least the really critical ones. 

Emma Tang: [15:48] Yes, I would say almost everything. So, everything that I mentioned that comes through Mongo, or comes through Kafka, all datasets coming in that way will automatically have a schema representation, either in Thrift or Proto. And we have some libraries that autogen Scala libraries for each model, basically, which users can then use in their type safe Spark jobs. So that's how we ensure there's consistency there. 

Utsav Shah: [16:15] Okay, that makes sense. So being the Engineering Manager for pretty much one of the most, it seems like business-critical teams, the company must be pretty stressful.

Emma Tang: [16:27] It can get stressful, that's true. Like you mentioned, when jobs fail, it can fail any time of the day, which can be stressful when you're firefighting. I would also say, increasingly, and this touches upon another subject that we might talk about later, which is future and stuff like that, a trend that I'm seeing a lot is increased data governance, data lineage, data security type work. There's just more regulation these days about how data is stored, how data is moved around, what kind of data you can keep around. So that also adds a lot of pressure to data teams as well. Not only do you have to do your job of making sure the infrastructures are running, that you're actually fulfilling the business needs, but you also have these security and privacy regulatory things that you have to start implementing solutions around as well. So, I think there is, in the last few years, a little bit of a double whammy of pressure on data teams. So, I definitely think it’s a ripe solution for some startup to come in and help solve.

Utsav Shah: [17:34] So maybe to dig into that a bit, is an example of that something like European data has to be processed in a European data center? Is that an example of that?

Emma Tang: [17:44] Yeah, an example is, if a country comes out and says, “Oh, we're going to implement data locality rules. All payment data needs to live in a data center in my country.” And then you might have to spend a lot of time actually working with a regulator to figure out what does that actually mean? Does it mean just storage? Does it mean storage on online databases but also S3? Does it mean compute, like data in fly? And if it is compute, is there a time limit? Can you keep it around for 24 hours, and then ship it over by then? There's a lot of different kinds of possible specifications here. So that's an example of a rule. Another thing is GDPR, you have to encrypt different things in ways. So that's another example.

Utsav Shah: [18:33] Is there a particular country that has the most annoying rules? I don't know if you want to share. Or one particular rule that you had to deal with?

Emma Tang: [18:47] I think honestly, these are great things for the consumer and for businesses too. These are things that we should have in the world. They do add technical challenges to the teams that have to sell for them. An example of one that was a little tricky to solve was the India data locality rule that’s coming out, specifically for payment data. And yeah, basically, if you already have a company and architecture that has scaled to a certain point, then migrating away from that architecture to make data locality work can be a bit challenging. But we still figured out a really great solution, so I'm very happy about that. 

Utsav Shah: [19:28] And you can't even fall back to the US when something like India because it’s literally halfway around the world, so the latency must be just really high.

Emma Tang: [19:38] Yeah, so luckily, we're all AWS so latency is not as bad as you think it is, even though, yes, there is some bit of that latency.

Utsav Shah: [19:50] Okay. Yeah, I just remember trying to SSH into my work production servers when I was visiting back. It was like, “I can't work here at all, there’s no point. [20:00] It's just too slow.”

Emma Tang: [20:03] Yeah, it could be like that. We did a lot of studies before we even attempted with any solution, to compare the latency characteristics. And in the end, it was actually much better than we assumed so, it was great.

Utsav Shah: [20:13] Yeah, that is interesting. So how did the work of your team evolve over time? Because I'm guessing in the beginning, there were teams just writing Spark jobs, and at some point, somebody decided, “We need to really invest in this space and make sure there's a [platform-ish 20:33] layer.’ So, I'm guessing that's where your team is. How did that work evolve over time?

Emma Tang: [20:37] Yeah, no, that's a good question. So, I'm going to step back a bit and talk a little bit about why would someone want to have a data org. Is it okay to not have a data org at all, and just figure out whatever solution that works for the business need? And I think the analogy for me is having planes around when most of the transportation is our trains. You can usually get to places on trains, and it's relatively okay, but there are some places that you cannot get to, like, from North America to Europe, you can't get there just on a train, you have to have a plane. But if it's a really short distance, if you're going from San Jose to San Francisco, maybe a plane is a little bit too extra, probably a train is okay. That's an analogy, I think. Big data, it's going to solve problems that your current architecture just cannot solve. It's just not possible. So, you're just losing out on capabilities and possible business products and scalability if you don't even consider it. You just will be kind of left out in the wind, in the dust if that happens. 

So the question is, when do you want to consider doing it? And like I mentioned before, when you've noticed that there are make or break situations in terms of business needs, the products that you want to make, you want to run machine learning, for example, you definitely to set up big data in order to even do that. Let's say you have a real-time business need for processing some large volume data, you probably need to set up Kafka. If your production database is falling over because you're running queries on it, you should probably set up a data warehouse like a Redshift, or a Snowflake. So, these are the times when you need to consider it. But I would really recommend not waiting until the breaking point. Investigate earlier, see if you can integrate it into it smoothly so that your architecture will just work well with whatever big data systems that you introduce. 

Okay, now, sorry, I'm going to go back to your original question, which is how has work evolved on big data at Stripe? So, I think I'll just start out by saying there is a couple of different philosophies about how data infrastructure groups should be organized. And I think, especially on the point of where do you put data infrastructure engineers and data engineers? And to clarify, in terms of terminology, I think the phrase data engineer is actually interpreted quite differently at different companies. My personal definition is folks who are setting up the Hadoop clusters or Spark or basically writing these autogen for schema data models, I think these folks, I call them data infrastructure engineers. And the folks who are using the infrastructure to write the jobs and pipelines, I call them data engineers. So, I think that's how I differentiate the two. And a lot of the organizational principles revolve around how do you organize a relationship with these two groups? 

One philosophy is basically have all of them in the same org. So, you have data infrastructure engineers, and then you have data engineers. And data engineers will help production engineers write their pipelines, but they report up to the same folks as the data infrastructure engineers do. Another philosophy is, let's say the data engineers are actually inside the product works, right? They report up to the product works, but they just know data engineering, and they will write the pipelines for the product works. And the third is there's no data engineer. There's data infrastructure engineers, and there's product engineers, but the product engineers have data engineering capabilities, they know how to do these things, they are more generalists. So, I think those are the main three philosophies. 

And Stripe definitely started out as data engineers and data infrastructure are one and the same. When it was small, the people who spun up the infrastructure are also the ones who knew how to write this, so they were doing both jobs. Eventually, when we started growing, it was just very inefficient to keep doing that because they're fundamentally very different skill sets. Understanding production data, understanding the business logic, and how to combine data sources to yield the kind of insights that people want from this is not the same job as making sure the Hadoop cluster doesn't fall over [25:00] from load. So eventually we started separating the two and that's where we were met with the hard decision of should data engineering live within our org, which is kind of how it was in the past, versus trying to split it out outside. And we've experimented with different kinds of ways in the past, but I think we ended up in a hybrid model in the end, where we have an organization under data science, which takes care of the most complicated business models and the most complicated analytics, and they will write the pipelines for the other products, basically. And they report up to data science. 

And then on the other side, most other pipelines, not the top level most complicated data models, like most other pipelines on the product side are written by product engineers. And as data info’s folks like myself and my team, we make sure the tooling at Stripe is some of the best tooling around. If you want to write a data job, you want to schedule a job, it's super easy. Observability into your job should be super easy, testing should be super easy. That's our job to make sure the barrier is really low so that we don't need to hire specialized data engineers to kind of tackle this problem. 

I can talk on and on about this topic but what I think is important to call out is there's a couple of reasons why having a separate data engineering org may not be the best solution for a lot of different companies. One is data engineers are actually really hard to hire. If you think about it, what do data engineers do, in my definition of data engineers? They're the ones translating production logic into Scala or Python or whatever you write your data pipelines in, so they're literally translating things. So, it doesn't really bring a really big sense of satisfaction when you're not producing a lot of things. You're literally a translation layer. And these folks are also highly technical as well. There's no reason why they need to just write pipeline jobs, they can move to other roles as well. So traditionally, they're really hard to hire. And we've observed this at Stripe as well, they're just really hard to hire. 

And the other thing is when a problem has a software solution, versus a hire people solution, you should always try to bias towards the software solution a little bit, because, at some point, your company is going to grow big enough that hiring a million data engineers is just not very tenable. First up, they are hard to hire. And second, it's expensive to hire them. So, if you take the resources that you would, and put them into developing the best software solution possible, I think that's a more long-term sustainable path forward.

Utsav Shah: [27:31] That's so interesting. And I think that certainly makes sense. I don't think I had a good understanding of what a data engineer does before I heard you talk about it. And yeah, how did you even go about evaluating whether somebody is a good data engineer or not, or whether somebody even fits that archetype of what you're looking for?

Emma Tang: [27:50] So I personally don't hire for data engineers, I hire for data infrastructure engineers, which is, again, a very different skill set. I think for data engineers, the folks who write these pipelines, they have to have a curiosity into the business, and the product, to really want to understand why the data should be combined the way they do, be creative about coming out with solutions. I would definitely say they need to be a lot more invested in the business side of things.

Utsav Shah: [28:19] That makes sense. It's like somebody who's interested in how the product works and why people are buying the product. And they want to--

Emma Tang: [28:24] Yeah, working closely with product teams. Yeah, exactly.

Utsav Shah: [28:28] And yeah, you hire for data infra engineers, these seem like more your standard infrastructure engineers interested in building big systems and seeing them work and maintaining them. Is that correct?

Emma Tang: [28:38] Yeah. So, we definitely want to see folks who not only can dig very deep and can successfully operate in very large codebases, because all of the technology we use - Hadoop, Spark, Kafka - they're all very, very complicated codebases, and you need to be very familiar with digging around in that kind of code. I think the other side that we really like seeing is some DevOps experience or at least the potential affinity for the kind of DevOps work that we sometimes have to do. If someone just really hates DevOps, doesn't want to go onto the server to see what the logs are, that’s not the greatest thing. So, it's at once technical depth, but also operational excellence.

Utsav Shah: [29:21] That also seems hard to hire really good people. 

Emma Tang: [29:24] Yeah. 

Utsav Shah: [29:25] But at least the role is clear, I would say, in the industry what this person does. Interesting.

Emma Tang: [29:31] Yeah. Honestly, all the companies are going after the same pool of people. It’s really funny. If you’ve been in this industry long enough, you basically know everybody else who’s in the industry. You just check in with them once in a while, “Hi, are you ready yet?” Check it every six months or so.

Utsav Shah: [29:46] Yeah. So, you're definitely looking for people who commit to Hadoop and all that, for example? Or that would be like an ideal.

Emma Tang: [29:51] Not necessarily Hadoop. 

Utsav Shah: [29:52] Okay.

Emma Tang: [29:53] I would say people with some distributed computation experience is usually helpful. It doesn't have to be Hadoop. You can be [30:00] a Kafka expert, or you can be a Scalding expert or whatever, and still be very successful at a company that does other types of distributed computation.

Utsav Shah: [30:10] Okay. And how do you evaluate whether you're doing a good job and you're helping the business enough? Clearly, when things aren't burning down, that's great. But what kind of quality metrics or like, “Are we shipping enough?” How do you measure yourself?

Emma Tang: [30:25] That's a really good question. So, we have a couple of mechanisms to kind of make sure that we're keeping ourselves honest. We definitely take a look at the runtimes of jobs themselves. We want to make sure that even though data is always growing, we still want to maintain a neutral, if not, lower runtime, over time. That means we keep on making optimizations on the frameworks that we're providing. We keep on providing really useful helper methods so that you can avoid processing more data than you need to. We introduce things like Iceberg data format so there's more optimized reading and writing. So, we're trying really hard. Ultimately, we're trying to make sure that we get to the results as fast as possible, as cheaply as possible. And then the other thing is, are we addressing business needs? We have check-ins with all of the user teams at Stripe, usually, at least every quarter, if not more frequently, where we really listen to what their business needs are and see if there's any ways that we can introduce new systems that help address their really painful, burning need. So, for example, Stripe Capital, they were looking at potentially, Spark ML, as well as data science was interesting. So, this will be a framework that we potentially will introduce to solve the business need. And we will check in periodically to see if what we're doing is fast enough and solves their problems for them. 

I think a lot of other metrics include false failure rates. We want to make sure that those get reduced. We want to count how many data validation errors that we hit. We want to reduce the false positives there, and also just reduce the number of validation errors in general. And, Stripe, in general, is very focused on metrics. So, we have a lot of KPIs. So as an org, we actually have a dashboard of like 10 or 20 different metrics that we track internally to make sure we're doing our job right. So those are kind of the things that we keep ourselves honest with. But again, like you mentioned, there's a lot of different things that are coming in that are not really well captured by metrics, for example, security or data locality. So, it’s definitely a complicated business.

Utsav Shah: [32:38] Yeah, it sounds like it's one of the hardest technical challenges, trying to deal with all of this data and all of these different constraints and making sure you're doing your job right. It sounds like a pretty hard technical challenge. 

Emma Tang: [32:49] Yeah, but that's the fun part. 

Utsav Shah: [32:52] Yeah. The best part about Stripe is it enables so much stuff. I really like the mission of the community which is, I think improving the GDP of the internet, like enabling or increasing the GDP of the internet. I think that's such a great mission statement. Yeah, and there's so many technical details behind that to get all that working. 

Emma Tang: [33:13] Yeah, definitely. Definitely. 

Utsav Shah: [13:15] So then maybe you can explain to me as a layman, why Snowflake is so popular? And what did it solve in the industry that other people were not doing?

Emma Tang: [33:25] Yeah. So, this is something that I personally feel pretty strongly about, which is SQL-ification of everything. If you think about initially what big data looked like, it's like MapReduce jobs. So, you write jobs in Java, they're really verbose, you have to write sometimes your own serialization, deserialization code. It's just extremely complex and you need to know a lot before you can even be productive and write anything that does any transformation. So then comes other kinds of possibilities, like Spark, for example. Spark initially, also, you can only write in JVM languages, so Scala or Java, and you also needed to write a lot, but they had a lot more helper functions. And then they introduced Spark DataFrames, which is a little bit more robust, a little bit more rails on it. They kind of modeled the Python DataFrames kind of thing. And then there's Spark SQL. And out of all of these, guess which one has the most excitement and increased adoption trends? It's the one that's easiest for folks to use. So, lowering the barrier to entry to big data has been a long-term trend. Hive is very popular for this reason as well. You can write SQL on it. And then Snowflake, Redshift, and the technologies like this are just one and the same. You can do massive data computation with SQL using familiar kind of interfaces and familiar tools. So that's why it's super powerful.

I personally have never worked with Snowflake myself so I can't go too deep in here, but my understanding is that [35:00] there's a lot of advantages to the technology itself. It's also compute and storage is also separated, which is a really great thing. I think initially, Redshift, at least now, they also separate, but in the beginning, it was not separate. So, if the hardware is only like that big, and you can't separate compute and storage, there's only so much data you can put on it, and there's only so much data you can use. And that's not the case with Snowflake. There's also locking associated with Redshift. It took us forever, you can't really do your own thing, and it's pretty expensive, whereas Snowflake, you don’t have that issue. So, I am bullish on any technology that comes out with a SQL solution that's really easy to use, very difficult to mess up, and that can crunch large amounts of data. I'm bullish on any type of technology there. So, Snowflake is just an example of that.

Utsav Shah: [35:51] Okay, that makes sense. And it's more about just the ease of use, and maybe the speed or something like that might be [Inaudible 35:56]

Emma Tang: [35:56] Yeah, ease of use, and the possibility of crunching large amounts of data. You can't do the same on a Postgres database. Postgres is also easy to use, but it's like, you don't have the same capacity.

Utsav Shah: [36:07] That makes sense. And so, you mentioned this earlier, but what do you think are some startups that are missing in this space? Something that still needs improvement. You said verification is one thing.

Emma Tang: [36:17] Yeah. So, I'll just start from the top in terms of just general trends, I think not necessarily all of them are startups. Some of them are just literally further improvements of the open-source repos that we have right now. So, in terms of batch distributed computation, I kind of hinted at this, but we're seeing a lot of database optimizations being borrowed, and then transformed for distributed computation. So, a lot of the stuff that we learned a few decades ago that we applied to databases, we're trying to reapply them again to this field. So, for example, columnar format, column stats, table metadata, SQL query optimization, all of these have arrived for big data. Spark SQL and Spark Data Frame have really great optimization engines that's moving in that direction. There's another thing that is kind of taking the big data world by storm, it's table formats. And honestly, what that is, is, before, data is just data, it can be anything really. But table formats kind of bring a little bit more structure to this. So, for example, the one that we use that Stripe is called Apache Iceberg. Basically, you can think of it as a wrapper around Parquet data. It basically has some extra metadata around to describe the data that it actually is referring to. And that is very powerful, that allows for things like schema evolution, or partition evolution, or table versioning, and also optimized performance, because you have data like oh, [Inaudible 37:56] and stuff like that. So, it's super powerful. I think that's where the industry is going. 

I think some other trends that I see are, again, privacy and security. And over here, I think there's just been a lot of demand on data teams too. At every company, we’re facing similar problems, which is how do we make sure that data is tracked, that lineage is clear? How do we make sure that PII data is kept away, stored securely? How do we tokenize different data if we don't need them to be everywhere? A lot of similar problems at all these companies and data catalogs, data lineage products, and data governance products are becoming much more interesting and necessary. LinkedIn had this Datahub data catalog product that were released, and Lyft also released Amundsen. So definitely a lot of companies have released their open-source thing, but there's no one company that's stood out and said, “Oh, we're going to put a stake in this field, and we're going to take over the marketplace.” So, I still think it's a ripe place to explore. 

I personally would really like to see kind of a GUI pipeline scheduler. Airflow is definitely a step in the right direction. It has a very feature-full UI, you can do a lot of stuff on it, and that's one of the reasons it's so popular. But I think it could be one step further. The definitions of these DAGs and the jobs themselves, potentially, at least a simple version of it could be a drag and drop. There's no reason why you definitely need to write Python to write as an operator. So, I would like to see something in that direction. And kind of adjacent to that is data pipeline observability and operability. I think, also Airflow offers a little bit of this, but it's really hard. Like you mentioned, what if a job fails in the middle of the night, who gets paged? Can they be smarter about who's getting paged? If this pipeline fails, can there be an estimate about the final job and [40:00] what the delay and landing time is? Which validation failed? How did the schema change from yesterday to today? All of these things could be better captured with better tooling. Right now, it's very ad hoc. You go in and look at the S3 bucket, and you say, “Oh, the data yesterday was like this.” So, I think there's a lot of potential there as well.

Utsav Shah: [40:18] The way you’re describing it, it's like the 2000s, or the at least 2010s, when there wasn't as many options like Prometheus and Datadog, and all of these things, which is now available in production, but seems like it's still not available for data infrastructure. 

Emma Tang: [40:34] Exactly. That’s exactly the pain point. 

Utsav Shah: [40:37] And just the idea of using drag and drop UIs instead of Python described schema, that makes so much sense. But then you get the advantage of version control, and all of that like with Python. But still, I can totally see, it should be really easy for someone maybe who doesn't understand how to write code at all, they only know how to use SQL, to be able to configure and write a data job. 

Emma Tang: [40:59] Yeah. 

Utsav Shah: [40:59] Is that possible today? Is there someone who only knows how to run SQL or write SQL? Can they write a data job at Stripe today? Or do you think the tools just aren't there yet?

Emma Tang: [41:12] Currently as Stripe, in order to write a proper pipeline, it's not really possible. You can write kind of pseudo pipelines based on Presto, or Redshift, but in terms of all the robust capabilities that we offer, most of it is only through code. We are moving in that direction, though. So, we wrote a bespoke kind of framework within Stripe that allows folks to write pure SQL and that SQL will get translated into a Spark job. So, you don't have to write Scala at all but you need to write a little bit of Python in order to schedule it. So that still is there. So really, the next step is if we can even get rid of that little bit of Python, and have a GUI-based solution, then that could be much more powerful. 

So a lot of companies do have solutions like this, though. Honestly, I don't know how other companies do it but these other companies that use Hive primarily, I think they also write pipelines with Hive, and that's a pure SQL solution. So, I'm pretty sure there isn't a UI solution at other companies. I think Facebook has a solution that's UI-based. Uber also has a solution that is based on Airflow, but they put a custom kind of interface on top that is much more UI-focused. So, everybody's kind of moving in the same direction at these companies. It's just that there is no unified solution yet.

Utsav Shah: [42:40] Basically, make it easier to operate on really large-scale data, and remove as many barriers of entry as possible. 

Emma Tang: [42:47] Mm-hmm.

Utsav Shah: [42:48] Okay. Yeah, taking a step back a bit, what got you interested in this space? Traditionally, at least I hadn't heard of data munging, and all of that even in University and I hadn't heard it until I'd been working for like, a year or two years and I had to mess with Hive and try to understand how our user was doing something. And this was like, internal users. And that's when I got interested in making pretty dashboards and all that at max. What got you interested in working on this space?

Emma Tang: [43:13] That's a really good question. So, I think I just got lucky in my previous job. Before Stripe, I was at a smaller company. We were acquired by a larger company, but we kind of stayed nuclear within the big company. And it was small enough where I got a chance to work on basically everything. So, I started on Salt Stack and then I moved towards the backend, and then towards infrastructure. And our infrastructure team comprises both of just normal infrastructure, like our edge servers, because it's AdTech, and then we also have the data infrastructure segment as well. And I've just found myself really gravitating towards the data infrastructure side of things. I think the fact that it's so powerful was really just really astounding. Once you've seen the magic of a big data job work, there is something magical about it. I think the other thing is distributed computation is fundamentally like-- I don't think. I think it's a second-order kind of software problem. Like you're not dealing with one server and how it's breaking, you have to deal with like 1000 servers and how they work in concert, and how they talk to each other, and how it works. So, I think that it was interesting in terms of the technical challenge side of things, so I just definitely found myself gravitating towards it. 

I think also the timing was a little bit lucky. I started putting a lot more time into Spark around 2015, 2016. That was when Spark became more stable, I think, from 1.3, 1.6, and then 2.0. It started getting massively more adoption as well. So, I think there was a wave of interest and I was riding that wave as well. So, I went to Spark Summits, and you meet all the people that are super excited about this field as well and a lot of like-minded [45:00] folks. So, I kind of got swept up in the storm that happened at that time, and just kept on working in this field. And it's definitely just, I still think intellectually, challenge-wise, it's one of the best places to be. It's just so interesting. It's endlessly interesting. It's moving very fast.

Utsav Shah: [45:19] That's really cool. And what's your suggestion on somebody who's trying to break into the field right now? Like, if they're just in university, let's say. Or maybe they have a few years of experience, but they find these problems really fascinating. How would somebody transition into this? Or how would somebody start working on data infra or become a data engineer?

Emma Tang: [45:36] Yeah. So, I would advise folks like this. Most companies these days do have big data capabilities. Try to get yourself opportunities to work using the data infrastructure, even if you're not working on the data infrastructure itself. Basically, if there's a chance to write a pipeline for your team, and usually there is, your team probably needs metrics, right? Your team probably needs to know some business insights and how they need to operate. So, if an opportunity arises, take that opportunity, and run with it, and talk with the data infrastructure folks at your company, and kind of learn what their day-to-day is, and see if there's anything you can help and contribute with. If you see that, oh, the way of doing this is kind of repetitive and there's a helper function that you can write on the infrastructure side that could help a lot of other people, try to do that. Honestly, writing libraries is a huge part of what data infrastructure does. We just abstract away things so that our users have an easier time too in writing code. So, I think just being exposed to areas is going to be super important. 

What I often see instead is when the opportunity arises to jump on these data challenges that are kind of separate from your work, a lot of folks just shrug away. They're like, “Oh, this is not related to what I do. What I do is Ruby, I do not want to learn Scala. This is totally different.” And I think that's like you’re kind of doing yourself a disservice. If you don't even try it, how do you know? And also, you want to bet on your future. You don't know what's going to be hot in the future, you just know Ruby. It might be good to expose yourself to a few more types of technology. So, I would say just take the opportunities as they come and just go forward with it and try different things.

Utsav Shah: [47:27] Very good. Yeah, you've answered a lot of questions on big data and everything and I don't think I have any more questions for you.

Emma Tang: [47:34] Awesome. Maybe I'll leave with a couple of common pitfalls that I've seen when people spin up their data infrastructure. So, I would say the number one thing is, people trying to do super bespoke solutions like they implement something on their own. They're like, “Oh, I need a scheduler, let me implement something on my own. It's going to be super cool.” It's usually going to be cool, and it will serve your needs in the beginning, but maintaining it and adapting it to your growing business need is going to be a lot harder than if you had an army of open-source people working on the same problem. So, I would say choose popular, widely used frameworks, see how many stars and how many Stack Overflow questions people have answered, just go with the most boring, common solution. That’s usually a good call. 

The other thing is probably, if you can afford it, maybe consider using out-of-the-box solutions on the cloud providers, like use EMR on AWS, instead of spinning up your own Hadoop cluster, that probably isn’t worth it. At some point, you can probably just move it in-house, it’s not that hard. You do want to make sure that you're using a technology that is going to be replaceable. I think TCP data flow is much harder to bring in-house than EMR AWS. But yeah, so definitely do your own research. I think the other thing I would call out, and this is much harder to do, so only do it if you absolutely can. But try to make your data immutable, especially the data writing to big data sources that you're going to use downstream. If necessary, just provide a separate path for your offline data than from your online data. They don't need to be exactly the same. I mean, the data needs to be the same, but the format does not need to be exactly the same. So, try to make it immutable if possible. And last but not least, I would say try to have a single source of truth for your data. You don't want to have a situation where the source of truth is separated in different places, like, “Oh, there are some tables that are most accurate on Redshift. There are some tables that are most accurate in S3. There are some tables that are most accurate in these different archives but not these other archives.” So definitely try to standardize and make sure that there is a very easy-to-use multipurpose data lake instead of trying to federate different sources together in the future. That's just such a pain.

Utsav Shah: [49:49] Yeah, so S3 should be the source of truth and has the most accurate stuff always and everybody in the org should basically be aware of something like that. Is that a good way to summarize that?

Emma Tang: [49:57] Yeah.

Utsav Shah: [49:58] Okay, and [50:00] that brings me to another question which you mentioned was, use AWS EMR or use these tools instead of trying to host it open source, at least until the cost becomes prohibitive. Do you know what the markup is, and at what point you should think about, you know, “I shouldn't use Redshift and I should try hosting my own Presto,” for example? At what point do you think it makes sense to start thinking about that? And how much is the markup generally? I'm sure you all have done cost evaluations and these things. 

Emma Tang: [50:25] Yeah. I don't recall exactly what the markup was. 

Utsav Shah: [50:30] Just like an [Inaudible 50:31], like how much it makes--

Emma Tang: [50:33] It’s definitely less than 10% of your entire spend...

Utsav Shah: [50:35] Oh, interesting.

Emma Tang: [50:36] Yeah, on the cluster. So probably much less than that, but don't quote me on this. We haven't looked at this in a while. But I think it's mostly a matter of, honestly, when people move on to their own clusters, usually, it's more of a business need like they need to install some custom software, that just they cannot do with AWS. Or they need to sync the data into some bizarre place, EMR just does not accommodate that. It's usually those reasons that people move off. And the savings is another justification for moving it off. So, it really does depend on your business need. And on the flip side, it does depend on how much resources you have. If you have the engineers, and you could possibly do this, then by all means evaluate. But a lot of companies, they might not even ever get to that point where they can afford to hire specialized data infrastructure engineers to try and spin this up. So, it really depends on the situation.

Utsav Shah: [51:29] So it's more like a feature gap, rather than a cost-saving measure is why you would consider moving in-house. That's pretty interesting. 

Emma Tang: [51:36] That's what I observed. Yeah. 

Utsav Shah: [51:38] Okay. Like, there's some very weird place, you need to put data that AWS doesn't support. I mean, I'm just trying to think because it sounds so abstract to me, what kind of things would they not support? What kind of integrations would they not support? 

Emma Tang: [51:51] So one thing they don't support is if you want to do Datadog.

Utsav Shah: [51:56] Okay, interesting.

Emma Tang:[Cross talk 51:57]] At least last time we evaluated it. They run on specific versions of Linux, and it just doesn't accommodate our cost of installation of Datadog. So that was one reason we couldn’t do it. There's also security concerns as well. You know, these boxes, they're spinning up and down, they’re processing all of this data, and they're ephemeral. So, there are potential security concerns with that as well.

Utsav Shah: [52:23] Interesting. Yeah, it sounds a little less abstract. So now you were saying something like you can add alerts to whether you're getting enough data on time, and that's why you would put some of your data in Datadog. Okay. That makes sense. Cool.

Emma Tang: [52:35] Yeah. 

Utsav Shah: [52:37 Thank you. I learned a lot of stuff about big data and managing big data in general. 

Emma Tang: [52:43] Yeah, I'm just spilling all of it out. 

Utsav Shah: [52:47] Yeah, and I think it's going to be great to have this recording. And these people are going to be familiar with how much work there is in managing all these infrastructures. 

Emma Tang: [52:57] Yeah, cool. Thanks for having me again.

Utsav Shah: [52:59 Yeah, thank you.

Emma Tang: [53:00] All right, bye.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

John Micco is a Cloud Transformation Architect at VMWare, where he works on CI/CD systems. He’s worked in the CI/CD space for nearly twenty years at various companies like Netflix and Google, he’s authored several research papers, and was a keynote speaker at the International Conference of Software Testing and Verification (ICST). Our collaboration helped shape my work on Athena at Dropbox.

This episode is extremely technical. We talk about the management of CI servers and systems in large companies, common problems faced, and themes in emerging solutions. Large-scale CI/CD management is far from being commoditized due to the custom configuration of every company’s build tooling.

Apple Podcasts | Spotify | Google Podcasts

Highlights

0:00 - What is a “Cloud Transformation Architect”?

6:00 - Sharing knowledge in the CI/CD space

7:00 - A comparison between starting a car company and starting a software company, and why standardization on tools is so much trickier in software

11:30 - The scale of testing systems. Managing a system that handled 10k RPS of test result data. The quadratic growth in compute resources required to manage a testing system

18:44 - “Only 7% of code changes had problems discovered by tests”

23:36 - Is unit testing overrated?

30:00 - At what point can companies no longer afford to run all tests for every code change

35:00 - What is Culprit Finding / Regression ID? What is auto-rollback / test quarantine?

41:00 - Culprit Finding at Dropbox

45:00 - Dealing with flakiness at the test layer vs. the platform layer. An ongoing challenge for VMWare

60:00 - Increase in demand for CI compute when developers don’t have any meetings

65:00 - Bazel migration at VMWare

73:00 - Developing an interest in CI/CD systems. And why it’s an exciting space where there’s a lot of innovation, and why more innovation needs to happen

80:00 - How can someone tell whether they’re investing enough in developer productivity / their CI/CD experience for engineers? The trade-offs that companies should consider when deciding to run tests.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Barak Schoster is the CEO of BridgeCrew, a cloud security platform that was just acquired by Palo Alto Networks. He’s also the maintainer of Checkov, a popular static code analysis tool for infrastructure-as-code.

In this episode, we discuss both aspects - the experience running a DevOps company and a popular open-source tool.

Apple Podcasts | Spotify | Google Podcasts

Highlights

1:40 - The story and history of BridgeCrew.

9:30 - Why should engineers run both Checkov and BridgeCrew checks in their infrastructure? In other words - why is static analysis of infrastructure config files not enough?

15:00 - The BridgeCrew VSCode plugin

17:00 - The community response towards Checkov (it’s grown from 50 checks to over 500 checks in one year)

20:00 - The software design behind Checkov made it easy for the community to contribute. Awareness of good software design principles is important, but also responsiveness to community needs - for example, Barak helped out with a refactoring effort to make additional cloud providers (like GCP) easier to check for

25:00 - Fostering an open-source community to ensure inclusivity

30:00 - Future of security in software organizations - the simplification that’s bound to happen

34:30 - Advice for founders of DevOps companies

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev