Software at Scale

Bharat Mediratta is the first Coinbase Fellow. Previously, he was a Distinguished Engineer at Google, CTO at AltSchool, and CTO at Dropbox.

Apple Podcasts | Spotify | Google Podcasts

We focus this episode on the role of a senior technical individual contributor in a technology company and contrast that with the role of a technology executive (like a CTO) in a public company. We talk about how to explore what the right position is for someone, what drives their success, how to drive impact as a technical leader, and the trade-offs that have to be explored by both senior individual contributors and leaders.

Highlights

01:00 - The story behind switching from a CTO back to an individual contributor.

08:00 - What does a director/VP/CTO’s day look like? How do they drive impact?

14:00 - If a manager has to make effective decisions and has input from individual contributors, why do they need to have a computer science background? Why do tech companies frown on hiring non-technical (MBA) leaders?

21:00 - A manager is certainly thinking about business priorities, and a senior technical individual contributor also has to think about business priorities. But how much? What’s their approach in driving towards business outcomes? How much backing does a technical leader need from management in order to be effective?

28:00 - When should Heads of Engineering think about hiring senior technical individual contributors? At what stage/what shape of problems should they be dealing with?

32:00 - The initial transition from senior engineering at Google to CTO of AltSchool

47:00 - For the next person exploring a move from IC to management: how to go about exploring the move? What should they keep in mind?

Transcript

Bharat [02:00]: Yes, Hey, thanks for having me back. I enjoyed our first podcast, it just reminded me of so much fun that I've had in my career doing great things, especially when I was at Google, and it's nice to be back. And yes, and thanks on the role, it seems maybe like an odd turn, I think, to go from being a Senior Executive Organizational leader at a big public company to being an individual contributor. But in many ways, for me, it felt like a very natural transition, and in fact, I know a lot of other people who have done this. I think to understand, like, what's going on here, you kind of have to wind the clock back a little bit, I spent most of my career doing this kind of arc where I would join a company as an engineer, I would become a technical lead, then I would become an organizational lead of some kind at the line manager. And then I would realize that I just didn't enjoy the things that I was doing on the organizational leadership side. And I would always be trying to steal time to work on the technology, partially, I think, because I was more comfortable with it. But also, I think what was clear is that it gave me a lot of joy and energy.

And I would just then get to a point in these companies where I didn't want to keep going on the trajectory that I was on, but I didn't feel like it was acceptable to go back to being an individual contributor. So when I got to Google in 2004, I also once again joined as an engineer, and I committed myself that I wasn't going to become a manager, I wasn't going to become a team lead. And that lasted actually for about three years but I am not good at ignoring a vacuum or ignoring problems and at Google, there was this amazing opportunity to take on a team that was important to the company and lead it. And so I thought, I'll try it again maybe I've grown up, maybe things have changed and I'd say in 2008 2009, it put me onto like a 10-year trajectory of becoming a Senior Engineering Leader. And there were a lot of parts of it that I was pretty good at, I'm a humanist, I think relationally, I enjoy people, I enjoy getting to know people and understanding, I enjoy that aspect of organizational leadership.

But there are also a large number of challenges to that role that Co-Op as your team gets larger, that take a lot of your time and energy, your primary tool stops being your compiler and your editor, and your primary tool starts being like Google Slides and email and meetings. And over time, I found that as I got better and better at being a technical organizational leader, I just found that I was enjoying my day-to-day less and less, and at first, I thought this is just [05:00] par for the course, this is just what you have to do to have a big impact and I felt like I did have a big impact. But I got to the point where I realized it used to be that I enjoyed my job, I didn't feel like I worked, I was blessed to be getting paid to do something I'd love to do and would do for free. And I kind of shifted more towards a model where I didn't love everything I had to do every day and in many cases, there'll be entire days, but I loved very little about what I did. And it was important and I felt the need and the value in it but I also was feeling myself slowly getting more and more demoralized, or less and less enthusiastic about the mission I was on.

And after I left Dropbox, I did some soul searching; I thought to myself, well, why am I doing this? I'm at a point in my career where I have achieved many of the career goals I set out to achieve and I'm just not enjoying myself as much as I would like to. And why is that? I didn't feel like I had to do something, even though I didn't enjoy it anymore, kind of reached that level where it's okay to not necessarily work every day. And so I decided to myself, I will go see what I enjoy doing when there is no pressure and no structure and no money, but only time. And I started working on things that I was passionate about and I found myself diving into finance and I found myself advising a lot of startups and startup founders, and I found myself learning about financial instruments and learning a lot more about cryptocurrency. Now I've been a cryptocurrency enthusiast, since 2013 or so, when I bought my first Bitcoin on Coinbase. And so, I spend a bunch of time at the beginning of this year learning about DAX, and DeFI and how SciFi could shift to DeFI, and ultimately, I started getting attracted to some of the missions of some of these big cryptocurrency companies. And when Coinbase reached out, the conversation with them was kind of awesome, I chatted with Manish Gupta, the VP of engineering there, and he and I had a great conversation where he wanted me to come in and be an engineering organizational leader. 

That's where his need is, it's one of his significant needs. And we had this great conversation about what his calendar looks like and Manish is a very important, very influential figure Coinbase, in his calendar is wall to wall craziness. And I was like, look, Manish, that's not what I'm looking for right now and he's very smart guys he's like, well, what are you looking for, and when I told him was, I want to show up and learn from some of the best people in the industry who've been investing in this space and I want to understand how Coinbase is strategically increasing economic freedom. And I want to increase my skill set and then I want to find some hard problems at Coinbase and go apply myself there but I want to do it without the day-to-day burden of organizational leadership, the day-to-day burden of a stack of meetings, I want to be able to go back to my roots as a technologist and go find some big problems, and where better to do it than the company that's at the forefront of how these crypto exchanges are working. And one that's so innovative, and so together Manish and I hammered out this role and the role description, which for me feels like something which will bring me joy every day, and will give me time and space to go investigate. 

And it's funny I have been talking with lots of CTOs and SVPs events over the last few months as I head into this role. And every single one of them, when I tell them, this is what I'm doing they all get that look in their face and go like, wow, that would be so nice. They'll be like, so great they like, lay down my burden of what I'm doing I was just talking to a guy yesterday, he just like, wow, and that sounds like so much fun. Because when you're in it, when you're running a big org, it's big, and it's meaningful, but it's also so strategic, it's hard to accomplish anything on a day-to-day basis. It's hard to feel that tactical win, it's hard to be close to the technology, and you’re just getting pulled so much into the corporate strategy, the organizational leadership aspects of it. So this for me is a welcome break and a return to my roots and an opportunity to go learn from a great team and get to apply my skills towards something which I think does have a big meaning in the world. And do it as an engineer again, which is some of the things that give me energy, and who knows where it leads few years down the road. But for today, I'm excited to be there working on this. 

Utsav [09:39]: It sounds like a lot of fun. And the first question that I have is around for ICS like me who've never stepped into a management role. We can understand or empathize like what a line manager does, but what is a director, what is an SVP supposed to do? Why are their calendars full of meetings [10:00] every day? How are they supposed to drive impact were just like meeting people? I know that that's how they drive impact but what does success look like? 

Bharat [10:08]: Yes. Well, I think from the fundamentals, if you look at it, and you say, hey, let's say you've got a big company that spends $100 million a year on product engineering design, let's just simplify, let's just say you're spending $100 million in engineering. You'd want that 100 million dollar to go into the pockets of engineers, and you'd want those engineers to build great things but whatever that's like, 400 engineers, that’s a lot of engineers. And those engineers need guidance, career coaching, they need a little bit of hand-holding, they're going to be at different levels of experience, and they need to be aligned, and they need to be focused. So, they all need line managers, so if every line manager is having reports that are like 12%, or 13%, of your team then automatically goes into management. And then you still have like 50, to 60 managers then and they need managers, so now, again, rule of seven, and then you're still going to have like 20, directors and they need managers and the VPS, and so on. And so at every level, the line manager has to understand how to marshal engineering talent and get them focused on working the right way so that the majority of their work leads to a result. But if you go up a level, and you have a director, whose primary tool is a series of line managers, and the director’s roles to get those managers aligned, you've got these groups that need to work together.

And for the most part, they're all heads down in their area so someone's got to work on that broader alignment. And so now you've got a director who has seven line managers, these line managers send in reports, you've got about, 60 70 people reporting up to this director, and then you have a VP who has five or six of these directors, and each of these directors is now wielding a pretty large amount of work product, roadmap influence, and you've got to get those directors aligned, and the farther you go up, the different the problem becomes. Now, your director is making sure that you're operating efficiently and reasonably aligned, so at Sun Microsystems used to say, tightly aligned, loosely coupled, so that everyone's headed in the same direction, but they're not current always jostling with each other because they're crammed in there. So, that's a different skill set at the director level, it requires you to be very thoughtful about how do you achieve this broader set of goals with this group of people, and you're always dealing with the problems that are bubbling up to you or the problems that your best leads under, you can't handle. And you handle the ones that you can, that you pass the rest up the chain to your VP, now your VP is not like, three 400 people. And the toughest problems there are bubbling up and your VP is now dealing largely with directors. 

And so your VP has a very kind of images that you're like, trying to play chess with boxing gloves on, you don't have the kind of fine-grained control, you have to be much more like, Hey, move this set of pieces over there, move that to the pieces over there, you can't reach in past the director, past the line manager and influence an individual, you have to be thinking and kind of these broad movements, and it just goes up and up enough. So you can kind of imagine how the skill set of a line engineer is very specific in their domain, the skill set of a manager is a completely different skill set around relationally, helping engineers develop and cultivating them to become increasingly better and growing. And then, of course, the skill set of a director is different, it's strategically getting larger organizations to work effectively together to achieve company-level goals. And the director has to sit in this zone where they understand what every line engineer is doing but also understand what the C suite members are saying when they start talking about business goals, someone's got to bridge that gap, and they tend to sit in that middle spot. Then you've got the VPS above who are essentially strictly organizational leaders, it helps that they're very technically if they have a good technical underpinning, but they are around figuring out what organizational ships do I need to make? What key goals do I need to set? What top-level decisions do I need to make to get the team to move effectively? The further up the chain you go, the more your job becomes to be an efficient decision factory.

As a line engineer, your job is to produce code quickly. As you go up, you have to make tougher and tougher decisions and that's a very different skill set. So over time, people, I think get opportunities to move into management to apply some [15:00] of these things, but not everyone loves it because you have to make some tough decisions. And the decisions are generally along the lines of deciding what you're not going to do. It's very easy to make a non-decision and do a little bit of everything but that way leads to mediocrity and failure. The toughest thing for a leader to do is to hold a line and say, we are not going to do this, I know what's attractive, I know it looks lucrative, but it's not what we're doing, we may come back to it later, we are instead going to marshal and focus our energies on this set of things over here, which are small and focused, we can get it done and it will move the needle. So, as you think about people moving up that arc, they have to accept that their primary tool will change, their majority motion will change, it'll stop being you talking to the computer, it'll be you talking to other people. And then their primary set of outcomes and accountability will change too, because it will shift more and more towards what the business needs, versus what the technology can deliver.

Utsav [15:58]: Yes, that makes a lot of sense. And how in-tune do you need to be with technology itself? Like, we often say that there has to be somebody with like a serious background and stuff to be all kinds of like managers, directors, VPS, but they're going further and further away from technology. But why is it still important for them to understand, as the technology or is it important?

Bharat [16:21]: I think it is important, because they say in the army, when you become a general and you get that star put on your lapel, it's the last time people tell you the truth. A friend of mine, the USGS final telling me is similar and leadership in different ways, the more you go up the leadership chain, the more levels there are between you and the engineers in the ground, the more the messages from the engineers in the ground are filtered and interpreted for you. And they kind of have to be because the engineer on the ground doesn't have the necessary broad context. They just see like, hey, this thing is working this thing isn't and they may be complaining, hey, we don't have enough resources to finish this project and this projects important. But you as a leader might know that while that project is important, it's not a business priority, and you're going to defund it and D prioritize it, you're going to make a tough decision. And so you have to have some level of filtering but the problem is sometimes the real signal is thrown out in that filter. So as an engineering leader, it's helpful to know the base level technology and the fundamentals, such that when a message comes up to you, you can give it the sniff test, does that sound right? When people are talking about procurement or Dropbox, we've talked about procurement for hard drives. When you know, the size of the hard drive? And then you ask yourself, well, if we're trying to store zettabytes of data, why do we need 5x petabytes of storage? 

Well, then you need to start like that's the kind of thing whereas a senior leader, you have to start thinking about, okay, well, how much of it is lost to the operating system, how much of it is lost to shredding, and redundancy, how much of it is lost to disaster recovery, etcetera, what's a reasonable multiple? So you have to have some sense of it in your head so you know, that 3x to 4x is reasonable, but 10x is unreasonable and so the more that you understand those things, the better. That doesn't mean that you have to be like, a hard drive firmware software developer somewhere in your path, it does mean you need to commit to understanding the fundamentals of the space that you're in. So, I would say that it's a bit of a false dichotomy to say that you can't be a good engineering leader unless you have been an engineer in the past, I would say, for you to be a good anything leader; you need to understand the fundamentals of the domain that you're leading. And when you go further and further up in the chain, you tend to start taking on things that are related. So for example, I was an engineering leader at Google but then I took on capacity planning. Now, capacity planning is related to engineering but it's a very different field where you have to understand supply chains and fabrication cycles and duty cycles and capitalization cycles. And so there are a lot of things to learn and I found that when I took that on, I had to go hit the books and learn a huge amount of stuff to like, be effective. And I think that's just par for the course, that's what you have to do.

Utsav [19:22]: Okay, yes, that makes sense. Now, you mentioned the job of, a director or a VP is just being an efficient decision factory. You're making tougher decisions at every level and you can also see how that's immediately impactful. Somebody's doing a project but you decide that you're not going to fund this project; you're going to fund another project because it's a higher business priority and that makes total sense. Now, when you transition to being a senior IC or when you are a super senior IC, how do you make a similar amount of impact? And is it even possible to make a similar amount of impact [20:00] what changes, at least in that impact sense?

Bharat [20:05]: Yes, it's a good question, one argument is that to have a big impact, you have to do more than what you would do with your own two hands, you have to marshal a team and that usually speaks to leadership or management. But in the technical realm, I think you can have a very large impact by setting technical direction. If you think about the role of a CTO, it is to understand the arc of the world, the understand the needs of the business, the arc of the business, and then to figure out where those things intersect, where do the challenges of the world? How far out do we have to go for our new technology to satisfy that? There's an intersection point out in the future where you're like, hey, if we proceed along this trajectory, with this many people building this technology, then by this time down the road, we solve that world problem, it becomes monetizable, it floats the business, we're in a good place, and the role of the CTO among other things, to find that intersection and figure out well, how can we bring that in sooner? How can we do it six months, sooner, or a year sooner? 

Because in many business contexts, being able to deliver it six months sooner is all the difference between a successful business and that is carved out, no real market niche, and the business in second place that just can't quite keep up. So, you kind of need to understand how all those pieces fit together to do this, and very often, that requires you to be deeply invested in technical strategy, and setting the technical direction. So, in a large company, like Google, or Coinbase well, there are large companies like Coinbase, massive companies that Google. There's a lot of opportunity for impact by being a technical leader, who understands how all these pieces fit together and can start setting technical strategy that accelerates the business towards where it needs to go. And especially for companies Coinbase scale, I think there's just a lot of opportunity still to focus on this, you get to the scale of, say, a Google or an Amazon or Microsoft or Facebook. There are a lot of things that are so big and so deeply entrenched, that it's quite difficult to like, make big technical changes. So, one of the things that excited me about Coinbase is there's still a lot of time and room and space to make that happen.

Utsav [22:31]: Yes, I think that makes sense. And as somebody who's setting technical direction, how much are you thinking about the business priorities? You probably are, but how much do you weigh that in versus other things like organizational alignment?

Bharat [22:47]: It's a good question. Well, it's an interesting question, because I'm kind of going back to this role after a long hiatus. So, one answer would be like, hey, ask me in a month, two months, when I've had a chance to find my footing. I think that what will likely happen is that I will partner with the organizational leadership, so I've got the time and the boots on the ground to go understand some of the technical challenges, I do not have the double-edged sword of organizational leadership which will simultaneously give me the autonomy to go do some of the work myself, but also would come with the drag of having to maintain that organization and lead it. So my sense is, how this will work out is I'll partner with Manish and I'll keep him informed of what I'm seeing. And I'll start making suggestions and proposals and shifts and adjustments to what we're doing now because he will have the ability to shift the organization in certain directions but it's a little too soon to tell.

Utsav [24:00]: That makes sense. Is it generally true, it seems to me that as senior IC, you need some kind of air cover and backing from senior management to drive maximum impact, you can have all of the ideas in the world, but unless they get funded, or they're discussed, and there's somebody who's saying, you know what, we should fund some of these ideas. There's not that much impact we can make.

Bharat [24:29]: It is always beneficial to be able to bring leadership along with any decision that you make, I would say it's not just leadership, it's the whole organization. No engineer is in a vacuum, no engineer has the kind of authority and frankly, it takes time to build trust and credibility. I'm stepping into a space where there are a large number of very smart people who worked [25:00] hard and understood the complexities of the problem space. Most systems are sophisticated solutions to complex problems and it's easy to mistake that sophistication for unnecessary complexity if you don't understand the problem. So step one in all of these cases is to go understand the problem, go get a sense for what they're solving and why, go understand the solution, and then try to understand where is it correct in terms of its sophistication? And where it could be adjusted, where it's maybe a little too unnecessarily complex? Or maybe it could be simplified? Or maybe we can re-evaluate the invariance of the system? What was the design for? What is it solving? Are those things still true? Are there ways to make shifts, and in all cases, that's a dialogue, that's a understand frame out the problem, get people into a room, and have them understand it? 

And my sense is generally, the right solutions, there are emerge it, you converge it and the things that hold you back, are essential when it's just going to be prohibitively difficult, or maybe people have invested in certain areas, and it feels like too big, correction, there's always a level of change management to get from the trajectory or on to a different trajectory. I'm not so worried about that, my sense, and one of the reasons why I've enjoyed the process of getting to know Coinbase is that I've found them to be the folks I've talked to, and I talked extensively to folks in the team before I joined. I found them to be very open to this and very humble and very excited about the prospect of making some changes and looking to bring in some DNA from the outside to go focus on some of the challenges they have. And frankly, I'm excited to just get people to collaborate with on this. So, I'm not too worried about that, I'm not worried that, I'll show up, I'll have a bunch of great ideas, and no one will listen. I am honestly more concerned about getting the time to be able to invest in learning and understanding the great work they've done already. So that when I get to the point where I'm thinking about changes to make I understand the problem and the solution.

Utsav [27:26]: Okay, I think that makes sense. And when you step into such a role, is there a particular team that you join? Are you reporting to the same line manager that other engineers are reporting to, how does that work? And what do you think makes sense, I guess?

Bharat [27:42]: So, I begged for a bit of an exception to the normal case where you would join the team and work on that problem. And the reason why I begged for this exception is that I felt that one of the advantages to bringing in someone as a more Senior Technical IC is to give them the time and space to go make their assessment. And so that, I asked for that it's a privilege and I appreciate the consideration that Coinbase offered to let me go have a little flexibility to make my assessment. And then I will be in a position to have a real discussion with the leadership team about what's the right thing to work on, I've found, Manish, and the rest of the team to be very thoughtful about creating that space. And to be honest, when you're stepping into a company that's been doing work like this for eight or nine years, it's difficult to be able to figure out this stuff quickly. 

If even people way better than me, and there are people way better than me could come in and figure it all out in a couple of months, then they haven't gotten nearly far enough. For a system this large, this complex, it's a $50 billion company, that's one of the top crypto exchanges in the world in an emerging space, that's been doing a huge amount of pioneering and is aggressively moving. There's a tremendous amount of sophistication that solution they have, it just takes time. So, I sense that if I was going to do a more conventional roll out of the gate, I would just be picking one thing now and going and focusing on it but I asked for their indulgence to let me take a step back and look at it from altitude. And spend the first quarter or two doing that, so that I could be more productive down the road and I view this as a long-term engagement so, and so there's Coinbase. So, it makes sense to carve out some time upfront, because it becomes a very small percentage of where I and hopefully many other people who follow the same path can add value over time.

Utsav [29:50]: That makes sense. Let's say that I'm ahead of engineering and I'm thinking about, okay, I have a bunch of line managers, let's say it's not a large company. Like [30:00] 30 engineers, 40 engineers. At what point should I be thinking about bringing in or even having, technical leaders or Senior Technical IC, does it just not make sense until you have an extremely complex application? Does it not make sense until you have 200 engineers? What's the framework you should be using to think about this?

Bharat [30:23]: What is the value of Senior Technical engineers? Well, they're experienced so they've kind of, they have seen things that work and don't work in the past; they have enough understanding of the fundamentals of prior different situations. Very often, one of the real roles of technical leadership is deciding what don't do now what you do, like pruning bad paths from the decision tree, which saves you a ton of time. And so I would say that if you're the head of engineering, and you're the work that you need to do, and the operating specifications of it are well defined, you may not need, you may just need people to go execute against your roadmap, and which is, by the way, hard enough to find in today's industry. And that's not to say that you're not getting a talent, you can always use a talent, I think you should always be aiming for a talent. But it's more in a situation where you're trying to do growth into an unknown area, you're trying to go up by an order of magnitude, you're trying to move into a space, where you don't necessarily have the skills in house, most companies, if they 10x, a couple of times rapidly get into a space where they simply don't have the engineering talent in house to do this. Because when you take a product and you make a 10x, larger, at Google, we always used to say, when you make a problem 10x larger, it ceases to be the same problem, it becomes a very different problem it becomes about the scale that you're moving into. 

So, if you're a company that's trying to grow rapidly, or pushes into new domains, or understand things differently, it's helpful to have a deep bench of senior technical talent who can go off and investigate, building complex systems. That's not the kind of thing that you would normally conventionally put on the roadmap, you need people who have the time and the space to go explore and come up with new and innovative ideas. So I do think that like, it also depends on, are you a product company? Or are you a platform company? Are you a technology company? So for example, Google fundamentally was a technology company. And so they invested in technologists to go explore all kinds of spaces and then they were good at recognizing odd this will be useful when we 10x again, this will be useful when we 10x again, let's keep these things, let's double down on our investment, some of these spaces. As a result, Google was always well prepared, when we grew, our systems and our technology could keep up. Many companies when they grow in the scale, it's hard for their systems and technology to keep up because they don't have that deep bench of people who's already been kind of ranging out in front, fairing out new things to do. And so then when their growth kicks in, they wind up in a scramble to keep up with that growth. Dropbox was great at Dropbox grew fast in you know, early 2010, 2011, 2012. And they had a world-class technology team that was able to stay ahead of that growth curve, which is not easy, and an event investing and a lot of people to go build large scale systems ahead of the demand. The problem is if you build those large-scale systems ahead of the demand, and the demand doesn't show up, then you've wasted that investment. And if you don't build ahead of that curve, and the curve does show up, then you're going to fall over so it's a tricky calculus. 

Utsav [34:02]: I think the whole idea of you have to think about how much ambiguity you have in your roadmap and that helps you inform that decision makes a ton of sense to me. Could you share publicly the kind of things you'd be thinking of at Coinbase? The kind of maybe projects not specific projects, but just the kind of work you're thinking about, like infrastructure work, exploring new things, you think it's just off the table? 

Bharat [34:33]: Well, I want to be respectful to Coinbase and not divulge any of the cool things that they're doing behind the scenes. So, let me just say this I feel comfortable with a pretty broad charter and I think all of those things are on the table for things that we could talk about. They're all areas where Coinbase I think, is growing rapidly and innovating and so, I currently I'm not trying to narrow [35:00] my options down right now. But let's talk in a couple of months after I've gotten my boots in the ground, and then we could see what I could share with you.

Utsav [35:10]: Cool. What does it mean to be a CTO of a small company or start-up when you started with old school? That's such a different role from being a super senior IC Google and what mindset shift did you have?

Bharat [35:26]: Well, it's tricky, because an old school we're solving a humanity scale soft problem I think it's deceptive to think that you can solve these types of problems with technology. Technology can help but technology is the tool, and technology has a wide range I mean, like, banging, flint, and steel together in the woods to light a fire is a technology. Rocket fuel and fire to put a rocket on Mars is also technology but if you're lost in the woods, you don't want to Falcon nine, you want to Flint and Tinder and steel, you want the right technology for the job. So, part of the role at old school was, well, what is the right technology for the problem and a lot of what I did at old school was understanding the problem to get a technology solution that made sense for where we needed to be over a certain period. And I would say that, unlike many Silicon Valley companies, the old school was technology second or maybe even technology third, after the pedagogical model. And after the curriculum design and the content design, and the operational model of running schools, then you have the technology, it's kind of scaffolding and a structure and a utility, that helps make things easier.

And so, for a CTO of a small company, I do think you want to be thinking very carefully about what is the problem to be solved? Where is the correct place for technology to be injected, such that technology supports and drives the mission and the business? And then how do I Marshal the correct team to go get it done efficiently? And very often, that means, a buddy of mine just joined a very small company, the CTO, he's probably going to start by looking at the problem assessing where it's going to be say,  the scope of the problem today. Let's say you have 10,000 users today, and you think that 18 months from now, you're going to have 100,000 users, and three years from now you're going to have 300,000 users? Well, you don't need to build a system anytime soon that can support a million users. So don't go chasing massive scale efficiency, start thinking about, well, what's the operational characteristic of your users? Are you high transaction volume? Is it okay to be lost? Some systems care, a huge amount around integrity, some systems care, usually around response time. You start thinking to yourself, what are the different decision points you can make? And then you can start getting into driving decisions like, well, okay. 

Are we a product or a platform? I was talking to a CTO recently and my big question for him is like, if you cannot answer whether or not you're a product or platform, you don't know where to marshal your resources. So very often, what the early-stage CTO has to do is to go figure out, okay, what are we? And what technology do we need to have? And what characteristics do we need to have by 18 months to 36 months out? And how do I build out an architecture that is neither too much nor too little, that achieves the goals of the business in a way that makes sense? Some businesses need to be high volume, very fast response time, some are pretty low volume and can have a reasonable risk, medium response time some can be heavier, some to be lighter, some need to be models, or you could iterate quickly, we need to be able to push a feature every couple of days. Some of them are the types of systems where you don't want to push features every couple of days. Your users want stability, they're like, utilities, they expected not to change all that often, and they prioritize stability over rapid injection of features. So very often, the CTO needs to decide, okay, once I know what those characteristics are, then I can make architectural decisions, then I can start thinking about what frameworks or technologies I need. Do I want to be native on mobile? Or do I want to be cross-platform on mobile? All of these decisions, I think an early stage CTO should be thinking about in the vein of where are we today? Where are we going to be before our next rates? Because very often, what you're trying to do is get to the point where you raise the next round of money.

Now, if you're the kind of business where you [40:00] don't necessarily need to raise a bunch of money, and you're looking at a longer-term arc and you're thinking, okay, well, we are going to run this, and we are very confident that we're going to get to 100 million users, which is a large number. It's not a billion-plus user, like at Google scale, but most companies have to get to 100 million users or over the million static 100 million users at a certain transaction volume, you got to figure, how many actives you're going to have? How big does your serving stack need to be, etcetera? So very often, the early-stage CTO is, got to be thinking about it in terms of stages and tears, and figuring out at every stage what you need, and then building out a roadmap to get there. So, we're going to do these architectural decisions in the early stage, and then when we start pushing past some of these thresholds, if our users get more active, or we get more users, or we're trying to monetize, then we're going to make this series of different decisions, and we have some decision points. And that's hard that requires discipline, requires hygiene, it requires a lot of careful thought, on the part of an early-stage CTO, and you know that CTO is also probably coding, and probably working very closely with the engineer and understands all the engineering challenges and solutions. And that CTO is also working closely with the CEO and their other peers and business partners to make sure that they understand where the business is heading and what's coming. That way, the CEO is not off selling products that engineering can't build and engineering is not building the wrong things, because they didn't understand what the CEO's vision for the company is. So, you kind of sit in that spot and it's tough, you spent a lot of time as an early-stage CTO, running around, dealing with a bunch of fires. But you wind up being probably the only person who can translate what's happening in the business down to the line engineers.

Utsav [42:04]: That makes sense. One of the distinctions you spoke about was being a product versus a platform, why is that important for a CTO or even somebody like an engineer joining that company to know?

Bharat [42:17]: So imagine you're the CEO, and you've got this app that you built, and you're going and selling it. And your customers like, this is awesome we would like you to build a version of that app for this slightly different business. The CEO is like, well, great we've got this one app, we have a team that's built it, how hard can it be, I'll go make a deal to sell it slightly differently to this other adjacent business. Now you go back to the team, and the team has two choices. If the team has built the app like a product, they don't have an easy way, the app is very tightly tuned, they've made a whole bunch of assumptions about what they're building the domain, and those assumptions are valuable. Because by making assumptions, you simplify your code, your code does not have to have a lot of abstractions, it does not have to be super flexible, it does one thing and it does that thing well. But now if you want that code to do something different, it's quite hard, because you have to go and find all the places where you made those assumptions, which are time-saving and you'd have to generalize them, you'd be like it could be this or that, and then you start getting into really weird edge cases. So it's difficult to take a specific product, and then transfer the value of it over into another specific product, you kind of have to rebuild the new product from scratch. But if you build a platform, then most of the value is in the platform and the app itself is merely a conduit to the value in the platform, using, say API.

Now the downside to this is if the CEO comes back and wants to add a feature to the product that can be harder because you've got to go add the feature to the platform, then you got to come to add the feature to the product, you've got to plumb between the app and the platform to get it to work. And it's probably not exactly what the CEO wants, because the platform itself has to be generic enough to be flexible and the app is kind of doing a translation there. So, you lose a little bit of that like hyper-specific, amazing app specificity but if the CEO then goes and makes a sale into an adjacent space, you're then writing a new app, which sits on top of the same platform, and you get to translate much of the platform value over into the new app. So if things like identity, and storage and networking and configuration, and basic UI modules, all those things are part of your platform, then reassembling those Lego pieces into a whole new beast is not so difficult you can do it quickly. It will also be not as hyper-specific, but it will also be a lot more stable and reliable and you get a lot of reuse in your engineering team. So, you're [45:00] always making some of these tradeoffs and it's important to know which ones you're making at any given time because that way if your CEO is about to go into making a sale, you want them to understand, how difficult will it be for your team to achieve the result.

If you build the product, and the CEO is now excitedly selling that product in a way that was not designed to support as the CTO, you want to be able to give guidance and yes, we can do that, if it's good for the business, we'll do it. But because of certain assumptions we made along the way, it will take you a non-intuitively long period to make that happen. And that's where the CEO needs to understand, okay but if I shifted a little bit and sold something slightly different, instead of selling to this customer, I sold to that customer and I got similar value, but much easier, lower hanging fruit in terms of cost, then I get a huge benefit that way. So, very often you as the CTO become the translation layer and that's why it's important to start asking yourself these questions, so if you're a CTO upfront, I would be talking to the CEO, what kind of sales model do you want? What things are we willing to trade off, explaining to your partner CEO, the difference between a platform and a product important, so that you can kind of co-create the solution together? You can say, okay, yes, we're going to be a platform because we anticipate having to be flexible about our domain and we're willing to trade off some hyper specificity in our app, versus saying, hey, no, we're going to be an app, because we're going to be a product because we don't believe we're going to shift domains and we want the hyper specificity in our product and we want to make a bunch of simplifying assumptions upfront. But these are all things that you need to negotiate with your peers so that you don't wind up kind of making bad assumptions going down a path and then finding that the business needs necessitate you doing something which you kind of ruled out based on early assumptions, and then you have to go back and do some very heavy lifting to get back on track.

Utsav [47:12]: That makes sense. And in fact, I feel like it can be applied to things within the app, parts of the app could be more platforms is that you can easily make shifts there.

Bharat [47:21]: Right. And by the way, it's a very common issue among engineers, when they don't know what they're going to be asked to do to build more generalized systems. It's like you're building your first app, and you have to build an authentication system. One approach is to do something very, very specific, hard code everything because this is the only time you're ever going to do it. But very often, people are like, well, I might get called on to reuse this in some other way so, I will bake in an abstraction layer at this moment. And when you bake in that abstraction layer, you are future-proofing the system, but it comes at the expense of a bunch of additional work every time you use it. And so, you potentially win big under certain circumstances, but you pay a tax every day that you use it. I am in favor of don't pay the tax, do the dumbest thing that can work, let the business needs evolve, and then once you understand truly what the business needs, go re-architected as necessary because that way you don't waste time upfront but you know that if certain business decisions are made, you have a set of work that you're going to have to do. And that's not the worst thing in the world because you can plan for it, you can make a series of decisions down the road that says, okay, if and when we cross this threshold, and we make the sale, we need to hire a team to go do this large scale refactoring, and pull our identification authentication system under separate cover, put it behind an API, rebuild the thing from the top down and by the way, clean up a lot of stuff in the process. That's a reasonable thing to do and it's better to do that in a more top-down way approach, than to kind of back your engineers into a corner where they're defensive all the time in every area.

Utsav [49:12]: And maybe as like a final question to wrap up, let's say that you are like an IC who's thinking for the first time, should I be transitioning into management? I'm not so sure whether I'll actually like all of the work that is in management, but I feel like I can go up my career have a larger impact. As someone who has made this transition to and from multiple times, what is the first thing that you would recommend? Is it just explore it, try it, see if you don't like it come back. Is there anything else you would tell people to think about?

Bharat [49:44]: You know, it's funny, I've had this conversation so many times with different people and I think a lot of people go into management for different reasons. So, whenever someone comes to me say, hey, I wanted to be a manager. My first question to them is always why? What motivates [50:00] you to do this? And a lot of times people what they're saying, pretty much very few people come to me and they're like, I love management and that's just what I want to do. Very rarely do I get that answer I'm sure there's a lot of people who feel that way but I don't get that very often. Mostly, what I get is people saying look, I want my career to progress and all the people who I think are succeeding are in the management of some kind. So, therefore, I must be in management of some kind and once I'm in management, then I will be succeeding. And the tricky thing is that it's not like engineering flows into management, it's not like it's the next stage in your career. That's saying work hard as an engineer, and then you become a manager, it's like saying, work hard as a painter, and then you become a ballerina, and they’re completely unrelated in so many ways. There are different motions, different skill sets, they require you to care about different things some people like it, and some people don't. So, the only way to know is to get some exposure to it and there's a lot of ways to get exposure to it. One of the ways to get exposure to it is to talk to a lot of managers, what do you like? What do you not like? What is your day like?

Walkthrough the calendar, a lot of times I dissuade people from management merely by showing them my calendar. Let's look at my calendar, let's see what I'm doing today, okay, today, I have eight hours of meetings, by the way, right now, they're all on zoom. And that might be 10, to 14 meetings, a lot of 30 minutes one on one, some breaks, each one of them, has a certain amount of context for the individual. Here's what I'm trying to achieve in each of those meetings, here's like, is it relational? is a strategic? Is it transactional? What's going on in that meeting, what am I trying to achieve? And then you look at it like day over day, week, over a week, month, over a month, quarter over quarter, and you're like, okay, here's what it's like to be a manager, here's what I achieved, here are the goals and you're the outcomes and here's what I do to get there. And I think some people look at this, and they're like, I don't think I would enjoy that and some people look at this, and they're like, I don't think I'd enjoy it but I have to try it to know because it feels like the right way to progress. And it's a very personal choice, a lot of people don't know till they try it. The challenge is that because management seems like a step up and a large number of organizations do kind of glorify their management team so it feels like a step up. 

Very often people don't feel like they can go from a management role to a non-management role, without leaving the company. And so one of the things that I would say, it's not so much for the people who are leaping, it would be for the company, I would say, listen, make it clear that management is not the route to success, doing what you do well, and having an impact is the route to success. You can switch between a management track and a non-management track, but that's not the relevant piece. It is taking what you're great at, and getting to do that as much as you can in a place that has access so, that people feel comfortable trying it without having felt like they have to leave the company, once they don't like it. That makes it I think, easier for people to find out if they like it if they're good at it. And the other thing I would tell people is when you realize that it's not for you don't keep doing it, go talk to your leadership team, have a candid conversation, and say, okay, I tried this and now I need a break for a while. And a good leadership team will recognize your value proposition, and we'll find a way to make that work. So that you're not so ugly tied up in the like, well, I got promoted to being a manager and if I don't keep managing, it's going to be a demotion, even though I don't like it and I don't want to keep doing it and I'm doing it poorly, and I think that's the trap a lot of people fall into that I see.

Utsav [54:02]: Yes, I think that makes sense it should be a lateral move, and it should be super easy to transition back. And that's what we'll even have the organization give bad management away, especially people who are not motivated about it.

Bharat [54:14]: You know what they say, people leave good companies because of bad management and stay a lot longer than they should because of good management, people's relationships to their managers are paramount it's important. And anyone who goes into management should take a lot of management training. There's just a lot of things that are not intuitive, and you don't want to learn it on the job you should be trained in it it's important.

Utsav [54:43]: Well, thank you so much for being a guest I feel like I've again learned a lot. 

Bharat [54:48]: Thanks buddy. It was fun I enjoyed it.

Utsav [54:50]: Yes, and I'm excited to hear about your role in a few months.

Bharat 54:56]: I'm excited to dive in; I think Coinbase is a great company. So [55:00] I'd be happy to give you an update once I can answer some of those questions. 

Utsav [55:06]: And congrats again. 

Bharat [55:09]: Alright. Thank you.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Sugu Sougoumarane is the CTO and Co-Founder of PlanetScale, a database as a service platform company. Previously, he worked at PayPal and YouTube on databases and other scalability problems, and he’s one of the creators of Vitess, a graduated CNCF database project that served as YouTube’s metadata storage for several years.

Apple Podcasts | Spotify | Google Podcasts

We discuss his early days at PayPal and YouTube, their initial scalability problems, the technology that helped them scale out their products, the motivation behind Vitess, its evolution into PlanetScale, and some of the innovative ideas coming out of PlanetScale today.

Highlights

5:00 - Being interviewed, and hired by Elon Musk at X.com. Working with some of the PayPal mafia like Max Levchin and Reid Hoffman.

9:00 - Solving PayPal’s unbalanced books via a better Activity Report

15:00 - PayPal’s Oracle database and the initial idea of sharding.

20:00 - Early YouTube architecture and culture, and the consequences of explosive growth.

24:00 - Sharding YouTube’s database.

32:00 - The story behind Vitess. It all started with a spreadsheet.

40:00 - How a user with 250,000 auto-generated videos caused YouTube to go down. How Vitess fixed that, and the implications of a SQL proxy.

45:00 - The motivation behind keeping Vitess open source, and interest from other companies dealing with similar problems. The motivation behind going from Vitess to PlanetScale

53:00 - How PlanetScale can improve on some traditional relational database developer experience. How NoSQL was never actually about getting rid of SQL and was more about skipping traditionally rigid schemas. MongoDB’s support for SQL, and PlanetScale’s approach to solving schema changes.

58:00 - Technical innovations coming out of PlanetScale.

1:05:00 - Running databases in containers and the implications

Transcript

Utsav Shah: [00:15] Hey, welcome to another episode of the Software at Scale podcast. Before we get started, I would just like to ask a quick favor from any listeners who have been enjoying the show. Please leave a review on Apple Podcasts. That will help me out a lot. Thank you.

Joining me today is Sugu, the CTO and Co-Founder of PlanetScale, a managed database as a service platform company. Previously he worked at YouTube for a really long time from 2006 to 2018 on scalability and other things, including Vitess and Open Source CNCF project, and before that he was at PayPal. Welcome.

Sugu Sougoumarane: [00:53] Thank you. Glad to be here.

Utsav: [00:56] Could you maybe tell me a little bit of, you know, when you got interested in databases and MySQL? Cause it seems, at least from my understanding you've been interested in distributed systems, my sequel databases for like a really long time.

Sugu: [01:11] Yeah. So, I would say, the first serious involvement with databases started when I joined Informix in 1993. That was the time when there were three huge database companies, Informix versus Cyber's versus Oracle. And I guess eventually Oracle won the war, but that was the first time when I came in touch with databases. I specifically did not work with the engine itself, but I worked on a development tool called 4GL, which was popular those days; not there anymore. So that was actually my first introduction to databases.

Utsav:

[02:08] Okay, and I guess you took this skillset or this interest when you moved to YouTube from PayPal. So, what were you doing at PayPal initially and what made you decide that you want to work at YouTube?

Sugu: [02:25] So PayPal, I guess when I moved from Informix to PayPal it was mainly because this was around 2000. This was the time when the internet boom was happening, and it was clearly obvious that Informix had fallen behind it; like they were still trying to make a client-server work and that was pretty much a dying technology. That's when I kind of decided to make a change. It was somewhat of a career change when you realize that the technology, you're working on is not going to be around much longer. I don't know if other people saw it then, but it was pretty obvious to me. So that's the time when I decided that, you know. I kind of made a decision to start back from the beginning because you'd be surprised in year 99, not many people knew how a website worked. Now, it's probably common knowledge, but I had no idea how a website worked. 03:36 inaudible requests come in, you know, what servers do what, and those things. So, it was all kind of unknown science to me. So, for me, it was like I'm starting a new career; like even within software engineering, each of these things sounds like a different career, you know, there are choices. It is as if I stopped being software engineer A and now, I'm going to become software engineer B. So, the only skills I could carry over were my ability to write programs. Beyond that, I knew nothing. And I had to learn all those things from scratch again at PayPal. By the way, I did not join PayPal directly. I actually joined X.com, which was founded by Elon. So, Elon actually hired me and then later, X.com and PayPal merged, and that's how we became PayPal. 

Utsav:

[04:41] I think if you can tell us a little bit about Elon Musk, I feel like this podcast will just shoot up in popularity because of anything to do with Elon Musk now.

Sugu: [04:50] Alright. There is this story I think I've said that it's not really commonly known, but at least, I'd say I might've said that in one place, which is that, actually, I failed the interview. I went for the interview and Elon thought I was not worthy. They decided to pass on me. But then I was so excited when I found out, at that time I didn't know who he was, this was the year 2000. I mean he had just sold his previous company, but there were many other people who had done that. But when I found out what he was building, I got so excited that after I heard that he passed on me, I sent him an email saying why he should hire me. And somehow that email convinced him to bring me back for a second round. And then I got hired and the rest is history.

Utsav:

[05:55] So interesting. What is it like working with him? Did you work closely with him? How big was X.com at that time?

Sugu: [06:02] It was very small. I think we were about 15 engineers, 12, 10, 15 engineers. I think the total was 50, but most of the other people were customer support type of thing because it was a bank, right, so there was a lot of customer support. Yeah, at that time, it's surprising many of those people that I worked with, not just Elon, are now celebrities like Jeremy Stoppelman who was like sitting right next to me. I remember asking, Jeremy Stoppelman is the CEO of Yelp in case you want to look it up. So, I remember I went there and he was the guy next to me. So, I said, "Hey, you know where's the bathroom?" [06:53] "Where do you get coffee?", and sometimes you say, "Yeah, it's there", sometimes you say, "I don't know". And finally, I asked him, "So how long have you been here?" "Oh, I joined yesterday". I literally joined the day after Jeremy; I got hired. Yeah, so those were good times. So, Elon himself, by the way, like SpaceX, he used to talk about that [07:21] as his dream, you know, going into space. And he's really inspiring when he talks about it; like he starts talking and everybody just shuts up and listens because he used to talk about him being broke. There was a time when he was broke, where he said he didn't have money for food. Did you know that Elon was broke once in a while?

Utsav:

[07:49] I feel like I've read about it or seen some YouTube video where; now there are so many YouTube videos where they take quotes of Elon and it's like inspirational stuff. I think I've seen some of those. That's so fascinating. And yeah, now that clearly became the PayPal mafia with so many people making so many successful companies and stuff. Like, did you work closely with any of the others, like Jeremy Stoppelman? 

Sugu: [08:15] Yeah. There were lots of them. So, I actually worked with Max a lot more right after the merger. So Max, definitely, Reid Hoffman, although I don't think I interacted much with Reid Hoffman, there is Roelof, for sure, Steve 'Chairman' Chen, obviously that's how I ended up at YouTube. My memory fails me, but if you look at the mafia, yeah, I have worked with all of them.

Utsav:

[08:52] Did you work on the Igor program, like the fraud detection stuff, which I've read about publicly?

Sugu: [08:58] Oh, no, I did not. I worked on some fraud systems, but not Igor specifically. I am trying to remember who worked on it. It was probably Russel Simmons.

Utsav:

[09:14] So what did you do at PayPal? I am curious.

Sugu: [09:18] So the thing is, this is a good lesson, the struggle I was having initially at PayPal was recognition. There were a couple of things. I was kind of an unknown there, nobody knew who I was, and I was part of the merger, right? I came from the X.com PayPal merger, and there was a lot of back and forth about which technology to use. Eventually, we decided to continue with the original PayPal technology and stuff using the X.com technology. So, the struggle I had was, how do I prove myself to be worthy. So, in software engineering, you have to show that you are good and then they give you good projects. You don't get to choose the projects because you get assigned projects, right? So at least that's how the situation was then. And by the fact that I was an unknown, all the good work was going to the people who were already recognized for how good they were. So, there was this one tool called The Activity Report, which basically tallied the total of all transactions on a daily basis, and say this is the total in this category and credit card category, ACH category, and that kind of stuff, and used to report those numbers. But the auditors came in and they said, you know, with these transaction numbers that this report produces, you need to tally them against your bank accounts; against the total balances that users have. The first time the auditors came, there were like a few million dollars off, you know?

And they're asking like, where is all this money? I don't know if you remember when PayPal initially was founded, the business model was to make money off of the float. When we transfer money, there’s a float. We'll basically invest that money and whatever money we make out of that. So, people are like, 'oh, it must be the float', you know, that difference was because of the float and the auditors were like, 'no, that explanation is not sufficient. You need to show the numbers and know that they add up and that they tally’. And so, finally, we figured out where all it is on the tally. But the problem was that PayPal was software written by engineers and how do you create transactions? You insert a row in the transaction table and commit, you insert the row in the transaction table, update a user's balance, and then do a commit, right?

So that's how you do it. But sometimes the transactions go pending, in which case we don't update the balance. Sometimes we complete the transaction, and then after that, you update the balance. And guess what happens when you start doing these things. There's this thing called the software bugs that creep up. And so, every once in a while, after they release the transactions wouldn't add up to the user's balance. And sometimes they won't tally with what's actually in the bank accounts. And so, every few days, the activity report is used to produce discrepancies. And because the software was written inorganically, there was no easy way to find out like you have, you know, million transactions. How do you figure out which one went wrong? The numbers don't add up to the totals here. And every time this used to happen an engineer was assigned to go and troubleshoot.

It was the most hated job because it's like finding a needle in a haystack. Like you'd say, 'Ah, activity report conflict. Now I got assigned to do this. Why is my life so bad?' You know? So that was the situation. And then, at one point in time, I said, you know what? I'll own this problem. I'd say, 'Just give me this problem, forget it. I will investigate this every time it happens. And I will also fix the software to make it easier.' So, I took on this and I rewrote the entire program, such that, to the extent that, at the end of its maturity, it would find out exactly, previously it was all totals, that at the end, once it got fully evolved, it will say 'this particular account has this discrepancy and most likely this is the transaction that caused it'.

So, it will produce a report or a list of conflicts with a very clear clue about where the problem is, and then you spend five minutes and you know what went wrong. It would find software bugs in it because sometimes somebody forgot to update the balance. It'll say, 'Hey, you created a transaction and didn't update the balance, you know, so here go fix it'. So that it used to do. And eventually what had happened was because it was so solid, it kind of became foundational for PayPal. Like people used to ask for reports about, you know, transactions, and they would say, 'How do you know this is correct?' and we'd say, 'Oh, I validated it against a not-so-good activity report'. 'Okay, then it's good. I trust these numbers. So that kind of stuff. So, making that software and taking that pain away from all the other engineers, got me accepted as one of the cool engineers. 

Utsav:

[15:12] So interesting. How big was the engineering group at that time? 

Sugu: [15:19] So the PayPal by the X.com site, there were about 15 and on PayPal's side there were also 15. So, we were about 30 to 40 engineers.

Utsav:

[15:28] And once you got accepted by these people, what were your next projects, I guess? 

Sugu: [15:33] So having gotten accepted, I became the kind of person who 'd say, 'If you give a problem to Sugo, you know, you consider it a salty kind of thing'. So, which means that anytime PayPal had a struggle where you say, 'Oh, you need the top brains on this', I used to get called. And the biggest problems later were all about scalability. So, I kind of moved into the PayPal architecture team, and used to solve all problems related to scalability. And the scalability was always the Oracle database, because it was a single instance database, and every time, the only way we could scale it was vertically. So, the idea of sharding was actually born there for us. We needed to shard this database. But it eventually succeeded only on YouTube. We didn't succeed at sharding at PayPal itself, but we kind of knew how to do it. We kind of learned; we went through the learning process when we were at PayPal about how to shard a database, but by then the [16:46] had taken place. There was a drift and lots of things took place after that.

Utsav:

[16:50] That's so interesting. And is that where you met Mike?  You were discussing some of these stories, [16:55]

Sugu: [16:56] Yeah, Mike was reporting to me. So, there was Mike, there was another person called Matt Rizzo. They were the top engineers, you know, one of the top engineers at PayPal. We still use some terms that only Mike and I would understand. We'd say, 'Oh, this is APCD right?' and then we'd laugh. Nobody else, I would say, would understand what that means. [17:28] ACH process completed deposits, Bash tool that he wrote. You probably know that Mike is really, really good with Bash.

Utsav:

[17:39] Yeah. He knows his Bash.

Sugu: [17:42] He knows his batch. That was kind of my fault or doing because there was a time when I just told him a problem, you own this problem, you have to solve it. And he'd say, 'Do you think Bash is a good idea? Sounds cool'. I said, 'If you think it's a good idea, it's a good idea. It's up to you. It's your decision'. So, he went and he is still scarred by the fact that I let him do it and said, 'Why did you let me, why didn't you tell me it was a mistake?' 

Utsav:

[18:15] I can imagine this very horrible process at PayPal, like adding and subtracting money and it's all written in bash.

Sugu: [18:23] That wasn't written in Bash, but there were some tools that we wrote in Bash. Yeah, the money part was all SQL, SQL+ actually. [18:36] I guess, that could be called the Kubernetes of those days. You were writing an orchestration tool and that we wrote it in bash.

Utsav:

[18:46] Okay, that's so interesting. And then you're saying that the idea of sharding was born at PayPal, or like the idea of sharding or my SQL database, like you figured out how to do it. Why would you say you did not complete it just because there was not enough time or, at that point did you decide to leave and go to YouTube?

Sugu: [19:04] There were a lot of reasons, some of which were changes in management at that time. There were the PayPal engineers and there were the eBay management that had acquired us and the relationships were very strained. So, it was kind of a very difficult thing. So, there was no coordinated, what do you call it effort? There was no focused, coordinated effort towards solving it. It was all over the map. There were a lot of committees spending a lot of time in conference rooms, discussing things, not coming to conclusions, you know, that kind of stuff where there were just 'too many cooks in the kitchen' type of thing. I think the core first kind of figured out how this needed to be done, but we were not able to, you know, push that idea forward, but you finally made that happen at YouTube. So, which proved, you know, it kind of absolved us of you know what you're doing.

Utsav:

[20:20] So, yeah, maybe you can talk about the earlier days at YouTube. What was YouTube like in 2006? How many people were there? Was this like pre-acquisition, post-acquisition? I don't know the exact -

Sugu: [20:29] -just around the time of acquisition? So actually, around the time when YouTube was founded, I actually went to India to open the PayPal India office. But I used to visit YouTube every time I visited. I came to the US and I used to come pretty often. So, I've been closely following, at some point in time, Steve, and Matt kept hounding me saying, 'What the hell are you doing there? This thing is catching fire. You need to come and join us right now'. So finally, I came back and joined them. But yeah, the YouTube culture was kind of a carryover from PayPal. It was very similar, lots of heroics, you know, lots of, like each person was an impact player. Each person made a huge difference. Like one person owns all of the search, that kind of stuff. One person owned the entire browsing experience. There were only like 10 engineers. So, everybody owned a very important piece of the stack.

Utsav:

[21:50] Okay, and what were some of the early problems that you ran into? Or like, I guess what were the fires that you were fixing as soon as you started?

Sugu: [21:59] So the one big thing I remember, there was this rule at YouTube where you join, you have to commit on your first date.

Utsav:

[22:11] Okay.

Sugu: [22:12] I broke that rule. I said, I'm not going to commit on the first day. But I spent time learning the code and produced a huge PR, a huge pull request, that we wrote an entire module because it was badly written. So, the biggest problem I found, I felt at least at that time, was because the code was organically grown and incrementally improved, it was going to run out of steam as we added more features. So, I felt that we needed, you know, to clean it up so that you can put a better foundation underneath so that you can add more features to it. So that's basically the first work that I did. But then later after that, the bigger problem also became scalability because we were single, we were uncharted. There was only one database that was called Main and we had to do the re-sharding. So that was the biggest challenge that we solved. We solved the re-sharding [23:22] test later, and adopted that sharding method. But when we first started, it was at the app layer.

Utsav:

[23:33] Okay, at what scale did you have to start thinking about sharding? How many engineers were there? How much data was going in and what was being stored in this database? Was this just everything of YouTube’s metadata in this one large database?

Sugu: [23:48] Yes, all the metadata, not the videos themselves because the videos themselves were video files and they were distributed to CDN, but all other metadata, like video title, likes, dislikes, everything else was stored in my SQL database. It was running out of capacity very soon, and the hardware wouldn't have kept up. Vertical scaling wouldn't have taken us much further. And that's when we said, 'Okay, we need to shard this'. We made a very tough call, which is that we are going to break cross short, transactional integrity. It was actually not as big a deal as we thought it was going to be because of the nature of the app, maybe because users just mostly work with their profile. There were a few things about comments because I posted a comment to you, so where do you put that comment? and some little bit of issues around those types of things, but otherwise, there was still quite a bit of rewriting because an application that asses an unsharded database and makes it [25:10] is non-trivial no matter how you put it. We had to build a layer under the app and write everything, change everything to go through that layer, follow certain rules. And then we changed that layer to accommodate sharding. That's basically how we solved the problem.

Utsav:

[25:27] Yeah. That layer has to basically decide which shard to go to. So, you were just running out of physical disc space on that My SQL box which seemed like the problem?

Sugu: [25:37] I'm trying to remember, whether it was a disc, it might've been a CPU actually.

Utsav:

[25:42] Okay.

Sugu: [25:44] I think it was the CPU. It's either CPU or memory. I think we could have a disc. I'm trying to remember if it's CPU, memory or IOPS, most likely memory, because of working set size issues. I could be wrong.

Utsav:

[26:14] I'm just curious, you know, do you know how many videos you had in YouTube at that time when you started running, [26:20]

Sugu: [26:21] I remember celebrating. Let's see it was either a hundred million or 1 billion, I don't know.

Utsav:

[26:31] Okay, 1 billion videos and that's when you start running out of space? [26:35] total video, that just shows you the part of like my SQL and like [26:40]

Sugu: [26:42] Yeah, it's amazing that I don't know the difference between a hundred million and 1 billion, you know, but something, yeah.

Utsav:

[26:52] Sounds like [26:53] I don't even know how many videos will be on YouTube now. I can't even imagine.

Sugu: [26:59] The reason why it works is that most videos are inactive, right? You just insert a row and move on, and so it is more a question of what is a working set, right? How many of those videos are popular? They come and live-in memory and we had a Memcached layer also in the front, and so it was more a question of how many popular videos people are trying to hit and can all that fit in memory?

Utsav:

[27:31] Okay, that makes a lot of sense. Then you could put a caching layer in front so that you don't hit the database for all of these popular videos. That's so interesting. And, given all of this, it makes sense to me that you can shard this because videos, you can totally imagine, it's relatively easy to shard, it sounds like. But yeah, what do you think about resharding? That seems like, how do you solve recharging? That sounds like a very interesting problem. And maybe just to add some color to what I think resharding means, it's like you have a data on like my SQL box A and you want to move some of it to my SQL box B or like split from one database to two and two to four. And you have to somehow live transfer data from the database, like, how do you solve that?

Sugu: [28:19] So, there's the principle and there is a mechanic, right? Typically, the mechanic is actually more, kind of straightforward, which is typically when you start sharding, the first thing you do is actually pull tables out. Like if there are 300 tables, you'll say, you know, these 20 tables can live in a separate database. It's not really called sharding or resharding cause you're not sharding, you're splitting. So, the first thing people usually do is split a database into smaller parts because then each one can grow on its own. The sharding aspect comes when a few sets of tables actually themselves become so big that they cannot fit in a database. So that's when you shard and when you make the sharding decision, what you do is, the tables that are related, that are together, you shard them the same way.

Like in YouTube's case, when we did the sharding, we kept the users and their videos together in the same shard. So that was actually a conscious decision that users and videos stay together. There is actually a sharding model that allows you to represent that as a relationship. Now that we test actually exposes, which is basically there's some theory crafting behind there. But from a common-sense perspective, you can always use about 9 users and their videos, being together. So that's the mechanic part.

On the principal part actually, the way you think about sharding is basically beyond the scale, you have to actually rethink your application as being fully distributed with independent operators working by themselves, working on their own. That means that some features you often have to forego for the sake of scalability. In other words, I mean, like crazy features, like what is the real time value of the total number of videos in the system? I want that up to the millisecond. That type of question is unreasonable for a sharding system. You could probably do a select count star, against a small database and can actually do it. But if you are having tens of thousands of servers answering that question becomes harder, right? And then at that time we are to start making trade-offs like, is it really important that you have to know it up to the, like, what if it is up to the minute? Is that good enough?

So those kinds of traders here make at a high level and most people do because you cannot go to that scale unless you make these tradeoffs. But once you make these trade-offs, sharding becomes actually very natural, because the one thing that you make, for example, if you are going to have the entire world of your customer, you are going to have, you know, 7 billion rows in a user account. And sometimes many of them create multiple accounts. So, you are going to have billions of rows. You cannot build features that put them all together. You have to build features that keep them separate. And as soon as your features follow that pattern, then sharding also kind of drives that same decision. So, it’s kind of more or less becomes natural.

Utsav:

[32:21] Okay, that makes sense to me, and it seems like you said that you have a sharding solution that predated the tests. So maybe you can talk about the motivation behind the test, like, and maybe the year that this was in like, well, if you have any numbers on, you know, at what point did you seem [32:36]

Sugu: [32:36] [32:37] I'm finding myself surprised that there are details I should know that I don't know, like the number of videos we had when we were sharding, right? It was 2007, 2007 is, actually wow,15,14 years ago. So, yeah it has been many years. So, in 2007 was I think when we did our first sharding, if I remember correctly. Maybe 2008? Around 2007 or 2008, one of those few years was the first time when we did sharding. But the test was not born because we needed to do sharding, obviously, because it was already sharded it was born because we couldn't keep up with the number of outages that the database was causing. That was actually why it was born. It was quite obvious that as the system was scaling, in spite of being sharded, there were a large number of things broken within the system that needed fixing for it to work.

And the system overall, when I say system meets end to end the entire system, and that includes the engineers and their development process. So that's how big the problem was, but we didn't solve the entire problem. We said from a database perspective, what can we do to improve things, right? So that's kind of the scope of the problem. And more specifically Mike actually, took off, and I think, spent some time in, Dana Street coffee at Mountain View and came up with the spreadsheet, where he described, he kind of listed every outage that we have had and what solution and how did we solve it? Is there, what would be the best remedy to avoid such an outage? So, when we actually then sat down and studied that spreadsheet, it was obvious that a new proxy layer at any build and that is how we test was born.

And the whole idea is to protect the database from the developer. For example, they were like, at that time 200 developers, if you look at it right, the developers, don't intentionally write [35:10] and a developer doesn't write a bad query all the time. But if a developer wrote a bad query, only once a year with 200 developers, 50 weeks a year, you do the math. How often do we see outages in the database? Almost every day. That's what it amounted to; we were seeing outages almost every day. And very often they were actually put 'quote and quote', bad queries coming from that one user role that they will never repeat, but they've fulfilled their quota for the year, you know? And so, what we did was we found common patterns where that would cause data.

So, the big change that we made was if I wrote a bad query, I have to pay the price for it, not the entire team, right? Now today, if I wrote a bad query, that query runs on my SQL, it takes on the entire database. So, we wrote with tests in such a way that if you wrote a bad query, your query would fail. So, in other words, what we used to look at is how long is the query running? If the query is running too long, we kill it. If the query is fetching too many roles, we return an error. So those kinds of defenses we added in with tests early on and that pretty much took it a long way. The other big feature that we deployed was connection pooling, which my SQL was very bad at. The new SQL is slightly better. It's still not as good. So, the connection pooling feature was a lifesaver for them too.

Utsav:

[37:00] That makes a lot of sense to me. And maybe you can tell us a little bit about why my SQL didn't have the capability of defending itself. So, you might imagine like, you know, just from somebody who has no my SQL experience, they can just be like a [inaudible 37:16].

Sugu: [37:17] It's very simple. The reason is because my SQL has, what do you call, claimed itself to be a relational database. And the relational database is required, you know, to not fail. So, we at YouTube could be opinionated about what query should work and what query should not work. That freedom my SQL doesn't have. Every query that is given to it, it has to return the results, right? So that's the rule that you had to, to be qualified as a relational database, you had to fulfill that pool and that pool was its curse.

Utsav:

[38:01] Okay, but you could configure maybe like a session timer or something, or like a query timer, or would that lose deference which is not good enough basically.

Sugu: [38:09] They actually added those features much later, but they didn't. Yeah. The newer, my SQLs now do have those features, but I don't know how- but they are all behind. They are not as usable because you have to set it in a query parameter, you have to set it in your session, you know, so it's not as, like by default you just install my SQL, and start running it. You won't have this property, you have to configure it, which is another hurdle to cross. Whereas in Vitess you installed the test, you'll start failing, you know, people will complain like, we see the opposite problem, right? Oh, [38:53] My SQL, why is the test failing it? Because you are trying to pitch a million rows. If you want to do 10 million rows, you have to tell the tests that you want to get 10 million rows and then it will give you them. But if you just send a query, it won't give you a million rows.

Utsav:

[39:10] Yeah. What do you think was like the one really important? Which as you said, there's connection pooling, it's also limited, bad where he's like, how did it figure out that a query was not one, maybe heuristic is it's turning too many rows and it would just fail fast. What were some other key problems?

Sugu: [39:27] Oh, there were, we added so many, the coolest one was, when engineers wrote code, they are, like me too, I'm guilty of the same thing. You think how many videos could a user possibly upload? And we were thinking manually in the [39:51]. For me to upload a video, I have to go make it, edit it and then upload it. You know, we will have, you know, 2000 videos where we'll be a number and, you know, look at how old YouTube was. It was two years old. How many videos can you produce in two years? So, 2000 videos. So, selecting all videos of users, not a problem, right? So, we never put a limit on how many videos we fetched. Then we selected a video by using, and then there was one user who ran a bot. I think there's no other way. Well now there are accounts that have billions of videos, but at that time that user had 250,000 videos.

And that wasn't a problem per se, because that I couldn't have been around, but it became a problem when that account got high, got listed on a YouTube stream page. Yeah. The index page, right. The landing page. So, which means that every person that went to youtube.com, you should a query that pool $250,000, because there was, so the, one of the changes we made was that, any query that, that's why we added the limit, plus like, where you say, if you send an aquari with no limit class, we will put a limit. And if that limit is exceeded, we'll just return your arrow thing that you're trying to fish to NATO. So, there was one protection, but the cooler protection that we did was, there are some queries that are just inherently expensive, right? Like this one, if this, I think in this case, I think it might've been a select count star.

If it's a select count, it doesn't fish to 50,000 rows, but it scans to 50,000 rows. So, the, the feature that we did was if a query got spam, like because of the fact that it's coming from the front page, which means that my school is receiving like a very high QPS at the same period in that case, what we do is we check if the query is already learning from some other request, if it is, if there are 10 other requests of the same query, they are not sent to my SQL. We wait for them to return and return that same dessert to all those requests.

Utsav:

[42:21] Okay. Would that be problematic from a data consistency perspective? Probably not. Yeah,

Sugu: [42:26] It is actually, yes, it is but it is no worse than eventual consistency. Right. So, if you are reading, if you are reading from a replica, no problem. Right. If you are doing something transactional. So, the rule that we had at YouTube was if you're doing something transactional, so there's also another reason like the rule is if you read a rope, asse that rule is already stayed. Why? Because as soon as you read the rule, somebody could have gone and changed. So that stainless guarantee kind of carried over into this will also is that when you do a select, when you do a select of the query, it may be slightly outdated because one thing is you could be reading from the previous period that is running, or sometimes it could be going to a replica.

So it wasn't, it was something that nobody even noticed that we had made this change, but it's like, we've never had an outage later to, you know, very spam after we did that feature. Yeah. But more and more, more importantly, the rule that we followed was if you want to make sure that the road doesn't change after you read it, you have to select for update. So, if you plan to update a row and update based on the value that you have read, you have to select for update. So that's a rule that all engineers followed.

Utsav:

[43:56] Yeah. It makes sense. And it's better to just be explicit about something like that.

Sugu: [44:00] Yeah. Yeah. Because of the fact that the, the, my SQL NVCC model tells you, right. The MVCC, the consistently it's called consistently, that's actually the actual practical Lim is something else. but that model basically tells you that the role that my simple serves to you is technically obsolete. 

Utsav:

[44:24] Okay, and then it just makes sense to me. And you’ll open source with us. I remember hearing the reason is you never wanted to write another way to test. So, you decided to open source it from day one. Can you maybe tell me a little bit about, you know, the motivations behind making it a product? So, it's a great piece of software, which I used to buy YouTube from my understanding for like a really, really long time. At what point did you decide, you know, we should actually make this available for other companies to use. And what is the transformation of the liquid test for PlanetScale? 

Sugu: [44:58] So even before PlanetScale, even as early as 2014, 2015, we felt that we should start promoting the test outside of YouTube. And one of the reasons was, it wasn't a very serious thing, but it was kind of a secondary motivation. We felt that even for YouTube, for the test to be credible, even within YouTube, this product needs to be used by other people beyond YouTube. In other words, you know, for a company to trust a product, a lot of other people must be using it. That was kind of the motivation that made us think that we should encourage other companies to also use, [45:46]. And actually, Flip card was the first one. I think it was 2015 or 16 that they approached us and they said, hey, it looks like this will perfectly solve the problem that we are trying to solve, and they were the first adopters of the test. 

Utsav: [46:05] Okay, so that makes sense. And that's like it was open source and you felt that other companies should use it 2015, it sounded like Flip card was interested, but then what is the transformation of from then to, you know, let's make it a product that people can buy because this it's an open-source technology that, how do you go from that to product?

Sugu: [46:28] Yeah. So, I can talk about how PlanetScale came about, right? So, the other thing that started happening over time as the test started evolving as a project companies like Flip card started relying on it, Slack started using it, Square started using it. And they were all scared of using the test. Why, because this, this is basically defining technology. It starts something, it's basically a lifetime commitment type of change. And the question in their mind is what, like, what's the guarantee that, like what YouTube is interested in making sure that, you know, it will continue to work for slack, right? Why, like they were, a large number of companies were worried about that part, where, you know, you do focus his videos by like, how can we trust the technology?

How can we trust the future of our company, right on technology? What if you abandon the project and that kind of stuff? So, overtime, I could sense that hesitancy growing among people. And this was one of the contributing factors to us starting planet scale, where we say, we are going to start a company at a company that is going to make a commitment, you know, to stand behind this open-source project. So, when we made that announcement that we are starting the company, there was a sigh of relief throughout the test community. You know, finally, I can-

Utsav: [48:07] -depend on this really important.

Sugu: [48:08] I can depend on this project. It's not like it is here to stay. There is a company who is going to make sure that this project stays healthy.

Utsav: [48:18] That makes sense. Yeah.

Sugu: [48:21] There were other factors also, but this was one of the, definitely one of the major contributing factors towards making. So, I think at the end of the day, this is something, this is generally a problem in open source. You know, that, you say open source is free, but there is an economy behind it because the engineer's time is not free. And for me to spend time on an open source, I have to be paid money because, you know, I have my family to take care of, you know, I need money to live. And so, I'm expecting engineers to contribute to open source in their free time. it can work for smaller projects, you know, but once the project becomes beyond certain sites, it is not a part-time, it cannot be a part-time contribution. It is a full-time contribution and the engineer has to be paid money for it. And so there has to be some kind of economy behind that.

Utsav: [49:32] That makes a lot of sense. And I think it also tells me that the problem that slack and square and these other companies are facing was just so large that they were, that they wanted to use the project, given all of these issues with, you know, we don't know the future of this project, but there was no other solution for them given that existing, my SQL stack and their hyper-growth and the kind of problems that they'd be dealing with.

Sugu: [49:59] Yeah, that's right. So that is something I still wonder about, right. If, if not retested, what could have, is that an ultimate, you know, solution, the alternative doesn't look as good either because the alternative would be a closed source product. Like, you know, that scares you, that even more scary, because what if the company COVID out of business, right. At least, in the case of Vitess the source code is there. If planet scale goes away, slack can employ engineers to take over that source code and continue to operate so that confidence is there. So that is one big advantage of something being open source, gives higher confidence that, you know, in the worst-case scenario, they have a way to move forward.

Utsav: [50:53] And when you, when you finally decided to make a company and a product, like, did you, what are some things you've learned along the way that, you know, you need to have in with tests for more people to be able to use it at a first impression to me, it's like, there are not millions of companies running into these scale issues, but like, what are some interesting things that you've learned along the way? Yeah.

Sugu: [51:14] All I can say is I am still learning. Yeah, every time I come there, I realize, oh, you know what I thought was important. It's not really that important. like somebody asked me, if you are agonizing so much about data integrity, why is Mongo DB so popular? Right? I mean, Mongo DB is pretty solid now, as far as I can tell, but it was not solid when it became popular. Right. They made it solid after it became popular. So, one good thing. If he doesn't, did not, for the longest time did not care about data integrity, but people still flock to it. So, there is always a trade- off in what is important. And you have to find out what that is and you're to basically meet people where they are.

And in the case of the test, it’s actually usability, approachability, and usability. That is a bigger hurdle to cross before you cannot up your tests. And that is a big, much bigger hurdle than, you know, the data integrity guarantees that have people looking for. So those are for example, one lesson, where you have to, so the one thing we test is we used to listen to, we used to, we have our ears open. You're constantly listening to people, giving us feedback about what is not working for them and fixing those problems. It turns out that is insufficient. Who are we not listening to? We are not listening to the person, the quiet person that drives over the weekend, doesn't work for them and quietly walks away. Right? That's that voice we never heard. We only heard the voice of somebody who tried to use the test and like went past the problem of not being able to use it, got to use it, found a missing feature and is asking for it. So, we've been focusing on very specific features about how to make the test, but we completely forgot about, you know, how to make it more approachable. Right. So those are the problems we are now solving at PlanetScale. Okay.

Utsav: [53:41] That makes a lot of sense. Yeah. Maybe you can tell us about one usability feature that you feel was extremely important that your live build.

Sugu: [53:49] The biggest one for example, is a schema deployment. Subconsciously every developer will cringe. When you say, use a database and manage your schema, use a database, they would like, but as soon as you say schema, almost every developer cringes, because, oh my God, I have to deal with that. Bureaucracy, like send it to for approval. DBA is going to detect it. our schema is going to break something. Some app is going to break all those headaches I have to deal with. So, what we have done at planet scale is give you an experience very similar to what you would do as a developer with your source code. You do the same thing with your database. You can take your database ratchet, applied, schema changes, tested, and then have a workflow to much it back into production and very natural. And it looks almost similar to how you develop your source code, right? So, it's a very natural workflow and it applies to the database and you don't have to deal with, you know, that whole bureaucracy, it's all part of this nice, nice workflow. It handles conflicts. If multiple people want to change the schema at the same time, it pipelines them the correct way. And if they cannot go together, we will flag it for you. So those are really, really nice feature. And that seems to be really going well with the developers. 

Utsav: [55:28] Interesting. Yeah. And I think it speaks to, you know, why databases like Mongo DBS are popular?

Sugu: [55:33] Yeah. No schema is not a no SQL that made Mongo DBS when it's the no schema part,

Utsav: [55:43] Being able to just tweak, add one more field, not be blocked on like a team to make sure not exactly worry about back-filling and all of that. It is a game changer and maybe a little ashamed to admit actually, like I use Mongo DB at this job and it's not half bad. It's pretty good.

Sugu: [56:00] Yeah. Yeah. Now they are trying to bring a skill back to Mongo DB and people like it. Right. So, the real problem was actually schemas, the fact that you can't just add something and move on. It's so hard to do it in a database. 

Utsav: [56:21] Yeah, and maybe you can tell me today, like, what's the difference between VTS open source and PlanetScale? Like what is the extra stuff?

Sugu: [56:29] They are very orthogonal, right? Because what we are building in PlanetScale is a beautiful developer experience. And, what we test is giving you is the other part that, most people that have a good developer experience MIS, which is, a solid, scalable database in the backend, like we could build this developer experience on top of stock, my SQL, but people are hesitating to adopt that because they know that it won't scale. At some point of time, I'm going to hit a limit and I'm going to spend the 90% of management energy trying to figure out how to scale this. That's what companies experience when that problem is taken away from you. Right. You get a good developer experience, but when it comes to scaling with us, have you covered?

Utsav: [57:23] Yeah. That makes a lot of sense. And does that mean that as a developer, let's say I want to start a startup, the model. I have a little bit of experience with my SQL. Can I just start with PlanetScale on day one?

Sugu: [57:35] What would I get? That's exactly what [inaudible]. Yeah. You basically spend almost no time configuring your database. You start, you said, just go to planets, click, like you click a button, you instantly get a database and then you just start developing on it. So, zero bad years, which is what you want in a startup.

Utsav: [58:02] And, and the experience I get is just stock my sequel to begin with, except for it would have all of these things like automatic limits on queries. And it would let me shard as soon as I become too big,

Sugu: [58:15] As soon as yeah, exactly. But interesting. Yeah. So basically, what they have to do is make sure that, you know, kind of guide you towards good programming practices. You know what I mean? If you're just running unbounded queries, it's going to tell you, this is not good. So had bounced your [inaudible 58:36], that kind of stuff. So, we can be opinionated that way.

Utsav: [56:37] So I'm curious about, you know, all of the technical innovation that I see your Twitter feed and YouTube talk about, you know, you're a handler, I'll have like an automatic benchmarking tool. You're like optimizing, protocol handling and letting go. What are some of the keys, like other innovations that you all are doing at that PlanetScale?

Sugu: [58:56] So there is one, I don't know if you've seen the blog series about consensus, generalized consensus. Okay. So, I think that's a decent idea. I feel like it's a decent innovation. What we are doing is there's Paxus raft. You know, those in my eyes are rigid algorithms and they have rigid assumptions about how nodes should talk to each other. And those algorithms were opted for industry best practice. But if you flip it around, what problems are they solving, right? You identify the problem. And then you start with the problem first and come in and say, this is the problem I want to solve. What system would you build? I don't think a raft or Paxus would be the systems we would have been. Right. And what problems are we trying to solve?

We are trying to solve the problem of durability, right? I have a distributed system and I say, save this data. And the system says, I have saved your data. And the guarantee that I want is that the system doesn't lose my data, essentially. That is the problem that all these consensus systems solve. But then when I come in here, I'm in a cloud environment, I'm on AWS. I have the ones that have regions, right. I can say for me, durability means my data is across two Lords. Or my, my notion of UWP is data is, at least in one other zone or one of the regions. Right? So, if you specify a system that can make sure that whenever you ask, ask people for the right data, it makes sure that the data has reached all those nodes or regions. And then it gives you, and then later, if there are failures, it recovers based on the fact that the data is elsewhere, right? So that is what the thing says. So, if you look at it, top-down, you come up with a very different approach to the problem where, in which case raft and Paxus are just one way to solve it., so what I have explained is what is the generic approach behind solving the problem of durability, how Paxus and raft solve it, but how we can build other systems that are more effective at more accurately meeting the driving requirements that are coming from the business.

Utsav: [1:01:39] This reminds me of semi sync replication and like my SQL.

Sugu: [1:01:44] Exactly. So, you could build a consensus system using my SQL semi application that gives you the same guarantees that raft gives you, for example.

Utsav: [1:01:55] Okay.

Sugu: [1:01:56] But then the theory behind this is kind of what I've explained in the blog feed. So that I think is one great innovation. And the benchmarking is another, is another good one, that may be a PhD that can make them out of that. Or at least a MTech thesis. I don't know what he's going to do, but float on is he's actually a student who has done an awesome job with them. So, some research work is going to come out of what you've done. Yeah.

Utsav: [1:02:28] That's awesome to hear, because you don't generally hear of smaller companies and like academy work coming out of them, but plan skills, exactly. The kind of place where something like that would come out.

Sugu: [1:02:40] Yeah. There's a lot more there. The other cool one is, the V replication or the materialization, feature, which is to be able to materialize anything into anything., and, it works so well within the model of, the Tessa sharding scheme, that you can actually, run, that is a foundation for a large number of migrations. It's a foundation for applying. DDLs like today in planet scale, you say, deploy this video, we do it in an offline fashion as in like with no downtime and the safeties, you can even rework it. Right. So those, so the way I see it is a DDL is a form of migration. And it's just one way of expressing a migration as a combined, but there are bigger migrations, very moving a table from one database to another where you Rashad a table. These are all DDLs of some sort. And if you can express them as videos, and if the system can do it offline in the background with no downtime and with reversibility, I think that is a very powerful mechanism to make available for you to use this. Like me, often some of these projects take months, years. Yeah, you can now do them in the test with just a simple command. 

Utsav: [1:04:16] Yeah, this seems like the kind of things that are just completely unheard of or impossible in any kind of other data system that you have.

Sugu: [1:04:22] Yeah, near stone because it's science fiction, this is science fiction. 

Utsav: [1:04:31] And I also let that. That test can also be deployed within Kubernetes. So, you can get all of the benefits of, you know, like the same way as you get benefits of Kubernetes with stateless services that things can just move around. You can also use that to your benefit with tasks where your data can be moved around and stored in multiple places. Am I completely misremembering stuff for like, [cross-talking 1:04:59]?

Sugu: [1:05:00] No, it's not like I can reveal to you that planet scale, the tests are all hosted in Kubernetes or everything runs in Kubernetes. And, we have a very large number of clusters. We may already be, or, may soon be, you know, the largest number of key spaces in the world. All in Kubernetes, the entire thing is in Kubernetes.

Utsav: [1:95:29] So, that's like at least traditionally, so do you run on containers or is it just like, yeah, that's a tradition, yeah. If people say that you shouldn't run stateful stuff in containers and clearly bucking that trend. So maybe you can just talk through that a little bit.

Sugu: [1:05:47] Yeah, totally. So that's, so the, the, the history goes behind the history behind this goes all the way back to, how we test evolve him YouTube and Google, right. On YouTube, originally, we wrote with testing our own dedicated hardware, but we were required to migrate it to Google's work, which is Google's cloud. And, Vitess had to, and the problem was that my sequel was not a first-class citizen in Google sport. Which means that you couldn't use it as a service. There was one team we actually collaborated with the team, but it was still not a first-class citizen. So, the way we did this was we actually landed with tests as, because, so the only way we could run with tests inside Google work was as a stateless application.

So, we actually added the durability, survivability of nodes going down in, and Vitess such that anytime, the cloud can come and take down your server and we test should handle that. So that wasn't with Tessa's DNA that we had to build, and make it part of the test. So then later when Kubernetes came out, you had already for it because it was already running in what I call a data hostile environment., and therefore, like, because of the fact that we land at a massive scale inside Google board, we were able to, we have the confidence that it will run well. And he has shown them doesn't run well so far.

Utsav: [1:07:34] Okay. Does that mean that the, my sequels basically live outside the Kubernetes and then you have a large part of like all of the tests it's running?

Sugu: [1:07:39] Inside the container. 

Utsav: [1:07:40] Okay, Interesting. 

Sugu: [1:07:43] Yeah, it's my read on my sequels inside parts. We are not the first ones to do it by the way jd.com, HubSpot, and, there's another company. I forgot their name. They've been doing it for many years.

Utsav: [1:08:03] Okay, and what are the benefits of running it inside a pot? I guess your perspective,

Sugu: [1:08:06] Ah, manageability, right? Because it's a uniform environment. You don't treat tests as, you know, a special case like today, if you run a Kubernetes, that's the Kubernetes setup where you run your application and then you have to manage the database using different tools and technologies and methodologies. If you're running everything in Kubernetes, then you have a uniform set of management tools that work the same way. You want to add something, just push the ambles.

Utsav: [1:08:40] That makes sense. And then you have to manage draining and basically hydrating nodes or pods. And you, do you have a custom logic to set that up?

Sugu: [1:08:49] There is actually an operator that actually manages these things. We are actually trying to simplify that, you know, because we test, does have a lot of things for that, that we built when we were at war. I feel like at least our planet scale operator does a lot more than it should. It doesn't have to but yeah. There's some, a little bit of blue code is needed. I would say

Utsav: [1:09:18] That makes sense. Yeah. But it certainly sounds like science fiction reading all of your data running on these pods that can be moved around and changed. Anytime you don't even know which machine has all of your data, it sounds in the future.

Sugu: [1:09:34] Yeah, it sounds like the future, but it is the future because there is no other way you can manage 10,000 nodes, right? Like if you're, if you're going from the unicorn world where there is this one server that you have to watch carefully to, it's going to be everywhere. It's going to be lots and lots of servers. How do you scale it? So, there's a term that one of our engineers used to use, you want off one manageability, which means that, as you scale the nodes, you shouldn't have to scale your team.

Utsav: [1:10:12] That makes a lot of sense. Yeah. And what does your day-to-day job look like now? Like, what is your role or how has it changed over time? I'm guessing initially you must have already got it

Sugu: [1:10:21] Every time. Yeah. Like, it keeps changing. So, like until, initially I spent like 99% of my time on with deaths, like the early days of planet scale, the last seven, eight months, I spent a lot of time on our internal operator because there was one particular part that was broken that I said, okay, I'll go fix it. So that I'm actually winding down. So, I have to, so I, like, I have a feeling ... eventually what will happen is I think I will have to spend a lot more time publishing stuff. I have not been publishing enough. Like people have been asking me, when are you going to finish this consensus series? I still haven't had time to do it. So probably spend more time doing that, and probably also speak at conferences. I scaled that back a bit because I was too busy, not writing code internally, so it keeps changing. So, it will remain like that over because it's the way I see it is whatever is needed. It's like PlanetScale, right. Whatever is more important. That's the problem I am working on.

Utsav: [1:11:44] Yep. That makes sense. And in some ways, it's like going back to your role at PayPal, they're just working on the important projects.

Sugu: [1:11:52] Yeah, yeah, yeah. Keep it afloat.

Utsav: [1:11:59] Yeah. And maybe to wrap things up a little bit, like I have one question for you, which is, have you seen the shape of your customers change over time? Like, you know, I'm, I'm sure initially with, with testers, like super large customers, but have you started seeing now with all of these usability improvements, smaller and smaller teams and companies like, you know, what we just want to future proof ourselves.

Sugu: [1:12:18] Yeah, totally. It's night and day. That type of like, you know, some of like, even the words they use are very different. Like things like, you know, previous customers are more like very correctness, durability, scalability, and now like the newer customers talk about, you know, these two factors are not working, you know, I have this Ruby program. I have this rails thing. Even the language is very different. Yeah. It does change.

Utsav: [1:12:54] Well, I think that that's a great sign though. Cause you're basically attracting a lot of people who traditionally would have never even thought about using something like with ESSA plan scale. So that's a great thing

Sugu: [1:13:04] To hear. Yeah. Yeah. Yeah.

Utsav: [1:13:08] Anyways, well, thanks so much for being a guest. I think this was, yeah, this was a lot of fun for me. I love that trip down memory lane and understanding, knowing how the product is supposed to work and yeah. Again, thank you so much. 

Sugu: [1:13:12] Yeah. Thank you.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Tammy Butow is a Principal SRE at Gremlin, an enterprise Chaos Engineering platform that makes it easy to build more reliable applications in order to prevent outages, innovate faster, and earn customer trust. She’s also the co-founder of Girl Geek Academy, an organization to encourage women to learn technology skills. She previously held IC and management roles in SRE at Dropbox and Digital Ocean.

Apple Podcasts | Spotify | Google Podcasts

In this episode, we talk about reliability engineering and Chaos Engineering. We talk about the growing trend of outages across the internet and their underlying reasons. We explore common themes in outages, like marketing events and lack of budgets/planning, the impact of such outages on businesses like online retailers, and how tools and methodologies from Chaos Engineering and SRE can help.

Highlights

01:00 - Starting as the seventh employee at Gremlin

04:00 - An analysis of recent outages and their root causes.

09:00 - A mindset shift on software reliability

14:00 - If you’re suddenly in charge of the reliability of thousands of MySQL databases, what do you do? How do you measure your own success?

25:00 - Why is it important to know exactly how many nodes your service requires to run reliably?

30:00 - What attracts customers to Chaos Engineering? Do prospects get concerned when they hear "chaos” or “failure as a service”?

43:00 - Regression testing failure in CI/CD

51:00 - Trends of interest in Chaos Engineering over time.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Itiel Schwartz is the CTO and Co-Founder of Komodor, a Kubernetes troubleshooting platform. Previously, he was one of the founding engineers at Rookout.

Apple Podcasts | Spotify | Google Podcasts

We discuss two major themes in this episode: the rise of Kubernetes as a popular orchestration platform, and the need for using an integrated service to understand and debug Kubernetes deployments.

Highlights

9:30 - When should a startup consider using a more heavy-duty system like Kubernetes, vs. managed platforms like AWS Fargate? What are the advantages of using Kubernetes over these platforms?

18:00 - What are the new developments in the Kubernetes world? Why it may make sense in the future to run stateful services like databases on Kubernetes. Open Policy Agent

25:00 - The motivation behind starting a Kubernetes focused company

28:00 - What’s the biggest gap that Kubernetes users face while debugging their deployments? And how does Komodor help with that?

39:00 - The surprising rise of Observability teams across smaller companies

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Tramale Turner is the Head of Engineering, Traffic at Stripe. Previously, he was a Senior Engineering Manager at F5 Networks and a Senior Manager at Nintendo.

Apple Podcasts | Spotify | Google Podcasts

This episode has an unexpectedly deep dive into security and compliance at Stripe. We discuss Stripe’s philosophy and approach towards building secure systems, achieving compliance standards like PCI, and complex requirements like data locality laws.

Highlights

05:00 - Growth at Stripe

09:00 - A sampling of challenges involved in being a payments provider

11:00 - Stripe API traffic is much lower than the traditional large companies with Traffic teams like Google/Facebook/Netflix. Why does Stripe need a Traffic team/group?

16:00 - Stripe’s innovative approach with an embargo for credit card numbers from most of their platform. Idempotency keys.

20:00 - Compliance automation at Stripe!

30:00 - Should the entire organization need to know and care about compliance? Or should teams provide internal platforms to abstract away compliance concerns?

36:00 - Data Governance and locality laws.

45:00 - Security’s relationship with Compliance, and how Stripe thinks about security

53:00 - How to build teams that need to achieve such lofty goals?

Transcript

(Best effort. Find my contact info at /about to report any errors).

Utsav Shah: Hey, Tramale, welcome to another episode of the Software at Scale podcast and thank you for joining me. If you could tell listeners your background and, first of all, the origin of your name, which I think is extremely interesting. 

Tramale Turner: Yes, sure. So first of all, thank you for having me. My name is Tramale Turner. And the origin of my name, I think, unless my father was lying to me, is as follows. When I was born, he had this image of the Magi in his head, the three wise men as it's colloquially known. And he thought, “Okay, three men, tri-male.” Didn't like the “I” and so he changed the “I” to an “A”, and I thus became Tramale.

Utsav Shah: Cool, very cool. And so a lot of your experience has been in the networking and traffic space. Right now, you're the Head of Engineering of Traffic at Stripe. And previously, you worked at F5 Networks. So what got you interested in this space? If you could just tell us a little bit about your story? What got you into this space? And what do you think of it? Clearly, you like it?

Tramale Turner: Well, it's interesting. In fact, I hadn't been in specifically this functional area for the majority of my career, but it has been my focus for the last, let's say, five years. I started off as a software engineer. I was a software engineering student at the University of Pennsylvania. I left Penn after my third year, moved to Japan, and worked as a software engineer in Tokyo, working on what I think we now call digital marketing and or web development, or sometimes interactive media, but didn't really have a name at that point. And I was designing software for interactive CDs, for websites and what have you. I ultimately left Japan, came back to the United States, was in the Bay for a little bit, commuting between San Francisco and Tokyo, then moved back home to start a company founded by myself and with investment from the folks that I worked for in Japan. 

So I started a series S corp in Michigan and built this product called [Inaudible 02:32], and [Inaudible 02:33] in Japanese just means the Festival of the gods. [Inaudible 02:37] was, for all intents purposes, another social networking service. But this was the age of GeoCities and Yahoo, and social networking services at scale was not really a thing, so to speak. And so our idea, which we thought was novel, was to build an interactive, full media experience where you could have communities and create communities. My co-workers, the people I hired, and the people who, in fact, invested all met on IRC. And so we have this image of people just being able to have real-time experiences and real-time chat in our mind as we were building this. And it was a pretty fun experience as you might imagine running a startup and being sort of the lead technical person. I was way too young to be a CTO of any shape, but I effectively was that. And we showed that off at Macworld and got an investor interested in actually buying the IP. That person bought out the company. I made a little bit of money, bought a house, then went through a couple of failed startups also, in this domain of interactive media, interactive marketing. Ultimately ended up at the Volkswagen Group. Spent nine years at the Volkswagen Group working initially in marketing, strangely enough, working for the Volkswagen brand, and then transitioning more into technology. Traveled all around the world for Volkswagen, mostly all around Germany, and then Latin America. Moved to Puebla, Mexico for a year and then to Herndon, Virginia for almost two years before I left Volkswagen and join Nissan again at an advertising agency captive inside of Nissan and Franklin, Tennessee. 

I lived in Nashville, commuted to Franklin, and worked on the Infiniti brand. Did that for six whole months before I got a ping from this little video game company in the Pacific Northwest called Nintendo, your listeners probably have never heard of that. Went to Nintendo working as an engineering manager for this consumer online and publishing team as it became known, working on payment services, account services, and developer support. So if you've ever used a Wii U, 3DS, or a Nintendo Switch, and you've had an NNID or Nintendo Network identifier, that was the service that my team built, and that I helped create, and also responsible for payments within the e-shop, and so on. 

So left Nintendo, joined F5, spent 13 pretty fun months at F5 building a brand new set of teams therein when I got a DM in my LinkedIn from the Seattle Site Lead for Stripe who said I should come over for lunch. That was two and a half years ago. I went over on a Taco Tuesday, and I never left and have been a part of the Traffic team and became the leader of the Traffic team for my entire tenure at Stripe, two and a half years now.

Utsav Shah: Okay, how big was Stripe? I have, first of all, so many questions, especially around your experience with Nintendo. But I'm going to ask you how big was Stripe when you join it? And how big do you think it is now, just approximately?

Tramale Turner: Yeah, when I joined Stripe, it was approximately 1000 people. And it is approximately four times the size of that now and growing so much. 

Utsav Shah: You’ve grown so much over the last few years.

Tramale Turner: Yeah. Yeah, especially the last 18 months, I would say, we've experienced phenomenal growth. I can't say specifically how many people are there now but as I said, about four times an increase, but still larger than that and growing rapidly.

Utsav Shah: Did your experience at Nintendo, especially working on the payments platform and stuff make you I guess recognize how important the problem is? Or how hard and complex the problem that Stripe is solving? Or is it just like, a combination of things?

Tramale Turner: I love this question. So when I got that invitation to lunch, I of course knew what Stripe was. I follow patio11 on Twitter and had seen his posts and Patrick Olsen's posts on Hacker News, but I pretty much in my mind just saw Stripe as a payment services provider or what we sort of colloquially call a PSP. They have a payment gateway, they connect you to credit card acquirers, and no big deal, you can accept credit card payments online. And I had dealt with Chase payment tech. intimately while at Nintendo and so yeah, I was very familiar with how the structure of those agreements worked and both the utility and some of the failure modes and fault domains that exist when dealing with a payment gateway. And that's what I walked through the door thinking that I was going to experience. Someone was going to talk to me about joining this thing that's going to, you know, in their words, I'm sure, change everything or transform it. But what I found out was something completely different, and it's super interesting. 

The person that I was having lunch with that day, a gentleman named Brian Delahunty, was talking to me about Lyft, who is a user of Stripe, and talking me through all of the different use cases. And I won't belabor the point all of the various products and services that Lyft makes use of, but as he was talking, a light went on, and I started to see, “Oh, it's not just a payment services provider. This is a company that's trying to democratize access to economic enablement. And not trying to do it just for a certain segment of the population, like startups or developers or large enterprises. It's for everyone, like literally anyone that has access to a connection and can get access to the API,” which is, again, if you have a computer or a phone and have connectivity, you can do that. You can then create a business. And what really occurred to me in that moment was, you could build something that could support putting food on your table every night, without a whole lot of effort. And when I got that concept, when it started to make sense to me, I said to myself, I couldn't imagine not being a part of it.

Utsav Shah: Wow, that's a great story. And I think there's also a lot of complexity there in something as simple as a Lyft transaction because the person driving the Lyft might be from some other country, the company Lyft is incorporated in some other country, and all of the rules and regulations surrounding all of that, making sure you have enough capital, it just sounds like such an interesting technical problem, plus, you're helping the world. 

Tramale Turner: Yeah, I mean, to your point, right. So if we can, because it's important, let's just quickly break it down. You as a platform provider, who has perhaps these 1099 or whatever the regulation is within your country, independent contractors who are driving on your behalf, you want to be able to accept funds somehow, right? So you need the scaffolding so that people can pay you money, right? And you can get some remunerative benefit from the service that you're providing. You want to be able to identify those people, to your point, right? You don't know who that person is when they say they want to be a Lyft, driver, an Uber driver, or Gojek driver, or whomever. And so that identity question is super important as well. You want to be able to manage the platform, right? So being able to do the core service that you enable, in Lyft’s case, being able to enable carshare. But with that rideshare, the core primitives for enabling rideshare, you also have to worry about the saving up movement of money. And oh, if you're operating in different countries, what about foreign exchange, right? And then you have to pay those people. And you want to make sure that you have enough money in whatever account that you're managing to do that. And those people who get paid, maybe they want to do that instantly, maybe you have a payment method by which they can have whatever money they've made in the day, immediately transferred to that payment method, maybe a debit card, right? And then they can go, as I said, to their local bodega or to their grocery store and buy the groceries that they need to put food on their table that night, right. And so when I saw that, in real time, and then it started to make sense to me, I started to see Stripe for so much more than what I think most people perceived it to be. And I think the story is starting to, you know, speak volumes for itself these days but there's still a lot more that we can teach the world about what we have to offer. And I'm really excited about the opportunity to do that going forward.

Utsav Shah: Yeah, I think the mission, increasing the GDP of the internet, it's so simple, but it makes so much sense. Yeah, so I've seen some of the numbers that Patrick has posted about on Twitter, and the API volume of Stripe, it’s charges, right? It's not going to be, like QPS2, a free service or anything like that. So I've seen it roughly translated like five to 6000 QPS. That's what I've just done the math from whatever numbers he shared. And I'm sure it's maybe twice or thrice of that. But it doesn't seem like an inherently super high traffic service. And maybe you can correct me if I'm wrong, and I know you might not be able to share everything publicly but at that point, when you joined the team, why was there a need for a Traffic team? What was the goal of the Traffic team? And how has that goal expanded over time?

Tramale Turner: Yeah, I think it's a really interesting question. You're wrong about the number of QPS, but that's okay. It's okay to be wrong. I won't tell you specifically what that specific metric is but what I will say, which I think is both something intuitive, but also important to consider is that when you're thinking about user attachment to a service, typically you're thinking about, I mean, in this modern age, we think about hyperscale services. So we think about Facebook, or we think about Netflix, we think about users who are engaging at volume, or at scale, as we like saying the industry, with consumer-oriented, sometimes skewing towards entertainment services, right. And so making sure that you have cash content as close as possible to that user, think about the Netflix core team, for instance, and the work that they do to make sure that the CDNs and the edge of the network is performant and robust and resilient. Those edge network teams that they have, and they do have multiple teams, they sort of make sense, again, intuitively. 

But the other thing that you would consider is that if you have to do a retry, or if there is a network partition or packet loss of some shape or form that, you know, you kind of for free, get with TCP/IP guarantee of delivery, right? So you can retry again, TCP/IP, layer seven will sort of take care of some of the nuance of what does retry actually look like or what does effective and efficient packet delivery look like. And so you can sort of fake it and you can buffer streams or you can sort of jitter delivery of content. And all of that makes sense again, for most folks, I think and certainly, people who are listening to this podcast are very familiar with the vagaries of content delivery and billing things like CDNs. But when you start talking about money, it becomes a completely different game. So it's not so simple to just retry a request, because you may inadvertently double charge someone. That's a really bad thing. Or you may inadvertently double pay someone, also really bad. Really bad for the user, really bad for the organization providing the service, and potentially in violation of regulatory restrictions or regulations that you have to comport to in order to continue doing your business. 

So when you ask why, or when did a “Traffic team” become an integral part of Stripe’s engineering organization, from the very beginning. Absolutely important to understand what happens at the edge of the network, terminating TLS because, of course, everything is TLS encrypted and protected, making sure that whatever was in that payload goes saliently to its destination, and that the response goes saliently back to the requester. So that sort of table stakes. But you also want to be quite careful because you're collecting credit card numbers. And so most payment services providers have something called a cardholder data environment. That's where all of those pans or the primary count numbers, the credit card numbers that you have sort of sit at rest. In most cases, at least in businesses that are doing it in a PCI-compliant way, which is hopefully all of them, you never want anyone who doesn't need to see that primary account number, have access to it. And so how do you do all of the work that you need to do to make the credit card “part of the business” work? You send forward probably some tokenized representation of that credit card number. And that's what the majority of the business deals with, only that tokenized representation. And then they communicate back to the cardholder data environment and the cardholder data environment will communicate with the acquiring partner, and the banks and to make sure that there are funds available, and that they can commit the charge. And that is part of what the Traffic team is responsible for, amongst many other things at Stripe.

Utsav Shah: Okay, so that's super interesting. Let me just clarify this. So there is some service or something at the edge, which takes the actual credit card information, does some kind of hash or tokenization of it, and then the rest of the service [in the factory 17:27] at Stripe never has to worry about potentially dealing with PCI because they will have to at some level, but they don't have to worry about actually holding the credit card information because your edge service has taken care of that. And there's only one data store somewhere that knows how to map from your token to an actual credit card number. Roughly, does that sound accurate?

Tramale Turner: Yeah. So the only thing I would correct there is that there is a service that understands the semantics of translating between an actual credit card number and a tokenized credit card number. But where are those credit card numbers to that rest, let's just say, without going too much into specifics, that there is high resiliency and robustness of persistence to make sure that, one, there is as little latency as possible for the transaction. Because we want the user and the user being the partner of Stripe, the person that signed up to Stripe, the organization that signed up to Stripe, and their customer to have a really good experience. We want wherever they are on the globe for them to imagine that Stripe might have been founded in their country because it's so fast and because it's so effective in closing those transactions. And so how we persist, and how we translate from raw pan to tokenized pan is a really fundamentally interesting distributed systems problem that I think we're pretty darn good at executing against. But yeah, effectively, your summary is broadly correct with, as I said, a couple of nuances that I would correct.

Utsav Shah: Okay, that makes sense. And I've also seen speaking to the other problem you were talking about with requests not needing to be doubled, it's really bad to retry a request without thinking too hard about it, because you don't want to double charge people, or you don't want to double pay someone. I've seen something interesting in the Stripe API Docs about item potency keys, where you let users specify a key. And I'm guessing what that means is some service at Stripe maintains a database of every single request that comes in. And from the API Docs, it looks like y’all garbage collect after like 24 hours or something. So you need to store every single request that comes in, if it has an item potency key, and that's how you make sure not to retry things. 

Tramale Turner: That's exactly right. You got it. 

Utsav Shah: Yeah. And is that something that your team does?

Tramale Turner: We support the effective routing of those API calls that are item potent. But there are other teams that are actually responsible for making sure that the integrity of a call that has an item potency key attached to it fulfills the service requests, and to make sure that whatever mutation occurs only occurs once.

Utsav Shah: Okay, so what else am I missing? Is there anything else that is interesting like that edge service you're talking about, which you have to think about, as--? Let's say that you are pitching to an engineer about the technical challenges that your team has to face. What can you talk about publicly? 

Tramale Turner: Yeah, so what I like to talk about is the fact that regardless of what any candidate has heard about an edge team in other organizations, and there are, I think, to your earlier point… well actually, I don't know that we covered this point but there are edge teams sort of spiraled throughout the industry. I brought up Netflix, but there's the GFE team, the Google Front End team at Google, Amazon has a similarly shaped team that works on edge primitives. All of the hyperscalers do. Many of the scaled, if you will, used to be startups and are now big companies, Spotify, Shopify, you name it, they probably all have something similar. Even Lyft has an edge team. Typically, those teams are very small, because they have a very targeted and very specific set of services. Many of them that I've experienced, work on things like, you know, maybe there's an envoy sidecar proxy that at the edge, they want to make sure they're delivering service requests to some service mesh effectively and efficiently. And they worry only about again, that initial TLS termination and making sure that whatever they have instantiated at the edge is robust, resilient, scalable, etc. If a team goes deeper, like I think the Netflix team does, maybe there are several teams within the edge construct that worry about CDNs, worry about API gateways, and worry about also just making sure that there's effective service to service communications, and maybe they worry about some other core distributed systems primitives like leadership election, and what have you. 

Our team does all of those things. And in addition to the network sort of core features, functions, and primitives that we proffer to the organization, and to make sure that the API is highly available and resilient, we also worry about this notion of compliance. And why is that? So I mentioned that we're responsible for the cardholder data environment and the cardholder data environment is the thing that makes Stripe PCI compliant. And PCI DSS is a compliance regime that is a consortium of acquirers and a lot of folks in its current state that care about the integrity and the security of folks’ credit card information, and to make sure that as you're building services online, offline, the amalgam of the two, that you're doing so safely. Because what no one wants to ever have happen is that you're on the front page of the news because you leak credit card numbers, or someone was able to break into a system and exploit those numbers and then trade them on the dark web or some horrible story like that. So every organization that deals with credit cards has to have some level of PCI compliance, be that online or offline. 

And we are really good at the PCI compliance motion at Stripe, I would say, fairly exceptional. And the team that looks after a lot of that work, from an engineering perspective is the Traffic team. That's right. So because we got really good at that, the organization sort of looked at us and said, “Would you be interested in thinking about how you can help remove the toil of a lot of these other compliance and regulatory concerns we have?” So sock one, sock two, future things that we might be thinking about, that I can't talk too directly about here. But as an example, there are things like FINRA, HIPAA FedRAMP, all types of regulatory concerns. And what's interesting about those regulatory concerns is that they all require some level of evidence to show that you are compliant with the regulation. And that evidence collection motion, when you start to zoom out and look at it, a lot of the things that are being asked were quite similar. And so as engineers, you look at that and say, “Oh, well, these are declarative statements. And when we have declarative statements, that means we can probably automate the thing. We can come up with a state machine or we can come up with some system by which we are automating the collection of this evidence and maybe even automating the reporting of our attestation of the correctness of that evidence. And so that's one of the things that the Traffic team at Stripe also looks after. 

And then we have yet another thing that we're working on that is kind of in the shape of an infrastructure primitive as a product. And I can't talk too much about that, but I will say that it is not completely orthogonal to that notion of compliance and that notion of just being very assertive about protecting the integrity of information that one might share with one's customers as an organization, as a business. And so I'm hopeful that we'll be talking about that in the next less than a year, I hope, depending on the velocity by which we build these primitives, but it's something I'm super excited about. And I really can't wait for Stripe’s users to hear about it.

Utsav Shah: Cool. And I don't know if you've ever spoken about the thing that I'm doing right now. I'm working for a compliance automation company. Yeah, I don't know if we've ended up talking about what I've been working on since we spoke last. I've been working for a compliance automation company, so all of this sounds super exciting to me. And it's interesting that the Traffic team has ended up in charge. But I guess that somewhat makes sense as well, given that y'all are the team that has to worry about making sure your integrations in a sense, with the rest of the company are compliant. I can also totally imagine that… how do I frame this? Any sort of regulatory issue should be caught at the earliest layer possible, and that could end up being the trafficking. You can imagine if there is a payment that is being made, again, in your Lyft example, in a way that isn't compliant, you don't want to find out right at the end. You want to find out right at the beginning. I don't know if that makes sense.

Tramale Turner: No, it totally makes sense to me. And I think yes, and… I think that when we think about compliance, one of the things that a lot of organizations in my experience tend to do is, well, everyone in the organization sort of steps away from compliance, and they're like, “Oh, that belongs to the regulatory--" Like, maybe there's literally a compliance organization typically reporting to the CFO or to the chief legal officer, and the engineers in the organization sort of sigh and they're like, “I don't want to deal with this toilsome evidence collection process. It's just super disruptive. And I understand it's necessary, but it's not something that gives me joy.” And what I love about that is that it's boring. Boring, but everyone who participates in this ecosystem, and not just for payments, as I mentioned, like, if you're doing health care information, you're dealing with HIPAA, if you're trying to sell to the federal government, you're dealing with FedRAMP, and so on, and so forth. So I mean, you know this because you're working on a startup that has seen the opportunity, so much opportunity in this space. And what I'm working on is just making sure that we're really good for Stripe internally, with all the compliance and regulatory motions that we have to comport to, but I totally see, totally see the opportunity for platforms and products. 

In fact, we know this to be already something that there are many companies that have tooling and or platforms and services that support. So GRC tooling, which people may be familiar with, which is governance, risk, and compliance, there are a ton of vendors. ServiceNow is one of the biggest SaaS vendors, for instance, that has a GRC tool where evidence is supposed to rest and then you can use that resting state of evidence to support these different regulatory needs and concerns. So there's opportunity here, and I'm excited to just have a team that is really well versed in sort of the complexities of compliance, as well as having the experience to know how to build let's say primitives, then services and platforms to help accelerate getting a lot of that work done. That very necessary, but maybe very boring work done.

Utsav Shah: So in terms of goals, would it be success if no one in the rest of the organization had to care about compliance and it all got kind of platformized by your team? Or would it be success if people still have to do it, but it's super seamless for them? And maybe if you could just talk concretely like how do you enable compliance when you have like a 4,000 person company that has to think about it pretty much because it's bread and butter for your business?

Tramale Turner: Yeah, I think that's an excellent question. I think success for me, looks like being very rigorous around understanding what the toil cost of all of the compliance motions that we invest in today are. Again, these are stay-in-business motions, they're not things that are optional. They're not things that you can choose not to do. If you want to stay in business and you want to continue doing business in certain markets or within certain industries or proffering certain services, you must do these things. And so it's a really easy equation to look and see, “Okay, how many people are committed to--" You know, let's constrain it down. Let's just talk about PCI DSS. “How many people are committed every year to collecting evidence, cleaning evidence, sitting with an internal auditor, sitting with a QSA, and making sure that everything that we're supposed to be doing to secure this data, this credit card data that we're holding, is correct.” That's literally the question. Is it correct? And you can look at all of the effort that you expend against that, and then start to see, “Okay, what parts of this are redundant? What parts of this are things, to the earlier point, could be collapsed into declarative statements that we could tell a piece of software to go and do?”

So a really simple, just very basic version of an example of that is, one of the things you want to be able to see is access logs that tell you when a persistent store that has sensitive data was accessed. You could easily have an operator go to a system, type a set of commands, copy those commands, show the date, and show that the operator is pulling that information to show those logs of when something was accessed and or modified. You could also ask a computer to do that. And the computer could probably do it more effectively, more efficiently. And with that saliency of automation, you can then start to see, “Okay, what is the time we just saved from having the operator do that?” So those are sort of like the basic conversations and the really easy starter primitives that one would want to start building around, like the automation of evidence collection. 

Where I think it goes further is, “Okay, well, can you also automate the reporting piece as well? Can you take all of that data that you've collected, if you're collecting it regularly, and just on-demand generate the attestation of compliance?” I think you can. I think that that's actually something quite feasible. And if you get to that point, and you've solved it for PCI DSS, can you solve it for PCI P2PE? Can you solve it for FedRAMP? Can you solve it for FINRA, etc, etc.? And my conjecture at this moment - conjecture, because I'm improving it - is that you can do that, and I am solving that problem internally for Stripe as part of the infrastructure organization. And I am hopeful because we all tend to be quite ambitious at Stripe, that that's something that if we get really good at it, who knows? Who knows what we might do with it going forward as a potential service offering?

Utsav Shah: That makes a lot of sense. And you can imagine, I think everybody that needs to maintain credit card information needs to be PCI compliant. So offering that as “You know what? If you have Stripe, you can also get reports that prove that you've been compliant with the rate.” I don't know how feasible that is because I don't have any familiarity with PCI, but it seems like that is certainly an approach that is exciting. It's also scary because you'd be a competitor of the company [Inaudible 34:30] and I don’t want that.

Tramale Turner: Well, I must say, just to be clear, for users of Stripe, we do handle PCI compliance on their behalf, but there are different levels of PCI compliance and there are different types of PCI compliance. I mentioned P2PE, which allows you to do things like tap on a phone or use the NFC chip within your smartphone to have it act as a point of sales device. I think that you're going to see as different types of online or near online, I think what we traditionally call offline, but they are offline only in as much as there's a person who is present, but everything that's happening after the payment method is made available to the retailer, is something that's happening in the digital space, happening online. 

So I think as you see the proliferation of those things, maybe not so much in North America, because North America is just strange in the world. But certainly, globally, there are all kinds of new payment methods coming up every single day. You're going to also see governments and regulatory bodies saying, “Okay, we kind of want to make sure that we have a handle on what's happening here because we want to protect our consumers, we want to protect our citizens, in many cases, to make sure that they're not being taken advantage of or that we're not allowing the proliferation of crime or criminal enterprises by virtue of these new payment methods that are coming online.” A great example of that is what we've seen with digital currency, or crypto recently, and its enablement of some crime vectors that clearly are not optimal. And regulatory bodies are trying to figure out in the present moment, in real-time, right now, how to put some controls around some of those criminal enterprise vectors.

Utsav Shah: Yeah. And as part of maybe a follow up to that there's also-- All right, this is from my conversation with Emma at Stripe recently, have you noticed, or is your team also thinking about data governance stuff - credit card information of a particular country should only live in that country - because it seems like there's more and more laws around that? Is that kind of stuff or something that you have to think about?

Tramale Turner: Absolutely. So it's already public knowledge because I think there were several articles published about it, but one very deliberate way that we're making investments in that space-- And just to sort of add clarity, for folks who may not be familiar, there is a notion of data residency, data locality or data sovereignty. And what does that mean? That means that a regulatory and or political body might say, “For users who are using online services to make payments and to buy goods and services, who reside within our municipality, our country, whatever domain that we control, we would want their data to remain within the borders of the country.” Or if there is a processing function, some type of compute that's occurring, where you're capturing that payment method and doing something to remove or add funds to it like that, that processing must happen only within the borders that they control. And they may say, at least on the surface, that they're doing that for, indeed, to protect the user from potential criminal activity, to be able to easily have access to those services, should there be some nefarious activity, and they need to collect evidence from those services much more saliently or easily. 

And then there, if you go deeper, is something of the notion of protectionism. So there may be other competitors within that country that are trying to compete with more globally established players, and the country or the regulatory body may want to support their own natively grown industries in order to allow them to scale and or to provide, maybe to some definition of better, “better” services to their citizenry. So that's what we're talking about. 

And Stripe recently - and I should caveat that recently means on the order of two and a half years - has been thinking about this for India. So India passed regulations that basically said that any payment service provider that's operating within the country has to keep any Indian citizen’s data that is used against those services resident within the country. Simple as that. So what does that really mean concretely? If I am shopping online with my credit card, I'm in India and I'm an Indian citizen, or I'm using RuPay or I'm using some other payment method that is either global or local to India, any information pertaining to that payment method, if it initiates in India must stay in India. And there are some caveats like you have a little bit of a grace to process outside of the country, but any data that is processed can't persist outside of the country for more than 24 hours and things like that. And so if you've built, for instance, on a hyperscaler, and let's say all of your services are in US-East-1. If you're using AWS, that presents an issue, right, because you now need to figure out how to move those services regionally. And then not to degrade the services of any of your users who are not in that country, but also for the users in that country, make it somewhat seamless and make it also seem as though all of those services that you proffer to everyone else are available wherever those sort of more constrained borders might be. 

And so that's a tough problem. It's not just a tough problem. It's actually a really difficult problem. But Stripe recently announced that we solved that problem for India. And if you're watching the FinTech news, as I think many folks are these days, you'll see that we'll be making future announcements about other locations that we've solved it for in the coming months. And it's something that we have been investing a lot of time and rigorous effort in making sure that we do correct. Again, not just for the countries that have data sovereignty rules, either pending or already in existence, but also to make sure that we don't degrade services for the modulo those countries rest of the world. So I really love this topic because I worked on India. And I'm really happy with not only how we addressed the issue, but the many excellent engineers that just dove deeply into the problem, and rigorously worked to affect a really nontrivial change to, I'm hoping, the light of our users in India.

Utsav Shah: That just sounds so amazing. Stripe sounds like to me as a compliant state machine where you have to continuously inject laws and changes in laws and make the system work. I don't know if you could talk anything about the actual implementation. So I know, at a very basic level, you have to run some, or a significant percentage of services in India. And that makes sense. I don't know if you can talk about anything else, or where the complexity of the implementation comes in.

Tramale Turner: Yeah, I can't talk about specifics, but I can talk about what the-- I think any reasonable person and certainly anyone that's dealt with distributed systems and understands convergence and understands, I think, misnamed eventual consistency, it’s eventual convergence really, I think the thing that makes all of this difficult is state, at the end of the day, where does data rest and making sure that you can attest that there is a compliant resting state of that data, per whatever the regulation says it needs to be. I mean, my friends who focus on the computing primitives will throw stones at me, I'm sure, but I actually see compute in actually my own area, networking, as much easier. It's not that difficult for me to come up with smart route maps and tag a request with sort of a locality primitive and say that, “Hey, I want you to direct this traffic only to these regions.” I mean, we've been doing that sort of, if you will, network, directive, and topological decisioning forever, right? Like since the inception of the internet, more or less That's not a true statement. But it's not that difficult. But dealing with making sure that data that you-- I mean, remember, these are corporations and corporations need to be able to close their books and understand how much revenue they've generated and understand if there are nefarious people trying to attack the system, how they're trying to attack the system. And you need reams and reams of data. And you need to be able to go through all of your data and understand what that data is telling you about what's happening within the system to get state about the system, which is why we call them stateful systems, right? 

That is a majorly, huge, huge, huge part of the issue. That's the big rock. And we at Stripe have spent a lot of time, I think, bending our core persistence primitives to a point where-- I don't know how every sort of storage problem within our industry looks in comparison. But I would say there are probably very few organizations that with the tools and services that we make use of, are pushing them to their utility limits as we are. I would argue that probably only the hyperscalers are doing the type of computer science, the type of distributed systems focus that we are investing in in order to comply with these regulations. And I know for a fact that it's a big deal at AWS. I'm here in Seattle and so I have a lot of friends that work at that particular hyperscaler. And I would imagine that my friends down the street at Google, also, similarly, are dealing with how best to comport and comply with these laws and regulations.

Utsav Shah: Yeah, and maybe taking a step back from compliance, what compliance really [Inaudible 46:18] a bunch of things to make. What compliance means is, “Are you complying with our set of laws? But the laws are there for a reason. They're not there just the impede progress, for no reason. But a lot of compliance is around making sure that you're keeping your data secure, and also having processes around that. So the flipside of compliance is really security. And I'm trying to understand if there's anything you can talk about with regards to the kind of security practice and security measures and maybe security implementations y’all have to think about, because I'm sure you're dealing with tons of actors trying to steal data from your systems by just fuzzing the API, like “Maybe I can just get access to something that I'm not supposed to access.? How do you think about that? And how much is your team responsible for that?

Tramale Turner: Yeah, it's a good question. You keep asking me these great questions that I can't actually give you direct answers to. Let me put it this way. We're dealing with money. And money, as we all know, creates an attractive attack vector for assailants. And I think, not just criminals who are trying to perhaps find a way to exploit the system for some remunerative benefit, but also people who would just want to deny others access to that democratization of economic freedom and enablement that I spoke to at the beginning, the thing that actually gets me up and excited about Stripe every morning. And so what I will say is that Stripe has already, and is continuing to invest in one of the best and most rigorous security teams and individuals building platforms and services to protect the integrity of our business that I've ever had the pleasure of working with. The leader of security at Stripe Niels Provos was a long-serving Google Googler, was a Google Vice President of Security, left Google to come to Stripe. And Niels has been building, I think, an amazing team of practitioners, that every time I get an opportunity to interface with them, and I do so frequently, I'm thoroughly impressed. And I actually consider my team to be an extension in some ways of the security team. Security is a different pillar at Stripe than infrastructure, but clearly, they conjoin and clearly, they're intersectional. And I am one of those points of intersection because I support this CDE, this Tier 0 service that is incredibly a lot of things. It's incredibly secure, for sure, but also incredibly important to the viability. Stripe doesn't exist without the CDE, so to speak. 

So one thing that I can talk about that we do that I think would be pretty obvious to anyone that deals with hyperscaled or highly scaled services, is that we are very concerned about what's happening at the edge of the network. To your point, who's sort of poking at the API? And what sort of things are they doing, layer three, four, or seven to try to either deny access to the API or to try to do strange things that the API doesn't expect? So you mentioned fuzzing but there are all types. If you were to look at request logs of Stripe, you would see all types of attempts at just doing strange escape vectors, trying to manipulate URL paths, using really odd URI access methods. It would not surprise, I think, most people, but would also really, really freak you out if you saw how frequently and how often folks are trying to attack just this one organization. And you think about all of the payment service providers that are out there. I am hopeful that they all have security teams and engineering teams more broadly, that are just as rigorous around security as we are. 

But again, to the point, this notion of denial of service is a big deal, we all know that. And so we make use of certain AWS primitives like AWS Shield to protect that layer three and four but for layer seven, we have sort of our own, it's not necessarily a WAF, a web application firewall, but it is a platform, a tool kit that we have to address different attack vectors that we have both seen and that we anticipate and expect against the API. And we insert primitives into that platform to allow us to do things like throttling or to deny API keys that maybe have leaked for whatever reason. Or if we see card testing, which is something that frequently happens within the industry - trying to test the card to see that it's usable and then to steal funds from that card once you find out that it is usable. We can identify that type of activity and then block it immediately. So that's just the sampling. But trust me, if I were able to, I could probably talk about for hours, many of the different types of security mitigations, remediations, and defense in-depth things that we invest in at Stripe.

Utsav Shah: Yeah, so you're talking about web application firewalls. And I think, publicly, I've seen just systems that block based on IP addresses or certain URL patterns. And what you're describing, at least what it sounds like to me is a whole platform of being able to specify all of these different rules or these different criteria, like, “Oh, looks like an API token for someone has gotten leaked, or there's just suspicious activity,” and being able to automatically disable those things. And that makes a lot of sense. And also, it makes sense that you don't want to be DDoSed, since you have a lot of people depending on you for their business. And I'm sure when Stripe goes down, a lot of companies are upset and are calling you up. 

Tramale Turner: Indeed, we try not to go down. And I would argue we're pretty good at that. Our availability is staggering when you look at it, five nines. And we say for the network, and I don't want to over speak here, but I think just mathematically for the API to be available at five nines, you can imagine what the network, at least for minutely availability we tried to achieve. Just to be clear, I didn't state what that SLI is so no one at Stripe come at me. I didn't reveal that publicly. But again, we have very high, great expectations, as it were. I've been quoting a lot of Dickens at work recently because wherever I look, and whenever I see a plan, or whenever I see an ambition at Stripe, and I see how it implicates my team, I go, “Wow, we've got a lot of great expectations.”

Utsav Shah: Yeah, maybe the last set of questions are around building teams to service these lofty goals. When you have a small team, and I'm just talking about like 50 people, 100 people in an entire company, there are a few trusted individuals that you can think about, you know, “This person will take charge of these things.” But once your company grows so big, you have to think about just expanding from individuals or even teams at that point. How do you make sure your infrastructure stays at five nines? You know, if there is a lot of attrition on the team, how do you set yourself up for success? And what kind of processes and what kind of people are you looking to hire? What are the things you're thinking about when you have to think about, you know, “How do I make sure that my APIs stay at five nines for next year?”

Tramale Turner: I love this because there's no easy answer. There's no sort of canonical response that says, “This is what you do.” If there were, everyone would be copying it, and everyone would be doing it. Here's my experience, and then I'll tell you what I specifically do. My experience is that you, first of all, face the reality of what you just said, people are going to churn. At Stripe, for instance, we're huge proponents of internal mobility. And we actually encourage people to move teams after 12 months or so within a role, assuming, you know, decent performance. And I think the good thing about that is that it encourages teams to be very rigorous about their processes, about making sure that they have run books for their services, that they build highly reliable, highly resilient services, that they look to see if you will build a moat of protection around faults. 

The old head of infrastructure at Stripe-- I should say the former head of infrastructure, because he’s not that old of a gent, is Will Larsen and Lethain all over the internet. And Will wrote this book recently, An Elegant Puzzle under Strike Press. And I encourage folks to read it. I'll make fun of Will, here, I find Will's writing incredibly dense, and you'll probably have to read the book three times. But it's a good book. And Will talks a lot about organizational structures and dealing with the vagaries of what will happen within an infrastructure as it intersects with those organizational structures. So you're talking about how technical complexity meets organizational complexity. And the thing that you have to hold as a base principle is that you don't know anything. It is the sort of Socratic paradox, “I know that I don't know anything.” And for all of your experience, and for all of the things that you've done, you're not going to build the perfect most reliable, never faults system. So how do you get around that? You prepare for eventual failure modes, you prepare for understanding what those fault domains look like. AWS could go down. An AWS region could fail tomorrow. What do you do when that happens? And how do you recover from that type of epic disaster? Do you even have a plan? Do you have a run book? Have you done a run day? Have you done or game day or whatever your company might call it? Have you tried to test your assumptions and validate whatever mitigations that you have in place? And if you haven't, you're doing it wrong. 

And so what I do is I try to hire the best. Absolutely. I look for the best possible folks that I can and I never sort of rest on the laurels. I have a ton of great folks that work in Traffic, I'm still looking for greater folks because I want to constantly challenge the notion that we have solved for the problem set, the problem domain that we are responsible for. I listen, - I try to - as much as I talk and hopefully more so than I talk because it's really about learning about the environment, learning about experiences and learning about what people are observing, and then synthesizing a sort of mental model of what we should be doing and what we should be investing in, in order to address these risks. I test my assumptions rigorously. So we do game days, we make sure that if we synthetically fail something, how are we mitigating against that eventual occurrence, and so on, and so forth? So I think proper planning prevents poor performance, which is sort of something you steal from the MBAs is something that really is relevant to engineering and engineering disciplines as well. If you're not planning, then you're not doing it right.

Utsav Shah: That makes a lot of sense to me. And the one question that comes from there is, how do you balance what's practical versus what should be correct on principle? A simple example is, you should make sure that your database backups work. I think almost no one would argue that that's something you should do. But then there are so many things that could go wrong. And thinking about the AWS example, I know a lot of companies would be like, “Oh, if AWS goes down, it's fine if I go down as well.” I'm sure that's not true at larger companies and companies like Stripe, but there's also a prioritization game that you have to play. So is there any framework you think about through then like, “How do I prioritize?” Is it just based on how much risk something has towards my business versus what's the likelihood of this happening? What else is there that you think about? 

Tramale Turner: Well, I mean, when you think about it from that perspective, it's almost like thinking about-- We were talking a lot about security today. And when you talk about security mitigations, you talk about threat models, and you talk about what's the probability of something occurring? And what's interesting about probability is that you're not saying-- I mean, it's very rare that you're saying the probability is zero, right? You're going to have some percentage of likelihood of occurrence. And so the probability of an AWS region going down is very, very low. But it has happened, it happened in 2012. We all remember the big EBS outage that took out US-East completely. It wasn't just US East one, it was like the entire East Coast went down. And it was a horrible day. Amazon learned a ton from that. And that shape of failure mode is not likely to occur again but the fact remains that a transit zone could go down, some core EBS primitive could go down, bad things can and will happen. And so when you talk about how to be principled around that, it's just acknowledgment of the fact that there's enough entropy in the world that random things, Black Swan events will happen. And if you are being rigorous about-- You know, I think, to the thing that you're driving towards, there's different levels of rigor and expectations depending upon the type of service that you're delivering. For us, we have to take every eventuality and every possibility quite seriously, again, because of the type of information that we're dealing with. We're dealing with people's livelihoods. 

We go back to the beginning of our conversation. If I fail to be rigorous about my pursuit and about the things that I'm responsible for, and that person somewhere in Bangalore, or somewhere in Kenya, or somewhere in Dublin, can't put food on their table that evening, that's my fault, I own that failure. And I have to have empathy with that user because I wouldn't want that to be me, and I wouldn't want to have an organization that my livelihood depends upon to not have that type of concern in mind as they're doing what they do every day. And so when we talk about principles, that's the way that I say my principles-- Like of course, Stripe has a canonical set of leadership and operating principles, as does every company of that scale. But I think even beyond the leadership and operating principles, which are very good, it's just about being human and caring about the fact that these services go beyond just bits and bytes, that they touch people at the end of the day. And so the humanist that’s in me says it's just about caring for my fellow man and woman and making sure that I do everything that I can to ensure that they have a good day.

Utsav Shah: Yeah, that makes sense. And it's very easy to get lost in numbers like four nines or five nines, but when you translate that, the actual human impact, that's what really shows you the importance of the work that you're doing. Yeah, well, this has been a lot of fun. Thank you so much for being a guest. And I hope to have you for a round two, maybe in a few years when you can talk about some of these topics more publicly.

Tramale Turner: All right. I will put that on the calendar. Thanks for having me. It was really fun.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Rajesh Venkataraman is a Senior Staff Engineer at Google where he works on Privacy and Personalization at Google Pay. He’s had experience building and maintaining search systems for a large part of his career. He worked on natural language processing at Microsoft, the cloud inference team at Google, and released parts of the search infrastructure at Dropbox.

Apple Podcasts | Spotify | Google Podcasts

In this episode, we discuss the nuances and technology behind search systems. We go over search infrastructure - data storage and retrieval, as well as search quality - tokenization, ranking, and more. I was especially curious about how image search and other advanced search systems work internally with constraints for low latency, high search quality, and cost-efficiency.

Highlights

08:00 - Getting started building a search system - where to begin? Some history.

13:30 - Why we should use different hardware for different parts of a high throughput search system

17:00 - What goes on behind the scenes in a search system when it has to incorporate a picture or a PDF? The rise of transformers, not the Optimus Prime kind.

We go on to discuss how transformers work at a very high level.

27:00 - The key idea for non-text search is being able to store, index, and search for vectors efficiently. Searches often involve nearest neighbor searches. Indexing involves techniques as simple as only storing the first few bits of each vector dimension in hashmaps.

34:00 - How search systems efficiently rebuild their inverted indices based on changing data; internationalization for search systems; search user interface design and research.

42:00 - How should a student interested in building a search system learn the best practices and techniques to do so?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Devdatta Akhawe is the Head of Security at Figma. Previously, he was Director of Security Engineering at Dropbox, where he led multiple teams on product security and abuse prevention.

Apple Podcasts | Spotify | Google Podcasts

In this episode, we discuss security for startups, as well as dive deep into some interesting new developments in the security realm like WebAuthn and BeyondCorp. We wrap things up with slightly philosophical points on the relationship between security and regulation.

Highlights

0:00 - What got Dev interested in computer security?

4:00 - Security for a startup. What framework should a CTO use to think about security as their startup gets its first customer?

7:30 - Trends in the security space. Increasing customer demand for security due to the multi-tenant nature of the cloud. Lateral movement attacks.

12:45 - BeyondCorp. “There’s BeyondCorp, and YOLO NoCorp”. NIST’s paper on it.

25:00 - How should I think about a Bug Bounty program as a startup founder? - Having a good “Vulnerability Disclosure Policy” is an extremely valuable first step

26:30 - Why would anyone report bugs if they weren’t being paid for them?

30:00 - Interesting security products that companies might want to buy :)

34:30 - What is WebAuthn?

39:00 - How security and usability shouldn't be a trade-off

43:00 - Security regulations

47:00 - A repeat question - as a startup, what should I do to keep myself secure?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Laurent Ploix is an engineering manager on the Platform Insights team at Spotify. Previously, he was responsible for CI/CD at several Swedish companies, most recently as a Product Manager at Spotify, and a Continuous Integration Manager at Sungard.

Apple Podcasts | Spotify | Google Podcasts

Highlights

05:40 - How CI/CD has evolved from a niche practice to a standard and expected part of the development workflow today

12:00 - The compounding nature of CI requirements

14:00 - Workflow inflection points. At what point do companies need to rethink their engineering workflows as they grow? How that’s affected by the testing pyramid and the “shape” of testing at your particular organization

20:00 - How the developer experience breaks down “at scale” due to bottlenecks, the serial nature of tooling, and the “bystander effect”. Test flakiness.

28:00 - How should an engineering team decide to invest in foundational efforts vs. product work? The idea of technical debt with varying “interest rates”. For example, an old library that needs an upgrade doesn’t impact developers every day, but a flaky test that blocks merging code is much more disruptive

33:00 - The next iteration of CI/CD for companies when they can no longer use a managed platform like CircleCI for various reasons.

40:00 - How should we measure product velocity?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Sujay Jayakar was a Software Engineer at Microsoft Research where he worked on kernel bypass networking. He was previously a Principal Engineer at Dropbox where he worked on the migration of user data from S3 to the internal storage system (Magic Pocket), and the sync engine deployed to clients.

Apple Podcasts | Spotify | Google Podcasts

Highlights

05:00 - What framework do you use to decide to stop using S3 and store data in your own data centers? (the “Epic Exodus” story)

11:00 - Perfect Hashtables and how they are used in production systems to reduce memory use

14:00 - What is an OSD? (Object Storage Device). How does it work?

20:30 - SMR drives

30:00 - The actual data migration - how did it happen, and how does one validate that the data being transferred is correct.

33:00 - “S3 being overwhelmed”. That’s a string of words most software developers don’t expect to hear. What kind of overhead do kernels impose on networking, and why?

43:00 - What is Kernel Bypass Networking?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Colin Chartier is the co-founder and CEO of LayerCI. LayerCI speeds up web developers by providing unique VM-like environments for every commit to a codebase, which enables developers, product managers, QA, and other stakeholders to preview code changes extremely quickly and removes the need to spin up a local environment to showcase demos. This enables interesting workflows like designers signing off on pull requests.

Colin was previously the CTO of ParseHub and a software design lecturer at the University of Toronto.

The focus of this episode was on developer productivity, management of a CI system and company, and even a little bit of cryptocurrency mining.

Apple Podcasts | Spotify | Google Podcasts

Highlights

0:00 - What does LayerCI solve?

2:00 - CI is generally resource-intensive and slow. What makes LayerCI fast? A lot of similarities to the Android Zygote. We’ve even floated the idea of Python Zygotes at a previous job.

5:00 - The story behind LayerCI.

12:00 - The architecture that serves LayerCI. The cost of nested virtualization and each additional Hypervisor. OVH.

15:00 - Rate limiting. The impact of rising cryptocurrency prices on free tiers of CI providers - read more.

30:00 - The power of building high-quality infrastructure. How both developer tools like LayerCI, as well as low-code/no-code tools like Retool and Zapier are important for the future.

37:00 - Colin’s course for DevOps academy

47:00 - Hiring philosophy for startups

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev