Software at Scale

Nikita Gupta is a Co-Founder & CTO at Symba, a platform that helps manage talent development programs like internships.

Internships are one of the most effective ways for hiring at a software company, but there’s a lot of work that goes into managing successful interns. With hiring getting harder across the industry due to increased competition and funding, I thought it would be interesting to dive into understanding how to manage successful internship programs.

Highlights

0:30 - What is Symba?

1:30 - Starting with the hot-takes. So, are college degrees overrated now?

5:30 - Why do I need a software platform to manage internships?

8:50 - Why do companies generally need to manage 8 - 10 platforms for internships? What have you seen?

10:30 - As a software engineer or manager, how do I make my intern successful?

13:30 - Cadence of check-ins

16:30 - With remote interns, how do you build a successful community?

18:50 - How do I measure the success/efficacy of my internship program?

21:00 - How do I know that my intern mentors/hosts are doing a good job?

25:00 - What are some concrete steps that I can take to increase my intern pool’s diversity? What should I track?

27:30 - What are some trends in the intern hiring space?

32:00 - Government investments in internship programs

33:00 - What’s your advice to the first-time intern mentor/host?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Guy Eisenkot is a Senior Director of Product Management at BridgeCrew by Prisma Cloud and was the co-founder of BridgeCrew, an infrastructure security platform.

We deep dive into infrastructure security, Checkov, and BridgeCrew in this episode. I’ve personally been writing Terraform for the last few weeks, and it often feels like I’m flying blind from a reliability/security perspective. For example, it’s all too easy to create an unencrypted S3 bucket in Terraform which you’ll only find out about when it hits production (via security tools). So I see the need for tools that lint my infrastructure as code more meaningfully, and we spend some time talking about that need.

We also investigate “how did we get here”, unravel some infrastructure as code history and the story behind Checkov’s quick popularity. We talk about how ShiftLeft is often a painfully overused term, the security process in modern companies, and the future of security, in a world with ever-more infrastructure complexity.

Highlights

00:00 - Why is infrastructure security important to me as a developer?

05:00 - The story of Checkov

09:00 - What need did Checkov fulfil when it was released?

10:30 - Why don’t tools like Terraform enforce good security by default?

15:30 - Why ShiftLeft is a tired, not wired concept.

20:00 - When should I make my first security hire?

24:00 - Productizing what a security hire would do.

27:00 - Amazon CodeGuru but for security fixes - Smart Fixes.

33:00 - Is it possible to write infrastructure as code checks in frameworks like Pulumi?

37:00 - Not being an early adopter when it comes to infrastructure tools.

40:00 - The Log4J vulnerability, and the security world moving forward.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Tanmai Gopal is the founder of Hasura, an API as a service platform. Hasura lets you skip writing API layers and exposes automatic GraphQL APIs that talk to your database, trigger external actions, and much more.

We talk about the implementation of a “compiler as a service”, the implications of primarily having a Haskell production codebase, their experience with GraphQL, hiring product managers technical enough to build useful features, some new and upcoming Hasura features, and riff about the current state of front-end development and `npm` hell.

Highlights

00:20 - What does the name Hasura mean?

02:00 - What does Hasura do?

04:00 - Why build this layer of the stack?

08:00 - How to deal with authentication if APIs are exposed directly via the database.

26:00 - Does Hasura make production applications faster?

33:00 - JSON Aggregation in modern databases

38:00 - Why Haskell?

44:00 - How do you write quality Haskell? How does hiring for Haskell positions work out in practice?

55:00 - Application servers do much more than just talk to databases. How does Hasura provide escape hatches so that non-database interactions (for eg: talking to Stripe) work out?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Kailash Nadh is the CTO of Zerodha, India’s largest retail stockbroker. Zerodha powers a large volume of stock trades - ~15-20% of India’s daily volume which is significantly more daily transactions than Robinhood.

Apple Podcasts | Spotify | Google Podcasts

The focus of this episode is the technology and mindset behind Zerodha - the key technology choices, challenges faced, and lessons learned while building the platform over several years. As described on the company’s tech blog, Zerodha has an unconventional approach to building software - open source centric, relatively few deadlines, an incessant focus on resolving technical debt, and extreme autonomy to the small but efficient technology team. We dig into these and learn about the inner workings of one of India’s premier fintech companies.

Highlights

[00:43]: Can you describe the Zerodha product? Could you also share any metrics that demonstrate the scale, like the number of transactions or number of users?

Zerodha is an online stockbroker. You can download one of the apps and sign to buy and sell shares in the stock market, and invest. We have over 7 million customers, on any given day we have over 2 million concurrent users, and this week, we broke our record for a number of trades handled in a day - 14 million trades in a day, which represented over 20% of all Indian stock-trading activity.

[03:00] When a user opens the app at 9:15 in the morning to see trade activity and purchase a trade, what happens behind the scenes? Life of a Query, Zerodha Edition

[05:00] What exactly is the risk management system doing? Can you give an example of where it will block a trade?

What is the risk management system doing?

The most critical check is a margin check - whether you have enough purchasing power margins in your account. With equities, it’s a simple linear check of whether you have enough, but for derivatives, it’s about figuring out if you have enough margins. If you already have some futures and options in your account, the risk is variable based on that pre-existing amount.

What does the reconciliation process look like with the exchange?

We have a joke in our engineering team that we’re just CSV engineers since re-conciliation in our industry happens via several CSV files that are distributed at the end of the trading day.

[08:40] Are you still using PostgreSQL for storing data?

We still use (abuse) PostgreSQL with hundreds of billions of rows of data, sharded several ways

[09:40] In general, how has Zerodha evolved over time, from the v0 of the tech product to today?

From 2010 to 2013, there was no tech team, and Zerodha’s prime value add was a discount pricing model. We had vendor products that let users log in and trade, and the competition was on pricing. But they worked on 1/10,000th the scale that we operate on today, for a tiny fraction of the userbase. To give a sense of their maturity, they only worked on Internet Explorer 6.

So in late 2014, we built a reporting platform that replaced this vendor-based system. We kept on replacing systems and dependencies, and the last piece left is the OMS - the Order Management System. We’ve had a project to replace this OMS ongoing for 2.5 years and are currently an running internal beta, and once this is complete, we will have no external dependencies.

The first version of Kite, written in Python, came out in 2015. Then, we rewrote some of the services in Go. We now have a ton of services that do all sorts of things like document verification, KYC, payments, banking, integrations, trading, PNL, number crunching and analytics, visualizations, mutual funds, absolutely everything you can imagine.

[13:55] Why is it so tricky to rebuild an Order Management System?

There’s no spec out there to build an Order or a Risk Management System. A margin check is based on mathematical models that take a lot of different parameters into account.

We’re doing complex checks that are based on mathematical models that we’ve reverse-engineered after years of experience with the system, as well as developing deep domain knowledge in the area.

And once we build out the system, we cannot simply migrate away from the old system due to the high consequences of potential errors. So we need to test and migrate piecemeal from the system. 

[13:55] One thing you notice when Zerodha is how fast it feels compared to standard web applications. This needs focus on both backend and frontend systems. To start with, how do you optimize your backends for speed?

When an application is slow (data takes more than a second to load), it’s perceptible, and can be annoying for users. So we're very particular about making everything as fast as possible, and we’ve set high benchmarks for ourselves. We set an upper limit of mean latency for users to be no more than 40 milliseconds, which seems to work well for us, given all the randomness from the internet. Then, all the code we write has to meet this benchmark.

In order to make this work, there’s no black magic, just common sense principles. For the core flow of the product, everything is retrieved from in-memory databases, and nothing touches disk in the hot path of a request.

Serialization is expensive. If you have a bunch of orders and you need to send those back, serializing and deserializing takes time. So when events take place, like a new order being placed, we serialize once and store the result in an in-memory database. And then when an HTTP request comes in from a user, instead of a database lookup and various transforms, the application reads directly from in-memory databases and writes it to the browser.

Then, we have a few heuristics. For fetching really old reports that <2% of users use, it’s okay for those to be slow. Those will happen in separate paths so that they don’t block the more frequent kind of requests.

Finally, we’ve written all these services with Golang, which is fast out of the box, provides a reasonably good developer experience, and has good concurrency primitives. We’re careful with memory allocations and pool resources wherever applicable.

[24:00] Zerodha also seems to have skipped the React world, by going with Flutter on mobile and Vue on the web. Can you speak to that decision for mobile apps?

We initially built the iOS app in React Native about 3-4 years ago. I’m not sure how things are today, but the application was fairly slow. The bread and butter of a trading application is a bunch of ticking numbers to show stock values, and we experienced a bunch of frame drops while trying to render those. You’d think that it’d be trivial to show that with low latency in 2017, but we were experiencing 5-10 frame/s, and Indian smartphones weren’t extremely powerful then. We also ran into several library/dependency breakages.

We then randomly ran across Flutter, which in pre-alpha, and not a lot had been written in it. Picking a bleeding-edge technology is very risky, and we wanted to evaluate the risk carefully, so we built out a full-blown prototype of the Kite app that had all the parts we thought would be bottlenecks, like web socket connections, updating numbers, list views, navigation, transitions. We learned Dart (which you need to use Flutter). Once we built out the prototype, it was clear that the performance and experience with Flutter was significantly better than React Native. So we made that very early decision to ditch React Native and adopt Flutter.

We figured that even if Flutter got killed, we’d benefit from using Flutter for a few years and would eventually move to something else, and traded-off the risk. We launched the iOS application, fixed up issues, killed our Android application, and rolled out a Flutter version.

[28:30] How about not going with React on the website, and going with Vue?

We built our v1 with Angular, but we found it tricky to use for our small team, and there was the major version break fiasco. It was overly complicated even after months of use. We decided to skip out on sunk costs and decided to evaluate something new. Picking Vue over React was primarily a judgment call, the template system reminded of us Django and it felt easier to work with compared to wrapping HTML in JSX and function calls.

[30:30] How do you verify that your systems are correct and consistent? Could you walk us through the time when commodity prices went negative, what happened with your systems?

When commodity prices went negative, we lost a bunch of money, just like many other brokers and institutions. Thankfully, the Indian exchanges shut down trading after a while, and I think the exchanges themselves weren’t equipped to handle negative commodity prices.

The nature of the stock market is that it’s extremely complex and unquantifiable, and the complexity comes from human psychology and nature which is hard to account for. A bit of news could come up that could shake up the market. There’s price volatility, but with India, also regulatory financial volatility. Regulations come and change how brokers work overnight. It’s all correct in spirit to improve things for Indian investors, but massive changes nonetheless. Some changes completely alter how broking works in India, all with a month’s notice. So change here is the only constant, and change management is complex, slow, and risky.

For the technical stuff, we do the standard unit tests, integration tests, but we do a ton of QA after. So after developers have validated changes, the application is handed to all the various domain experts across the company test and QA changes. Their job is to try and break the system. This is very important as there are a lot of behavior changes that are tricky to quantify. For example, a regulation might come in that requires stock splits in a certain way, and it’s hard to back-test since it’s never been implemented that way before.

Due to the inherent complexity and rate of change, it’s not feasible to implement a stock market’s model in a test. So the combination of automated testing and manual QA by domain experts is our first step, after which we release an internal beta, and we slowly ramp up. Thankfully, this has worked for us.

[35:30] Release Cycles and Technical Decision Making at Zerodha

In terms of our release cycles, we move slowly and with care. If we feel that technical debt is mounting, we will pause feature development and address the debt. We’ve rewritten core systems several times when the benefits were apparent.

One of the really unique things about Zerodha is that technical decisions are entirely driven by technical folk. There’s no business folk who come and say - don’t fix that system but add this new feature instead. There’s no pressure to ship features, we don’t commit to shipping features every quarter, and there are no absurd goals. We agree that critical bug fixes should be fixed in a timely way. We will implement features only if it makes sense. Sometimes, if a feature makes business sense, but the system is not ready for it, and there might be a hacky way to implement it. We never add hacks, instead, we clean up the debt, make the system amenable to the feature, and only then add it.

This sounds slow, but it pays off in the long run. Because we’ve never let technical debt mount, and never compromised on hacks, we’ve been able to build things faster. After every refactor, the next set of features end up being implemented extremely quickly. Even with continuous regulatory changes, we’re always able to keep up and implement whatever’s necessary.

Ironically, we’ve shipped a lot of things fast and well by slowing down.

[38:00] What’s different about Zerodha that it allows its tech team full autonomy?

It’s common sense, really. The technology team does not have a full understanding of the business side of things, so it lets the business decide what to build. Likewise, the business does not really have context on debt and demand technical changes.

If you never pause to clear technical debt, it grows exponentially, and the system becomes a burden. I think if people had empathy to understand that concerns like tech debt were legitimate, software companies would likely be much more productive than they are now.

The other side of this is deadlines. Technical people find it extremely difficult to come up with deadlines due to the complexity of the space. Even as domain experts, the estimated time for a small task might be weeks, and a seemingly tricky task could just take hours. The core is to let technical people make these decisions.

[42:00] As a technical person, how do I know that there’s too much technical debt? What frameworks can I use to understand that I should probably invest in the foundation, and how do I develop that intuition?

Intuition is the unquantifiable summation of past experiences. The more experienced you are, the more you can develop your intuition. But there are simple metrics you can use to make these decisions if you’re a competent developer.

When you find it hard to collaborate, hard to ship new things, there are consistent performance bottlenecks, these are simple, commonplace signs that something’s wrong. If you realize that if certain parts of your code were slightly more modular, we could have shipped these 4-5 features faster. It’s contextual, but you know it when you see it. These indicators, such as difficulties, annoyances, and bottlenecks, indicate debt and burden.

[45:00] As a final question, if I’m a software engineer looking for a job where they value technical quality, how would you suggest I evaluate from the outside? 

First and most importantly, make sure to cut through the hype, and join a company only because others think it’s a good idea to do so. Look past your biases and evaluate in a data-driven way. Most importantly, the kind of software produced by the company is the best indicator of the culture of the company. Also, try to find resources that serve as an indicator of culture and engineering practices.

The reality is that most software engineering is really, really boring work. And innovation generally comes in spurts. And once you innovate, you need to build it into a usable system, which involves a lot of boilerplate. So most software engineering is boring, and that realization only comes in with experience. Once you know that, you will be in a better position to make trade-offs, and you’ll look for companies with a better culture or other parameters that are important to you in your decision-making.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Ganesh Datta is the CTO and co-founder of Cortex, a microservice management platform.

Apple Podcasts | Spotify | Google Podcasts

We continue the age-old monolith/microservice debate and dig into why companies seem to like services so much (I’m generally cautious about such migrations). Ganesh has a ton of insights into developer productivity and tooling to make engineering teams successful that we dive into.

Highlights

00:00 - Why solve the service management problem?

06:00 - When to drive a monolith → service migration? What inflection points should one think about to make that decision?

08:30 - What would Ganesh do his next service migration?

10:30 - What tools are useful when migrating to services?

12:00 - Standardizing infrastructure to facilitate migrations. How much to standardize (à la Google), to letting team make their own decisions (à la Amazon)?

17:30 - How does a tool like Cortex help with these problems?

21:30 - How opinionated should such tools be? How much user education is part of building such tools?

27:00 - What are the key cultural components of successful engineering teams?

31:00 - Tactically, what does good service management look like today?

37:00 - What’s the cost/benefit ratio of shipping an on-prem product vs. a SaaS tool?

41:30 - What would your advice be for the next software engineer embarking on their monolith → microservice migration?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Johannes Schindelin is the maintainer (BDFL) of Git for Windows.

Apple Podcasts | Spotify | Google Podcasts

Git is a fundamental piece of the software community, and we get to learn the history and inner workings of the project in this episode. Maintaining a widely-used open source project involves a ton of expected complexity around handling bug reports, deprecations, and inclusive culture, but also requires management of inter-personal relationships, ease of contribution, and other aspects that are fascinating to learn about.

Highlights

00:06 - How did Johannes end up as the maintainer of Git for Windows?

06:30 - The Git community in the early days. Fun fact: Git used to be called `dircache`

08:30 - How many downloads does Git for Windows get today?

10:15 - Why does Git for Windows a separate project? Why not make improvements to Git itself?

24:00 - How do you deprecate functionality when there are millions of users of your product and you have no telemetry?

30:00 - What does being the BDFL of a project mean? What does Johannes day-to-day look like?

33:00 - What is GitGitGadget? How does it make contributions easier?

41:00 - How do foster an inclusive community of an open-source project?

50:00 - What’s next for Git?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Guido van Rossum is the creator of the Python programming language and a Distinguished Engineer at Microsoft.

Apple Podcasts | Spotify | Google Podcasts

We discuss Guido’s new work on making CPython faster (PEP 659), Tiers of Python Interpreter Execution, and high impact, low hanging fruit performance improvements.

Highlights

(an edited summary)

[00:21] What got you interested in working on Python performance?

Guido: In some sense, it was probably a topic that was fairly comfortable to me because it means working with a core of Python, where I still feel I know my way around. When I started at Microsoft, I briefly looked at Azure but realized I never enjoyed that kind of work at Google or Dropbox. Then I looked at Machine Learning, but it would take a lot of time to do something interested with the non-Python, and even Python-related bits.

[02:31] What was different about the set of Mark Shannon’s ideas on Python performance that convinced you to go after them?

Guido: I liked how he was thinking about the problem. Most of the other approaches around Python performance like PyPy and Cinder are not suitable for all use cases since they aren’t backward compatible with extension modules. Mark has the perspective and experience of a CPython developer, as well as a viable approach that would maintain backward compatibility, which is the hardest problem to solve. The Python Bytecode interpreter is modified often across minor releases (for eg: 3.8 → 3.9) for various reasons like new opcodes, so modifying that is a relatively safe approach.

Utsav: [09:45] Could you walk us through the idea of the tiers of execution of the Python Interpreter?

Guido: When you execute a program, you don't know if it's going to crash after running a fraction of a millisecond, or whether it's going to be a three-week-long computation. Because it could be the same code, just in the first case, it has a bug. And so, if it takes three weeks to run the program, maybe it would make sense to spend half an hour ahead of time optimizing all the code that's going to be run. But obviously, especially in dynamic languages like Python, where we do as much as we can without asking the user to tell us exactly how they need it done, you just want to start executing code as quickly as you can. So that if it's a small script, or a large program that happens to fail early, or just exits early for a good reason, you don't spend any time being distracted by optimizing all that code.

So, what we try to do there is keep the bytecode compiler simple so that we get to execute the beginning of the code as soon as possible. If we see that certain functions are being executed many times over, then we call that a hot function, and some definition of “hot”. For some purposes, maybe it's a hot function if it gets called more than once, or more than twice, or more than 10 times. For other purposes, you want to be more conservative, and you can say, “Well, it's only hot if it's been called 1000 times.”

The specializing adaptive compiler (PEP 659) then tries to replace certain bytecodes with bytecodes that are faster, but only work if the types of the arguments are specific types. A simple hypothetical example is the plus operator in Python. It can add lots of things like integers, strings, lists, or even tuples. On the other hand, you can't add an integer to a string. So, the optimization step - often called quickening, but usually in our context, we call it specializing - is to have a separate “binary add” integer bytecode, a second-tier bytecode hidden from the user. This opcode assumes that both of its arguments are actual Python integer objects, reaches directly into those objects to find the values, adds those values together in machine registers, and pushes the result back on the stack. 

The binary adds integer operation still has to make a type check on the arguments. So, it's not completely free but a type check can be implemented much faster than a sort of completely generic object-oriented dispatch, like what normally happens for most generic add operations.

Finally, it's always possible that a function is called millions of times with integer arguments, and then suddenly a piece of data calls it with a floating-point argument, or something worse. At that point, the interpreter will simply execute the original bytecode. That's an important part so that you still have the full Python semantics.

Utsav [18:20] Generally you hear of these techniques in the context of JIT, a Just-In-Time compiler, but that’s not being implemented right now.

Just-In-Time compilation has a whole bunch of emotional baggage with it at this point that we're trying to avoid. In our case, it’s unclear what and when we’re exactly compiling. At some point ahead of program execution, we compile your source code into bytecode. Then we translate the bytecode into specialized bytecode. I mean, everything happens at some point during runtime, so which part would you call Just-In-Time?

Also, it’s often assumed that Just-In-Time compilation automatically makes all your code better. Unfortunately, you often can't actually predict what the performance of your code is going to be. And we have enough of that with modern CPUs and their fantastic branch prediction. For example, we write code in a way that we think will clearly reduce the number of memory accesses. When we benchmark it, we find that it runs just as fast as the old unoptimized code because the CPU figured out access patterns without any of our help. I wish I knew what went on in modern CPUs when it comes to branch prediction and inline caching because that is absolute magic.

Full Transcript

Utsav: [00:14] Thank you, Guido, for joining me on another episode of the Software at Scale podcast. It's great to have you here. 

Guido: [00:20] Great to be here on the show. 

Utsav: [00:21] Yeah. And it's just fun to talk to you again. So, the last time we spoke was at Dropbox many, many years ago. And you got retired, and then you decided that you wanted to do something new. And you work on performance now at Microsoft, and that's amazing. So, to start off with, I just want to ask you, you could pick any project that you wanted to, based on some slides that I've seen. So, what got you interested in working on Python performance?

Guido: [00:47] In some sense, it was probably a topic that was fairly comfortable to me because it means working with a core of Python, where I still feel I know my way around. Some other things I considered briefly in my first month at Microsoft, I looked into, “Well, what can I do with Azure?”, and I almost immediately remembered that I was not cut out to be a cloud engineer. That was never the fun part of my job at Dropbox. It wasn't the fun part of my job before that at Google either. And it wouldn't be any fun to do that at Microsoft. So, I gave up on that quickly. I looked in machine learning, which I knew absolutely nothing about when I joined Microsoft. I still know nothing, but I've at least sat through a brief course and talked to a bunch of people who know a lot about it. And my conclusion was actually that it's a huge field. It is mostly mathematics and statistics and there is very little Python content in the field. And it would take me years to do anything interesting with the non-Python part and probably even with the Python part, given that people just write very simple functions and classes, at best in their machine learning code. But at least I know a bit more about the terminology that people use. And when people say kernel, I now know what they mean. Or at least I'm not confused anymore as I was before.

Utsav: [02:31] That makes sense. And that is very similar to my experience with machine learning. Okay, so then you decided that you want to work on Python performance, right? And then you are probably familiar with Mark Shannon's ideas?

Guido: [02:43] Very much so. Yeah.

Utsav: [02:44] Yeah. So, was there anything different about the set of ideas that you decided that this makes sense and I should work on a project to implement these ideas?

Guido: [02:55] Mark Shannon's ideas are not unique, perhaps, but I know he's been working on for a long time. I remember many years ago, I went to one of the earlier Python UK conferences, where he gave a talk about his PhD work, which was also about making Python faster. And over the years, he's never stopped thinking about it. And he sort of has a holistic attitude about it. Obviously, the results remain to be seen, but I liked what he was saying about how he was thinking about it. And if you take PyPy, it has always sounded like PyPy is sort of a magical solution that only a few people in the world understand how it works. And those people built that and then decided to do other things. And then they left it to a team of engineers to solve the real problems with PyPy, which are all in the realm of compatibility with extension modules. And they never really solved that. 

[04:09] So you may remember that there was some usage of PyPy at Dropbox because there was one tiny process where someone had discovered that PyPy was actually so much faster that it was worth it. But it had to run in its own little process and there was no maintenance. And it was a pain, of course, to make sure that there was a version of PyPy available on every machine. Because for the main Dropbox application, we could never switch to PyPy because that depended on 100 different extension modules. And just testing all that code would take forever. 

[04:49] I think since we're talking about Dropbox, Pyston was also an interesting example. They've come back actually; you've probably heard that. The Pyston people were much more pragmatic, and they've learned from PyPy’s failures. 

[05:04] But they have always taken this attitude of, again, “we're going to start with CPython,” which is good because that way they are sort of guaranteed compatibility with extension modules. But still, they make these huge sets of changes, at least Pyston one, and they had to roll back a whole bunch of things because, again, of compatibility issues, where I think one of the things, they had a bunch of very interesting improvements to the garbage collection. I think they got rid of the reference counting, though. And because of that, the behavior of many real-world Python programs was completely changed. 

[05:53] So why do I think that Mark's work will be different or Mark's ideas? Well, for one, because Mark has been in Python core developer for a long time. And so, he knows what we're up against. He knows how careful we have with backwards compatibility. And he knows that we cannot just say get rid of reference counting or change the object layout. Like there was a project that was recently released by Facebook basically, was born dead, or at least it was revealed to the world in its dead form, CI Python (Cinder), which was a significantly faster Python implementation, but using sort of many of the optimizations came from changes in object layout that just aren't compatible with extension modules. And Mark has sort of carved out these ideas that work on the bytecode interpreter itself. 

[06:58] Now, the bytecode is something where we know that it's not going to sort of affect third-party extension modules too much if we change it, because the bytecode changes in every Python release. And internals of the interpreter of the bytecode interpreter, change in every Python release. And yes, we still run into the occasional issue. Every release, there is some esoteric hack that someone is using that breaks. And they file an issue in the bug tracker because they don't want to research or they haven't yet researched what exactly is the root cause of the problem, because all they know is their users say, “My program worked in Python 3.7, and it broke in Python 3.8. So clearly, Python 3.8 broke something.” And since it only breaks when we're using Library X, it must be maybe Library X's fault. But Library X, the maintainers don't know exactly what's going on because the user just says it doesn't work or give them a thousand-line traceback. And they bounce it back to core Python, and they say, “Python 3.8 broke our library for all our users, or 10% of our users,” or whatever. 

[08:16] And it takes a long time to find out, “Oh, yeah, they're just poking inside one of the standard objects, using maybe information they gleaned from internal headers, or they're calling a C API that starts with an underscore.” And you're not supposed to do that. Well, you can do that but then you pay the price, which is you have to fix your code at every next Python release. And in between, sort of for bug fix releases like if you go from 3.8.0 to 3.8.1, all the way up to 3.8.9, we guarantee a lot more - the bytecodes stay stable. But 3.9 may break all your hacks and it changes the bytecode. One thing we did I think in 3.10, was all the jumps in the bytecode are now counted in instructions rather than bytes, and instructions are two bytes. Otherwise, the instruction format is the same, but all the jumps jump a different distance if you don't update your bytecode. And of course, the Python bytecode compiler knows about this. But people who generate their own bytecode as a sort of the ultimate Python hack would suffer.

Utsav: [09:30] So the biggest challenge by far is backwards compatibility.

Guido: [09:34] It always is. Yeah, everybody wants their Python to be faster until they find out that making it faster also breaks some corner case in their code.

Utsav: [09:45] So maybe you can walk us through the idea of the tiers of execution or tiers of the Python interpreter that have been described in some of those slides.

Guido: [09:54] Yeah, so that is a fairly arbitrary set of goals that you can use for most interpreted languages. 

Guido: [10:02] And it's actually a useful way to think about it. And it's something that we sort of plan to implement, it's not that there are actually currently tiers like that. At best, we have two tiers, and they don't map perfectly to what you saw in that document. But the basic idea is-- I think this also is implemented in .NET Core. But again, I don't know if it's sort of something documented, or if it's just this is how their optimizer works. So, when you just start executing a program, you don't know if it's going to crash after running a fraction of a millisecond, or whether it's going to be a three-week-long computation. Because it could be the same code, just in the first case, it has a bug. And so, if it takes three weeks to run the program, maybe it would make sense to spend half an hour ahead of time optimizing all the code that's going to be run. But obviously, especially in dynamic language, and something like Python, where we do as much as we can without asking the user to tell us exactly how they need it done, you just want to start executing the code as quickly as you can. So that if it's a small script, or a large program that happens to fail early, or just exits early for a good reason, you don't spend any time being distracted by optimizing all that code. 

[11:38] And so if this was a statically compiled language, the user would have to specify that basically, when they run the compiler, they say, “Well, run a sort of optimize for speed or optimize for time, or O2, O3 or maybe optimized for debugging O0.” 

In Python, we try not to bother the user with those decisions. So, you have to generate bytecode before you can execute even the first line of code. So, what we try to do there is keep the bytecode compiler simple, keep the bytecode interpreter simple, so that we get to execute the beginning of the code as soon as possible. If we see that certain functions are being executed many times over, then we call that a hot function, and you can sort of define what's hot. For some purposes, maybe it's a hot function if it gets called more than once, or more than twice, or more than 10 times. For other purposes, you want to be more conservative, and you can say, “Well, it's only hot if it's been called 1000 times.” 

[12:48] But anyway, for a hot function, you want to do more work. And so, the specializing adaptive compiler, at that point, tries to replace certain bytecodes with bytecodes that are faster, but that work only if the types of the arguments are specific types. A simple example but pretty hypothetical is the plus operator in Python at least, can add lots of things. It can add integers, it can add floats, it can add strings, it can list or tuples. On the other hand, you can't add an integer to a string, for example. So, what we do there, the optimization step - and it's also called quickening, but usually in our context, we call it specializing - is we have a separate binary add integer bytecode. And it's sort of a second-tier bytecode that is hidden from the user. If the user asked for the disassembly of their function, they will never see binary add integer, they will also always see just binary add. But what the interpreter sees once the function has been quickened, the interpreter may see binary add integers. And the binary add integer just assumes that both of its arguments, that's both the numbers on the stack, are actual Python integer objects. It just reaches directly into those objects to find the values, adds those values together in machine registers, and push the result back on the stack. 

[14:35] Now, there are all sorts of things that make that difficult to do. For example, if the value doesn't fit in a register for the result, or either of the input values, or maybe even though you expected it was going to be adding two integers, this particular time it's going to add to an integer and a floating-point or maybe even two strings. 

[15:00] So the first stage of specialization is actually… I'm blanking out on the term, but there is an intermediate step where we record the types of arguments. And during that intermediate step, the bytecode actually executes slightly slower than the default bytecode. But that only happens for a few executions of a function because then it knows this place is always called with integers on the stack, this place is always called with strings on the stack, and maybe this place, we still don't know or it's a mixed bag. And so then, the one where every time it was called during this recording phase, it was two integers, we replace it with that binary add integer operation. The binary adds integer operation, then, before it reaches into the object, still has to make a type check on the arguments. So, it's not completely free but a type check can be implemented much faster than a sort of completely generic object-oriented dispatch, like what normally happens for the most generic binary add operations. 

[16:14] So once we've recorded the types, we specialize it based on the types, and the interpreter then puts in guards. So, the interpreter code for the specialized instruction has guards that check whether all the conditions that will make the specialized instruction work, are actually met. If one of the conditions is not met, it's not going to fail, it's just going to execute the original bytecode. So, it's going to fall back to the slow path rather than failing. That's an important part so that you still have the full Python semantics. And it's always possible that a function is called hundreds or millions of times with integer arguments, and then suddenly a piece of data calls it with a floating-point argument, or something worse. And the semantics still say, “Well, then it has to do with the floating-point way.

Utsav: [17:12] It has to deoptimize, in a sense.

Guido: [17:14] Yeah. And there are various counters in all the mechanisms where, if you encounter something that fails the guard once, that doesn't deoptimize the whole instruction. But if you sort of keep encountering mismatches of the guards, then eventually, the specialized instruction is just deoptimized and we go back to, “Oh, yeah, we'll just do it the slow way because the slow way is apparently the fastest, we can do.” 

Utsav: [17:45] It's kind of like branch prediction.

Guido: [17:47] I wish I knew what went on in modern CPUs when it comes to branch prediction and inline caching because that is absolute magic. And it's actually one of the things we're up against with this project, because we write code in a way that we think will clearly reduce the number of memory accesses, for example. And when we benchmark it, we find that it runs just as fast as the old unoptimized code because the CPU figured it out without any of our help. 

Utsav: [18:20] Yeah. I mean, these techniques, generally you hear them in a context of JIT, a Just-In-Time compiler, but y’all are not implementing that right now.

Guido: [18:30] JIT is like, yeah, in our case, it would be a misnomer. What we do expect to eventually be doing is, in addition to specialization, we may be generating machine code. That's probably going to be well past 3.11, maybe past 3.12. So, the release that we still have until October next year is going to be 3.11, and that's where the specializing interpreters going to make its first entry. I don't think that we're going to do anything with machine code unless we get extremely lucky with our results halfway through the year. But eventually, that will be another tier. But I don't know, Just-In-Time compilation has a whole bunch of emotional baggage with it at this point that we're trying to avoid.

Utsav: [19:25] Is it baggage from other projects trying it?

Guido: [19:29] People assume that Just-In-Time compilation automatically makes all your code better. It turns out that it's not that simple. In our case, compilation is like, “What exactly is it that we compile?” At some point ahead of time, we compile your source code into bytecode. Then we translate the bytecode into specialized bytecode. I mean, everything happens at some point during runtime, so which thing would you call Just-In-Time? 

Guido: [20:04] So I'm not a big fan of using that term. And it usually makes people think of feats of magical optimization that have been touted by the Java community for a long time. And unfortunately, the magic is often such that you can't actually predict what the performance of your code is going to be. And we have enough of that, for example, with the modern CPUs and their fantastic branch prediction.

Utsav: [20:35] Speaking of that, I saw that there's also a bunch of small wins y'all spoke about, that y’all can use to just improve performance, things like fixing the place of __dict__ in objects and changing the way integers are represented. What is just maybe one interesting story that came out of that?

Guido: [20:53] Well, I would say calling Python functions is something that we actually are currently working on. And I have to say that this is not just the Microsoft team, but also other people in the core dev team, who are very excited about this and helping us in many ways. So, the idea is that in the Python interpreter, up to and including version 3.10, which is going to be released next week, actually, whenever you call a Python function, the first thing you do is create a frame object. And a frame object contains a bunch of state that is specific to that call that you're making. So, it points to the code object that represents the function that's being called, it points to the globals, it has a space for the local variables of the call, it has space for the arguments, it has space for the anonymous values on the evaluation stack. But the key thing is that it’s still a Python object. And there are some use cases where people actually inspect the Python frame objects, for example, if they want to do weird stuff with local variables. 

[22:18] Now, if you're a debugger, it makes total sense that you want to actually look at what are all the local variables in this frame? What are their names? What are their values and types? A debugger may even want to modify a local variable while the code is stopped in a breakpoint. That's all great. But for the execution of most code, most of the time, certainly, when you're not using a debugger, there's no reason that that frame needs to be a Python object. Because a Python object has a header, it has a reference count, it has a type, it is allocated as its own small segment of memory on the heap. It's all fairly inefficient. Also, if you call a function, then you create a few objects, then from that function, you call another function, all those frame objects end up scattered throughout the entire heap of the program. 

[23:17] What we have implemented in our version of 3.11, which is currently just the main branch of the CPython repo, is an allocation scheme where when we call a function, we still create something that holds the frame, but we allocate that in an array of frame structures. So, I can't call them frame objects because they don't have an object header, they don't have a reference count or type, it's just an array of structures. This means that unless that array runs out of space, calls can be slightly faster because you don't jump around on the heap. And allocation sort of is to allocate the next frame, you compare two pointers, and then you bump one counter, and now you have a new frame structure. And so, creation, and also deallocation of frames is faster. Frames are smaller because you don't have the object header. You also don't have the malloc overhead or the garbage collection overhead. And of course, it's backwards incompatible. So, what do we do now? Fortunately, there aren't that many ways that people access frames. And what we do is when people call an API that returns a frame object, we say, “Okay, well sure. Here's the frame in our array. Now we're going to allocate an object and we're going to copy some values to the frame object,” and we give that to the Python code. So, you can still introspect it and you can look at the locals as if nothing has changed. 

[25:04] But most of the time, people don't look at add frames. And this is actually an old optimization. I remember that the same idea existed in IronPython. And they did it differently. I think for them, it was like a compile-time choice when the bytecode equivalent in IronPython was generated for a function, it would dynamically make a choice whether to allocate a frame object or just a frame structure for that call. And their big bugaboo was, well, there is a function you can call sys dunder __getFrame__ and it just gives you the frame object. So, in the compiler, they were looking, were you using the exact thing named system dunder __getFrame__ and then they would say, “Oh, that's getFrame, now we're going to compile you slightly slower so you use a frame object.” We have the advantage that we can just always allocate the frame object on the fly. But we get similar benefits. And oh, yeah, I mentioned that the frame objects are allocated in array, what happens if that array runs out? Well, it's actually sort of a linked list of arrays. So, we can still create a new array of frames, like we have space for 100 or so which, in many programs, that's plenty. And if your call stack is more than 100 deep, we'll just have one discontinuity, but the semantics are still the same and we still have most of the benefits.

Utsav: [26:39] Yeah, and maybe as a wrap-up question, there are a bunch of other improvements happening in the Python community for performance as well, right? There's Mypyc, which we're familiar with, which is using types, Mypy types to maybe compiled code to basically speed up. Are there any other improvements like that, that you're excited about, or you're interested in just following?

Guido: [27:01] Well, Mypyc is very interesting. It gives much better performance boost, but only when you fully annotate your code and only when you actually follow the annotations precisely at runtime. In Mypy, if you say, “This function takes two integers,” and it returns an integer, then if you call it with something else, it's going to immediately blow up. It'll give you a traceback. But the standard Python semantics are that type annotations are optional, and sometimes they're white lies. And so, the types that you see at runtime may not actually be compatible with the types that were specified in the annotations. And it doesn't affect how your program executes. Unless you sort of start introspecting the annotations, your program runs exactly the same with or without annotations. 

[28:05] I mean, there are a couple of big holes that are in the type system, like any. And the type checker will say, “Oh, if you put any, everything is going to be fine.” And so, using that, it's very easy to have something that is passed, an object of an invalid type, and the type checker will never complain about it. And our promise is that the runtime will not complain about it either unless it really is a runtime error. Obviously, if you're somehow adding an integer to a string at runtime, it's still going to be a problem. But if you have a function that, say, computes the greatest common divisor of two numbers, which is this really cute little loop, if you define the percent operator in just the right way, you can pass in anything. I think there are examples where you can actually pass it to strings, and it will return a string without ever failing. 

[29:07] And so basically, Mypyc does things like the instance attributes are always represented in a compact way where there is no dunder __dict__. The best that we can do, which we are working on designing how we're actually going to do that, is make it so that if you don't look at the dunder __dict__ attribute, we don't necessarily have to store the instance attributes in a dictionary as long as we preserve the exact semantics. But if you use the dunder __dict__, at some later point, again, just like the frame objects, we have to materialize a dictionary. And Mypyc doesn't do that. It's super-fast if you don't use dunder __dict__. If you do use dunder __dict__, it just says, “dunder __dict__ not supported in this case.” 

[29:59] Mypyc really only compiles a small subset of the Python language. And that's great if that's the subset you're interested in. But I'm sure you can imagine how complex that is in practice for a large program.

Utsav: [30:17] It reminds me of JavaScript performance when everything is working fast and then you use this one function, which you're not supposed to use to introspect an object or something, and then performance just breaks down. 

Guido: [30:29] Yeah, that will happen. 

Utsav: [30:31] But it's still super exciting. And I'm also super thankful that Python fails loudly when you try to add a number in the string, not like JavaScript,

Guido: [30:41] Or PHP, or Perl.

Utsav: [30:44] But yeah, thank you so much for being a guest. I think this was a lot of fun. And I think it walked through the performance improvement y’all are trying to make in an accessible way. So, I think it’s going to be useful for a lot of people. Yeah, thank you for being a guest.

Guido: [30:58] My pleasure. It’s been a fun chat.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Abhay Venkatesh is a Software Engineer at Anduril Industries where he focuses on infrastructure and platform engineering.

Apple Podcasts | Spotify | Google Podcasts

We focus this episode on drone engineering - exploring the theme of “If I wanted to start my own technology project/company that manages drones, what technology bits would I need to know?” We discuss the commoditization of drone hardware, the perception stack, testing and release cycles, simulation software, software invariants, defensive software architecture, and wrap up with discussing the business models behind hardware companies.

Highlights

1:56 - Are we getting robot cleaners (other than Roomba) anytime soon?

5:00 - What should I do if I want to build a technology project/company that leverages drones? Where should I be innovating?

7:30 - What does the perception stack for a drone look like?

13:30 - Are drones/robots still programmed in C++? How is Rust looked at in their world?

18:30 - What does software development look like for a company that deploys software on drones? What are the testing/release processes like?

20:30 - How are simulations used? Can game engines be used for simulations to test drones? Interestingly - since neural networks perceive objects and images very differently from how brains do it, adapting drone perception to work on a game engine is actually really hard.

26:30 - Drone programming can be similar to client-side app development. But you have to write your own app store/auto-update infrastructure. Testing new releases manually is the largest bottleneck in releases.

30:00 - Defensive programming for drones - how do you ensure safety? What is the base safety layer that needs to be built for a drone? “Return to Base” logic - often separated out into a different CPU.

33:00 - How do hardware businesses look different from traditional SaaS businesses?

38:00 - What are some interesting trends in hardware that Abhay is excited about?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Derrick Stolee is a Principal Software Engineer at GitHub, where he focuses on the client experience of large Git repositories.

Apple Podcasts | Spotify | Google Podcasts

Subscribers might be aware that I’ve done some work on client-side Git in the past, so I was pretty excited for this episode. We discuss the Microsoft Windows and Office repository’s migrations to Git, recent performance improvements to Git for large monorepo, and more.

Highlights

lightly edited

[06:00] Utsav: How and why did you transition from academia to software engineering?

Derrick Stolee: I was teaching and doing research at a high level and working with really great people. And I found myself not finding the time to do the work I was doing as a graduate student. I wasn't finding time to do the programming and do these really deep projects. I found that the only time I could find to do that was in the evenings and weekends because that's when other people weren't working, who could collaborate with me on their projects and move those projects forward. And then, I had a child and suddenly my evenings and weekends aren't available for that anymore.

And so the individual things I was doing just for myself and for, you know, that was more programming oriented, fell by the wayside. I'd found myself a lot less happy with that career. And so I decided, you know what, there are two approaches I could take here. One is I could spend the next year or two winding down my collaborations and spinning up more of this time to be working on my own during regular work hours. Or I could find another job and I was going to set out.

And, I lucked out that Microsoft has an office here in Raleigh, North Carolina, where we now live. This is where Azure DevOps was being built and they needed someone to help solve some graph problems. So it was really nice that it happened to work out that way. I know for a fact that they took a chance on me because of their particular need. I didn't have significant professional experience in the industry.

[21:00] Utsav: What drove the decision to migrate Windows to Git?

The Windows repository moving to Git was a big project driven by Brian Harry, who was the CVP of Azure DevOps at the time. Previously, Windows used this source control system called Source Depot, which was a fork of Perforce. No one knew how to use this version control system until they got there and learned on the job, and that caused some friction in terms of onboarding people.

But also if you have people working in the windows code base for a long time, they only learn this version control system. They don't know Git and they don't know what everyone else is using. And so they're feeling like they're falling behind and they're not speaking the same language when they talk to somebody else who's working commonly used version control tools. So they saw this as a way to not only update the way their source control works to a more modern tool but specifically allow this more free exchange of ideas and understanding.

The Windows Git repository is going to be big and have some little tweaks here and there, but at the end of the day, you're just running Git commands and you can go look at StackOverflow to solve questions as opposed to needing to talk to specific people within the Windows organization and how to use this version control tool.

Transcript

Utsav Shah: Welcome to another episode of the Software at Scale Podcast, joining me today is Derek Stolee, who is a principal software engineer at GitHub. Previously, he was a principal software engineer at Microsoft, and he has a Ph.D. in Mathematics and Computer Science from the University of Nebraska, welcome. 

Derek Stolee: Thanks, happy to be here. 

Utsav Shah: So a lot of work that you do on Git, from my understanding, it's similar to the work you did in your Ph.D. around graph theory and stuff. So maybe you can just walk through the initial like, what got you interested in graphs and math in general?

Derek Stolee: My love of graph theory came from my first algorithms class in college my sophomore year, just doing simple things like path-finding algorithms. And I got so excited about it, I started clicking around Wikipedia constantly, I just read every single article I could find on graph theory. So I learned about the four-color theorem, and I learned about different things like cliques, and all sorts of different graphs, the Peterson graph, and I just kept on discovering more. I thought this is interesting to me, it works well with the way my brain works and I could just model these things while [unclear 01:32]. And as I kept on doing more, for instance, graph theory, and combinatorics, my junior year for my math major, and it's like I want to pursue this. Instead of going into the software, I had planned with my undergraduate degree, I decided to pursue a Ph.D. in first math, then I split over to the joint math and CS program, and just worked on very theoretical math problems but I also would always pair it with the fact that I had this programming background and algorithmic background. 

So I was solving pure math problems using programming, and creating these computational experiments, the thing I call it was computational competent works. Because I would write these algorithms to help me solve these problems that were hard to reason about because the cases just became too complicated to hold in your head. But if you could quickly write a program, to then over the course of a day of computation, discover lots of small examples that can either answer it for you or even just give us a more intuitive understanding of the problem you're trying to solve and that was my specialty as I was working in academia.

Utsav Shah: You hear a lot about proofs that are just computer-assisted today and you could just walk us through, I'm guessing, listeners are not math experts. So why is that becoming a thing and just walk through your thesis read in super layman terms, what do you do?

Derek Stolee: There are two very different ways what you can mean when you say I have automated proof, there are some things like Coke, which are completely automated formal logic proofs, which specify all the different axioms and the different things I know to be true. And the statement I want to prove and constructs the sequence of proof steps, what I was focused more on was taking a combinatorial problem. For instance, do graphs with certain sub-structures exist, and trying to discover those examples using an algorithm that was finely tuned to solve those things, so one problem was called uniquely Kr saturated graphs. A Kr was essentially a set of our vertices where every single pair was adjacent to each other and to be saturated means I don't have one inside my graph but if I add any missing edge, I'll get one. And then the uniquely part was, I'll get exactly one and now we're at this fine line of doing these things even exist and can I find some interesting examples. And so you can just do, [unclear 04:03] generate every graph of a certain size, but that blows up in size. 

And so you end up where you can get maybe to 12 vertices, every graph of up to 12 vertices or so you can just enumerate and test. But to get beyond that, and find the interesting examples, you have to be zooming in on the search space to focus on the examples you're looking for. And so I generate an algorithm that said, Well, I know I'm not going to have every edge, so it's fixed one, parents say, this isn't an edge. And then we find our minus two other vertices and put all the other edges in and that's the one unique completion of that missing edge. And then let's continue building in that way, by building up all the possible ways you can create those sub-structures because they need to exist as opposed to just generating random little bits and that focus the search space enough that we can get to 20 or 21 vertices and see this interesting shapes show up. From those examples, we found some infinite families and then used regular old-school math to prove that these families were infinite once we had those small examples to start from.

Utsav Shah: That makes a lot of sense and that tells me a little bit about how might someone use this in a computer science way? When would I need to use this in let's say, not my day job but just like, what computer science problems would I solve given something like that?

Derek Stolee: It's always asking a mathematician what the applications of the theoretical work are. But I find whenever you see yourself dealing with a finite problem, and you want to know what different ways can this data be up here? Is it possible with some constraints? So a lot of things I was running into were similar problems to things like integer programming, trying to find solutions to an integer program is a very general thing and having those types of tools in your back pocket to solve these problems is extremely beneficial. And also knowing integer programming is still NP-hard. So if you have the right data shape, it will take an exponential amount of time to work, even though there are a lot of tools to solve most cases, when your data looks aren't particularly structured to have that exponential blow up. So knowing where those data shapes can arise and how to take a different approach can be beneficial.

Utsav Shah: And you've had a fairly diverse career after this. I'm curious, what was the difference? What was the transition from doing this stuff to get or like developer tools? How did that end up happening?

Derek Stolee: I was lucky enough that after my Ph.D. was complete, I landed a tenure track job in a math and computer science department, where I was teaching and doing research at a high level and working with great people. I had the best possible accountant’s workgroup, I could ask for doing interesting stuff, working with graduate students. And I found myself not finding the time to do the work I was doing as a graduate student, I wasn't finding time to do the programming and do these deep projects I wanted, I had a lot of interesting math project projects, I was collaborating with a lot of people, I was doing a lot of teaching. But I was finding that the only time I could find to do that was in evenings and weekends because that's when other people weren't working, who could collaborate with me on their projects and move those projects forward. And then I had a child and suddenly, my evenings and weekends aren't available for that anymore. And so the individual things I was doing just for myself, and for that we're more programming oriented, fell by the wayside and found myself a lot less happy with that career. And so I decided, there are two approaches I could take here; one is I could spend the next year or two, winding down my collaborations and spinning up more of this time to be working on my own during regular work hours, or I could find another job. 

And I was going to set out, but let's face it, my spouse is also an academic and she had an opportunity to move to a new institution and that happened to be soon after I made this decision. And so I said, great, let's not do the two-body problem anymore, you take this job, and we move right in between semesters, during the Christmas break, and I said, I will find my job, I will go and I will try to find a programming job, hopefully, someone will be interested. And I lucked out that, Microsoft has an office here in Raleigh, North Carolina, where we now live and they happen to be the place where what is now known as Azure DevOps was being built. And they needed someone to help solve some graph theory problems in the Git space. So it was nice that it happened to work out that way and I know for a fact that they took a chance on me because of their particular need. I didn't have significant professional experience in the industry, I just said, I did academics, so I'm smart and I did programming as part of my job, but it was always about myself. So, I came with a lot of humility, saying, I know I'm going to learn to work with a team. in a professional setting. I did teamwork with undergrad, but it's been a while. 

So I just come in here trying to learn as much as I can, as quickly as I can, and contribute in this very specific area you want me to go into, and it turns out that area they needed was to revamp the way Azure Repos computed Git commit history, which is a graph theory problem. The thing that was interesting about that is the previous solution is that they did everything in the sequel they'd when you created a new commit, he would say, what is your parent, let me take its commit history out of the sequel, and then add this new commit, and then put that back into the sequel. And it took essentially a sequel table of commit IDs and squashes it into a varbinary max column of this table, which ended up growing quadratically. And also, if you had a merge commit, it would have to take both parents and interestingly merge them, in a way that never matched what Git log was saying. And so it was technically interesting that they were able to do this at all with a sequel before I came by. 

But we need to have the graph data structure available, we need to dynamically compute by walking commits, and finding out how these things work, which led to creating a serialized commit-graph, which had that topological relationship encoded in concise data, into data. That was a data file that would be read into memory and very quickly, we could operate on it and do things topologically sorted. And we could do interesting File History operations on that instead of the database and by deleting these Database entries that are growing quadratically, we saved something like 83 gigabytes, just on the one server that was hosting the Azure DevOps code. And so it was great to see that come into fruition.

Utsav Shah: First of all, that's such an inspiring story that you could get into this, and then they give you a chance as well. Did you reach out to a manager? Did you apply online? I'm just curious how that ended up working? 

Derek Stolee: I do need to say I have a lot of luck and privilege going into this because I applied and waited a month and didn't hear anything.  I had applied to the same group and said,  here's my cover letter, I heard nothing but then I have a friend who was from undergrad, who was one of the first people I knew to work at Microsoft. And I knew he worked at this little studio as the Visual Studio client editor and I said, well, this thing, that's now Azure DevOps was called Visual Studio online at the time, do you know anybody from this Visual Studio online group, I've applied there, haven't heard anything I'd love if you could get my resume on the top list. And it turns out that he had worked with somebody who had done the Git integration in Visual Studio, who happened to be located at this office, who then got my name on the top of the pile. And then that got me to the point where I was having a conversation with who would be my skip-level manager, and honestly had a conversation with me to try to suss out, am I going to be a good team player? 

There's not a good history of PhDs working well with engineers, probably because they just want to do their academic work and work in their space. I remember one particular question is like,  sometimes we ship software and before we do that, we all get together, and everyone spends an entire day trying to find bugs, and then we spend a couple of weeks trying to fix them, they call it a bug bash, is that something you're interested in doing? I'm 100% wanting to be a good citizen, good team member, I am up for that. I that's what it takes to be a good software engineer, I will do it. I could sense the hesitation and the trepidation about looking at me more closely but it was overall, once I got into the interview, they were still doing Blackboard interviews at that time and I felt unfair because my phone screen interview was a problem. I had assigned my C Programming students as homework, so it's like sure you want to ask me this, I have a little bit of experience doing problems like this. So I was eager to show up and prove myself, I know I made some very junior mistakes at the beginning, just what's it like to work on a team? What's it like to check in a change and commit that pull request at 5 pm? And then go and get in your car and go home and realize when you are out there that you had a problem? And you've caused the bill to go red? Oh, no, don't do that. So I had those mistakes, but I only needed to learn them once. 

Utsav Shah: That's amazing and going to your second point around [inaudible 14:17], get committed history and storing all of that and sequel he also go, we had to deal with an extremely similar problem because we maintain a custom CI server and we try doing Git [inaudible 14:26] and try to implement that on our own and that did not turn out well. So maybe you can walk listeners through like, why is that so tricky? Why it is so tricky to say, is this commit before another commit is that after another commit, what's the parent of this commit?  What's going on, I guess?

Derek Stolee: Yes the thing to keep in mind is that each commit has a list of a parent or multiple parents in the case of emerging, and that just tells you what happened immediately before this. But if you have to go back weeks or months, you're going to be traversing hundreds or 1000s of commits and these merge commits are branching. And so not only are we going deep in time in terms of you just think about the first parent history is all the merge all the pull requests that have merged in that time. But imagine that you're also traversing all of the commits that were in the topic branches of those merges and so you go both deep and wide when you're doing this search. And by default, Git is storing all of these commits as just plain text objects, in their object database, you look it up by its Commit SHA, and then you go find that location in a pack file, you decompress it, you go parse the text file to find out the different information about, what's its author-date, committer date, what are its parents, and then go find them again, and keep iterating through that. And it's a very expensive operation on these orders of commits and especially when it says the answer's no, it's not reachable, you have to walk every single possible commit that is reachable before you can say no. 

And both of those things cause significant delays in trying to answer these questions, which was part of the reason for the commit-graph file. First again, it was started when I was doing Azure DevOps server work but it's now something it's a good client feature, first, it avoids that going through to the pack file, and loading this plain text document, you have to decompress and parse by just saying, I've got it well-structured information, that tells me where in the commit-graph files the next one. So I don't have to store the whole object ID, I just have a little four-byte integer, my parent is this one in this table of data, and you can jump quickly between them. And then the other benefit is, we can store extra data that are not native to the commit object itself, and specifically, this is called generation number. The generation number is saying, if I don't have any parents, my generation number is one, so I'm at level one. 

But if I have parents, I'm going to have one larger number than the maximum most parents, so if I have one parent is; one, now two, and then three, if I merge, and I've got four and five, I'm going to be six. And what that allows me to do is that if I see two commits, and one is generation number 10, and one is 11, then the one with generation number 10, can't reach the one with 11 because that means an edge would go in the wrong direction. It also means that if I'm looking for the one with the 11, and I started at 20, I can stop when I hit commits that hit alright 10. So this gives us extra ways of visiting fewer commits to solve these questions.

Utsav Shah: So maybe a basic question, why does the system care about what the parents of a commit are why does that end up mattering so much?

Derek Stolee: Yes, it matters for a lot of reasons. One is if you just want to go through the history of what changes have happened to my repository, specifically File History, the way to get them in order is not you to say, give me all the commits that changed, and then we sort them by date because the commit date can be completely manufactured. And maybe something that was committed later emerged earlier, that's something else. And so by understanding those relationships of where the parents are, you can realize, this thing was committed earlier, it landed in the default branch later and I can see that by the way that the commits are structured to these parent relationships. And a lot of problems we see with people saying, where did my change go, or what happened here, it's because somebody did a weird merge. And you can only find it out by doing some interesting things with Git log to say, this merge caused a problem and cause your file history to get mixed up and if somebody resolved the merging correctly to cause this problem where somebody change got erased and you need to use these social relationships to discover that.

Utsav Shah: Should everybody just be using rebase versus merge, what's your opinion?

Derek Stolee: My opinion is that you should use rebase to make sure that the commits that you are trying to get reviewed by your coworkers are as clear as possible. Present a story, tell me that your commits are good, tell me in the comments just why you're trying to do this one small change, and how the sequence of commits creates a beautiful story that tells me how I get from point A to point B. And then you merge it into your branch with everyone else's, and then those commits are locked, you can't change them anymore. Do you not rebase them? Do you not edit them? Now they're locked in and the benefit of doing that as well, I can present this best story that not only is good for the people who are reviewing it at the moment, but also when I go back in history and say, why did I change it that way? You've got all the reasoning right there but then also you can do things like go down Do Git log dash the first parent to just show me which pull requests are merged against this branch. And that's it, I don't see people's commits. I see this one was merged, this one was merged, this one was merged and I can see the sequence of those events and that's the most valuable thing to see.

Utsav Shah: Interesting, and then a lot of GitHub workflows, just squash all of your commits into one, which I think is the default, or at least a lot of people use that; any opinions on that, because I know the Git workflow for development does the whole separate by commits, and then merge all of them, do you have an opinion, just on that?

Derek Stolee: Squash merges can be beneficial; the thing to keep in mind is that it's typically beneficial for people who don't know how to do interactive rebase. So their topic match looks like a lot of random commits that don't make a lot of sense. And they're just, I tried this and then I had a break. So I fixed a bug, and I kept on going forward, I'm responding to feedback and that's what it looks like. That's if those commits aren't going to be helpful to you in the future to diagnose what's going on and you'd rather just say, this pull request is the unit of change. The squash merge is fine, it's fine to do that, the thing I find out that is problematic as a new user is also then don't realize that they need to change their branch to be based on that squash merge before they continue working. Otherwise, they'll bring in those commits again, and their pull request will look very strange. So there are some unnatural bits to using squash merge, that require people to like, let me just start over from the main branch again, to do my next work. And if you don't remember to do that, it's confusing.

Utsav Shah: Yes, that makes a lot of sense. So going back to your story, so you started working on improving, get interactions in Azure DevOps? When did the whole idea of let's move the windows repository to get begin and how did that evolve?

Derek Stolee: Well, the biggest thing is that the windows repository moving to get was decided, before I came, it was a big project by Brian Harry, who was the CVP of Azure DevOps at the time. Windows was using this source control system called source depot, which was a literal fork of Perforce. And no one knew how to use it until they got there and learn on the job. And that caused some friction in terms of well, onboarding people is difficult. But also, if you have people working in the windows codebase, for a long time, they learn this version control system. They don't know what everyone else is using and so they're feeling like they're falling behind. And they're not speaking the same language as when they talk to somebody else who's working in the version control that most people are using these days. So they saw this as a way to not only update the way their source control works to a more modern tool but specifically Git because it allowed more free exchange of ideas and understanding, it's going to be a mono repo, it's going to be big, it's going to have some little tweaks here and there. 

But at the end of the day, you're just running Git commands and you can go look at Stack Overflow, how to solve your Git questions, as opposed to needing to talk to specific people within the windows organization, and how to use this tool. So that, as far as I understand was a big part of the motivation, to get it working. When I joined the team, we were in the swing of let's make sure that our Git implementation scales, and the thing that's special about Azure DevOps is that it's using, it doesn't use the core Git codebase, it has a complete reimplementation of the server-side of Git in C sharp. So it was rebuilding a lot of things to just be able to do the core features, but is in its way that worked in its deployment environment and it had done a pretty good job of handling scale. But the issues that the Linux repo was still a challenge to host. At that time, it had half a million commits, maybe 700,000 commits, and it's the site number of files is rather small. But we were struggling especially with the commit history being so deep to do that, but also even when they [inaudible 24:24] DevOps repo with maybe 200 or 300 engineers working on it and in their daily work that was moving at a pace that was difficult to keep up with, so those scale targets were things we were daily dealing with and handling and working to improve and we could see that improvement in our daily lives as we were moving forward.

Utsav Shah: So how do you tackle the problem? You're on this team now and you know that we want to improve the scale of this because 2000 developers are going to be using this repository we have two or 300 people now and it's already not like perfect. My first impression is you sit and you start profiling code and you understand what's going wrong. What did you all do?

Derek Stolee: You're right about the profiler, we had a tool, I forget what it's called, but it would run on every 10th request selected at random, it would run a dot net profiler and it would save those traces into a place where we could download them. And so we can say, you know what Git commit history is slow. And now that we've written it in C sharp, as opposed to a sequel, it's the C sharp fault. Let's go see what's going on there and see if we can identify what are the hotspots, you go pull a few of those traces down and see what's identified. And a lot of it was chasing that like, I made this change. Let's make sure that the timings are an improvement, I see some outliers over here, they're still problematic, we find those traces and be able to go and identify that the core parts to change. Some of them are more philosophical, we need to change data structures, we need to introduce things like generation numbers, we need to introduce things like Bloom filters for filed history, nor to speed that up because we're spending too much time parsing commits and trees. 

And once we get to the idea that once we're that far, it was time to essentially say, let's assess whether or not we can handle the windows repo. And I think would have been January, February 2017. My team was tasked with doing scale testing in production, they had the full Azure DevOps server ready to go that had the windows source code in it didn't have developers using it, but it was a copy of the windows source code but they were using that same server for work item tracking, they had already transitioned, that we're tracking to using Azure boards. And they said, go and see if you can make this fall over in production, that's the only way to tell if it's going to work or not. And so a few of us got together, we created a bunch of things to use the REST API and we were pretty confident that the Git operation is going to work because we had a caching layer in front of the server that was going to avoid that. And so we just went to the idea of like, let's have through the REST API and make a few changes, and create a pull request and merge it, go through that cycle. 

We started by measuring how often developers would do that, for instance, in the Azure DevOps, and then scale it up and see where be going and we crashed the job agents because we found a bottleneck. Turns out that we were using lib Git to do merges and that required going into native code because it's a C library and we couldn't have too many of those running, because they each took a gig of memory. And so once this native code was running out, things were crashing and so we ended up having to put a limit on how that but it was like, that was the only Fallout and we could then say, we're ready to bring it on, start transitioning people over. And when users are in the product, and they think certain things are rough or difficult, we can address them. But right now, they're not going to cause a server problem. So let's bring it on. And so I think it was that a few months later that they started bringing developers from source depot into Git.

Utsav Shah: So it sounds like there was some server work to make sure that the server doesn't crash. But the majority of work that you had to focus on was Git inside. Does that sound accurate?

Derek Stolee: Before my time in parallel, is my time was the creation of what's now called VFS Forget, he was GVFs, at the time, realized that don't let engineers name things, they won't do it. So we've renamed it to VFS forget, it's a virtual file system Forget, a lot of [inaudible 28:44] because the source depot, version that Windows is using had a virtualized file system in it to allow people to only download a portion of the working tree that they needed. And they can build whatever part they were in, and it would dynamically discover what files you need to run that build. And so we did the same thing on the Git side, which was, let's make the Git client let's modify in some slight ways, using our fork of Git to think that all the files are there. And then when a file is [inaudible 29:26] we look through it to a file system event, it communicates to the dot net process that says, you want that file and you go download it from the Git server, put it on disk and tell you what its contents are and now you can place it and so it's dynamically downloading objects. 

This required aversion approach protocol that we call the GVFs protocol, which is essentially an early version of what's now called get a partial clone, to say, you can go get the commits and trees, that's what you need to be able to do most of your work. But when you need the file contents into the blob of a file, we can download that as necessary and populate it on your disk. The different thing is that personalized thing, the idea that if you just run LS at the root directory, it looks like all the files are there. And that causes some problems if you're not used to it, like for instance, if you open the VS code in the root of your windows source code, it will populate everything. Because VS code starts crawling and trying to figure out I want to do searching and indexing. And I want to find out what's there but Windows users were used to this, the windows developers; they had this already as a problem. 

So they were used to using tools that didn't do that but we found that out when we started saying, VFS forget is this thing that Windows is using, maybe you could use it to know like, well, this was working great, then I open VS code, or I ran grep, or some other tool came in and decided to scan everything. And now I'm slow again, because I have absolutely every file in my mana repo, in my working directory for real. And so that led to some concerns that weren’t necessarily the best way to go. But it did specifically with that GFS protocol, it solved a lot of the scale issues because we could stick another layer of servers that were closely located to the developers, like for instance, get a lab of build machines, let's take one of these cache servers in there. So the build machines all fetch from there and there you have quick throughput, small latency. And they don't have to bug the origin server for anything but the Refs, you do the same thing around the developers that solved a lot of our scale problems because you don't have these thundering herds of machines coming in and asking for all the data all at once.

Utsav Shah: If we had a super similar concept of repository mirrors that would be listening to some change stream every time anything changed on a region, it would run GitHub, and then all the servers. So it's remarkable how similar the problems that we're thinking about are. One thing that I was thinking about, so VFS Forget makes sense, what's the origin of the FS monitor story? So for listeners, FS monitor is the File System Monitor in Git that decides whether files have changed or not without running [inaudible 32:08] that lists every single file, how did that come about?

Derek Stolee: There are two sides to the story; one is that as we are building all these features, custom for VFS Forget, we're doing it inside the Microsoft slash Git fork on GitHub working in the open. So you can see all the changes we're making, it's all GPL. But we're making changes in ways that are going fast. And we're not contributing to upstream Git to the core Git feature. Because of the way VFS Forget works, we have this process that's always running, that is watching the file system and getting all of its events, it made sense to say, well, we can speed up certain Git operations, because we don't need to go looking for things. We don't want to run a bunch of L-stats, because that will trigger the download of objects. So we need to refer to that process to tell me what files have been updated, what's new, and I created the idea of what's now called FS monitor. And people who had built that tool for VFS Forget contributed a version of it upstream that used Facebook's watchman tool and threw a hook. 

So it created this hook called the FS monitor hook, it would say, tell me what's been updated since the last time I checked, the watchmen or whatever tools on their side would say,  here's the small list of files that have been modified. You don't have to go walking all of the hundreds of 1000s of files because you just change these [inaudible 0:33:34]. And the Git command could store that and be fast to do things like Git status, we could add. So that was something that was contributed just mostly out of the goodness of their heart, we want to have this idea, this worked well and VFS Forget, we think can be working well for other people in regular Git, here we go and contributing and getting it in. It became much more important to us in particular when we started supporting the office monitor repo because they had a similar situation where they were moving from their version of source depot into Git and they thought VFS Forget is just going to work.

The issue is that the office also has tools that they build for iOS and macOS. So they have developers who are on macOS and the team has just started by building a similar file system, virtualization for macOS using kernel extensions. And was very far along in the process when Apple said, we're deprecating kernel extensions, you can't do that anymore. If you're someone like Dropbox, go use this thing, if you use this other thing, and we tried both of those things, and none of them work in this scenario, they're either too slow, or they're not consistent enough. For instance, if you're in Dropbox, and you say, I want to populate my files dynamically as people ask for them. The way that Dropbox in OneNote or OneDrive now does that, the operating system we decided I'm going to delete this content because the disk is getting too big. You don't need it because you can just get it from the remote again, that inconsistency was something we couldn't handle because we needed to know that content once downloaded was there. And so we were at a crossroads of not knowing where to go. But then we decided, let's do an alternative approach, let's look at what the office monorepo is different from the windows monitor repo. 

And it turns out that they had a very componentized build system, where if you wanted to build a word, you knew what you needed to build words, you didn't need the Excel code, you didn't need the PowerPoint code, you needed the word code and some common bits for all the clients of Microsoft Office. And this was ingrained in their project system, it’s like if you know that in advance, Could you just tell Git, these are the files I need to do my work in to do my build. And that’s what they were doing in their version of source depot, they weren't using a virtualized file system and their version of source depot, they were just enlisting in the projects I care about. So when some of them were moving to Git with VFS Forget, they were confused, why do I see so many directories? I don't need them. So what we did is we decided to make a new way of taking all the good bits from VFS forget, like the GVFs protocol that allowed us to do the reduced downloads. But instead of a virtualized file system to use sparse checkout is a good feature and that allows us you can say, tell Git, only give me within these directories, the files and ignore everything outside. And that gives us the same benefits of working as the smaller working directory, than the whole thing without needing to have this virtualized file system. But now we need that File System Monitor hook that we added earlier. 

Because if I still have 200,000 files on my disk, and I edit a dozen, I don't want to walk with all 200,000 to find those dozen. And so the File System Monitor became top of mind for us and particularly because we want to support Windows developers and Windows process creation is expensive, especially compared to Linux; Linux, process creation is super-fast. So having hooky run, that then does some shell script stuff to come to communicate to another process and then come back. Just that process, even if it didn't, you don't have to do anything. That was expensive enough to say we should remove the hook from this equation. And also, there are some things that watchman does that we don't like and aren't specific enough to Git, let's make a version of the file system monitor that is entrenched to get. And that's what my colleague Jeff Hosteller, is working on right now. And getting reviewed in the core Git client right now is available on Git for Windows if you want to try it because the Git for Windows maintainer is also on my team. And so we only get an early version in there. But we want to make sure this is available to all Git users. There's an imputation for Windows and macOS and it's possible to build one for Linux, we just haven't included this first version. And that's our target is to remove that overhead. I know that you at Dropbox got had a blog post where you had a huge speed up just by replacing the Perl script hook with a rusted hook, is that correct?

Utsav Shah: With the go hook not go hog, yes, but eventually we replace it with the rust one.

Derek Stolee: Excellent. And also you did some contributions to help make this hook system a little bit better and not fewer bucks. 

Utsav Shah: I think yes, one or two bugs and it took me a few months of digging and figuring out what exactly is going wrong and it turned out there's this one environment variable which you added to skip process creation. So we just had to make sure to get forest on track caches on getting you or somebody else edited. And we just forced that environment variable to be true to make sure we cache every time you run Git status. So subsequent with Git statuses are not slow and things worked out great. So we just ended up shipping a wrapper that turned out the environment variable and things worked amazingly well. So, that was so long ago. How long does this process creation take on Windows? I guess that's one question that I have had for you for while, why did we skip writing that cache? Do you know what was slow but creating processes on Windows?

Derek Stolee: Well, I know that there are a bunch of permission things that Windows does, it has many backhauls about can you create a process of this kind and what elevation privileges do you exactly have. And there are a lot of things like there that have built up because Windows is very much about re maintaining backward compatibility with a lot of these security sorts of things. So I don't know all the details I do know that it's something around the order of 100 milliseconds. So it's not something to scoff at and it's also the thing that Git for windows, in particular, has difficulty to because it has to do a bunch of translation layers to take this tool that was built for your Unix environment, and has dependencies on things like shell and Python, and Perl and how to make sure that it can work in that environment. That is an extra cost like if windows need to pay over even a normal windows process. 

Utsav Shah: Yes, that makes a lot of sense and maybe some numbers on I don't know how much you can share, like how big was the windows the office manrico annual decided to move from source depot to get like, what are we talking about here?

Derek Stolee: The biggest numbers we think about are like, how many files do I have, but I didn't do anything I just checked out the default branch should have, and I said, how many files are there? And I believe the windows repository was somewhere around 3 million and that uncompressed data was something like 300 gigabytes of like that those 3 million files taking up that long. I don't know what the full size is for the office binary, but it is 2 million files at the head. So definitely a large project, they did their homework in terms of removing large binaries from the repository so that they're not big because of that, it's not like it's Git LSS isn't going to be the solution for them. They have mostly source code and small files that are not the reason for their growth. The reason for their growth is they have so many files, and they have so many developers moving, it moving that code around and adding commits and collaborating, that it's just going to get big no matter what you do. And at one point, the windows monorepo had 110 million Git objects and I think over 12 million of those were commits partly because they had some build machinery that would commit 40 times during its build. So they rein that in, and we've set to do a history cutting and start from scratch and now it's not moving nearly as quickly, but it's still very similar size so they've got more runways.

Utsav Shah: Yes, maybe just for comparison to listeners, like the numbers I remember in 2018, the biggest repositories that were open-source that had people contributing to get forward, chromium. And remember chromium being roughly like 300,000 files, and there were like a couple of chromium engineers contributing to good performance. So this is just one order of magnitude but bigger than that, like 3 million files, I don't think there's a lot of people moving such a large repository around especially with the kind of history with like, 12 million objects it's just a lot. What was the reaction I guess, of the open-source community, the maintainers of getting stuff when you decided to help out? Did you have a conversation to start with they were just super excited when you reached out on the mailing list? What happened?

Derek Stolee: So for full context, I switched over to working on the client-side and contributed upstream get kind of, after all of the DFS forget was announced and released as open-source software. And so, I can only gauge what I saw from people afterward and people I've become to know since then, but the general reaction was, yes, it's great that you can do this, but if you had contributed to get everyone would benefit and part of the things were, the initial plan wasn't ever to open source it or, the goal was to make this work for Windows if that's the only group that ever uses it that was a success. And it turns out, we can maybe try to say it, because we can host the windows source code, we can handle your source code was kind of like a marketing point for Azure Repos and that was a big push to put this out there and say in the world, but to say like, well, it also needs this custom thing that's only on Azure Repos and we created it with our own opinions that wouldn't be up to snuff with the Git project. 

And so, things like FS monitor and partial clone are direct contributions from Microsoft engineers at the time that we're saying, here's a way to contribute the ideas that made VFS forget work to get and that was an ongoing effort to try to bring that back but it kind of started after the fact kind of, hey, we are going to contribute these ideas but at first, we needed to ship something. So we shipped something without working with the community but I think that over the last few years, is especially with the way that we've shifted our stance within our strategy to do sparse check out things with the Office monitor repo, we've much more been able to align with the things we want to build, we can build them for upstream Git first, and then we can benefit from them and then we don't have to build it twice. And then we don't have to do something special that's only for our internal teams that again, once they learn that thing, it's different from what everyone else is doing and we have that same problem again. So, right now the things that the office is depending on our sparse Checkout, yes, they're using the GVFs protocol, but to them, you can just call it partial clone and it's going to be the same from their perspective. And in fact, the way we've integrated it for them is that we've gone underneath the partial clone machinery from upstream Git and just taught it to do the GVFS protocol. So, we're much more aligned with because we know things are working for the office, upstream, Git is much more suited to be able to handle this kind of scale.

Utsav Shah: And that makes a ton of sense and given that, it seems like the community wanted you to contribute these features back. And that's just so refreshing, you want to help out someone, I don't know if you've heard of those stories where people were trying to contribute to get like Facebook has like this famous story of trying to continue to get a long time ago and not being successful and choosing to go in Mercurial, I'm happy to see that finally, we could add all of these nice things to Git.

Derek Stolee: And I should give credit to the maintainer, Junio Hamano, and people who are now my colleagues at GitHub, like Peff Jeff King, and also other Git contributors at companies like Google, who took time out of their day to help us learn what's it like to be a Git contributor, and not just open source, because open source merging pull requests on GitHub is a completely different thing than working in the Git mailing list and contributing patch sets via email. And so learning how to do that, and also, the level of quality expert expected is so high so, how can we navigate that space has new contributors, who have a lot of ideas, and are motivated to do this good work. But we needed to get over a hump of let's get into this community and establish ourselves as being good citizens and trying to do the right thing.

Utsav Shah: And maybe one more selfish question from my side. One thing that I think Git could use is some kind of login system, where today, if somebody checks in PII into our repository into the main branch, from my understanding, it's extremely hard to get rid of that without doing a full rewrite. And some kinds of plugins for companies where they can rewrite stuff or hide stuff on servers, does GitHub have something like that?

Derek Stolee: I'm not aware of anything on the GitHub or Microsoft side for that, we generally try to avoid it by doing pre received books, or when you push will reject it, for some reason, if we can, otherwise, it's on you to clear up the data. Part of that is because we want to make sure that we are maintaining repositories that are still valid, that are not going to be missing objects. I know that Google source control tool, Garrett has a way to obliterate these objects and I'm not exactly sure how it works to then say they get clients are fetching and cloning and they say,  I don't have this object it'll complain, but I don't know how they get around that. And with the distributed nature of Git, it's hard to say that the Git project should take on something like that, because it is centralizing things to such a degree that you have to say, yes, you didn't send me all the objects you said you were going to, but I'll trust you to do that anyway, that trust boundary is something that gets cautious to violate. 

Utsav Shah: Yes, that makes sense and now to the non-selfish questions, maybe you can walk through listeners, why does it need to bloom filter internally?

Derek Stolee: Sure. So let's think about commit history is specifically when, say you're in a Java repo, a repo that uses the Java programming language, and your directory structure mimics your namespace. So if you want to get to your code, you go down five directories before you find your code file. Now in Git that's represented as I have my commit, then I have my route tree, which describes the root of my working directory and then I go down for each of those directories I have another tree object, tree object, and then finally my file. And so when we want to do a history query, say what things have changed this file, I go to my first commit, and I say, let's compare it to its parent and I'm going to the root trees, well, they're different, okay they're different. Let me open them up find out which tree object they have at that first portion of the path and see if those are different, they're different let me keep going and you go all the way down these five things, you've opened up 10 trees in this diff, to parse these things and if those trees are big, that's expensive to do. 

And at the end, you might find out, wait a minute the blobs are identical way down here but I had to do all that work to find out now multiply that by a million. And you have to find out that this file that was changed 10 times in the history of a million commits; you have to do a ton of work to parse all of those trees. So, the Bloom filters come in, in a way to say, can we guarantee sometimes, and in the most case that these commits, did not change that path, we expect that most commits did not change the path you're looking for. So what we do is we injected it in the commit-graph file because that gives us a quick way to index, I'm at a commit in a position that's going to graph file, I can understand where this Bloom filter data is. And the Bloom filter is storing which paths were changed by that commit and a bloom filter is what's called a probabilistic data structure. So it doesn't list those paths, which would be expensive, if I just actually listed, every single path that changed at every commit, I would have this sort of quadratic growth again, in my data would be in the gigabytes, even for a small repo. 

But with the Bloom filter, I only need 10 bits per path so it's compact. The thing we sacrifice is that sometimes it says yes, to a path that is the answer is no but the critical thing is if it says no, you can be sure it's no, and its false-positive rate is 2%, at the compression settings we're using so I think about the history of my million commits 98% of them will this Bloom filter will say no, it didn't change. So I can immediately go to my next parent, and I can say this commit isn't important so let's move on then the sparse any trees, 2% of them, I still have to go and parse them and the 10 that changed it they'll say yes. So, I'll parse them, I'll get the right answer but we've significantly subtracted the amount of work we had to do to answer that query. And it's important when you're in these big monitor repos because you have so many commits, that didn't touch the file, you need to be able to isolate them.

Utsav Shah: At what point or like at what repository number of files, because the size of file that thing you mentioned, you can just use LFS for that should solve a lot of problems with the number of files, that's the problem. At what number of files, do I have to start thinking about okay; I want to use these good features like sparse checkout and the commit graphs and stuff? Have you noticed a tipping point like that?

Derek Stolee: Yes, there are some tipping points but it's all about, can you take advantage of the different features. So to start, I can tell you that if you have a recent version of Git saved from the last year, so you can go to whatever repository you want, and run, Git, maintenance, start, just do that in every [inaudible 52:48] is going to moderate size and that's going to enable background maintenance. So it's going to turn off auto GC because it's going to run maintenance on a regular schedule, it'll do things like fetch for you in the background, so that way, when you run Git fetch, it just updates the refs and it's really fast but it does also keep your commit graph up to date. Now, by default, it doesn't contain the Bloom filters, because Bloom filters is an extra data sink and most clients don't need it, because you're not doing these deep queries that you need to do at web-scale, like the GitHub server. The GitHub server does generate those Bloom filters so when you do a File History query on GitHub, it's fast but it does give you that commit-graph thing so you can do things like Git log graph fast. 

The topological sorting has to do for that, it can use the generation numbers to be quick, as opposed to before printers, it would take six seconds to do that just to show 10 commits, on the left few books had to walk all of them, so now you can get that for free. So whatever size repo is, you can just run that command, and you're good to go and it's the only time you have to think about it run at once now your posture is going to be good for a long time. The next level I would say is, can I reduce the amount of data I download during my clones and fetches and that the partial clones for the good for the site that I prefer blob fewer clones, so you go, Git clone, dash filter, equals blob, colon, none. I know it's complicated, but it's what we have and it just says, okay, filter out all the blobs and just give me the commits and trees that are reachable from the refs. And when I do a checkout, or when I do a history query, I'll download the blobs I need on demand. So, don't just get on a plane and try to do checkouts and things and expect it to work that's the one thing you have to be understanding about. But as long as you are relatively frequently, having a network connection, you can operate as if it's a normal Git repo and that can make your fetch times your cleaning time fast and your disk space a lot less. 

So, that's kind of like the next level of boosting up your scale and it works a lot like LFS, LFS says, I'm only going to pull down these big LFS objects when you do a checkout and but it uses a different mechanism to do that this is you've got your regular Git blobs in there. And then the next level is okay, I am only getting the blobs I need, but can I use even fewer and this is the idea of using sparse checkout to scope you’re working directory down. And I like to say that, beyond 100,000 files is where you can start thinking about using it, I start seeing Git start to chug along when you get to 100,000 200,000 files. So if you can at least max out at that level, preferably less, but if you max out at that level that would be great sparse checkout is a way to do that the issue right now that we're seeing is, you need to have a connection between your build system and sparse Checkout, to say, hey, I work in this part of the code, what files I need. 

Now, if that's relatively stable, and you can identify, you know what, all the web services are in this directory, that's all I care about and all the client code is over there, I don't need it, then a static gets merged Checkout, will work, you can just go Git's sparse checkout set, whatever directories you need, and you're good to go. The issue is if you want to be close, and say, I'm only going to get this one project I need, but then it depends on these other directories and those dependencies might change and their dependencies might change, that's when you need to build that connection. So office has a tool, they call scooper, that connects their project dependency system to sparks Checkout, and will help them automatically do that but if your dependencies are relatively stable, you can manually run Git sparse checkout. And that's going to greatly reduce the size of your working directory, which means Git's doing less when it runs checkout and that can help out.

Utsav Shah: That's a great incentive for developers to keep your code clean and modular so you're not checking out the world and eventually, it's going to help you in all these different ways and maybe for a final question here. What are you working on right now? What should we be excited about in the next few versions of Git?

Derek Stolee: I'm working on a project this whole calendar year, and I'm not going to be done with it to the calendar year is done called the Sparse Index. So it's related to sparse checkout but it's about dealing with the index file, the index file is, if you go into your Git repository, go to dot Git slash index. That file is index is a copy of what it thinks should be at the head and also what it thinks is in your working directory, so when it doesn't get status, it's walked all those files and said, this is the last time it was modified or when I expected was modified. And any difference between the index and what's actually in your working tree, Git needs to do some work to sync them up. And normally, this is just fast, it's not that big but when you have millions of files, every single file at the head has an entry in the index. Even worse, if you have a sparse Checkout, even if you have 100,000 of those 2 million files in your working directory, the index itself has 2 million entries in it, just most of them are marked with what's called the Skip Worksheet that says, don't write this to disk. So for the office monitor repo, this file is 180 megabytes, which means that every single Git status needs to read 180 gigabytes from disk, and with the LFS monitor going on, it has to go rewrite it to have the latest token from the LFS monitor so it has to rewrite it to disk. 

So, this takes five seconds to run a Git status, even though it didn't say much and you just have to like load this thing up and write it back down. So the sparse index says, well, because we're using sparse checkout in a specific way called cone mode, which is directory-based, not path file-based, you can say, well, once I get to a certain directory, I know that none of its files inside of it matter. So let's store that directory and its tree object in the index instead, so it's a kind of a placeholder to say, I could recover all the data, and all the files that would be in this directory by parsing trees, but I don't want it in my index, there's no reason for that I'm not manipulating those files when I run a Git add, I'm not manipulating them, I do Git commit. And even if I do a Git checkout, I don't even care; I just want to replace that tree with whatever I'm checking out what it thinks the tree should be. It doesn't matter for what the work I'm doing and for a typical developer in the office monorepo; this reduces the index size to 10 megabytes. So it's a huge shrinking of the size and it's unlocking so much potential in terms of our performance, our Git status times are now 300 milliseconds on Windows, on Linux, and Mac, which are also platforms, we support for the office monitor repo, it's even faster. 

So that's what I'm working on the issue here is that there's a lot of things in Git that care about the index, and they explore the index as a flat array of entries and they're always expecting those to be filenames. So all these things run the Git codebase that needs to be updated to say, well, what happens if I have a directory here? What's the thing I should do? And so, all of the ideas of what is the sparse index format, have been already released in two versions of Git, and then there's also some protections and say, well, if I have a sparse index on disk, but I'm in a command that has an integrated, well, let me parse those trees to expand it to a full index before I continue. And then at the end, I'll write a sparse index instead of writing a full index and what we've been going through is, let's integrate these other commands, we've got things like status, add, commit, checkout, those things are all integrated, we got more on the way like merge, cherry-pick, rebase. And these things all need different special care to make it to work but it's unlocking this idea that when you're in the office monitoring who after this is done, and you're working on a small slice of the repo, it's going to feel like a small repo. 

And that is going to feel awesome. I'm just so excited for developers to be able to explore that we have a few more integrations; we want to get in there. So that we can release it and feel confident that users are going to be happy. The issue being that expanding to a full index is more expensive than just reading the 180 megabytes from disk, if I just already have it in the format; it's faster than being to parse it. So we want to make sure that we have enough integrations that most scenarios users do are a lot faster, and only a few that they use occasionally get a little slower. And once we have that, we can be very confident that developers are going to be excited about the experience.

Utsav Shah: That sounds amazing the index already has so many features like the split index, the shared index, I still remember trying to like Wim understands when you're trying to read a Git index, and it just shows you as the right format and this is great. And do you think at some point, if you had all the time, and like a team of 100, people, you'd want to rewrite Git in a way that it was aware of all of these different features and layered in a way where all the different commands did not have to think about these different operations, since Git get a presented view of the index, rather than have to deal with all of these things individually?

Derek Stolee: I think the index because it's a list of files, and it's a sorted list of files, and people want to do things like replace a few entries or scan them in a certain order that it would benefit from being replaced by some sort of database, even just sequel lite would be enough. And people have brought that idea up but because this idea of a flat array of in-memory entries is so ingrained in the Git code base, that's just not possible. To do the work to layer on top, an API that allows the compatibility between the flat layer and it's something like a sequel, it's just not feasible to do, we would just disrupt users, it would probably never get done and just cause bugs. So, I don't think that that's a realistic thing to do but I think if we were to redesign it from scratch, and we weren't in a rush to get something out fast, that we would be able to take that approach. And for instance, you would sparse index, so I update one file after we write the whole index that is something I'll have to do it's just that it's smaller now. But if I had something like a database, we could just replace that entry in the database and that would be a better operation to do but it's just not built for that right now.

Utsav Shah: Okay. And if you had one thing that you would change about Git architecture like the code architecture, what would you change?

Derek Stolee: I think there are some areas where we could do some plug ability, which would be great. The code structure is flat, most of the files are just C files in the root directory and it'd be nice if they were componentized a little bit better. We had API layers that could be operating. So we could do things like swap out how refs are stored more easily, or how to swap out how the objects are stored and it is less coupled to a lot of the things across the built-ins and other things. But I think the Git project is extremely successful for its rather humble beginnings, it started as Linus Torvalds, creating a version control system for the Linux Kernel things are in a couple of weekends or however long he took a break to do that. And then people just got excited about it started contributing it and you can tell, looking at the commit messages from 2005 2006 that this was the Wild West, people were just fast in replacing code and building new things and it didn't take very long, definitely by 2010 2011 to get code base is much more solid in its form and composition. And the expectations of contributors to write good commit messages and do small changes here and there have already been built at that time a decade ago. So Git is solid software at this point, and it's very mature, so making these big drastic changes are hard to do. But I'm not going to fault it for that at all, it's good to be able to operate slowly and methodically to be able to build something and improve something that's used by millions of people you just got it, treat it with the respect and care it deserves.

Utsav Shah: If you think of software today as you run into bugs and so many different things, but Git is something that pretty much I think all developers use the most probably, and you don't even think of Git having bugs. You think, okay, I messed up using Git, you don't think that we'll get that something interesting. And if it turned out that Git had all sorts of bugs that people will run into, I don't even know what their experience would be like. They just get frustrated and they stop programming or something but yes, well, thank you for being a guest I think I learned a lot of stuff on the show. I hope listeners appreciate that as well and thank you for being a guest.

Derek Stolee: Thank you so much it was great to have these chats. I'm always happy to talk about Git, especially at scale and it's been a thing I've been focusing on for the last five years, and I'm happy to share the love. 

Utsav Shah: I might ask you for like another episode in a year or so once like sparse indexes are out.

Derek Stolee: Excellent. Yeah, I'm sure we'll have lots of teachers who had directions.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Maju Kuruvilla is the CTO and COO of Bolt, a startup that offers quick online checkout technology to retailers. Previously, he was VP and GM at Amazon Global Mile, in charge of Amazon’s global logistics and Amazon Prime fulfillment operations amongst other things.

Apple Podcasts | Spotify | Google Podcasts

Highlights

00:30 - What does a VP at Amazon even do? The day-to-day experience of a VP/GM at Amazon. I think I’ve asked enough people this question that I finally have a vague sense of what these engineering leaders do (I think).

04:00 - Managing global logistics in one of the world’s largest logistics companies in the middle of a pandemic

09:00 - Shipping software quickly when you’re a large company. Two pizza teams with a twist.

16:00 - The role of software in global logistics. Amazon’s epic migration off Oracle databases. How to get thousands of people interested in migration or similar work.

25:00 - Launching Amazon Prime Now in 111 days (21 days more than what Jeff Bezos mandated).

38:00 - The complexity behind a checkout operation in an online store. Tax operations, compliance (!), and other complexities.

46:00 - A tech stack to solve the checkout problem.

51:00 - Building trust, relationships, and making an impact as an engineering leader in a new company. Everyone wants to hire great people, but what does that really mean?

Transcript

Utsav Shah: Welcome to another episode of the Software at Scale Podcast, joining me today is Maju Kuruvilla, who is the CTO and now CEO of Bolt. Previously, he was VP at Global Mile, which is an organization at Amazon that was in charge of global fulfillment. Thank you for joining me.

Maju Kuruvilla: Thanks for inviting me, Utsav. It's great to be here.

Utsav Shah: Maybe we can start with what exactly does a VP of Global Mile at Amazon do? How many people are reporting to you eventually and what is your day-to-day look like at that time?

Maju Kuruvilla:  So I had few different roles at Amazon, the last role was the VP of Global Mile. Before that, I was a VP of the Worldwide Fulfillment technology team and the difference is when I was VP of the Worldwide Fulfillment technology team, we were responsible for all tech and products that Amazon uses in our fulfillment centers worldwide and so that's kind of global responsibility. And then a move to Global Mile, which was a little bit more a general manager role, where I was responsible for not just the technology and products, but also the operations, the sales, all of the different components of that as an end-to-end business. 

In either of those roles were global had more than 1000s of engineers and a lot of product managers? And even when I was managing the Worldwide Fulfillment team and even hardware teams and networking teams, all of them are involved as part of the team. The difference in the Global Mile was more being responsible for a P&L and an entire business, it is a little bit different than the Global Fulfillment side and I'm happy to explain on either side but just want to put the difference out there, what's up.

Utsav Shah: So maybe you can just explain both of those things, I'm sure you're looking at P&L and then going into specifics understanding of what's going on? What does your day-to-day look like? I don't know if you can talk about any specific projects that you did, I think that would be super interesting to know about.

Maju Kuruvilla: When I took on the Global Mile role; first of all, what Global Mile is, before it used to be called Global Logistics, it's all the logistics that is done to connect between countries. So whenever an item moves between countries and has to cross a border, that's where global, logistics or global mile comes into the picture. Most of the items that we sell in the US or Europe comes from manufacturing countries like Asia or China. And so when we have to bring those items from there to the US, that's part of the global logistics role. So I took on this role right before the pandemic hit and the pandemic initially hit China, which I was responsible for running the China operations at that time. 

And then when the pandemic then expanded and spread out to the Western countries, and then the rest of the world, and running global logistics during that time was very challenging and exciting. At the same time, there are a lot of changes that got disrupted that even today are not restored to what it was before. So the global supply chain is still recovering from all the problems that have happened since the beginning of COVID. So since I took on the role, it was largely doing catch-up on how to do, what do we do with the China operations? How do we manage our people there? How do we get the operations up and running? How do we keep our people safe during this process? 

And then, when all the planes stop flying or the passenger flight stop flying, the majority of the cargo space was not available after that. So we had to figure out how to create more air capacity between countries and then there was this big backlog of things that were coming from China all over the world, all the way from the challenges at the beginning of the port in China to the port of Los Angeles. For example, you could see lines of ships that are waiting to get unloaded and just managing all of that process [5:00] and reinventing and figuring out how to solve the global logistics problem in the middle of the pandemic was the highlight of my time at Global Logistics.

Utsav Shah: Sounds like a fun on boarding project.

Maju Kuruvilla: It's hard to get trained for global logistics anywhere because Amazon does a lot of that. On top of that, dealing with that during a pandemic was certainly something I was not ready for but like with every challenge and every job; you just have to figure things out. And they've created a lot of great opportunities for innovations, we did a lot of things that we wouldn't have done at the speed at which we did if we were not presented with a constraint. So a lot of innovations came out of that, a lot of new capabilities came out of that, and it was great to have a strong team at Amazon, where everyone responded fast, and started building things fast and get [unclear 06:08] operational in a very short time.

Utsav Shah: Are you at liberty to share any of those innovations that you talked about? 

Maju Kuruvilla:  A few I can certainly share; one is the air capacity problem, like, around 45% of air cargo capacity comes from the belly cargo from the passenger flight. So when passenger flights stopped flying, you essentially lost 45% of the global air cargo capacity, it’s just completely gone. And so how do we recreate that, so we had to create, start thinking of running out on charters. So we started renting planes, like 747  to fly from China to the US and Europe, from the US to export countries and so we started releasing planes and started flying, which we never had before and that was something we created last minute and started flying. 

An interesting story there was we started even getting the cargo planes for lease became very expensive and because everyone was trying to do something similar, we started renting even VIP jets. At one point, we rented a VIP jet that even I have never been on but then we were putting packages on it and shipping all over the world. So creating that kind of air capacity in a very short time and then building that into a framework that can be used for longer-term was something amazing and quick we did and whenever we have to do that a lot of things need to come together. 

One is, do you need to have the right technology for us to organize all the different products at source, figure out the route at which we all need to go, and then how do we fill this capacity when on the airplanes. You put something called a ULD which is a box that you fill all the items in and that's what you load and unload. And so you have this complex math problem of how do we fill it because you want to mix the right amount of weight and cube so that you can maximize the utilization of that. And you just have to come up with all these algorithms quickly to maximize that and then you have to do the safety to make sure that the things that you don't want to get on a plane don't get on a plane. 

And the same thing, once it reaches the destination, how do we unload it, and then from there, how does it go into all the different distribution centers, so creating the technology for that into the end and also rocking the operational process, so we can run that end-to-end, and then making sure that the business is ready to run through all of that. Creating all of that in a matter of few weeks is the speed and scale sometimes we have to run with.

Utsav Shah: First of all, I didn't even know that 45% of all cargo capacity is on passenger flights so that fact itself blew my mind. But it's also interesting that there's so much software involved to ship something like this out. One thing that always fascinated me is that even though Amazon is such a large company, it can operate and ship some things like these so fast, and maybe it's like a secret sauce, how does Amazon get that done? And we've all heard about the two pizza teams and all of that but if there's something else, like something that you've seen is a super important part of the culture.

Maju Kuruvilla: One is what you just mentioned, which is the two-pizza team but the other aspect which at least I think allows Amazon to move [10:00] fast is the decision-making capability. So whenever at Amazon, we have to make a decision, we call it one word or a two word or decision. And one word or decision is something that you make a decision and there is no going back, so you have to be extremely thoughtful on whether you make that decision and decide to walk through that door. Whereas a duet or decision is a decision you can make and then if you don't like it, you can always walk back. 

And so for us to move fast, we have to create a lot of [inaudible 10:36] decisions, and then allow the teams and the people involved to make them because it's okay, if they make a mistake, they can always walk back. And decentralizing that decision-making, allowing people, enabling them to make that decision, and then giving them a framework where most of the decisions are just too [inaudible10:58] decisions where they can walk back so that it creates a very fast decision making. And execute and enabling people to make those decisions fast and then allow them to verify if this is working or not so that they can walk back. 

That kind of decentralized enablement is critical for companies to move fast and Amazon certainly takes advantage of that quite a bit, so people are not afraid to make decisions and people are not afraid to fail. Because the culture is that you can make decisions, and learn from it and if something is wrong, you need to work back from it, so long as you can do that properly, then that's great. And that is where I feel like most of the companies get stuck because nobody knows who will make a decision. Everyone is kind of marveling at the decision to a higher level. And there is somebody who's trying to make a decision that is not very close to all the action and even if it's a very good decision, it just takes a long time. 

So we say sometimes, a fast repair wrong decision made fast, might be better than the right decision made very slow as long as it's a two-word artist.

Utsav Shah: Yes, that makes sense, and if you can walk through the example of renting airplanes, who would make that decision? Would a VP or a GM decide that we have to rent out this airplane, how would that bubble down just so that you can walk listeners through a project like that? Who would be making decisions, at what level?

Maju Kuruvilla: Again, these are not qualified anywhere, per se and so whenever you make a decision, it is one thing to make a decision, the other thing is to notify and let people know, this is what we are doing and this is going to be the implications of that. In this scenario, it was brought up to me by the team itself, so they were like, capacity is down, and we got to come up with some of the new ideas. So they started coming up with options and now the challenge here was, whenever you create capacity in any supply chain if there is a little bit of a chicken and egg problem, do you create capacity first, or you create demand first. 

And when you build a supply chain, most of the time, you have to create capacity for us because if you create demand for us, then demand is just going to wait and you create a terrible experience for people. So you have to build capacity first and so in this particular case, it was more like, we didn't have enough demand to charter our planes. But if you can build it and create an infrastructure around that and make it reliable, are more people going to use it. And so it was a decision that people bubbled up, it was largely too for me to make that choice at that point. And I made a very strong recommendation to our leadership, and they were like, go for it and the decision were made in less than six hours, and then off we go. 

And now that we decide that we are going to double the capacity and we are going to figure this out, then it's all about execution.  Now, that's only a small part of the decision-making, then there is the decision-making that happens every single day, like how many do we need, which days we need to run? And when at the beginning and the source and the destination, do you have the right operations aligned and which days you don't want to come and wait somewhere? And so how do you know that there's a lot of decision-making that happens at that level? And then it's like, what do you put on the plane, how do you prioritize? 

Do you prioritize an Amazon retail item? Do you [15:00] prioritize a seller's item, do we prioritize protective equipment for our associates. We even transported some of the equipment for hospitals, and some others just so that we can help during the middle of a pandemic. So, there are decision makings that happen at all levels. And the key there is not making one big decision right now that it's all about when you think it's in when you are working in a very fast-paced environment like this. There's a lot of micro-level decision making and if anybody hesitates to decide because they feel like okay, now somebody is going to beat me up, and I don't know what exactly it means, or I don't have all the answers, then you can move fast. 

But if everybody knows that, that's okay, I can't explain this. And it's fine. And I have no I and my everybody's going to understand or at least, and if I did something wrong, that's a great learning experience for me, then you can move fast. So it’s not a decision making at a particular level and what kind, it's enabling that a culture of freeing people from the fear of failure, and allowing people to focus on what are we trying to do? How do we achieve it fast? And anything and everything is available and possible to do that? It’s the culture of this decision-making that needed to move fast.

Utsav Shah:  That's interesting for me to hear and can you apply this a little bit to a large software project that you all did so that it's more relatable to people? We're more used to trying to ship a large piece of software? 

Maju Kuruvilla: Quite a few examples are coming to my head but one is a very complex and very technical project, even though the outcome is fairly simple. So Amazon has built the entire software on the Oracle Database platform over decades and then we were struggling and started running into constraints on Oracle and there were scaling issues because what could scale vertically, Amazon wants to scale more horizontally? So we wanted to move out of vertically scaling systems and getting more horizontally scalable systems? So the bottom line is, how do we move out of Oracle, and get to different platforms, whether it's dynamo DB or Aurora, or any of them some kind of a horizontally scalable solution. 

Now, doing this the entire Amazon had to go through it but for the fulfillment, which is one of the earliest teams at Amazon, it is a big ordeal. How do we now go and change your database where you have; first of all, fulfillment cannot stop right now, every day, you're fulfilling millions of units, and everything needs to move fast. But on the other side, you want to change the database, which you have been relying on for decades and this is not a small load, you're applying heavy load on this completely new technology and it's an all-in-one database. Everybody has their tables, and there are dependencies across all these tables, and how do we make it all happen in some kind of a sequential way and we want to get it all done in one year.

So, this is one of those projects, where we thought it was impossible, and then we said, alright, let's go after it because our leadership all agree that this is the right thing to do for the company. And some teams were able to take their data and go and find new destinations for them and that was fine; that simple those are the 20% of the use case of that all locked. everybody else, there were a ton of features needed, some needed transactional support, that are dependent on the ACID properties on a database to kind of do whatever that feature they are trying to do. And there are also interdependencies where if this error is moved, then that service also needs to move because they share the same data and you cannot shift in different ways and this is a project. 

I spent a lot of time, a whole year working with every single team and you have to imagine I have around 1000 plus engineering teams, a few 100 individual teams, and hundreds of services that need to move out of this. And so this is why again, another one [20:00] where we decentralized decision making where we tell people that these are the things that need to be done and you can go ahead and get your pieces done and if you cannot, then you need to come back with a proposal, and how do we collaborate between the teams? And also, a lot of times there were decisions that need to be made where do we go from a relational database like Oracle to a completely no sequel database? 

Or we go from a relational database to another relational database like Postgres or an Aurora, kind of a data store? And how do we manage all of that journey? And again, this is where every team had to make hundreds and 1000s of micro-decisions on their side and they have to figure out how to collaborate across all the people. And then there were times, where some of those things were not happening like there was no plan. And so I still remember me and one of the Distinguished Engineer in the team, we know we had to tell people if something is not going to work, you need to tell us right now, you need to raise your hand because if you keep thinking that it's going to work, and if it doesn't work on last minute, we won't be able to help. 

So you need to ask us early enough that we can help but there are a few instances where we had to go and help. There are a few instances, we had to innovate and come up with completely new technology so that we can solve that by running that kind of an experience of moving the entire worldwide fulfillment to Amazon that has been running 20 years or so in Oracle. And in one year, moved them completely out of Oracle to a new platform and we got it done and it was no small feat and we got it run and scale just fine for a week and it was a huge outcome for the company. So I don't think running something like that in one year is something a lot of other companies may struggle to achieve, something like that in a short time.

Utsav Shah:  That sounds like a hard migration way or that you said that 20% of use cases are easy, like half of them will probably get done and there'll be so many stragglers with special cases and one-offs, it just sounds extremely painful to drive.  How do you set up incentives in the right way to make sure that people wanted to do like because I'm sure a lot of it is grunt work and it's a little bit of an ordeal as you said? So one thing you mentioned was, you have a Distinguished Engineer, going to help everyone but people have so many competing priorities, they want to ship new things. How do you make it easier for these teams to prioritize this work?

Maju Kuruvilla: It's all about prioritization, so you have two people look at what's important, you can say something is important but then people will pay attention to where you are spending your time. Right now, if people see that I as a leader or anybody who was a leader, spending most of their time on a project then people know that is important. It's not that they are saying it, they're doing it, and then provide with structure or guidance. For example, we created a small Tiger team and they knew how to go and audit every single team's plan, not just their migration plan but also their peak readiness plan and they will give out a report to me, and then I can review that. 

And if they are not ready, even if they think they are ready and if the team that's auditing did not feel like that's ready, they will come back with that and there are different mechanisms we put in place to enable teams to do that. Now, that example is more a hard groundwork of just migration, if I switch to a different example, which is the launch of prime now that is more exciting new work. Amazon wanted to get into fast delivery and there was this big plan that was created on how to do that and when debase [unclear 24:27], he reviewed a plan and he was, well, this is great, I love it go and make it happen in 90 days. So when you bait in your bases, you get to say things like that, go and make an entirely new experience of allowing people to get things within one hour in 90 days. 

This is said when same-day delivery is not even a thing, nobody would see our delivery is not even a thing and nobody know what it means. And so I was responsible for all the fulfillment aspect of that which turned out to be very complex, [25:00] how do we deliver something less than an hour when usually our fulfillment systems are designed to deliver in two days or less. So there is another one where we assembled a core team, we divvy up the function, and we decided to build, take some of the components of the fulfillment, created a lighter version of our fulfillment stack, and build.  Some teams built the front end, new app, how the call connectivity? How do we manage payments? How do we manage fraud? 

And then, in the end, how do we even create technologies within the fulfillment center where you can pick and box all these things in a lightning-fast way. And then we launched that in New York, right in the middle of the city during Christmas, and I was there because we work hard on it and I delivered, we call it rideshare. You can go with the people who are delivering and see the experience for the customer and I did that for some of the initial ones and it was fascinating to watch the people's reactions when they get it. Sometimes we delivered things within eight minutes of when a customer first presses the order button and you can just see the stars and customers' eyes when something just shows up within eight minutes of something they ordered and mind you back in the day, these things didn't exist. 

So it was a magical experience for everyone; so we got it done. Not just my team but the collective team recording was done in 11 days, from the basis meeting, even though here that 90 days, we got it done in 11 days, and, and I don't even know how we got it done in under 11 days and basis, we're still happy but that's the speed at which you move. And if you need to move that kind of speed, you are innovating so many things; you are making decisions on so many items. But, it's not one person making all of the decisions, the entire team making those decisions, and allowing everyone to move faster.

Utsav Shah:  Just from reading one of the books on Amazon and understanding your stories, it seems the idea is to find each level so that it has enough people, so that's not what the blocker is. But then make sure there is fast decision-making and accountability. So projects get done on time, rather than not funding each level enough that it will just take forever to get stuff done because people feel they're super or something like that.

Maju Kuruvilla: Well, whenever you have a big problem it's very hard to solve a big problem as is. So number one is let's agree on the problem and make sure that solving this problem makes sense for everybody and this is the right thing to do for the company, for the customer, and we have the right know-how to make it happen. And once we do that, then the next question is, how do we divvy up this problem into small chunks? If you want to move the mountain, it's extremely hard to move the mountain but assuming everybody can take a piece of rock from it and 1000s and millions of people if you can all assemble, then we can move the mountain. 

That's the concept of swarming a group of people around a huge problem not attacking the whole problem as one but divvying up the problem into smaller components. Now, divvying up the components is an art, you have to find out how to divvy it up because divvying up is not like, I will do this part or that part. Each of the smaller components needs to be fully functional by itself so that the team knows if they make it, they know how to test it, and how can you know if it's a car? Each of the components is like a tire and if you build the tire properly, you can have a tire team, you can have a wheel team and the good thing about that is, that team can continue to obsess over that and make it better every day. 

Today you can come up with some new tread wearing and all of that and tomorrow you can come up with some nails, but they have something they build that is complete in its sense as a component and that is something they can continue [30:00] to make better throughout their life, so we call that a pizza team. Or at Amazon, we call the pizza team and what that means is that team has ownership of a component that is complete and relevant [unclear 30:20]. So it's not a short term, as a project I will do this piece and that piece is more long-term ownership. And in the long-term ownership, I own this component, and I will continue to make that component better and I know how this component fits into all the other larger pieces but I don't care. 

So I am not going to worry about how the headlights are going to work or something, all those other things, I know it's all there but I don't need to worry about it. I'm not constrained by it because I built the right contract, if it's a tire, I just need to fit it on the right wheel and then I'm good, and then what happens after that I don't care. And I am going to obsess over this tire and all the materials that are on that forever and so we'll make it better over time. That's the concept, how do we break it down into components where each of the teams can own one of those pieces and then they can obsess over it? Every day, they have 24 7 365 days, they just tried to make it better and when a lot of components come together, you assemble to solve the larger problem you're going after.

Utsav Shah:  And I'm guessing the holistic vision on the assembly is management's responsibility along with the senior engineers and the principal engineers who will give guidance.

Maju Kuruvilla: Yes, more than management it is that the senior engineers and the principal engineers, and a distinguished engineers because that's the way at least Amazon is structured is you have this pizza team that it's called two pizza team because suddenly, you should be able to feed a team with two pizzas. And so the magic number is somewhere between seven and 11, that's the number of people you should have in a team and that team should have everybody you need to solve the problem with that component. So sometimes the team may have hardware people, software people, data science people, it cannot be just a software team; it's a team, what do you need to solve that problem, you need all of them in there. 

And then whenever you have a principal engineer, their responsibility is to look at across multiple components like this, and see how they all come together and also the senior engineers in the teams, even though they are part of the team, they look across and negotiate and make sure that all the pieces are coming together well. But the rest of the teams, are headstand focused on what they need to build, and how to and they all have a metric that they are trying to make better and Amazon call it a fitness function. So you are looking at that fit fitness function and trying to make sure that you're continuing to make progress on that.

Utsav Shah:  And then that's some quality metric which is indicative of this team's component functioning properly or not.

Maju Kuruvilla:  It's more than that, this team's component being the best. So,  it's not a functional metric, it's a fitness function means this is the most important thing that you can measure to see if they are doing the ultimate best, they can do.

Utsav Shah: That's interesting, and then you can imagine that some people might even have an NPS score or something eventually getting tracked as part of that fitness function. And then finally, management reviews, all of these different teams is fitness functions to make sure that everything is coming together and then they can deliver the final large piece which is a huge prime now.

Maju Kuruvilla: And usually there is a operate operational plan, so that that happens every year, you see how everything is going to come together and Amazon also has heavy documentation, culture, writing and reading are important. And reading is as important as writing by the way because I have seen a lot of companies tend to say we have a writing culture but most of them don't read. So I feel writing is one of those networking effects. If people don't read then writers have no incentive to write properly but Amazon has a very good doc in writing and reading culture and what that means is whenever you have to solve let's say, [unclear 34:43], another product was we did a computer vision system to completely automate the inventory, counting process in all fulfillment centers, it used to be a very manual [35:00] process. 

Before people used to count, it’s late, you're closing a store, you have to count the entire inventory all the time for compliance purposes, and also have to make sure that the virtual and the physical things are connected. And we completely get on with a new project where computer vision systems constantly monitor things as robots are moving things across the field and that can replace the whole process manual counting process. And again, a team of 11 people made that happen from an idea, they pitch to making it happen for momentum, Amazon's providing fulfillment centers worldwide, just loving people just made it happen. And it saves a lot of money for the company and automated a lot of processes but its more pizza team coming together. 

And so when you have to do that, the first things you write is what do you call a press release document or a PR FAQ document? And it's a one-page that clearly says, what's the problem? Why do we think we are the right people to solve? Is this the right problem to solve? And if you solve it, what will be the experience for the customer, so you have to write the experience for the customer, past solving this problem before you start doing anything. So that's what the press releases, you are writing down how the experience is going to be after the fact before you start any work, which is very powerful, by the way, because most of the time when you have to write that from a customer angle, a lot of things could become very clear, things we didn't think through will become very obvious because you might be solving either part of a problem or it's part of a bigger problem. 

And a customer only cares if the whole thing is off, not a piece of the result. So going to explain that customer experience is very powerful and then from that, there is a sequence of FAQs and usually, it's only one page, by the way. PR FAQ is just one page on the press release and then a lot of questions and that document is the first thing everybody writes and reviews. And so any of this brothers prime now, whether it's this computer vision-based inventory, counting any of that's the first thing people do, they write in a document, and then that gets reviewed by leadership. And then if someone reads and then finally says, approve, and you can go and start a new pizza team for that, or you going to allocate resources for that. And then you can go and do what there might be multiple pizza teams come together to solve that and you explain that in your plan. 

That's our plan, and then that's fine, you get a proof and then people will go off from that point, and make things out.

Utsav Shah: So switching gears maybe a little bit, one of the things that Amazon is good at is making sure the checkout experience is super smooth,  there's the one-click option, which I think Amazon has a patent on, so it's not easy for other companies to do something similar. But a lot of reasons why people like using Amazon is also the smooth checkout experience plus, it's super reliable. So just recently, I tried to buy a DELL laptop from Dell, and twice they just canceled my order and I had to place it through Amazon. I don't know how much Dell has to pay Amazon for it, but they just lost the commission on that laptop. So can you walk through why it's so important for checkout to be so seamless, intuitively it makes sense, you don't want people to be abandoning stuff in their car, but any numbers or anything whatsoever about why that checkout experience has to be as convenient as it is?

Maju Kuruvilla: It's an interesting story and the whole online commerce story is unfolding in front of our eyes as we speak, this is the time for e-commerce. Companies like Amazon created this online buying experience and people started trusting buying online before people were worried about the security and safety of their staff and the quality of the things that they might get. Amazon solved all of that and got people to buy online without thinking twice about it. What we are seeing now is the consumers are used to it, now pandemic accelerated the whole process too, pandemic accelerated e-commerce adoption, almost 10 years ahead of the previous base. So if we didn't have a pandemic ended up taking another 10 years for us to be where we are, but it accelerated 10 years. 

So now people are [40:00] buying, especially the newer generation, folks buying online are not a big deal. They don't even think twice about it but what comes with that is people also want a different experience, people want to buy, they will continue to buy from Amazon but they also want to go and buy from other places. Because there are a lot of different merchants and brands that want to provide a very unique experience for customers. And people having that experience is not just about buying a product as it's about that whole experience of connecting yourself with the brand and that the experience of buying and then all the post-purchase experience after that on being connected with that brand. 

So sometimes that relationship is more than just buying something and more people want that, and especially the newer generations are more into that process. The challenge with most of those margins is that, how do you provide a simple seamless checkout experience like Amazon? Because Amazon has, like you said, an amazing checkout experience and how do you provide that because people are used to it now, so you don't need to beat Amazon on that. But at least people, you need to provide a similar experience that everywhere else and this is where companies like Bolt come in. I'll speak about Bolt a little bit here, where we can provide that checkout experience that people are used to, and make that one-click checkout experience where you come to our site, and you just click one button, and it's yours. 

And if you can provide that experience, people will start engaging with brands and merchants a lot more than if they have to go through the whole kind of high friction buying process. Because when you think of buying, it's a funnel and the first is research, and then there is a discovery of product, then there is intent to buy. That's why most of the time [unclear 42:11] on and then there is the conversion and at every point, you are adding friction, and if you can have where people can back off from the process. But if you can remove all of that, it's when you have to randomly a movie, and you have to go all the way to a blockbuster or stand in line, pick up a video and come home and watch it, that seems like a lot of friction versus just sitting at home and click Netflix and boom.  

People do watch more Netflix because of the ease of it, than like Hollywood or [unclear 42:51] Hollywood, or blockbuster or somebody like that because there's a lot of friction in that and the same people just do more because it's convenient. And that applies even in a checkout where we want to eliminate all that friction so that when people want their intent to buy is their conversion. There's nothing that stands between that and beyond just buying things on the brands and their website. There is also one more step beyond that. This is where most of the people are moving to and that is called social commerce where people want to buy things as soon as they see something, it's called buying at the point of discovery. 

So as you see an advertisement, you see a video, you will see an influencers media or you're reading a review, and you see something and you want to buy it, can you just click and buy it right there with one click, or it's a link that takes you back to some site and you have to buy it there now by going through all the process. So simplifying this whole process, whether that's buying from a website, or buying at the point of discovery at any surface, is going to be very critical in the future. And in fact, that's going to be the expectation for a lot of people as we move more and more into the e-commerce journey and that's what companies like Bolt provide out of the box for merchants so they can provide this experience to their customers.

Utsav Shah: And maybe even talk to us a little bit about the impact of going through that funnel, if I'm a shopkeeper today who just set up my site without using Bolt or an optimized lead what abandonment rates and stuff would I expect? I know it's going to be different for each person, but I'm just trying to understand how much does that convenience factor play into online purchases?

Maju Kuruvilla: It's substantial. So, it depends on the merchant, depends on the category, depends on the customer [45:00] and we have several case studies that we have listed on our website. But in some of the good use cases, you see an 80% increase in conversion, when you get a one-click checkout experience that's on the highest end. But it's very powerful because there are a couple of reasons; One, is it just the friction, you don't need to go and do something else to make it happen and number two is, it's just the safe, that people feel their safety and privacy. Do I want to give my information to all the websites out there? Or if I just give it to this one party I trust, and it insists that my identity is just there, and all my information is there but it enables me to log in everywhere, it's the single sign-on concept everywhere, that provides more comfort and safety for customers. So it's one side is the friction, the other side is feeling having a safe way to buy from anywhere.

Utsav Shah: And I think one thing you spoke about was the fulfillment stack, building out all of this technology that provides fulfillment, maybe you can walk us through the checkout stack if you have to build a checkout system like this.  At a glance, you're adding something to a cart, and you're retrieving that cart, and you're making a purchase but how does this work? How do you make it go one-click? And what are some challenges? Because I can imagine there are all sorts of things to worry about as users might click by mistake, there's fraud and stuff, how do you solve all of these problems?

Maju Kuruvilla: Great question. So when you think of checkout when from the outside, it seemed like a fairly simple process, you add something to the cart, there's a checkout button, you click on it, and it asks you for payments and few things and it's done. Checkout is one of the most complex parts in a commerce stack and the reason for that is until it is checkout, it's just browsing, you are just adding things here and there, nothing needs consistency until at that point but when it's come to checkout, it's real. The checkout system needs to check for inventory, it needs to check for taxes, you need to look for coupons, it needs to look for the pricing at that time, it needs to look for all the shipping options, anything and everything that you need to check is all done at checkout. 

So checkout needs to call every single system that an e-commerce stack has to make sure that it all comes together so you can present it to the customer. What does it take to buy the thing and when they can get it and so it is an extremely complex function and so behind that scene, there is the UX how we provide a seamless user experience. For example, at Bolt, we obsess over how every single pixel works in that checkout? How does it work on the website? How does it work on a mobile? How can we make sure that it's so optimized that our customers feel the whole process like a breeze but then there is the payment gateway? People may want to use a variety of different payment instruments; payments are the whole world is changing so fast that there is some new payment method or an alternate payment method coming out, or every other week nowadays. 

So how do we keep provide merchants and integration into the entire payment world? And how do we place all of that so that the cost the customers can choose it in the right way, which is the other layer of complexity? But then there is the identity of the user itself, how do we do provide one click? We need to save that user's name and all the payment details and all the information so that we can do that. So, a company like Bolt, what we have is, we use that user in all the shopper's data as a network. So think of all the different shoppers who are connected to our shopping accounts network and everybody can contribute to that network and everybody can benefit from that network if they use Bolt checkout. 

So the Bolt checkout system is built on our accounts network that is continuing to grow and evolve and as people are shopping and all of these different merchants, people are getting added to this network. So as a new merchant coming brand new use Bolt, they have access to this whole network that's created through the shared network that is created so far, and they can provide one-click checkout to every single user in that [50:00] network. So what we are finding is this virtuous cycle of as we get more merchants, we get more accounts into our network and as we have more accounts in the network, you get more one-click checkout transactions, and then more one-click means more merchants want to sign up for us. 

So that virtuous cycle that's happening and that's added is accelerating, kind of Bolt's growth for now. But those are the different layers, all the way from a UI to all the complexities around that, but fundamentally built on this shared accounts network that's truly powering the one-click experience for everybody.

Utsav Shah: So as an end-user, do I know I'm using Bolt when I'm buying something [unclear 50:46] website, or is it just like opaque to me?

Maju Kuruvilla: No, you are buying it through Bolt; however, we don't try to brand it as a completely different brand because the merchant is buying us and we want to seamlessly integrate into Martin's ecosystem, so that we want to look more like, we are enabling the merchant, and we want to stay away between the customer and the merchant because we want to provide an experience as seamless as possible. However, for users they need to know that this is Bolt so that they can trust us, they know that it's powered by Bolt, creating an icon with Bolt, or logging in with Bolt, so that they know where it is, and they can control their information, they can manage it, and they can be at peace that we are taking care of that information and in one safe place. 

Utsav Shah: So that makes sense to me. Now, I'm curious to learn, you were running divisions or organizations at large companies like Amazon before, you can't just take every single good idea you have and apply it at a much smaller company. But what are some things that you feel you've changed in the first few months you were there and what are some things that you feel extremely strongly about? Even at a much smaller company when the Bolt is not super small, clearly, but what are just some learning’s that you feel you had to apply?

Maju Kuruvilla: First and foremost, I would say, it doesn't matter what's the size of the company, you have to be connected with the customer and every single person all the way. For engineering, it doesn't matter whether you're an engineer, whether you are an accountant, or it doesn't matter what your discipline is, if you are in a company, your mission, you're passionate about it, you need to be very connected with your customer, you need to attend to some customer calls, you need to attend some support calls, you need to be on-call, sometimes, you need to be right in the thick of things. So big, small, doesn't matter and that I'm passionate about, and I'll force to make sure that everyone is deeply connected to that. 

And number two is hiring great people whether it’s; again, your company is only as good as the people you have. So hiring great people and taking care of great people is the highest priority of any leader across big, large mall doesn't matter; all companies. So, for example, when I came in, hiring more people into Bolt we are growing fast, so we wanted to bring in a lot of great people. I spend a lot of time in hiring and meeting with our people, for example, in my first 90 days, I met with every single person in the engineering and the product organization, had a one-on-one with them as simple three questions on what's working well at Bolt? What's not working well? And what are they hoping that I'll be doing to help? 

Three questions to every single person involved in engineering and products, I enter my first 90 days and I did that north of 100 interviews in my first 90 days, to both sides of the world, you have to take care of your great people and understand the people and what they want, what's working, what's not working so you can fix them. And continue to bring in great people into the company and that to me, knowing the customer and having a great team are the two things I hold dear to me, it doesn't matter where I go.

Utsav Shah: Look, that makes a lot of sense to me but how do you [55:00] gauge the right people? How do you know somebody is good or not? Is there something in the interview you do or is this just a quality? You see, I don't know, I'm just curious, everybody talks about having great people, what does that mean?

Maju Kuruvilla: There are two aspects where I look for personally, when we look for hiring great people, one is the basic table stakes, which are operational excellence, their technical skills,  and all of that, which I think almost every company probably look for, which is great. But then there is the other side and this is where I will say beyond culture, I tend to look for people who are system thinkers, people who can think into it, people whenever they hear a problem, it's not just like, how do I solve this piece of the problem, but take a step back and look at, what's going on here. And what is the right way to solve it is maybe the solution is very different than what's obvious at the very beginning. 

And so people who can take a step back and look into and have that system thinking. And what that means is, truly solve the problem and sometimes not solving the problem you solve. But solving the original problem that even caused the problem, what you are seeing right now and so I tend to focus a lot during the interviews on things that like, what are some of the things they did? And I asked questions like, why did they do that? And why did that particular problem, the solution was the right one? How did they think that will solve whatever they're looking for? Because what I'm finding is that people can think more, can have that system level, an end-to-end, and end up solving innovative ways, than people just attack individual issues and solve them by themselves. 

And so I know, it's not that scientific way to think but it's a mindset, I have found that the people tend to create much resolving longstanding impact than most of the others.

Utsav Shah: Yes, I think that makes a lot of sense, systems thinking framework, thinking people can understand the problem more holistically, than looking at the smaller things. But yes, I think this was a lot of useful information and we're almost on time. Thank you so much for being a guest, I certainly feel I learned a lot about a company that is pretty much in a black box, sometimes from the outside. I had no idea about so many things, about how fulfillment works, check out, thank you so much.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev