Software at Scale

Software at Scale

Download on the App Store

All Software at Scale episodes

  • Software at Scale 40 - Talent Management with Nikita Gupta

    Nikita Gupta is a Co-Founder & CTO at Symba, a platform that helps manage talent development programs like internships.

    Internships are one of the most effective ways for hiring at a software company, but there’s a lot of work that goes into managing successful interns. With hiring getting harder across the industry due to increased competition and funding, I thought it would be interesting to dive into understanding how to manage successful internship programs.

    Highlights

    0:30 - What is Symba?

    1:30 - Starting with the hot-takes. So, are college degrees overrated now?

    5:30 - Why do I need a software platform to manage internships?

    8:50 - Why do companies generally need to manage 8 - 10 platforms for internships? What have you seen?

    10:30 - As a software engineer or manager, how do I make my intern successful?

    13:30 - Cadence of check-ins

    16:30 - With remote interns, how do you build a successful community?

    18:50 - How do I measure the success/efficacy of my internship program?

    21:00 - How do I know that my intern mentors/hosts are doing a good job?

    25:00 - What are some concrete steps that I can take to increase my intern pool’s diversity? What should I track?

    27:30 - What are some trends in the intern hiring space?

    32:00 - Government investments in internship programs

    33:00 - What’s your advice to the first-time intern mentor/host?



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev
    36 min
  • Software at Scale 39 - Infrastructure Security with Guy Eisenkot

    Guy Eisenkot is a Senior Director of Product Management at BridgeCrew by Prisma Cloud and was the co-founder of BridgeCrew, an infrastructure security platform.

    We deep dive into infrastructure security, Checkov, and BridgeCrew in this episode. I’ve personally been writing Terraform for the last few weeks, and it often feels like I’m flying blind from a reliability/security perspective. For example, it’s all too easy to create an unencrypted S3 bucket in Terraform which you’ll only find out about when it hits production (via security tools). So I see the need for tools that lint my infrastructure as code more meaningfully, and we spend some time talking about that need.

    We also investigate “how did we get here”, unravel some infrastructure as code history and the story behind Checkov’s quick popularity. We talk about how ShiftLeft is often a painfully overused term, the security process in modern companies, and the future of security, in a world with ever-more infrastructure complexity.

    Highlights

    00:00 - Why is infrastructure security important to me as a developer?

    05:00 - The story of Checkov

    09:00 - What need did Checkov fulfil when it was released?

    10:30 - Why don’t tools like Terraform enforce good security by default?

    15:30 - Why ShiftLeft is a tired, not wired concept.

    20:00 - When should I make my first security hire?

    24:00 - Productizing what a security hire would do.

    27:00 - Amazon CodeGuru but for security fixes - Smart Fixes.

    33:00 - Is it possible to write infrastructure as code checks in frameworks like Pulumi?

    37:00 - Not being an early adopter when it comes to infrastructure tools.

    40:00 - The Log4J vulnerability, and the security world moving forward.



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev
    46 min
  • Software at Scale 38 - Hasura with Tanmai Gopal

    Tanmai Gopal is the founder of Hasura, an API as a service platform. Hasura lets you skip writing API layers and exposes automatic GraphQL APIs that talk to your database, trigger external actions, and much more.

    We talk about the implementation of a “compiler as a service”, the implications of primarily having a Haskell production codebase, their experience with GraphQL, hiring product managers technical enough to build useful features, some new and upcoming Hasura features, and riff about the current state of front-end development and `npm` hell.

    Highlights

    00:20 - What does the name Hasura mean?

    02:00 - What does Hasura do?

    04:00 - Why build this layer of the stack?

    08:00 - How to deal with authentication if APIs are exposed directly via the database.

    26:00 - Does Hasura make production applications faster?

    33:00 - JSON Aggregation in modern databases

    38:00 - Why Haskell?

    44:00 - How do you write quality Haskell? How does hiring for Haskell positions work out in practice?

    55:00 - Application servers do much more than just talk to databases. How does Hasura provide escape hatches so that non-database interactions (for eg: talking to Stripe) work out?



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev
    1 hr 10 min
  • Software at Scale 37 - Building Zerodha with Kailash Nadh

    Kailash Nadh is the CTO of Zerodha, India’s largest retail stockbroker. Zerodha powers a large volume of stock trades - ~15-20% of India’s daily volume which is significantly more daily transactions than Robinhood.

    Apple Podcasts | Spotify | Google Podcasts

    The focus of this episode is the technology and mindset behind Zerodha - the key technology choices, challenges faced, and lessons learned while building the platform over several years. As described on the company’s tech blog, Zerodha has an unconventional approach to building software - open source centric, relatively few deadlines, an incessant focus on resolving technical debt, and extreme autonomy to the small but efficient technology team. We dig into these and learn about the inner workings of one of India’s premier fintech companies.

    Highlights

    [00:43]: Can you describe the Zerodha product? Could you also share any metrics that demonstrate the scale, like the number of transactions or number of users?

    Zerodha is an online stockbroker. You can download one of the apps and sign to buy and sell shares in the stock market, and invest. We have over 7 million customers, on any given day we have over 2 million concurrent users, and this week, we broke our record for a number of trades handled in a day - 14 million trades in a day, which represented over 20% of all Indian stock-trading activity.

    [03:00] When a user opens the app at 9:15 in the morning to see trade activity and purchase a trade, what happens behind the scenes? Life of a Query, Zerodha Edition

    [05:00] What exactly is the risk management system doing? Can you give an example of where it will block a trade?

    What is the risk management system doing?

    The most critical check is a margin check - whether you have enough purchasing power margins in your account. With equities, it’s a simple linear check of whether you have enough, but for derivatives, it’s about figuring out if you have enough margins. If you already have some futures and options in your account, the risk is variable based on that pre-existing amount.

    What does the reconciliation process look like with the exchange?

    We have a joke in our engineering team that we’re just CSV engineers since re-conciliation in our industry happens via several CSV files that are distributed at the end of the trading day.

    [08:40] Are you still using PostgreSQL for storing data?

    We still use (abuse) PostgreSQL with hundreds of billions of rows of data, sharded several ways

    [09:40] In general, how has Zerodha evolved over time, from the v0 of the tech product to today?

    From 2010 to 2013, there was no tech team, and Zerodha’s prime value add was a discount pricing model. We had vendor products that let users log in and trade, and the competition was on pricing. But they worked on 1/10,000th the scale that we operate on today, for a tiny fraction of the userbase. To give a sense of their maturity, they only worked on Internet Explorer 6.

    So in late 2014, we built a reporting platform that replaced this vendor-based system. We kept on replacing systems and dependencies, and the last piece left is the OMS - the Order Management System. We’ve had a project to replace this OMS ongoing for 2.5 years and are currently an running internal beta, and once this is complete, we will have no external dependencies.

    The first version of Kite, written in Python, came out in 2015. Then, we rewrote some of the services in Go. We now have a ton of services that do all sorts of things like document verification, KYC, payments, banking, integrations, trading, PNL, number crunching and analytics, visualizations, mutual funds, absolutely everything you can imagine.

    [13:55] Why is it so tricky to rebuild an Order Management System?

    There’s no spec out there to build an Order or a Risk Management System. A margin check is based on mathematical models that take a lot of different parameters into account.

    We’re doing complex checks that are based on mathematical models that we’ve reverse-engineered after years of experience with the system, as well as developing deep domain knowledge in the area.

    And once we build out the system, we cannot simply migrate away from the old system due to the high consequences of potential errors. So we need to test and migrate piecemeal from the system. 

    [13:55] One thing you notice when Zerodha is how fast it feels compared to standard web applications. This needs focus on both backend and frontend systems. To start with, how do you optimize your backends for speed?

    When an application is slow (data takes more than a second to load), it’s perceptible, and can be annoying for users. So we're very particular about making everything as fast as possible, and we’ve set high benchmarks for ourselves. We set an upper limit of mean latency for users to be no more than 40 milliseconds, which seems to work well for us, given all the randomness from the internet. Then, all the code we write has to meet this benchmark.

    In order to make this work, there’s no black magic, just common sense principles. For the core flow of the product, everything is retrieved from in-memory databases, and nothing touches disk in the hot path of a request.

    Serialization is expensive. If you have a bunch of orders and you need to send those back, serializing and deserializing takes time. So when events take place, like a new order being placed, we serialize once and store the result in an in-memory database. And then when an HTTP request comes in from a user, instead of a database lookup and various transforms, the application reads directly from in-memory databases and writes it to the browser.

    Then, we have a few heuristics. For fetching really old reports that <2% of users use, it’s okay for those to be slow. Those will happen in separate paths so that they don’t block the more frequent kind of requests.

    Finally, we’ve written all these services with Golang, which is fast out of the box, provides a reasonably good developer experience, and has good concurrency primitives. We’re careful with memory allocations and pool resources wherever applicable.

    [24:00] Zerodha also seems to have skipped the React world, by going with Flutter on mobile and Vue on the web. Can you speak to that decision for mobile apps?

    We initially built the iOS app in React Native about 3-4 years ago. I’m not sure how things are today, but the application was fairly slow. The bread and butter of a trading application is a bunch of ticking numbers to show stock values, and we experienced a bunch of frame drops while trying to render those. You’d think that it’d be trivial to show that with low latency in 2017, but we were experiencing 5-10 frame/s, and Indian smartphones weren’t extremely powerful then. We also ran into several library/dependency breakages.

    We then randomly ran across Flutter, which in pre-alpha, and not a lot had been written in it. Picking a bleeding-edge technology is very risky, and we wanted to evaluate the risk carefully, so we built out a full-blown prototype of the Kite app that had all the parts we thought would be bottlenecks, like web socket connections, updating numbers, list views, navigation, transitions. We learned Dart (which you need to use Flutter). Once we built out the prototype, it was clear that the performance and experience with Flutter was significantly better than React Native. So we made that very early decision to ditch React Native and adopt Flutter.

    We figured that even if Flutter got killed, we’d benefit from using Flutter for a few years and would eventually move to something else, and traded-off the risk. We launched the iOS application, fixed up issues, killed our Android application, and rolled out a Flutter version.

    [28:30] How about not going with React on the website, and going with Vue?

    We built our v1 with Angular, but we found it tricky to use for our small team, and there was the major version break fiasco. It was overly complicated even after months of use. We decided to skip out on sunk cost…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev
    49 min
  • Software at Scale 36 - Decomposing Monoliths with Ganesh Datta

    Ganesh Datta is the CTO and co-founder of Cortex, a microservice management platform.

    Apple Podcasts | Spotify | Google Podcasts

    We continue the age-old monolith/microservice debate and dig into why companies seem to like services so much (I’m generally cautious about such migrations). Ganesh has a ton of insights into developer productivity and tooling to make engineering teams successful that we dive into.

    Highlights

    00:00 - Why solve the service management problem?

    06:00 - When to drive a monolith → service migration? What inflection points should one think about to make that decision?

    08:30 - What would Ganesh do his next service migration?

    10:30 - What tools are useful when migrating to services?

    12:00 - Standardizing infrastructure to facilitate migrations. How much to standardize (à la Google), to letting team make their own decisions (à la Amazon)?

    17:30 - How does a tool like Cortex help with these problems?

    21:30 - How opinionated should such tools be? How much user education is part of building such tools?

    27:00 - What are the key cultural components of successful engineering teams?

    31:00 - Tactically, what does good service management look like today?

    37:00 - What’s the cost/benefit ratio of shipping an on-prem product vs. a SaaS tool?

    41:30 - What would your advice be for the next software engineer embarking on their monolith → microservice migration?



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev
    44 min
  • Software at Scale 35 - Maintaining Git with Johannes Schindelin

    Johannes Schindelin is the maintainer (BDFL) of Git for Windows.

    Apple Podcasts | Spotify | Google Podcasts

    Git is a fundamental piece of the software community, and we get to learn the history and inner workings of the project in this episode. Maintaining a widely-used open source project involves a ton of expected complexity around handling bug reports, deprecations, and inclusive culture, but also requires management of inter-personal relationships, ease of contribution, and other aspects that are fascinating to learn about.

    Highlights

    00:06 - How did Johannes end up as the maintainer of Git for Windows?

    06:30 - The Git community in the early days. Fun fact: Git used to be called `dircache`

    08:30 - How many downloads does Git for Windows get today?

    10:15 - Why does Git for Windows a separate project? Why not make improvements to Git itself?

    24:00 - How do you deprecate functionality when there are millions of users of your product and you have no telemetry?

    30:00 - What does being the BDFL of a project mean? What does Johannes day-to-day look like?

    33:00 - What is GitGitGadget? How does it make contributions easier?

    41:00 - How do foster an inclusive community of an open-source project?

    50:00 - What’s next for Git?



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev
    56 min
  • Software at Scale 34 - Faster Python with Guido van Rossum

    Guido van Rossum is the creator of the Python programming language and a Distinguished Engineer at Microsoft.

    Apple Podcasts | Spotify | Google Podcasts

    We discuss Guido’s new work on making CPython faster (PEP 659), Tiers of Python Interpreter Execution, and high impact, low hanging fruit performance improvements.

    Highlights

    (an edited summary)

    [00:21] What got you interested in working on Python performance?

    Guido: In some sense, it was probably a topic that was fairly comfortable to me because it means working with a core of Python, where I still feel I know my way around. When I started at Microsoft, I briefly looked at Azure but realized I never enjoyed that kind of work at Google or Dropbox. Then I looked at Machine Learning, but it would take a lot of time to do something interested with the non-Python, and even Python-related bits.

    [02:31] What was different about the set of Mark Shannon’s ideas on Python performance that convinced you to go after them?

    Guido: I liked how he was thinking about the problem. Most of the other approaches around Python performance like PyPy and Cinder are not suitable for all use cases since they aren’t backward compatible with extension modules. Mark has the perspective and experience of a CPython developer, as well as a viable approach that would maintain backward compatibility, which is the hardest problem to solve. The Python Bytecode interpreter is modified often across minor releases (for eg: 3.8 → 3.9) for various reasons like new opcodes, so modifying that is a relatively safe approach.

    Utsav: [09:45] Could you walk us through the idea of the tiers of execution of the Python Interpreter?

    Guido: When you execute a program, you don't know if it's going to crash after running a fraction of a millisecond, or whether it's going to be a three-week-long computation. Because it could be the same code, just in the first case, it has a bug. And so, if it takes three weeks to run the program, maybe it would make sense to spend half an hour ahead of time optimizing all the code that's going to be run. But obviously, especially in dynamic languages like Python, where we do as much as we can without asking the user to tell us exactly how they need it done, you just want to start executing code as quickly as you can. So that if it's a small script, or a large program that happens to fail early, or just exits early for a good reason, you don't spend any time being distracted by optimizing all that code.

    So, what we try to do there is keep the bytecode compiler simple so that we get to execute the beginning of the code as soon as possible. If we see that certain functions are being executed many times over, then we call that a hot function, and some definition of “hot”. For some purposes, maybe it's a hot function if it gets called more than once, or more than twice, or more than 10 times. For other purposes, you want to be more conservative, and you can say, “Well, it's only hot if it's been called 1000 times.”

    The specializing adaptive compiler (PEP 659) then tries to replace certain bytecodes with bytecodes that are faster, but only work if the types of the arguments are specific types. A simple hypothetical example is the plus operator in Python. It can add lots of things like integers, strings, lists, or even tuples. On the other hand, you can't add an integer to a string. So, the optimization step - often called quickening, but usually in our context, we call it specializing - is to have a separate “binary add” integer bytecode, a second-tier bytecode hidden from the user. This opcode assumes that both of its arguments are actual Python integer objects, reaches directly into those objects to find the values, adds those values together in machine registers, and pushes the result back on the stack. 

    The binary adds integer operation still has to make a type check on the arguments. So, it's not completely free but a type check can be implemented much faster than a sort of completely generic object-oriented dispatch, like what normally happens for most generic add operations.

    Finally, it's always possible that a function is called millions of times with integer arguments, and then suddenly a piece of data calls it with a floating-point argument, or something worse. At that point, the interpreter will simply execute the original bytecode. That's an important part so that you still have the full Python semantics.

    Utsav [18:20] Generally you hear of these techniques in the context of JIT, a Just-In-Time compiler, but that’s not being implemented right now.

    Just-In-Time compilation has a whole bunch of emotional baggage with it at this point that we're trying to avoid. In our case, it’s unclear what and when we’re exactly compiling. At some point ahead of program execution, we compile your source code into bytecode. Then we translate the bytecode into specialized bytecode. I mean, everything happens at some point during runtime, so which part would you call Just-In-Time?

    Also, it’s often assumed that Just-In-Time compilation automatically makes all your code better. Unfortunately, you often can't actually predict what the performance of your code is going to be. And we have enough of that with modern CPUs and their fantastic branch prediction. For example, we write code in a way that we think will clearly reduce the number of memory accesses. When we benchmark it, we find that it runs just as fast as the old unoptimized code because the CPU figured out access patterns without any of our help. I wish I knew what went on in modern CPUs when it comes to branch prediction and inline caching because that is absolute magic.

    Full Transcript

    Utsav: [00:14] Thank you, Guido, for joining me on another episode of the Software at Scale podcast. It's great to have you here. 

    Guido: [00:20] Great to be here on the show. 

    Utsav: [00:21] Yeah. And it's just fun to talk to you again. So, the last time we spoke was at Dropbox many, many years ago. And you got retired, and then you decided that you wanted to do something new. And you work on performance now at Microsoft, and that's amazing. So, to start off with, I just want to ask you, you could pick any project that you wanted to, based on some slides that I've seen. So, what got you interested in working on Python performance?

    Guido: [00:47] In some sense, it was probably a topic that was fairly comfortable to me because it means working with a core of Python, where I still feel I know my way around. Some other things I considered briefly in my first month at Microsoft, I looked into, “Well, what can I do with Azure?”, and I almost immediately remembered that I was not cut out to be a cloud engineer. That was never the fun part of my job at Dropbox. It wasn't the fun part of my job before that at Google either. And it wouldn't be any fun to do that at Microsoft. So, I gave up on that quickly. I looked in machine learning, which I knew absolutely nothing about when I joined Microsoft. I still know nothing, but I've at least sat through a brief course and talked to a bunch of people who know a lot about it. And my conclusion was actually that it's a huge field. It is mostly mathematics and statistics and there is very little Python content in the field. And it would take me years to do anything interesting with the non-Python part and probably even with the Python part, given that people just write very simple functions and classes, at best in their machine learning code. But at least I know a bit more about the terminology that people use. And when people say kernel, I now know what they mean. Or at least I'm not confused anymore as I was before.

    Utsav: [02:31] That makes sense. And that is very similar to my experience with machine learning. Okay, so then you decided that you want to work on Python performance, right? And then you are probably familiar with Mark Shannon's ideas?

    Guido: [02:43] Very much so. Yeah.

    Utsav: [02:44] Yeah. So, was there anything different about the set of ideas that you decided …



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev
    32 min
  • Software at Scale 33 - Drone Engineering with Abhay Venkatesh

    Abhay Venkatesh is a Software Engineer at Anduril Industries where he focuses on infrastructure and platform engineering.

    Apple Podcasts | Spotify | Google Podcasts

    We focus this episode on drone engineering - exploring the theme of “If I wanted to start my own technology project/company that manages drones, what technology bits would I need to know?” We discuss the commoditization of drone hardware, the perception stack, testing and release cycles, simulation software, software invariants, defensive software architecture, and wrap up with discussing the business models behind hardware companies.

    Highlights

    1:56 - Are we getting robot cleaners (other than Roomba) anytime soon?

    5:00 - What should I do if I want to build a technology project/company that leverages drones? Where should I be innovating?

    7:30 - What does the perception stack for a drone look like?

    13:30 - Are drones/robots still programmed in C++? How is Rust looked at in their world?

    18:30 - What does software development look like for a company that deploys software on drones? What are the testing/release processes like?

    20:30 - How are simulations used? Can game engines be used for simulations to test drones? Interestingly - since neural networks perceive objects and images very differently from how brains do it, adapting drone perception to work on a game engine is actually really hard.

    26:30 - Drone programming can be similar to client-side app development. But you have to write your own app store/auto-update infrastructure. Testing new releases manually is the largest bottleneck in releases.

    30:00 - Defensive programming for drones - how do you ensure safety? What is the base safety layer that needs to be built for a drone? “Return to Base” logic - often separated out into a different CPU.

    33:00 - How do hardware businesses look different from traditional SaaS businesses?

    38:00 - What are some interesting trends in hardware that Abhay is excited about?



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev
    42 min
  • Software at Scale 32 - Derrick Stolee: Principal Software Engineer, GitHub

    Derrick Stolee is a Principal Software Engineer at GitHub, where he focuses on the client experience of large Git repositories.

    Apple Podcasts | Spotify | Google Podcasts

    Subscribers might be aware that I’ve done some work on client-side Git in the past, so I was pretty excited for this episode. We discuss the Microsoft Windows and Office repository’s migrations to Git, recent performance improvements to Git for large monorepo, and more.

    Highlights

    lightly edited

    [06:00] Utsav: How and why did you transition from academia to software engineering?

    Derrick Stolee: I was teaching and doing research at a high level and working with really great people. And I found myself not finding the time to do the work I was doing as a graduate student. I wasn't finding time to do the programming and do these really deep projects. I found that the only time I could find to do that was in the evenings and weekends because that's when other people weren't working, who could collaborate with me on their projects and move those projects forward. And then, I had a child and suddenly my evenings and weekends aren't available for that anymore.

    And so the individual things I was doing just for myself and for, you know, that was more programming oriented, fell by the wayside. I'd found myself a lot less happy with that career. And so I decided, you know what, there are two approaches I could take here. One is I could spend the next year or two winding down my collaborations and spinning up more of this time to be working on my own during regular work hours. Or I could find another job and I was going to set out.

    And, I lucked out that Microsoft has an office here in Raleigh, North Carolina, where we now live. This is where Azure DevOps was being built and they needed someone to help solve some graph problems. So it was really nice that it happened to work out that way. I know for a fact that they took a chance on me because of their particular need. I didn't have significant professional experience in the industry.

    [21:00] Utsav: What drove the decision to migrate Windows to Git?

    The Windows repository moving to Git was a big project driven by Brian Harry, who was the CVP of Azure DevOps at the time. Previously, Windows used this source control system called Source Depot, which was a fork of Perforce. No one knew how to use this version control system until they got there and learned on the job, and that caused some friction in terms of onboarding people.

    But also if you have people working in the windows code base for a long time, they only learn this version control system. They don't know Git and they don't know what everyone else is using. And so they're feeling like they're falling behind and they're not speaking the same language when they talk to somebody else who's working commonly used version control tools. So they saw this as a way to not only update the way their source control works to a more modern tool but specifically allow this more free exchange of ideas and understanding.

    The Windows Git repository is going to be big and have some little tweaks here and there, but at the end of the day, you're just running Git commands and you can go look at StackOverflow to solve questions as opposed to needing to talk to specific people within the Windows organization and how to use this version control tool.

    Transcript

    Utsav Shah: Welcome to another episode of the Software at Scale Podcast, joining me today is Derek Stolee, who is a principal software engineer at GitHub. Previously, he was a principal software engineer at Microsoft, and he has a Ph.D. in Mathematics and Computer Science from the University of Nebraska, welcome. 

    Derek Stolee: Thanks, happy to be here. 

    Utsav Shah: So a lot of work that you do on Git, from my understanding, it's similar to the work you did in your Ph.D. around graph theory and stuff. So maybe you can just walk through the initial like, what got you interested in graphs and math in general?

    Derek Stolee: My love of graph theory came from my first algorithms class in college my sophomore year, just doing simple things like path-finding algorithms. And I got so excited about it, I started clicking around Wikipedia constantly, I just read every single article I could find on graph theory. So I learned about the four-color theorem, and I learned about different things like cliques, and all sorts of different graphs, the Peterson graph, and I just kept on discovering more. I thought this is interesting to me, it works well with the way my brain works and I could just model these things while [unclear 01:32]. And as I kept on doing more, for instance, graph theory, and combinatorics, my junior year for my math major, and it's like I want to pursue this. Instead of going into the software, I had planned with my undergraduate degree, I decided to pursue a Ph.D. in first math, then I split over to the joint math and CS program, and just worked on very theoretical math problems but I also would always pair it with the fact that I had this programming background and algorithmic background. 

    So I was solving pure math problems using programming, and creating these computational experiments, the thing I call it was computational competent works. Because I would write these algorithms to help me solve these problems that were hard to reason about because the cases just became too complicated to hold in your head. But if you could quickly write a program, to then over the course of a day of computation, discover lots of small examples that can either answer it for you or even just give us a more intuitive understanding of the problem you're trying to solve and that was my specialty as I was working in academia.

    Utsav Shah: You hear a lot about proofs that are just computer-assisted today and you could just walk us through, I'm guessing, listeners are not math experts. So why is that becoming a thing and just walk through your thesis read in super layman terms, what do you do?

    Derek Stolee: There are two very different ways what you can mean when you say I have automated proof, there are some things like Coke, which are completely automated formal logic proofs, which specify all the different axioms and the different things I know to be true. And the statement I want to prove and constructs the sequence of proof steps, what I was focused more on was taking a combinatorial problem. For instance, do graphs with certain sub-structures exist, and trying to discover those examples using an algorithm that was finely tuned to solve those things, so one problem was called uniquely Kr saturated graphs. A Kr was essentially a set of our vertices where every single pair was adjacent to each other and to be saturated means I don't have one inside my graph but if I add any missing edge, I'll get one. And then the uniquely part was, I'll get exactly one and now we're at this fine line of doing these things even exist and can I find some interesting examples. And so you can just do, [unclear 04:03] generate every graph of a certain size, but that blows up in size. 

    And so you end up where you can get maybe to 12 vertices, every graph of up to 12 vertices or so you can just enumerate and test. But to get beyond that, and find the interesting examples, you have to be zooming in on the search space to focus on the examples you're looking for. And so I generate an algorithm that said, Well, I know I'm not going to have every edge, so it's fixed one, parents say, this isn't an edge. And then we find our minus two other vertices and put all the other edges in and that's the one unique completion of that missing edge. And then let's continue building in that way, by building up all the possible ways you can create those sub-structures because they need to exist as opposed to just generating random little bits and that focus the search space enough that we can get to 20 or 21 vertices and see this interesting shapes show up. From those examples, we found some infinite families and then used reg…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev
    1 hr 7 min
  • Software at Scale 31 - Maju Kuruvilla: CTO/COO, Bolt

    Maju Kuruvilla is the CTO and COO of Bolt, a startup that offers quick online checkout technology to retailers. Previously, he was VP and GM at Amazon Global Mile, in charge of Amazon’s global logistics and Amazon Prime fulfillment operations amongst other things.

    Apple Podcasts | Spotify | Google Podcasts

    Highlights

    00:30 - What does a VP at Amazon even do? The day-to-day experience of a VP/GM at Amazon. I think I’ve asked enough people this question that I finally have a vague sense of what these engineering leaders do (I think).

    04:00 - Managing global logistics in one of the world’s largest logistics companies in the middle of a pandemic

    09:00 - Shipping software quickly when you’re a large company. Two pizza teams with a twist.

    16:00 - The role of software in global logistics. Amazon’s epic migration off Oracle databases. How to get thousands of people interested in migration or similar work.

    25:00 - Launching Amazon Prime Now in 111 days (21 days more than what Jeff Bezos mandated).

    38:00 - The complexity behind a checkout operation in an online store. Tax operations, compliance (!), and other complexities.

    46:00 - A tech stack to solve the checkout problem.

    51:00 - Building trust, relationships, and making an impact as an engineering leader in a new company. Everyone wants to hire great people, but what does that really mean?

    Transcript

    Utsav Shah: Welcome to another episode of the Software at Scale Podcast, joining me today is Maju Kuruvilla, who is the CTO and now CEO of Bolt. Previously, he was VP at Global Mile, which is an organization at Amazon that was in charge of global fulfillment. Thank you for joining me.

    Maju Kuruvilla: Thanks for inviting me, Utsav. It's great to be here.

    Utsav Shah: Maybe we can start with what exactly does a VP of Global Mile at Amazon do? How many people are reporting to you eventually and what is your day-to-day look like at that time?

    Maju Kuruvilla:  So I had few different roles at Amazon, the last role was the VP of Global Mile. Before that, I was a VP of the Worldwide Fulfillment technology team and the difference is when I was VP of the Worldwide Fulfillment technology team, we were responsible for all tech and products that Amazon uses in our fulfillment centers worldwide and so that's kind of global responsibility. And then a move to Global Mile, which was a little bit more a general manager role, where I was responsible for not just the technology and products, but also the operations, the sales, all of the different components of that as an end-to-end business. 

    In either of those roles were global had more than 1000s of engineers and a lot of product managers? And even when I was managing the Worldwide Fulfillment team and even hardware teams and networking teams, all of them are involved as part of the team. The difference in the Global Mile was more being responsible for a P&L and an entire business, it is a little bit different than the Global Fulfillment side and I'm happy to explain on either side but just want to put the difference out there, what's up.

    Utsav Shah: So maybe you can just explain both of those things, I'm sure you're looking at P&L and then going into specifics understanding of what's going on? What does your day-to-day look like? I don't know if you can talk about any specific projects that you did, I think that would be super interesting to know about.

    Maju Kuruvilla: When I took on the Global Mile role; first of all, what Global Mile is, before it used to be called Global Logistics, it's all the logistics that is done to connect between countries. So whenever an item moves between countries and has to cross a border, that's where global, logistics or global mile comes into the picture. Most of the items that we sell in the US or Europe comes from manufacturing countries like Asia or China. And so when we have to bring those items from there to the US, that's part of the global logistics role. So I took on this role right before the pandemic hit and the pandemic initially hit China, which I was responsible for running the China operations at that time. 

    And then when the pandemic then expanded and spread out to the Western countries, and then the rest of the world, and running global logistics during that time was very challenging and exciting. At the same time, there are a lot of changes that got disrupted that even today are not restored to what it was before. So the global supply chain is still recovering from all the problems that have happened since the beginning of COVID. So since I took on the role, it was largely doing catch-up on how to do, what do we do with the China operations? How do we manage our people there? How do we get the operations up and running? How do we keep our people safe during this process? 

    And then, when all the planes stop flying or the passenger flight stop flying, the majority of the cargo space was not available after that. So we had to figure out how to create more air capacity between countries and then there was this big backlog of things that were coming from China all over the world, all the way from the challenges at the beginning of the port in China to the port of Los Angeles. For example, you could see lines of ships that are waiting to get unloaded and just managing all of that process [5:00] and reinventing and figuring out how to solve the global logistics problem in the middle of the pandemic was the highlight of my time at Global Logistics.

    Utsav Shah: Sounds like a fun on boarding project.

    Maju Kuruvilla: It's hard to get trained for global logistics anywhere because Amazon does a lot of that. On top of that, dealing with that during a pandemic was certainly something I was not ready for but like with every challenge and every job; you just have to figure things out. And they've created a lot of great opportunities for innovations, we did a lot of things that we wouldn't have done at the speed at which we did if we were not presented with a constraint. So a lot of innovations came out of that, a lot of new capabilities came out of that, and it was great to have a strong team at Amazon, where everyone responded fast, and started building things fast and get [unclear 06:08] operational in a very short time.

    Utsav Shah: Are you at liberty to share any of those innovations that you talked about? 

    Maju Kuruvilla:  A few I can certainly share; one is the air capacity problem, like, around 45% of air cargo capacity comes from the belly cargo from the passenger flight. So when passenger flights stopped flying, you essentially lost 45% of the global air cargo capacity, it’s just completely gone. And so how do we recreate that, so we had to create, start thinking of running out on charters. So we started renting planes, like 747  to fly from China to the US and Europe, from the US to export countries and so we started releasing planes and started flying, which we never had before and that was something we created last minute and started flying. 

    An interesting story there was we started even getting the cargo planes for lease became very expensive and because everyone was trying to do something similar, we started renting even VIP jets. At one point, we rented a VIP jet that even I have never been on but then we were putting packages on it and shipping all over the world. So creating that kind of air capacity in a very short time and then building that into a framework that can be used for longer-term was something amazing and quick we did and whenever we have to do that a lot of things need to come together. 

    One is, do you need to have the right technology for us to organize all the different products at source, figure out the route at which we all need to go, and then how do we fill this capacity when on the airplanes. You put something called a ULD which is a box that you fill all the items in and that's what you load and unload. And so you have this complex math problem of how do we fill it because you want to mix the right amount of weight and cube so th…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev
    59 min