Software at Scale

David Cramer is the co-founder and CTO of Sentry, a monitoring platform helps every developer diagnose, fix, and optimize the performance of their applications. Before this, he worked at Dropbox and Disqus.

Apple Podcasts | Spotify | Google Podcasts

We discuss the evolution of Sentry from both a technical and business perspective: the architecture, some outages, hiring, improving the customer experience, and more.

Highlights

Notes are highlighted

0:00 - The history of Sentry

10:00 - The initial architecture of Sentry. It started off as a Django plugin, and would write to the production database.

12:10 - Keeping software simple.

14:39 - The scalability of PostgreSQL. MVCC.

19:00 - Sharding databases (or not) at Sentry. Multi-tenancy of customer’s exceptions. The overkill nature of most solutions.

23:30 - Transaction ID Wraparound - the source of Sentry’s largest outage. Customer reactions to large outages. Robinhood and Gamestop.

35:00 - Scaling out engineering quality when hiring more engineers. Hiring engineers in charge of developer productivity. Setting engineering culture, and making an impact as the CTO.

44:30 - Challenges while growing out the company to more than a hundred employees and over a million users

47:00 - Hiring outside of your network, and managing attrition.

54:30 - Transitioning from CEO to CTO

66:00 - Maintaining transparency with customers in a devops company

68:00 - The evolution of developer tools companies in the last few years.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Beyang Liu is the CTO and co-founder of Sourcegraph, a code search and intelligence tool for developers. He used to be a software engineer at Palantir, and a research assistant at Stanford University.

Apple Podcasts | Spotify | Google Podcasts

Highlights

1:00 - The story behind Sourcegraph

6:12 - How is Sourcegraph different from a tool like grep (or ripgrep)?

14:25 - Social code search via links are a bigger deal than one might imagine

17:00 - Engineering challenges in building Sourcegraph.

29:30 - The product feedback loop

38:30 - The role of a CTO in an early stage company, and how that’s evolved over time. Maintaining quality as the engineering team gets bigger

56:00 - Learnings behind starting a developer tools business.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Farhan Thawar is VP Engineering for Channels and Mobile at Shopify. Previously he was the CTO and Co-Founder at Helpful, CTO Mobile at Pivotal and VP, Engineering at Pivotal Labs. He was named one of Toronto's 25 most powerful people.

Apple Podcasts | Spotify | Google Podcasts

We discussed frameworks for making life decisions, the structure of engineering organizations, setting up successful hiring pipelines and healthy organizations, committing to large engineering decisions (like React Native for Shopify mobile apps), pair programming, growing engineers and leaders, and more.

Highlights

1:28 - A framework for making long term decisions. “Work with smart people on hard problems”.

9:00 - The varying experiences of being an engineering leader at both small and large companies.

12:30 - The Roles and Responsibilities of a VP of Engineering

16:30 - Diving into the decision of going with React Native for Shopify mobile applications. What’s the process to make that decision?

26:12 - Should large scale decisions like these be bottom up or top down? What’s the tradeoff.

31:00 - How much should an engineering organization invest into platform, vs. customer facing work? Why Farhan thinks they might have over-engineered their systems at Helpful.

35:30 - Measuring developer productivity. A well reasoned, first principles argument against using commit count

40:00 - Managing 120 direct reports at Pivotal, and how to keep an open mind about the actual scaling limits of engineering leaders. The value of scheduled vs. unscheduled 1:1s.

49:30 - Pair programming, especially in a remote setting

52:00 - Hiring 2021 engineers in 2021 - managing that hiring pipeline

64:00 - High leverage activities to drive high quality hiring

67:00 - How to think about attrition in engineering organizations? Is there a magical percentage to keep track of? Segmentation is key.

72:00 - The best way to grow engineers and engineering leaders at a company

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Charity Majors is the CTO of Honeycomb, a platform that helps engineers understand their own production systems through observability. Honeycomb is very different from traditional monitoring tools like Wavefront as it is built for data with high cardinality and high dimensionality, which can instantly speed up debugging of many problems.

Apple Podcasts | Spotify | Google Podcasts

NOTE: This episode has some explicit language.

We talk about observability, monitoring, building your own database for a particular use case, starting a developer tool startup, having the right oncall culture, getting to fifteen minute deployments and more.

Highlights

Notes are italicized

05:00 - High cardinality and high dimensionality in Honeycomb. Data retention in Honeycomb - 60 days. Many monitoring systems, like Dropbox’s Vortex, downsample data in two weeks

13:00 - Observability driven development. The impact of deploying code within 15 minutes of it being merged in. Synchronous and asynchronous engineering workflows

19:00 - Setting up oncall rotations. What the size of a rotation should be

21:00 - How often should someone on a 24/7 oncall rotation be woken up? Once or twice. But there are exceptions. The impractical nature of some of Google SRE book’s “Being Oncall” chapter. Oncall for managers

31:00 - Why are monitoring tools so ubiquitous compared to observability tools?

36:00 - Observability & Tracing. What the future of observability infrastructure might look like

40:00 - What will the job of an SRE look like in the future? The split of roles in software engineering organizations in the future

43:00 - Shipping code faster makes engineers happier. How do you ensure your engineering organization is healthy, and the metrics to use. Learned helplessness in engineering organizations, and leadership failures

51:00 - Building internal tools in-house vs using external tools. The large impact that designers at Honeycomb have had on the product.

58:00 - The story of starting Honeycomb. Creating a “Minimum Lovable Product”. A description of Honeycomb internal architecture. Dealing with tail latencies.

71:00 - Continuous Deployment and releasing code quickly. Use calendly.com/charitym if you want to chat with Charity about continuous deployment best practices or anything else.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Indranil Gupta (Indy) is a Professor of Computer Science at the University of Illinois, Urbana Champaign. He leads the DPRG (Distributed Protocols Research Group) and runs popular Cloud Computing MOOCs on Coursera. His work has inspired software that runs in many production services, like Serf in Nomad by Hashicorp. and Uber’s Ringpop.

Apple Podcasts | Spotify | Google Podcasts

We discussed how academia drives progress in distributed systems, a bit of blockchains, distributed systems and machine learning, doing quality research, working as a visiting researcher in industry, cluster schedulers, and how to choose between going into industry vs. academia.

Highlights

0:00 - Going into academia. “Make a list of pros and cons, throw away that list, and go with your gut”.

6:30 - The acceptance of distributed systems in early 2000s vs. today

7:30 - The emergence of blockchain and how the world treats it. “Re-looking the wheel, vs re-inventing the wheel”

12:30 - Differences in Computer Science research in industry and academia, especially with distributed systems.

21:00 - The inspiration for solving the reconfiguration problem came from open source bug reports. “Directions that seem daunting for folks in industry since the solutions are unclear can be tackled by academia”. Making progress towards solving online reconfiguration problem.The similarity of research to intern projects.

25:30 - How to pick an area to do research in.

31:00 - Writing a good paper is “90% good communication and 90% good ideas”.

31:20 - SWIM - a paper that’s had an outsized industry impact due to good ideas that were well explained

38:00 - “What are you excited about in distributed systems today?” - Distributed Systems + X. Machine Learning, Agriculture, etc. For examples, dealing with malicious workers in distributed machine learning (example)

47:00 - Training and inference with new machine learning systems like GPT3 becomes a distributed systems problem. Partitioning of data in Tensorflow.

56:00 - How can industry and academia collaborate so that industry produces more research?

66:00 - What Indy learnt by working in industry for a year (at Google)? Borg at Google (a predecessor to Kubernetes). Omega + Mesos

73:00 - Advice for a student evaluating a choice between academia and industry

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Kyle is a Software Engineer at Dropbox. He’s currently working in the Core Sync area in charge of the Dropbox Sync Engine and other related components. He was previously the tech lead of the Build Infrastructure team, the tech lead of the Android team at Dropbox, and a senior engineer at Google.

Kyle has been thinking about developer productivity for a while, and his tenure at Dropbox (~9 years) gives him a unique insight into the origins and consequences of some large technical decisions, the value of code review, and making developers more effective in general.

Listen on Apple Podcasts or Spotify.

We discussed technical migrations, test quarantine, the purpose of code review, multi-round code reviews, the popularity and community of Rust, the properties of Go, good software design, the design of the Dropbox sync engine, the role of a tech lead, and deciding between staying an IC vs. going into management.

Highlights

2:20 - Build Infrastructure at Dropbox. Dropbox has an in house CI Server that was written by David Cramer (CTO of Sentry). Dropbox’s internal CI Server might have been the first user of Sentry. How Bazel came about at Dropbox.

5:40 - Bazel might have been an overcorrection at Dropbox. The danger of copying Google for internal tools.

8:23 - The precision of Bazel and infrastructural investments to keep it running. Moving things into the Bazel world could be challenging.

17:30 - How should a decision maker decide between investing in a customized but potentially superior developer experience, vs sticking with open source tools or external best practices? For example: Git vs. Mercurial

24:00 - Enforcing the right amount of “best practices” on the rest of engineering.

26:30 - One way to think about developer tools in your organization.

33:40 - The issues you run into if you apply a systems engineering mindset to developer tools, compared to a product development mindset.

37:42 - What test quarantine is, and why an engineering organization needs it.

47:40 - Viewpoints on code review and how they’ve evolved over time. The consequences of not catching a bug in code review.

56:20 - Maintaining the reliability of applications that are being deployed on millions of desktop hosts and Android devices. User reports were often wrong.

62:30 - Being a long time language nerd, and the code design of a complex sync engine. Not using standard library hashmaps due to non determinism.

77:50 - A discussion on Golang.

88:00 - Contrasting Golang and Rust. Diffused benefits vs. specific benefits.

96:50 - Being a good tech lead, and deciding between staying an individual contributor, vs. going into management. Being smart vs. being obsessive as a reason for success in software, compared to other things. The difficulty of being a Tech Lead Manager (TLM).

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Akshay Shekhar is a software engineer working on Amazon HoneyCode, a no code tool to build applications. He’s been in a few different teams at Amazon, and he used to be a maintainer of Elementary OS (and his name is still on the team page).

We talk about the experience of being an open source developer, working in startups, corporate tech blogs, evaluating starting a startup vs. staying in a tech company, moving from India to Canada, AWS organizational structure, a comparison between technology choices and communism, building reliable systems, and the technical challenges faced in building a no code platform.

Apple Podcasts | Spotify | Google Podcasts

To send feedback: Email | Twitter

Note: Thanks for all the feedback. I’ve set up a Google form for guest suggestions if you/someone you know would be a good guest for the show. Merry Christmas, and Happy Holidays!

Highlights

Notes are Italicized

0:00 - Introduction

4:45 - How Akshay got into open source development

7:01 - Canonical emailed Akshay a CD since downloading it would have taken a month at that time. Those were the days..

Elementary OS is its own separate OS.

8:45 - A controversial Elementary OS blog post.

15:39 - College Experience, and internships.

21:20 - Pivoting to Amazon. Akshay was inspired to join Amazon from a blog post about some scale issues (The only thing I could dig up was this). We talk about the hidden impact of corporate tech blogs. Mostly in agreement with Dan Luu’s take on it.

24:00 - Starting off at Amazon Data Extraction as an intern. Solving address matching.

31:50 - Technology at a large company that can easily be spun into a startup (and evaluating whether one should join a startup).

37:00 - Contrasting the Google and Amazon approach on building software. “It’s like Communism vs. Capitalism”. Yes, I really like this question.

45:00 - Working with research engineers in Amazon, and unnecessary SIMD optimizations.

48:00 - Moving to AWS Networking. Switching teams at Amazon is very normalized. "For the first couple of months, when you switch teams, don’t speak. Listen”.

53:30 - “Respect what came before” from Principal Engineering Community Tenets. Chesterton’s Fence.

55:10 - Akshay worked on a machine database for AWS

57:30 - What kind of organizational structure facilitates the various number of offerings that AWS provides? How is the system kept reliable when there’s so many moving parts that could fail? Being customer obsessed is key.

65:40 - Working on Amazon HoneyCode. The technical challenges that come from designing an application that lets users design applications. It sounds like a fascinating problem.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Bharat Mediratta was a Distinguished Engineer at Google, CTO at Altschool, and CTO at Dropbox. At Google, he worked on GWS (Google Web Server), a system that I’ve always been curious about, especially since its Wikipedia entry calls it “one of the most guarded components of Google's infrastructure”.

In this podcast, we discuss GWS, bootstrapping a culture of testing at Google, breaking up services to be more manageable, monorepos, build systems, the ethics of software at scale, and more. We spent almost an hour and a half, and didn’t even manage to cover his experiences at Altschool or Dropbox (which hopefully will be covered in a follow up).

Listen on Apple Podcasts or Spotify.

Highlights

Notes are italicized.

0:20 - Background - Childhood interests in technology. His dad was a director at ADE, India. His dad recruited APJ Abdul Kalam, arguably one of India’s most popular Presidents, and kick-started his career.

6:10 - Studying tech in university. Guru Meditation errors.

10:50 - Working at Sun Microsystems as a first job.

12:30 - Transitioning from being a programmer to a leader, and thinking about project plans and deadlines.

14:15 - Working on side projects for the company (a potential inspiration for 20% projects at Google?)

15:30 - Moving from Sun to a few startups to Google. How did 20% projects start?

16:50 - Google News as a 20% project. Apparently 20% projects has its own wikipedia page.

18:24 - Did 20% time require management approval?

19:30 - TK at Google Cloud, and how the management model compares to early Google

21:00 - Declining an offer from Google at 2002, and going to VA Linux instead.

22:28 - Growth at Google from 2004 onwards.

24:28 - Hiring at Google at that time. “A players hire A players, B players hire C players”.

24:55 - Culture Fit (indoctrination)? Two weeks of “fairly intense education”, a Noogler project, and a general investment of time and money to help explain the Google way of doing things. It wasn’t accidental. I went through this in 2016 and definitely learnt a bunch, especially from an intriguing talk called “Life of a Query”.

27:22 - Culturally integrating acquisitions successfully. YouTube as an example.

28:40 - Differences between Google and YouTube, and other acquisitions like Motorola Mobility.

30:20 - Search/Google Web Server (GWS) only had 3 nines of availability? The difference between a forager and a refiner (in terms of programming)

31:15 - What was GWS? Server responsible for Google Home and Google Search.

32:20 - There was only one infrastructure engineer on GWS at the time (who wanted to switch), but about a hundred engineers made changes to it every week.

33:10 - Starting with writing unit tests for this system.

33:40 - “They” used to call GWS “the neck of Google”. Extremely critical, but also extremely fragile. Search results and 98% of revenue came through this system. One second of downtime implied revenue loss. Rewriting was infeasible.

34:50 - How to use unit tests to create a culture of shared understanding. Bharat released a manifesto that basically said “all changes to GWS required unit tests”. This caused massive consternation at the time.

36:10 - A quick example on how to enforce unit tests on new code. If an engineer didn’t add a new unit test, Bharat would write the test for them, which often would be failing due to a bug in engineer’s code. This led to a culture where engineers realized the value of writing these tests (and implicitly

39:23 - New Googlers were taught to write unit tests, so that new engineers would spread a culture of writing tests. “Oh, everyone writes unit tests at Google”.

41:50 - “What kind of features were those hundreds of engineers adding to GWS?”. An example - adding UPS tracking numbers automatically showed you UPS tracking results. These were all quiet launches.

Some of the software design around experimentation towards Google search might have influenced Optimizely’s design.

45:00 - Google’s search page in 2007 was pure HTML. In 2009, it was completely AJAX based. This was a massive shift that happened transparently for users.

46:00 - “We wanted Search to be a utility. We wanted it to be the air you breath. You don’t turn on the faucet and worry that water doesn’t come out.”

47:40 - The evolution of GWS’s architecture. Initially, very monolithic. GWS would talk to indices, get results, rank results, and send back HTML. This eventually was broken into layers. Each layer had responsibility, and the plan was to stick to that.

49:00 - “You could find one line of code in GWS that had C code, C++ code, HTML, Javascript, and CSS output”. Wow.

The number one query at Google at the time was “Yahoo” - a navigational search query.

50:00 - Google Instant was rolled out in 2010. Internally, this was called “Google Psychic”, cause it was pretty good at predicting what users wanted to search.

51:50 - “A rewrite would have been a disaster”. GWS was essentially refactored from inside out every 18 months for 11 years. The first one - was breaking out ranking from GWS to another service.

57:00 - YouTube knew that if it convinced enough people to get better internet, Google would make more revenue.

59:00 - Search grew from 500-1000 people in 2004, to 3000 people in 2010.

59:30 - How exactly did search ranking work, technically and organizationally? The Long Click.

61:40 - Google ran 20+ experiments to figure out the best shade of blue on the Search page. This might seem silly, but it helps at scale, since it could potentially find the shade that would help the most colorblind individuals.

67:50 - Hate speech from Google search, and the ethical quandaries around building a humanity scale system.

70:30 - Improving iteration speed and developer productivity for these systems

71:50 - Google had an ML model for search results back in 2004 that was competitive with the hand-built systems, but didn’t end up using it, due to the lack of understandability. This has definitely changed now. I had read that document during my internship, but was surprised to learn that Google had a working ML model for ranking since 2004.

73:30 - Service Oriented Architecture at Google. Enabled GWS from C to C++ and divest itself from some responsibilities. But Google stuck with a monorepo, compared to Amazon.

76:40 - Components in the Monorepo + Blaze (Bazel) helped Google scale build times and reduce iteration speed. Components is the most interesting piece, since to my understanding, it hasn’t been written about much externally.

78:00 - The scale and complexity of the monorepo.

79:40 - The 400,000 line Makefile, and the start of Blaze.

82:00 - What were the benefits of “Components”?

84:00 - The project to multi-thread GWS, when it was serving 5 - 10 billion search queries a day. It started off as a practical joke.

91:00 - It’s rarely only about the technology. It’s about culture and team cohesion.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

This episode contains an interview with Christine Dodrill, ex Senior Software Reliability Expert at Lightspeed.

We discuss Kubernetes, Spectre/Meltdown, configuration languages, a controversial testing philosophy, autoscaling (auto-failing), technical problems vs social problems, monoliths, Conway’s Law and Canada.

Listen on Apple Podcasts or Spotify.

Highlights

Notes are italicized.

5:34 - Stack Overflow might become actively harmful if you’re working on WebAssembly or something sufficiently niche.

7:40 - Spectre/Meltdown caused 20-40% slowdown at one workplace, which led to some interesting projects, like the aforementioned WebAssembly work

11:30 - What’s up with the title Senior Site Reliability Expert?

Apparently, you can’t call yourself an “engineer” in Canada, you need to go through some kind of process which software developers don’t need to bother.

14:40 - It’s questionable how much software developers are “engineers” in the first place.

16:52 - YAML allows 8 values for boolean true and false, such as “no”, and “on”, which conflict with ISO code for Norway and Province code for Ontario. Maybe Starlark is an answer. Christine uses Dhall with promising results.

Dhall looks like Haskell. It has a strong type system, with variables, functions, and imports, but otherwise a config language. They have a Kubernetes package.

20:30 - Nix the language. An example to configure a website.

24:00 - Experiences building internal tools that interact with other internal tools. Developers tend to have strange environments. One developer would only keep source code on a thumbdrive, and that would cause a few issues.

27:20 - Compliance requirements can be a useful to stop developers from security snafus.

29:30 - Experience with Kubernetes.

30:40 - Kubernetes Autoscaling out of the box is a great way to cause downtime. Experiences on the Metrics team at Heroku which worked on autoscaling. Most applications tend to be I/O bound to the database, so autoscaling tends to become “auto-failing” and cause more problems than it solves.

37:00 - PostgreSQL, PgBouncer, and Transaction ID wraparound. External postmortem.

40:48 - “A lot of document databases are solutions looking for problems”

45:30 - “Continuous Deployment can be a double edged sword”

47:20 - “A lot of unit testing methodology I’ve seen is kind of fundamentally wrong. A fake version of the world will only let you see how fake your world is”.

53:30 - Experience with tiered deployments - stage, QA, and production.

58:40 - Exploring the model where product engineers only build features, and SREs focus on reliability.

A conclusion is that some governance is probably required to prevent a complexity explosion.

60:30 - Monoliths are pretty great. Eventually, Conway’s Law takes place. Incongruities in products or APIs often reflect team boundaries.

68:00 - Buzzwords at big companies.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Welcome to the first Software at Scale podcast. This episode contains an interview with Alexey Ivanov, Principal Engineer, Infrastructure at Dropbox.

The motivation for yet another software podcast is to let software builders share technical decisions, opinions, and stories in an informal way. Personal blogs and corporate engineering blogs are extremely informative, but often require high activation energy to be published. This podcast instead tries to replicate bar conversations with grizzled senior engineers reminiscing about horrors of their systems and what they’ve learnt over the years.

In this podcast, we discuss object storage, load balancing, build systems like Bazel, Nginx/Envoy config management, monoliths, services, gRPC, and more.

Highlights

Notes are italicized.

0:00 - Introduction

2:20 - Experience working on Object Storage at Yandex in 2012

3:55 - LevelDB wasn’t efficient enough for avatar storage, presumably due to record size. The failure mode was memory consumption, and didn’t seem to work well on spinning drives at the time, so they built a custom storage backend.

5:55 - RocksDB/WiredTiger might be more appropriate for such a use case today. In general, today, it makes sense to take off the shelf components, unless it involves the core of the business and requires innovation. Other examples - Figma and browser based multiplayer design, and Dropbox with Magic Pocket.

8:55 - Experience working on Server Team at Dropbox in 2015. Teams were fairly broad (Server Team, Client Team), and a new Systems Engineering team was created as a lower layer for Server Team that focused on the edge network and runtime concepts like service discovery. Service Discovery at Dropbox today is fairly sophisticated.

10:45 - Dropbox’s stack in 2015. Stateless systems weren’t as mature as the stateful ones, and there might have been a little duct tape involved. David Mah’s talk on securing user data at SRECon is worth listening to.

12:53 - Initial, DNS based service discovery, and Nginx config management and generation via Python and Jinja2.

15:09 - A DNS outage story.

17:30 - Monoliths aren’t that bad, and many successful businesses start off with monoliths.

17:51 - Dropbox doesn’t use the term “microservice”, neither does it encourage too many tiny services. Services shouldn’t be too small or big.

20:10 - How to reasonably manage configs for Envoy - learnings from Nginx config generation. Object Oriented Python that generates a protobuf config helps with a declarative and reusable config format. Materializing to protobuf helps avoid a lot of bugs, like “no” in YAML.

23:05 - “Config languages eventually converge to Turing Complete languages”. Pick the appropriate language based on the engineer who will need to edit these the most. Concretely - Python for Traffic engineers for configurability, a stripped down YAML format for product engineers that need to perform small scoped tasks like adding new routes.

24:50 - What sparked the Nginx to Envoy migration? The killer feature was the community and the ability to participate in the development process. Shout out to Matt Klein, the leader of the Envoy project, for fostering an inclusive community.

28:30 - Envoy aligns with the Google way of development that Dropbox adopted - it works well with gRPC and builds with Bazel. gRPC at Dropbox; Bazel at Dropbox.

29:10 - Initially, Bazel was an unpopular decision at Dropbox, but it ended up as one of the best decisions made. The hermeticity guarantees and the build graph are extremely useful features for incremental builds and tests, tracking down dependencies and keeping deployments (and the deployment system) simple.

“Bazel is like a sewer. You get out of it what you put into it” - Mike Solomon

32:10 - How should someone decide whether Bazel is the right choice for their company? It’s definitely not for everyone. When it starts becoming painful to manage multi language builds, it might become worth it.

36:00 - The “Google” way of development (monorepo, one build system), is very different from the “Amazon” style of team independence in all software decisions. Which approach is better, and why?

The Google style seems to work well for midsize companies that don’t have infinite resources to paper over the inefficiencies and duplication in the Amazon development style. Somewhat alternative opinion: the Google Platform Rant.

40:20 - Why did Dropbox decide to build their own monitoring system as recently as 2019 instead of using something off the shelf?

The answer is mostly cost efficiency. The volume of metrics logged would make an external solution prohibitive. Magic Pocket probably logs a lot of metrics.

44:00 - What’s a project that you’re most proud of?

The transition from systems engineers just managing Nginx configs to rolling out Dropbox’s edge network and the first ten Points of Presence was awesome. That work hit the sweet spot of technical innovation and direct, measurable improvement for users.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev