The Backend Engineering Show with Hussein Nasser

NodeJS Jan 2021 released its security update and its time to go through them!   Resources https://nodejs.org/en/blog/vulnerability/january-2021-security-releases/use-after-free in TLSWrap (High) (CVE-2020-8265) HTTP Request Smuggling in nodejs (Low) (CVE-2020-8287) OpenSSL - EDIPARTYNAME NULL pointer de-reference (CVE-2020-1971)

On Jan 4th 2021 7:14 PST All Slack services went down. This video is an early report of the incident and speculation of what might have caused this outage. We still don’t know what caused the outage, we will wait for the full incident report from slack and I'll make a video once that's up.  https://status.slack.com/

The SolarWinds hack is one of the largest highly coordinated and intelligent attempt to hit enterprise companies. In this video, I briefly explain how smart this is.

While working on a Postgres docker container executing some queries I noticed that my index-only scan query is hitting the heap which it shouldn't. After digging deep I found that it's the shared memory that docker allocates by default. Defaults are never enough, very interesting train of thought that I thought I’d share with you  The Blog I found that helped me find it https://blog.makandra.com/2018/11/investigating-slow-postgres-index-only-scans/

Let us discuss what I'm excited for in Backend Tech in 2021 and answer your great questions

In this video, I go through my process of how I design and architect full software from A-Z. This is part of a Twitter thread that you guys seem to enjoy so I decided to make a video on the topic. Although the spec I generate is usually Backend oriented this is applicable for all software.  Twitter thread https://twitter.com/hnasr/status/1339021983195918337?s=20

In this video, I go through how I overcome procrastination as a software engineer.  What is Procrastination? 0:00How to Defeat  * Reward based system - a reward after achieving  1:20* discipline, remembering why started this, your goal 3:16* Professional - I need to do the work and ship 6:10

2020 retrospective

0:00 Intro

1:00 Summary of the Outage

4:00 Detailed Analysis of the Incident Report

On Dec 14 2020 Google across the globe suffered from an outage that lasted 45 minutes nobody could access most of Google services.

Google has released a detailed incident report discussing the outage, what caused it, technical details on their internal service architecture and what did they do to mitigate and prevent this from happening in this in the future

In this video, I want to take a few minutes to summarize the report and then go into a detailed analysis. You can find youtube chapters to jump to the interesting part of the video. pick your favorite drink, sit back relax, and enjoy. Let's get started.

let's start with an overview of how the google id service works, the client connects to Google authentication service to get authenticated or retrieve account information

The account information is stored in a distributed manner between the different service ids for redundancy.

when an update is made to an account on the leader node, the existing data in all nodes are marked as outdated, this is done for security reasons. Let’s say you updated your credit card info, privated your profile or deleted a comment, it is extremely dangerous to serve that outdated information. This was the key to the outage.

The updated account is then replicated based on Paxos Consensus protocol.

The user id service has a storage quota controlled by an automated quota management solution when the storage usage of the service changes.

the quota is maintained accordingly either reduced or increased based on the demand ..

So What Exactly Happened that caused the outage?

In October 2020, google migrated their quota management to a new system and registered the id service with the new system.

however some parts of the old system remained hooked up specifically the parts regarding the reading of the service usage. And because the service is registered to the new system, the old qouta system reported 0 usage as it should. So when the new quota manement asked its service for its usage it was incorrectly reporting 0.

Nothing happened for a while since there was a grace period, but that period expired on December

Thats when the new quota system kicked and saw the id service with 0 usage and started reducing the qouta for the id service down .. you are not using it why waste?

The quota kept reducing until the service had no space left.

This has caused updates to the leader node to fail, which caused all data to go out of date in all nodes which in turn escalated globally to what we have seen.

Resource

https://status.cloud.google.com/incident/zall/20013

The Backend Engineering Show Live with Hussein Nasser episode 10 we discuss many great questions!! Indexing Woes, The Secret to Backend Interviews, What is on my Bookshelf? Backend Engineering Show