The Swyx Mixtape

Listen to BCS insider: https://www.listennotes.com/podcasts/better-call-saul/103-better-call-saul-insider-5X3QinSj9mV/ (55mins in)

Opening song https://www.youtube.com/watch?v=1ouIqZJ1elE

• https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ 
• https://www.youtube.com/watch?v=fifVuhgvQQ8
• https://lexfridman.com/liv-boeree/ (1h19mins in)

What sphinx of cement and aluminum bashed open their skulls and ate up their brains and imagination?

Moloch! Solitude! Filth! Ugliness! Ashcans and unobtainable dollars! Children screaming under the stairways! Boys sobbing in armies! Old men weeping in the parks!

Moloch! Moloch! Nightmare of Moloch! Moloch the loveless! Mental Moloch! Moloch the heavy judger of men!

Moloch the incomprehensible prison! Moloch the crossbone soulless jailhouse and Congress of sorrows! Moloch whose buildings are judgment! Moloch the vast stone of war! Moloch the stunned governments!

Moloch whose mind is pure machinery! Moloch whose blood is running money! Moloch whose fingers are ten armies! Moloch whose breast is a cannibal dynamo! Moloch whose ear is a smoking tomb!

Moloch whose eyes are a thousand blind windows! Moloch whose skyscrapers stand in the long streets like endless Jehovahs! Moloch whose factories dream and croak in the fog! Moloch whose smoke-stacks and antennae crown the cities!

Moloch whose love is endless oil and stone! Moloch whose soul is electricity and banks! Moloch whose poverty is the specter of genius! Moloch whose fate is a cloud of sexless hydrogen! Moloch whose name is the Mind!

Moloch in whom I sit lonely! Moloch in whom I dream Angels! Crazy in Moloch! Cocksucker in Moloch! Lacklove and manless in Moloch!

Moloch who entered my soul early! Moloch in whom I am a consciousness without a body! Moloch who frightened me out of my natural ecstasy! Moloch whom I abandon! Wake up in Moloch! Light streaming out of the sky!

Moloch! Moloch! Robot apartments! invisible suburbs! skeleton treasuries! blind capitals! demonic industries! spectral nations! invincible madhouses! granite cocks! monstrous bombs!

They broke their backs lifting Moloch to Heaven! Pavements, trees, radios, tons! lifting the city to Heaven which exists and is everywhere about us!

Visions! omens! hallucinations! miracles! ecstasies! gone down the American river!

Dreams! adorations! illuminations! religions! the whole boatload of sensitive bullshit!

Breakthroughs! over the river! flips and crucifixions! gone down the flood! Highs! Epiphanies! Despairs! Ten years’ animal screams and suicides! Minds! New loves! Mad generation! down on the rocks of Time!

Real holy laughter in the river! They saw it all! the wild eyes! the holy yells! They bade farewell! They jumped off the roof! to solitude! waving! carrying flowers! Down to the river! into the street!

• https://www.listennotes.com/podcasts/cartoon-avatars/ep-38-anchor-ceofounder-4m-eDu8Ng75/ (59mins in)
• https://stratechery.com/2022/instagram-tiktok-and-the-three-trends/

Listen to the Copyblogger podcast: https://www.listennotes.com/podcasts/the-copyblogger/rather-than-being-helpful-be-jdsk3ni7upa/ (15mins in)

Tim's article: https://www.timstodz.com/coping-with-imposter-syndrome/

Listen to Indie Hackers: https://www.listennotes.com/podcasts/indie-hackers/041-an-optimistic-nihilists-ldribhqGTOM/ (20mins in)

special! the making of this mixtape was livestreamed today if you're interested in the behind the scenes process

Also recommend listening to the Dropbox talk that he mentions: https://www.youtube.com/watch?v=J8UwcyYT3z0

Transcripts

Courtland Allen
19:43
What about in the early days? Because I know for a lot of founders, those first few months where you're not sure that this is something that's going to work out or be worthwhile can be pretty nerve racking. I know you started off with CoderPad as a side project, but you eventually decided to make it full time, so where there any bumps in the road, or challenges, or any insights that you had growing from zero dollars to, what was it, $4,000/month?

Vincent Woo
20:3
Yeah, I quit it when I hit $4,000 MRR.

Courtland Allen
20:6
What was that decision like?

Vincent Woo
20:8
This is going to sound really stupid, and I had to explain this in the YC interview as well, they were like why $4,000, you haven't quit yet? I was like, I'm quitting at exactly $4,000 MRR, and they were like, why? I was like, okay, here's a reason... it's stupid. It's because when I hit $40 MRR, I posted on Facebook as a joke, haha my business makes $40/month guys, isn't that funny? Then when I hit $400 I was like haha, guess what, I made 10x what I made the last time I posted, that's crazy bro. Then I thought, oh shit, if I do this again at $4,000 that's actually kind of real stakes money, so I might as well quit then. Also $4,000 kind of pays for rent and stuff, so that's why. There's no reason to it, I just did it because I felt like it. I could've quit at any number, I mean if I quit at the beginning it would have been fine too, it didn't really matter.

Courtland Allen
20:57
In San Francisco $4,000 pays for literally just rent, haha.

Vincent Woo
21:1
I had a roommate, we were splitting a one bedroom you know, I had the converted living room kind of situation, one of the shitty old Victorian's in a basement.

Courtland Allen
21:11
Was it a hard transition going from your developer salary to just $4,000/month...

Vincent Woo
21:17
No.

Courtland Allen
21:17
Would you say that you were motivated to... what was pushing you the most, just to increase your revenue or to...?

Vincent Woo
21:21
These motivational questions are hard for me, I don't think I'm like most people. Why was I doing what I did? The truth is, I don't know. I don't actually believe that most people know why they do what they do. I was doing a thing because it seemed like the right thing to do but... okay, I'm going to take a moment to explain. I'm what you might call an optimistic nihilist, like I don't think anything's really real, up to and including money. Money is like a dead person's face painted on a green piece of paper... that it signifies material wealth to me is almost amazing. That, that system actually works to me is terrifying and awesome at the same time. So yeah, I thought it would be fun to make more money, but I knew abstractly that if I failed at CoderPad, literally the worst possible thing that could happen to me... is I would just get a job. Which I had proven that I had been able to do at least a couple times before that, so I wasn't worried about it. There was no anxiety for, because to me this is all a big game.

Courtland Allen
22:22
Yeah, it's almost like a role playing game. Where you're essentially leveling up and acquiring skills, and to what end... I don't know haha.

Vincent Woo
22:31
I think of it more like an open world exploration game, like GTA or whatever. Like just seeing how much you can get away with before everyone figures out that you have no idea what you're doing, and you're just making everything up as you go along.

Courtland Allen
22:41
So one of the reasons I asked you about your motivation was because one of the earlier things that I saw you in was actually a video where you gave a talk at Dropbox.

Courtland Allen
23m 9s
Well that's what I wanted to ask, why give a talk like that?

Vincent Woo
23m 12s
It was to provoke the audience.

Courtland Allen
23m 14s
Did you want to tell them that...

Vincent Woo
23m 17s
That they were doing their lives wrong? Yes.

Courtland Allen
23m 18s
How did they take that? They seemed pretty supportive.

Vincent Woo
23m 21s
They liked it. I mean it was tongue in cheek, obviously. It was in this building, probably on this floor, just a different room, it's Stripe now. We could do the same thing if you want... could come back and do the same talk again, in the same room. Why did I do that? I mean...

Courtland Allen
23m 40s
The impression that I got watching it was that it seems like it was a core principle of yours that people should do this, or that it's better for the world if more people do that.

Vincent Woo
23m 51s
There's more of the premise for the talk than necessarily a core belief of my personality. I mean I was invited to do a talk and they even payed me, it was crazy. This was a best topic I could come up with, so I tried to make it compelling, but I also gave reasons to not start a business. I actually think there are tons of reasons to not do it. Many people I think are unsuited for it, and also it's not terribly pleasant in a lot of ways, so I don't know necessarily that I actually recommend everybody who's listening to start a business. I know that's sort of the premise of Indie Hackers, and also it's acquisition by Stripe is sort of, how do you put it? Grow the GDP of the internet.

Courtland Allen
24m 27s
Exactly, nailed it.

Vincent Woo
24m 28s
Right, so readers at home, if you can start a business, do that, but also maybe don't. It's not easy, it's a lot of work and there are a lot of things that valuable in life that have nothing to do with money, that's how I'd put that.

Courtland Allen
24m 44s
I think one of the cooler things that you touched on, that I've also found to be true is that a lot of people who would love to start a business don't, just because they've never even considered it as an option. Especially being the smaller, Indie Hacker type business where you're just making money and you're not trying to be a unicorn, especially if you're a developer, that doesn't get advertised as much. Do you think that's changing now-a-days?

Vincent Woo
25m 7s
I think it must be changing. If there exists people like you, who's sole job is to promote this lifestyle, I would take as some indication that things are changing a bit. On the other hand, I touched on this in the talk, I think this is kind of cyclical. It used to be a cultural norm that everybody kind of wheeled and dealed, at least that's my impression. If you go in other countries that's way more true too. We probably hit peak corporatism and are trying to dial that back a little bit, I think is natural, it was probably inevitable in some respect that people would get upset with... we've had cultural satire and lampooning of corporate life for decades now. I remember my entire life I would watch stuff like Office Space, or cartoons...

Courtland Allen
25m 55s
Dilbert.

Vincent Woo
25m 56s
Yeah, no one paints a favorable light of corporate life anymore. There is no work that makes the work of an office seem noble. In some ways that's tragic because I don't think that's necessarily true, but on the other hand it reflects reality. I think the majority of of...

Listen to Switched On Pop: https://www.listennotes.com/podcasts/switched-on-pop/why-do-new-christmas-songs-my7-wjKMVZ4/ (5mins in)

Listen to devtools.fm: https://www.listennotes.com/podcasts/devtoolsfm/jori-lallo-linear-KS_Gn2v3hqV/

Transcript:

So one of the big differences between linear and other product products is that it has a local first approach where, uh, it does a lot of things locally in the app before, like communicating with the backend. Uh, why was this such an important baseline feature for you guys to get in? And how, how did you do it? If you can, uh, comment on that. So yeah, optimistic updates. I don't like, that's that, that's the story. first and offline readiness, or someone, they're almost like, like local first, like for sure. But like offline readiness for example is like a byproduct of like how, just how we build linear and we're able to like plug that in afterwards. So the goal is like build something that's really, really snappy and if you see a spinner or like a loader after your every interaction, um, you know, like, it's like if that's your foundation, like it's extremely hard to like change. So we select, build out the foundation from the get go to, uh, do, do updates locally first and then like likes us, communicate them, uh, distribute them like behind the scenes Um, of course some might fail, but like most of them, uh, they don't, uh, often the question is like, okay, like how do you do conflict resolution? Well, the short answer, like simple answer, like, we kind of like don't have to that often. Uh, so In a tool like Linear, most changes are addition on something that's like already happened. We're we're not a, we're not a Notion or Google Docs. You don't edit the same text block. Uh, of course you can edit like the same, uh, issue description at the same time. But reality is that that rarely, rarely ever happens. So hopefully we can like, solve that at some point. But right now the last update, like last update wins. Uh, so how it like technically happens is a little bit like similar to, for example, like how Dropbox builds their desktop, uh, application. Is that when you first, uh, log into linear, you get a larger dump, what we call a bootstrap, um, of kinda like all of your issues and like, like teams and users and that information. Uh, we load that first in memory, then persist in an index db in the background. Um, if you use the product that's the caching, uh, indicator that like pops up one after you're logged in. And after that, uh, we just do delta, delta syncs or like delta packages distributed through web sockets and those get up, uh, applied on the local, uh, copy of the store that you use. Uh, so we, we maintain a copy of the. Connect the whole, like the database, uh, in memory and then persisted in index db. We first update everything into that, uh, show everything in the client, then send those over. If you get something like those, like Delta packages over the wire, they select it applied like the UI react to those. Um, we use a little bit like, like it's a React application, react with type script. We use mobx under the hood, uh, to do all the updating. But, and like the developer experience for this is, uh, It is, uh, it is, it is different from a lot of like applications where you have to, like, we do all the heavy lifting for you. We do like the mutations and queries and uh, like we don't do queries, but like we do mutations. So whenever you like mutate something, we auto generate mutational mutations out of those and send those to the server. The server, uh, which then again, then creates the Delta packages for other clients. But all of that, uh, is abstracted from US engineers at Linear. So we basically only interact with the, with the store. That's the application state and you reach from it, UI updates, you change things to it. Uh, mutations are created in your background. So while it was like it's an engineering geat, uh, Tomas actually has a YouTube video about it. Uh, I can share it after the fact, but, um, It allows us to move faster while putting this upfront investment in place. Uh, it's, it's not like easy system . That's why we have and couple of other engineers working on it, uh, nowadays more, more or less full-time. Full-time as well. Like scaling this, uh, is, um, you know, like we knew like at first, like our customers were relatively small. They don't have that much data, so we can just read every, all of their information into the client. As we grow, like we know that, like, that's not gonna work out anymore. So then we started like chipping off on doing other more unique things, uh, on top of it to help it scale. Um, so that's a, it's an ongoing, ongoing thing as well. But it's kind like at first, you know, we served like the small startups, now we serve companies like cross stage companies of, you know, Brazil, retool like several hundred, uh, engineers. And then Cash App is also using us and they're, I don't remember like how big they are, but it's a sizeable, sizeable engineering organization. So we've been able to like, Scale, linear asset product? Uh, both like on from the product perspective, but also from the, kinda like the technical foundation perspective. And that's, that's in the end, like our goal, like we want linear to be the, the one tool that you use. Like if you're a small startup, like pick up using it, like don't think about it and it should, uh, get you throughout your, your like, scale IPO and beyond . So Really, I really love this. So it's like this really. Complex technical architecture that's required, but the, the thing that you're going for, the thing that you're pushing for is optimistic updates. You know, I want to do an action and feel like it instantly completed. And it's like, it's a difference between a performance first company and a company who thinks about performance after the fact. Because architecting a system or introducing that type of architecture, post shipping a product is a non-trivial endeavor. And, and it would've taken y'all a long time to get to that point. So it's like, uh, I don't know. That's, that's really cool to, to hear about. And yeah, it's, it's hard. Luckily nowadays there are starting to be some technical solutions for this. For example, repliCash, and I'm sure you you know, like bunch of like and, yeah, bunch of others. When we started four years ago, there's nothing Couch db, which, and Coach DB and Fire Base, but they only really, they didn't work for a, uh, kinda like multi-tenant team tool. It, they work really well. If you're like one user who switches between like different applications and your, your state follows you, your settings follow you. But for, for a tool like linear, like nothing existed, so we kinda like have to. Do the investment. And Tomas really went, went to town with it and I'm, I'm glad he did. Uh, I, I don't personally like have the engineering like capabilities like pull that off, but I'm really fortunate to be working with him and yeah, it's now, it's like one of the, like the defining things of, of linear. It's like the speed and the snappiness. We basically don't have, like, there's a couple of, there's a couple of spinner and like loading states in the application, but, uh, you, you, you can use the product and like try to find them yourself. I won't tell where they are. 

Listen to Streaming Audio: https://www.listennotes.com/podcasts/streaming-audio/streaming-analytics-and-real-XS_gvcBjY2p/ (30mins in)

Listen to the Changelog: https://changelog.com/podcast/510

Transcript

So elastic compute makes sense, and scaling down because you have like ephemeral on-demand resource usage, right? Like, all of a sudden, I have to answer a bunch of HTTP requests, and so my server has to do stuff, and then everybody leaves, and my website doesn’t get any hits, and I could scale that down. With databases, if I’ve got a one-gigabyte database, it’s just like, it’s always there. I mean, all that data is there, and I could access any part of it at any time, or I need to… And we don’t know which parts. So I have a hard time with database scaling to zero, unless you’re – I don’t know, just like stomaching the cost… Or tell us how that works with Neon. Are you just stomaching the costs of keeping that online, or are you actually scaling it down?

NIKITA SHAMGUNOV

We’re actually scaling that down. Let me explain how this works, and it may get quite technical. The first thing is what should be the enabling technology of scaling that down? If you’re just kind of thinking, “How would I build serverless Postgres?” and if you ask a person that is not familiar with database internals, they would say something like, “Well, I would put it in the VM maybe, or I would put it in the container, I would put that stuff into Kubernetes… Maybe I can change the size of the containers…” The issue with all that, as you start moving those containers around, it will start breaking connections, because databases like to have a persistent connection to them. And then you will be impacting your cache. Databases like to have a working set in memory, and if you don’t have a working set of memory, you’re paying the performance hit by bringing that data from cold storage to memory.

The third thing that you will find out, that if the database is large enough, it’s really, really hard to move database from host to host, because that involves data transfer, and data transfers are just long and expensive. And now you need to do it live, while the application is running and hitting the system. And so naively, you would arrive with something that you kind of proposed, like just stomach the costs. There is a better approach, though… And the better approach starts with an architectural change of separating of storage and compute.

If you look at how databases, storage works at the high level, it’s what is called a page-based storage; all the data in the database is split into 9-kilobyte pages. And the storage subsystem basically reads and writes those pages from disk, and caches those pages in memory. And then, kind of the upper-level system in the database lays out data on pages.

So now you can separate that storage subsystem, and move that storage subsystem away from Compute into a cloud service. And because that storage subsystem operates is relatively simple from the API standpoint - the API is “read a page, write into a page”, then you can make that part multi-tenant. And so now you start amortizing costs across all your clients. So if you make that multi-tenant, and you make that distributed, and distribute key-value stores - you know, we’ve been building them forever, so it’s not rocket science anymore - then you can make that key-value store very, very efficient, including being cost efficient. And cost efficiency comes from taking some of that data that’s stored there and offloading cold data into S3.

[20:13] Now, then it leaves out compute. And compute is the SQL query processor, and caching. So that, you can put in a VM. We actually started with containers, but we quickly realized that micro VMs such as Firecracker or Cloud-hypervisor is the right answer here. And those micro VMs have very, very nice properties to them. First of all, we can scale them to zero, and preserve the state. And they come back up really, really quickly. And so that allows to us to even preserve caches, if we shut that down.

The second thing that allows us to do is live-changing the amount of CPU and RAM we’re allocating to the VM. That’s where it gets really tricky, because we need to modify Postgres as well, to be able to adjust to suddenly you have more memory, or shrink down to “Oh, all of a sudden, I have less memory now.” And so if you all of a sudden have less memory, you need to release some of the caches, and release this memory into the operating system, and then we change the amount of memory available to the VM. And there’s a lot of cool technology there, with live-changing the amount of CPU, and there’s another one that’s called memory ballooning, that allows you to, at the end of the day, adjust the amount of memory available to Postgres.

And then you can live-migrate VMs from host to host. Obviously, if you put multiple VMs on a host, they all started growing, at some point, you don’t have enough space on the host. Now you do make a decision - which ones do you want to remove from the host? Maybe you have a brand new hosts available for them, with the space… But there is an application running, with a TCP connection, hitting that system> Storage is separate, so you only need to move the compute. And so now you’re not moving terabytes of data with moving Postgres, you’re just moving the compute part, which is really the caches, and caches only. But you need to perform a live migration here. So that’s what we’re doing with this technology that’s called Cloud Hypervisor, that supports live migrations. And the coolest part is, as you’re performing the live migration, you’re not even terminating the TCP connection. So you can have the workload keep hitting the system as you change the size of the VM for the computer up and down, as well as you can change the host for that VM, and the application just keeps running… So yeah, that’s kind of super-exciting technology.

JEROD SANTO

So do you have your own infrastructure that this is running on, or are you on top of a public cloud, or how does that all work?

NIKITA SHAMGUNOV

So we are on top of AWS. We know that we need to be on every public cl...

Listen to Stevey's podcast: https://youtubetranscript.com/?v=Wi8SL-Tot-8&t=1212

Transcript

so let me tell you about service meshes
kind of like the terminology just to get
everybody up to speed because i know
some of you haven't looked at this space
or haven't looked at it recently
you're going to hear two terms control
plane and data plane bandied about a lot
and it's very confusing at first okay
because first of all they are sort of
poorly named and second of all there is
actually a fair amount of overlap
between the two in the in the service
offerings that we have today all right
and in the tech stacks that we have
available so let me walk you through
them all right
so starting at the uh at the service
level so you have a bunch of services
maybe they're on vms maybe they're in
kubernetes maybe they're in nomad or
fargate or whatever right but you've got
services vms or containers and you want
to have them communicate with each other
all right
well having rather than having them all
communicate with each other
which obviously means you're going to
have to build like service discovery
logic into the service itself
so if i have a player service
let's say i have a game server and it
wants to go call the player service and
say is this player real okay if so give
me their give me their information give
me their credentials okay typical
service to service uh you know function
call rpc
all right well you could have the game
server say well i'm going to call the
service registry service to see uh where
the player service lives and then i'll
make a call to the player service right
but now you're building that i'm going
to call the service registry service
which is this other service right that
you would have to build or whatever or
use ncd like grab did or whatever
and then it has to call and get the
address of the player service and then
and then it makes the call and it's like
you've built
routing logic
and discovery logic into your actual
application logic which you do not want
you do not want that okay
so
almost immediately people started moving
to proxies
you have a proxy that's your local proxy
they call it a sidecar proxy in
kubernetes land because it actually runs
in your little cluster as another
service along alongside all of your
other services
and it handles all
network uh ingress and egress for you
so you the idea is that your application
only knows about the sidecar proxy right
so to your application the proxy is the
outside world if if you you know it
knows about the service locations and it
also knows about circuit breakers and
traffic splitting and load balancing and
scaling and everything else that we'll
talk about in a bit
and that proxy becomes the thing that
other people use to talk to your service
as well because your service may be a
cluster right and so people if people
want to send something to the player
service and there's a bunch of instances
of it your proxy is the one to choose
which one maybe maybe it interacts with
an external load balancer or maybe it
does the load balancing itself the proxy
does okay by doing the health checks on
its local service instances yeah
does this model make sense so as soon as
you get this basic model of the of the
sidecar proxy you've got a helper
service that goes along with every
cluster
and it knows about the services in that
cluster and it knows about the outside
world
and your cluster talks to the outside
world through the proxy and the outside
world talks to your cluster through the
proxy okay you can use nginx for that
and that's what dropbox is doing right
but these days people always almost
always use envoy or link or d there are
a couple of other options in addition to
those in nginx but i mean those are the
really popular ones okay
envoy is the the super industrial
strength
does everything swiss army knife amazing
data plane okay by the way those sidecar
proxies i'm going to introduce you now
to the to the second term you hear data
plane the other one being control plane
data planes is just all of your sidecar
proxies in aggregate because if you if
you've got a whole bunch of clusters
right uh or even a whole bunch of
services and you want proxies for each
of them then
that mesh of proxies
that are all talking to each other
to work out the service discovery and
the routing and everything on behalf of
the application services now you've
extracted all of that you know who who's
talking to who what where and how much
and all that you've extracted it into
your
sidecar proxies
that's your data plane
it's because the network data is going
through that and i think it's a terrible
name it should have been called the
network plane or the proxy point proxy
plane would have been an absolute great
name for it right
proxy plane but no they call it data
plane so it's completely confusing
because you'd think the data plane would
be either your application logic or it
would be the data layer behind your
application logic but no
so stupid name really stupid shame on
whoever chose that name really you just
you did a huge disservice to the
industry so if you patent yourself on
the back because you came up with a name
data plane like seriously like punch
yourself in the mouth okay it just it
was a bad name
naming you know naming stuff matters man
you don't want to confuse everybody for
the rest of their lives
whatever but the name is stuck and the
name is the name now and in fact there
are well we've been ahead of ourselves
here but they're even becoming universal
standards now for data plane uh
interfaces
so the data plane i mean like you're
just going to have to learn what data
plane means it means it's the proxy
layer okay the proxies that can uh could
load balance and they can they they
handle the network for you it's software
load balancing they actually in envoy
they actually communicate through a
protocol called a gossip protocol which
is a family protocols where they're sort
of like udp multicast where
everybody just kind of like spits out
the state and consumes the state and it
sort of floods the network
and it's eventually consistent
so that's one thing to know about envoy
is they chose an eventually consistent
model
if you'll recall
i said that etcd and technologies like
it like google's chubby or uh
zookeeper or uh even hashicorp console
they're all they're all key value stores
that are
um transactional highly available and
strongly consistent okay
uh and that actually makes them uh sort
of a pain to operate
uh in practice
all of the ones that i just mentioned
chubby is an interesting one google's
chubby it was probably the first uh mike
burrows i think uh did chubby and if you
haven't heard the name mike burrows
uh you really should know his name
because you know he's easily one of the
the people who had 10 people who've had
the most impact of google right
uh he's you know i don't know he's a de
or whatever
and uh and he he came up with chubby as
far as you know among other things and
chubby is um
chubby is distinguished as
having something like seven nines of
availability it was down for 30 seconds
in 10 years something like that
so um
so yeah and it's because google has a
core competency of operating chubby at
scale
right because it's the it's the central
you know key value service for service
uh discovery and information ...