The Swyx Mixtape

Full interview on the HOSS talks FOSS podcast: https://www.youtube.com/watch?v=8e3IC-lAgXs

Vitess: https://vitess.io/
Planetscale: https://planetscale.com/

Vitess was created in 2010 to solve the MySQL scalability challenges that the team at YouTube faced. This section briefly summarizes the sequence of events that led to Vitess' creation:

1. YouTube’s MySQL database reached a point when peak traffic would soon exceed the database’s serving capacity. To temporarily alleviate the problem, YouTube created a master database for write traffic and a replica database for read traffic.
2. With demand for cat videos at an all-time high, read-only traffic was still high enough to overload the replica database. So YouTube added more replicas, again providing a temporary solution.
3. Eventually, write traffic became too high for the master database to handle, requiring YouTube to shard data to handle incoming traffic. As an aside, sharding would have also become necessary if the overall size of the database became too large for a single MySQL instance.
4. YouTube’s application layer was modified so that before executing any database operation, the code could identify the right database shard to receive that particular query.

Vitess let YouTube remove that logic from the source code, introducing a proxy between the application and the database to route and manage database interactions. Since then, YouTube has scaled its user base by a factor of more than 50, greatly increasing its capacity to serve pages, process newly uploaded videos, and more. Even more importantly, Vitess is a platform that continues to scale.

Listen to the full episode on SERadio: https://www.se-radio.net/2020/06/episode-413-spencer-kimball-on-cockroachdb/

Previous two parter on Spencer: 

• https://swyx.io/podcast/learn-in-podcast/rss.xml
• https://swyx.io/podcast/learn-in-podcast/rss.xml

Listen to the full interview on SEDaily: https://softwareengineeringdaily.com/2021/06/28/timescale-time-series-databases-with-mike-freedman/

We originally created Timescale, really from our own need. Around that
time, 2014-2015, my co-founder and I, Ajay Kulkarni, who we go back many years, we resynced
up and we started thinking about it was a good time for both of us to think about what the next
challenges are that we want to tackle. It seemed to us that there was this emerging trend of
now, people talk about the digitization, or digital transformation. It feels like somewhat of an
analyst term, but I think, it's really responsive of what's happening, in that if you think about the
large, big IT revolution, it was about changing the back office. What was used to be on paper
was now in computers.

What we saw was somewhat the same thing happened to basically, every industry, from heavy
industry, to shipping, to logistics, to manufacturing, both discrete and continuous and home IoT.
Sometimes this gets blurred under IoT, but we also think about it more broadly as operational
technology, those which are not necessarily bits, but atoms. A big part of that was actually
collecting data of what those systems were doing. It's about sensors and data and whatnot.
When we do Initially looked at this problem, we were thinking about a type of data platform we
would want to build, to make it easy to collect and store and analyze that type of data. I think
that's a way that we're slightly different, or why our – what we ultimately built as our database
ended up being fairly different than a lot of other so-called time series databases. That's
because many of them arose out of IT monitoring, where they were trying to collect metrics from
servers, where we were originally thinking about collecting data more broadly from all these type
of applications and devices around your world.
When we started building it, it was originally focusing mostly on IoT. We quickly ran into this
problem that the existing databases out there and the time series databases out there were not
really designed for our problems. They were often much more limited, because they were
focusing on this narrow infrastructure monitoring problem, where the data maybe wasn't as
important. It was only a very specific type. Let's say, they stored only floats. They didn't have to
have extra metadata that they wanted to enrich their data to better understand what was going
on, like through joins.
After, basically working on this platform for about a year, we somewhat came to the conclusion
that we actually need to build somewhat of our own time series database that was focusing on
this more broad type of problem, and so that's what we do. That's what led the development of
what became Timescale.

JM: Today, what are the most common applications of a time series database?

Like and speak mostly about obviously, TimescaleDB, rather than – as I was
alluding to before, a lot of the other time series databases are much more narrowly focused on
IT monitoring, or observability. We really see our use cases across the field. We certainly see
cases of observability. In fact, we have subsequently built actually a separate product on top of Timescale called Promp scale, that is really used for initially Prometheus metrics, but more
broadly, to make it easier to store observability data with TimescaleDB.
We see still a lot of IoT. We see a lot of logistics. We see financial data and crypto data. We see
event sourcing. We see product and user analytics. We see people collecting data about how
users are using their SaaS platforms. We see gaming analytics, where companies are collecting
information about how people's virtual avatars are actually playing within the games. We see
music analytics. We like to think of the old way, used to find the pop stars, you went down to the
smoky club. Now you collect SoundCloud and Spotify streams, and you use that to identify who
the next breakout artist is going to be.
All of these are example of time series data. It's really what's so exciting to us as is it's such a
broad use case, so horizontal, because basically, it's all about collecting data at the finest
granularity you can.

Tell me about the initial architecture for TimescaleDB. You’re based off of
PostgresSQL. What was the reasoning around that decision?

I think, as you point out, Timescale is actually implemented as an extension on
PostgresSQL. Starting maybe 10 or 15 years ago, PostgresSQL started exposing low-level
hooks throughout its code base. This is not a plugin where you're running a little JavaScript
code. We have function pointers into – we get function hooks into the C. PostgresSQL is written
in C, and so TimescaleDB is, for the most part written in C. We have hooks throughout the code
base at the planner, at sometimes in the storage, at the execution nodes. We are able to insert
ourselves and do Lot of optimizations as part of the same process.

You could ask the question of why not just implement a new database from scratch? Why build
it on top of PostgresSQL? I think this really gets to that, we always viewed ourselves as, and we
hear this from our users and community all the time that we are – they are storing critical data
inside TimescaleDB, and they need it to, A, work and be reliable. They also need it to be – they
have a lot of use case requirements. It’s not this, again, narrow thing where you're collecting
one metrics, and all you're asking to do is figure out the min-max average of a certain metric.
You want to do fancy analysis. You want to do joins. You want to do sub queries. You want to do
correlations. You want to have views. You want the operational maturity of a database. You want
transactions, backup, and restore, and all of the replication and all of the above. Some people
say, it takes maybe 10 years, at least, to build a reliable database. We thought this was a great
way in order to immediately gain that level of reliability, we ourselves are huge fans of
PostgresSQL. It has such a great community. It also has such a large ecosystem.
The idea is that effectively, that entire ecosystem would work from us on day one. That means,
all of the tooling, all of the ORMs, all of your libraries would just work. If we support full SQL, not
SQL-ish. If you know how to use SQL, you could start using – and if your tools speak SQL, if
you're running Tableau, if you're running Power BI if you're running Grafana, if you're running
Superset, those all just start working on day one.
Now, the second part of it is, well, what does that mean to build a time series database on top of
PostgresSQL, which clearly was designed more as a traditional transactional database, OLTP
engine? Sometimes they talk about you think about this architecturally. What I mean by that is
you somewhat think about what your workloads look like and what that would mean from a
software architecture. Maybe I'll give you a very concrete example. Starting maybe 10 or 15
years ago, if you look at traditional databases, you started seeing the growth of what people
commonly now called as log structured merge trees, LSMs.
This is a data structure that goes back to the mid-90s, but I think you first saw Google, Jeff
Dean and Sanjay Ghemawat built something called LevelDB. The whole idea of an LSM tree
was, if you look at a workload that has a lot of updates, so with a lot of e-commerce
applications, with a lot of social networks, you're constantly updating things. Traditional
database, if you think about a disk...

Listen to more on the StackOverflow Podcast: https://stackoverflow.blog/2021/08/06/podcast-364-mark-porter-mongodb-database/

Transcript

markporter 
 

[00:00:00] swyx: This is Mark Porter, the CTO of Mongo DB on his personal journey from relational databases to Mongo DB.   

[00:00:06] Mark Porter: I am a relentless tech geek. I've loved tech my whole life. In fact, my Twitter handle is MarkLovesTech. I have used databases since I was 14 with some really ancient technologies started out on a 4k TRS 80 model one computer. 

We had to program it in assembly language because there wasn't enough memory to use the local basic copy. And I very quickly got into databases and I was talking to someone the other day and he pointed out something I'd never noticed, which is I've oscillated between using databases and building database. 

So I started out at Caltech and NASA using databases for space, data, and chip data. And then I built databases at Oracle versions, 5 6, 7, 8 for about 13 years. And then I used databases at NewsCorp for huge student data systems. And then I built databases at Amazon with Amazon RDS. Then I moved to Grab taxi, which is the Uber of Southeast Asia and use databases to deliver 15 million rides and meals a day, and then came back to Mongo DB. 

And here I am building databases again. I frankly can't get away from this thing.  

[00:01:20] Ben Popper: I love that story. I wonder. Does that mean. You know, at each point you had some sort of frustration or saw some sort of like opportunity for innovation, you know, you kind of would build something, then you'd be the user of it. 

Then you'd realize that like the next sort of turn of the wheel was coming. As you move between those jobs where new paradigms and databases and murders.  

[00:01:38] Mark Porter: Yeah. I mean, it's been really interesting. Half of my career. I've been the Bo and half my career. I've been the target. And I got to tell you that sometimes as a customer, you're not really happy being the target of what has been produced. 

Look, the reality is, is relational databases have been the modus operandi since 1970, when Cod first did his paper. And then Oracle was the first company that released them in 1979. They were actually known as relational technology back then and then changed their name later to Oracle. So the mission criticality of databases has never been in doubt. 

What has changed is the amount of data, the way we process that data. And what's really, really important. And it used to be duplication of data was important and things like that. And while that's still important, what's really important. Now is developer product. Bar none. That is job one for any mission critical software company is developer productivity and innovation  

[00:02:35] Ben Popper: makes a lot of sense. 

It does seem like data has become almost this, uh, overwhelming force for some companies. Ryan. I know if you have experience with this, but I've been getting a lot of pitches and, and talking with folks on the podcast and you know, it's gone from, we're using data to, we have data lakes and there's a data iceberg. 

And, you know, we're only sort of scratching the surface of what we might be able to do with this. Endless flow of unstructured data that we're collecting. And as you mentioned, yeah, a lot of times what they're looking to do is understand it in a way that allows them to enhance productivity or automate certain processes, which right now are very time labor intensive. 

Yeah. Yeah. At my previous job, I worked out on an article about data pipelines and, you know, ETL processes and that yeah. There's a becoming a separation, I think, between your production database and the database you use to gain insights, right? Then the production database has to be fast. But the insight database, it can be a little more flexible in how it produces data, right? 

[00:03:34] Mark Porter: Yeah. So we think about systems of record. We think about systems of insight and yeah. I mean, definitely different people want to do different things with the databases. And so what we do is we think about personas. Are you an analyst? Are you a developer? Are you an AI ML engineer? Are you a PhD data scientist? 

We always try to come at it from the customer and what they want to accomplish. Yeah,  

[00:03:56] Ben Popper: I think that's so interesting because as you said, obviously, databases have always been part of working in the world of software and computers, but increasingly there are these specialties that are very important in which are producing these really interesting results that themselves are devoted to data, as opposed to it being something that, you know, needs to be part of the larger process. 

Um, so mark, I wanted to touch on something, which is that you had a part of your career at AWS, which now, you know, has grown into. Quite a behemoth. Um, yeah. Just wondering if you can talk to us a little bit about what you learned there and maybe how some of that applies to the role you have at, at Mongo DB. 

[00:04:26] Mark Porter: Yeah. So I joined AWS as the general manager of AWS RDS, which at that time was probably the largest fleet of databases in the world. And that fleet grew just tremendously while I was there. It was, it was amazing, you know, just showing. That it's not just databases. It was managed databases that mattered. 

So RDS did not build any of its own databases, RDS vended. By the time I left over a million significantly more than a million Postgres, my SQL Maria DB, Oracle, and SQL server databases. And so the product that we produced was managing those databases and people love it when their database stays up. When the backups and restores work, when you can change parameters when fail over works and all those things. 

However, over time, as much as I loved running those databases, I became frustrated with how they were shackles almost on customer innovation and customer operability. And so we developed this system called Amazon Aurora, which changed out the storage system underneath Postgres in my SQL. Obviously we couldn't do that for the commercial databases and we made those databases so much more resilient, so much more durable, so much more available, but we kept running into the fundamental limit. 

Of a rigid architecture of high fail over times and a single primary architecture, which meant that the blast rate. Of a system going down or play in changing in Oracle database. I mean, it takes down a whole company and I can talk more about availability. In fact, you'll have trouble stopping. When you talk to you about availability, if you get me started  

[00:06:09] Ben Popper: well, I mean, that's, that's the, uh, the big thing about a no SQL is, is availability, right? 

The replicability, the speed of access. Yeah, for folks who don't know, let let's lay out the value prop here. Like what is sort of the difference between the two and why would you prefer one over the other? You know, you mentioned shackles. I love that word, but yeah. You know, what are the limitations that it allows you to avoid when you, when you move to a new SQL and I gue...

Watch on https://www.youtube.com/watch?v=iJIc16aqpO8&t=642s

The Petersens: https://www.youtube.com/c/ThePetersens/about

• Take Me Home: https://www.youtube.com/watch?v=qap9Qm-Q894
• Jolene: https://www.youtube.com/watch?v=viQx4KDivPY
• Landslide: https://www.youtube.com/watch?v=joUwy8lpvP0

Listen to his full interview on the Lex Fridman podcast: https://www.youtube.com/watch?v=1YbcB6b4A2U

Bryan's wiki: https://en.wikipedia.org/wiki/Bryan_Johnson_(entrepreneur)

Listen to the Cloud Giants podcast: https://www.listennotes.com/podcasts/cloud-giants/jeff-lawson-co-founder-and-sFi-ad_etHQ/#

Listen to the NFX podcast: https://www.nfx.com/post/the-insider-story-of-waze/ (28 mins in)

Listen to the OSS Startup Podcast for the full episode.

References:

- Heptio's acquisition in 2018 "It’s not clear how many customers Heptio worked with but they included large, tech-forward businesses like Yahoo Japan."