Every company has a large number of routine data workflows. These data workflows involve spreadsheets, CSV files, and tedious manual work to be done by a knowledge worker.  For example, data might need to be taken from Salesforce, filtered for new customers, and piped into Mailchimp. Or perhaps you need to sort all your customers

At Airbnb, infrastructure management is standardized across the organization. Platform engineering teams build tools that allow the other teams throughout the organization to work more effectively. A platform engineering team handles problems such as continuous integration, observability, and service discovery. Other teams throughout a company use the tools that a platform engineering team builds. For

A gig economy application generates lots of notifications.  There is SMS, mobile phone updates, emails, and native application updates. If you order a ride from Uber, you might receive a text message and a push notification at the same time. If an app overloads the user with notifications, the user might end up annoyed and

In a modern data platform, distributed streaming systems are used to read data coming off of an application in real-time. There are a wide variety of streaming systems, including Kafka Streams, Apache Samza, Apache Flink, Spark Streaming, and more.  When Eric Anderson joined the show back in 2016, he was working at Google on Google

Distributed stream processing frameworks are used to rapidly ingest and aggregate large volumes of incoming data. These frameworks often require the application developer to write imperative logic describing how that data should be processed.  For example, a high volume of clickstream data that is getting buffered to Kafka needs to have a stream processing system

A cloud provider gives developers access to virtualized server infrastructure. When a developer rents this infrastructure via an API call, a virtual server is instantiated on physical machines. That virtual server needs to be made addressable through the allocation of an IP address to make it reachable from the open Internet. When the virtual server

A data pipeline is a series of steps that takes large data sets and creates usable results from them. At the beginning of a data pipeline, a data set might be pulled from a database, a distributed file system, or a Kafka topic. Throughout a data pipeline, different data sets are joined, filtered, and statistically

A data warehouse provides low latency access to large volumes of data.  A data warehouse is a crucial piece of infrastructure for a large company, because it can be used to answer complex questions involving a large number of data points. But a data warehouse usually cannot hold all of a company’s data at any

Machine learning applications are widely deployed across the software industry.  Most of these applications used supervised learning, a process in which labeled data sets are used to find correlations between the labels and the trends in that underlying data. But supervised learning is only one application of machine learning. Another broad set of machine learning

Distributed stream processing systems are used to read large volumes of data and perform operations across those data streams.  These stream processing systems often build off of the MapReduce algorithm for collecting and aggregating large volumes of data, but instead of processing a calculation over a single large batch of data, they process data on