Defining Kafka

Apache Kafka is an open-source stream-processing software platform which aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Anna McDonald, one of the Apache Kafka experts we speak with in this episode, describes it like this, “The easiest way to describe it is a durable log. And that makes it different from most all other messaging systems, where a message goes in, you have to broadcast it out. And as soon as it's consumed, it's gone. And Apache Kafka, it lives until your retention period, which is fantastic. ” Ricardo Ferreria, another expert on Apache Kafka, adds to this definition, “Kafka is not actually a messaging technology, it’s more of a data streaming technology.”

How Kafka is different from other messaging brokers

• Pull-based  — Kafka data consumers  fetch data when, how, and how fast they need it, without affecting performance. In push-based messaging technologies, the broker pushes messages to consumers, creating potential bottlenecks and scalability issues.

Treat Kafka as your “single source of truth”

Both of our experts today talk about treating Kafka as your “single source of truth,” your system of record. Ricardo explains further, “You can have multiple different consumers interested in the same data set, but each one of them are using the data set differently. So when you start using Kafka this way, you start building architectures that are not only super resilient and scalable, but also it is a very good replacement for very expensive end databases.”

Try Apache Kafka on Platfporm.sh today to get your “single source of truth” 

Platform.sh on social media

Watch, listen, and subscribe to the Platform.sh Deploy Friday podcast:

Platform.sh is a robust, reliable hosting platform that gives development teams the tools to build and scale applications efficiently. Whether you run one or one thousand websites, you can focus on creating features and functionality with your favorite tech stack and leave managing infrastructure and processes to us.