Kafka is a distributed log for producers and consumers to publish messages to each other. We’ve done many shows about Kafka as a key building block for distributed systems, but we often leave out the discussion of the complexities of setting up Kafka and monitoring it. Kafka deployments can be a complex piece of software to manage.

Tom Crayford is an engineer at Heroku, where he helped engineer the recent Heroku Kafka product, which is a managed version of Apache Kafka. Our conversation explored the use cases of Kafka and how to build Kafka as a cloud service at scale. For more information about Heroku Kafka, check out an upcoming webinar.

Full disclosure: Heroku is a sponsor of Software Engineering Daily. That said, this is a topic I am genuinely interested in–it is often difficult to get cloud providers to talk in detail about how they are architecting their services.

The post Managed Kafka with Tom Crayford appeared first on Software Engineering Daily.