It has been said that in distributed messaging, there are two hard problems: 2) exactly once delivery, 1) guaranteed order of messages and 2) exactly once delivery. Apache Kafka® has offered exactly once processing since version 0.11, which allows properly configured producers and consumers to make the guarantee that each message will be processed exactly one time. 

In this episode, Kafka Streams engineer Guozhang Wang walks through the implementation of transactional messaging in Kafka in some detail, including the idempotent producer API, the transaction coordinator responsible for managing the transaction log and consumer configurations. It’s a complex topic, but he takes us through it carefully and completely.

EPISODE LINKS

• Transactions in Apache Kafka 
• Enabling Exactly Once in Kafka Streams 
• KIP-98: Exactly Once Delivery and Transactional Messaging
• KIP-129: Streams Exactly-Once Semantics

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

• 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating.
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Stanislav Kozlovski joins us to discuss common pitfalls when using Kafka consumers and a new KIP that promises to make consumer restarts much smoother.

EPISODE LINKS

• KIP-345: Static consumer membership 
• KIP-211: Documents the current behavior of offset expiration
• Watch the video version of this podcast
  
  

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

• 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating.
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Microservices are pretty ubiquitous these days. Really “SOA done right,” they reimagine the services pattern in the context of the world we live in today, nearly two decades since the first big service-oriented systems hit production. But what have we learned in this time? There are plenty of war stories. System designers have explored different architectural patterns—REST, events and databases of all types. 

In this podcast, Tim Berglund and Ben Stopford explore the event-driven paradigm and how it relates to the microservice architectures we build today. Ben dives deep into coupling, evolution and challenges of our increasingly data-oriented culture. He also talks about the future, where data are events and events are data, and touches on real-time architectures that retain the decoupling properties needed to be pluggable, and to evolve. Powerful stuff.

EPISODE LINKS

• Designing Event-Driven Systems 
• Building a Microservices Ecosystem with Kafka Streams and KSQL

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

• 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating.
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

After several years of development, librdkafka has finally reached 1.0! It remains API compatible with older versions of the library, so you won’t need to make any changes to your application. There are, however, several important new features like the idempotent producer, sparse broker connections, support for the vaunted KIP-62 and a complete makeover for the C#/.NET client.

EPISODE LINKS

• librdkafka v1.0.0 release notes

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

• 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating.
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Do metrics for detecting clients from old versions actually exist? Or is Gwen making features up? This and more useful advice is coming up on today's episode of Ask Confluent.

EPISODE LINKS

• The Java property that will refresh DNS cache frequently: java.security.Security.setProperty(“networkaddress.cache.ttl” , “60");
• Improvements to DNS lookups in Confluent Platform 5.1.2 (Apache Kafka 2.1.1):
  • KAFKA-7755
  • KAFKA-7890
• More reasons to upgrade to Confluent Platform 5.1.2
• Monitoring clients with old versions:
  • KIP-188 has lots of important new metrics 
• If you are worried about “down-conversion” as discussed in Ask Confluent #5, you want to monitor: MBean: kafka.server:type=BrokerTopicMetrics,name=FetchMessageConversionsPerSec,topic=([-.\w]+)
• KIP-188 also added a metric for temp memory usage (memory used for conversion and compression) that can be useful
• In KIP-272, we’ve added version tag to request metrics, so you can see how many requests per sec you get from each version
• Recommendations for Kafka Summit NYC
• Watch the video version of this podcast

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

• 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating.
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Nick Dearden explains the five stages of streaming maturity. They are not denial, anger, bargaining, depression and acceptance—that’s the Kübler-Ross model, and it’s for bad things. This one is for awesome things, and takes you from the first streaming project you ever build all the way to a state where an entire organization is transformed to think in terms of real-time, event-driven systems. If you have ever found yourself trying to get streaming technology adopted, this episode is for you!

EPISODE LINKS

• Five Stages to Streaming Platform Adoption

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

• 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating.
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Kubernetes provides all the building blocks needed to run stateful workloads, but creating a truly enterprise-grade Apache Kafka® platform that can be used in production is not always intuitive. In this episode, Tim Berglund and Viktor Gamov address some of the challenges and pitfalls of managing Kafka on Kubernetes at scale. They also share lessons learned from the development of the Confluent Operator for Kubernetes, and answer questions like:
-What is Kubernetes?
-What are stateful workloads?
-Why are they hard?
-Will Confluent Operator make it easier?

EPISODE LINKS

• Join the #kubernetes Slack channel
• Kafka on Kubernetes: Does it really have to be “The Hard Way”?

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

• 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating.
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

We all know that feeling of waiting when your ride is running late. Leslie Kurt shares about how you can use KSQL to calculate the difference between the expected arrival time and real-time updates of a bus as it executes its route. Listen as Leslie walks you through fundamental concepts like KTables, Kafka Streams, persistent queries and Confluent MQTT Proxy, as well as other use cases that involve a similar mechanism of capturing Unix timestamps and performing a stream processing operation on these timestamps.

EPISODE LINKS

• About KSQL
• Stream Processing Cookbook
• KSQL Recipe: Calculating Bus Delay Time

For more, you can check out ksqlDB, the successor to KSQL.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

• 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating.
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

When you are dealing with streaming data, it might seem like tables are things that dwell in the far-off land of relational databases, outside of Apache Kafka and your event streaming system. But then the Kafka Streams API gives us the KTable abstraction, which lets us create tabular views of data in Kafka topics.

Apache Kafka 2.1 featured an interesting change to the table API—commonly known to the world as KIP-328—that gives you better control over how updates to tables are emitted into destination topics. What might seem like a tiny piece of minutia gives us an opportunity to explore important parts of the Streams API, and unlocks some key new use cases. Join John Roesler for a clear explanation of the whole thing.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

• 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating.
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Tim Berglund chats with System Engineer Pascal Vantrepote about a KSQL recipe he created based on a real-life customer use case in the financial services industry. They also discuss the advantages of KSQL, such as its expressiveness and ease of deployment in places where you’re not already writing a Java application.

EPISODE LINKS

• About KSQL 
• Stream Processing Cookbook
• KSQL Recipe: Data Routing Joined with a KTable

For more, you can check out ksqlDB, the successor to KSQL.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

• 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating.
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.