It’s Halloween again, which means Anna McDonald (Staff Technical Account Manager, Confluent) is back for another spooktacular episode of Streaming Audio.

In this episode, Anna shares six of the most spine-chilling, hair-raising  Apache Kafka® JIRAs from the past year. Her job is to help hunt down problems like these and dig up skeletons like: 

• Early death causes epoch time travel
• Attack of the clones
• Missing snapshot file leads to madness
• Shrink inWriteLock time to avoid maiming cluster performance
• Older groups are forced to flatline
• Ghost segment haunts for eternity 

If JIRAs are undead monsters, Anna is practically a zombie slayer. Get a haunting taste of the horrors that she's battled with as she shares about each of these Kafka updates. Keep calm and scream on in today’s special episode of Streaming Audio!

EPISODE LINKS

• Kafka: A Modern Distributed System
• From Eager to Smarter in Apache Kafka Consumer Rebalances by Sophie Blee-Goldman
• The Magical Rebalance Protocol of Apache Kafka (Strange Loop)
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

It’s the first work-from-home episode of Ask Confluent, where Gwen Shapira (Core Kafka Engineering Leader, Confluent) virtually sits down with Apache Kafka® expert Anna McDonald (Staff Technical Account Manager, Confluent) to answer questions from Twitter. 

Find out Anna’s favorite Kafka Improvement Proposal (KIP), which  will start to use racially neutral terms in the Kafka community and in our code base, as well as answers to the following questions: 

• If you could pick any one KIP from the backlog that hasn't yet been implemented and have it immediately available, which one would you pick?
• Are we able to arrive at any formula for identifying the consumer/producer throughput rate in Kafka with the given hardware specifications (CPU, RAM, network, and disk)? 
• Does incremental cooperative rebalancing also work for general Kafka consumers in addition to Kafka Connect rebalancing?

They also answer how to determine throughput and achieve your desired SLA by using partitions. 

EPISODE LINKS

• Watch Ask Confluent #18: The Toughest Questions ft. Anna McDonald
• From Eager to Smarter in Apache Kafka Consumer Rebalances
• Streaming Heterogeneous Databases with Kafka Connect – The Easy Way
• Keynote: Tim Berglund, Confluent | Closing Keynote Presentation | Kafka Summit 2020
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

Wouldn’t it be awesome if there was a language as elegant as Spring Boot is as a framework? In this episode of Streaming Audio, Tim Berglund talks with Josh Long, Spring developer advocate at VMware about Kotlin, about the productivity-focused language from our friends at JetBrains, and how it works with Spring Boot to make the experience leaner, cleaner, and easy to use.

Josh shares how the Spring and Kotlin teams have worked hard to make sure that Kotlin and Spring Boot are a first-class experience for all developers trying to get to production faster and safer. 

They also talk about the issues that arise when wrapping one set of APIs with another, as often arises in the Spring Framework: when APIs should leak, when they should not, and how not to try to be a better Kafka Streams when the original is working well enough. 

EPISODE LINKS

• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

Neil Buesing, an Apache Kafka® community stalwart at Object Partners, spends his days building things out of Kafka and helping others do the same. Today, he discusses the concept of a CoE (center of excellence), and how a CoE is integral to attain and sustain world-class performance, business value, and success in a business. Neil talks us through how to make a CoE successful, the importance of event streaming, how to better understand streaming technologies, and how to best utilize CoE for your needs. This includes evangelizing Kafka, building a Proof of Value (PoV) with team members, defining deliverables as part of that CoE, and understanding how to implement Kafka into your organization. 

EPISODE LINKS

• EoS in Kafka: Listen up, I will only say this once! by Jason Gustafson 
• The Magical Rebalance Protocol of Apache Kafka by Gwen Shapira 
• Chair-throwing meme that was discussed at end of episode 
• Apache Kafka and Confluent Platform Reference Architecture
• Benchmark Your Dedicated Apache Kafka Cluster on Confluent Cloud
• Optimizing Your Apache Kafka Deployment
• Cluster sizing
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

Ever wonder what it's like to intern at a place like Confluent? How about working with Kafka Streams and creating your own KIP? Well, that's exactly what we discuss on today's episode with Leah Thomas. Leah Thomas, who first interned as a recruiter for Confluent, quickly realized that she was enamored with the problem solving the engineering team was doing, especially with Kafka Streams. 

The next time she joined Confluent's intern program, she worked on the Streams team and helped bring KIP-450 to life. With KIP-450, Leah started learning Apache Kafka® from the inside out and how to better address the user experience. She discusses her experience with getting a KIP approved with the Apache Software Foundation and how she dove into solving the problem of hopping windows with sliding windows instead.

EPISODE LINKS

• Range: How Generalists Triumph in a Specialized World
• Confluent Careers
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

The feature-rich release of Confluent Platform 6.0, based on Apache Kafka® 2.6, introduces Tiered Storage, Self-Balancing Clusters, ksqlDB 0.10, Admin REST APIs, and Cluster Linking in preview. These features enhance the platform with greater elasticity, improved cost effectiveness, infinite data retention, and global availability so that you can simplify management operations, reduce the cost of adopting Kafka, and focus on building event streaming applications.

EPISODE LINKS

• Confluent Platform 6.0 Release Notes
• Introducing Confluent Platform 6.0
• Download Confluent Platform 6.0
• Watch the video version of this podcast
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

Bobby Calderwood (Founder, Evident Systems) discusses event streaming, event modeling, and event-driven architecture. He describes the emerging visual language and process, how to effectively understand and teach what events are, and some of Bobby's own use cases in the field with oNote, Evident System’s new SaaS platform for event modeling. Finally, Bobby emphasizes the power of empowering and informing the community on how best to integrate event streaming with the outside world.

EPISODE LINKS

• Building Information Systems Using Event Modeling 
• Real-Time Payments with Clojure and Apache Kafka ft. Bobby Calderwood
• Event modeling leaders Adam Dymitruk and Greg Young
• Good Enough Software is by Definition Good Enough written by Greg Young
• oNote
• Event modeling
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

Did you know that a team of 900 developers at Wix is using Apache Kafka® to maintain 1,500 microservices? Tim Berglund sits down with Natan Silnitsky (Backend Infrastructure Engineer, Wix) to talk all about how Wix benefits from using an event streaming platform. 

Wix (the website that’s made for building websites) is designing a platform that gives people the freedom to create, manage, and develop their web presence exactly the way they want as they look to move from synchronous to asynchronous messaging. 

In this episode, Natan and Tim talk through some of the vital lessons learned at Wix through their use of Kafka, including common infrastructure, at-least-once processing, message queuing, and monitoring. Finally, Natan gives Tim a brief overview of the open source project Greyhound and how it's being used at Wix. 

EPISODE LINKS

• github.com/wix/greyhound
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

This year, Confluent turns six! In honor of this milestone, we are taking a very special moment to celebrate with Gwen Shapira by highlighting the top six things everyone should know about Apache Kafka®:

1. Clients have metrics
2. Bug fix releases/Kafka Improvement Proposals (KIPs)
3. Idempotent producers and how they work
4. Kafka Connect is part of Kafka and Single Message Transforms (SMTs) are worth not missing out on
5. Cooperative rebalancing 
6. Generating sequence numbers and how Kafka changes the way you think

Listen as Tim and Gwen talk through the importance of Kafka Connect, cooperative rebalancing protocols, and the promise (and warning) that your data architecture will never be the same. As Gwen puts it, “Kafka gives you the options, but it's up to you how you use it.”

EPISODE LINKS

• KIP-415: Incremental Cooperative Rebalancing in Kafka Connect
• Why Kafka Connect? ft. Robin Moffatt 
• Confluent Hub 
• Creativity Inc
• Fifth Discipline 
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

With the explosion of real-time data, Apache Kafka and event stream processing (ESP) have grown in proliferation, with event streaming technology becoming the de facto technology transforming businesses across numerous verticals. Gwen Shapira (Engineering Leader, Confluent), Ben Stopford (Senior Director, OCTO, Confluent), and Michael Noll (Principal Technologist, Confluent) meet up to talk all about their last five years at Confluent and the changes they’ve seen in event streaming. They discuss what they were doing with Apache Kafka® before they arrived at Confluent, challenges in event streaming challenges that have arisen, and their favorite use cases. They then talk through what they think the Kafka community is undervaluing and where they think event streaming will be in the next five years. 

EPISODE LINKS

• Tim’s Budapest Drone Footage 
• Rolling Kafka Upgrades and Confluent Cloud ft. Gwen Shapira
• Distributed Systems Engineering with Apache Kafka ft. Gwen Shapira
• Improving Fairness Through Connection Throttling in the Cloud with KIP-402 ft. Gwen Shapira
• Ask Confluent
• Apache Kafka Fundamentals: The Concept of Streams and Tables ft. Michael Noll
• Ben Stopford on Microservices and Event Streaming
• The Portable Wonder Synthesizer 
• Children's Hospital of Atlanta: Helping Healthcare with Apache Kafka and KSQL ft. Ramesh Sringeri
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)