Streaming Audio: Apache Kafka® & Real-Time Data

Optimizing Apache JVMs for Apache Kafka


Listen Later

Java Virtual Machines (JVMs) impact Apache Kafka® performance in production. How can you optimize your event-streaming architectures so they process more Kafka messages using the same number of JVMs? Gil Tene (CTO and Co-Founder, Azul) delves into JVM internals and how developers and architects can use Java and optimized JVMs to make real-time data pipelines more performant and more cost effective, with use cases.

Gil has deep roots in Java optimization, having started out building large data centers for parallel processing, where the goal was to get a finite set of hardware to run the largest possible number of JVMs. As the industry evolved, Gil switched his primary focus to software, and throughout the years, has gained particular expertise in garbage collection (the C4 collector) and JIT compilation. The OpenJDK distribution Gil's company Azul releases, Zulu, is widely used throughout the Java world, although Azul's Prime build version can run Kafka up to forty-percent faster than the open version—on identical hardware. 

Gil relates that improvements in JVMs aren't yielded with a single stroke or in one day, but are rather the result of many smaller incremental optimizations over time, i.e. "half-percent" improvements that accumulate. Improving a JVM starts with a good engineering team, one that has thought significantly about how to make JVMs better. The team must continuously monitor metrics, and Gil mentions that his team tests optimizations against 400-500 different workloads (one of his favorite things to get into the lab is a new customer's workload). The quality of a JVM can be measured on response times, the consistency of these response times including outliers, as well as the level and number of machines that are needed to run it. A balance between performance and cost efficiency is usually a sweet spot for customers.

Throughout the podcast, Gil goes into depth on optimization in theory and practice, as well as Azul's use of JIT compilers, as they play a key role in improving JVMs. There are always tradeoffs when using them: You want a JIT compiler to strike a balance between the work expended optimizing and the benefits that come from that work. Gil also mentions a new innovation Azul has been working on that moves JIT compilation to the cloud, where it can be applied to numerous JVMs simultaneously.

EPISODE LINKS

  • A Guide on Increasing Kafka Event Streaming Performance
  • Better Kafka Performance Without Changing Any Code
  • Watch the video version of this podcast
  • Kris Jenkins’ Twitter
  • Streaming Audio Playlist 
  • Join the Confluent Community
  • Learn more with Kafka tutorials, resources, and guides at Confluent Developer
  • Live demo: Intro to Event-Driven Microservices with Confluent
  • Use PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)   
...more
View all episodesView all episodes
Download on the App Store

Streaming Audio: Apache Kafka® & Real-Time DataBy Confluent, founded by the original creators of Apache Kafka®

  • 4.8
  • 4.8
  • 4.8
  • 4.8
  • 4.8

4.8

44 ratings


More shows like Streaming Audio: Apache Kafka® & Real-Time Data

View all
Planet Money by NPR

Planet Money

30,892 Listeners

The Changelog: Software Development, Open Source by Changelog Media

The Changelog: Software Development, Open Source

285 Listeners

Talk Python To Me by Michael Kennedy

Talk Python To Me

586 Listeners

Software Engineering Daily by Software Engineering Daily

Software Engineering Daily

631 Listeners

AWS Podcast by Amazon Web Services

AWS Podcast

201 Listeners

DataFramed by DataCamp

DataFramed

268 Listeners

Tech Lead Journal by Henry Suryawirawan

Tech Lead Journal

12 Listeners

System Design by Wes and Kevin

System Design

93 Listeners

Postgres FM by Nikolay Samokhvalov and Michael Christofides

Postgres FM

20 Listeners

Kubernetes for Humans by Komodor

Kubernetes for Humans

2 Listeners

Learn System Design by Ben Kitchell

Learn System Design

32 Listeners