Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov

In this episode, Kai Waehner (Senior Systems Engineer, Confluent) defines machine learning in depth, describes the architecture of his dream machine learning pipeline, shares about its relevance to Apache Kafka®, Kafka Connect, ksqlDB, and the related ecosystem, and discusses the importance of security and fraud detection. He also covers Kafka use cases, including an example of how Kafka Streams and TensorFlow provide predictive analytics for connected cars.

EPISODE LINKS

• How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka
• Learn about Apache Kafka
• Learn about Kafka Connect
• Learn about ksqlDB, the successor to KSQL
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Streamlining banking technology to help smaller banks and credit unions thrive among financial giants is top of mind for Bobby Calderwood (Founder, Evident Systems), who started out in programming, transitioned to banking, and recently launched Evident Real-Time Payments. 

Payments leverages Confluent Cloud to help banks of all sizes transform to real-time banking services from traditionally batch-oriented, bankers’ hours operational mode. This is achieved through Apache Kafka® and the Kafka Streams and Kafka Connect APIs with Clojure using functional programming paradigms like transducers. 

Bobby also shares about his efforts to help financial services companies build their next-generation platforms on top of streaming events, including interesting use cases, addressing hard problems that come up in payments, and identifying solutions that make event streaming technology easy to use within established banking structures. 

EPISODE LINKS

• Toward a Functional Programming Analogy for Microservices
• Event Modeling: Designing Modern Information Systems
• Finovate Fall/Evident Systems
• The REPL Podcast: 30: Bobby Calderwood on Kafka and Fintech
• Clojure Transducers
• Rich Hickey’s Twitter
• David Nolen's Twitter
• Stuart Halloway’s Twitter
• Chris Redinger’s Twitter
• Tim Ewald’s LinkedIn
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Jay Kreps (original Co-creator of Apache Kafka and CEO, Confluent) introduces ksqlDB, an event streaming database. As the successor to KSQL, ksqlDB seeks to unify the multiple systems involved in stream processing into a single, easy-to-use solution for building event streaming applications.

ksqlDB offers support for running connectors in an embedded mode, in addition to support for both push and pull queries. Push queries allow you to subscribe to changing query results as new events occur, while pull queries allow you to look up a particular value at a single point in time. To use a ride-sharing app as an example, there is both a continuous feed of the current position of the driver (a push query) and the ability to look up current values such as the price of the ride (a pull query). 

Databases are still effective in their own realms, and ksqlDB is not intended as a replacement. Rather, ksqlDB enables you to build event streaming applications with the same ease and familiarity of building traditional applications on a relational database. It simplifies the underlying architecture for these applications so you can build powerful, real-time systems with just a few SQL statements.

EPISODE LINKS

• Learn about ksqlDB on the blog
• Watch the demo to see ksqlDB in action
• Get started with ksqlDB
• Follow ksqlDB on Twitter
• Why Kafka Connect? ft. Robin Moffatt
• Contributing to Open Source with the Kafka Connect MongoDB Sink ft. Hans-Peter Grahsl
• Connecting to Apache Kafka with Neo4j
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

“It’s one thing to get a distributed system up and running. It’s another thing to get a distributed system up and running well.” Ansible keeps your Apache Kafka® deployment, management, and installation consistent, and it enables you to implement best practices that make it easy to get started. Justin Manchester (Platform DevOps Engineer, Confluent) and Viktor Gamov (Developer Advocate, Confluent) discuss the problems that Ansible is trying to solve, enabling collaboration and optimizing all components for top performance.

EPISODE LINKS

• Learn more about Ansible
• Follow Viktor Gamov on Twitter
• Follow Justin Manchester on Twitter
• The Easiest Way to Install Apache Kafka and Confluent Platform – Using Ansible
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Everything is moving to the cloud, which makes it increasingly important to secure your cloud infrastructure and minimize the threat of potential attackers. With a virtual private cloud (VPC)—your own private network in the cloud that you can launch your own instances into—this can be done with VPC Peering, connecting VPCs together to create a path between them to keep your data safe and accessible to you alone. Although typically performed in a single cloud provider, it is possible to do in more than one—think of it as your cloud router

Daniel LaMotte (Site Reliability Engineer, Confluent) walks through the details of cloud networking and VPC peering: what it is, what it does, and how to launch a VPC in the cloud, plus the difference between AWS PrivateLink and AWS Transit Gateway, CIDR, and its accessibility across cloud providers.  

EPISODE LINKS

• VPC Peering in Confluent Cloud
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Migrating from traditional ETL tools to an event streaming platform is a process that Stewart Bryson (CEO and founder, Red Pill Analytics), is no stranger to. In this episode, he dispels misconceptions around what “streaming ETL” means, and explains why event streaming and event-driven architectures compel us to rethink old approaches:

• Not all data is corporate data anymore
• Not all data is relational data anymore
• The cost of storing data is now negligible

Supporting modern, distributed event streaming platforms, and the shift of focus from on-premises to the cloud introduces new use cases that focus primarily on building new systems and rebuilding existing ones. 

From Kafka Connect and stack applications to the importance of tables, events, and logs, Stewart also discusses Gradle and how it’s being used at Red Pill Analytics. 

EPISODE LINKS

• Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation
• Deploying Kafka Streams and KSQL with Gradle – Part 2: Managing KSQL Implementations
• Deploying Kafka Streams and KSQL with Gradle – Part 3: KSQL User-Defined Functions and Kafka Streams 
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Several definitions of a fully managed Apache Kafka® service have floated around, but Ricardo Ferreira (Developer Advocate, Confluent) breaks down what it truly means and why every developer should care. Addressing a handful of questions around Apache Kafka®, Confluent Cloud, hosted solutions, and how they all work, Ricardo describes the benefits of using a fully managed service as a means of simplifying the lives of developers and letting them get back to building—which is why they started out as developers in the first place! 

EPISODE LINKS

• The Rise of Managed Services for Apache Kafka
• Excerpt from The Beginner's Guide to Mathematica, Version 4
• Jay Kreps’ keynote at Kafka Summit SF 2019
• Neha Narkhede’s keynote at Kafka Summit London 2019
• Demos by Ricardo Ferreira
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

In today's spooktacular episode of Streaming Audio, Anna McDonald (Technical Account Manager, Confluent) discusses six of the scariest Apache Kafka® JIRAs. Starting with KAFKA-6431: Lock Contention in Purgatory, Anna breaks down what purgatory is and how it’s not something to fear or avoid. 

Next, she dives into KAFKA-8522: Tombstones Can Survive Forever, where she explains tombstones, compacted topics, null values, and log compaction. Not to mention there’s KAFKA-6880: Zombie Replicas Must Be Fenced, which sounds like the spookiest of them all. 

KAFKA-8233, which focuses on the new TestTopology mummy (wrapper) class, provides one option for setting the topology through your Kafka Screams Streams application. As Anna puts it, "This opens doors for people to build better, more resilient, and more interesting topologies." 

To close out the episode, Anna talks about two more JIRAs: KAFKA-6738, which focuses on the Kafka Connect dead letter queue as a means of handling bad data, and the terrifying KAFKA-5925 on the addition of an executioner API. 

EPISODE LINKS

• KAFKA-6431: Lock Contention in Purgatory
• KAFKA-8522: Tombstones Can Survive Forever
• KAFKA-6880: Zombie Replicas Must Be Fenced
• KAFKA-8233: Helper Classes to Make it Simpler to Write Test Logic with TopologyTestDriver 
• KAFKA-6738: Kafka Connect Handling of Bad Data
• KAFKA-5925: Adding Records Deletion Operation to the New Admin Client API
• Streaming Apps and Poison Pills: Handle the Unexpected with Kafka Streams
• Data Modeling for Apache Kafka – Streams, Topics & More with Dani Traphagen
• Distributed Systems Engineering with Apache Kafka ft. Jason Gustafson
• Kafka Streams Topology Visualizer
• Follow Anna McDonald on Twitter
• Follow Mitch Henderson on Twitter
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

From change data capture (CDC) to business development, connecting Apache Kafka® environments, and customer success stories, Graham Hainbach discusses the possibilities of data integration with Kafka and Attunity using Replicate, Compose, and Enterprise Manager. He also shares real-life examples of how Attunity best leverages Kafka in their systems.

EPISODE LINKS

• Apache Kafka Transaction Data Streaming for Dummies
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Colin McCabe shares about what it’s like being a distributed systems engineer on the Core Kafka team at Confluent, where he has worked previously, and how that led to his interest in Apache Kafka®. As an active member of the Apache open source community, he describes that the community is a place that both welcomes newcomers and fosters different ideas that help make the product the best that it can be for everyone.

Being a distributed systems engineer versus a full stack engineer comes with its own unique challenges. Colin offers some advice for those interested in working with Kafka and what the interview process is like at Confluent. It’s not all about what you know, but rather how you collaborate and contribute to the team, and how you get to the answer. Part of finding the answer is getting involved with Apache projects themselves by engaging with others and helping with bug fixes as much as possible, because it’ll help you gain a better grasp on a technology that is ever-changing.

EPISODE LINKS

• KIP-500: Apache Kafka Without ZooKeeper ft. Colin McCabe and Jason Gustafson
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.