Streaming Audio: Apache Kafka® & Real-Time Data

Reddit Sentiment Analysis with Apache Kafka-Based Microservices


Listen Later

How do you analyze Reddit sentiment with Apache Kafka® and microservices? Bringing the fresh perspective of someone who is both new to Kafka and the industry, Shufan Liu, nascent Developer Advocate at Confluent, discusses projects he has worked on during his summer internship—a Cluster Linking extension to a conceptual data pipeline project, and a microservice-based Reddit sentiment-analysis project. Shufan demonstrates that it’s possible to quickly get up to speed with the tools in the Kafka ecosystem and to start building something productive early on in your journey.

Shufan's Cluster Linking project extends a demo by Danica Fine (Senior Developer Advocate, Confluent) that uses a Kafka-based data pipeline to address the challenge of automatic houseplant watering. He discusses his contribution to the project and shares details in his blog—Data Enrichment in Existing Data Pipelines Using Confluent Cloud.

The second project Shufan presents is a sentiment analysis system that gathers data from a given subreddit, then assigns the data a sentiment score. He points out that its results would be hard to duplicate manually by simply reading through a subreddit—you really need the assistance of AI. The project consists of four microservices:

  1. A user input service that collects requests in a Kafka topic, which consist of the desired subreddit, along with the dates between which data should be collected
  2. An API polling service that fetches the requests from the user input service, collects the relevant data from the Reddit API, then appends it to a new topic
  3. A sentiment analysis service that analyzes the appended topic from the API polling service using the Python library NLTK; it calculates averages with ksqlDB
  4. A results-displaying service that consumes from a topic with the calculations

Interesting subreddits that Shufan has analyzed for sentiment include gaming forums before and after key releases; crypto and stock trading forums at various meaningful points in time; and sports-related forums both before the season and several games into it. 

EPISODE LINKS

  • Data Enrichment in Existing Data Pipelines Using Confluent Cloud
  • Watch the video version of this podcast
  • Kris Jenkins’ Twitter
  • Streaming Audio Playlist 
  • Join the Confluent Community
  • Learn more with Kafka tutorials, resources, and guides at Confluent Developer
  • Live demo: Intro to Event-Driven Microservices with Confluent
  • Use PODCAST100 to get an additional $100 of free Confluent Cloud usage (details
...more
View all episodesView all episodes
Download on the App Store

Streaming Audio: Apache Kafka® & Real-Time DataBy Confluent, founded by the original creators of Apache Kafka®

  • 4.8
  • 4.8
  • 4.8
  • 4.8
  • 4.8

4.8

44 ratings


More shows like Streaming Audio: Apache Kafka® & Real-Time Data

View all
Planet Money by NPR

Planet Money

30,892 Listeners

The Changelog: Software Development, Open Source by Changelog Media

The Changelog: Software Development, Open Source

285 Listeners

Talk Python To Me by Michael Kennedy

Talk Python To Me

586 Listeners

Software Engineering Daily by Software Engineering Daily

Software Engineering Daily

631 Listeners

AWS Podcast by Amazon Web Services

AWS Podcast

201 Listeners

DataFramed by DataCamp

DataFramed

268 Listeners

Tech Lead Journal by Henry Suryawirawan

Tech Lead Journal

12 Listeners

System Design by Wes and Kevin

System Design

93 Listeners

Postgres FM by Nikolay Samokhvalov and Michael Christofides

Postgres FM

20 Listeners

Kubernetes for Humans by Komodor

Kubernetes for Humans

2 Listeners

Learn System Design by Ben Kitchell

Learn System Design

32 Listeners