Abstract

Data engineers, data analysts, and big data developers are looking to evolve their analytics from batch to real-time so their companies can learn about what their customers, applications, and products are doing right now and react promptly.

This whitepaper discusses the evolution of analytics from batch to real-time.

It describes how services such as Amazon Kinesis Streams, Amazon Kinesis Firehose, and Amazon Kinesis Analytics can be used to implement real- time applications, and provides common design patterns using these services.

Introduction

Businesses today receive data at massive scale and speed due to the explosive growth of data sources that continuously generate streams of data.

Whether it is log data from application servers, clickstream data from websites and mobile apps, or telemetry data from Internet of Things (IoT) devices, it all contains information that can help you learn about what your customers, applications, and products are doing right now.

Having the ability to process and analyze this data in real-time is essential to do things such as continuously monitor your applications to ensure high service uptime and personalize promotional offers and product recommendations.

Real-time processing can also make other common use cases, such as website analytics and machine learning, more accurate and actionable by making data available to these applications in seconds or minutes instead of hours or days.

Real-time Application Scenarios

There are two types of use case scenarios for streaming data applications:

Evolving from Batch to Streaming Analytics You can perform real-time analytics on data that has been traditionally analyzed using batch processing in data warehouses or using Hadoop frameworks.

The most common use cases in this category include data lakes, data science, and machine learning.

You can use streaming data solutions to continuously load real-time data into your data lakes.

You can also update machine learning models more frequently as new data becomes available, ensuring accuracy and reliability of the outputs.

For example, Zillow uses Amazon Kinesis Streams to collect public record data and MLS listings, and then provide home buyers and sellers with the most up-to-date home value estimates in near real-time.

Zillow also sends the same data to its Amazon Simple Storage Service (S3) data lake using Kinesis Streams so that all the applications work with the most recent information.

Building Real-Time Applications You can use streaming data services for real-time applications such as application monitoring, fraud detection, and live leaderboards.

These use cases require millisecond end-to-end latencies—from ingestion, to processing, all the way to emitting the results to target data stores and other systems.