April 20, 2020

Building Real Time Applications On Streaming Data With Eventador

50 minutes

Summary

Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor. Eventador is a managed platform designed to let you focus on using the data that you collect, without worrying about how to make it reliable. In this episode Eventador Founder and CEO Kenny Gorman describes how the platform is architected, the challenges inherent to managing reliable streams of data, the simplicity offered by a SQL interface, and the interesting projects that his customers have built on top of it. This was an interesting inside look at building a business on top of open source stream processing frameworks and how to reduce the burden on end users.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, a 40Gbit public network, fast object storage, and a brand new managed Kubernetes platform, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. And for your machine learning workloads, they’ve got dedicated CPU and GPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!

Your host is Tobias Macey and today I’m interviewing Kenny Gorman about the Eventador streaming SQL platform

Interview

Introduction

How did you get involved in the area of data management?

Can you start by describing what the Eventador platform is and the story

behind it?

How has your experience at ObjectRocket influenced your approach to streaming SQL?

How do the capabilities and developer experience of Eventador compare to other streaming SQL engines such as ksqlDB, Pulsar SQL, or Materialize?

What are the main use cases that you are seeing people use for streaming SQL?

How does it fit into an application architecture?

What are some of the design changes in the different layers that are necessary to take advantage of the real time capabilities?

Can you describe how the Eventador platform is architected?

How has the system design evolved since you first began working on it?

How has the overall landscape of streaming systems changed since you first began working on Eventador?

If you were to start over today what would you do differently?

What are some of the most interesting and challenging operational aspects of running your platform?

What are some of the ways that you have modified or augmented the SQL dialect that you support?

What is the tipping point for when SQL is insufficient for a given task and a user might want to leverage Flink?

What is the workflow for developing and deploying different SQL jobs?

How do you handle versioning of the queries and integration with the software development lifecycle?

What are some data modeling considerations that users should be aware of?

What are some of the sharp edges or design pitfalls that users should be aware of?

What are some of the most interesting, innovative, or unexpected ways that you have seen your customers use your platform?

What are some of the most interesting, unexpected, or challenging lessons that you have learned in the process of building and scaling Eventador?

What do you have planned for the future of the platform?

Contact Info

Blog

@kennygorman on Twitter

kgorman on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.

Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.

If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.

To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat

Links

Eventador

Oracle DB

Paypal

EBay

Semaphore

MongoDB

ObjectRocket

RackSpace

RethinkDB

Apache Kafka

Pulsar

PostgreSQL Write-Ahead Log (WAL)

ksqlDB

Podcast Episode

Pulsar SQL

Materialize

Podcast Episode

PipelineDB

Podcast Episode

Apache Flink

Podcast Episode

Timely Dataflow

FinTech == Financial Technology

Anomaly Detection

Network Security

Materialized View

Kubernetes

Confluent Schema Registry

Podcast Episode

ANSI SQL

Apache Calcite

PostgreSQL

User Defined Functions

Change Data Capture

Podcast Episode

AWS Kinesis

Uber AthenaX

Netflix Keystone

Ververica

Rockset

Podcast Episode

Backpressure

Keen.io

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Support Data Engineering Podcast

...more

View all episodes

By Tobias Macey

4.6

135135 ratings