In this podcast episode, Ralph and I interview Ben, the CEO and founder of Popsink. Ben shares his background, starting his data journey at Amazon and later becoming the head of data at SIXT, a well-known European car rental company. Ben explains that the motivation behind creating Popsink was frustration with the current data tools, particularly in the realm of serving operations and facing the complexities of serving end users. He wanted to go beyond traditional table-based approaches and create a more optimal solution. Popsink aims to provide a familiar experience for users from the database world, implementing technologies like Flink for transformation, Connect for CDC (change data capture), and Red Panda for event bus purposes.

Overall, this podcast episode gives insights into the origin story of Popsink and its goal of revolutionizing the data industry by offering a comprehensive and user-friendly solution for serving operations and delivering data to end users.

SUP! Hubert’s Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

In this podcast, I’m joined by Ting and Jove at Current 23 at the StartupHub. Ting and Jove are two founders of Timeplus, a streaming database company. Timeplus recently announced their open source version, called Proton, under Apache Linux 2. It provides a docker image for easy setup on laptops. While the SAS version focuses on usability with a user-friendly interface and community-based deployment, the open source version aims to collect feedback from developers and provides a secure primary interface with no web UI. The podcast continues with Jove explaining that the open source version of Timeplus is compact and can be customized based on user preferences, including different Linux versions and binary downloads. Our discussion highlights the comprehensive nature of the streaming database and its various capabilities, which are user-friendly and accessible for both developers and non-technical users.

Speedb (pronounced “speedy bee”) is an embedded key-value storage engine that is a fork of RocksDB. At Current23, I caught up with the founders as they introduced many features on top of RocksDB and managed a large community of RocksDB and Speedb users. They emphasize the project's open-source nature and the community's acceptance of contributions. Their current work focuses on performance improvement, ease of use, and resource management. Streaming is a focus for Speedb, highlighting the growing market for event streaming databases. Speedb is a drop-in replacement for RocksDB that is faster and more efficient than RocksDB due to a new patented LSM tree technology.

In this podcast, Ralph, my co-host and I interview Eric Broda. In this interview with Eric Broda, co-author of the book "Implementing Data Mesh," the focus is on the technical aspects of implementing a data mesh. The interview touches on the book's content, emphasizing its technical and hands-on nature, providing tools, frameworks, and code to help organizations implement a data mesh successfully. Eric's perspective on data mesh is discussed, highlighting potential conflicts with other viewpoints in the field, particularly in defining domains and data products as well as how they are consumed. Despite these differences, the interview emphasizes the common goal of solving practical problems rather than getting caught up in pedantic debates. The role of sponsors and Conways' law in shaping data products is also briefly explored, along with the concept of chargeback as a financial consideration.

Conway’s Law

In simple terms, it means that the way people and teams interact and collaborate will influence the design and organization of the software they create. For example, if different teams don't communicate well, it can lead to a fragmented and poorly integrated software system. Conversely, effective collaboration can result in a more cohesive and well-designed software product.

SUP! Hubert’s Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

In this podcast episode, host I interview Robert Zych, a software engineer at Raft with nearly 20 years of experience in software and data engineering. Robert talks about his engineering background and why he is so passionate about the Apache Pinot project. Robert explains that Apache Pinot is a game changer in real-time analytics, offering scalability, performance, and powerful features like UPSERT and the StarTree index. These capabilities, Robert states, are lacking in other data warehouses he has worked with, making Pinot a compelling choice.

Robert also highlights the importance of data quality when using Pinot for real-time analytics, emphasizing the need for accurate and complete data to build solid metrics. As an expert in the Pinot community, Robert shares the top three critical features he believes are essential for a real-time analytical use case. He also explains how Pinot makes Kafka queryable and emphasizes the importance of good-quality data for building solid metrics. Additionally, Robert discusses the ability to ingest data on the fly and add indexes to support query patterns, which is particularly useful for platform teams like the one he worked with at DoorDash.

SUP! Hubert’s Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

30-Second Summary: Tim Sehn on Dolt and Data ManagementIn this interview, Tim Sehn discusses Dolt, a revolutionary data management system. Dolt challenges traditional data sharing barriers by proposing the concept of "renting data" as easily as renting a computer. Dolt's novel approach involves Prolly trees, enabling lightning-fast data access. With use cases spanning from version control for complex spreadsheets to replicating MySQL, Dolt bridges data and versioning seamlessly. It's gaining traction, with significant funding and a growing user base, and is set to focus on edge case compatibility in the coming months.

3-Minute Summary: Tim Sehn on Dolt: Transforming Data ManagementTim Sehn's interview unveils the groundbreaking paradigm of Dolt, a data management platform that challenges conventional limitations. Drawing an intriguing analogy between renting computers and renting data, Sehn proposes a paradigm shift in data sharing. Addressing the historical difficulty of data sharing, he introduces the idea of branching and merging in data, akin to version control systems.Central to Dolt's architecture is the innovative Nom operating system and the Prolly tree structure, characterized by nodes with content addresses. Dolt initially focused on data sharing and introduced features like import-export mechanisms. Prolly trees, with their exceptional speed, became instrumental in Dolt's evolution.Sehn introduces three primary use cases or modes of Dolt. The "Git for data" mode, comparable to Artifactory, caters to intricate spreadsheets and CSV data, facilitating functions like diffing and merging. Another mode replaces MySQL, featuring version control for databases, and allows SQL interactions with data. The third mode replicates MySQL, synchronizing data between different instances. Dolt also tackles machine learning use cases and potentially serves as a feature store.The interview delves into DoltHub's role, akin to the GitHub of data, and introduces DoltSQL as well. Sehn emphasizes Dolt's adaptability to Data Mesh concepts and hints at its potential compatibility with Postgres in the future.As Dolt gains traction, Sehn envisions focusing on compatibility, especially in edge cases, over the next six months. Notably, Dolt's capacity to trace historical changes in data has implications for government, audits, and data governance.In essence, Tim Sehn's interview showcases Dolt as a dynamic disruptor in data management, pioneering novel ways of sharing, versioning, and evolving data, ultimately transforming the landscape of data handling and accessibility.

Hubert’s Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Our guest today is Gilad Kleinman, CEO of Epsio. Epsio has emerged from the world of academia with over a decade of experience dedicated to one mission: optimizing database queries. With the explosion of data in OLTP (Online Transaction Processing) databases and the increasing complexity of queries, our guest emphasizes the need to keep things simple.But that's not all. They're on the path to revolutionize the database world, embracing the concept of HTAP (Hybrid Transactional/Analytical Processing) and recognizing that queries aren't just ad hoc; they're recurring and essential.They've found a way to bridge the gap between OLTP and analytical queries without reinventing the wheel. Instead of building a new database from scratch (a task compared to "changing a car's engine while it's running"), they're integrating analytical queries seamlessly into existing databases.How do they do it? By plugging into OLTP, leveraging Change Data Capture (CDC), and utilizing foreign tables and federated tables. They create views that seamlessly connect OLTP data to the world of analytics, with streaming materialized views persisting in their streaming engine.It's all about externalized materialized views, a concept that promises to reshape how we approach data optimization and analytics. In today's episode, we'll explore their innovative approach, learn about their process, and discover how their external engine, Epsio, operates as a separate process. We'll also uncover their method of writing results back to the database.

Hubert’s Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Our guests today are the co-founders of WarpStream: Richard Artoul and Ryan Worl. WarpStream is a company that's creating waves in the world of data streaming and storage. Their innovative approach to distributed systems has caught the attention of tech enthusiasts everywhere.

Richie, one of the co-founders, is no stranger to the tech world, with an impressive background that includes stints at M3 Uber and Chronosphere. And his serendipitous meeting with his co-founder, Ryan, at a Percona live conference, laid the foundation for WarpStream.

Ryan, the other half of this dynamic duo, had the spark for WarpStream even before their journey with Datadog. Together, they've set out to revolutionize the way we handle data, and today, they're here to give us a glimpse into their world.

WarpStream, with its Kafka-compatible, entirely new architecture implemented in a Go binary, has been turning heads. They're introducing the concept of "Bring Your Own Cloud," pointing directly to S3 buckets, and eliminating the need for local disks.

Join us as we explore their unique deployment model, where agents become brokers and can be deployed across multiple Availability Zones. These stateless agents maintain a distributed cache, ensuring efficient data handling.

In this episode, we'll also uncover the costs, features currently missing, and the innovative use cases that make WarpStream a game-changer for high-volume data streams.

And of course, we'll touch on the future – including the "Bring Your Own Cloud" strategy, their take on the concept of a "lakehouse," and how WarpStream aims to bridge the gap between batch and streaming processing.

Hubert’s Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Today, we have an exclusive interview with a visionary in the field, Hojjat, the CEO of DeltaStream and one of the original creators of ksqlDB.

DeltaStream is at the forefront of revolutionizing data processing with its serverless stream processing platform. One of the key challenges Hojjat highlights is the management of multiple clusters, which can become a cumbersome task for many organizations. With DeltaStream, the user doesn't have to worry about nodes and servers. It's all about simplifying the process and allowing businesses to focus on what matters most – their data.

We delve into the distinction between stream processing and streaming databases, uncovering the importance of stateful processing and materialization, drawing parallels with the traditional Postgres model.

We also get a sneak peek into FlinkSQL, offering a comprehensive view of DeltaStream's capabilities in the world of stream processing and data management.

Hubert’s Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

In this episode, we talk to Seth Wiseman the Director of Field Engineering at Materialize and Apache Flink Committer about streaming databases. He is not a fan of the term streaming database - "Streaming is just an implementation of a database."

Enter Materialize. The name itself carries the promise of a new era. The market is growing hungry to understand the potential of materialized views. While the perception is that streaming might be complex, it's worth noting that companies like Uber have invested significantly in technologies like Apache Flink, boasting a team of 30 engineers solely dedicated to this endeavor. The market demand for such solutions has never been more apparent, but curiously, the "streaming engineer" title remains relatively niche.

As we delve deeper, we uncover the internal distinctions between Flink and Materialize, tracing the evolution of streaming databases. Flink plays the role of the compute, while Materialize aims to build out the database itself. The journey from Spark to Flink reveals unique internals, showcasing the distinction between pipeline architectures and the streaming paradigm. Seth provides an exciting perspective on streaming and streaming databases in general and how Materialize differs from Flink and Spark.

Hubert’s Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. Expense it brah.