The announcement that Confluent will now support seamless materialization of Apache Kafka topics as Iceberg Tables (aka Tableflow) has gotten the streaming world and lakehouse worlds rubbing their chins 🤔 and cleaning their monocles 🧐.

Jack Vanlightly says in his post:

There are three principal paradigm shifts going on:

1. The rise of object storage as the…

The data mesh concept is a decentralized approach to data architecture. The core pillar of this concept is reassigning data ownership to the engineering team that produced it - or the origin domain. This shift in ownership is dependent on the other three pillars that make up the data mesh concept. Here are all the pillars of a more decentralized approac…

RAG stands for Retrieval-Augmented Generation. It’s a technique that enhances the responses returned by a large language model (LLM) to be more accurate. LLMs alone can answer many questions provided to them. But to respond with accurate answers and cite sources, the LLM needs help to do some research. In this post, I’ll show how to do this in real-time.

As part of the massive shift into AI, vector databases have been increasing in popularity. Also known as vectorized databases, they play a crucial role in the context of AI, so it’s important to understand how they work. To do so, we’ll need first to understand what vectors are.

Vectors are arrays of numbers that represent unstructured d…

Gunnar’s 1 billion row challenge has gone viral, so I also wanted to take on the challenge. This is technically a Java challenge. Some of the benchmarks for the Java implementations tested so far are really good, but I wanted to use something off the shelf that supports SQL, can be secured, and has SaaS available. Also, since the goal is to aggregate a …

In this podcast Ralph and I interview a former colleague of mine, Kai, who has extensive experience in the data streaming and real-time events space. Kai highlights the top five trends for data streaming with Kafka and Flink, including data sharing, data contracts for governance, serverless stream processing, multi-cloud adoption, and the use of generative AI in real-time contexts. We discuss the role of generative AI in providing accurate answers and the importance of real-time data integration for contextual recommendations, using the example of travel and flight cancellations. We also delve into the role of Flink as a stream processor in ensuring the accuracy and freshness of data for semantic searches and generative AI applications.

We also delve into the idea of streaming databases and whether the market is ready to embrace them. We discuss the need for data contracts and data governance to understand the flow of data through systems, as well as the responsibility of the data engineering team in creating embeddings. We also discuss integrating large language models with other applications using technologies like Kafka and provide examples of how generative AI can be integrated into existing business processes. The interview touches on the concept of a "lake house" and the separation of compute and storage for real-time analytics. The guest also highlights Confluent's approach to building Kafka in a cloud-native way and their focus on the streaming side, while emphasizing the need for accessible stream processing solutions for ordinary database users.

Incremental View Maintenance (IVM) provides a method for keeping materialized views current by calculating and applying only the incremental changes, as opposed to the complete recomputation of contents performed by the REFRESH MATERIALIZED VIEW command.

A materialized view is a database object that stores the result set of…

Two methodologies stand out when building an analytical data model: One Big Table (OBT) and Star Schema. Let’s quickly go over them.

"One Big Table" (OBT) is a database design approach in which all the data is stored in a large table. There are no relations between tables in this schema, and all the information is contained within a single st…

Do you watch a lot of physics YouTube videos? After watching so many of them myself, I stumbled upon a fascinating connection between stream processing in the world of data and the theory of relativity. As events journey through the concept of time, we can draw intriguing parallels between these seemingly unrelated concepts.

Just as the theory of relativ…

The book I recently, “Quit” by , has me thinking about how to apply quitting to how we sell software.

I spent years as a pre-sales engineer in one of the highest performing sales regions of a previous company. But before this, I knew nothing of bing a sales engineer. I was a heads-down, burnt-out data/platform engineer. When I was approached to do pre-sales, I took it knowing I needed a change. When I look back, I really wish I had a quick introduction to the technical selling methodologies. The “quitting” ideas that Annie Duke talked about in her book actually are the essence of the purpose of these methodologies that no experienced salesperson would actually tell you.

Annie talks about how humans psychologically are biased to “gut things out” and to do whatever it takes to win. Especially in sales, you would think that the LFG attitude was required to win. But when you start putting structure around the approach to selling, you’ll find out quickly that the LFG attitude is a weapon used sparingly for moments where a win is in sight. Before that, you need to find ways to quit.

I once said to a CEO, “We should always be finding ways to disqualify an opportunity.” He was confused by this statement for a second and then realized why it was true. He realized that we shouldn’t be wasting time on prospects that will not close in 3 months.

Having a plan to quit is exactly what Duke conveys in her book. Duke talks about the consequences of not knowing when to quit. These consequences are apparent in boxing or mountain climbing, it’s important to know when to turn back. Your life depends on it. These are some of the examples Duke uses to explain her approach to quitting that resonates with the reader. It’s because they are life threatening decisions.

Sales aren’t life threatening. But the approach is the same. How do we know if an opportunity will close in 3 months? We need a sales methodology that can quickly determine when to “quit” on an opportunity so that we can move on to the next one. It allows the business to have predictable revenue. These exist today like the very popular BANT and MEDDICC qualification questions.

BANT is the methodology that needs to be qualified before engaging with a pre-sales engineer. Account reps should know what the budget is, if the person they are speaking to has the authority to make a purchase if the prospect has a true need for your product, and lastly, if there is a timeline in which the prospect needs your product. Without any one of these, the deal will have a lower probability of closing so why waste time? If one of these qualifiers is not fulfilled then the deal should not go to the next stage. But if it does, then someone (either the pre-sales engineer or the account rep) is going to allocate some time to find the answers to these qualifiers which will extend the deal cycle, causing revenue to get pushed out to a later quarter, and the business to be less predictable.

MEDDICC is a sales methodology that really is used by the account executive. But it is just as important for you as the pre-sales engineer to know well. This will allow you to build strategic plans for each deal.

There are variants to this acronym like MEDDPICC where P could stand for Process or Paper Processes. This methodology brings structure and predictability to the selling processes. Account executives follow this method to qualify to the next steps of the sales cycle.

As you might have guessed, quitting in sales in this post means quitting a deal if it’s not qualified no matter how much the deal is worth. In sales, you want to make sure that you’re putting most of your effort on the deals that are highly qualified because they have the highest probability to close.

Both BANT and MEDDICC are qualifying methodologies for the account executive. In my time as a sales engineer, we derived a pre-sales version of this called CLUSSTTERS.