
Sign up to save your podcasts
Or


Supabase just acquired OrioleDB, a storage engine for PostgreSQL.
Oriole gets creative with MVCC! It uses an UNDO log rather than keeping multiple versions of an entire data row (tuple). This means when you update data, Oriole tracks the changes needed to "undo" the update if necessary. Think of this like the "undo" function in a text editor. Instead of keeping a full copy of the old text, it just remembers what changed. This can be much smaller. This also saves space by eliminating the need for a garbage collection process.
It also has a bunch of additional performance boosters like data compression, easy integration with data lakes, and index-organized tables.
Show notes:
Chris Gwilliams:
Nicolay Gerold:
00:42 Introduction to OrioleDB
04:38 The Undo Log Approach
08:39 Improving Performance for High Throughput Databases
11:08 My take on OrioleDB
OrioleDB, storage engine, Postgres, table access methods, undo log, high throughput databases, automated features, new use cases, S3, data migration
Today’s guest is Antonio Bustamante, a serial entrepreneur who previously built Kite and Silo and is now working to fix bad data. He is building bem, the data tool to transform any data into the schema your AI and software needs.
bem.ai is a data tool that focuses on transforming any data into the schema needed for AI and software. It acts as a system's interoperability layer, allowing systems that couldn't communicate before to exchange information. Learn what place LLMs play in data transformation, how to build reliable data infrastructure and more.
"Surprisingly, the hardest was semi-structured data. That is data that should be structured, but is unreliable, undocumented, hard to work with."
"We were spending close to four or five million dollars a year just in integrations, which is no small budget for a company that size. So I was pretty much determined to fix this problem once and for all."
"bem focuses on being the system's interoperability layer."
"We basically take in anything you send us, we transform it exactly into your internal data schema so that you don't have to parse, process, transform anything of that sort."
"LLMs are a 30% of it... A lot of it is very, very like thorough validation layers, great infrastructure, just ensuring reliability and connection to our user systems.”
"You can use a million token context window and feed an entire document to an LLM. I can guarantee you if you don't, semantically chunk it out before you're not going to get the right results.”
"We're obsessed with time to value... Our milestone is basically five minute onboarding max, and then you're ready to go."
Antonio Bustamante
bem.ai
Nicolay Gerold:
Semi-structured data, Data integrations, Large language models (LLMs), Data transformation, Schema interoperability, Fault tolerance, Validation layers, System reliability, Schema evolution, Enterprise software, Data pipelines.
Chapters
00:00 The Problem of Integrations
05:58 Building Fault Tolerant Systems
13:51 Versioning and Semantic Validation
27:33 BEM in the Data Ecosystem
34:40 Future Plans and Onboarding
Imagine a world where data bottlenecks, slow data loaders, or memory issues on the VM don't hold back machine learning.
Machine learning and AI success depends on the speed you can iterate. LanceDB is here to to enable fast experiments on top of terabytes of unstructured data. It is the database for AI. Dive with us into how LanceDB was built, what went into the decision to use Rust as the main implementation language, the potential of AI on top of LanceDB, and more.
"LanceDB is the database for AI...to manage their data, to do a performant billion scale vector search."
“We're big believers in the composable data systems vision."
"You can insert data into LanceDB using Panda's data frames...to sort of really large 'embed the internet' kind of workflows."
"We wanted to create a new generation of data infrastructure that makes their [AI engineers] lives a lot easier."
"LanceDB offers up to 1,000 times faster performance than Parquet."
Chang She:
LanceDB:
Nicolay Gerold:
Chapters:
00:00 Introduction to LanceDB
02:16 Building LanceDB in Rust
12:10 Optimizing Data Infrastructure
26:20 Surprising Use Cases for LanceDB
32:01 The Future of LanceDB
LanceDB, AI, database, Rust, multimodal AI, data infrastructure, embeddings, images, performance, Parquet, machine learning, model database, function registries, agents.
From the publisher's feed