For the Sake of Search

Search is changing fast. AI has raised user expectations, vector search has brought in a wave of new builders, and more teams are realizing that similarity alone does not mean relevance. In this episode, we’re joined by Dom Couldwell, Product Manager for OpenSearch at IBM, to talk about intent-aware search. What does “intent” mean, how teams can uncover it, and why search needs to be treated as a product, not just infrastructure. We’ll explore query intent, product intent, journey intent, expensive vs. efficient search strategies, and the growing identity crisis search faces in the age of AI. This episode is brought to you by https://relevan.dev.

How can we determine what users are looking for when they search? We’re joined by Tito Sierra, Director of Product Management at Yahoo, to break down how modern search engines interpret queries at scale. Most companies get this wrong and we'll discuss why. Lets dive into query understanding, intent classification, and the real product and engineering work behind making search feel “smart.” If you’ve ever wondered why search fails, or how to fix it, this episode lays the foundation. This episode is brought to you by relevan.dev - search that sells.

HNSW and IVF in the same index? This week we have James (Xiaofan Luan) the VP of Engineering from Zilliz to talk about some of the magic that makes Milvus special. They use all sorts of techniques from indexes with multiple retrieval sources and the Starling algorithm.

Make sure to check out Milvus at https://milvus.io and the Starling paper at: https://arxiv.org/abs/2401.02116

How can you be sure your search experience is good? All it takes is the right tools. Eric and I walk through what the "right tools" are for the search engineers tool belt. Make sure to check out Quepid for testing your relevancy! https://github.com/o19s/quepid. Also, make sure to check out Haystack the Search Relevance conference!

So much recently I've seen companies obsessing over fine tuning. Should you fine tune your language models? Nicolay from AISBACH Data Solutions joins me this week as we talk through the many challenges that come when you fine tune language models. He's spent the last several years working with customers to get the most out of their LLM's and has some insights to share.

You wouldn't hire a network engineer to build your front end would you? Then why hire just any software engineer for search?! In this episode we talk with Brian Pedersen founder of TheSearchBar.ai to talk a bit more about the challenges of hiring search engineers.

There tends to be a big disconnect between companies that want to hire search engineers and engineers that are looking for companies that value their skills. Thankfully Brian is here to help give some advice so that everyone can get what they are looking for!

Elasticsearch is open source again? Maybe? Join Nick (@skeptrune) and I as we discuss what Elasticsearchs new license means for its users. Along with this we take a dive into Trieve (trieve.ai) a source available framework for building search and RAG workflows. How better to test a new product than rebuilding search for Hacker News?!

We talk with Philippe from ParadeDB about search on Postgres. Historically search on Postgres has been really bad but it doesn't have to be anymore. Let's go into a bit of the why search on Postgres is the way it is. This will help us understand why it got to where it is and how ParadeDB is making it better.

Session Notes:

• Why ParadeDB picked AGPL for their License
• Putting DuckDB into Postgres