
Sign up to save your podcasts
Or


In this episode, we talk data-driven search optimizations with Charlie Hull.
Charlie is a search expert from Open Source Connections. He has built Flax, one of the leading open source search companies in the UK, has written “Searching the Enterprise”, and is one of the main voices on data-driven search.
We discuss strategies to improve search systems quantitatively and much more.
Key Points:
Resources mentioned:
Charlie Hull:
Nicolay Gerold:
search results, search systems, assessing, evaluation, improvement, data quality, user behavior, proactive, test dataset, search engine optimization, SEO, search quality, metadata, query classification, user intent, search results, metrics, business objectives, user objectives, experimentation, continuous improvement, data modeling, embeddings, machine learning, information retrieval
00:00 Introduction
01:35 Challenges in Measuring Search Relevance
02:19 Common Mistakes in Search System Assessment
03:22 Methods to Measure Search System Performance
04:28 Human Evaluation in Search Systems
05:18 Leveraging User Interaction Data
06:04 Implementing AI for Search Evaluation
09:14 Technical Components for Assessing Search Systems
12:07 Improving Search Quality Through Data Analysis
17:16 Proactive Search System Monitoring
24:26 Balancing Business and User Objectives in Search
25:08 Search Metrics and KPIs: A Contract Between Teams
26:56 The Role of Recency and Popularity in Search Algorithms
28:56 Experimentation: The Key to Optimizing Search
30:57 Offline Search Labs and A/B Testing
34:05 Simple Levers to Improve Search
37:38 Data Modeling and Its Importance in Search
43:29 Combining Keyword and Vector Search
44:24 Bridging the Gap Between Machine Learning and Information Retrieval
47:13 Closing Remarks and Contact Information
Welcome back to How AI Is Built.
We have got a very special episode to kick off season two.
Daniel Tunkelang is a search consultant currently working with Algolia. He is a leader in the field of information retrieval, recommender systems, and AI-powered search. He worked for Canva, Algolia, Cisco, Gartner, Handshake, to pick a few.
His core focus is query understanding.
**Query understanding is about focusing less on the results and more on the query.** The query of the user is the first-class citizen. It is about figuring out what the user wants and than finding, scoring, and ranking results based on it. So most of the work happens before you hit the database.
**Key Takeaways:**
- The "bag of documents" model for queries and "bag of queries" model for documents are useful approaches for representing queries and documents in search systems.
- Query specificity is an important factor in query understanding. It can be measured using cosine similarity between query vectors and document vectors.
- Query classification into broad categories (e.g., product taxonomy) is a high-leverage technique for improving search relevance and can act as a guardrail for query expansion and relaxation.
- Large Language Models (LLMs) can be useful for search, but simpler techniques like query similarity using embeddings can often solve many problems without the complexity and cost of full LLM implementations.
- Offline processing to enhance document representations (e.g., filling in missing metadata, inferring categories) can significantly improve search quality.
**Daniel Tunkelang**
- [LinkedIn](https://www.linkedin.com/in/dtunkelang/)
- [Medium](https://queryunderstanding.com/)
**Nicolay Gerold:**
- [LinkedIn](https://www.linkedin.com/in/nicolay-gerold/)
- [X (Twitter)](https://twitter.com/nicolaygerold)
- [Substack](https://nicolaygerold.substack.com/)
Query understanding, search relevance, bag of documents, bag of queries, query specificity, query classification, named entity recognition, pre-retrieval processing, caching, large language models (LLMs), embeddings, offline processing, metadata enhancement, FastText, MiniLM, sentence transformers, visualization, precision, recall
[00:00:00] 1. Introduction to Query Understanding
[00:05:30] 2. Query Representation Models
[00:12:00] 3. Query Specificity and Classification
[00:19:30] 4. Named Entity Recognition in Query Understanding
[00:24:00] 5. Pre-Retrieval Query Processing
[00:28:30] 6. Performance Optimization Techniques
[00:33:00] 7. Advanced Techniques: Embeddings and Language Models
[00:39:00] 8. Practical Implementation Strategies
[00:44:00] 9. Visualization and Analysis of Query Spaces
[00:47:00] 10. Future Directions and Closing Thoughts - Emerging trends in query understanding - Key takeaways for search system engineers
[00:53:00] End of Episode
Today we are launching the season 2 of How AI Is Built.
The last few weeks, we spoke to a lot of regular listeners and past guests and collected feedback. Analyzed our episode data. And we will be applying the learnings to season 2.
This season will be all about search.
We are trying to make it better, more actionable, and more in-depth. The goal is that at the end of this season, you have a full-fleshed course on search in podcast form, which mini-courses on specific elements like RAG.
We will be talking to experts from information retrieval, information architecture, recommendation systems, and RAG; from academia and industry. Fields that do not really talk to each other.
We will try to unify and transfer the knowledge and give you a full tour of search, so you can build your next search application or feature with confidence.
We will be talking to Charlie Hull on how to systematically improve search systems, with Nils Reimers on the fundamental flaws of embeddings and how to fix them, with Daniel Tunkelang on how to actually understand the queries of the user, and many more.
We will try to bridge the gaps. How to use decades of research and practice in iteratively improving traditional search and apply it to RAG. How to take new methods from recommendation systems and vector databases and bring it into traditional search systems. How to use all of the different methods as search signals and combine them to deliver the results your user actually wants.
We will be using two types of episodes:
We will be starting with episodes next week, looking at the first, last, and overarching action in search: understanding user intent and understanding the queries with Daniel Tunkelang.
I am really excited to kick this off.
I would love to hear from you:
Yeah, let me know in the comments or just slide into my DMs on Twitter or LinkedIn.
I am looking forward to hearing from you guys.
I want to try to be more interactive. So anytime you encounter anything unclear or any question pops up in one of the episode, give me a shout and I will try to answer it to you and to everyone.
Enough of me rambling. Let’s kick this off. I will see you next Thursday, when we start with query understanding.
Shoot me a message and stay up to date:
In this episode of "How AI is Built," host Nicolay Gerold interviews Jonathan Yarkoni, founder of Reach Latent. Jonathan shares his expertise in extracting value from unstructured data using AI, discussing challenging projects, the impact of ChatGPT, and the future of generative AI. From weather prediction to legal tech, Jonathan provides valuable insights into the practical applications of AI across various industries.
Key Takeaways
Key Quotes
"I think we're going to see another wave in 2024 and another one in 2025. And people are familiarized. That's kind of the wave of 2023. 2024 is probably still going to be a lot of internal use cases because it's a low risk environment and there was a lot of opportunity to be had."
"To really get to production reliably, we have to have these tools evolve further and get more standardized so people can still use the old ways of doing production with the new technology."
Jonathan Yarkoni
Nicolay Gerold:
Chapters
00:00 Introduction: Extracting Value from Unstructured Data
03:16 Flexible Tailoring Solutions to Client Needs
05:39 Monitoring and Retraining Models in the Evolving AI Landscape
09:15 Generative AI: Disrupting Industries and Unlocking New Possibilities
17:47 Balancing Immediate Results and Cutting-Edge Solutions in AI Development
28:29 Dream Tech Stack for Generative AI
unstructured data, textual data, automation, weather prediction, data cleaning, chat GPT, AI disruption, legal, education, software engineering, marketing, biotech, immediate results, cutting-edge solutions, tech stack
This episode of "How AI Is Built" is all about data processing for AI. Abhishek Choudhary and Nicolay discuss Spark and alternatives to process data so it is AI-ready.
Spark is a distributed system that allows for fast data processing by utilizing memory. It uses a dataframe representation "RDD" to simplify data processing.
When should you use Spark to process your data for your AI Systems?
→ Use Spark when:
→ Consider alternatives when:
Spark isn't always necessary. Evaluate your specific needs and resources before committing to a Spark-based solution for AI data processing.
In today’s episode of How AI Is Built, Abhishek and I discuss data processing:
Abhishek Choudhary:
Nicolay Gerold:
In this episode, Nicolay talks with Rahul Parundekar, founder of AI Hero, about the current state and future of AI agents. Drawing from over a decade of experience working on agent technology at companies like Toyota, Rahul emphasizes the importance of focusing on realistic, bounded use cases rather than chasing full autonomy.
They dive into the key challenges, like effectively capturing expert workflows and decision processes, delivering seamless user experiences that integrate into existing routines, and managing costs through techniques like guardrails and optimized model choices. The conversation also explores potential new paradigms for agent interactions beyond just chat.
Key Takeaways:
Key Quotes:
Rahul Parundekar:
Nicolay Gerold:
00:00 Exploring the Potential of Autonomous Agents
02:23 Challenges of Accuracy and Repeatability in Agents
08:31 Capturing User Workflows and Improving Prompts
13:37 Tech Stack for Implementing Agents in the Enterprise
agent development, determinism, user experience, agent paradigms, private use, human-agent interaction, user workflows, agent deployment, human-in-the-loop, LLMs, declarative ways, scalability, AI Hero
In this conversation, Nicolay and Richmond Alake discuss various topics related to building AI agents and using MongoDB in the AI space. They cover the use of agents and multi-agents, the challenges of controlling agent behavior, and the importance of prompt compression.
When you are building agents. Build them iteratively. Start with simple LLM calls before moving to multi-agent systems.
Main Takeaways:
Richmond Alake:
Nicolay Gerold:
00:00 Reducing the Scope of AI Agents
01:55 Seamless Data Ingestion
03:20 Challenges and Considerations in Implementing Multi-Agents
06:05 Memory Modeling for Robust Agents with MongoDB
15:05 Performance Optimization in AI Agents
18:19 RAG Setup
AI agents, multi-agents, prompt compression, MongoDB, data storage, data ingestion, performance optimization, tooling, generative AI
In this episode, Kirk Marple, CEO and founder of Graphlit, shares his expertise on building efficient data integrations.
Kirk breaks down his approach using relatable concepts:
Kirk Marple:
Nicolay Gerold:
Chapters
00:00 Building Integrations into Different Tools
00:44 The Two-Sided Funnel Model for Data Flow
04:07 Using Well-Defined Interfaces for Faster Integration
04:36 Managing Feeds and State with Actor Models
06:05 The Importance of Data Normalization
10:54 Tech Stack for Data Flow
11:52 Progression towards a Kappa Architecture
13:45 Reusability of Patterns for Faster Integration
data integration, data sources, data flow, two-sided funnel model, canonical format, stream of ingestible objects, competing consumer model, well-defined interfaces, actor model, data normalization, tech stack, Kappa architecture, reusability of patterns
In our latest episode, we sit down with Derek Tu, Founder and CEO of Carbon, a cutting-edge ETL tool designed specifically for large language models (LLMs).
Carbon is streamlining AI development by providing a platform for integrating unstructured data from various sources, enabling businesses to build innovative AI applications more efficiently while addressing data privacy and ethical concerns.
Derek Tu:
Nicolay Gerold:
Key Takeaways:
00:00 Introduction and Optimizing Embedding Models
03:00 The Evolution of Carbon and Focus on Unstructured Data
06:19 Customer Progression and Target Group
09:43 Interesting Use Cases and Handling Different Data Representations
13:30 Chunking Strategies and Normalization
20:14 Approach to Chunking and Choosing a Vector Database
23:06 Tech Stack and Recommended Tools
28:19 Future of Carbon: Multimodal Models and Building a Platform
Carbon, LLMs, RAG, chunking, data processing, global customer base, GDPR compliance, AI founders, AI agents, enterprises
In this episode, Nicolay sits down with Hugo Lu, founder and CEO of Orchestra, a modern data orchestration platform. As data pipelines and analytics workflows become increasingly complex, spanning multiple teams, tools and cloud services, the need for unified orchestration and visibility has never been greater.
Orchestra is a serverless data orchestration tool that aims to provide a unified control plane for managing data pipelines, infrastructure, and analytics across an organization's modern data stack.
The core architecture involves users building pipelines as code which then run on Orchestra's serverless infrastructure. It can orchestrate tasks like data ingestion, transformation, AI calls, as well as monitoring and getting analytics on data products. All with end-to-end visibility, data lineage and governance even when organizations have a scattered, modular data architecture across teams and tools.
Key Quotes:
Hugo Lu:
Nicolay Gerold:
00:00 Introduction to Orchestra and its Focus on Data Products
08:03 Unified Control Plane for Data Stack and End-to-End Control
14:42 Use Cases and Unique Applications of Orchestra
19:31 Retaining Existing Dev Workflows and Best Practices in Orchestra
22:23 Event-Driven Architectures and Monitoring in Orchestra
23:49 Putting Data Products First and Monitoring Health and Usage
25:40 The Future of Data Orchestration: Stream-Based and Cost-Effective
data orchestration, Orchestra, serverless architecture, versatility, use cases, maturity levels, challenges, AI workloads
From the publisher's feed