How AI Is Built

Modern search systems face a complex balancing act between performance, relevancy, and cost, requiring careful architectural decisions at each layer.

While vector search generates buzz, hybrid approaches combining traditional text search with vector capabilities yield better results.

The architecture typically splits into three core components:

1. ingestion/indexing (requiring decisions between batch vs streaming)
2. query processing (balancing understanding vs performance)
3. analytics/feedback loops for continuous improvement.

Critical but often overlooked aspects include query understanding depth, systematic relevancy testing (avoid anecdote-driven development), and data governance as search systems naturally evolve into organizational data hubs.

Performance optimization requires careful tradeoffs between index-time vs query-time computation, with even 1-2% improvements being significant in mature systems.

Success requires testing against production data (staging environments prove unreliable), implementing proper evaluation infrastructure (golden query sets, A/B testing, interleaving), and avoiding the local maxima trap where improving one query set unknowingly damages others.

The end goal is finding an acceptable balance between corpus size, latency requirements, and cost constraints while maintaining system manageability and relevance quality.

"It's quite easy to end up in local maxima, whereby you improve a query for one set and then you end up destroying it for another set."

"A good marker of a sophisticated system is one where you actually see it's getting worse... you might be discovering a maxima."

"There's no free lunch in all of this. Often it's a case that, to service billions of documents on a vector search, less than 10 millis, you can do those kinds of things. They're just incredibly expensive. It's really about trying to manage all of the overall system to find what is an acceptable balance."

Search Pioneers:

• Website
• GitHub

Stuart Cam:

• LinkedIn

Russ Cam:

• Github
• LinkedIn
• X (Twitter)

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

00:00 Introduction to Search Systems 00:13 Challenges in Search: Relevancy vs Latency 00:27 Insights from Industry Experts 01:00 Evolution of Search Technologies 03:16 Storage and Compute in Search Systems 06:22 Common Mistakes in Building Search Systems 09:10 Evaluating and Improving Search Systems 19:27 Architectural Components of Search Systems 29:17 Understanding Search Query Expectations 29:39 Balancing Speed, Cost, and Corpus Size 32:03 Trade-offs in Search System Design 32:53 Indexing vs Querying: Key Considerations 35:28 Re-ranking and Personalization Challenges 38:11 Evaluating Search System Performance 44:51 Overrated vs Underrated Search Techniques 48:31 Final Thoughts and Contact Information

Today we are talking to Michael Günther, a senior machine learning scientist at Jina about his work on JINA Clip.

Some key points:

• Uni-modal embeddings convert a single type of input (text, images, audio) into vectors
• Multimodal embeddings learn a joint embedding space that can handle multiple types of input, enabling cross-modal search (e.g., searching images with text)
• Multimodal models can potentially learn richer representations of the world, including concepts that are difficult or impossible to put into words

Types of Text-Image Models

1. CLIP-like Models
   • Separate vision and text transformer models
   • Each tower maps inputs to a shared vector space
   • Optimized for efficient retrieval
2. Vision-Language Models
   • Process image patches as tokens
   • Use transformer architecture to combine image and text information
   • Better suited for complex document matching
3. Hybrid Models
   • Combine separate encoders with additional transformer components
   • Allow for more complex interactions between modalities
   • Example: Google's Magic Lens model

Training Insights from Jina CLIP

1. Key Learnings
   • Freezing the text encoder during training can significantly hinder performance
   • Short image captions limit the model's ability to learn rich text representations
   • Large batch sizes are crucial for training embedding models effectively
2. Training Process
   • Three-stage training approach: 
     • Stage 1: Training on image captions and text pairs
     • Stage 2: Adding longer image captions
     • Stage 3: Including triplet data with hard negatives

Practical Considerations

• Similarity Scales
  • Different modalities can produce different similarity value scales
  • Important to consider when combining multiple embedding types
  • Can affect threshold-based filtering
• Model Selection
  • Evaluate models based on relevant benchmarks
  • Consider the domain similarity between training data and intended use case
  • Assessment of computational requirements and efficiency needs

Future Directions

1. Areas for Development
   • More comprehensive benchmarks for multimodal tasks
   • Better support for semi-structured data
   • Improved handling of non-photographic images
2. Upcoming Developments at Jina AI
   • Multilingual support for Jina ColBERT
   • New version of text embedding models
   • Focus on complex multimodal search applications

Practical Applications

• E-commerce
  • Product search and recommendations
  • Combined text-image embeddings for better results
  • Synthetic data generation for fine-tuning
• Fine-tuning Strategies
  • Using click data and query logs
  • Generative pseudo-labeling for creating training data
  • Domain-specific adaptations

Key Takeaways for Engineers

1. Be aware of similarity value scales and their implications
2. Establish quantitative evaluation metrics before optimization
3. Consider model limitations (e.g., image resolution, text length)
4. Use performance optimizations like flash attention and activation checkpointing
5. Universal embedding models might not be optimal for specific use cases

Michael Guenther

• LinkedIn
• X (Twitter)
• Jina AI
• New Multilingual Embedding Modal

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

00:00 Introduction to Uni-modal and Multimodal Embeddings 00:16 Exploring Multimodal Embeddings and Their Applications 01:06 Training Multimodal Embedding Models 02:21 Challenges and Solutions in Embedding Models 07:29 Advanced Techniques and Future Directions 29:19 Understanding Model Interference in Search Specialization 30:17 Fine-Tuning Jina CLIP for E-Commerce 32:18 Synthetic Data Generation and Pseudo-Labeling 33:36 Challenges and Learnings in Embedding Models 40:52 Future Directions and Takeaways

Imagine a world where data bottlenecks, slow data loaders, or memory issues on the VM don't hold back machine learning.

Machine learning and AI success depends on the speed you can iterate. LanceDB is here to to enable fast experiments on top of terabytes of unstructured data. It is the database for AI. Dive with us into how LanceDB was built, what went into the decision to use Rust as the main implementation language, the potential of AI on top of LanceDB, and more.

"LanceDB is the database for AI...to manage their data, to do a performant billion scale vector search."

“We're big believers in the composable data systems vision."

"You can insert data into LanceDB using Panda's data frames...to sort of really large 'embed the internet' kind of workflows."

"We wanted to create a new generation of data infrastructure that makes their [AI engineers] lives a lot easier."

"LanceDB offers up to 1,000 times faster performance than Parquet."

Change She:

• LinkedIn
• X (Twitter)

LanceDB:

• X (Twitter)
• GitHub
• Web
• Discord
• VectorDB Recipes

Nicolay Gerold:

• LinkedIn
• X (Twitter)

00:00 Introduction to Multimodal Embeddings
00:26 Challenges in Storage and Serving
02:51 LanceDB: The Solution for Multimodal Data
04:25 Interview with Chang She: Origins and Vision
10:37 Technical Deep Dive: LanceDB and Rust
18:11 Innovations in Data Storage Formats
19:00 Optimizing Performance in Lakehouse Ecosystems
21:22 Future Use Cases for LanceDB
26:04 Building Effective Recommendation Systems
32:10 Exciting Applications and Future Directions

Today’s guest is Mór Kapronczay. Mór is the Head of ML at superlinked. Superlinked is a compute framework for your information retrieval and feature engineering systems, where they turn anything into embeddings.

When most people think about embeddings, they think about ada, openai.

You just take your text and throw it in there.

But that’s too crude.

OpenAI embeddings are trained on the internet.

But your data set (most likely) is not the internet.

You have different nuances.

And you have more than just text.

So why not use it.

Some highlights:

1. Text Embeddings are Not a Magic Bullet

➡️ Pouring everything into a text embedding model won't yield magical results ➡️ Language is lossy - it's a poor compression method for complex information

1. Embedding Numerical Data

➡️ Direct number embeddings don't work well for vector search ➡️ Consider projecting number ranges onto a quarter circle ➡️ Apply logarithmic transforms for skewed distributions

1. Multi-Modal Embeddings

➡️ Create separate vector parts for different data aspects ➡️ Normalize individual parts ➡️ Weight vector parts based on importance

A Multi-Vector approach can help you understand the contributions of each modality or embedding and give you an easier time to fine-tune your retrieval system without fine-tuning your embedding models by tuning your vector database like you would a search database (like Elastic).

Mór Kapronczay

• LinkedIn
• Superlinked
• X (Twitter)

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

00:00 Introduction to Embeddings 00:30 Beyond Text: Expanding Embedding Capabilities 02:09 Challenges and Innovations in Embedding Techniques 03:49 Unified Representations and Vector Computers 05:54 Embedding Complex Data Types 07:21 Recommender Systems and Interaction Data 08:59 Combining and Weighing Embeddings 14:58 Handling Numerical and Categorical Data 20:35 Optimizing Embedding Efficiency 22:46 Dynamic Weighting and Evaluation 24:35 Exploring AB Testing with Embeddings 25:08 Joint vs Separate Embedding Spaces 27:30 Understanding Embedding Dimensions 29:59 Libraries and Frameworks for Embeddings 32:08 Challenges in Embedding Models 33:03 Vector Database Connectors 34:09 Balancing Production and Updates 36:50 Future of Vector Search and Modalities 39:36 Building with Embeddings: Tips and Tricks 42:26 Concluding Thoughts and Next Steps

Today we have Jessica Talisman with us, who is working as an Information Architect at Adobe. She is (in my opinion) the expert on taxonomies and ontologies.

That’s what you will learn today in this episode of How AI Is Built. Taxonomies, ontologies, knowledge graphs.

Everyone is talking about them no-one knows how to build them.

But before we look into that, what are they good for in search?

Imagine a large corpus of academic papers. When a user searches for "machine learning in healthcare", the system can:

• Recognize "machine learning" as a subcategory of "artificial intelligence"
• Identify "healthcare" as a broad field with subfields like "diagnostics" and "patient care"
• We can use these to expand the query or narrow it down.
• We can return results that include papers on "neural networks for medical imaging" or "predictive analytics in patient outcomes", even if these exact phrases weren't in the search query
• We can also filter down and remove papers not tagged with AI that might just mention it in a side not.

So we are building the plumbing, the necessary infrastructure for tagging, categorization, query expansion and relexation, filtering.

So how can we build them?

1️⃣ Start with Industry Standards • Leverage established taxonomies (e.g., Google, GS1, IAB) • Audit them for relevance to your project • Use as a foundation, not a final solution

2️⃣ Customize and Fill Gaps • Adapt industry taxonomies to your specific domain • Create a "coverage model" for your unique needs • Mine internal docs to identify domain-specific concepts

3️⃣ Follow Ontology Best Practices • Use clear, unique primary labels for each concept • Include definitions to avoid ambiguity • Provide context for each taxonomy node

Jessica Talisman:

• LinkedIn

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

00:00 Introduction to Taxonomies and Knowledge Graphs 02:03 Building the Foundation: Metadata to Knowledge Graphs 04:35 Industry Taxonomies and Coverage Models 06:32 Clustering and Labeling Techniques 11:00 Evaluating and Maintaining Taxonomies 31:41 Exploring Taxonomy Granularity 32:18 Differentiating Taxonomies for Experts and Users 33:35 Mapping and Equivalency in Taxonomies 34:02 Best Practices and Examples of Taxonomies 40:50 Building Multilingual Taxonomies 44:33 Creative Applications of Taxonomies 48:54 Overrated and Underappreciated Technologies 53:00 The Importance of Human Involvement in AI 53:57 Connecting with the Speaker 55:05 Final Thoughts and Takeaways

ColPali makes us rethink how we approach document processing.

ColPali revolutionizes visual document search by combining late interaction scoring with visual language models. This approach eliminates the need for extensive text extraction and preprocessing, handling messy real-world data more effectively than traditional methods.

In this episode, Jo Bergum, chief scientist at Vespa, shares his insights on how ColPali is changing the way we approach complex document formats like PDFs and HTML pages.

Introduction to ColPali:

• Combines late interaction scoring from Colbert with visual language model (PoliGemma)
• Represents screenshots of documents as multi-vector representations
• Enables searching across complex document formats (PDFs, HTML)
• Eliminates need for extensive text extraction and preprocessing

Advantages of ColPali:

• Handles messy, real-world data better than traditional methods
• Considers both textual and visual elements in documents
• Potential applications in various domains (finance, medical, legal)
• Scalable to large document collections with proper optimization

Jo Bergum:

• LinkedIn
• Vespa
• X (Twitter)
• PDF Retrieval with Vision Language Models
• Scaling ColPali to billions of PDFs with Vespa

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

00:00 Messy Data in AI 01:19 Challenges in Search Systems 03:41 Understanding Representational Approaches 08:18 Dense vs Sparse Representations 19:49 Advanced Retrieval Models and ColPali 30:59 Exploring Image-Based AI Progress 32:25 Challenges and Innovations in OCR 33:45 Understanding ColPali and MaxSim 38:13 Scaling and Practical Applications of ColPali 44:01 Future Directions and Use Cases

Today, we're talking to Aamir Shakir, the founder and baker at mixedbread.ai, where he's building some of the best embedding and re-ranking models out there. We go into the world of rerankers, looking at how they can classify, deduplicate documents, prioritize LLM outputs, and delve into models like ColBERT.

We discuss:

• The role of rerankers in retrieval pipelines
• Advantages of late interaction models like ColBERT for interpretability
• Training rerankers vs. embedding models and their impact on performance
• Incorporating metadata and context into rerankers for enhanced relevance
• Creative applications of rerankers beyond traditional search
• Challenges and future directions in the retrieval space

Still not sure whether to listen? Here are some teasers:

• Rerankers can significantly boost your retrieval system's performance without overhauling your existing setup.
• Late interaction models like ColBERT offer greater explainability by allowing token-level comparisons between queries and documents.
• Training a reranker often yields a higher impact on retrieval performance than training an embedding model.
• Incorporating metadata directly into rerankers enables nuanced search results based on factors like recency and pricing.
• Rerankers aren't just for search—they can be used for zero-shot classification, deduplication, and prioritizing outputs from large language models.
• The future of retrieval may involve compound models capable of handling multiple modalities, offering a more unified approach to search.

Aamir Shakir:

• LinkedIn
• X (Twitter)
• Mixedbread.ai

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

00:00 Introduction and Overview 00:25 Understanding Rerankers 01:46 Maxsim and Token-Level Embeddings 02:40 Setting Thresholds and Similarity 03:19 Guest Introduction: Aamir Shakir 03:50 Training and Using Rerankers (Episode Start) 04:50 Challenges and Solutions in Reranking 08:03 Future of Retrieval and Recommendation 26:05 Multimodal Retrieval and Reranking 38:04 Conclusion and Takeaways

Text embeddings have limitations when it comes to handling long documents and out-of-domain data.

Today, we are talking to Nils Reimers. He is one of the researchers who kickstarted the field of dense embeddings, developed sentence transformers, started HuggingFace’s Neural Search team and now leads the development of search foundational models at Cohere. Tbh, he has too many accolades to count off here.

We talk about the main limitations of embeddings:

• Failing out of domain
• Struggling with long documents
• Very hard to debug
• Hard to find formalize what actually is similar

Are you still not sure whether to listen? Here are some teasers:

• Interpreting embeddings can be challenging, and current models are not easily explainable.
• Fine-tuning is necessary to adapt embeddings to specific domains, but it requires careful consideration of the data and objectives.
• Re-ranking is an effective approach to handle long documents and incorporate additional factors like recency and trustworthiness.
• The future of embeddings lies in addressing scalability issues and exploring new research directions.

Nils Reimers:

• LinkedIn
• X (Twitter)
• Website
• Cohere

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

text embeddings, limitations, long documents, interpretation, fine-tuning, re-ranking, future research

00:00 Introduction and Guest Introduction 00:43 Early Work with BERT and Argument Mining 02:24 Evolution and Innovations in Embeddings 03:39 Constructive Learning and Hard Negatives 05:17 Training and Fine-Tuning Embedding Models 12:48 Challenges and Limitations of Embeddings 18:16 Adapting Embeddings to New Domains 22:41 Handling Long Documents and Re-Ranking 31:08 Combining Embeddings with Traditional ML 45:16 Conclusion and Upcoming Episodes

Hey! Welcome back.

Today we look at how we can get our RAG system ready for scale.

We discuss common problems and their solutions, when you introduce more users and more requests to your system.

For this we are joined by Nirant Kasliwal, the author of fastembed.

Nirant shares practical insights on metadata extraction, evaluation strategies, and emerging technologies like Colipali. This episode is a must-listen for anyone looking to level up their RAG implementations.

"Naive RAG has a lot of problems on the retrieval end and then there's a lot of problems on how LLMs look at these data points as well."

"The first 30 to 50% of gains are relatively quick. The rest 50% takes forever."

"You do not want to give the same answer about company's history to the co-founding CEO and the intern who has just joined."

"Embedding similarity is the signal on which you want to build your entire search is just not quite complete."

Key insights:

• Naive RAG often fails due to limitations of embeddings and LLMs' sensitivity to input ordering.
• Query profiling and expansion: 
  • Use clustering and tools like latent Scope to identify problematic query types
  • Expand queries offline and use parallel searches for better results
• Metadata extraction: 
  • Extract temporal, entity, and other relevant information from queries
  • Use LLMs for extraction, with checks against libraries like Stanford NLP
• User personalization: 
  • Include user role, access privileges, and conversation history
  • Adapt responses based on user expertise and readability scores
• Evaluation and improvement: 
  • Create synthetic datasets and use real user feedback
  • Employ tools like DSPY for prompt engineering
• Advanced techniques: 
  • Query routing based on type and urgency
  • Use smaller models (1-3B parameters) for easier iteration and error spotting
  • Implement error handling and cross-validation for extracted metadata

Nirant Kasliwal:

• X (Twitter)
• LinkedIn
• Search in the LLM Era for AI Engineers (course)

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

query understanding, AI-powered search, Lambda Mart, e-commerce ranking, networking, experts, recommendation, search

In this episode of How AI is Built, Nicolay Gerold interviews Doug Turnbull, a search engineer at Reddit and author on “Relevant Search”. They discuss how methods and technologies, including large language models (LLMs) and semantic search, contribute to relevant search results.

Key Highlights:

• Defining relevance is challenging and depends heavily on user intent and context
• Combining multiple search techniques (keyword, semantic, etc.) in tiers can improve results
• LLMs are emerging as a powerful tool for augmenting traditional search approaches
• Operational concerns often drive architectural decisions in large-scale search systems
• Underappreciated techniques like LambdaMART may see a resurgence

Key Quotes:

"There's not like a perfect measure or definition of what a relevant search result is for a given application. There are a lot of really good proxies, and a lot of really good like things, but you can't just like blindly follow the one objective, if you want to build a good search product." - Doug Turnbull

"I think 10 years ago, what people would do is they would just put everything in Solr, Elasticsearch or whatever, and they would make the query to Elasticsearch pretty complicated to rank what they wanted... What I see people doing more and more these days is that they'll use each retrieval source as like an independent piece of infrastructure." - Doug Turnbull on the evolution of search architecture

"Honestly, I feel like that's a very practical and underappreciated thing. People talk about RAG and I talk, I call this GAR - generative AI augmented retrieval, so you're making search smarter with generative AI." - Doug Turnbull on using LLMs to enhance search

"LambdaMART and gradient boosted decision trees are really powerful, especially for when you're expressing your re-ranking as some kind of structured learning problem... I feel like we'll see that and like you're seeing papers now where people are like finding new ways of making BM25 better." - Doug Turnbull on underappreciated techniques

Doug Turnbull

• LinkedIn
• X (Twitter)
• Web

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

Chapters

00:00 Introduction and Guest Introduction 00:52 Understanding Relevant Search Results 01:18 Search Behavior on Social Media 02:14 Challenges in Defining Relevance 05:12 Query Understanding and Ranking Signals 10:57 Evolution of Search Technologies 15:15 Combining Search Techniques 21:49 Leveraging LLMs and Embeddings 25:49 Operational Considerations in Search Systems 39:09 Concluding Thoughts and Future Directions