How AI Is Built

How AI Is Built

By Nicolay GeroldTechnology
Download on the App Store

How AI Is Built episodes

  • #019 Data-driven Search Optimization, Analysing Relevance

    In this episode, we talk data-driven search optimizations with Charlie Hull.

    Charlie is a search expert from Open Source Connections. He has built Flax, one of the leading open source search companies in the UK, has written “Searching the Enterprise”, and is one of the main voices on data-driven search.

    We discuss strategies to improve search systems quantitatively and much more.

    Key Points:

    1. Relevance in search is subjective and context-dependent, making it challenging to measure consistently.
    2. Common mistakes in assessing search systems include overemphasizing processing speed and relying solely on user complaints.
    3. Three main methods to measure search system performance: 
      • Human evaluation
      • User interaction data analysis
      • AI-assisted judgment (with caution)
    4. Importance of balancing business objectives with user needs when optimizing search results.
    5. Technical components for assessing search systems: 
      • Query logs analysis
      • Source data quality examination
      • Test queries and cases setup

    Resources mentioned:

    • Quepid: Open-source tool for search quality testing
    • Haystack conference: Upcoming event in Berlin (September 30 - October 1)
    • Relevance Slack community
    • OpenSource Connections

    Charlie Hull:

    • LinkedIn
    • X (Twitter)

    Nicolay Gerold:

    • ⁠LinkedIn⁠
    • ⁠X (Twitter)

    search results, search systems, assessing, evaluation, improvement, data quality, user behavior, proactive, test dataset, search engine optimization, SEO, search quality, metadata, query classification, user intent, search results, metrics, business objectives, user objectives, experimentation, continuous improvement, data modeling, embeddings, machine learning, information retrieval

    00:00 Introduction
    01:35 Challenges in Measuring Search Relevance
    02:19 Common Mistakes in Search System Assessment
    03:22 Methods to Measure Search System Performance
    04:28 Human Evaluation in Search Systems
    05:18 Leveraging User Interaction Data
    06:04 Implementing AI for Search Evaluation
    09:14 Technical Components for Assessing Search Systems
    12:07 Improving Search Quality Through Data Analysis
    17:16 Proactive Search System Monitoring
    24:26 Balancing Business and User Objectives in Search
    25:08 Search Metrics and KPIs: A Contract Between Teams
    26:56 The Role of Recency and Popularity in Search Algorithms
    28:56 Experimentation: The Key to Optimizing Search
    30:57 Offline Search Labs and A/B Testing
    34:05 Simple Levers to Improve Search
    37:38 Data Modeling and Its Importance in Search
    43:29 Combining Keyword and Vector Search
    44:24 Bridging the Gap Between Machine Learning and Information Retrieval
    47:13 Closing Remarks and Contact Information

    52 min
  • #018 Query Understanding: Doing The Work Before The Query Hits The Database

    Welcome back to How AI Is Built. 

    We have got a very special episode to kick off season two. 

    Daniel Tunkelang is a search consultant currently working with Algolia. He is a leader in the field of information retrieval, recommender systems, and AI-powered search. He worked for Canva, Algolia, Cisco, Gartner, Handshake, to pick a few. 

    His core focus is query understanding.  

    **Query understanding is about focusing less on the results and more on the query.** The query of the user is the first-class citizen. It is about figuring out what the user wants and than finding, scoring, and ranking results based on it. So most of the work happens before you hit the database. 

    **Key Takeaways:**

    - The "bag of documents" model for queries and "bag of queries" model for documents are useful approaches for representing queries and documents in search systems.
    - Query specificity is an important factor in query understanding. It can be measured using cosine similarity between query vectors and document vectors.
    - Query classification into broad categories (e.g., product taxonomy) is a high-leverage technique for improving search relevance and can act as a guardrail for query expansion and relaxation.
    - Large Language Models (LLMs) can be useful for search, but simpler techniques like query similarity using embeddings can often solve many problems without the complexity and cost of full LLM implementations.
    - Offline processing to enhance document representations (e.g., filling in missing metadata, inferring categories) can significantly improve search quality.

    **Daniel Tunkelang**

    - [LinkedIn](https://www.linkedin.com/in/dtunkelang/)
    - [Medium](https://queryunderstanding.com/)

    **Nicolay Gerold:**

    - [⁠LinkedIn⁠](https://www.linkedin.com/in/nicolay-gerold/)
    - [⁠X (Twitter)](https://twitter.com/nicolaygerold)
    - [Substack](https://nicolaygerold.substack.com/)

    Query understanding, search relevance, bag of documents, bag of queries, query specificity, query classification, named entity recognition, pre-retrieval processing, caching, large language models (LLMs), embeddings, offline processing, metadata enhancement, FastText, MiniLM, sentence transformers, visualization, precision, recall

    [00:00:00] 1. Introduction to Query Understanding

    • Definition and importance in search systems
    • Evolution of query understanding techniques

    [00:05:30] 2. Query Representation Models

    • The "bag of documents" model for queries
    • The "bag of queries" model for documents
    • Advantages of holistic query representation

    [00:12:00] 3. Query Specificity and Classification

    • Measuring query specificity using cosine similarity
    • Importance of query classification in search relevance
    • Implementing and leveraging query classifiers

    [00:19:30] 4. Named Entity Recognition in Query Understanding

    • Role of NER in query processing
    • Challenges with unique or tail entities

    [00:24:00] 5. Pre-Retrieval Query Processing

    • Importance of early-stage query analysis
    • Balancing computational resources and impact

    [00:28:30] 6. Performance Optimization Techniques

    • Caching strategies for query understanding
    • Offline processing for document enhancement

    [00:33:00] 7. Advanced Techniques: Embeddings and Language Models

    • Using embeddings for query similarity
    • Role of Large Language Models (LLMs) in search
    • When to use simpler techniques vs. complex models

    [00:39:00] 8. Practical Implementation Strategies

    • Starting points for engineers new to query understanding
    • Tools and libraries for query understanding (FastText, MiniLM, etc.)
    • Balancing precision and recall in search systems

    [00:44:00] 9. Visualization and Analysis of Query Spaces

    • Discussion on t-SNE, UMAP, and other visualization techniques
    • Limitations and alternatives to embedding visualizations

    [00:47:00] 10. Future Directions and Closing Thoughts - Emerging trends in query understanding - Key takeaways for search system engineers

    [00:53:00] End of Episode

    54 min
  • Season 2 Trailer: Mastering Search

    Today we are launching the season 2 of How AI Is Built.

    The last few weeks, we spoke to a lot of regular listeners and past guests and collected feedback. Analyzed our episode data. And we will be applying the learnings to season 2.

    This season will be all about search.

    We are trying to make it better, more actionable, and more in-depth. The goal is that at the end of this season, you have a full-fleshed course on search in podcast form, which mini-courses on specific elements like RAG.

    We will be talking to experts from information retrieval, information architecture, recommendation systems, and RAG; from academia and industry. Fields that do not really talk to each other.

    We will try to unify and transfer the knowledge and give you a full tour of search, so you can build your next search application or feature with confidence.

    We will be talking to Charlie Hull on how to systematically improve search systems, with Nils Reimers on the fundamental flaws of embeddings and how to fix them, with Daniel Tunkelang on how to actually understand the queries of the user, and many more.


    We will try to bridge the gaps. How to use decades of research and practice in iteratively improving traditional search and apply it to RAG. How to take new methods from recommendation systems and vector databases and bring it into traditional search systems. How to use all of the different methods as search signals and combine them to deliver the results your user actually wants.

    We will be using two types of episodes:

    1. Traditional deep dives, like we have done them so far. Each one will dive into one specific topic within search interviewing an expert on that topic.
    2. Supplementary episodes, which answer one additional question; often either complementary or precursory knowledge for the episode, which we did not get to in the deep dive.

    We will be starting with episodes next week, looking at the first, last, and overarching action in search: understanding user intent and understanding the queries with Daniel Tunkelang.

    I am really excited to kick this off.

    I would love to hear from you:

    • What would you love to learn in this season?
    • What guest should I have on?
    • What topics should I make a deep dive on (try to be specific)?

    Yeah, let me know in the comments or just slide into my DMs on Twitter or LinkedIn.

    I am looking forward to hearing from you guys.

    I want to try to be more interactive. So anytime you encounter anything unclear or any question pops up in one of the episode, give me a shout and I will try to answer it to you and to everyone.

    Enough of me rambling. Let’s kick this off. I will see you next Thursday, when we start with query understanding.

    Shoot me a message and stay up to date:

    • ⁠LinkedIn⁠
    • ⁠X (Twitter)
    5 min
  • #017 Unlocking Value from Unstructured Data, Real-World Applications of Generative AI

    In this episode of "How AI is Built," host Nicolay Gerold interviews Jonathan Yarkoni, founder of Reach Latent. Jonathan shares his expertise in extracting value from unstructured data using AI, discussing challenging projects, the impact of ChatGPT, and the future of generative AI. From weather prediction to legal tech, Jonathan provides valuable insights into the practical applications of AI across various industries.

    Key Takeaways

    • Generative AI projects often require less data cleaning due to the models' tolerance for "dirty" data, allowing for faster implementation in some cases.
    • The success of AI projects post-delivery is ensured through monitoring, but automatic retraining of generative AI applications is not yet common due to evaluation challenges.
    • Industries ripe for AI disruption include text-heavy fields like legal, education, software engineering, and marketing, as well as biotech and entertainment.
    • The adoption of AI is expected to occur in waves, with 2024 likely focusing on internal use cases and 2025 potentially seeing more customer-facing applications as models improve.
    • Synthetic data generation, using models like GPT-4, can be a valuable approach for training AI systems when real data is scarce or sensitive.
    • Evaluation frameworks like RAGAS and custom metrics are essential for assessing the quality of synthetic data and AI model outputs.
    • Jonathan’s ideal tech stack for generative AI projects includes tools like Instructor, Guardrails, Semantic Routing, DSPY, LangChain, and LlamaIndex, with a growing emphasis on evaluation stacks.

    Key Quotes

    "I think we're going to see another wave in 2024 and another one in 2025. And people are familiarized. That's kind of the wave of 2023. 2024 is probably still going to be a lot of internal use cases because it's a low risk environment and there was a lot of opportunity to be had."

    "To really get to production reliably, we have to have these tools evolve further and get more standardized so people can still use the old ways of doing production with the new technology."

    Jonathan Yarkoni

    • LinkedIn
    • YouTube
    • X (Twitter)
    • Reach Latent

    Nicolay Gerold:

    • ⁠LinkedIn⁠
    • ⁠X (Twitter)

    Chapters

    00:00 Introduction: Extracting Value from Unstructured Data 
    03:16 Flexible Tailoring Solutions to Client Needs 
    05:39 Monitoring and Retraining Models in the Evolving AI Landscape 
    09:15 Generative AI: Disrupting Industries and Unlocking New Possibilities 
    17:47 Balancing Immediate Results and Cutting-Edge Solutions in AI Development 
    28:29 Dream Tech Stack for Generative AI

    unstructured data, textual data, automation, weather prediction, data cleaning, chat GPT, AI disruption, legal, education, software engineering, marketing, biotech, immediate results, cutting-edge solutions, tech stack

    37 min
  • #016 Data Processing for AI, Integrating AI into Data Pipelines, Spark

    This episode of "How AI Is Built" is all about data processing for AI. Abhishek Choudhary and Nicolay discuss Spark and alternatives to process data so it is AI-ready.

    Spark is a distributed system that allows for fast data processing by utilizing memory. It uses a dataframe representation "RDD" to simplify data processing.

    When should you use Spark to process your data for your AI Systems?

    → Use Spark when:

    • Your data exceeds terabytes in volume
    • You expect unpredictable data growth
    • Your pipeline involves multiple complex operations
    • You already have a Spark cluster (e.g., Databricks)
    • Your team has strong Spark expertise
    • You need distributed computing for performance
    • Budget allows for Spark infrastructure costs

    → Consider alternatives when:

    • Dealing with datasets under 1TB
    • In early stages of AI development
    • Budget constraints limit infrastructure spending
    • Simpler tools like Pandas or DuckDB suffice

    Spark isn't always necessary. Evaluate your specific needs and resources before committing to a Spark-based solution for AI data processing.

    In today’s episode of How AI Is Built, Abhishek and I discuss data processing:

    • When to use Spark vs. alternatives for data processing
    • Key components of Spark: RDDs, DataFrames, and SQL
    • Integrating AI into data pipelines
    • Challenges with LLM latency and consistency
    • Data storage strategies for AI workloads
    • Orchestration tools for data pipelines
    • Tips for making LLMs more reliable in production

    Abhishek Choudhary:

    • LinkedIn
    • GitHub
    • X (Twitter)

    Nicolay Gerold:

    • ⁠LinkedIn⁠
    • ⁠X (Twitter)
    47 min
  • #015 Building AI Agents for the Enterprise, Agent Cost Controls, Seamless UX

    In this episode, Nicolay talks with Rahul Parundekar, founder of AI Hero, about the current state and future of AI agents. Drawing from over a decade of experience working on agent technology at companies like Toyota, Rahul emphasizes the importance of focusing on realistic, bounded use cases rather than chasing full autonomy.

    They dive into the key challenges, like effectively capturing expert workflows and decision processes, delivering seamless user experiences that integrate into existing routines, and managing costs through techniques like guardrails and optimized model choices. The conversation also explores potential new paradigms for agent interactions beyond just chat.

    Key Takeaways:

    • Agents need to focus on realistic use cases rather than trying to be fully autonomous. Enterprises are unlikely to allow agents full autonomy anytime soon.
    • Capturing the logic and workflows in the user's head is the key challenge. Shadowing experts and having them demonstrate workflows is more effective than asking them to document processes.
    • User experience is crucial - agents must integrate seamlessly into existing user workflows without major disruptions. Interfaces beyond just chat may be needed.
    • Cost control is important - techniques like guardrails, context windowing, model choice optimization, and dev vs production modes can help manage costs.
    • New paradigms beyond just chat could be powerful - e.g. workflow specification, state/declarative definition of desired end-state.
    • Prompt engineering and dynamic prompt improvement based on feedback remain an open challenge.

    Key Quotes:

    • "Empowering users to create their own workflows is essential for effective agent usage."
    • "Capturing workflows accurately is a significant challenge in agent development."
    • "Preferences, right? So a lot of the work becomes like, hey, can you do preference learning for this user so that the next time the user doesn't have to enter the same information again, things like that."

    Rahul Parundekar:

    • AI Hero
    • AI Hero Docs

    Nicolay Gerold:

    • ⁠LinkedIn⁠
    • ⁠X (Twitter)

    00:00 Exploring the Potential of Autonomous Agents

    02:23 Challenges of Accuracy and Repeatability in Agents

    08:31 Capturing User Workflows and Improving Prompts

    13:37 Tech Stack for Implementing Agents in the Enterprise

    agent development, determinism, user experience, agent paradigms, private use, human-agent interaction, user workflows, agent deployment, human-in-the-loop, LLMs, declarative ways, scalability, AI Hero

    36 min
  • #014 Building Predictable Agents through Prompting, Compression, and Memory Strategies

    In this conversation, Nicolay and Richmond Alake discuss various topics related to building AI agents and using MongoDB in the AI space. They cover the use of agents and multi-agents, the challenges of controlling agent behavior, and the importance of prompt compression.

    When you are building agents. Build them iteratively. Start with simple LLM calls before moving to multi-agent systems.

    Main Takeaways:

    1. Prompt Compression: Using techniques like prompt compression can significantly reduce the cost of running LLM-based applications by reducing the number of tokens sent to the model. This becomes crucial when scaling to production.
    2. Memory Management: Effective memory management is key for building reliable agents. Consider different memory components like long-term memory (knowledge base), short-term memory (conversation history), semantic cache, and operational data (system logs). Store each in separate collections for easy access and reference.
    3. Performance Optimization: Optimize performance across multiple dimensions - output quality (by tuning context and knowledge base), latency (using semantic caching), and scalability (using auto-scaling databases like MongoDB).
    4. Prompting Techniques: Leverage prompting techniques like ReAct (observe, plan, act) and structured prompts (JSON, pseudo-code) to improve agent predictability and output quality.
    5. Experimentation: Continuous experimentation is crucial in this rapidly evolving field. Try different frameworks (LangChain, Crew AI, Haystack), models (Claude, Anthropic, open-source), and techniques to find the best fit for your use case.

    Richmond Alake:

    • LinkedIn
    • Medium
    • Find Richmond on MongoDB
    • X (Twitter)
    • YouTube
    • GenAI Showcase MongoDB
    • MongoDB AI Stack

    Nicolay Gerold:

    • ⁠LinkedIn⁠
    • ⁠X (Twitter)

    00:00 Reducing the Scope of AI Agents

    01:55 Seamless Data Ingestion 

    03:20 Challenges and Considerations in Implementing Multi-Agents

    06:05 Memory Modeling for Robust Agents with MongoDB

    15:05 Performance Optimization in AI Agents

    18:19 RAG Setup

    AI agents, multi-agents, prompt compression, MongoDB, data storage, data ingestion, performance optimization, tooling, generative AI

    33 min
  • Data Integration and Ingestion for AI & LLMs, Architecting Data Flows | changelog 3

    In this episode, Kirk Marple, CEO and founder of Graphlit, shares his expertise on building efficient data integrations.

    Kirk breaks down his approach using relatable concepts:

    1. The "Two-Sided Funnel": This model streamlines data flow by converting various data sources into a standard format before distributing it.
    2. Universal Data Streams: Kirk explains how he transforms diverse data into a single, manageable stream of information.
    3. Parallel Processing: Learn about the "competing consumer model" that allows for faster data handling.
    4. Building Blocks for Success: Discover the importance of well-defined interfaces and actor models in creating robust data systems.
    5. Tech Talk: Kirk discusses data normalization techniques and the potential shift towards a more streamlined "Kappa architecture."
    6. Reusable Patterns: Find out how Kirk's methods can speed up the integration of new data sources.
    7. Kirk Marple:

      • LinkedIn
      • X (Twitter)
      • Graphlit
      • Graphlit Docs
      • Nicolay Gerold:

        • ⁠LinkedIn⁠
        • ⁠X (Twitter)
        • Chapters

          00:00 Building Integrations into Different Tools

          00:44 The Two-Sided Funnel Model for Data Flow

          04:07 Using Well-Defined Interfaces for Faster Integration

          04:36 Managing Feeds and State with Actor Models

          06:05 The Importance of Data Normalization

          10:54 Tech Stack for Data Flow

          11:52 Progression towards a Kappa Architecture

          13:45 Reusability of Patterns for Faster Integration

          data integration, data sources, data flow, two-sided funnel model, canonical format, stream of ingestible objects, competing consumer model, well-defined interfaces, actor model, data normalization, tech stack, Kappa architecture, reusability of patterns

          15 min
        • #013 ETL for LLMs, Integrating and Normalizing Unstructured Data

          In our latest episode, we sit down with Derek Tu, Founder and CEO of Carbon, a cutting-edge ETL tool designed specifically for large language models (LLMs).

          Carbon is streamlining AI development by providing a platform for integrating unstructured data from various sources, enabling businesses to build innovative AI applications more efficiently while addressing data privacy and ethical concerns.

          • "I think people are trying to optimize around the chunking strategy... But for me, that seems a bit maybe not focusing on the right area of optimization. These embedding models themselves have gone just like, so much more advanced over the past five to 10 years that regardless of what representation you're passing in, they do a pretty good job of being able to understand that information semantically and returning the relevant chunks." - Derek Tu on the importance of embedding models over chunking strategies
          • "If you are cost conscious and if you're worried about performance, I would definitely look at quantizing your embeddings. I think we've probably been able to, I don't have like the exact numbers here, but I think we might be saving at least half, right, in storage costs by quantizing everything." - Derek Tu on optimizing costs and performance with vector databases

          Derek Tu:

          • LinkedIn
          • Carbon

          Nicolay Gerold:

          • ⁠LinkedIn⁠
          • ⁠X (Twitter)

          Key Takeaways:

          • Understand your data sources: Before building your ETL pipeline, thoroughly assess the various data sources you'll be working with, such as Slack, Email, Google Docs, and more. Consider the unique characteristics of each source, including data format, structure, and metadata.
          • Normalize and preprocess data: Develop strategies to normalize and preprocess the unstructured data from different sources. This may involve parsing, cleaning, and transforming the data into a standardized format that can be easily consumed by your AI models.
          • Experiment with chunking strategies: While there's no one-size-fits-all approach to chunking, it's essential to experiment with different strategies to find what works best for your specific use case. Consider factors like data format, structure, and the desired granularity of the chunks.
          • Leverage metadata and tagging: Metadata and tagging can play a crucial role in organizing and retrieving relevant data for your AI models. Implement mechanisms to capture and store important metadata, such as document types, topics, and timestamps, and consider using AI-powered tagging to automatically categorize your data.
          • Choose the right embedding model: Embedding models have advanced significantly in recent years, so focus on selecting the right model for your needs rather than over-optimizing chunking strategies. Consider factors like model performance, dimensionality, and compatibility with your data types.
          • Optimize vector database usage: When working with vector databases, consider techniques like quantization to reduce storage costs and improve performance. Experiment with different configurations and settings to find the optimal balance for your specific use case.

          00:00 Introduction and Optimizing Embedding Models

          03:00 The Evolution of Carbon and Focus on Unstructured Data

          06:19 Customer Progression and Target Group

          09:43 Interesting Use Cases and Handling Different Data Representations

          13:30 Chunking Strategies and Normalization

          20:14 Approach to Chunking and Choosing a Vector Database

          23:06 Tech Stack and Recommended Tools

          28:19 Future of Carbon: Multimodal Models and Building a Platform

          Carbon, LLMs, RAG, chunking, data processing, global customer base, GDPR compliance, AI founders, AI agents, enterprises

          37 min
        • #012 Serverless Data Orchestration, AI in the Data Stack, AI Pipelines

          In this episode, Nicolay sits down with Hugo Lu, founder and CEO of Orchestra, a modern data orchestration platform. As data pipelines and analytics workflows become increasingly complex, spanning multiple teams, tools and cloud services, the need for unified orchestration and visibility has never been greater.

          Orchestra is a serverless data orchestration tool that aims to provide a unified control plane for managing data pipelines, infrastructure, and analytics across an organization's modern data stack.

          The core architecture involves users building pipelines as code which then run on Orchestra's serverless infrastructure. It can orchestrate tasks like data ingestion, transformation, AI calls, as well as monitoring and getting analytics on data products. All with end-to-end visibility, data lineage and governance even when organizations have a scattered, modular data architecture across teams and tools.

          Key Quotes:

          • Find the right level of abstraction when building data orchestration tasks/workflows. "I think the right level of abstraction is always good. I think like Prefect do this really well, right? Their big sell was, just put a decorator on a function and it becomes a task. That is a great idea. You know, just make tasks modular and have them do all the boilerplate stuff like error logging, monitoring of data, all of that stuff.”
          • Modularize data pipeline components: "It's just around understanding what that dev workflow should look like. I think it should be a bit more modular." Having a modular architecture where different components like data ingestion, transformation, model training are decoupled allows better flexibility and scalability.
          • Adopt a streaming/event-driven architecture for low-latency AI use cases: "If you've got an event-driven architecture, then, you know, that's not what you use an orchestration tool for...if you're having a conversation with a chatbot, like, you know, you're sending messages, you're sending events, you're getting a response back. That I would argue should be dealt with by microservices."

          Hugo Lu:

          • LinkedIn
          • Newsletter
          • Orchestra
          • Orchestra Docs

          Nicolay Gerold:

          • ⁠LinkedIn⁠
          • ⁠X (Twitter)

          00:00 Introduction to Orchestra and its Focus on Data Products

          08:03 Unified Control Plane for Data Stack and End-to-End Control

          14:42 Use Cases and Unique Applications of Orchestra

          19:31 Retaining Existing Dev Workflows and Best Practices in Orchestra

          22:23 Event-Driven Architectures and Monitoring in Orchestra

          23:49 Putting Data Products First and Monitoring Health and Usage

          25:40 The Future of Data Orchestration: Stream-Based and Cost-Effective

          data orchestration, Orchestra, serverless architecture, versatility, use cases, maturity levels, challenges, AI workloads

          29 min

        About How AI Is Built

        From the publisher's feed

        Real engineers. Real deployments. Zero hype. We interview the top engineers who actually put AI in production. Learn what the best engineers have figured out through years of experience. Hosted by…