In this episode of Neural Search Talks, Andrew Yates (Assistant Prof at the University of Amsterdam) Sergi Castella (Analyst at Zeta Alpha), and Gabriel Bénédict (PhD student at the University of Amsterdam) discuss the prospect of using GPT-like models as a replacement for conventional search engines.

References

Timestamps:

Andrew Yates (Assistant Prof at University of Amsterdam) and Sergi Castella (Analyst at Zeta Alpha) discuss the paper "Task-aware Retrieval with Instructions" by Akari Asai et al. This paper proposes to augment a conglomerate of existing retrieval and NLP datasets with natural language instructions (BERRI, Bank of Explicit RetRieval Instructions) and use it to train TART (Multi-task Instructed Retriever).  

📄 Paper: https://arxiv.org/abs/2211.09260

🍻 BEIR benchmark: https://arxiv.org/abs/2104.08663

📈 LOTTE (Long-Tail Topic-stratified Evaluation, introduced in ColBERT v2): https://arxiv.org/abs/2112.01488

Timestamps: 

00:00 Intro: "Task-aware Retrieval with Instructions"

02:20 BERRI, TART, X^2 evaluation

04:00 Background: recent works in domain adaptation

06:50 Instruction Tuning 08:50 Retrieval with descriptions

11:30 Retrieval with instructions

17:28 BERRI, Bank of Explicit RetRieval Instructions

21:48 Repurposing NLP tasks as retrieval tasks

23:53 Negative document selection

27:47 TART, Multi-task Instructed Retriever

31:50 Evaluation: Zero-shot and X^2 evaluation

39:20 Results on Table 3 (BEIR, LOTTE)

50:30 Results on Table 4 (X^2-Retrieval)

55:50 Ablations

57:17 Discussion: user modeling, future work, scale

Marzieh Fadaee — NLP Research Lead at Zeta Alpha — joins Andrew Yates and Sergi Castella to chat about her work in using large Language Models like GPT-3 to generate domain-specific training data for retrieval models with little-to-no human input. The two papers discussed are "InPars: Data Augmentation for Information Retrieval using Large Language Models" and "Promptagator: Few-shot Dense Retrieval From 8 Examples".

InPars: https://arxiv.org/abs/2202.05144

Promptagator: https://arxiv.org/abs/2209.11755

Timestamps:

00:00 Introduction

02:00 Background and journey of Marzieh Fadaee

03:10 Challenges of leveraging Large LMs in Information Retrieval

05:20 InPars, motivation and method

14:30 Vanilla vs GBQ prompting

24:40 Evaluation and Benchmark

26:30 Baselines

27:40 Main results and takeaways (Table 1, InPars)

35:40 Ablations: prompting, in-domain vs. MSMARCO input documents

40:40 Promptagator overview and main differences with InPars

48:40 Retriever training and filtering in Promptagator

54:37 Main Results (Table 2, Promptagator)

1:02:30 Ablations on consistency filtering (Figure 2, Promptagator)

1:07:39 Is this the magic black-box pipeline for neural retrieval on any documents

1:11:14 Limitations of using LMs for synthetic data

1:13:00 Future directions for this line of research

Andrew Yates (Assistant Professor at the University of Amsterdam) and Sergi Castella (Analyst at Zeta Alpha) discus the two influential papers introducing ColBERT (from 2020) and ColBERT v2 (from 2022), which mainly propose a fast late interaction operation to achieve a performance close to full cross-encoders but at a more manageable computational cost at inference; along with many other optimizations.

📄 ColBERT: "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT" by Omar Khattab and Matei Zaharia. https://arxiv.org/abs/2004.12832

📄 ColBERTv2: "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction" by Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. https://arxiv.org/abs/2112.01488

📄 PLAID: "An Efficient Engine for Late Interaction Retrieval" by Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. https://arxiv.org/abs/2205.09707

📄 CEDR: "CEDR: Contextualized Embeddings for Document Ranking" by Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. https://arxiv.org/abs/1904.07094

🪃 Feedback form: https://scastella.typeform.com/to/rg7a5GfJ

Timestamps:

00:00 Introduction

00:42 Why ColBERT?

03:34 Retrieval paradigms recap

08:04 ColBERT query formulation and architecture

09:04 Using ColBERT as a reranker or as an end-to-end retriever

11:28 Space Footprint vs. MRR on MS MARCO

12:24 Methodology: datasets and negative sampling

14:37 Terminology for cross encoders, interaction-based models, etc.

16:12 Results (ColBERT v1) on MS MARCO

18:41 Ablations on model components

20:34 Max pooling vs. mean pooling

22:54 Why did ColBERT have a big impact?

26:31 ColBERTv2: knowledge distillation

29:34 ColBERTv2: indexing improvements

33:59 Effects of clustering compression in performance

35:19 Results (ColBERT v2): MS MARCO

38:54 Results (ColBERT v2): BEIR

41:27 Takeaway: strong specially in out-of-domain evaluation

43:59 Qualitatively how do ColBERT scores look like?

46:21 What's the most promising of all current neural IR paradigms

49:34 How come there's still so much interest in Dense retrieval?

51:09 Many to many similarity at different granularities

53:44 What would ColBERT v3 include?

56:39 PLAID: An Efficient Engine for Late Interaction Retrieval

Contact: [email protected]

How much of the training and test sets in TREC or MS Marco overlap? Can we evaluate on different splits of the data to isolate the extrapolation performance?

In this episode of Neural Information Retrieval Talks, Andrew Yates and Sergi Castella i Sapé discuss the paper "Evaluating Extrapolation Performance of Dense Retrieval" byJingtao Zhan, Xiaohui Xie, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma.

📄 Paper: https://arxiv.org/abs/2204.11447

❓ About MS Marco: https://microsoft.github.io/msmarco/

❓About TREC: https://trec.nist.gov/

🪃 Feedback form: https://scastella.typeform.com/to/rg7a5GfJ  

Timestamps: 

00:00 Introduction 

01:08 Evaluation in Information Retrieval, why is it exciting 

07:40 Extrapolation Performance in Dense Retrieval 

10:30 Learning in High Dimension Always Amounts to Extrapolation 

11:40 3 Research questions 

16:18 Defining Train-Test label overlap: entity and query intent overlap 

21:00 Train-test Overlap in existing benchmarks TREC 

23:29 Resampling evaluation methods: constructing distinct train-test sets 

25:37 Baselines and results: ColBERT, SPLADE

29:36 Table 6: interpolation vs. extrapolation performance in TREC 

33:06 Table 7: interplation vs. extrapolation in MS Marco 

35:55 Table 8: Comparing different DR training approaches 

40:00 Research Question 1 resolved: cross encoders are more robust than dense retrieval in extrapolation 

42:00 Extrapolation and Domain Transfer: BEIR benchmark. 

44:46 Figure 2: correlation between extrapolation performance and domain transfer performance 

48:35 Broad strokes takeaways from this work 

52:30 Is there any intuition behind the results where Dense Retrieval generalizes worse than Cross Encoders? 

56:14 Will this have an impact on the IR benchmarking culture? 

57:40 Outro   

Contact: [email protected]

Andrew Yates (Assistant Professor at the University of Amsterdam) and Sergi Castella i Sapé discuss the recent "Open Pre-trained Transformer (OPT) Language Models" from Meta AI (formerly Facebook). In this replication work, Meta developed and trained a 175 Billion parameter Transformer very similar to GPT-3 from OpenAI, documenting the process in detail to share their findings with the community. The code, pretrained weights, and logbook are available on their Github repository (links below). 

Links 

❓Feedback Form: https://scastella.typeform.com/to/rg7a5GfJ

📄 OPT paper: https://arxiv.org/abs/2205.01068

👾 Code: https://github.com/facebookresearch/metaseq

📒 Logbook: https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf

✍️ OPT Official Blog Post: https://ai.facebook.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/  

OpenAI Embeddings API: https://openai.com/blog/introducing-text-and-code-embeddings/

Nils Reimers' critique of OpenAI Embeddings API: https://medium.com/@nils_reimers/openai-gpt-3-text-embeddings-really-a-new-state-of-the-art-in-dense-text-embeddings-6571fe3ec9d9 

Timestamps: 

00:00 Introduction and housekeeping: new feedback form, ACL conference highlights 

02:42 The convergence between NLP and Neural IR techniques 

06:43 Open Pretrained Transformer motivation and scope, reproducing GPT-3 and open-sourcing 

08:16 Basics of OPT: architecture, pre-training objective, teacher forcing, tokenizer, training data 

13:40 Preliminary experiments findings: hyperparameters, training stability, spikiness 

20:08 Problems that appear at scale when training with 992 GPUs

23:01 Using temperature to check whether GPUs are working

25:00 Training the largest model: what to do when the loss explodes? (which happens quite often)

29:15 When they switched away from AdamW to SGD

32:00 Results: successful but not quite GPT-3 level.

Toxicity? 35:45 Replicability of Large Language Models research. Was GPT-3 replicable? What difference does it make?

37:25 What makes a paper replicable?

40:33 Directions in which large Language Models are applied to Information Retrieval

45:15 Final thoughts and takeaways

We discuss Conversational Search with our usual cohosts Andrew Yates and Sergi Castella i Sapé; along with a special guest Antonios Minas Krasakis, PhD candidate at the University of Amsterdam. 

We center our discussion around the ConvDR paper: "Few-Shot Conversational Dense Retrieval" by Shi Yu et al. which was the first work to perform Conversational Search without an explicit conversation to query rewriting step.

Timestamps:

00:00 Introduction

00:50 Conversational AI and Conversational Search

05:40 What makes Conversational Search challenging

07:00 ConvDR paper introduction

10:10 Passage representations

11:30 Conversation representations: query rewriting

19:12 ConvDR novel proposed method: teacher-student setup with ANCE

22:50 Datasets and benchmarks: CAsT, CANARD

25:32 Teacher-student advantages and knowledge distillation vs. ranking loss functions

28:09 TREC CAsT and OR-QuAC

35:50 Metrics: MRR, NDCG, holes@10

44:16 Main Results on CAsT and OR-QuAC (Table 2)

57:35 Ablations on combinations of loss functions (Table 4)

1:00:10 How fast is ConvDR? (Table 3)

1:02:40 Qualitative analysis on ConvDR embeddings (Figure 4)

1:04:50 How has this work aged? More recent works in similar directions: Contextualized Quesy Embeddings for Conversational Search.

1:07:02 Is "end-to-end" the silver-bullet for Conversational Search?

1:10:04 Will conversational search become more mainstream?

1:18:44 Latest initiatives for Conversational Search

Andrew Yates and Sergi Castella discuss the paper titled "Transformer Memory as a Differentiable Search Index" by Yi Tay et al at Google. This work proposes a new approach to document retrieval in which document ids are memorized by a transformer during training (or "indexing") and for retrieval, a query is fed to the model, which then generates autoregressively relevant doc ids for that query.

Paper: https://arxiv.org/abs/2202.06991

Timestamps:

00:00 Intro: Transformer memory as a Differentiable Search Index (DSI)

01:15 The gist of the paper, motivation

4:20 Related work: Autoregressive Entity Linking

7:38 What is an index? Conventional vs. "differentiable"

10:20 Indexing and Retrieval definitions in the context of the DSI

12:40 Learning representations for documents

17:20 How to represent document ids: atomic, string, semantically relevant

22:00 Zero-shot vs. finetuned settings

24:10 Datasets and baselines

27:08 Dinetuned results

36:40 Zero-shot results

43:50 Ablation results

47:15 Where could this model be useds?

52:00 Is memory efficiency a fundamental problem of this approach?

55:14 What about semantically relevant doc ids?

60:30 Closing remarks 

Contact: [email protected]

In this third episode of the Neural Information Retrieval Talks podcast, Andrew Yates and Sergi Castella discuss the paper "Learning to Retrieve Passages without Supervision" by Ori Ram et al.  

Despite the massive advances in Neural Information Retrieval in the past few years, statistical models still overperform neural models when no annotations are available at all. This paper proposes a new self-supervised pertaining task for Dense Information Retrieval that manages to beat BM25 on some benchmarks without using any label.  

Paper: https://arxiv.org/abs/2112.07708 

Timestamps:

00:00 Introduction

00:36 "Learning to Retrieve Passages Without Supervision"

02:20 Open Domain Question Answering

05:05 Related work: Families of Retrieval Models

08:30 Contrastive Learning

11:18 Siamese Networks, Bi-Encoders and Dual-Encoders

13:33 Choosing Negative Samples

17:46 Self supervision: how to train IR models without labels.

21:31 The modern recipe for SOTA Retrieval Models

23:50 Methodology: a new proposed self supervision task

26:40 Datasets, metrics and baselines

\33:50 Results: Zero-Shot performance

43:07 Results: Few-shot performance

47:15 Practically, is not using labels relevant after all?

51:37 How would you "break" the Spider model?

53:23 How long until Neural IR models outperform BM25 out-of-the-box robustly?

54:50 Models as a service: OpenAI's text embeddings API

Contact: [email protected]

We discuss the Information Retrieval publication "The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes" by Nils Reimers and Iryna Gurevych, which explores how Dense Passage Retrieval performance degrades as the index size varies and how it compares to traditional sparse or keyword-based methods.

Timestamps:

00:00 Co-host introduction

00:26 Paper introduction

02:18 Dense vs. Sparse retrieval

05:46 Theoretical analysis of false positives(1)

08:17 What is low vs. high dimensional representations

11:49 Theoretical analysis o false positives (2)

20:10 First results: growing the MS-Marco index

28:35 Adding random strings to the index

39:17 Discussion, takeaways

44:26 Will dense retrieval replace or coexist with sparse methods?

50:50 Sparse, Dense and Attentional Representations for Text Retrieval

Referenced work:

Sparse, Dense and Attentional Representations for Text Retrieval by Yi Luan et al. 2020.