summaries: This paper studies pre-training of Siamese Transformer encoders for the zero-shot dense retrieval problem. The author consider contrastive learning loss function to optimize the encoders and investigate variants of ICT for data augmentations. While not much novelty in the techincal sides, the author indeed present solid empirical study on zero-shot retrieval benchmark as well as extension to the few-shot evaluation. This paper proposes to use cropping strategy to train a neural model for IR. It is compared to BM25 and several other neural models, including BERT and the ones using inverse cloze task. The goal of cropping is to train a model that fits better the IR tasks. It is hypothesized that this task is more similar to IR than other pre-training strategies. The method is tested on a set of benchmarks of zero-shot retrieval and MSMARCO with training data. It is shown than the proposed method fine-tuned on MSMARCO can outperform the compared methods, including BM25. When no fine-tuning is used, the method is almost at par with BM25.  This paper presents a contrastive learning approach for unsupervised text retrieval. As the approach is unsupervised, similar to the Inverse Cloze Task (ICT) approach, the authors propose to do independent cropping of a paragraph to generate a pair of "pseudo-question" and "pseudo-document", which is used to train the model. The model is trained using contrastive learning following the MoCo algorithm. Experiments are performed on the recent BEIR benchmark for zero-shot and few-shot retrieval evaluation. The proposed approach often performs better than the BM25 algorithm on zero-shot retrieval on the BEIR benchmark. There are also some ablations presented to analyze different aspects of model training such as the effect of number of negative examples, training data, data augmentations etc.   This paper proposed a contrastive learning-based method for zero-shot dense IR. Based on the contrastive learning framework, 3 positive pair construction methods and 2 negative pair construction methods are used and compared. Evaluations are performed on the BEIR benchmark, with the best average result achieved compared to a list of IR (both sparse and dense) baselines in the zero-shot setting.
abstractText: Information retrieval is an important component in natural language processing, for knowledge intensive tasks such as question answering and fact checking. Recently, information retrieval has seen the emergence of dense retrievers, based on neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new domains or applications with no training data, and are often outperformed by term-frequency methods such as BM25 which are not supervised. Thus, a natural question is whether it is possible to train dense retrievers without supervision. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers, and show that it leads to strong retrieval performance. More precisely, we show on the BEIR benchmark that our model outperforms BM25 on 8 out of 14 datasets. Furthermore, when a few thousands examples are available, we show that fine-tuning our model on these leads to strong improvements compared to BM25. Finally, when used as pre-training before fine-tuning on the MS MARCO dataset, our technique obtains state-of-the-art results on the BEIR benchmark.
authors.name: TRIEVAL WITH
authors.name: CONTRASTIVE LEARNING
references.authors: Payal Bajaj Daniel Campos Nick Craswell Li Deng Jianfeng Gao Xiaodong Liu Rangan Majumder Andrew McNamara Bhaskar Mitra Tri Nguyen
references.title: Ms marco: A human generated machine reading comprehension dataset
references.venue: arXiv preprint arXiv:1611.09268,
references.authors: Adam Berger Rich Caruana David Cohn Dayne Freitag Vibhu Mittal
references.title: Bridging the lexical chasm: statistical approaches to answer-finding
references.venue: In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval,
references.authors: Mathilde Caron Ishan Misra Julien Mairal Priya Goyal Piotr Bojanowski Armand Joulin
references.title: Unsupervised learning of visual features by contrasting cluster assignments
references.venue: arXiv preprint arXiv:2006.09882,
references.authors: Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin
references.title: Emerging properties in self-supervised vision transformers
references.venue: arXiv preprint arXiv:2104.14294,
references.authors: Wei-Cheng Chang Felix X Yu Yin-Wen Chang Yiming Yang Sanjiv Kumar
references.title: Pre-training tasks for embedding-based large-scale retrieval
references.venue: arXiv preprint arXiv:2002.03932,
references.authors: Danqi Chen Adam Fisch Jason Weston Antoine Bordes
references.title: Reading Wikipedia to answer open-domain questions
references.venue: In Proc. ACL,
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey Hinton
references.title: A simple framework for contrastive learning of visual representations
references.venue: In International conference on machine learning,
references.authors: Scott Deerwester Susan T Dumais George W Furnas Thomas K Landauer Richard Harshman
references.title: Indexing by latent semantic analysis
references.venue: Journal of the American society for information science,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: BERT: Pre-training of deep bidirectional transformers for language understanding
references.venue: In Proc. NAACL,
references.authors: Hongchao Fang Sicheng Wang Meng Zhou Jiayuan Ding Pengtao Xie
references.title: Cert: Contrastive self-supervised learning for language understanding
references.venue: arXiv preprint arXiv:2005.12766,
references.authors: Luyu Gao Jamie Callan
references.title: Unsupervised corpus aware language model pre-training for dense passage retrieval
references.venue: arXiv preprint arXiv:2108.05540,
references.authors: Daniel Gillick Alessandro Presta Gaurav Singh Tomar
references.title: End-to-end retrieval in continuous space
references.venue: arXiv preprint arXiv:1811.08008,
references.authors: John M Giorgi Osvald Nitski Gary D Bader Bo Wang
references.title: Declutr: Deep contrastive learning for unsupervised textual representations
references.venue: arXiv preprint arXiv:2006.03659,
references.authors: Kelvin Guu Kenton Lee Zora Tung Panupong Pasupat Ming-Wei Chang
references.title: Realm: Retrievalaugmented language model pre-training
references.venue: arXiv preprint arXiv:2002.08909,
references.authors: Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross Girshick
references.title: Momentum contrast for unsupervised visual representation learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Sebastian Hofstätter Sheng-Chieh Lin Jheng-Hong Yang Jimmy Lin Allan Hanbury
references.title: Efficiently teaching an effective dense retriever with balanced topic aware sampling
references.venue: arXiv preprint arXiv:2104.06967,
references.authors: Po-Sen Huang Xiaodong He Jianfeng Gao Li Deng Alex Acero Larry Heck
references.title: Learning deep structured semantic models for web search using clickthrough data
references.venue: In Proceedings of the 22nd ACM international conference on Information & Knowledge Management,
references.authors: Samuel Humeau Kurt Shuster Marie-Anne Lachaux Jason Weston
references.title: Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring
references.authors: Gautier Izacard Edouard Grave
references.title: Leveraging passage retrieval with generative models for open domain question answering
references.venue: arXiv preprint arXiv:2007.01282,
references.authors: Gautier Izacard Edouard Grave
references.title: Distilling knowledge from reader to retriever for question answering
references.venue: In International Conference on Learning Representations,
references.authors: Gautier Izacard Fabio Petroni Lucas Hosseini Nicola De Cao Sebastian Riedel Edouard Grave
references.title: A memory efficient baseline for open domain question answering
references.venue: arXiv preprint arXiv:2012.15156,
references.authors: Yacine Jernite Samuel R Bowman David Sontag
references.title: Discourse-based objectives for fast unsupervised sentence representation learning
references.venue: arXiv preprint arXiv:1705.00557,
references.authors: Jeff Johnson Matthijs Douze Hervé Jégou
references.title: Billion-scale similarity search with gpus
references.venue: IEEE Transactions on Big Data,
references.authors: Karen Sparck Jones
references.title: A statistical interpretation of term specificity and its application in retrieval
references.venue: Journal of documentation,
references.authors: Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer
references.title: Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension
references.venue: In Proc. ACL,
references.authors: Vladimir Karpukhin Barlas Oğuz Sewon Min Ledell Wu Sergey Edunov Danqi Chen Wen-tau Yih
references.title: Dense passage retrieval for open-domain question answering
references.venue: arXiv preprint arXiv:2004.04906,
references.authors: Omar Khattab Christopher Potts Matei Zaharia
references.title: Relevance-guided supervision for openqa with colbert
references.authors: Tom Kwiatkowski Jennimaria Palomaki Olivia Redfield Michael Collins Ankur Parikh Chris Alberti Danielle Epstein Illia Polosukhin Jacob Devlin Kenton Lee
references.title: Natural questions: a benchmark for question answering research
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Jungmin Kwon Jeongseop Kim Hyunseo Park In Kwon Choi
references.title: ASAM: adaptive sharpnessaware minimization for scale-invariant learning of deep neural networks
references.venue: arXiv preprint arXiv:2102.11600,
references.authors: Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut
references.title: Albert: A lite bert for self-supervised learning of language representations
references.authors: Kenton Lee Ming-Wei Chang Kristina Toutanova
references.title: Latent retrieval for weakly supervised open domain question answering
references.venue: In Proc. ACL,
references.authors: Mike Lewis Marjan Ghazvininejad Gargi Ghosh Armen Aghajanyan Sida Wang Luke Zettlemoyer
references.title: Pre-training via paraphrasing
references.venue: arXiv preprint arXiv:2006.15020,
references.authors: Jimmy Lin Rodrigo Nogueira Andrew Yates
references.title: Pretrained transformers for text ranking: Bert and beyond
references.venue: arXiv preprint arXiv:2010.06467,
references.authors: Ilya Loshchilov Frank Hutter
references.title: Decoupled weight decay regularization, 2019
references.authors: Yi Luan Jacob Eisenstein Kristina Toutanova Michael Collins
references.title: Sparse, dense, and attentional representations for text retrieval
references.venue: arXiv preprint arXiv:2005.00181,
references.authors: Xueguang Ma Kai Sun Ronak Pradeep Jimmy Lin
references.title: A replication study of dense passage retriever
references.authors: Christopher D Manning Hinrich Schütze Prabhakar Raghavan
references.title: Introduction to information retrieval
references.venue: Cambridge university press,
references.authors: Yu Meng Chenyan Xiong Payal Bajaj Saurabh Tiwary Paul Bennett Jiawei Han Xia Song
references.title: Coco-lm: Correcting and contrasting text sequences for language model pretraining
references.venue: arXiv preprint arXiv:2102.08473,
references.authors: Tomas Mikolov Ilya Sutskever Kai Chen Greg S Corrado Jeff Dean
references.title: Distributed representations of words and phrases and their compositionality
references.venue: In Advances in neural information processing systems,
references.authors: Bhaskar Mitra Nick Craswell
references.title: An introduction to neural information retrieval
references.venue: Foundations and Trends® in Information Retrieval,
references.authors: Tri Nguyen Mir Rosenberg Xia Song Jianfeng Gao Saurabh Tiwary Rangan Majumder Li Deng
references.title: Ms marco: A human generated machine reading comprehension
references.authors: Rodrigo Nogueira Kyunghyun Cho
references.title: Passage re-ranking with bert
references.venue: arXiv preprint arXiv:1901.04085,
references.authors: Hamid Palangi Li Deng Yelong Shen Jianfeng Gao Xiaodong He Jianshu Chen Xinying Song Rabab Ward
references.title: Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J Liu
references.title: Exploring the limits of transfer learning with a unified text-to-text transformer
references.venue: arXiv preprint arXiv:1910.10683,
references.authors: Nils Reimers Iryna Gurevych
references.title: Sentence-bert: Sentence embeddings using siamese bert-networks
references.venue: arXiv preprint arXiv:1908.10084,
references.authors: Stephen Robertson Hugo Zaragoza
references.title: The probabilistic relevance framework: BM25 and beyond
references.venue: Now Publishers Inc,
references.authors: Stephen E Robertson Steve Walker Susan Jones Micheline M Hancock-Beaulieu Mike Gatford
references.title: Okapi at TREC-3
references.venue: NIST Special Publication Sp,
references.authors: Devendra Singh Sachan Mostofa Patwary Mohammad Shoeybi Neel Kant Wei Ping William L Hamilton Bryan Catanzaro
references.title: End-to-end training of neural retrievers for open-domain question answering
references.venue: arXiv preprint arXiv:2101.00408,
references.authors: Devendra Singh Sachan Siva Reddy William Hamilton Chris Dyer Dani Yogatama
references.title: End-to-end training of multi-document reader and retriever for open-domain question answering, 2021b
references.authors: Yelong Shen Xiaodong He Jianfeng Gao Li Deng Grégoire Mesnil
references.title: Learning semantic representations using convolutional neural networks for web search
references.venue: In Proceedings of the 23rd international conference on world wide web,
references.authors: Nandan Thakur Nils Reimers Andreas Rücklé Abhishek Srivastava Iryna Gurevych
references.title: Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models
references.venue: arXiv preprint arXiv:2104.08663,
references.authors: James Thorne Andreas Vlachos Christos Christodoulopoulos Arpit Mittal
references.title: Fever: a large-scale dataset for fact extraction and verification
references.venue: arXiv preprint arXiv:1803.05355,
references.authors: Guillaume Wenzek Marie-Anne Lachaux Alexis Conneau Vishrav Chaudhary Francisco Guzmán Armand Joulin Edouard Grave
references.title: CCNet: Extracting high quality monolingual datasets from web crawl data
references.venue: In Proceedings of the 12th Language Resources and Evaluation Conference,
references.authors: Zhirong Wu Yuanjun Xiong Stella X Yu Dahua Lin
references.title: Unsupervised feature learning via nonparametric instance discrimination
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Zhuofeng Wu Sinong Wang Jiatao Gu Madian Khabsa Fei Sun Hao Ma
references.title: Clear: Contrastive learning for sentence representation
references.venue: arXiv preprint arXiv:2012.15466,
references.authors: Lee Xiong Chenyan Xiong Ye Li Kwok-Fung Tang Jialin Liu Paul Bennett Junaid Ahmed Arnold Overwijk
references.title: Approximate nearest neighbor negative contrastive learning for dense text retrieval
references.authors: Sohee Yang Minjoon Seo
references.title: Is retriever merely an approximator of reader
references.venue: arXiv preprint arXiv:2010.10999,
references.authors: Tiancheng Zhao Xiaopeng Lu Kyusong Lee
references.title: Sparta: Efficient open-domain question answering via sparse transformer matching retrieval
references.venue: arXiv preprint arXiv:2009.13013,
sections.heading: 1 INTRODUCTION
sections.text: Document retrieval is the task of finding relevant documents in a large collection to answer specific queries. This is an important task by itself and a core component of many natural language processing (NLP) problems, such as open domain question answering (Chen et al., 2017) or fact checking (Thorne et al., 2018). Traditionally, retrieval systems, or retrievers, leverage lexical similarities to match queries and documents, using, for instance, TF-IDF or BM25 weighting (Robertson & Zaragoza, 2009). These approaches, based on exact matches between tokens of the queries and documents, suffer from the lexical gap and do not generalize well (Berger et al., 2000). By contrast, approaches based on neural networks allow learning beyond lexical similarities, resulting in state-ofthe-art performance on question answering benchmarks, such as MS-MARCO (Nguyen et al., 2016) or NaturalQuestion (Kwiatkowski et al., 2019).
  As training neural networks requires large amount of data, their strong retrieval results were possible for domains and applications where large training datasets are available. In the case of retrieval, creating these datasets requires manually matching queries to the relevant documents in the collection. This is hardly possible when the collection contains millions or billions of element, resulting in many scenarios where only a few in-domain examples, if any, are available. A potential solution is to train a dense retriever on a large retrieval dataset such as MS MARCO, and then apply it to new domains, a setting refered to as zero-shot. Unfortunately, in this setting dense retrievers are often outperformed by classical methods based on term-frequency, which do not require supervision (Thakur et al., 2021).
  Thus, a natural alternative to transfer learning is unsupervised learning, which raises the following question: is it possible to train dense retrievers without supervision, and match the performance of BM25? Training dense retrievers without supervision can be achieved by using a pretext task that approximates retrieval. Given a document, one can generate a synthetic query and then train the network to retrieve the original document, among many others, given the query. The inverse Cloze task (ICT), proposed by Lee et al. (2019) to pre-train retrievers, uses a given sentence as query and predict the context surrounding it. While showing promising results as pre-training (Chang et al., 2020; Sachan et al., 2021a), this approach still lags behind BM25 when used as a zero-shot retriever.
  ICT is strongly related to contrastive learning (Wu et al., 2018), which has been widely applied in computer vision (Chen et al., 2020; He et al., 2020). In particular, computer vision models trained with the latest contrastive learning methods lead to features well suited to retrieval (Caron et al., 2021). In this work, we thus propose to revisit how well contrastive learning performs to train dense retrievers without supervision. We want to investigate how much the recent developments from computer vision, such as MoCo, can improve dense retrievers. Here, our goal is not to develop new techniques, but to determine how far we can go by pushing existing methods to their limits.
  In this paper, we thus make the following contributions. First, we show that contrastive learning can lead to strong unsupervised retrievers: our model achieves Recall@100 results competitive with BM25 on most of the BEIR benchmark. Second, in a few-shot setting, we show that our model benefits from few training examples, and obtains better results than transfering models from large datasets such as MS MARCO. Third, when used as a pre-training method before fine-tuning on MS MARCO, our technique leads to state-of-the-art retrieval performance on the BEIR benchmark. Finally, we perform ablations to motivate our design choices, and show that cropping works better than the inverse Cloze task.
sections.heading: 2 RELATED WORK
sections.text: In this section, we briefly review relevant work in information retrieval, and application of machine learning to this problem. This is not an exhaustive review, and we refer the reader to Manning et al. (2008), Mitra et al. (2018) and Lin et al. (2020) for a more complete introduction to the field.
  Term-frequency based information retrieval. Historically, in information retrieval, documents and queries are represented as sparse vectors where each element of the vectors corresponds to a term of the vocabulary. Different weighing schemes have been proposed, to determine how important a particular term is to a document in a large dataset. One of the most used weighing scheme is known as TF-IDF, and is based on inverse document frequency, or term specificity (Jones, 1972). BM25, which is still widely used today, extends TF-IDF (Robertson et al., 1995). A well known limitation of these approaches is that they rely on exact match to retrieve documents. This lead to the introduction of latent semantic analysis (Deerwester et al., 1990), in which documents are represented as low dimensional dense vectors.
  Neural network based information retrieval. Following the successful application of deep learning methods to natural language processing, neural networks techniques were introduced for information retrieval. Huang et al. (2013) proposed a deep bag-of-words model, in which representations of queries and documents are computed independently. A relevance score is then obtained by taking the dot product between representations, and the model is trained end-to-end on click data from a search engine. This method was later refined by replacing the bag-of-words model by convolutional neural networks (Shen et al., 2014) or recurrent neural network (Palangi et al., 2016). A limitation of bi-encoders is that queries and documents are represented by a single vector, preventing the model to capture fine-grained interactions between terms. Nogueira & Cho (2019) thus introduced a cross-encoder model, based on a pre-trained BERT model (Devlin et al., 2019), which jointly encodes queries and documents. The application of a strong pre-trained model, as well as the cross-encoder architecture, lead to important improvement on the MS-MARCO benchmark (Bajaj et al., 2016).
  The methods described in the previous paragraph were applied to re-rank documents, which were retrieved with a traditional IR system such as BM25. Gillick et al. (2018) first studied whether continuous retrievers, based on bi-encoder neural models, could be viable alternative to re-ranking. In the context of question answering, Karpukhin et al. (2020) introduced a dense passage retriever (DPR) based on the bi-encoder architecture. This model is initialized with a BERT network, and trained discriminatively using pairs of queries and relevant documents, with hard negatives from BM25. Xiong et al. (2020) further extended this work by mining hard negatives with the model itself during optimization, and trained on the MS-MARCO dataset. Once a collection of documents, such as Wikipedia articles, is encoded, retrieval is performed with a fast k-nearest neighbors library such as FAISS Johnson et al. (2019). To alleviate the limitations of bi-encoders, Humeau et al. (2019) introduces the poly-encoder architecture, where documents are encoded by multiple vectors. Similarly, Khattab et al. (2020) proposes the ColBERT model, which keeps a vector representation for each term of the queries and documents. To make the retrieval tractable, the term-level function
  is approximated to first retrieve an initial set of candidates, which are then re-ranked with the true score. In the context of question answering, knowledge distillation has been used to train retrievers, either using the attention scores of the reader of the downstream task as synthetic labels (Izacard & Grave, 2021), or the relevance score from a cross encoder (Yang & Seo, 2020). Luan et al. (2020) compares, theoretically and empirically, the performance of sparse and dense retrievers, including bi-, cross- and poly-encoders. Dense retrievers, such as DPR, can lead to indices that weigh close to 100 GB when encoding document collections such as Wikipedia. Izacard et al. (2020) shows how to compress such indices, with limited impact on performance, making them more practical to use.
  Self-supervised learning for NLP. Following the success of word2vec (Mikolov et al., 2013), many self-supervised techniques have been proposed to learn representation of text. Here, we briefly review the ones that are most related to our approach: sentence level models and contrastive techniques. Jernite et al. (2017) introduced different objective function to learn sentence representations, including next sentence prediction and binary order prediction. These objectives were later used in pre-trained models based on transformers, such as BERT (Devlin et al., 2019) and AlBERT (Lan et al., 2019). In the context of retrieval, Lee et al. (2019) introduced the inverse cloze task (ICT), whose purpose is to predict the context surrounding a span of text. Guu et al. (2020) integrated a bi-encoder retriever model in a BERT pre-training scheme. The retrieved documents are used as additional context in the BERT task, and the whole system is trained end-to-end in an unsupervised way. Similarly, Lewis et al. (2020) proposed to jointly learn a retriever and a generative seq2seq model, using self-supervised training. Chang et al. (2020) compares different pre-training tasks for retrieval, including the inverse cloze task. Constrastive learning was introduced in computer vision by Wu et al. (2018), followed by several modifications to improve the training (He et al., 2020; Chen et al., 2020; Caron et al., 2020). In the context of natural language processing, Fang et al. (2020) proposed to apply MoCo where positive pairs of sentences are obtained using back-translation. Different works augmented the masked language modeling objective with a contrastive loss (Giorgi et al., 2020; Wu et al., 2020; Meng et al., 2021). Finally, SBERT (Reimers & Gurevych, 2019) uses a siamese network similar to contrastive learning to learn a BERT-like model that is adapted to matching sentence embeddings. Their formulation is similar to our work but requires aligned pairs of sentences to form positive pairs while we propose to use data augmentation to leverage large unaligned textual corpora. Concurrent to this work, Gao & Callan (2021) have also shown the potential of contrastive learning for information retrieval; building on the same observation that both tasks share a similar structure.
sections.heading: 3 METHOD
sections.text: In this section, we describe how to train a dense retriever with no supervision. We review the model architecture and then describe contrastive learning - a key component of its training.
  The objective of a retriever is to find relevant documents in a large collection for a given query. Thus, the retriever takes as input the set of documents and the query and outputs a relevance score for each document. A standard approach is to encode each query–document pair with a neural network (Nogueira & Cho, 2019). This procedure requires re-encoding every document for any new query and hence does not scale to large collections of documents. Instead, we propose to use a bi-encoder architecture, where documents and queries are encoded independently (Huang et al., 2013; Karpukhin et al., 2020). One can compute the relevance score by taking the dot product (or cosine similarity) between the document’s representation and the representation of the query. More precisely, given a pair of query q and document d, we encode each of them independently using the same model, fθ, parametrized by θ. The relevance score s(q, d) is then the dot product of the resulting representations:
  s(q, d) = 〈fθ(q), fθ(d)〉.
  In practice, we use a transformer network for fθ to embed both queries and documents. The representation fθ(q) (resp. fθ(d)) for a query (resp. document) is obtained by averaging the hidden representations of the last layer. Following previous work on dense retrieval with neural networks, we use the BERT base uncased architecture and refer the reader to Devlin et al. (2019) for more details.
sections.heading: 3.1 UNSUPERVISED TRAINING ON UNALIGNED DOCUMENTS
sections.text: In this section, we describe our unsupervised training pipeline. We briefly review the loss function traditionally used in contrastive learning. We then discuss obtaining positive pairs from a single text document, a critical ingredient for this training paradigm.
sections.heading: 3.1.1 CONTRASTIVE LEARNING
sections.text: Contrastive learning is an approach that relies on the fact that every document is, in some way, unique. This signal is the only information available in the absence of manual supervision. The resulting algorithm learns by discriminating between documents, using a contrastive loss (Wu et al., 2018). This loss compares either positive (from the same document) or negative (from different documents) pairs of document representations. More formally, given a positive pair of representations (q, k+) and a set of negative pairs (q, ki)i=0..K , the contrastive InfoNCE loss is defined as:
  L(q, k+) = exp(s(q, k+)/τ)∑K i=0 exp(s(q, ki)/τ) ,
  where τ is a temperature parameter. This loss encourages the relevance score of similar examples to be high and that of dissimilar examples to be low. Another interpretation of this loss function is the following: given the query representation q, the goal is to recover, or retrieve, the representation k+ corresponding to the positive document, among all the negatives ki. In the following, we refer to the left-hand side representations in the score s as queries and the right-hand side representations as keys.
sections.heading: 3.1.2 BUILDING POSITIVE PAIRS FROM A SINGLE DOCUMENT
sections.text: A crucial element of contrastive learning is how to build positive pairs from a single input. In computer vision, this step relies on applying two independent data augmentations to the same image, resulting in two “views” that form a positive pair (Wu et al., 2018; Chen et al., 2020). While we consider similar independent text transformation, we also explore dependent transformations designed to reduce the correlation between views.
  Inverse Cloze Task is a data augmentation that generates two mutually exclusive views of a document, introduced in the context of retrieval by Lee et al. (2019). The idea is to take a span of tokens to form one view, and its complementary to form the other view. More precisely, given a sequence of text (w1, ..., wn), ICT samples a span a, b, where 1 ≤ a < b ≤ n, uses the tokens of the span as the query and the rest of the tokens as the document (or key). In the original implementation by Lee et al. (2019) the span corresponds to a sentence, and is kept in the document 10% of the time to encourage lexical matching.
  Independent cropping is a common independent data augmentation used for images where views are generated independently by cropping the input. In the context of text, cropping is equivalent to sampling a span of tokens. This strategy thus samples independently two spans from a document to form a positive pair. As opposed to the inverse Cloze task, in cropping both views of the example correspond to contiguous subsequence of the original data. A second difference between cropping and ICT is the fact that the task is symmetric: both the queries and documents follow the same distribution. Independent cropping can also lead to overlap between the two views of the data, hence encouraging the network to learn exact matches between the query and document, in a way that is similar to lexical matching methods like BM25. In practice, we can either fix the length of the span for the query and the key, or sample them.
  Additional data augmentation. Finally, we also consider additional data augmentations such as random word deletion, random word replacement or random word masking. We use these perturbation in addition to random cropping.
sections.heading: 3.1.3 BUILDING LARGE SET OF NEGATIVE PAIRS
sections.text: An important aspect of contrastive learning is to maintain a large set of negative pairs. Most standard frameworks differ from how the negatives are handled, and we briefly describe two of them, SimCLR and MoCo, that we use in this work.
  Negative pairs within a batch. A first solution, called SimCLR (Chen et al., 2020), is to generate the negative pairs by using the other examples from the same batch: each example in a batch is transformed twice to generate positive pairs, and we generate negative pairs by using the views from the other examples in the batch. In that case, the gradient is back-propagated through the representations of both the queries and the keys. A downside of this approach is that it requires extremely large batch sizes to work well. This technique was also considered when using ICT to pre-train retrievers by Lee et al. (2019).
  Negative pairs across batches. An alternative approach is to store representations from previous batches in a queue and use them as negative examples in the loss (Wu et al., 2018). This allows for smaller batch size but slightly changes the loss by making it asymmetric between “queries” (one of the view generated from the elements of the current batch), and “keys” (the other generated views, as well as the elements stored in the queue). Gradient is only backpropagated through the “queries”, and the representation of the “keys” are considered as fixed. In practice, the features stored in the queue from previous batches comes form previous iterations of the network. This leads to a drop of performance when the network rapidly changes during training. Instead, He et al. (2020) proposed to generate representations of keys from a second network that is updated more slowly. This approach, called MoCo, considers two networks: one for the keys, parametrized by θk, and one of the query, parametrized by θq. The parameters of the query network are updated with backpropagation and stochastic gradient descent, like in SimCLR, while the parameters of the key network, or Momentum encoder, is updated from the parameters of the query network by using a exponential moving average:
  θk ← mθk + (1−m)θq, where m is the momentum parameter that takes its value in [0, 1].
sections.heading: 4 EXPERIMENTS
sections.text: In this section, we empirically evaluate contrastive learning as a way to train dense retrievers. First, we compare our best models to the state-of-the-art on competitive retrieval benchmarks for both zero-shot and few-shot retrieval. Then, we provide an ablation study to motivate the technical choices leading to our best retriever, shedding some light on the important ingredients of our approach. We give technical details about our models in Appendix A.
sections.heading: 4.1 BEIR BENCHMARK
sections.text: The BEIR benchmark, introduced by Thakur et al. (2021), contains 18 retrieval datasets with a focus on diversity. Each dataset is made of a set of queries, the corresponding relevant documents and a large collection of documents to retrieve from. These datasets correspond to nine different retrieval tasks, such as fact checking, question answering, entity retrieval or citation prediction. Moreover, the benchmark covers a large number of domains, such as Wikipedia, news articles, scientific publications or social media posts. Finally, there is also diversity in terms of documents and queries length, with queries ranging from 3 to 190 words and documents from 11 to 630 words. Most datasets from the BEIR benchmark do not contain a training set, and the focus of the benchmark is zero-shot retrieval. However, most machine learning based retrievers are still trained on supervised data, such as the large scale retrieval dataset MS-MARCO (Bajaj et al., 2016). Following standard practice, we report two metrics on this benchmark, the nDGC@10 and the Recall@100. These two metrics are complementary and both important: nDCG focuses on the ranking of the top 10 retrieved documents, and is good at evaluating documents returned to human, for example in a search engine. On the other hand, the Recall@100 is relevant to evaluate retrievers that are included in marchine learning systems, such as question answering. Indeed, such models can process hundreds of documents, and ignore the ranking of these documents (Izacard & Grave, 2020).
sections.heading: 4.2 BASELINES
sections.text: First, we compare our approach to the BM25 ranking function, which does not require any training data. However, the relevance of individual terms depends on the inverse document frequency, which is computed over each document collection independently. In that sense, the ranking function used in the BM25 baseline is adapted to each dataset on which retrieval is performed. We also compare
  to machine learning based retrievers, which can be classified in three categories: sparse, dense and late-interaction. For sparse methods, we compare to docT5query (Nogueira et al., 2019), which is a document expansion technique based on the T5 model (Raffel et al., 2019). The second sparse model we use as baseline is SPARTA (Zhao et al., 2020), which computes sparse representations of documents using a BERT model. More precisely, the ranking function of SPARTA is based on interactions between terms of the document and the query. Because the representations of the query terms are non-contextualized, the scores can be pre-computed when indexing the database. For dense methods, we use DPR (Karpukhin et al., 2020) and ANCE (Xiong et al., 2020) as baselines, which are bi-encoder models trained on supervised data such as NaturalQuestions or MS-MARCO. We also include the TAS-B technique (Hofstätter et al., 2021), which performs distillation from a cross-encoder to a bi-encoder model. Finally, we compare to GenQ, which uses a generative query model to create synthetic query-document pairs, and is applied to each dataset. Thus, a different GenQ model is used for each dataset of the BEIR benchmark. For late-interaction methods, we use ColBERT (Khattab et al., 2020), which computes pairwise scores between contextualized representations of the query and the document. To make this approach scalable, this scoring function is only applied on documents obtained by applying an approximation of the full score. Interestingly, it should be noted that BM25 is a very competitive baseline in the zero-shot setting, outperforming all dense retrievers on average over the BEIR benchmark. Only the re-ranking models and the query expansion (docT5query), which is also based on BM25, obtain better results than BM25.
sections.heading: 4.3 RESULTS
sections.text: In Table 1, we report the nDCG@10 on the BEIR benchmark for different retrievers. The corresponding Recall@100 performances can be found in Table 7 of appendix. We individually report results on each dataset as well as the average (“Avg.”) over 14 datasets of the BEIR Benchmark (excluding 3 for license reasons). We omit retrievers that use a re-ranker during inference since it is orthogonal to our focus and can be applied to any method.
  We observe that contrastive pre-training leads to strong performance among retrievers: our model obtains state-of-the art performance when finetuned on the MS-MARCO dataset. It should be noted that our finetuning procedure on MS-MARCO is simpler than for other dense retrievers: we use a simple strategy for negative mining and do not use distillation. Our model would probably also benefits from improvements proposed by these retrievers, but this is beyond the scope of this paper.
  finetuning SciFact NFCorpus FiQA
  Dataset size 809 2,590 5,500
  BM25 - 66.5 32.5 23.6
  BERT - 75.2 29.9 26.1 Ours - 84.0 33.9 31.5
  BERT MS-MARCO 80.9 33.2 30.9 Ours MS-MARCO 84.1 34.9 34.4
  We also illustrate the benefit of our dense retriever compared to BM25 by evaluating it in a few shot setting, where we have access to a small number of training queries and relevant documents to finetune locally the retriever. This setting is common in practice, and lexical based methods, like BM25, cannot benefit leverage the small training sets to adapt its weights. In Table 2, we report the nDCG@10 results on the three datasets from the BEIR benchmark associated with the smallest training sets. These training sets range from 809 to almost 5,500 queries. We observe that on these small datasets, our pre-training leads to significantly better results than BERT pre-training only, even when BERT model is finetuned on MS-MARCO. Our pre-trained model also outperforms BM25, showing the advantage of dense retriever over lexical methods in the low-shot setting.
  Finally, we compare the performance of fully unsupervised models, i.e., without finetuning on MS-MARCO or other annotated data. In Table 8 we report the retrieval performance of our model on two standard question answering benchmarks: NaturalQuestions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017). In Figure 1 we report the performance of unsupervised models on the BEIR benchmark. Interestingly, we observe that, in this setting, our pre-training can lead to a model with competitive retrieval performance compared to BM25. Our approach outperforms previously proposed unsupervised dense retrievers. On question answering benchmarks it also outperforms a strong BM25 baseline (Ma et al., 2021) by 3% for the Recall@100, and is on par with BM25 on TriviaQA for the Recall@100. On datasets of the BEIRbenchmark, we observe that our unsupervised retriever is competitive with BM25 on every datasets, but TREC-COVID and Tóuche-2020. These results show the potential of contrastive learning to develop fully unsupervised dense retrievers.
sections.heading: 5 ABLATION STUDIES
sections.text: In this section, we investigate the influence of different design choices on our method. In these ablations, all the models are pre-trained on Wikipedia for 200k gradient steps, with a batch size of
  2,048 (on 32 GPUs). Each fine-tuning on MS-MARCO takes 20k gradient steps with a batch size of 512 (on 8 GPUs), using AdamW and no hard negatives.
  MoCo vs. SimCLR. First, we compare the two contrastive pre-training methods: MoCo and SimCLR. As in SimCLR, the number of negative examples is equal to the batch size, we train models with a batch size of 4,096 and restrict the queue in MoCo to the same number of elements. This experiment measures the effect of using of momentum encoder for the keys instead of the same network as for the queries. Using a momentum also prevents from backpropagating the gradient through the keys. We report results, with fine-tuning on MS-MARCO in Table 5. We observe that the difference of performance between the two methods is small, especially after fine-tuning on MS-MARCO. This validates our choice of MoCo as our contrastive learning framework, since it scales to a larger number of negative examples without the need to increase the batch size.
  Number of negative examples. Next, we study the influence of the number of negatives used in the contrastive loss, by varying the queue size of the MoCo algorithm. We consider values ranging from 2,048 to 131,072, and report results in Figure 2. We see that on average over the BEIR benchmark, increasing the number of negatives leads to better retrieval performance, especially in the unsupervised setting. However, we note that this effect is not equally strong for all datasets.
  Data augmentations. Third, we compare different ways to generate pairs of positive examples from a single document or chunk of text. In particular, we compare random cropping, which leads to pairs with overlap, and the inverse cloze task, which was previously considered to pre-train retrievers. Interestingly, as shown in Table 4, the random cropping strategy strongly outperforms the inverse cloze task in our setting. We argue that overlap between the two views of the data is beneficial, as it allows the network to learn lexical matching, similarly to BM25. We also investigate
  whether additional data perturbations, such as random word deletion or replacement, are beneficial for retrieval. After fine-tuning on MS-MARCO, it seems that such data perturbations slightly degrade the performance of dense retrievers.
  Training data. Finally, we study the impact of the pre-training data on the performance of our retriever, by training on Wikipedia, CCNet or a mix of both sources of data. We report results in Table 5, and observe that there is no clear winner between the two data sources. Unsurprisingly, pre-training on Wikipedia leads to better performance on datasets for which documents also come from Wikipedia, such as NaturalQuestions or FEVER. On the other hand, on datasets from different domains than Wikipedia such as FiQA or ArguAna, training on the more diverse CCNet data leads to better results. To get the best of both worlds, we consider to strategies to mix the two data sources. In the “50/50%” strategy, examples are sampled uniformly across domain, meaning that half the batches are from Wikipedia and the other half from CCNet. In the “uniform” strategy, examples are sampled uniformly over the union of the dataset. Since CCNet is significantly larger than Wikipedia, this means that most of the batches are from CCNet.
  Impact of fine-tuning on MS-MARCO. To isolate the impact of our pre-training method as opposed to the impact of our fine-tuning on MS-MARCO, we propose to apply the same fine-tuning to the BERT base uncased model. We report results in Table 6, and observe that when applied to BERT, our fine-tuning leads to results that are lower than the state-of-the-art. Hence, we believe that most of the improvement compared to the state-of-the-art retrievers can be attributed to our contrastive pre-training strategy.
  DISCUSSION
  In this work, we propose to explore the limits of contrastive pre-training to learn dense text retrievers. We use the MoCo technique, which allows us to train with a large number of negative examples. We make several interesting observations: first, while the inverse Cloze task has been introduced in the context of retrieval, we show that independent random cropping leads to better results. Second, we show that neural networks trained without supervision using contrastive learning exhibits good (albeit not state-of-the-art) retrieval performance. Based on these observations, we trained a model that obtains state-of-the-art performance on the BEIR benchmark when fine-tuned on MS-MARCO.
sections.heading: A TECHNICAL DETAILS
sections.text: Contrastive pre-training. For the model with fine-tuning on MS-MARCO (respectively, without fine-tuning), we use the MoCo algorithm He et al. (2020) with a queue of size 65536 (resp. 131,072), a momentum value of 0.999 (resp. 0.9995) and a temperature of 0.05. We use the random cropping data augmentation, with documents of 256 tokens and span sizes sampled between 10% and 50% (resp. 5% and 50%) of the document length. We also apply word deletion with a probability of 20%. We optimize the model with the AdamW (Loshchilov & Hutter, 2019) optimizer, with learning rate of 10−4 (resp. 5 · 10−5), batch size of 2,048 and 500,000 (resp. 200,000) steps. We initialize the network with the publicly available BERT base uncased model. For training data, we use a data from Wikipedia (resp. a mix between Wikipedia and CC-net data (Wenzek et al., 2020), where half the batches are sampled from each source).
  Fine-tuning on MS-MARCO. When fine-tuning our model on MS-MARCO, we use the ASAM optimizer (Kwon et al., 2021), with a learning rate of 10−5 and a batch size of 1024 with a temperature of 0.05, also used during pretraining. We train an initial model with random negative examples for 20000 steps, mine hard negatives with this first model, and re-train a second model with those. Each query is associated with a gold document and a negative document, which is a random document in the first phase and a hard negative 10% of the time in the second phase. All documents from a batch are used as negatives for each query.
title: TOWARDS UNSUPERVISED DENSE INFORMATION RE-