Recent research demonstrates the effectiveness of using fine-tuned language models (LM) for dense retrieval. However, dense retrievers are hard to train, typically requiring heavily engineered fine-tuning pipelines to realize their full potential. In this paper, we identify and address two underlying problems of dense retrievers: i) fragility to training data noise and ii) requiring large batches to robustly learn the embedding space. We use the recently proposed Condenser pre-training architecture, which learns to condense information into the dense vector through LM pre-training. On top of it, we propose coCondenser, which adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space. Experiments on MS-MARCO, Natural Question, and Trivia QA datasets show that coCondenser removes the need for heavy data engineering such as augmentation, synthesis, or fil-tering, and the need for large batch training. It shows comparable performance to RocketQA, a state-of-the-art, heavily engineered system, using simple small batch fine-tuning. 1 1 Introduction Building upon the advancements of pre-trained language models (LM; Devlin et al. (2019); Liu et al. (2019)), dense retrieval has become an effective paradigm for text retrieval (Lee et al., 2019; Chang et al., 2020; Karpukhin et al., 2020; Qu et al., 2021; Gao et al., 2021a; Zhan et al., 2022). Recent research has however found that fine-tuning dense retrievers to realize their capacity requires carefully designed fine-tuning techniques. Early works include iterative negative mining (Xiong et al., 2021) and multi-vector representations (Luan et al., 2020). The recent RocketQA system (Qu et al., 2021) sig-nificantly improves the performance of a dense retriever by designing an optimized fine-tuning 1 Our code is available at https://github.com/ luyug/Condenser . pipeline that includes i) denoising hard negatives, which corrects mislabeling, and ii) large batch training. While this is very effective, the entire pipeline is very heavy in computation and not feasible for people who do not have tremendous hardware resources, like those in academia. In this paper, we ask, instead of directly using the pipeline, can we take the insights of RocketQA to perform language model pre-training such that the pre-trained model can be easily fine-tuned on any target query set. Concretely, we ask what the optimized training in RocketQA solves. We hypothesize that typical LMs are sensitive to mislabeling, which can cause detrimental updates to the model weights. Denoising can effectively remove the bad samples and their updates. On the other hand, for most LMs, the CLS vectors are either trained with a simple task (Devlin et al., 2019) or not explicitly trained at all (Liu et al., 2019). These vectors are far from being able to form an embedding space of passages (Lee et al., 2019). The large training batches in RocketQA help the LM to stably learn to form the full embedding space. To this end, we want to pre-train an LM such that it is locally noise-resistant and has a well-structured global embedding space. For noise resistance, we borrow the Condenser pre-training architecture (Gao and Callan, 2021), which performs language model pretraining actively conditioning on the CLS vector. It produces an information-rich CLS representation that can robustly condense an input sequence. We then introduce a simple corpus level contrastive learning objective: given a target corpus of documents to retrieve from, at each training step sample text span pairs from a batch of documents and train the model such that the CLS embeddings of two spans from the same document are close and spans from different documents are far apart. Combining the two, we propose coCondenser pre-training, which unsupervisedly learns a corpus-aware pre-trained model for dense retrieval. In this paper, we test coCondenser pre-training on two popular corpora, Wikipedia and MSMARCO. Both have served as information sources for a wide range of tasks. This popularity justi-fies pre-training models specifically for each of them. We directly fine-tune the pre-trained coCondenser using small training batches without data engineering. On Natural Question, TriviaQA, and MS-MARCO passage ranking tasks, we found that the resulting models perform on-par or better than RocketQA and other contemporary methods. Dense Retrieval Transformer LM has advanced the state-of-the-art of many NLP tasks (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lan et al., 2020) including dense retrieval. Lee et al. (2019) are among the first to demonstrate the effectiveness of Transformer dense retrievers. They proposed a simple Inverse Cloze Task (ICT) method to further pre-train BERT (Devlin et al., 2019). Follow-up works explored other pretraining tasks (Chang et al., 2020) as well end-to-end co-training of reader and retriever (Guu et al., 2020). Karpukhin et al. (2020) is the first to discover that careful fine-tuning can learn effective dense retriever directly from BERT. Later works then started to investigate ways to further improve fine-tuning (Xiong et al., 2021; Qu et al., 2021). Among them, Qu et al. (2021) proposed the RocketQA fine-tuning pipeline which hugely advanced the performance of dense retrievers. Until very recently, pre-training for dense retrieval has been left unexplored. A concurrent work, DPR-PAQ (Oguz et al., 2021), revisits pretraining and proposes domain matched pre-training, using a 65-million-size synthetic QA pair dataset generated with pre-trained Natural Question and Trivia QA pipelines to pre-train dense retrievers. This paper uses a recently proposed dense retrieval pre-training architecture, Condenser (Gao and Callan, 2021). Unlike previous works that design pre-training tasks, Condenser explored the idea of designing a special pre-training architecture to improve representation effectiveness. One reason why dense retrieval is of immediate great value is that there is a rich literature that studies efficient dense retrieval for first stage retrieval (Johnson et al., 2017; Guo et al., 2020). There are also mature dense retrieval libraries, such as FAISS (Johnson et al., 2017). By pre-encoding the corpus into a MIPS index, retrieval can run online with millisecond-level latency (Johnson et al., 2017; Guo et al., 2020). Contrastive Learning Contrastive learning has become a very popular topic in computer vision (Chen et al., 2020; He et al., 2020). Recent works have brought the idea to natural language processing to learn high-quality sentence representation (Giorgi et al., 2020; Wu et al., 2020). In this work, we use contrastive learning to do pretraining for dense retrieval. Different from earlier work, instead of individual representations (Giorgi et al., 2020), we are interested in the full learned embedding space, which we will use to warm start the retriever. The large batch requirement had been a limiting factor in contrastive learning (Chen et al., 2020) under resource-limited setups where GPU (accel-erator) memory is not sufficiently large. In general, this extends to any training procedure that uses contrastive loss, including dense retrieval pretraining (Guu et al., 2020; Chang et al., 2020). Gao et al. (2021b) recently devised a gradient cache technique that upper-bounds peak memory usage of contrastive learning to almost constant. In subsection 3.3, we show how to adapt it for coCondenser pre-training. In this section, we first give a brief review of Condenser. Then we discuss how to extend it to coCondenser and how to perform memory-efficient coCondenser pre-training. In this paper, we adopt a special pre-training architecture, Condenser (Gao and Callan, 2021). Condenser is a stack of Transformer blocks. As shown in Figure 1, these Transformer blocks are divided into three groups, early backbone encoder layers, late backbone encoder layers, and head layers. An input x = [ x 1 , x 2 , .. ] is first prepended a CLS, embedding, and run through the backbone layers. [ h 0 cls ; h 0 ] = Embed ([ CLS ; x ]) (1) [ h earlycls ; h early ] = Encoder early ([ h 0 cls ; h 0 ]) (2) [ h latecls ; h late ] = Encoder late ([ h earlycls ; h early ]) (3) The head takes the CLS representation from the late layers but using a short circuit, the token representations from the early layers. This late-early pair then runs through the head's Transformer blocks. The head's outputs are then used to do masked language model (MLM; Devlin et al. (2019)) training. To utilize the capacity of the late layers, Condenser is forced to learn to aggregate information into the CLS token, which will then participate in the LM prediction. Leveraging the rich and effective training signal produced by MLM, Condenser learns to utilize the powerful Transformer architecture to generate dense CLS representations. We hypothesize that with this LM objective typically used to train token representations now put on the dense CLS representation, the learned LM gains improved robustness against noise. While Condenser can be trained on a diverse collection of corpora to produce a universal model, it is not able to solve the embedding space issue: while information embedded in the CLS can be non-linearly interpreted by the head, inner products between these vectors still lack semantics. Consequently, they do not form an effective embedding space. To this end, we augment the Condenser MLM loss with a contrastive loss. Unlike previous work that pre-trains on artificial query passage pairs, in this paper, we propose to simply pre-train the passage embedding space in a query-agnostic fashion, using a contrastive loss defined over the target search corpus. Concretely, given a random list of n documents [ d 1 , d 2 , ..., d n ] , we extract randomly from each a pair of spans, [ s 11 , s 12 , ..., s n 1 , s n 2 ] . These spans then form a training batch of coCondenser. Given a span s ij 's corresponding late CLS representation h ij , its corpus-aware contrastive loss is defined over the batch as shown below. (6) Familiar readers may recognize this as the contrastive loss from SimCLR (Chen et al., 2020), for which we use random span sampling as augmentation. Others may see a connection to noise contrastive estimation (NCE). Here we provide an NCE narrative. Following the spirit of the distributional hypothesis, passages close together should have similar representations while those in different documents should have different representations. Here we use random spans as surrogates of passages and enforce the distributional hypothesis through NCE, as word embedding learning in Word2Vec (Mikolov et al., 2013). We can also recognize this as a span-level language model objective, or skip-span. Denote span s ij 's Condenser MLM loss L mlm ij . The batch's loss is defined as an average sum of MLM and contrastive loss, or from an alternative perspective, word and span LM loss. The RocketQA pipeline uses supervision and large-batch training to learn the embedding space. We would also like to run large-batch unsupervised pretraining to construct effective stochastic gradient estimators for the contrastive loss in Equation 6. To remind our readers, this large-batch pre-training happens only once for the target search corpus. We will show that this allows effective small batch fine-tuning on task query sets. However, due to the batch-wise dependency of the contrastive loss, it requires fitting the large batch into GPU (accelerator) memory. While this can be done naively with interconnected GPU nodes or TPU pods, which can have thousands of gigabytes of memory, academia, and smaller organizations are often restricted to machines with 2845 four commercial GPUs. To break the memory constraint and perform effective contrastive learning, we adjust the gradient caching technique (Gao et al., 2021b) for our setup. We describe the procedure here for people who want to perform coCondenser pre-training but have limited resources. Denote L co = (cid:80) i (cid:80) j L coij , we can write Equation 7 as, L = 1 2 n [ L co + (cid:88) i (cid:88) j L mlm ij ] (8) The spirit of gradient caching is to decouple representation gradient and encoder gradient computation. Before computing the model weight update, we first run an extra backbone forward for the entire batch. This provides numerical values of [ h 11 , h 12 , ...., h n 1 , h n 2 ] , from which we compute: v ij =  h ij (cid:88) i (cid:88) j L coij =  L co h ij (9) i.e. the contrastive loss gradient with respect to the CLS vector. We store all these vectors in a gradient cache, C = [ v 11 , v 12 , .., v n 1 , v n 2 ] . Using v ij , denote the model parameter  , we can write the derivative of the contrastive loss as shown below. We can then write the gradient of Equation 8. Since v ij is already in the cache C , each summation term now only concerns span s ij and its activation, meaning that we can compute the full batch's gradient in an accumulation fashion over small sub-batches. In other words, the full batch no longer needs to concurrently reside on the GPUs. At the end of pre-training, we discard the Condenser head, keeping only the backbone layers. Consequently, the model reduces to its backbone, or effectively a Transformer Encoder. We use the backbone weights to initialize query encoder f q and passage encoder f p . Each outputs the last layer CLS. Recall that they have already been warmed up in pre-training. Query and passage encoders are supervisedly fine-tuned on the target task's training set. We train with a supervised contrastive loss and compute for query q , negative log likelihood of a positive document d + against a set of negatives { d  1 , d  2 , ..d  l .. } . We run a two-stage training as described in the DPR (Karpukhin et al., 2020) toolkit. As shown in Figure 2b, in the first stage, the retrievers are trained with BM25 negatives. The first-stage retriever is then used to mine hard negatives to complement the negative pool. The second stage retriever trains with the negative pool generated in the first round. This is in contrast to the multi-stage pipeline of RocketQA shown in Figure 2a. In this section, we first describe the implementation details of coCondenser pre-training. We then conduct dense retrieval experiments to test the effectiveness of fine-tuned coCondenser retrievers. The coCondenser pre-training starts with vanilla BERT and goes in two stages, universal Condenser pre-training and corpus aware coCondenser pretraining. In the first stage, we pre-train a Condenser and warm start the backbone layers with pre-trained 12-layer BERT base weights (Devlin et al., 2019). The backbone uses an equal split, 6 early layers, and 6 late layers. The Condenser pre-training uses the same data as BERT: English Wikipedia and the BookCorpus. The first stage Condenser pretraining takes roughly a week on 4 RTX 2080 Ti GPUs or 2 days on a v3-8 cloud TPU. The Condenser model from stage one, including both backbone and head, is taken to warm start stage two coCondenser pre-training on the target corpus (Wikipedia or MS-MARCO web collection). We keep the Condenser architecture unchanged in the second step. We use AdamW optimizer with a learning rate 1e-4, weight decay of 0.01, and linear learning rate decay. Each model weight update uses 2K documents. We train using gradient 2846 cache update, as described in subsection 3.3. We used the released Condenser model for the first stage. The second stage takes roughly 2 days on 4 RTX 2080 Ti GPUs or 19 hours on a v3-8 cloud TPU. Our GPU implementations are based on Py-torch (Paszke et al., 2019) and TPU implementations on JAX (Bradbury et al., 2018). After the second stage finishes, we discard the Condenser head, resulting in a model of the exact same architecture as BERT base . Next, we fine-tune the learned coCondenser to test retrieval performance. Following RocketQA, we test on Natural Question and MS-MARCO passage ranking. We also report performance on Trivia QA, whose pre-processed version is released in DPR. Dataset We use MS-MARCO passage ranking (Bajaj et al., 2018), Natural Question(NQ; Kwiatkowski et al. (2019)) and Trivia QA(TQA; Joshi et al. (2017)). MS-MARCO is constructed from Bing's search query logs and web documents retrieved by Bing. Natural Question contains questions from Google search. Trivia QA contains a set of trivia questions. We report official metrics MRR@10, Recall@1000 for MS-MARCO, and Recall at 5, 20, and 100 for NQ and TQA. Data Preparation We use Natural Question, Trivia QA, and Wikipedia cleaned and released with DPR toolkit (Karpukhin et al., 2020). NQ and TQA each have about 60K training data postprocessing. Similarly, we use the MS-MARCO corpus released with RocketQA open-source code (Qu et al., 2021). For reproducibility, we use the official relevance file instead of RocketQA's extended one, which has about 0.5M training queries. The BM25 negatives for MS-MARCO are taken from the official training triples. Training MS-MARCO models are trained with Tevatron toolkit (Gao et al., 2022) using AdamW with a 5e-6 learning rate, linear learning rate schedule, and batch size 64 for 3 epochs. NQ and TQA models are trained with the DPR toolkit following published hyperparameters by Karpukhin et al. (2020). All models are trained on one RTX 2080 Ti. We added gradient caching to DPR to deal with memory constraints. The models are trained only on each task's corresponding training set. We note that RocketQA is trained on a concatenation of several datasets (Qu et al., 2021). Model Validation Since for dense retrieval, validating a checkpoint requires encoding the full corpus, evaluating a checkpoint becomes very costly. Due to our computation resource limitation, we follow the suggestion in the DPR toolkit and take 2847 Method MS-MARCO Dev Natural Question Test Trivia QA Test MRR@10 R@1000 R@5 R@20 R@100 R@5 R@20 R@100 BM25 18.7 85.7 -59.1 73.7 -66.9 76.7 DeepCT 24.3 90.9 ---docT5query 27.7 94.7 ---GAR -60.9 74.4 85.3 73.1 80.4 85.7 DPR --74.4 85.3 -79.3 84.9 ANCE 33.0 95.9 -81.9 87.5 -80.3 85.3 ME-BERT 33.8 ---RocketQA 37.0 97.9 74.0 82.7 88.5 -Condenser 36.6 97.4 -83.2 88.4 -81.9 86.2 DPR-PAQ BERT base 31.4 -74.5 83.7 88.6 --BERT large 31.1 -75.3 84.4 88.9 --RoBERTa base 32.3 -74.2 84.0 89.2 --RoBERTa large 34.0 -76.9 84.7 89.2 -coCondenser 38.2 98.4 75.8 84.3 89.0 76.8 83.2 87.3 Table 1: Retrieval performance on MSMARCO dev, Natural Question test and Trivia QA test. the last model training checkpoint. We do the same for MS-MARCO. Comparison Systems We take RocketQA (Qu et al., 2021), the state-of-the-art fine-tuning technique, as our main baseline. We borrowed several other baselines from the RocketQA paper, including lexical systems BM25, DeepCT (Dai and Callan, 2019), DocT5Query (Nogueira and Lin, 2019) and GAR (Mao et al., 2020); and dense systems DPR (Karpukhin et al., 2020), ANCE (Xiong et al., 2021), and ME-BERT (Luan et al., 2020). We also included the concurrent work DPR-PAQ (Oguz et al., 2021), which pre-trains using a 65-million-size synthetic QA pair dataset. The pre-training data is created by using retriever-reader pairs trained on Natural Question and Trivia QA. Designing the synthesis procedure also requires domain knowledge, thus under the context of this paper, we refer to this as a semi-supervised pre-training method. We include 4 DPR-PAQ variants based on base/large architectures of BERT/RoBERTa models. Finally, we fine-tune a Condenser model which is produced in the first stage of pre-training. Table 1 shows development (dev) set performance for MS-MARCO passage ranking and test set performance for Natural Question and Trivia QA. Across three query sets, dense systems show superior performance compared to sparse systems. We also see a big performance margin between systems involving either careful fine-tuning or pre-training (RocketQA, DPR-PAQ, Condenser, coCondenser) over earlier dense systems. This result confirms recent findings that low dimension embeddings possess a strong capacity for dense retrieval, a capacity however hard to exploit naively. coCondenser shows small improvements over RocketQA. Importantly, this is achieved with greatly reduced computation and data engineering effort in fine-tuning. Notably on MS-MARCO, coCondenser reduced the RocketQA's 4096 batch size to 64 (Table 5). A comparison of the two training pipelines of RocketQA and coCondenser can be found in Figure 2. Comparison with DPR-PAQ shows several interesting findings. Combining large semi-supervised pre-training with the better and larger LM RoBERTa large , DPR-PAQ achieves the best results on Natural Question. On the other hand, when starting from BERT (base/large), DPR-PAQ shows similar performance to coCondenser, which is based on BERT base . This suggests that large-scale semi-supervised pre-training is still the way to go to get the very best performance. However, when computational resources are limited and a large pre-training set is missing, the unsupervised coCondenser is a strong alternative. On the other hand, as it moves to MS-MARCO where DPR-PAQ's pre-training supervision becomes distant, 2848 MS-MARCO Passage Ranking Leaderboard Rank Method MRR@10 1 Adaptive Batch Scheduling + CoCondenser 43.1 2 coCondenser* 42.8 MS-MARCO Document Ranking Leaderboard Rank Method MRR@100 1 UniRetriever 44.0 2 coCondenser + MORES+* 43.6 Table 2: Performance on the MS-MARCO leader-boards with reranking. we observe that DPR-PAQ becomes less effective than RocketQA and coCondenser. The comparison between Condenser and coCondenser demonstrates the importance of the contrastive loss in coCondener: coCondenser can be robustly fine-tuned thanks to its pre-structured embedding space, allowing it to have better Recall (fewer false negatives) across all datasets. Due to the leaderboard nature of MS-MARCO Eval set, we cannot do ablation studies on it but have only made two submissions. We follow other top-performing systems and add some form of reranker. For the passage ranking leaderboard, we rerank the top 1000 retrieved passages with an ensemble of ERNIE (Sun et al., 2020) and RoBERTa. We also fine-tuned a coCondenser on the MS-MARCO document ranking dataset. As passage retrieval is the focus of this paper, we retrieve based on the first passage of 512 tokens. The top100 are reranked by a fast modular reranker (Gao et al., 2020). Performance of best systems and ours are recorded in Table 2. At the time of this paper's submission, both of our systems are the 2nd best on the two leaderboards. For passages ranking, we are excited to see that other people are able to further improve coCondenser with additional fine-tuning techniques. For document, we leave the study of retrieval beyond first passage to future work and refer readers to other leaderboard systems. Recall the two desired properties of coCondernser are local noise resistance to mislabeling and a well-structured, pre-trained embedding space. In this section, to investigate the former, we introduce and compare with a knowledge distillation (Hin-ton et al., 2015) setup where we substitute noisy hard labels with soft labels from a cross-encoder. For the latter, we measure the quality of 1st stage retriever mined negatives to see if coCondenser embedding can help find related but not relevant hard negatives. We also provide ablation studies of loss components and, pre-training and fine-tuning stages. To analyze the local robustness of coCondenser, we introduce a knowledge distillation upper-bound model: instead of training using the noisy labels, we first train a cross encoder and then fine-tune a coCondenser model using soft labels generated from the cross-encoder. Unlike Qu et al. (2021) that uses cross encoder for filtering, here we directly expose the logits as soft labels. Concretely, given cross encoder g , a batch of M queries { q 1 , q 2 , .., q M } , each paired with N passages (positive and hard negatives) { p 11 , .., p 1 N , .., p MN } , for a query q l we define its soft target distribution T , T ij = softmax j ( g ( q l , d ij )) if i = l else 0 (15) i.e., the soft labels are normalized logits from g for the local passages and 0 for the rest. Let S ij = softmax ij ( s ( q, p ij )) , the normalized bi-encoder similarities. Loss is defined as Kullback-Leibler divergence between S and T , L kd = DKL ( S || T ) (16) This setup a) focuses on improving labels for local hard negatives and positives while b) avoids evaluating cross encoder g for in-batch negatives. In Table 3, we compare coCondenser trained with MS-MARCO Dev Model Label MRR@10 R@100 R@1K BERT Hard 33.4 85.1 95.4 coCondenser Hard 38.2 91.3 98.4 coCondenser Soft 39.1 91.9 98.6 Table 3: Fine-tuning with hard v.s. soft labels. the original hard labels and the cross-encoder generated soft labels. Using soft labels indeed produces some improvement. On the other hand, without help from the cross-encoder, coCondenser still yields performance within small margins, showing coCondenser's superior local noise resistance. Intuitively, a globally better-structured embedding space will be less likely to collapse, producing a more accurate set of mined negatives for the second stage retriever to learn over. In other words, the 2849 2nd stage retriever will produce less unexpected top (hard) negatives. To quantitatively measure this, we propose a new metric, top n neighborhood recall at depth k (nb-recall n @ k ): for a query, the coverage over the 2nd stage retriever's top n candidates by the 1st stage retrievers top k candidates, nb-recall n @ k = { stage1 top k }  { stage2 top n } n (17) Essentially, this measures how the mined hard negatives agree with the actual negatives, or simply, how well the 1st stage retriever locating hard negatives. We measure neighborhood recall of BERT, Condenser and coCondenser retrievers, averaged over all MS-MARCO Dev queries, for n = 50 , 100 and various k values, in Figure 3, We see consis-55.00 65.00 75.00 85.00 95.00 50 100 150 200 250 BERT Condenser coCondenser nb-recall n=50 Depth K 55.00 65.00 75.00 85.00 95.00 100 200 300 400 500 BERT Condenser coCondenser nb-recall n=100 Figure 3: MS-MARCO Dev Neighborhood Recall tent higher nb-recall of Condenser over BERT and coCondenser over Condenser. The former comes from stronger CLS representation while the latter is due to the globally better-structured embedding space. In this section, we conduct an ablation study to understand the second stage pre-training loss com-ponents' com-ponents' influence on the final quality. In particular, we consider a Condenser model further pre-trained with only the contrastive loss. In Table 4, we see that further pre-training Condenser with only contrastive loss leads to better recall but similar MRR. The contrastive loss learns a better embedding space but by itself cannot keep the CLS locally discriminative. The original Condenser is able to rank better locally, producing similar MRR with fewer recalled passages. When both contrastive and Condenser MLM loss are used, we see improvements on all metrics. This again stresses the importance of the Condenser MLM loss during the second contrastive learning stage. We seek to understand the contribution of each pre-training and fine-tuning stage of coCondenser retriever. We consider pre-trained Condenser from the first stage and coCondenser from the second stage. For each, we consider retrievers trained with and without hard negatives (HN). For reference, we compare also with various RocketQA training stages. Results are shown in Table 5. We see Method Batch Size MS-MARCO Dev MRR@10 R@1000 RocketQA Cross-batch negatives 8192 33.3 -+ Hard negatives 4096 26.0 -+ Denoising 4096 36.4 -+ Data augmentation 4096 37.0 97.9 coCondenser Condenser w/o HN 64 33.8 96.1 + Hard negatives 64 36.6 97.4 coCondenser w/o HN 64 35.7 97.8 + Hard negatives 64 38.2 98.4 Table 5: MS-MARCO Dev performance for various training stages RocketQA and coCondenser. that each stage of RocketQA is critical. As each is added, performance improves steadily. On the other hand, this also suggests the full pipeline has to be executed to get the best performance. In comparison, we see Condenser with hard negatives has 2850 performance very close to the full RocketQA system. Condenser with hard negatives also has better MRR than coCondenser without hard negatives, meaning that Condenser from the first pre-training stage is already very strong locally but the embedding space trained from a relatively cold start is still not optimal, causing global misses. Adding the corpus aware loss, coCondenser without hard negatives has Recall very close to the full RocketQA system, using only a size 64 batch. This again confirms our hypothesis that fine-tuning can benefit from a pre-trained passage embedding space. Further adding hard negatives, we get the strongest coCondenser system that is both locally and globally effective. Note that all Condenser systems achieve their performance without denoising, showing the superior noise resistance capability learned using the Condenser architecture. Practically, our systems also do not require data augmentation, which removes engineering effort in designing augmentation techniques and defining augmentation data. This paper introduces coCondenser , an unsupervised corpus-aware language model pre-training method. We demonstrate proper pre-training can establish not only language understanding ability but also corpus-level representation power. Leveraging the Condenser architecture and a corpus aware contrastive loss, coCondenser acquires two important properties for dense retrieval, noise resistance, and structured embedding space. This corpus-aware pre-training needs to be done once for a search corpus and is query agnostic. The learned model can be shared among various types of end task queries. Experiments show that coCondenser can drastically reduce the costs of fine-tuning a strong dense retriever. We also find coCondenser yields performance close or similar to semi-supervised pre-trained models that are several times larger. Importantly, coCondenser provides a hands-off way to pre-train a very effective LM for dense retrieval. In particular, it effectively removes the effort for designing and testing pre-training as well as fine-tuning techniques. With our models, practitioners can use limited resources to train dense retrieval systems with state-of-the-art level performance. Future works may also investigate integrating additional pre-training and/or fine-tuning methods to further improve performance. This research was supported by NSF grant IIS-1815528. Any opinions, findings, and conclusions in this paper are the authors' and do not necessarily reflect those of the sponsors. The authors would like to thank Google's TPU Research Cloud (TRC) for access to Cloud TPUs and the anonymous reviewers for the reviews.
abstain abstain method abstain abstain objective objective abstain abstain other abstain abstain other abstain abstain abstain abstain method objective objective abstain abstain abstain abstain method method abstain objective objective abstain abstain abstain method result other other other other other other other other other abstain other other other other other other method method other other other abstain method method method abstain abstain abstain other abstain abstain abstain abstain method abstain abstain method method method abstain abstain method abstain method abstain method method abstain abstain abstain method method method abstain abstain method method method method method method method method method abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain method method abstain method method abstain abstain abstain abstain method abstain method abstain method abstain method method method method abstain method method method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain other other abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain objective abstain abstain abstain abstain result abstain abstain method abstain other other other