We consider the large-scale query-document retrieval problem: given a query (e.g., a question), return the set of relevant documents (e.g., paragraphs containing the answer) from a large document corpus. This problem is often solved in two steps. The retrieval phase first reduces the solution space, returning a subset of candidate documents. The scoring phase then scores and re-ranks the documents. The algorithm used in the retrieval phase is critical. On the one hand, it needs to have high recall -otherwise some relevant documents won't even be considered in the scoring phase. On the other hand, it needs to be highly efficient, returning the candidate documents in time sublinear to the total number of documents. Unlike the scoring phase which witnessed significant advances recently due to the BERT-style cross-attention models, the retrieval phase remains less well studied: most previous works rely on the classic Information Retrieval (IR) methods such as BM-25 (token matching + TF-IDF weights). In this paper, we conduct a comprehensive study on different retrieval algorithms and show that the two-tower Transformer models with properly designed pre-training tasks can largely improve over the widely used BM-25 algorithm. The pre-training tasks we studied are Inverse Cloze Task (ICT), Body First Selection (BFS), Wiki Link Prediction (WLP) and the combination of them.
This paper studies a query-related document retrieval problem using a framework which they call “two-tower retrieval method”. The task is to learn query representation and document representation in order to retrieve query-related documents by the maximum inner product. This is a realistic setting for large-scale retrieval problem since it enables document representations to be computed once regardless of the question, and obtaining query-sensitive document representations is very expensive.
Then, the paper studies three different pretraining methods for this task, ICT (previously proposed by Lee et al 2019), and BFS & WLP (proposed by this paper). 
For evaluation, the paper considers the retrieval task of question answering, based on SQuAD and Natural Questions. The combination of ICT, BFS and WLP achieves remarkable improvement over the number of baselines including BM25 and other neural-based models.
The strength of this paper is that it includes comprehensive studies on the two-tower retrieval problem. In particular, they have conducted extensive ablation studies with different train/test ratios.
However, there are some notable weaknesses of this paper as follows.
First, the benchmark relies on the recall rate instead of the end task (open-domain QA). Recall rate is not a good way to evaluate the retrieval result since a system may retrieve text which contains the answer text but is not semantically related to the question. (I understand that the paper follows Admad et al (2019), but I believe this is not a published paper.) In addition, this paper did not empirically demonstrate the relatedness between the recall rate and the end performance. This makes it very hard to compare with other papers in open-domain QA, which has been extensively studied for a few recent years.
Second, despite comprehensive studies, the fact that ICT+BFS+WLP is almost the same as ICT (93.91 vs. 94.37) means that the method does not give improvement over ICT which was already proposed in the previous study.
Third, the gap between BM25 and ICT+BFS+WLP in Table 3 and 4 are very significant (e.g. 27 vs 94 on Natural Questions), but this doesn't seem to be consistent to Lee et al (2019). (There are some differences: (1) Lee et al (2019) compares BM25 vs. ICT, but according to this paper, ICT and ICT+BFS+WLP are similar. (2) Lee et al (2019) reports the end QA performance while this paper reports the recall rate, but one of the assumptions in this paper is that recall rate and the end performance is related.) What is the explanation for this discrepancy?
(I am happy to increase the rating if my concerns are resolved during rebuttals and/or the paper includes performance on the end QA performance.)
Some questions:
1) Section 4.1 says ICT is sentence-level, BFS is paragraph-level and WLP is document-level. What does it mean? I thought, according to Section 3, all methods are paragraph-level.
2) Section 4.1: it looks like Ahmad et al (2019)’s setting is actually not entirely open-domain. Their candidate sentences/paragraphs are much less than the entire Wikipedia. Did this paper also use the same set of the candidate? In that case, it should be clearly mentioned in the paper. In addition, the data statistics are different across two papers. Did Ahmad et al (2019) include only train set whereas this submission reports train+test? In case there is an official split of train/test, why were different splits used for evaluation?
3) Also regarding the split: for each split, how much was used for development? I believe data used for the development and test should be different. In fact, rather than experimenting on different ratios of train/test, is it possible to report on official test set, while splitting the train set into 90/10 for training and development? Or, split the entire data to 90/5/5 for training/development/test?
Update on Nov 15: The revised paper resolves most of my concerns, so I am updating the score from 3 to 6.