Open-domain questions are likely to be open-ended and ambiguous, leading to multiple valid answers. Existing approaches typically adopt the rerank-then-read framework, where a reader reads top-ranking evidence to predict answers. According to our empirical analysis, this framework faces three problems: first , to leverage a large reader under a memory constraint, the reranker should select only a few relevant passages to cover diverse answers, while balancing relevance and diversity is non-trivial; second , the small reading budget prevents the reader from accessing valuable retrieved evidence filtered out by the reranker; third , when using a generative reader to predict answers all at once based on all selected evidence, whether a valid answer will be predicted also pathologically depends on the evidence of some other valid answer(s). To address these issues, we propose to answer open-domain multi-answer questions with a recall-then-verify framework, which separates the reasoning process of each answer so that we can make better use of retrieved evidence while also leveraging large models under the same memory constraint. Our framework achieves state-of-the-art results on two multi-answer datasets, and predicts significantly more gold answers than a rerank-then-read system that uses an oracle reranker. Open-domain question answering (Voorhees, 1999; Chen et al., 2017) is a long-standing task where a question answering system goes through a large-scale corpus to answer information-seeking questions. Previous work typically assumes that there is only one well-defined answer for each question, or only requires systems to predict one correct answer, which largely simplifies the task. However, humans may lack sufficient knowledge or patience  *Corresponding author: Minlie Huang. Original Question: When did [You Don't Know Jack] come out? Interpretation #1: When did the first video game called [You Don't Know Jack] come out? Evidence #1: You Don't Know Jack is a video game released in 1995, and the first release in ... Answer #1: 1995 Interpretation #2: When did the Facebook game [You Don't Know Jack] come out on Facebook? Evidence #2: In 2012, Jackbox Games developed and published a social version of the game on Facebook ... Answer #2: 2012 Interpretation #3: When did the film [You Don't Know Jack] come out? Evidence #3: You Don't Know Jack premiered April 24, 2010 on HBO. Answer #3: April 24, 2010 Table 1: An example of open-domain multi-answer questions. to frame very specific information-seeking questions, leading to open-ended and ambiguous questions with multiple valid answers. According to Min et al. (2020b), over 50% of a sampled set of Google search queries (Kwiatkowski et al., 2019) are ambiguous. Figure 1 shows an example with at least three interpretations. As can be seen from this example, the number of valid answers depends on both questions and relevant evidence, which challenges the ability of comprehensive exploitation of evidence from a large-scale corpus. Existing approaches mostly adopt the rerank-then-read framework. A retriever retrieves hundreds or thousands of relevant passages which are later reranked by a reranker; a generative reader then predicts all answers in sequence conditioned on top-ranking passages. With a fixed memory constraint 1 , there is a trade-off between the size of the reader and the number of passages the reader can process at a time. According to Min et al. (2021), provided that the reranker is capable of 1 We follow Min et al. (2021) to constrain memory usage, which is usually a bottleneck of performance on open-domain question answering . selecting a small set of highly-relevant passages with high coverage of diverse answers, adopting a larger reader can outperform a smaller reader using more passages. However, as shown by Section 4.4, this framework is faced with three problems: first , due to the small reading budget, the reranker has to balance relevance and diversity, which is non-trivial as it is unknown beforehand that which answers should be distributed with more passages to convince the reader and which answers can be safely distributed with less to save the budget for the other answers; second , the reader has no access to more retrieved evidence that may be valuable but is filtered out by the reranker, while combining information from more passages was found to be beneficial to open-domain QA (Izacard and Grave, 2021b); third , as the reader predicts answers in sequence all at once, the reader learns pathological dependencies among answers, i.e., whether a valid answer will be predicted also depends on passages that cover some other valid answer(s), while ideally, prediction of a particular answer should depend on the soundness of associated evidence itself. To address these issues, we propose to answer open-domain multi-answer questions with a recall-then-verify framework. Specifically, we first use an answer recaller to predict possible answers from each retrieved passage individually; this can be done with high recall, even when using a weak model for the recaller, but at the cost of low precision due to insufficient evidence to support or refute a candidate. We then aggregate retrieved evidence relevant to each candidate, and verify each candidate with a large answer verifier. By separating the reasoning process of each answer, our framework avoids the problem of multiple answers sharing a limited reading budget, and makes better use of retrieved evidence while also leveraging strong large models under the same memory constraint. Our contributions are summarized as follows:  We empirically analyze the problems faced by the rerank-then-read framework when dealing with open-domain multi-answer QA. To address these issues, we propose to answer open-domain multi-answer questions with a recall-then-verify framework, which makes better use of retrieved evidence while also leveraging the power of large models under the same memory constraint. Open-domain QA requires question answering systems to answer factoid questions by searching for evidence from a large-scale corpus such as Wikipedia (Voorhees, 1999; Chen et al., 2017). The presence of many benchmarks has greatly promoted the development of this community, such as questions from real users like NQ (Kwiatkowski et al., 2019) and WEBQUESTIONS (Berant et al., 2013), and trivia questions like Quasar-T (Dhingra et al., 2017) and TriviaQA (Joshi et al., 2017). All these benchmarks either assume that each question has only one answer with several alternative surface forms, or only require a system to predict one valid answer. A typical question answering system is a pipeline as follows: an efficient retriever retrieves relevant passages using sparse (Mao et al., 2021; Zhao et al., 2021) or dense (Karpukhin et al., 2020; Xiong et al., 2021; Izacard and Grave, 2021a; Khattab et al., 2021) representations; an optional passage reranker (Asadi and Lin, 2013; Nogueira and Cho, 2019; Nogueira et al., 2020) further nar-rows down the evidence; an extractive or generative reader (Izacard and Grave, 2021b; Cheng et al., 2021) predicts an answer conditioned on retrieved or top-ranking passages. Nearly all previous work focused on locating passages covering at least one answer, or tried to predict one answer precisely. However, both Kwiatkowski et al. (2019) and Min et al. (2020b) reported that there is genuine ambiguity in open-domain questions, resulting in multiple valid answers. To study the challenge of finding all valid answers for open-domain questions, Min et al. (2020b) proposed a new benchmark called AMBIGQA where questions are annotated with as many answers as possible. In this new task, the passage reranker becomes more vital in the rerank-then-read framework, particularly when only a few passages are allowed to feed a large reader due to memory constraints. This is because the reranker has to ensure that top-ranking passages are highly relevant and also cover diverse answers. Despite state-of-the-art performance on AMBIGQA (Min et al., 2021), according to our empirical analysis, applying the rerank-then-read framework to open-domain multi-answer QA faces the following problems: balancing relevance and diversity is nontrivial for the reranker due to unknown effect on the 1826 performance of the subsequent reader; when using a large reader under a fixed memory constraint, the small reading budget prevents it from making use of more retrieved evidence that is valuable but filtered out; when using a generative reader to predict all answers in sequence based on all selected evidence, it learns pathological dependencies among answers. To address these issues, we propose to tackle this task with a recall-then-verify framework, which separates the reasoning process of each answer with a higher level of evidence usage while also leveraging large models under the same memory constraint. Some previous work argued that a reader can be confused by similar but spurious passages, resulting in wrong predictions. Therefore, they proposed answer rerankers (Wang et al., 2018a,b; Hu et al., 2019; Iyer et al., 2021) to rerank top predictions from readers. Our framework is related to answer reranking but with two main differences. First, a reader typically aggregates available evidence and already does a decent job of answer prediction even without answer reranking; an answer reranker is introduced to filter out hard false positive predictions from the reader. By contrast, our answer recaller aims at finding possible answers with high recall, most of which are invalid. Evidence focused on each answer is then aggregated and reasoned about by our answer verifier. It is also possible to introduce another model analogous to an answer reranker to filter out false positive predictions from our answer verifier. Second, answer reranking typically compares answer candidates to determine the most valid one, while our answer verifier selects multiple valid answers mainly based on the soundness of their respective evidence but without comparisons among answer candidates. Open-domain multi-answer QA can be formally defined as follows: given an open-ended question q , a question answering system is required to make use of evidence from a large-scale text corpus C and predict a set of valid answers { a 1 , a 2 , ..., a n } . Questions and their corresponding answer sets are provided for training. Evaluation To evaluate passage retrieval and reranking, we adopt the metric MRECALL @ k from (Min et al., 2021), which measures whether the topk passages cover at least k distinct answers (or n answers if the total number of answers n is less than k ). To evaluate question answering performance, we follow (Min et al., 2020b) to use F1 score between gold answers and predicted ones. In this section, we will briefly introduce the representative and state-of-the-art rerank-then-read pipeline from (Min et al., 2021) for open-domain multi-answer questions, and provide empirical analysis of this framework. Dense retrieval is widely adopted by open-domain question answering systems (Min et al., 2020a). A dense retriever measures relevance of a passage to a question by computing the dot product of their semantic vectors encoded by a passage encoder and a question encoder, respectively. Given a question, a set of the most relevant passages, denoted as B ( |B| (cid:28) |C| ), is retrieved for subsequent processing. To improve the quality of evidence, previous work (Nogueira et al., 2020; Gao et al., 2021) finds it effective to utilize a passage reranker, which is more expressive than a passage retriever, to rerank retrieved passages, and select the k best ones to feed a reader for answer generation ( k < |B| ). With a fixed memory constraint, there is a trade-off between the number of selected passages and the size of the reader. As shown by (Min et al., 2021), with good reranking, using a larger reader is more beneficial. To balance relevance and diversity of evidence, Min et al. (2021) proposed a passage reranker called JPR for joint modeling of selected passages. Specifically, they utilized T5-base (Raf-fel et al., 2020) to encode retrieved passages following (Izacard and Grave, 2021b) and decode the indices of selected passages autoregressively using a tree-decoding algorithm. JPR is designed to seek for passages that cover new answers, while also having the flexibility to select more passages covering the same answer, especially when there are less than k answers for the question. A reader takes as input the top-ranking passages, and predicts answers. Min et al. (2021) adopted a generative encoder-decoder reader initialized with T5-3b, and used the fusion-in-decoder method from (Izacard and Grave, 2021b) which efficiently ag-1827 gregates evidence from multiple passages. Specifi-cally, each passage is concatenated with the question and is encoded independently by the encoder; the decoder then attends to the representations of all passages and generates all answers in sequence, separated by a [SEP] token. To analyze performance of the rerank-then-read framework for open-domain multi-answer questions, we built a system that resembles the state-of-the-art pipeline from (Min et al., 2021) but with two differences 2 . First, we used the retriever from (Izacard and Grave, 2021a). Second, instead of using JPR, we used an oracle passage reranker (OPR): a passage p is ranked higher than another passage p (cid:48) if and only if 1) p covers some answer while p (cid:48) covers none 2) or both p and p (cid:48) cover or fail to cover some answer but p has a higher retrieval score. Following (Min et al., 2021), we retrieved |B| =100 Wikipedia passages, k =10 of which were selected by the reranker. Table 2 shows model performance on a representative multi-answer dataset called AMBIGQA (Min et al., 2020b). Compared with JPR, OPR is better in terms of reranking, with similar question answering results 3 . Though 3,670 diverse gold answers are covered by OPR on the dev set, the reader predicts only 1,554 of them. Our empirical analysis and findings are detailed as follows. (1) To leverage a large reader under a fixed memory constraint, a reranker should select only a few highly-relevant passages to cover diverse answers, while balancing relevance and diversity is nontrivial. As shown by Figure 1a (bottom), the num-2 Code and models from (Min et al., 2021) were not publicly available in the period of this work. 3 With the oracle knowledge of whether a passage contains a gold answer during reranking, OPR is probably still far from being a perfect reranker. Notably, we are not striving for a better rerank-then-read pipeline for multi-answer questions, but use OPR as a representative case to analyze the problems a rerank-then-read pipeline may face. ber of selected supporting passages 4 of predicted gold answers has a widespread distribution. There may be cases where redundant false positive evidence is selected and can be safely replaced with passages that cover other gold answers. However, it is non-trivial for the reranker to know beforehand whether a passage is redundant, and how many or which supporting passages of an answer are strong enough to convince the reader. (2) Multiple answers sharing a small reading budget prevents a reader from using more evidence that may be valuable but is filtered out by the reranker. Due to the shared reading budget, it is inevitable that some answers are distributed with less supporting passages. As shown by Figure 1a, a gold answer covered by OPR but missed by the reader generally has significantly less supporting passages fed to the reader (3.13 on average) than a predicted gold answer (5.08 on average), but not because of lacking available evidence. There is more evidence in retrieved passages for missed answers but filtered out by the reranker. As shown by Figure 1b, OPR has a much lower level of evidence usage for missed answers. (3) As the reader predicts answers all at once conditioned on all selected passages, whether a valid answer will be predicted also pathologically depends on evidence of some other valid answer(s), which partly accounted for the large number of 4 We abuse the use of supporting passages of an answer to refer to passages that cover the answer. gold answers missed by the reader. For verification, we attacked OPR's reader on the dev set of AMBIGQA as follows: a question is a target if and only if 1) it has a gold answer covered by OPR but missed by the reader 2) and it has a predicted gold answer whose supporting passages cover no other gold answer; a successful attack on a targeted question means that a missed answer is recovered after removing a subset of supporting passages of some predicted answer 5 without removing any supporting passage of the other gold answers. There are 179 targeted questions; for 43.6% of them, we successfully recovered at least one missed gold answer. Figure 3 shows the success rate breakdown on the number of answers covered by the reader's input, indicating that predictions tend to be brittle when the reader is fed with many diverse supporting passages. One possible explanation of the pathological dependencies is that the reader implicitly compares 5 Removed passages were replaced with the same number of top-ranking passages that cover no gold answer, so that the number of passages fed to the reader remained unchanged. the validity of answer candidates and predicts the most likely ones. However, for 40.0% of successfully attacked questions, according to OPR, supporting passages of recovered missed answers are more relevant than those removed passages of predicted answers. Notably, Min et al. (2020b) also had a similar observation on another rerank-then-read pipeline, i.e., it is hard to argue that the predicted answers are more likely than the missed ones. To avoid the issues faced by the rerank-then-read framework, we propose a recall-then-verify framework, which separates the reasoning process of each answer so that answers (1) can be individually distributed with maximum supporting passages allowed on the same hardware (2) and are predicted mainly based on their own evidence. Figure 2 shows our framework. Specifically, we first guess possible answers based on retrieved passages using an answer recaller, an evidence aggregator then aggregates evidence for each answer candidate, and finally, an answer verifier verifies each candidate and outputs valid ones. Our answer recaller, based on T5, is trained to predict all gold answer(s) in sequence (separated by a [SEP] token) from each retrieved positive passage p  B that cover some gold answer(s). We also train the recaller to predict the irrelevant token given a negative passage so that the recaller can filter out negative candidates; the number of negatives per positive used for training is denoted as  neg . The set of answer candidates recalled during inference is denoted as A = {  a 1 ,  a 2 , ...,  a m } . Though a passage may not contain strong enough evidence to support an answer, by exploiting semantic clues in the question and the passage (e.g., the answer type), it is sufficient for even a weak model to achieve high recall. However, this is at the cost of low precision, which necessitates answer verification based on more supporting passages. We aggregate evidence for each answer candidate from retrieved passages, which can be formulated as a reranking task, i.e., to rerank retrieved passages according to their relevance to a question-candidate pair, and select top-ranking ones for answer verification. Our evidence aggregator resembles OPR: for a specific candidate  a i , we encode the question-candidate pair with the retriever's question encoder; a passage p is ranked higher than another passage p (cid:48) if and only if 1) p covers  a i while p (cid:48) does not 2) or both p and p (cid:48) cover or fail to cover  a i but the semantic vector of p is closer to that of the question-candidate pair. We denote the topk relevant passages of  a i as E i . Given a candidate  a i and its evidence E i , our answer verifier, based on T5-3b, predicts whether  a i is valid, using the fusion-in-decoder method from (Izacard and Grave, 2021b). Each passage from E i is concatenated with the question and the candidate, and is encoded independently; the decoder then attends to the representations of all passages and is trained to produce the tokens right or wrong depending on whether the encoded candidate is valid or not 6 . During inference, we compute the validity score of a candidate by taking the normalized probability assigned to the token right: P ( a i is valid ) = exp( logit ( right | q,  a i , E i )) (cid:80) t { right , wrong } exp( logit ( t | q,  a i , E i )) (1) Candidates with their validity scores higher than a threshold  will be produced as final predictions. We conducted experiments on two multi-answer QA datasets, whose statistics are shown in Table 3. 6 We have tried other verbalizers such as yes and no, but found no significant difference. WEBQSP (Yih et al., 2016) is a semantic parsing dataset for knowledge base question answering, where answers are a set of entities in Freebase. Following (Min et al., 2021), we repurposed this dataset for textual QA based on Wikipedia 7 . AMBIGQA (Min et al., 2020b) originates from NQ (Kwiatkowski et al., 2019), where questions are annotated with equally valid answers from Wikipedia. We compare our recall-then-verify system with two state-of-the-art rerank-then-read systems. REFUEL (Gao et al., 2021) selects 100 top-ranking passages from 1,000 retrieved passages, and predicts answers with a reader based on BART large (Lewis et al., 2020). It also has a round-trip prediction mechanism, i.e., to generate disambiguated questions based on predicted answers, which are re-fed to the reader to recall more answers. JPR (Min et al., 2021) is a passage reranker which jointly models selected passages. With improved reranking performance, Min et al. (2021) selected only 10 passages from 100 retrieved passages, and used a reader based on T5-3b which is much larger and more powerful than REFUEL 's reader, while requiring no more memory resources than REFUEL . Our retrieval corpus is the English Wikipedia from 12/20/2018. We finetuned the dense retriever from (Izacard and Grave, 2021a) on each multi-answer dataset. The answer recaller and the answer verifier were initialized with T5-3b; both were pre-trained on NQ and then finetuned on each multi-answer dataset. neg was 0.1 when finetuning the recaller. We retrieved 100 passages for a question, and verified each candidate with k =10 passages. The threshold  for verification was tuned on the dev set based on the sum of F1 scores on all questions (F1 (all)) and questions with multiple answers (F1 (Multi)); the best  on WEB QSP/A MBIGQA are 7 Our train/dev split on WEBQSP is different from Min et al. (2021)'s, as their split was not publicly available in the period of this work. 0.8/0.5, respectively. Experiments with different model choices for the recaller and different values of  neg , k and  are shown in Section 6.5. Please refer to the Appendix for more implementation details. Memory Constraint: Min et al. (2021) considered a fixed hardware and trained a reader with the maximum number of passages. We follow this memory constraint, under which a reader/verifier based on T5-3b can encode up to 10 passages each of length no longer than 360 tokens at a time. Due to candidate-aware evidence aggregation and a fixed sufficient number of passages distributed to each candidate, our recall-then-verify framework can make use of most retrieved supporting passages (see our improvements over OPR in Figure 1b). With a higher level of evidence usage, our recall-then-verify system outperforms state-of-the-art rerank-then-read baselines on both multi-answer datasets, which is shown by Table 4. Though focused on multi-answer questions, our framework is also applicable to single-answer scenario and achieves state-of-the-art results on NQ. Please refer to the Appendix for more details. In this section, we present ablation studies on AMBIGQA. Please refer to the Appendix for results on WEBQSP, which lead to similar conclusions. Model Choices for the Answer Recaller As shown by Table 5, though T5-base is commonly recognized as a much weaker model than T5-3b, a recaller based on T5-base can achieve a high coverage of gold answers, leading to competitive end-to-end performance on the test set. Necessity of Verification To investigate whether the recaller has the potential to tackle multi-answer questions alone, we tuned the precision of the re-Recaller Verifier Dev Test T5  neg  |A| # Hit Recall Precision F1 F1 3b 10 -2.2 2068/1237 54.4/39.0 39.6/38.3 41.1/34.3 3b 5 -3.3 2206/1328 56.8/41.7 36.6/36.5 39.7/34.7 3b 1 -7.2 2714/1690 65.7/50.9 22.2/22.7 29.7/28.2 3b 0 -51.2 3364 / 2211 73.5 / 61.9 3.8/4.6 6.8/8.2 3b 0.1 -28.7 3288/2141 72.6/60.5 6.3/7.5 10.9/12.7 base 0.1 -48.4 3156/2056 70.0/57.9 3.3/4.1 6.0/7.5 3b 0.1 0.5 28.7 2046/1184 55.2/37.8 57.7 / 56.4 52.1 / 41.6 46.2 / 37.1 base 0.1 0.6 48.4 2051/1181 54.8/37.6 55.4/54.3 50.8/40.8 45.8/37.0 Table 5: Performance of recallers on AMBIGQA, trained with different models and  neg . caller by varying  neg . As shown in Table 5, with increased  neg , the recaller learns to recall answers more precisely but still significantly underperforms the overall recall-then-verify system. It is likely that the recaller is trained on false positive passages, which may mislead the recaller to be overconservative in filtering out hard negative passages. By contrast, using more evidence for verification is less likely to miss true positive evidence if there is any for a candidate, thus not prone to mislead the verifier. Reducing Answer Candidates Though only using our recaller for multi-answer QA falls short, the recaller can be trained to shrink down the number of candidates so that the burden on the verifier can be reduced. As shown by Table 5, a small value of  neg helps reduce answer candidates without significantly lowering recall. Effect of k Figure 4 shows the benefit of using more evidence for verification. As k increases from 1 to 10, there is a significant boost in F1 scores. Effect of  As shown by Figure 4a, the balance between recall and precision can be controlled by  : a lower  leads to higher recall and may benefit performance on questions with multiple answers. With k =10, our system outperforms the previous state-of-the-art system for a wide range of  . As shown by Figure 4b, under the best setups ( k =10,  =0.5), our system predicts 31.7% and 34.1% more gold answers than the system using OPR on all questions and questions with multiple answers, respectively. Dependencies among Answers Despite being candidate-aware, aggregated evidence E can also include supporting passages of some other gold answer(s). We therefore investigated how answer verification is affected by the evidence of the other gold answers. Specifically, we attacked the verifier as follows: a question-candidate pair is a target if and only if 1) the candidate  a i is a gold answer and 2) the aggregated evidence E i includes at least one supporting passage of some other gold answer(s) that do not cover  a i ; we removed an arbitrary subset of supporting passages of the other gold answer(s) at a time 8 without removing any supporting passages of  a i , and recorded the worst changes of the predicted validity scores of  a i . As shown by Figure 5, the changes are small, indicating that missed gold candidates with low scores are not mainly suppressed by some other answer(s), and that predicted gold candidates with high scores are verified 8 Removed passages were replaced with the same number of top-ranking passages that cover no gold answers. mainly based on their associated evidence. Among 3,288 recalled gold answers on the dev set of AMBIGQA, the answer verifier misses 1,242 of them and outputs 1,323 wrong predictions. We manually analyzed 50 random samples, 25 of which are missed gold answers and 25 are wrong predictions. Table 6 reports our analysis. For 76% of missed gold answers, our evidence aggregator actually aggregates straightforward true positive evidence. Among these missed answers with straightforward evidence, 58% of them have validity scores higher than 0.2 but lower than the threshold 0.5. We attacked the verifier on missed gold answers with their validity scores below 0.2 as in Section 6.5.2, and found that the maximum change of predicted scores on average is small (+0.04), indicating that the low scores can not be attributed to the negative distraction by the other gold answer(s). We conjecture that, as it is difficult even for human annotators to find all valid answers to an open-domain question (Min et al., 2020b), the verifier was trained to refute false negative candidates, resulting in unexpected low scores on some straightforward valid answers. Notably, 80% of our wrong predictions turn out to be false negatives: 52% of wrong predictions are semantically equivalent to some annotated answer but are superficially different (Si et al., 2021); 28% of wrong predictions are unannotated false negatives. Therefore, it is likely that our system is underrated. In this section, we analyze the time complexity of our framework during inference, make comparisons with the state-of-the-art rerank-then-read framework JPR, and show how to reduce the computation cost of a recall-then-verify system. For convenience, we denote the encoder length and decoder length as L p and L a , respectively. Recaller vs. Reranker The time complexity of answer recalling is O ( |B|  ( L 2 p + L a  L p + L 2 a )) , while that of passage reranking is O ( |B|  L 2 p + k  |B| L p + k 2 ) . As encoding dominates computation cost (whose time complexity is O ( |B|  L 2 p ) ), given the same model size and |B| , the time complexity of answer recalling and passage reranking is at the same level. Verifier vs. Reader The time complexity of answer verification is O ( |A| ( k  L 2 p + k  L p )) , while that of the reader is O ( k  L 2 p + L a  k  L p + L 2 a ) . As the reader decodes a sequence of length L a in an autoregressive way, while the decoding length of the verifier is only 1, the ratio between the inference time of the verifier and that of the reader should be much less than |A| . Evidence Aggregator Evidence aggregation is significantly faster than answer recalling and verification, as representations of Wikipedia passages are pre-computed. The time complexity is O ( |A|  ( L 2 p + |B|  log k )) where L 2 p comes from encoding a question-candidate pair with the retriever's question encoder, and |B|  log k comes from selecting the topk relevant passages for a candidate. One can adjust the computation cost of a recall-then-verify system, depending on how much inference efficiency is valued over precision and recall, by (1) choosing a recaller model of proper time complexity 9 , (2) tuning  neg to adjust the expected number of candidates |A| needed for verification, (3) or tuning the number of passages k used for verification. Table 7 shows QA performance and inference efficiency of our systems with different configurations. Replacing T5-3b with T5-base for the recaller is significantly faster in answer recalling, but is much less precise and produces more answer candidates with the same  neg , which increases the burden on the verifier and thus may fail to reduce the overall computation cost if  neg is not raised. By also increasing  neg and choosing a smaller k , as shown by the last row of Table 7, the overall time needed to answer a question on the dev set of AMBIGQA can be reduced to 1.88 sec on a single V100 GPU while also obtaining state-of-the-art F1 scores (50.7/38.2). By contrast, the rerank-then-read system from Min et al. (2021) using a T5-base JPR ( k =10) and a T5-3b reader is estimated to take 1.51 sec per question 10 with F1 scores of 48.5/37.6. In this paper, we empirically analyze the problems of the rerank-then-read framework for open-domain multi-answer questions, and propose the recall-then-verify framework, which separates the reasoning process of each answer so that 1) we can have a higher level of evidence usage 2) and predicted answers are mainly based on associated evidence and are more robust to distraction by evidence of the other gold answer(s), 3) while also leveraging large models under the same memory constraint. On two multi-answer datasets, our framework significantly outperforms rerank-then-read baselines with new state-of-the-art records. This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604) and the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2019GQG1 and 2020GQG0005. pro-10 The average inference time of JPR from Min et al. (2021) is independent of its parameters given a fixed number of encoded tokens and a fixed decoder length, which can be estimated with a randomly initialized JPR. The average inference time of JPR's reader was estimated with OPR's reader. pose a recall-then-verify framework that will hopefully benefit information-seeking users with an enhanced ability of comprehensive exploitation of evidence from a large-scale corpus. As our predictions are verified with textual knowledge, our system itself would not raise new significant ethical concerns. All the datasets as well as the retrieval corpus in our experiments have been widely used for research purposes, and to our knowledge, do not have any attached privacy and ethical issues.
abstain abstain other objective result abstain abstain abstain abstain objective objective objective objective objective objective objective abstain abstain abstain abstain abstain abstain abstain method abstain other objective objective result method objective objective other other other other other other other other other method objective other other abstain other objective method method method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method method abstain method abstain method result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain result result result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain other abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other objective other other abstain abstain abstain abstain abstain