To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively. In this paper, we study a hybrid approach for leveraging the strengths of both models. We apply novel techniques to enhance both extractive and generative readers built upon recent pretrained neural language models, and find that proper training methods can provide large improvements over previous state-of-the-art models. We demonstrate that an hybrid approach by combining answers from both readers can effectively take advantages of extractive and generative answer inference strategies and outperform single models as well as homogeneous ensembles. Our approach outperforms previous state-of-the-art models by 3 . 3 and 2 . 7 points in exact match on NaturalQuestions and TriviaQA respectively. Open-domain question answering (QA) has been a long standing problem in natural language understanding, information retrieval, and related fields (Chen and Yih, 2020). An typical open-domain QA system follows the retrieval-reader framework (Chen et al., 2017; Guu et al., 2020; Karpukhin et al., 2020), where the relevant passages are first retrieved from a large text corpus, and a reader module then navigates multiple passages for answer inference. In this work, we study two paradigms of reader modules, i.e. extractive (Karpukhin et al., 2020; Guu et al., 2020) and generative (Lewis et al., 2020; Izacard and Grave, 2021) readers. The extractive reader extracts contiguous spans from the retrieved passages whereas the generative reader sequentially decodes the answer string which might not be contained in the retrieved passages. Recent work on open-domain QA (Karpukhin et al., 2020; Guu et al., 2020; Lewis et al., 2020; Izacard and Grave, 2021) explores either an extractive reader or a generative reader exclusively. We hypothesize that extractive and generative readers adopt different answer inference strategies, thus a hybrid extractive/generative reader can be a better option for open-domain QA tasks. As shown in Figure 1, compared with prediction agreement among only generative or extractive readers (top-left and bottom-right), the cross prediction agreement between extractive and generative readers (bottom-left) is relatively low (< 50% ). It indicates that answers produced by those two types of models are different and they can be complementary to each other. Therefore, we propose a hybrid reader approach, UnitedQA, which is a simple ensemble approach to combine the predictions from extractive and generative readers. It achieves state-of-the-art results on NaturalQuestions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017). In UnitedQA, the extractive reader (UnitedQA-E) and generative reader (UnitedQA-G) are built upon the pretrained language models, ELECTRA (Clark et al., 2020) and T5 (Raffel et al., 2020), respectively. For the UnitedQA-E, we adopt a weakly-supervised training objective to address the noisy supervision issue caused by the heuristics-based labeling and incorporate the posterior differential regularization (PDR) (Cheng et al., 2021) to improve the model robustness. The UnitedQA-G follows the T5 Fusion-in-Decoder (FID) (Izacard and Grave, 2021) and we make two improvements: first, we add a group of attention bias parameters into the decoder cross-attention block to feature the ranking information of retrieved contexts; second, we add the adversarial training (Ju et al., 2019; Jiang et al., 2020; Pereira et al., 2021) to improve the model generalization ability. The experimental results highlight the effec-G-1 G-2 G-3 E-1 E-2 E-3 G1 G2 G3 E1 E2 E3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Figure 1: Pairwise prediction agreement ratio. tiveness of the simple hybrid approach of UnitedQA. With both improved extractive and generative readers, UnitedQA sets new state-of-the-art results on two popular open-domain QA datasets, i.e. 54 . 7 and 70 . 3 in exact match on NaturalQuestions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017), respectively. It is worth noting that our UnitedQA model not only outperforms each single model but also brings more pronounced improvements over homogeneous ensembles of either extractive or generative readers. Last, based on our analyses, UnitedQA-E and UnitedQA-G have advantages in different cases, suggesting they may use different reasoning strategies. In this section, we present the overall pipeline of the UnitedQA system, which consists of three components: Retrieval , Reading , and Re-ranking . First, the retrieval module fetches a list of relevant passages from a Wikipedia dump for a given question. Then, the module of hybrid readers produces answer candidates from the set of retrieved passages. Last, the re-ranking module combines the answer candidates with linear interpolation and produce the final answer. Retrieval Following Karpukhin et al. (2020), we consider two methods, BM25 and dense passage retrieval (DPR), for retrieving the support passages for a given question. For BM25, passages are encoded as bag of words (BOW), and inverse document frequencies are used as the ranking function. For DPR, passages and questions are represented as dense vectors based on two BERT (Devlin et al., 2019) models. The relevance score is then computed based on the dot production between the query and passage vectors. In this paper, we adopt the same implementation as Karpukhin et al. (2020) for retrieving passages. Specifically, the English Wikipedia dump from Dec. 20, 2018 is used as the source documents for retrieval, with the removal of semi-structured data, such as tables or lists. Each document is split into disjoint 100-word passages as the basic retrieval unit. The top-100 passages are then passed for reading. Reading We combine the generative reader and the extractive reader to produce answer candidates over the retrieved passages. Here, we only give a high-level description of our approach. More details regarding our improved extractive and generative models are presented in 2.1 and 2.2 respectively. The generative reader is based on a sequence-to-sequence model pre-trained in a forward-generation fashion on a large corpus, i.e. T5 (Raffel et al., 2020). Similar to Izacard and Grave (2021), the model takes the question and its relevant passages as input, and then generates the answer string token by token. Specifically, the concatenation of all retrieved passages and the corresponding question is used as the encoder input. Then, the decoder performs reasoning over the concatenation of all evidence through an attention mechanism. Following state-of-the-art extractive QA models (Devlin et al., 2019; Karpukhin et al., 2020), our extractive reader is based on a Transformer neural network pre-trained with a cloze style self-supervised objective, i.e. ELECTRA (Clark et al., 2020). Here, a pair of a given question and a support passage is jointly encoded into neural text representations. These representations are then used to define scores or probabilities of possible answer begin and end positions, which are in turn used to define probabilities over possible answer spans. Finally, the answer string probabilities are based on the aggregation over all possible answer spans from the entire set of support passages. In 2.1.2, we give the problem definition of open-domain QA for extractive reader. Then, we detail the improvements of UnitedQA-E in 2.1.2. Given a question q and a set of K retrieved passages p 1 , . . . , p K , a text encoder produces contextualized representations: h k 1 , ... h kT  R n for the question-passage pair (q , p k ) in the form of  [CLS] question [SEP] passage [SEP] , where [CLS] and [SEP] are special tokens for encoding inputs, T is the maximum sequence length of the input text, and h ki indicates the contextualized embedding of the i -th token in (q , p ). The extractive reader computes the span-begin score of the i -th token as s b ( i k ) = w Tb h ki using a weight vector w b  R d . The span-end score s e ( j k ) is defined in the same way. Thus, the probabilities of a start position i k and an end position j k are P b ( i k ) = exp( s b ( i k )) Z b , P e ( j k ) = exp( s e ( j k )) Z e , where Z b , Z e are normalizing factors defined by the corresponding probability space. The probability of an answer span from i k to j k is defined as P s ( i k , j k ) = P b ( i k ) P e ( j k ) . Here, we consider two probability spaces, passage level and multi-passage level , with the only difference in the computing of Z b , Z e . Specifically, the passage-level probability of each answer begin and end is computed by normalizing all possible positions in the respective passage, i.e. Z b = Z kb = (cid:80) I k  NULL exp( s b ( i )) , Z e = Z ke = (cid:80) I k  NULL exp( s e ( j )) , where I k is the set of all possible positions from the k -th passage and NULL indicates special positions if p k does not support answering the question. Similarly, the multi-passage level probability is computed by normalizing over each answer positions across all K relevant passages, i.e. Z b = Z  b = (cid:80) k (cid:80) I k exp( s b ( i )) , Z e = Z  e = (cid:80) k (cid:80) I k exp( s e ( j )) , respectively. Since there are usually multiple plausible mentions for open-domain QA, during training, it is typical to maximize either the marginal log-likelihood (MML) of all correct spans (Karpukhin et al., 2020) or the log-likelihood of the most likely correct span (HardEM) (Min et al., 2019). During inference, the prediction is made based on the candidate answer string score, obtaining as P a ( y ) = (cid:80) ( i,j ) Y P s ( i, j ) , where Y is the set of spans corresponding to the answer string y . In addition to better text representations from Clark et al. (2020), we consider two methods for improving the training of the extractive reader. Multi-objective for Weakly-supervised QA The multi-objective formulation is introduced in Cheng et al. (2020) for improving weakly supervised document-level QA. Different from Cheng et al. (2020) where only MML is considered for the multi-objective formulation, we found combining HardEM with MML is more effective for open-domain QA based on our experiments (4.1). Specifically, we combine a multi-passage HardEM loss with K passage-level MML losses over a batch of K passages LEXT = log max ( i,j ) P Ms ( i, j ) + 1 K (cid:88) k log (cid:88) ( i k ,j k ) P Ps ( i k , j k ) , (1) where P Ms , P Ps is the multi-passage level and passage level span probabilities respectively. Posterior Differential Regularization Due to the noisy supervision for open-domain QA (Chen et al., 2017), we investigate the posterior differential regularization (PDR) (Cheng et al., 2021) to improve the robustness of the extractive reader. Different from Cheng et al. (2021) where only clean supervision setting is considered, in this work, we apply PDR to the weakly supervised open-domain QA scenario. Given it is computationally expensive to enumerate all possible spans, we apply two separate regularization terms for the begin and end probabilities at the multi-passage level, respectively, LPDR = D ( P b ( i ) | P (cid:48) b ( i )) + D ( P e ( j ) | P (cid:48) e ( j )) , (2) where D ( | ) is the squared Hellinger distance, and P (cid:48) b , P (cid:48) e are the probabilities of start and end positions with additive input noise to the token embeddings. Specifically, we sample noise vectors (cid:15) 1 , . . . , (cid:15) T from N (0 , c 2 I ) , and add them to the token embeddings as the noisy input, i.e. v 1 + (cid:15) 1 , . . . , v T + (cid:15) T , where c is fixed to 1e  3 throughout our experiments. Based on this, the overall training objective for the extractive reader is L 1 = LEXT +  LPDR , (3) where  is a regularization scalar hyperparameter. Here, we first formally define the setup of generative reader for open-domain QA in  2.2.1 and then present our improvements in  2.2.2. Given a question q and a set of K retrieved passages p 1 , . . . , p K , the encoder model encodes each ( q , p k ) pair independently, and produces contextualized representation for each token: h ki  R d for the i -th token of the k -th pair. The decoder then performs attention over the concatenation of the representations of all the retrieved passages, and generates the answer string. Let x denote the input of the question and all retrieved passages x = (cid:0) ( q , p 1 ) , ..., ( q , p K ) (cid:1) , and y the answer string with its tokens as ( y 1 , ..., y N ) . The generative reader is trained to maximize a sequence-to-sequence objective for a given ( x , y ) , L ( x , y ;  ) = N (cid:88) i log P  ( y i | x , y 1: i  1 ) , (4) where  is the model parameter. During inference, a greedy decoding is used to produce the answer. Decoder Attention Bias The decoder in the T5 transformer model adopts a cross-attention mechanism to compute attention scores between the decoding answer tokens and all the retrieved passage tokens. Specifically, let y i  R d be the query vector of the i -th decoding token 1 , and m kj  R d be the key vector of the j -th token in ( ( q ) , p k ) . The multi-head cross-attention scores in T5 (Raffel et al., 2020) s ki,j is calculated as s ki,j = MultiHeadAtt ( y i , m kj )  R | Head | (5) where | Head | is the number of attention heads. However, it doesn't capture the relevance information of retrieved passages into the reader in (5). To add the relevance feature into the attention block, we revise (5) by incorporating the attention bias s ki,j = MultiHeadAtt ( y i , m kj ) + b k , (6) where b k  R | Head | is a trainable attention bias vector for all the tokens in the k -th retrieved passage. In the experiments, the maximum retrieved passages is by default set to 100 . Thus, the decoder attention bias introduces additional 100  | Head | parameters for each layer. Adversarial Training Adversarial training creates adversarial examples by adding small perturbations to the embedding layer. Assuming the word(-piece) embedding layer is parameterized by a matrix V  R | V | d , | V | is the vocabulary size, and d 1 we omit the layer notation for simplification Dataset Train Dev Test NQ 79168 8757 3610 TriviaQA 78785 8837 11313 EffcientQA 1800 Table 1: Number of questions in each QA dataset. is the embed-dimension. The adversarial embedding matrix  V can be obtained by g V =  VL ( x , y ;  ) , (7)  V = V + SG ( (cid:15)g V / || g V || 2 ) , (8) where SG (  ) is the stop-gradient operation. We use the adversarial embedding matrix  V to replace the original V in model parameters  , and obtain   . Thus the adversarial loss can be calculated as LAT ( x , y ;  ) = L ( x , y ;   ) . Therefore, the overall training objective of the generative reader is where  = 0 . 5 ,  = 0 . 5 in all of the exepriments. The UnitedQA system combines outputs from both extractive and generative models for a given question during inference. Since the output spaces of extractive and generative models are different, we use a simple linear interpolation based on best predictions from each model 2 . Denote the predicted strings from M extractive and N generative models as y E 1 , ..., y EM and y G 1 , ..., y GN , respectively. The hybrid prediction y  is obtained by argmax y Y  M (cid:88) m =1 1 ( y, y Em ) +  N (cid:88) n =1 1 ( y, y Gn ) , (11) where Y is the set of all predicted strings, 1 ( y, y (cid:48) ) is an indicator function and  = 0 . We use two representative QA datasets and adopt the same training/dev/testing splits as in previous 2 We have also tried a few more complex approaches for combining the extractive and generative models. For example, we first train an extractive model, and then append the top-k answer strings from the extractive model at the end of the input for training a generative model. None of them is as good as the simple ensemble approach. work (Lee et al., 2019; Karpukhin et al., 2020). Both datasets (see Table 1 for statistics) have been heavily studied in recent work (Lee et al., 2019; Min et al., 2019; Karpukhin et al., 2020; Guu et al., 2020). We follow the standard evaluation protocol and use exact match (EM) as the evaluation metric. NaturalQuestions (Kwiatkowski et al., 2019) is composed of questions by real users to Google Search, each with answers identified by human annotators in Wikipedia. The open-domain version of NaturalQuestions (Lee et al., 2019) only consider questions with short answers, i.e. answers with less than 5 tokens. In the NaturualQuestions, the questions are considered to be more information seeking given that the question askers didn't know the answer beforehand. In addition, we use another evaluation set, i.e. the dev set introduced recently by the EfficientQA competition (Min et al., 2021), which is constructed in the same way as the original NaturalQuestions dataset. TriviaQA (Joshi et al., 2017) contains trivia question-answer pairs that were scraped from the web. Different from NaturalQuestions, the questions here are written with known answers in mind. Specifically, the unfiltered set has been used for developing open-domain QA models. Implementation details For a fair comparison, we use the same retrieval module as Karpukhin et al. (2020) for NaturalQuestions and TriviaQA to mitigate the impact of retrieval difference. Specifi-cally, we use DPR (single) for NaturalQuestions and BM25+DPR (multi) for TriviaQA because of their best end-to-end performance (Karpukhin et al. 2020). For all the experiments, we use 8 and 16 V100-32GB for base and large model training respectively. We train our models with Adam optimizer of a linear scheduler with a warmup raito of 0.1. The extractive models are trained for up to 8 epochs with a learning rate of 2e  5 and a batch passage size per question of 16 . The generative models are trained for up to 10 epochs with a learning rate of 1e  4 , a batch size of 64 , and 100 retrieved passages per question for model training. We select  in { 4 , 8 } . After the best configuration is selected based on the dev set, we run our best models 3 times independently with different random seeds and report the median performance on the test set. We also report ensemble results which are based on the linear interpolation over answer predictions from the 3 models. Single Model Results: We first compare our models to two recent models, REALM (Guu et al., 2020) and RAG (Lewis et al., 2020), which are first pre-trained with different retrieval augmented objectives and then fine-tuned for open-domain QA. In addition, we include as baselines DPR (Karpukhin et al., 2020) and T5-FID (Izacard and Grave, 2021), both of which are based on the same retriever as ours. As shown in Table 2, both our extractive and generative models achieve new state-of-the-art results for both studied datasets. Compared with the recent state-of-the-art extractive model (DPR), our base model leads to pronounced 15% relative improvements for both NaturalQuestions ( +6 . 2 absolute improvement) and TriviaQA ( +8 . 4 absolute improvement). More importantly, UnitedQA-E base achieves comparable or even better performance with regard to generative models of larger size, i.e. RAG and T5-FID base . It highlights the importance of proper training strategies for open-domain QA models. Hybrid Model Results: In order to evaluate the advantage of the hybrid of the extractive and generative models (UnitedQA), we include two homogeneous ensemble baselines, one consisting of only extractive readers (UnitedQA-E++) and the other ensemble of exclusively generative models (UnitedQA-G++). For homogeneous ensemble cases, the three-way majority prediction is used. For the hybrid of extractive and generative readers, we select a three-model combination from the set of three generative and three extractive models based on the dev set. We observed that combining predictions from two generative models and one extractive model results in the best hybrid model for both datasets. As expected, all ensemble models show an improvement over their single model counterparts. However, the two homogeneous ensemble baselines, UnitedQA-E++ and UnitedQA-G++, only provide marginal gains over the corresponding best single models. The significant improvement brought by our proposed hybrid approach indicates the benefit of combining extractive and generative readers for open-domain QA. Discussion: Although the proposed hybrid approach has been shown to be highly effective for open-domain QA, we point out that the improved performance comes with increased computational cost. The best combination requires approximately three times the computational cost of a single generative model. Therefore, it would be interesting to explore more efficient hybrid methods, such as effective parameter sharing strategies or unified formulations. Another interesting future direction is to explore customized compression approaches for reducing the model size of retriever and reader separately or jointly through pruning (Han et al., 2016), quantization (Hubara et al., 2018), and knowledge distillation (Hinton et al., 2015). Specifically, given that the hybrid model is more effective, it is likely that a student model can learn more effectively from a hybrid teacher model via knowledge distillation for open-domain QA. In this section, we first carry out ablation study on the extractive and generative model improvements. Moreover, we aim to take a deeper look and understand the difference between the two models. In Table 3, we present ablation experiments on the effectiveness of different textual representations and methods for improving the extractive model UnitedQA-E base . Here, we focus on base models, i.e. BERT base and ELECTRA base . Note that the row UnitedQA-E base is the corresponding base model reported in Table 2. Compared with the MML-based multi-objective (Cheng et al., 2020), we find that a new multi-objective with HardEM at the multi-passage level and MML at the passage level is more effective for open-domain QA. In addition to the multi-objective training, there is a noticeable improvement brought by the regularization method (PDR) which indicates the importance of proper regularization for learning with noisy supervision. Last but not least, the large improvement of ELECTRA over BERT indicates the importance of deriving better text representations for weakly supervised NLP problems. For the UnitedQA-G, we present the ablation study on analyzing the effectiveness of decoder attention bias component and adversarial training mechanism in Table 4. Both techniques contribute to decent improvements over T5-FID with more pronounced gains brought by adversarial training. Here, we vary the number of retrieved passages during inference and report the evaluation results in terms of end-to-end QA exact match score of UnitedQA-E and UnitedQA-G along with the corresponding topk retrieval accuracy. The results are summarized in Table 5. As expected, when the number of retrieved passages increases, both topk retrieval accuracy and the end-to-end QA performance improve. However, there is a noticeable gap between the improvement of retrieving more passages (i.e., recall) and that of the corresponding end-to-end QA performance, especially for the extractive reader. This is likely caused by additional noise introduced with improved retrieval recall. Specifically, only half of the retriever improvement can be effectively utilized by the extractive model while the generative model can benefit more from retrieving more passages. This suggests that by concatenating all passages in vector space, the generative model are more effective in de-noising in comparison to the extractive model. Following Lewis et al. (2021), we carry out a breakdown evaluation of model performance over the NaturalQuestions and TriviaQA test sets. Given their superior performance, we again only consider our improved extractive and generative models, i.e. UnitedQA-E large and UnitedQA-G respectively. The evaluation is summarized in Table 6. In comparison to their corresponding overall performance, both the extractive and generative models achieve much better performance on the Overlap categories ( i.e. Question Overlap and Answer Overlap) for both NaturalQuestions and TrivaQA, which indicates that both models perform well for question and answer memorization. Different from question and answer memorization, there is a pronounced performance drop for both models on theAnswer Overlap Only category where certain amount of relevance inference capability is required to succeed. Lastly, we see that both extractive and generative models suffer some significant performance degradation for the No Overlap column which highlights model's generalization evaluation. Nevertheless, the extractive model demonstrate a better QA generalization by achieving a better overall performance on the No Overlap category for both datasets. Here, we conduct analyses into prediction errors made by the extractive and generative models based on automatic evaluation. For this study, we use the EfficientQA dev set (Min et al., 2021) which is constructed in the same way as the original NaturalQuestions dataset. Specifically, we group prediction errors into three categorizes: 1) common prediction errors made by both the extractive and generative models, 2) prediction errors made by the extractive model, 3) prediction errors produced by the generative model. In the following, we first carry out a manual inspection into the common errors. Then, we compare the prediction errors made by extractive and generative models, respectively. First of all, there is an error rate of 29% of those consensus predictions made by both extractive and generative models according to the automatic evaluation. Based on 30 randomly selected examples, we find that around 30% of those predictions are actually valid answers as shown in the top part of Table 7. In addition to predictions that are answers at different granularity or semantically equivalent ones, some of those prediction errors are likely caused by the ambiguity in questions. As the given example in Table 7, based on the specificity, the model prediction is also a valid answer. This high-Dataset Model Total Question Overlap No Question Overlap Answer Overlap Answer Overlap Only No Overlap NQ UnitedQA-G 52.3 72.2 40.5 62.7 45.4 34.0 UnitedQA-E 51.8 69.4 41.5 60.1 45.1 37.6 TriviaQA UnitedQA-G 68.6 88.4 62.5 78.1 69.6 44.5 UnitedQA-E 68.9 89.3 62.7 78.6 70.6 44.3 Table 6: Breakdown evaluation on NaturalQuestions (NQ) and TriviaQA based on test splits defined in (Lewis et al., 2021). lights the limitation of the current evaluation metric, which does not accurately estimate the existing open-domain QA system capabilities. As shown in the bottom part of Table 7, most of representative errors are due to the confusion of related concepts, entities or events that are mentioned frequently together with the corresponding gold answers. Next, all questions from the dev set are categorized based the WH question word, i.e. what, which, when, who, how, where . We then report the relative performance change of each WH category for both extractive and generative models over their corresponding overall prediction accuracy in Figure 2. First, it is easy to see that both extractive and generative models achieve the best performance for entity related who questions, which is likely to be the result of high ratio of samples of this type seen during training. In contrast, the answers to what questions can play a much richer syntactic role in context, making it more difficult for both extractive and generative models to perform well. Interestingly, the generative model exhibits the strength for temporal reasoning, whereas the extractive model does not. This difference suggests that it is worth exploring better temporal modeling strategies to improve the extractive model in the future. Open-domain QA Open-domain QA requires a system to answer questions based on evidence retrieved from a large corpus such as Wikipedia (Voorhees, 2000; Chen et al., 2017). Recent progress has been made towards improving evidence retrieval through both sparse vector models like TF-IDF or BM25 (Chen et al., 2017; Min et al., 2019), and dense vector models based on BERT (Lee et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Qu et al., 2021). Generally, the dense representations complement the sparse vector methods for passage retrieval as they can potentially give w h a t w h i c h w h e n w h o h o w w h e r e 0.10 0.05 0.00 0.05 0.10 R e l a t i v e A cc u r a c y Generative w h a t w h i c h w h e n w h o h o w w h e r e Extractive Figure 2: Relative accuracy of different WH questions. high similarity to semantically related text pairs, even without exact lexical overlap. Unlike most work focusing on a pipeline model, Lee et al. (2019) propose a pre-training objective for jointly training both the retrieval encoder and reader. It is further extended by Guu et al. (2020) with a dynamic update of the passage index during the training. Instead, in this work, we focus on a hybrid reader approach for open-domain QA. By simply combing answer predictions from extractive and generative models, our UnitedQA achieves significant improvements over state-of-the-art models. Reading Comprehension with Noisy Labels There has been a line of work on improving distantly-supervised reading comprehension models by developing learning methods and model architectures that can better use noisy labels. Most of them focus on the document-level QA, where all paragraphs share the same document context. Clark and Gardner (2018) propose a paragraph-pair ranking objective for learning with multiple paragraphs so that the model can distinguish relevant paragraphs from irrelevant ones. In (Lin et al., 2018), a coarse-to-fine model is proposed to handle label noise by aggregating information from relevant paragraphs and then extracting answers from selected ones. Min et al. (2019) propose a hard EM learning scheme where only passage-level loss is considered for document-level QA. More recently, different probabilistic assumptions with corresponding training and inference methods are examined in (Cheng et al., 2020) again for document-level QA with distant supervision. In our work, we further extend the multi-objective formulation proposed in (Cheng et al., 2020) with the hard EM learning (Min et al., 2019) for enhancing extractive open-domain QA, where the input passages are given by a retrieval model and are typically from different documents. In this study, we propose a hybrid model for open-domain QA, called UnitedQA, which combines the strengths of extractive and generative readers. We demonstrate the effectiveness of UnitedQA on two popular open-domain QA benchmarks, NaturalQuestions and TriviaQA. Our results show that the proposed UnitedQA model significantly outperforms single extractive and generative models as well as their corresponding homogeneous ensembles, and sets new state-of-the-art on both benchmarks. We also perform a comprehensive empirical study to investigate the relative contributions of different components of our model and the techniques we use to improve the readers. For future work, it would be interesting to explore model compression approaches for reducing the model size of retriever and reader separately or jointly through pruning, quantization, and knowledge distillation. We would like to thank the anonymous reviewers for valuable suggestions, Yuning Mao for valuable discussions and comments, and Microsoft Research Technology Engineering team for computing support.
abstain method objective objective result abstain abstain abstain abstain method abstain abstain objective abstain abstain objective abstain abstain abstain other abstain abstain abstain abstain abstain result method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain method abstain abstain abstain method method other abstain abstain abstain abstain method other abstain abstain abstain method abstain method method method method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method method method method abstain abstain method method method abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other method abstain other other other other other other abstain objective objective objective objective abstain other