We propose a multi-task learning framework to learn a joint Machine Reading Comprehension (MRC) model that can be applied to a wide range of MRC tasks in different domains. Inspired by recent ideas of data selection in machine translation, we develop a novel sample re-weighting scheme to assign sample-specific weights to the loss. Empirical study shows that our approach can be applied to many existing MRC models. Combined with contextual representations from pre-trained language models (such as ELMo), we achieve new state-of-the-art results on a set of MRC benchmark datasets. We release our code at https://github.com/ xycforgithub/MultiTask-MRC . Machine Reading Comprehension (MRC) has gained growing interest in the research community (Rajpurkar et al., 2016; Yu et al., 2018). In an MRC task, the machine reads a text passage and a question, and generates (or selects) an answer based on the passage. This requires the machine to possess strong comprehension, inference and reasoning capabilities. Over the past few years, there has been much progress in building end-to-end neural network models (Seo et al., 2016) for MRC. However, most public MRC datasets (e.g., SQuAD, MS MARCO, TriviaQA) are typically small (less than 100K) compared to the model size (such as SAN (Liu et al., 2018c,b) with around 10M parameters). To prevent over-fitting, recently there have been some studies on using pre-trained word embeddings (Pennington et al., 2014) and contextual embeddings in the MRC model training, as well as back-translation approaches (Yu et al., 2018) for data augmentation. Multi-task learning (Caruana, 1997) is a widely studied area in machine learning, aiming at better model generalization by combining training datasets from multiple tasks. In this work, we explore a multi-task learning (MTL) framework to enable the training of one universal model across different MRC tasks for better generalization. Intuitively, this multi-task MRC model can be viewed as an implicit data augmentation technique, which can improve generalization on the target task by leveraging training data from auxiliary tasks. We observe that merely adding more tasks cannot provide much improvement on the target task. Thus, we propose two MTL training algorithms to improve the performance. The first method simply adopts a sampling scheme, which randomly selects training data from the auxiliary tasks controlled by a ratio hyperparameter; The second algorithm incorporates recent ideas of data selection in machine translation (van der Wees et al., 2017). It learns the sample weights from the auxiliary tasks automatically through language models. Prior to this work, many studies have used upstream datasets to augment the performance of MRC models, including word embedding (Pen-nington et al., 2014), language models (ELMo) (Peters et al., 2018) and machine translation (Yu et al., 2018). These methods aim to obtain a robust semantic encoding of both passages and questions. Our MTL method is orthogonal to these methods: rather than enriching semantic embedding with external knowledge, we leverage existing MRC datasets across different domains, which help make the whole comprehension process more robust and universal. Our experiments show that MTL can bring further performance boost when combined with contextual representations from pre-trained language models, e.g., ELMo (Peters et al., 2018). To the best of our knowledge, this is the first work that systematically explores multi-task learning for MRC. In previous methods that use language models and word embedding, the external embedding/language models are pre-trained separately and remain fixed during the training of the MRC model. Our model, on the other hand, can be trained with more flexibility on various MRC tasks. MTL is also faster and easier to train than embedding/LM methods: our approach requires no pre-trained models, whereas back translation and ELMo both rely on large models that would need days to train on multiple GPUs (Jozefowicz et al., 2016; Peters et al., 2018). We validate our MTL framework with two state-of-the-art models on four datasets from different domains. Experiments show that our methods lead to a significant performance gain over single-task baselines on SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2017) and Who-Did-What (Onishi et al., 2016), while achieving state-of-the-art performance on the latter two. For example, on NewsQA (Trischler et al., 2017), our model surpassed human performance by 13.4 (46.5 vs 59.9) and 3.2 (72.6 vs 69.4) absolute points in terms of exact match and F1. The contribution of this work is three-fold. First, we apply multi-task learning to the MRC task, which brings significant improvements over single-task baselines. Second, the performance gain from MTL can be easily combined with existing methods to obtain further performance gain. Third, the proposed sampling and re-weighting scheme can further improve the multi-task learning performance. Studies in machine reading comprehension mostly focus on architecture design of neural networks, such as bidirectional attention (Seo et al., 2016), dynamic reasoning (Xu et al., 2017), and parallelization (Yu et al., 2018). Some recent work has explored transfer learning that leverages out-domain data to learn MRC models when no training data is available for the target domain (Golub et al., 2017). In this work, we explore multi-task learning to make use of the data from other domains, while we still have access to target domain training data. Multi-task learning (Caruana, 1997) has been widely used in machine learning to improve generalization using data from multiple tasks. For natural language processing, MTL has been successfully applied to low-level parsing tasks (Collobert et al., 2011), sequence-to-sequence learning (Lu-ong et al., 2015), and web search (Liu et al., 2015). More recently, (McCann et al., 2018) proposes to cast all tasks from parsing to translation as a QA problem and use a single network to solve all of them. However, their results show that multi-task learning hurts the performance of most tasks when tackling them together. Differently, we focus on applying MTL to the MRC task and show significant improvement over single-task baselines. Our sample re-weighting scheme bears some resemblance to previous MTL techniques that assign weights to tasks (Kendall et al., 2018). However, our method gives a more granular score for each sample and provides better performance for multitask learning MRC. We call our model Multi-Task-SAN (MT-SAN), which is a variation of SAN (Liu et al., 2018c) model with two main differences: i) we add a highway network layer after the embedding layer, the encoding layer and the attention layer; ii) we use exponential moving average (Seo et al., 2016) during evaluation. The SAN architecture and our modifications are briefly described below and in Section 5.2, and detailed description can be found in (Liu et al., 2018c). For most tasks we consider, our MRC model takes a triplet ( Q, P, A ) as input, where Q = ( q 1 , ..., q m ) , P = ( p 1 , ..., p n ) are the word index representations of a question and a passage, respectively , and A = ( a begin , a end ) is the index of the answer span. The goal is to predict A given ( Q, P ) . We map the word indices of P and Q into their 300-dim Glove vectors (Pennington et al., 2014). We also use the following additional information for embedding words: i) 16-dim part-of-speech (POS) tagging embedding; ii) 8-dim named-entity-recognition (NER) embedding; iii) 3-dim exact match embedding: f exact match ( p i ) = I ( p i  Q ) , where matching is determined based on the original word, lower case, and lemma form, respectively; iv) Question enhanced passage word embeddings: f align ( p i ) = (cid:80) j  i,j h ( GloVe ( q j )) , where  i,j = exp( h ( GloVe ( p j )) ,h ( GloVe ( q i ))) (cid:80) j (cid:48) exp( h ( GloVe ( p j (cid:48) )) ,h ( GloVe ( q i ))) (1) is the similarity between word p j and q i , and g (  ) is a 300-dim single layer neural net with Recti-fied Linear Unit (ReLU) g ( x ) = ReLU ( W 1 x ) ; v) Passage-enhanced question word embeddings: the same as iv) but computed in the reverse direction. To reduce the dimension of the input to the next layer, the 624-dim input vectors of passages and questions are passed through a ReLu layer to reduce their dimensions to 125. After the ReLU network, we pass the 125-dim vectors through a highway network (Srivastava et al., 2015), to adapt to the multi-task setting: g i = sigmoid ( W 2 p ti ) , p ti = ReLU ( W 3 p ti ) (cid:12) g i + g i (cid:12) p t i , where p t i is the vector after ReLU transformation. Intuitively, the highway network here provides a neuron-wise weighting, which can potentially handle the large variation in data introduced by multiple datasets. Both the passage and question encodings go through a 2-layer Bidirectional Long-Short Term Memory (BiLSTM, Hochreiter and Schmidhuber, 1997) network in this layer. We append a 600-dim CoVe vector (McCann et al., 2017) to the output of the lexicon encoding layer as input to the contextual encoders. For the experiments with ELMo, we also append a 1024-dim ELMo vector. Similar to the lexicon encoding layer, the outputs of both layers are passed through a highway network for multi-tasking. Then we concatenate the output of the two layers to obtain H q  R 2 d  m for the question and H p = R 2 d  n the passage, where d is the dimension of the BiLSTM. We fuse H p and H q through cross attention and generate a working memory in this layer. We adopt the attention function from (Vaswani et al., 2017) and compute the attention matrix as C = dropout (cid:16) f attention (  H q ,  H p ) (cid:17)  R m  n . We then use C to compute a question-aware passage representation as U p = concat ( H p , H q C ) . Since a passage usually includes several hundred tokens, we use the method of (Lin et al., 2017) to apply self attention to the representations of passage to rearrange its information:  U p = U p drop diag ( f attention ( U p , U p )) , where drop diag means that we only drop diagonal elements on the similarity matrix (i.e., attention with itself). Then, we concatenate U p and  U p and pass them through a BiLSTM: M = BiLSTM ([ U p ];  U p ]) . Finally, output of the BiLSTM (after concatenating two directions) goes through a highway layer to produce the memory. 3.5 Answer Module The base answer module is the same as SAN, which computes a distribution over spans in the passage. Firstly, we compute an initial state s 0 by self attention on H q : s 0  Highway (cid:18)(cid:80) j exp( w 4 H qj ) (cid:80) j (cid:48) exp w 4 H qj (cid:48)  H qj (cid:19) . The final answer is computed through T time steps. At step t  { 1 , ..., T  1 } , we compute the new state using a Gated Recurrent Unit (GRU, Cho et al., 2014) s t = GRU ( s t  1 , x t ) , where x t is computed by attention between M and s t  1 : x t = (cid:80) j  j M j ,  j = softmax ( s t  1 W 5 M ) . Then each step produces a prediction of the start and end of answer spans through a bilinear function: P begin t = softmax ( s t W 6 M ) , P end t = softmax ( s t W 7 M ) . The final prediction is the average of each time step: P begin = 1 T (cid:80) t P begin t , P end = 1 T (cid:80) t P end t . We randomly apply dropout on the step level in each time step during training, as done in (Liu et al., 2018c). During training, the objective is the log-likelihood of the ground truth: l ( Q, P, A ) = log P begin ( a begin ) + log P end ( a end ) . We describe our MTL training algorithms in this section. We start with a very simple and straightforward algorithm that samples one task and one mini-batch from that task at each iteration. To improve the performance of MTL on a target dataset, we propose two methods to re-weight samples according to their importance. The first proposed method directly lowers the probability of sampling from a particular auxiliary task; however, this probability has to be chosen using grid search. We then propose another method that avoids such search by using a language model. Suppose we have K different tasks, the simplest version of our MTL training procedure is shown in Algorithm 1. In each epoch, we take all the mini-batches from all datasets and shuffle them for Algorithm 1 Multi-task Learning of MRC Input: k different datasets D 1 , ..., DK , max epoch 1: Initialize the model M 2: for epoch = 1 , 2 , ... , max epoch do 3: Divide each dataset D k into N k mini-batches D k = { b k 1 , ..., b kN k } , 1  k  K 4: Put all mini-batches together and randomly shuffle the order of them, to obtain a sequence B = ( b 1 , ..., b L ) , where L = (cid:80) k N k 5: for each mini-batch b  B do 6: Perform gradient update on M with loss l ( b ) = (cid:80) ( Q,P,A )  b l ( Q, P, A ) 7: end for 8: Evaluate development set performance 9: end for Output: Model with best evaluation performance model training, and the same set of parameters is used for all tasks. Perhaps surprisingly, as we will show in the experiment results, this simple baseline method can already lead to a considerable improvement over the single-task baselines. One observation is that the performance of our model using Algorithm 1 starts to deteriorate as we add more and more data from other tasks into our training pool. We hypothesize that the external data will inevitably bias the model towards auxiliary tasks instead of the target task. To avoid such adverse effect, we introduce a mixture ratio parameter during training. The training algorithm with the mixture ratio is presented in Algorithm 2, with D 1 being the target dataset. In each epoch, we use all mini-batches from D 1 , while only a ratio  of mini-batches from external datasets are used to train the model. In our experiment, we use hyperparameter search to find the best  for each dataset combination. This method resembles previous methods in multi-task learning to weight losses differently (e.g., Kendall et al., 2018), and is very easy to implement. In our experiments, we use Algorithm 2 to train our network when we only use 2 datasets for MTL. The mixture ratio (Algorithm 2) dramatically improves the performance of our system. However, it requires to find an ideal ratio by hyperparameter search which is time-consuming. Furthermore, Algorithm 2 Multi-task Learning of MRC with mixture ratio, targeting D 1 Input: K different datasets D 1 , ..., DK , max epoch, mixture ratio  1: Initialize the model M 2: for epoch = 1 , 2 , ... , max epoch do 3: Divide each dataset D k into N k mini-batches D k = { b k 1 , ..., b kN k } , 1  k  K 4: S  { b 11 , ..., b 1 N 1 } 5: Randomly pick (cid:98) N 1 (cid:99) mini-batches from (cid:83) Kk =2 D k and add to S 6: Assign mini-batches in S in a random order to obtain a sequence B = ( b 1 , ..., b L ) , where L = N 1 + (cid:98) N 1 (cid:99) 7: for each mini-batch b  B do 8: Perform gradient update on M with loss l ( b ) = (cid:80) ( Q,P,A )  b l ( Q, P, A ) 9: end for 10: Evaluate development set performance 11: end for Output: Model with best evaluation performance the ratio gives the same weight to every auxiliary data, but the relevance of every data point to the target task can vary greatly. We develop a novel re-weighting method to resolve these problems, using ideas inspired by data selection in machine translation (Axelrod et al., 2011; van der Wees et al., 2017). We use ( Q k , P k , A k ) to represent a data point from the k th task for 1  k  K , with k = 1 being the target task. Since the passage styles are hard to evaluate, we only evaluate data points based on Q k and A k . Note that only data from auxiliary task ( 2  k  K ) is re-weighted; target task data always have weight 1. Our scores consist of two parts, one for questions and one for answers. For questions, we create language models (detailed in Section 5.2) using questions from each task, which we represent as LM k for the k -th task. For each question Q k from auxiliary tasks, we compute a cross-entropy score: H C,Q ( Q k ) =  1 m (cid:88) w  Q k log( LMC ( w )) , (2) where C  { 1 , k } is the target or auxiliary task, m is the length of question Q k , and w iterates over all words in Q k . It is hard to build language models for answers since they are typically very short (e.g., answers Dataset SQuAD(v1) NewsQA MS MARCO(v1) WDW # Training Questions 87,599 92,549 78,905 127,786 Text Domain Wikipedia CNN News Web Search Gigaword Corpus Avg. on SQuAD includes only one or two words in most cases). We instead just use the length of answers as a signal for scores. Let l ka be the length of A k , the cross-entropy answer score is defined as: H C,A ( A k ) =  log freq C ( l ka ) , (3) where freq C is the frequency of answer lengths in task C  { 1 , k } . The cross entropy scores are then normalized over all samples in task C to create a comparable metric across all auxiliary tasks: H (cid:48) C,Q ( Q k ) = H C,Q ( Q k )  min( H C,Q ) max( H C,Q )  min( H C,Q ) (4) H (cid:48) C,A ( A k ) = H C,A ( A k )  min( H C,A ) max( H C,A )  min( H C,A ) (5) for C  { 1 , 2 , ..., K } . For C  { 2 , ..., K } , the maximum and minimum are taken over all samples in task k . For C = 1 (target task), they are taken over all available samples. Intuitively, H (cid:48) C,Q and H (cid:48) C,A represents the similarity of text Q, A to task C ; a low H (cid:48) C,Q (resp. H (cid:48) C,A ) means that Q k (resp. A k ) is easy to predict and similar to C , and vice versa. We would like samples that are most similar from data in the target domain (low H (cid:48) 1 ), and most different (infor-mative) from data in the auxiliary task (high H (cid:48) k ). We thus compute the following cross-entropy difference for each external data: CED ( Q k , A k ) =( H (cid:48) 1 ,Q ( Q k )  H (cid:48) k,Q ( Q k ))+ ( H (cid:48) 1 ,A ( A k )  H (cid:48) k,A ( A k )) (6) for k  { 2 , ..., K } . Note that a low CED score indicates high importance. Finally, we transform the scores to weights by taking negative, and normalize between [0 , 1] : CED (cid:48) ( Q k , A k ) = 1  CED ( Q k , A k )  min( CED ) max( CED )  min( CED ) . Here the maximum and minimum are taken over all available samples and task. Our training algorithm is the same as Algorithm 1, but for mini-batch b we instead use the loss l ( b ) = (cid:88) ( P,Q,A )  b CED (cid:48) ( Q, A ) l ( P, Q, A ) (8) in step 6. Our experiments are designed to answer the following questions on multi-task learning for MRC: 1. Can we improve the performance of existing MRC systems using multi-task learning? 2. How does multi-task learning affect the performance if we combine it with other external data? 3. How does the learning algorithm change the performance of multi-task MRC? 4. How does our method compare with existing MTL methods? We first present our experiment details and results for MT-SAN. Then, we provide a comprehensive study on the effectiveness of various MTL algorithms in Section 5.4. At last, we provide some additional results on combining MTL with DrQA (Chen et al., 2017) to show the flexibility of our approach 1 . We conducted experiments on SQuAD (Rajpurkar et al., 2016), NewsQA(Trischler et al., 2017), MS MARCO (v1, Nguyen et al.,2016) and WDW (On-ishi et al., 2016). Dataset statistics is shown in Table 1. Although similar in size, these datasets are quite different in domains, lengths of text, and types of task. In the following experiments, we will validate whether including external datasets as additional input information (e.g., pre-trained language model on these datasets) helps boost the performance of MRC systems. We mostly focus on span-based datasets for MT-SAN, namely SQuAD, NewsQA, and MS MARCO. We convert MS MARCO into an answer-span dataset to be consistent with SQuAD and NewsQA, following (Liu et al., 2018c). For each question, we search for the best span using ROUGE-L score in all passage texts and use the span to train our model. We exclude questions with maximal ROUGE-L score less than 0.5 during training. For evaluation, we use our model to find a span in all passages. The prediction score is multiplied with the ranking score, trained following Liu et al. (2018a)'s method to determine the final answer. We train our networks using algorithms in Section 4, using SQuAD as the target task. For experiments with two datasets, we use Algorithm 2; for experiments with three datasets we find the re-weighting mechanism in Section 4.2 to have a better performance (a detailed comparison will be presented in Section 5.4). For generating sample weights, we build a LSTM language model on questions following the implementation of Merity et al. (2017) with the same hyperparameters. We only keep the 10,000 most frequent words, and replace the other words with a special out-of-vocabulary token. Parameters of MT-SAN are mostly the same as in the original paper (Liu et al., 2018c). We utilize spaCy 2 to tokenize the text and generate part-of-speech and named entity labels. We use a 2-layer BiLSTM with 125 hidden units as the BiLSTM throughout the model. During training, we drop the activation of each neuron with 0.3 probability. For optimization, we use Adamax (Kingma and Ba, 2014) with a batch size of 32 and a learning rate of 0.002. For prediction, we compute an exponential moving average (EMA, Seo et al. 2016) of model parameters with a decay rate of 0.995 and use it to compute the model performance. For experiments with ELMo, we use the model implemented by AllenNLP 3 . We truncate passage to contain at most 1000 tokens during training and eliminate those data with answers located after the 1000th token. The training converges in around 50 epochs for models without ELMo (similar to the single-task SAN); For models with ELMo, the convergence is much faster (around 30 epochs). In the following sub-sections, we report our results on SQuAD and MARCO development sets, The multi-task learning results of SAN on SQuAD are summarized in Table 2. By using MTL on SQuAD and NewsQA, we can improve the exact-match (EM) and F1 score by (2%, 1.5%), respectively, both with and without ELMo. The similar gain indicates that our method is orthogonal to ELMo. Note that our single-model performance is slightly higher than the original SAN, by incorporating EMA and highway networks. By incorporating with multi-task learning, it further improves the performance. The performance gain by adding MARCO is relatively smaller, with 1% in EM and 0.5% in F1. We conjecture that MARCO is less helpful due to its differences in both the question and answer style. For example, questions in MS MARCO are real web search queries, which are short and may have typos or abbreviations; while questions in SQuAD and NewsQA are more formal and well written. Using 3 datasets altogether provides another marginal improvement. Our model obtains the best results among existing methods that do not use a large language model (e.g., ELMo). Our ELMo version also outperforms any other models which are under the same setting. We note that BERT (Devlin et al., 2018) uses a much larger model than ours(around 20x), and we leave the performance of combining BERT with MTL as interesting future work. The results of multi-task learning on NewsQA are in Table 3. The performance gain with multitask learning is even larger on NewsQA, with over 2% in both EM and F1. Experiments with and without ELMo give similar results. What is worth noting is that our approach not only achieves new state-of-art results with a large margin but also surpasses human performance on NewsQA. Finally we report MT-SAN performance on MS MARCO in Table 4. Multi-tasking on SQuAD and NewsQA provides a similar performance boost in terms of BLEU-1 and ROUGE-L score as in the case of NewsQA and SQuAD. Our method does not achieve very high performance compared to previous work, probably because we do not apply common techniques like yes/no classification 4 The official submission for SQuAD v1.1 and MARCO v1.1 are closed, so we report results on the development set. According to their leaderboards, performances on development and test sets are usually similar. We also test the robustness of our algorithm by performing another set of experiments on SQuAD and WDW. WDW is much more different than the other three datasets (SQuAD, NewsQA, MS MARCO): WDW guarantees that the answer is always a person, whereas the percentage of Model SQuAD WDW MT-SAN (Single Task) 76.8, 84.5 77.5 MT-SAN (S+W) 77.6, 85.1 78.5 SOTA(Yang et al., 2016). such questions in SQuAD is 12.9%. Moreover, WDW is a cloze dataset, whereas in SQuAD and NewsQA answers are spans in the passage. We use a task-specific answer layer in this experiment and use Algorithm 2; the WDW answer module is the same as in AS Reader (Kadlec et al., 2016), which we describe in the appendix for completeness. Despite these large difference between datasets, our results (Table 5) show that MTL can still provide a moderate performance boost when jointly training on SQuAD (around 0.7%) and WDW (around 1%). Comparison of methods using external data. As a method of data augmentation, we compare our approach to previous methods for MRC in Table 6. Our model achieves better performance than back translation. We also observe that language models such as ELMo obtain a higher performance gain than multi-task learning, however, combining it with multi-task learning leads to the most significant performance gain. This validates our assump-tion that multi-task learning is more robust and is different from previous methods such as language modeling. In this section, we provide ablation studies as well as comparisons with other existing algorithms on the MTL strategy. We focus on MT-SAN without Model Performance SQuAD + MARCO EM,F1 Simple Combine (Alg. 1) 77.1, 84.6 Loss Uncertainty 77.3, 84.7 Mixture Ratio 77.8, 85.2 Sample Re-weighting 77.9,85.3 SQuAD + NewsQA + MARCO Simple Combine (Alg. 1) 77.6, 85.2 Loss Uncertainty 78.2, 85.6 Mixture Ratio 78.4, 85.7 Sample Re-weighting 78.8 , 86.0 Table 7: Comparison of different MTL strategies on MT-SAN. Table 7 compares different multi-task learning strategies for MRC. Both the mixture ratio (Sec 4.1) and sample re-weighting (Sec 4.2) improves over the naive baseline of simply combining all the data (Algorithm 1). On SQuAD+MARCO, they provide around 0.6% performance boost in terms of both EM and F1, and around 1% on all 3 datasets. We note that this accounts for around a half of our overall improvement. Although sample re-weighting performs similar as mixture ratio, it significantly reduces the amount of training time as it eliminates the need for a grid searching the best ratio. Kendal et al., (2018) use task uncertainty to weight tasks differently for MTL; our experiments show that this has some positive effect, but does not perform as well as our proposed two techniques. We note that Kendal et al. (as well as other previous MTL methods) optimizes the network to perform well for all the tasks, whereas our method focuses on the target domain which we are interested in, e.g., SQuAD. Sensitivity of mixture ratio. We also investigate the effect of mixture ratio on the model performance. We plot the EM/F1 score on SQuAD dev set vs. mixture ratio in Figure 1 for MT-SAN when trained on all three datasets. The curve peaks at  = 0 . 4 ; however if we use  = 0 . 2 or  = 0 . 5 , the performance drops by around 0 . 5% , well behind the performance of sample re-weighting. This shows that the performance of MT-SAN is sensitive to changes in  , making the hyperparameter search even more difficult. Such sensitivity suggests a preference for using our sample reweighting technique. On the other hand, the ratio Samples/Groups CED (cid:48) HQHA Examples (NewsQA) Q: Where is the drought hitting? implement. Analysis of sample weights. Dataset comparisons in Table 1 and performance in Table 2 suggests that NewsQA share more similarity with SQuAD than MARCO. Therefore, a MTL system should weight NewsQA samples more than MARCO samples for higher performance. We try to verify this in Table 8 by showing examples and statistics of the sample weights. We present the CED (cid:48) scores, as well as normalized version of question and answer scores (resp. ( H (cid:48) 1 ,Q  H (cid:48) k,Q ) and ( H (cid:48) 1 ,A  H (cid:48) k,A ) in (6), and then negated and normalized over all samples in NewsQA and MARCO in the same way as in (7)). A high HQ score indicates high importance of the question, and HA of the answer; CED (cid:48) is a summary of the two. We first show one example from NewsQA and one from MARCO. The NewsQA question is a natural question (similar to SQuAD) with a short answer, leading to high scores both in questions and answers. The MARCO question is a phrase, with a very long answer, leading to lower scores. From overall statistics, we also find samples in NewsQA have a higher score than those in MARCO. However, if we look at MARCO questions that start with when or who (i.e., probability natural questions with short answers), the scores go up dramatically. We proposed a multi-task learning framework to train MRC systems using datasets from different domains and developed two approaches to re-weight the samples for multi-task learning on MRC tasks. Empirical results demonstrated our approaches outperform existing MTL methods and the single-task baselines as well. Interesting future directions include combining with larger language models such as BERT, and MTL with broader tasks such as language inference (Liu et al., 2019) and machine translation. Yichong Xu has been partially supported by DARPA (FA8750-17-2-0130).
objective objective result objective other abstain abstain abstain abstain abstain abstain abstain objective abstain result objective abstain abstain abstain abstain abstain result objective abstain method method abstain result result objective objective abstain abstain other other objective other other other other abstain abstain method abstain method method abstain method abstain abstain method abstain abstain abstain other abstain abstain abstain other abstain abstain abstain abstain abstain abstain method method method method abstain abstain abstain method abstain other abstain abstain method abstain abstain method abstain abstain abstain abstain other abstain method abstain abstain abstain method abstain abstain method abstain abstain other abstain method method abstain abstain method method abstain abstain method abstain other abstain abstain abstain abstain abstain method abstain method abstain method abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain method method objective objective abstain other