Neural models have achieved great success on machine reading comprehension (MRC), many of which typically consist of two components: an evidence extractor and an answer predictor. The former seeks the most relevant information from a reference text, while the latter is to locate or generate answers from the extracted evidence. Despite the importance of evidence labels for training the evidence extractor, they are not cheaply accessible, particularly in many non-extractive MRC tasks such as YES/NO question answering and multi-choice MRC. To address this problem, we present a Self-Training method (STM), which supervises the evidence extractor with auto-generated evidence labels in an iterative process. At each iteration, a base MRC model is trained with golden answers and noisy evidence labels. The trained model will predict pseudo evidence labels as extra supervision in the next iteration. We evaluate STM on seven datasets over three MRC tasks. Experimental results demonstrate the improvement on existing MRC models, and we also analyze how and why such a self-training method works in MRC. Machine reading comprehension (MRC) has received increasing attention recently, which can be roughly divided into two categories: extractive and non-extractive MRC. Extractive MRC requires a model to extract an answer span to a question from reference documents, such as the tasks in SQuAD (Rajpurkar et al., 2016) and CoQA (Reddy et al., 2019). In contrast, non-extractive MRC infers answers based on some evidence in reference  Equal contribution  Corresponding author: Minlie Huang. documents, including Yes/No question answering (Clark et al., 2019), multiple-choice MRC (Lai et al., 2017; Khashabi et al., 2018; Sun et al., 2019), and open domain question answering (Dhingra et al., 2017b). As shown in Table 1, evidence plays a vital role in MRC (Zhou et al., 2019; Ding et al., 2019; Min et al., 2018), and the coarse-to-fine paradigm has been widely adopted in multiple models (Choi et al., 2017; Li et al., 2018; Wang et al., 2018) where an evidence extractor first seeks the evidence from given documents and then an answer predictor infers the answer based on the evidence. However, it is challenging to learn a good evidence extractor due to the lack of evidence labels for supervision. Manually annotating the golden evidence is expensive. Therefore, some recent efforts have been dedicated to improving MRC by leveraging noisy evidence labels when training the evidence extractor. Some works (Lin et al., 2018; Min et al., 2018) generate distant labels using hand-crafted rules and external resources. Some studies (Wang et al., 2018; Choi et al., 2017) adopt reinforcement learning (RL) to decide the labels of evidence. However, such RL methods suffer from unstable training. More distant supervision techniques are also used to refine noisy labels, such as deep probability logic (Wang et al., 2019), but they are hard to transfer to other tasks. Nevertheless, improving the evidence extractor remains challenging when golden evidence labels are not available. In this paper, we present a general and effective method based on Self-Training (Scudder, 1965) to improve MRC with soft evidence extraction when golden evidence labels are not available. Following the Self-Training paradigm, a base MRC model is iteratively trained. At each iteration, the base model is trained with golden answers, as well as noisy evidence labels obtained at the preceding it-Q: Did a little boy write the note? D: ... This note is from a little girl . She wants to be your friend. If you want to be her friend, ... A: No Q: Is she carrying something? D: ...On the step, I find the elderly Chinese lady, small and slight, holding the hand of a little boy. In her other hand, she holds a paper carrier bag. ... A: Yes Table 1: Examples of Yes/No question answering. eration. Then, the trained model generates noisy evidence labels, which will be used to supervise evidence extraction at the next iteration. The overview of our method is shown in Figure 1. Through this iterative process, the evidence is labeled automatically to guide the RC model to find answers, and then a better RC model benefits the evidence labeling process in return. Our method works without any manual efforts or external information, and therefore can be applied to any MRC tasks. Besides, the Self-Training algorithm converges more stably than RL. Two main contributions in this paper are summarized as follows: 1. We propose a self-training method to improve machine reading comprehension by soft evidence labeling. Compared with other existing methods, our method is more effective and general. 2. We verify the generalization and effectiveness of STM on several MRC tasks, including Yes/No question answering (YNQA), multiple-choice machine reading comprehension (MMRC), and open-domain question answering (ODQA). Our method is applicable to different base models, including BERT and DSQA (Lin et al., 2018). Experimental results demonstrate that our proposed method improves base models in three MRC tasks remarkably. Early MRC studies focus on modeling semantic matching between a question and a reference document (Seo et al., 2017; Huang et al., 2018; Zhu et al., 2018; Mihaylov and Frank, 2018). In order to mimic the reading mode of human, hierarchical coarse-to-fine methods are proposed (Choi et al., 2017; Li et al., 2018). Such models first read the full text to select relevant text spans, and then infer answers from these relevant spans. Extracting such spans in MRC is drawing more and more attention, though still quite challenging (Wang et al., 2019). Evidence extraction aims at finding evidential and relevant information for downstream processes in a task, which arguably improves the overall performance of the task. Not surprisingly, evidence extraction is useful and becomes an important component in fact verification (Zhou et al., 2019; Yin and Roth, 2018; Hanselowski et al., 2018; Ma et al., 2019), multiple-choice reading comprehension (Wang et al., 2019; Bax, 2013; Yu et al., 2019), open-domain question answering (Lin et al., 2018; Wang et al., 2018), multi-hop reading comprehension (Nishida et al., 2019; Ding et al., 2019), natural language inference (Wang et al., 2017; Chen et al., 2017), and a wide range of other tasks (Nguyen and Nguyen, 2018; Chen and Bansal, 2018). In general, evidence extraction in MRC can be classified into four types according to the training method. First, unsupervised methods provide no guidance for evidence extraction (Seo et al., 2017; Huang et al., 2019). Second, supervised methods train evidence extraction with golden evidence labels, which sometimes can be generated automatically in extractive MRC settings (Lin et al., 2018; Yin and Roth, 2018; Hanselowski et al., 2018). Third, weakly supervised methods rely on noisy evidence labels, where the labels can be obtained by heuristic rules (Min et al., 2018). Moreover, some data programming techniques, such as deep probability logic, were proposed to refine noisy labels (Wang et al., 2019). Last, if a weak extractor is obtained via unsupervised or weakly supervised pre-training, reinforcement learning can be utilized to learn a better policy of evidence extraction (Wang et al., 2018; Choi et al., 2017). For non-extractive MRC tasks, such as YNQA and MMRC, it is cumbersome and inefficient to annotate evidence labels (Ma et al., 2019). Although various methods for evidence extraction have been proposed, training an effective extractor is still a challenging problem when golden evidence labels are unavailable. Weakly supervised methods either suffer from the low performance or rely on too many external resources, which makes them difficult to transfer to other tasks. RL methods can indeed train a better extractor without evidence labels. However, they are much more complicated and unstable to train, and highly dependent on model pre-training. Our method is based on Self-Training, a widely used semi-supervised method. Most related studies follow the framework of traditional Self-Training (Scudder, 1965) and Co-Training (Blum and Mitchell, 1998), and focus on designing better policies for selecting confident samples. Co-Trade (Zhang and Zhou, 2011) evaluates the confidence of whether a sample has been correctly labeled via a statistic-based data editing technique (Zighed et al., 2002). Self-paced Co-Training (Ma et al., 2017) adjusts labeled data dynamically according to the consistency between the two models trained on different views. A reinforcement learning method (Wu et al., 2018) designs an additional Q-agent as a sample selector. The task of machine reading comprehension can be formalized as follows: given a reference document composed of a number of sentences D = { S 1 , S 2 ,    , S m } and a question Q , the model should extract or generate an answer  A to this question conditioned on the document, formally as The process can be decomposed into two components, i.e., an evidence extractor and an answer predictor. The golden answer A is given for training the entire model, including the evidence extractor and the answer predictor. Denote E i as a binary evidence label { 0 , 1 } for the i -th sentence S i , where 0 / 1 corresponds to the non-evidence/evidence sentence, respectively. An auxiliary loss on the evidence labels can help the training of the evidence extractor. The overview of our method is shown in Figure 1, which is an iterative process. During training, two data pools are maintained and denoted as U (unlabeled data) and L (labeled data). In addition to golden answers, examples in L are annotated with pseudo evidence labels. In contrast, there are only golden answers provided in U . At each iteration, the base model is trained on both data pools (two training arrows). After training, the model makes evidence predictions on unlabeled instances (the labeling arrow), and then Selector chooses the most confident instances from U to provide noisy evidence labels. In particular, the instances with newly generated evidence labels are moved from U to L (the moving arrow), which are used to supervise evidence extraction in the next iteration. This process will iterate several times. As shown in Figure 2, the overall structure of a base model consists of an encoder layer, an evidence extractor, and an answer predictor. The encoder layer takes document D and question Q as input to obtain contextual representation for each word. Denote h Di,j as the representation of the j -th word in S i , and h Qi as the representation of the i -th word in question Q . Our framework is agnostic to the architecture of the encoder, and we show improvements on two widely used encoding models, i.e., Transformer (with BERT, Devlin et al., 2019) and LSTM (with DSQA, Lin et al., 2018) in the experiments. The evidence extractor employs hierarchical attention, including tokenand sentence-level attention, to obtain the document representation h D . Token-level attention obtains a sentence vector by self-attention (Vaswani et al., 2017) within the words in a sentence, as follows: h Di = | S i | (cid:88) j  i,j h Di,j ,  i,j  exp( FS ( h Q , h Di,j )) , s Di = | S i | (cid:88) j  i,j h Di,j ,  i,j  exp( w s h Di,j + b s ) , where h Q is the sentence representation of the question. i,j refers to the importance of word j in sentence i , and so on for  i,j . w s and b s are learnable parameters. The attention function FS follows the bilinear form (Kim et al., 2018). Sentence-level attention identifies important sentences conditioned on the question in a soft way to get the summary vector ( h D ), as follows: h D = m (cid:88) i  i h Di ,  i  exp( FD ( h Q , s Di )) , where FD has the same bilinear form as FS with different parameters. i refers to the importance of the corresponding sentence. The answer predictor adopts different structures for different MRC tasks. For Yes/No question answering, we use a simple linear classifier to infer answers. For multiple-choice MRC, we use a Multiple Layer Perceptron (MLP) with Softmax to obtain the score of each choice. And for open-domain question answering, one MLP is used to predict the answer start, and another MLP is used to predict the end. We adopt two loss functions, one for task-specific loss and the other for evidence loss. The task-specific loss is defined as the negative log-likelihood (NLL) of predicting golden answers, formally as follows: LA ( D, Q, A ) =  log P (  A = A | D, Q ) , where  A denotes the predicted answer and A is the golden answer. When the evidence label E is provided, we can impose supervision on the evidence extractor. For the most general case, we assume that a variable number of evidence sentences exist in each sample ( Q, A, D ) . Inspired by the previous work (Nishida et al., 2019) that used multiple pieces of evidence, we calculate the evidence loss step by step. Suppose we will extract K evidence sentences. In the first step, we compute the loss of selecting the most plausible evidence sentence. In the second step, we compute the loss in the remaining sentences, where the previously selected sentence is masked and not counted in computing the loss at the second step. The overall loss is the average of all the step-by-step loss until we select out K evidence sentences. In this manner, we devise a BP-able surrogate loss function for choosing the top K evidence sentences. Formally, we have LE ( D, Q, E ) = 1 KK (cid:88) k =1 H ( D, Q, E, M k ) , where K is the number of evidence sentences, a pre-specified hyperparamter. M k = { M k 1 , M k 2 ,    , M km } and each M ki  { 0 , } is a sentence mask, where 0 means sentence i is not selected before step k , and  means selected. As M ki =  for the previously selected sentences, the attention weight on those sentences will be zero, in other words, they are masked out. Then, the step-wise loss can be computed as follows: H ( D, Q, E, M k ) =  log max i (  ki  E i ) , where  ki indicates the attention weight for sentence i , and E i  { 0 , 1 } is the evidence label for sentence i . The sentence with the largest attention weight will be chosen as the k -th evidence sentence. For each sentence i , M 1 i is initialized to be 0 . At each step k ( k > 1) , the mask M ki will be set to  if sentence i is chosen as an evidence sentence at the preceding step k  1 , and the mask remains unchanged otherwise. Formally, the mask is updated as follows: M ki = (cid:40)  i = argmax j (  k  1 j E j ) M k  1 i otherwise . During training, the total loss L is the combination of the task-specific loss and the evidence loss: L = (cid:88) ( D,Q,A )  U  LLA ( D, Q, A )+  (cid:88) ( D,Q,E )  LLE ( D, Q, E ) , (1) where  is a factor to balance the two loss terms. L and U denote the two sets in which instances with and without evidence labels, respectively. Note that the evidence label in L is automatically obtained in our self-training method. STM is designed to improve base MRC models via generating pseudo evidence labels for evidence extraction when golden labels are unavailable. STM works in an iterative manner, and each iteration consists of two stages. One is to learn a better base model for answer prediction and evidence labeling. The other is to obtain more precise evidence labels for the next iteration using the updated model. At each iteration, STM first trains the base model with golden answers and pseudo evidence labels from the preceding iteration using the total loss as defined Equation 1. Then the trained model can predict a distribution of pseudo evidence labels for each unlabelled instance ( D, Q, A ) , and decides  E as  E = argmin E (cid:48) LE ( D, Q, E (cid:48) ) . (2) Define the confidence of a labelled instance ( D, Q, A,  E ) as c ( D, Q, A,  E ) = exp(  LA ( D, Q, A ))  exp(  LE ( D, Q,  E )) . Selector selects the instances with the largest confidence scores whose LA ( D, Q, A ) and LE ( D, Q,  E ) are smaller than the prespecified thresholds. These labelled instances will be moved from U to L for the next iteration. In the first iteration (iteration 0), the initial labeled set L is set to an empty set. Thus the base model is supervised only by golden answers. In this case, the evidence extractor is trained in a distant supervised manner. The procedure of one iteration of STM is illustrated in Algorithm 1.  and (cid:15) are two thresholds (hyper-parameters). sort operation ranks the candidate samples according to their confidence scores s and returns the topn samples. n varies different datasets, and details are presented in the appendix . Algorithm 1 One iteration of STM Input: Training sets U, L ; Thresholds  and (cid:15) ; Number of generated labels n ; Weight of evidence loss  ; Output: Trained MRC model M ; Updated training sets U, L ; 1: Randomly initialize M ; 2: Train M on U and L ; 3: Initialize L (cid:48) =  ; 4: for each ( D, Q, A )  U do 5: l A = LA ( D, Q, A ) ; 6: Generate  E via Equation 2; 7: l  E = LE ( D, Q,  E ) ; 8: if l A  , l  E  (cid:15) then 9: s = c ( D, Q, A,  E ) ; 10: Add ( D, Q, A,  E, s ) to L (cid:48) ; 11: end if 12: end for 13: L (cid:48) = sort ( L (cid:48) , n ) ; 14: L = L  L (cid:48) , U = U \ L (cid:48) ; 15: return M, U, L ; 3.5 Analysis To understand why STM can improve evidence extraction and the performance of MRC, we revisit the training process and present a theoretical explanation, as inspired by (Anonymous, 2020). In Section 3.4, we introduce the simple labeling strategy used in STM. If there is no sample selection, the evidence loss can be formulated as L  t =  E x  p ( x ) EE  p t  1 ( E | x ) log p  t ( E | x ) , where x represents ( D, Q, A ) , and  t is the parameter of the t -th iteration. In this case, pseudo evidence labels E are randomly sampled from p  t  1 ( E | x ) to guide p  t ( E | x ) , and therefore minimizing L  t will lead to  t =  t  1 . As a matter of fact, the sample selection strategy in STM is to filter out the low-quality pseudo labels with two Model / Dataset CoQA MARCO BoolQ BERT-MLP 78.0 70.8 71.6 BERT-HA 78.8 71.3 72.9 BERT-HA+RL 79.3 70.3 70.4 BERT-HA+Rule 78.1 70.4 73.8 BERT-HA+STM 80.5  72.3  75.2  BERT-HA+Gold 82.0 N/A N/A Table 2: Classification accuracy on three Yes/No question answering datasets. In STM, f is a filter function with two pre-specified thresholds,  and (cid:15) . g is defined as argmax (Equa-tion 2). Compared with random sampling, our strategy tends to prevent  t from learning wrong knowledge from  t  1 . And the subsequent training might benefit from implicitly learning the strategy. In general, the strategy of STM imposes naive prior knowledge on the base models via the two distribution mappings, which may partly explain the performance gains. CoQA (Reddy et al., 2019) is a multi-turn conversational question answering dataset where questions may be incomplete and need historical context to get the answers. We extracted the Yes/No questions from CoQA, along with their histories, to form a YNQA dataset. BoolQ (Clark et al., 2019) consists of Yes/No questions from the Google search engine. Each question is accompanied by a related paragraph. We expanded each short paragraph by concatenating some randomly sampled sentences. MS MARCO (Nguyen et al., 2016) is a large MRC dataset. Each question is paired with a set of reference documents, and the answer may not exist in the documents. We extracted all Yes/No questions, and randomly picked some reference documents containing evidence 1 . To balance the ratio of Yes 1 The evidence annotation in a document is provided by the and No questions, we randomly removed some questions whose answers are Yes. RACE (Lai et al., 2017) consists of about 28,000 passages and 100,000 questions from English exams for middle (RACE-M) and high (RACE-H) schools of China. The average number of sentences per passage in RACE-M and RACE-H is about 16 and 17, respectively. DREAM (Sun et al., 2019) contains 10,197 multiple-choice questions with 6,444 dialogues, collected from English examinations. In DREAM, 85% of the questions require reasoning with multiple evidential sentences. MultiRC (Khashabi et al., 2018) is an MMRC dataset where the amount of correct options to each question varies from 1 to 10. Each question in MultiRC is annotated with evidence from its reference document. The average number of annotated evidence sentences for each question is 2.3. Quasar-T (Dhingra et al., 2017b) consists of 43,000 open-domain trivial questions, whose answers were extracted from ClueWeb09. For fair comparison, we retrieved 50 reference sentences from ClueWeb09 for each question the same as DSQA (Lin et al., 2018). We compared several methods in our experiments, including some powerful base models without evidence supervision and some existing methods (*+Rule/RL/DPL/STM), which improve MRC with noisy evidence labels. Experimental details are shown in the appendix. YNQA and MMRC : (1) BERT-MLP utilizes a BERT encoder and an MLP answer predictor. The predictor makes classification based on the BERT representation at the position of [CLS] . The parameters of the BERT module were initialized from BERT-base. (2) BERT-HA refers to the base model introduced in Section 3.2, which applies hierarchical attention over words and sentences. (3) Based on BERT-HA, BERT-HA+Rule supervises the evidence extractor with noisy evidence labels, which are derived from hand-crafted rules. We have explored three types of rules based on Jaccard similarity, integer linear programming original dataset. (ILP) (Boudin et al., 2015), and inverse term frequency (ITF) (Wang et al., 2019), among which ITF performed best in most cases. For simplicity, we merely provided experimental results with the rule of ITF. (4) Based on BERT-HA, BERT-HA+RL trains the evidence extractor via reinforcement learning, similar to (Choi et al., 2017). And (5) another deep programming logic (DPL) method, GPT+DPL (Wang et al., 2019), is complicated, and the source code is not provided. Thus we directly used the results from the original paper and did not evaluate it on BERT. ODQA : (1) For each question, DSQA (Lin et al., 2018) aggregates multiple relevant paragraphs from ClueWeb09, and then infers an answer from these paragraphs. (2) GA (Dhingra et al., 2017a) and BiDAF (Seo et al., 2017) perform semantic matching between questions and paragraphs with attention mechanisms. And (3) R 3 (Wang et al., 2018) is a reinforcement learning method that explicitly selects the most relevant paragraph to a given question for the subsequent reading comprehension module. Table 2 shows the results on the three YNQA datasets. We merely reported the classification accuracy on the development sets since the test sets are unavailable. BERT-HA+STM outperformed all the baselines, which demonstrates the effectiveness of our method. Compared with BERT-MLP, BERT-HA achieved better performance on all the three Model EM F1 GA (Dhingra et al., 2017a) 26.4 26.4 BiDAF (Seo et al., 2017) 25.9 28.5 R 3 (Wang et al., 2018) 35.3 41.7 DSQA (Lin et al., 2018) 40.7 47.6 +distant supervision 41.7 48.7 +STM 41.8  49.2  Table 4: Experimental results on the test set of Quasar-T. datasets, indicating that distant supervision on evidence extraction can benefit Yes-No question answering. However, compared with BERT-HA, BERT-HA+RL made no improvement on MARCO and BoolQ, possibly due to the high variance in training. Similarly, BERT-HA+Rule performed worse than BERT-HA on CoQA and MARCO, implying that it is more difficult for the rule-based methods (inverse term frequency) to find correct evidence in these two datasets. In contrast, our method BERT-HA+STM is more general and performed the best on all datasets. BERT-HA+STM achieved comparable performance with BERT-HA+Gold, which stands for the upper bound by providing golden evidence labels, indicating that the effectiveness of noisy labels in our method. Table 3 shows the experimental results on the three MMRC datasets. We adopt the metrics from the referred papers. STM improved BERT-HA consis-Model/Dataset CoQA MultiRC P@1 R@1 R@2 R@3 P@1 P@2 P@3 BERT-HA 20.0 28.2 49.8 62.5 62.3 55.2 46.6 +RL 5.2 10.5 22.3 32.9 24.0 25.3 24.7 +Rule 38.4 32.4 53.6 65.1 71.8 59.6 48.7 +STM (iter 1) 32.7 32.8 57.1 70.1 72.2 63.3 52.5 +STM (iter 2) 37.3 32.9 58.0 71.3 72.7 64.4 53.5 +STM (iter 3) 39.9 31.4 55.3 68.8 69.5 61.6 51.6 BERT-HA+Gold 53.6 33.7 59.5 73.4 74.5 65.9 54.8 Table 5: Evidence extraction evaluation on the development sets of CoQA and MultiRC. tently on RACE-H, MultiRC and DREAM in terms of all the metrics. However, the improvement on RACE-M is limited ( 1 . 0 gain on the test sets). The reason may be that RACE-M is much simpler than RACE-H, and thus, it is not challenging for the evidence extractor of BERT-HA to find the correct evidence on RACE-M. Table 4 shows the exact match scores and F1 scores on Quasar-T. Distant evidence supervision (DS) indicates whether a passage contains the answer text. Compared with the base models DSQA and DSQA+DS, DSQA+STM achieved better performance in both metrics, which verifies that DSQA can also benefit from Self-Training. Our method is general and can improve both lightweight and heavyweight models, like LSTM-based and BERT-based models, in different tasks. To evaluate the performance of STM on evidence extraction, we validated the evidence labels generated by several methods on the development sets of CoQA and MultiRC. Considering that the evidence of each question in MultiRC is a set of sentences, we adopted precision @ k and recall @ k as the metrics for MultiRC, which represent the precision and recall of the generated evidence labels, respectively, when k sentences are predicted as evidence. We adopted only precision @1 as the metric for CoQA as this dataset provides each question with one golden evidence sentence. Table 5 shows the performance of five methods for evidence labeling on the CoQA and MultiRC development sets. It can be seen that BERT-HA+STM outperformed the base model BERT-HA by a large margin in terms of all the metrics. As a result, the evidence extractor augmented with STM provided more evidential information for the answer predictor, which may explain the improvements of BERT-HA+STM on the two datasets. To examine whether error propagation exists and how severe it is in STM, we visualized the evolution of evidence predictions on the development set of CoQA (Figure 3). From the inside to the outside, the four rings show the statistic results of the evidence predicted by BERT-HA (iteration 0) and BERT-HA+STM (iteration 1, 2, 3). Each ring is composed of all the instances from the development set of CoQA, and each radius corresponds to one sample. If the evidence of an instance is predicted correctly, the corresponding radius is marked in green, otherwise in purple. Two examples are shown in the appendix due to space limit. Self-correction. As the innermost ring shows, about 80% of the evidence predicted by BERT-HA (iter 0) was incorrect. However, the proportion of wrong instances reduced to 60% after self-training (iter 3). More concretely, 27% of the wrong predictions were gradually corrected with high confidence within three self-training iterations, as exemplified by instance A in Figure 3. Error propagation. We observed that 4% of the evidence was mistakenly revised by STM, as exemplified by instance B in Figure 3. In such a case, the incorrect predictions are likely to be retained in the next iteration. But almost 50% of such mistakes were finally corrected during the subsequent iterations like instance C . This observation shows that STM can prevent error propagation to avoid catastrophic failure. To evaluate the improvement of STM over stronger pre-trained models, we employed RoBERTa-large (Liu et al., 2019) as the encoder in the base model. Table 6 shows the results on CoQA. STM significantly improved the evidence extraction (Evi. Acc) of the base model. However, the improvement on answer prediction (Ans. Acc) is marginal. One reason is that RoBERTa-HA achieved such a high performance that there was limited room to improve. Another possible explanation is that evidence information is not important for such stronger models to generate answers. In other words, they may be more adept at exploiting data bias to make answer prediction. In comparison, weaker pre-trained models, such as BERT-base, can benefit from evidence information due to their weaker ability to exploit data bias. We present an iterative self-training method (STM) to improve MRC models with soft evidence extraction, when golden evidence labels are unavailable. In this iterative method, we train the base model with golden answers and pseudo evidence labels. The updated model then generates new pseudo evidence labels, which can be used as additional supervision in the next iteration. Experiment results show that our proposed method consistently improves the base models in seven datasets for three MRC tasks, and that better evidence extraction indeed enhances the final performance of MRC. As future work, we plan to extend our method to other NLP tasks which rely on evidence finding, such as natural language inference. This work was jointly supported by the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096), and the National Key R&D Program of China (Grant No. 2018YFC0830200). We thank THUNUS NExT Joint-Lab for the support.
abstain abstain abstain method abstain abstain method objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective objective objective objective objective objective objective objective objective abstain abstain result abstain method abstain objective objective method result abstain objective other other other other other other other other other other other other other other other other other other method other other other other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain method method abstain method abstain method method method method method method method method method abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result method abstain objective objective other other other