Many NLP tasks such as tagging and machine reading comprehension (MRC) are faced with the severe data imbalance issue: negative examples significantly outnumber positive ones, and the huge number of easy-negative examples overwhelms training. The most commonly used cross entropy criteria is actually accuracy-oriented, which creates a discrepancy between training and test. At training time, each training instance contributes equally to the objective function, while at test time F1 score concerns more about positive examples. In this paper, we propose to use dice loss in replacement of the standard cross-entropy objective for data-imbalanced NLP tasks. Dice loss is based on the SrensenDice coefficient (Sorensen, 1948) or Tversky index (Tversky, 1977), which attaches similar importance to false positives and false negatives, and is more immune to the data-imbalance issue. To further alleviate the dominating influence from easy-negative examples in training, we propose to associate training examples with dynamically adjusted weights to deemphasize easy-negative examples. Experimental results show that this strategy narrows down the gap between the F1 score in evaluation and the dice loss in training. With the proposed training objective, we observe significant performance boosts over a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5, CTB6 and UD1.4 for the part of speech tagging task, and competitive or even better results on CoNLL03, OntoNotes5.0, MSRA and OntoNotes4.0 for the named entity recognition task along with the machine reading comprehension and paraphrase identification tasks. The code can be found at https://github.com/ShannonAI/ dice_loss_for_NLP . Task # neg # pos ratio CoNLL03 NER 170K 34K 4.98 OntoNotes5.0 NER 1.96M 239K 8.18 SQuAD 1.1 (Rajpurkar et al., 2016) 10.3M 175K 55.9 SQuAD 2.0 (Rajpurkar et al., 2018) 15.4M 188K 82.0 QUOREF (Dasigi et al., 2019) 6.52M 38.6K 169 Table 1: Number of positive and negative examples and their ratios for different data-imbalanced NLP tasks. Data imbalance is a common issue in a variety of NLP tasks such as tagging and machine reading comprehension. Table 1 gives concrete examples: for the Named Entity Recognition (NER) task (Sang and De Meulder, 2003; Nadeau and Sekine, 2007), most tokens are backgrounds with tagging class O . Specifically, the number of tokens with tagging class O is 5 times as many as those with entity labels for the CoNLL03 dataset and 8 times for the OntoNotes5.0 dataset; Data-imbalanced issue is more severe for MRC tasks (Rajpurkar et al., 2016; Nguyen et al., 2016; Ra-jpurkar et al., 2018; Kocisk`y et al., 2018; Dasigi et al., 2019) with the value of negative-positive ratio being 50-200, which is due to the reason that the task of MRC is usually formalized as predicting the starting and ending indexes conditioned on the query and the context, and given a chunk of text of an arbitrary length, only two tokens are positive (or of interest) with all the rest being background. Data imbalance results in the following two issues: (1) the training-test discrepancy : Without balancing the labels, the learning process tends to converge to a point that strongly biases towards class with the majority label. This actually creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function, whereas at test time, F1 gives equal weight to positive and negative examples; (2) the overwhelming effect of easy-negative examples . As pointed out by Meng et al. (2019), a significantly large number of negative examples also means that the number of easy-negative example is large. The huge number of easy examples tends to overwhelm the training, making the model not sufficiently learn to distinguish between positive examples and hard-negative examples. The cross-entropy objective (CE for short) or maximum likelihood (MLE) objective, which is widely adopted as the training objective for data-imbalanced NLP tasks (Lample et al., 2016; Wu et al., 2019; Devlin et al., 2018; Yu et al., 2018a; McCann et al., 2018; Ma and Hovy, 2016; Chen et al., 2017), handles neither of the issues. To handle the first issue, we propose to replace CE or MLE with losses based on the SrensenDice coefficient (Sorensen, 1948) or Tversky index (Tver-sky, 1977). The SrensenDice coefficient, dice loss for short, is the harmonic mean of precision and recall. It attaches equal importance to false positives (FPs) and false negatives (FNs) and is thus more immune to data-imbalanced datasets. Tversky index extends dice loss by using a weight that trades precision and recall, which can be thought as the approximation of the F  score, and thus comes with more flexibility. Therefore, we use dice loss or Tversky index to replace CE loss to address the first issue. Only using dice loss or Tversky index is not enough since they are unable to address the dominating influence of easy-negative examples. This is intrinsically because dice loss is actually a soft version of the F1 score. Taking the binary classification task as an example, at test time, an example will be classified as negative as long as its probability is smaller than 0.5, but training will push the value to 0 as much as possible. This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily. Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones. Inspired by the idea of focal loss (Lin et al., 2017) in computer vision, we propose a dynamic weight adjusting strategy, which associates each training example with a weight in proportion to (1  p ) , and this weight dynamically changes as training proceeds. This strategy helps deemphasize confident examples during training as their probability p approaches 1 , making the model attentive to hard-negative examples, and thus alleviates the dominating effect of easy-negative examples. Combing both strategies, we observe significant performance boosts on a wide range of data imbalanced NLP tasks. The rest of this paper is organized as follows: related work is presented in Section 2. We describe different proposed losses in Section 3. Experimental results are presented in Section 4. We perform ablation studies in Section 5, followed by a brief conclusion in Section 6. The idea of weighting training examples has a long history. Importance sampling (Kahn and Marshall, 1953) assigns weights to different samples and changes the data distribution. Boosting algorithms such as AdaBoost (Kanduri et al., 2018) select harder examples to train subsequent classi-fiers. Similarly, hard example mining (Malisiewicz et al., 2011) downsamples the majority class and exploits the most difficult examples. Oversampling (Chen et al., 2010; Chawla et al., 2002) is used to balance the data distribution. Another line of data resampling is to dynamically control the weights of examples as training proceeds. For example, focal loss (Lin et al., 2017) used a soft weighting scheme that emphasizes harder examples during training. In self-paced learning (Kumar et al., 2010), example weights are obtained through optimizing the weighted training loss which encourages learning easier examples first. At each training step, self-paced learning algorithm optimizes model parameters and example weights jointly. Other works (Chang et al., 2017; Katharopoulos and Fleuret, 2018) adjusted the weights of different training examples based on training loss. Besides, recent work (Jiang et al., 2017; Fan et al., 2018) proposed to learn a separate network to predict sample weights. The background-object label imbalance issue is severe and thus well studied in the field of object detection (Li et al., 2015; Girshick, 2015; He et al., 2015; Girshick et al., 2013; Ren et al., 2015). The idea of hard negative mining (HNM) (Girshick et al., 2013) has gained much attention recently. Pang et al. (2019) proposed a novel method called IoU-balanced sampling and Chen et al. (2019) designed a ranking model to replace the conventional classification task with an average-precision loss to alleviate the class imbalance issue. Sudre et al. (2017) addressed the severe class imbalance issue for the image segmentation task. They proposed to use the class re-balancing property of the Generalized Dice Loss as the training objective for unbalanced tasks. Shen et al. (2018) investigated the influence of Dice-based loss for multi-class organ segmentation using a dataset of abdominal CT volumes. Kodym et al. (2018) proposed to use the batch soft Dice loss function to train the CNN network for the task of segmentation of organs at risk (OAR) of medical images. Shamir et al. (2019) extended the definition of the classical Dice coefficient to facilitate the direct comparison of a ground truth binary image with a probabilistic map. In this paper, we introduce dice loss into NLP tasks as the training objective and propose a dynamic weight adjusting strategy to address the dominating influence of easy-negative examples. For illustration purposes, we use the binary classification task to demonstrate how different losses work. The mechanism can be easily extended to multi-class classification. Let X denote a set of training instances and each instance x i  X is associated with a golden binary label y i = [ y i 0 , y i 1 ] denoting the ground-truth class x i belongs to, and p i = [ p i 0 , p i 1 ] is the predicted probabilities of the two classes respectively, where y i 0 , y i 1  { 0 , 1 } , p i 0 , p i 1  [0 , 1] and p i 1 + p i 0 = 1 . The vanilla cross entropy (CE) loss is given As can be seen from Eq.1, each x i contributes equally to the final objective. Two strategies are normally used to address the the case where we wish that not all x i are treated equally: associating different classes with different weighting factor  or resampling the datasets. For the former, Eq.1 is adjusted as follows: Weighted CE =  1 N (cid:88) i  i (cid:88) j { 0 , 1 } y ij log p ij (2) where  i  [0 , 1] may be set by the inverse class frequency or treated as a hyperparameter to set by cross validation. In this work, we use lg( n  n t n t + K ) to calculate the coefficient  , where n t is the number of samples with class t and n is the total number of samples in the training set. K is a hyperparameter to tune. Intuitively, this equation assigns less weight to the majority class and more weight to the minority class. The data resampling strategy constructs a new dataset by sampling training examples from the original dataset based on human-designed criteria, e.g. extracting equal training samples from each class. Both strategies are equivalent to changing the data distribution during training and thus are of the same nature. Empirically, these two methods are not widely used due to the trickiness of selecting  especially for multi-class classification tasks and that inappropriate selection can easily bias towards rare classes (Valverde et al., 2017). SrensenDice coefficient (Sorensen, 1948; Dice, 1945), dice coefficient (DSC) for short, is an F1-oriented statistic used to gauge the similarity of two sets. Given two sets A and B , the vanilla dice coefficient between them is given as follows: DSC ( A, B ) = 2 | A  B | | A | + | B | (3) In our case, A is the set that contains all positive examples predicted by a specific model, and B is the set of all golden positive examples in the dataset. When applied to boolean data with the definition of true positive (TP), false positive (FP), and false negative (FN), it can be then written as follows: DSC = 2 TP 2 TP + FN + FP = 2 TP TP + FN TP TP + FP TP TP + FN + TP TP + FP = 2 Pre  Rec Pre+Rec = F 1 (4) For an individual example x i , its corresponding dice coefficient is given as follows: DSC ( x i ) = 2 p i 1 y i 1 p i 1 + y i 1 (5) As can be seen, a negative example ( y i 1 = 0 ) does not contribute to the objective. For smoothing purposes, it is common to add a  factor to both the nominator and the denominator, making the form to be as follows (we simply set  = 1 in the rest of Loss Formula (one sample x i ) CE  (cid:80) j { 0 , 1 } y ij log p ij WCE   i (cid:80) j { 0 , 1 } y ij log p ij DL 1  2 p i 1 y i 1 +  p 2 i 1 + y 2 i 1 +  TL 1  p i 1 y i 1 +  p i 1 y i 1 +  p i 1 y i 0 +  p i 0 y i 1 +  DSC 1  2(1  p i 1 ) p i 1  y i 1 +  (1  p i 1 ) p i 1 + y i 1 +  FL   i (cid:80) j { 0 , 1 } (1  p ij )  log p ij Table 2: Different losses and their formulas. We add +1 to DL, TL and DSC so that they are positive. this paper): DSC ( x i ) = 2 p i 1 y i 1 +  p i 1 + y i 1 +  (6) As can be seen, negative examples whose DSC is  p i 1 +  , also contribute to the training. Additionally, Milletari et al. (2016) proposed to change the denominator to the square form for faster convergence, which leads to the following dice loss (DL): DL = 1 N (cid:88) i (cid:20) 1  2 p i 1 y i 1 +  p 2 i 1 + y 2 i 1 +  (cid:21) (7) Another version of DL is to directly compute set-level dice coefficient instead of the sum of individual dice coefficient, which is easier for optimization: DL = 1  2 (cid:80) i p i 1 y i 1 +  (cid:80) i p 2 i 1 + (cid:80) i y 2 i 1 +  (8) Tversky index (TI), which can be thought as the approximation of the F  score, extends dice coefficient to a more general case. Given two sets A and B , tversky index is computed as follows: TI = | A  B | | A  B | +  | A \ B | +  | B \ A | (9) Tversky index offers the flexibility in controlling the tradeoff between false-negatives and false-positives. It degenerates to DSC if  =  = 0 . 5 . The Tversky loss (TL) is thus given as follows: TL = 1 N (cid:88) i (cid:20) 1  p i 1 y i 1 +  p i 1 y i 1 +  p i 1 y i 0 +  p i 0 y i 1 +  (cid:21) (10) 3.4 Self-adjusting Dice Loss Consider a simple case where the dataset consists of only one example x i , which is classified as positive as long as p i 1 is larger than 0.5. The computation of F 1 score is actually as follows: F1 ( x i ) = 2 I ( p i 1 > 0 . 5) y i 1 I ( p i 1 > 0 . 5) + y i 1 (11) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 D e r i v a ti v e s FL( =1) DL( =1) TL( =0.5) DSC Figure 1: An illustration of derivatives of the four losses. Comparing Eq.5 with Eq.11, we can see that Eq.5 is actually a soft form of F 1 , using a continuous p rather than the binary I ( p i 1 > 0 . 5) . This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily. Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones, which has a huge negative effect on the final F1 performance. To address this issue, we propose to multiply the soft probability p with a decaying factor (1  p ) , changing Eq.11 to the following adaptive variant of DSC: DSC ( x i ) = 2(1  p i 1 ) p i 1  y i 1 +  (1  p i 1 ) p i 1 + y i 1 +  (12) One can think (1  p i 1 ) as a weight associated with each example, which changes as training proceeds. The intuition of changing p i 1 to (1  p i 1 ) p i 1 is to push down the weight of easy examples. For easy examples whose probability are approaching 0 or 1, (1  p i 1 ) p i 1 makes the model attach significantly less focus to them. A close look at Eq.12 reveals that it actually mimics the idea of focal loss (FL for short) (Lin et al., 2017) for object detection in vision. Focal loss was proposed for one-stage object detector to handle foreground-background tradeoff encountered during training. It down-weights the loss assigned to well-classified examples by adding a (1  p )  factor, leading the final loss to be  (1  p )  log p . In Table 2, we summarize all the aforementioned losses. Figure 1 gives an explanation from the perspective in derivative: The derivative of DSC approaches zero right after p exceeds 0.5, which suggests the model attends less to examples once they are correctly classified. But for the other losses, the derivatives reach 0 only if the probability is exactly 1, which means they will push p to 1 as much as possible. We evaluated the proposed method on four NLP tasks, part-of-speech tagging, named entity recognition, machine reading comprehension and paraphrase identification. Hyperparameters are tuned on the corresponding development set of each dataset. More experiment details including datasets and hyperparameters are shown in supplementary material. Settings Part-of-speech tagging (POS) is the task of assigning a part-of-speech label (e.g., noun, verb, adjective) to each word in a given text. In this paper, we choose BERT (Devlin et al., 2018) as the backbone and conduct experiments on three widely used Chinese POS datasets including Chinese Treebank (Xue et al., 2005) 5.0/6.0 and UD1.4 and English datasets including Wall Street Journal (WSJ) and the dataset proposed by Ritter et al. (2011). We report the span-level micro-averaged precision, recall and F1 for evaluation. Baselines We used the following baselines:  Joint-POS: Shao et al. (2017) jointly learns Chinese word segmentation and POS. Lattice-LSTM: Zhang and Yang (2018) constructs a word-character lattice network. Bert-Tagger: Devlin et al. (2018) treats part-of-speech as a tagging task. Results Table 3 presents the experimental results on Chinese datasets. As can be seen, the proposed DSC loss outperforms the best baseline results by a large margin, i.e., outperforming BERT-tagger by +1.86 in terms of F1 score on CTB5, +1.80 on CTB6 and +2.19 on UD1.4. As far as we know, we are achieving SOTA performances on the three datasets. Focal loss only obtains a little performance improvement on CTB5 and CTB6, and the dice loss obtains huge gain on CTB5 but not on CTB6, which indicates the three losses are not consistently robust in solving the data imbalance issue. Table 4 presents the experimental results for English datasets. Settings Named entity recognition (NER) is the task of detecting the span and semantic category of entities within a chunk of text. Our implementation uses the current state-of-the-art model proposed by Li et al. (2019) as the backbone, and changes the MLE loss to DSC loss. Datasets that we use include OntoNotes4.0 (Pradhan et al., 2011), MSRA (Levow, 2006), CoNLL2003 (Sang and Meulder, 2003) and OntoNotes5.0 (Pradhan et al., 2013). We report span-level micro-averaged precision, recall and F1. Baselines We use the following baselines:  ELMo: a tagging model with pretraining from Peters et al. (2018). Lattice-LSTM: Zhang and Yang (2018) constructs a word-character lattice, only used in Chinese datasets. CVT: Clark et al. (2018) uses Cross-View Training(CVT) to improve the representations of a Bi-LSTM encoder. Bert-Tagger: Devlin et al. (2018) treats NER as a tagging task. Glyce-BERT: Wu et al. (2019) combines Chinese glyph information with BERT pretraining. BERT-MRC: Li et al. (2019) formulates NER as a machine reading comprehension task and achieves SOTA results on Chinese and English NER benchmarks. Results Table 5 shows experimental results on NER datasets. DSC outperforms BERT-MRC(Li et al., 2019) by +0.29, +0.96, +0.97 and +2.36 respectively on CoNLL2003, OntoNotes5.0, MSRA and OntoNotes4.0. As far as we are concerned, we are setting new SOTA performances on all of the four NER datasets. Settings The task of machine reading comprehension (MRC) (Seo et al., 2016; Wang et al., 2016; Wang and Jiang, 2016; Wang et al., 2016; Shen et al., 2017; Chen et al., 2017) predicts the answer span in the passage given a question and the passage. We followed the standard protocols in Seo et al. (2016), in which the start and end indexes of answer are predicted. We report Extract Match (EM) as well as F1 score on validation set. We use three datasets on this task: SQuAD v1.1, SQuAD v2.0 (Rajpurkar et al., 2016, 2018) and Quoref (Dasigi et al., 2019). QANet: Yu et al. (2018b) builds a model based on convolutions and self-attentions. Convolutions are used to model local interactions and self-attention are used to model global interactions. BERT: Devlin et al. (2018) scores each candidate span and the maximum scoring span is used as a prediction. XLNet: Yang et al. (2019) proposes a generalized autoregressive pretraining method that SQuAD v1.1 SQuAD v2.0 QuoRef Model EM F1 EM F1 EM F1 QANet (Yu et al., 2018b) 73.6 82.7 -34.41 38.26 BERT (Devlin et al., 2018) 84.1 90.9 78.7 81.9 58.44 64.95 BERT+FL 84.67 91.25 78.92 82.20 60.78 66.19 (+0.57) (+0.35) (+0.22) (+0.30) (+2.34) (+1.24) BERT+DL 84.83 91.86 78.99 82.88 62.03 66.88 (+0.73) (+0.96) (+0.29) (+0.98) (+3.59) (+1.93) BERT+DSC 85.34 91.97 79.02 82.95 62.44 67.52 (+1.24) (+1.07) (+0.32) (+1.05) (+4.00) (+2.57) XLNet (Yang et al., 2019) 88.95 94.52 86.12 88.79 64.52 71.49 XLNet+FL 88.90 94.55 87.04 89.32 65.19 72.34 (-0.05) (+0.03) (+0.92) (+0.53) (+0.67) (+0.85) XLNet+DL 89.13 95.36 87.22 89.44 65.77 72.85 (+0.18) (+0.84) (+1.10) (+0.65) (+1.25) (+1.36) XLNet+DSC 89.79 95.77 87.65 89.51 65.98 72.90 (+0.84) (+1.25) (+1.53) (+0.72) (+1.46) (+1.41) Table 6: Experimental results for MRC task. enables learning bidirectional contexts. Results Table 6 shows the experimental results for MRC task. With either BERT or XLNet, our proposed DSC loss obtains significant performance boost on both EM and F1. For SQuADv1.1, our proposed method outperforms XLNet by +1.25 in terms of F1 score and +0.84 in terms of EM. For SQuAD v2.0, the proposed method achieves 87.65 on EM and 89.51 on F1. On QuoRef, the proposed method surpasses XLNet by +1.46 on EM and +1.41 on F1. Settings Paraphrase identification (PI) is the task of identifying whether two sentences have the same meaning or not. We conduct experiments on the two widely-used datasets: MRPC (Dolan and Brockett, 2005) and QQP. F1 score is reported for comparison. We use BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019) as baselines. Results Table 7 shows the results. We find that replacing the training objective with DSC introduces performance boost for both settings, +0.58 for MRPC and +0.73 for QQP. It is interesting to see how differently the proposed objectives affect datasets imbalanced to different extents. We use the paraphrase identification dataset QQP (37% positive and 63% negative) for studies. To construct datasets with different imbalance degrees, we used the original QQP dataset to construct synthetic training sets with different positive-negative ratios. Models are trained on these different synthetic sets and then test on the same original test set. Original training set (original) The original dataset with 363,871 examples, with 37% being positive and 63% being negative  Positive augmentation (+ positive) We created a balanced dataset by adding positive examples. We first randomly chose positive training examples in the original training set as templates. Then we used Spacy 1 to retrieve entity mentions and replace them with new ones by linking mentions to their corresponding entities in DBpedia. The augmented set contains 458,477 examples, with 50% being positive and 50% being negative. Negative augmentation (+ negative) We created a more imbalanced dataset. The size of the newly constructed training set and 1 https://github.com/explosion/spaCy original + positive + negative negative + positive & negative BERT 91.3 92.27 90.08 89.73 93.14 BERT+FL 91.86(+0.56) 92.64(+0.37) 90.61(+0.53) 90.79(+1.06) 93.45(+0.31) BERT+DL 91.92(+0.62) 92.87(+0.60) 90.22(+0.14) 90.49(+0.76) 93.52(+0.38) BERT+DSC 92.11(+0.81) 92.92(+0.65) 90.78(+0.70) 90.80(+1.07) 93.63(+0.49) Table 8: The effect of different data augmentation ways for QQP in terms of F1-score. the data augmented technique are exactly the same as +negative , except that we chose negative training examples as templates. The augmented training set contains 458,477 examples, with 21% being positive and 79% being negative. Negative downsampling (negative) We down-sampled negative examples in the original training set to get a balanced training set. The down-sampled set contains 269,165 examples, with 50% being positive and 50% being negative. Positive and negative augmentation (+ positive & +negative) We augmented the original training data with additional positive and negative examples with the data distribution staying the same. The augmented dataset contains 458,477 examples, with 50% being positive and 50% being negative. Results are shown in Table 8. We first look at the first line, with all results obtained using the MLE objective. We can see that + positive outperforms original , and +negative underperforms original . This is in line with our expectation since + positive creates a balanced dataset while +negative creates a more imbalanced dataset. Despite the fact that -negative creates a balanced dataset, the number of training data decreases, resulting in inferior performances. DSC achieves the highest F1 score across all datasets. Specially, for +positive , DSC achieves minor improvements (+0.05 F1) over DL. In contrast, it significantly outperforms DL for +negative dataset. This is in line with our expectation since DSC helps more on more imbalanced datasets. The performance of FL and DL are not consistent across different datasets, while DSC consistently performs the best on all datasets. We argue that the cross-entropy objective is actually accuracy-oriented, whereas the proposed losses perform as a soft version of F1 score. To SST-2 SST-5 Model Acc Acc BERT+CE 94.90 55.57 BERT+DL 94.37 54.63 BERT+DSC 94.84 55.19 Table 9: The effect of DL and DSC on sentiment classification tasks. explore the effect of the dice loss on accuracy-oriented tasks such as text classification, we conduct experiments on the Stanford Sentiment Treebank (SST) datasets including SST-2 and SST-5. We fine-tuned BERT Large with different training objectives. Experimental results for SST are shown in Table 9. For SST-5, BERT with CE achieves 55.57 in terms of accuracy, while DL and DSC perform slightly worse (54.63 and 55.19, respec-tively). Similar phenomenon is observed for SST-2. These results verify that the proposed dice loss is not accuracy-oriented, and should not be used for accuracy-oriented tasks. As mentioned in Section 3.3, Tversky index (TI) offers the flexibility in controlling the tradeoff between false-negatives and false-positives. In this subsection, we explore the effect of hyperparameters (i.e.,  and  ) in TI to test how they manipulate the tradeoff. We conduct experiments on the Chinese OntoNotes4.0 NER dataset and English QuoRef MRC dataset. Experimental results are shown in Table 10. The highest F1 on Chinese OntoNotes4.0 is 84.67 when  is set to 0.6 while for QuoRef, the highest F1 is 68.44 when  is set to 0.4. In addition, we can observe that the performance varies a lot as  changes in distinct datasets, which shows that the hyperparameters ,  acturally play an important role in TI. In this paper, we propose the dice-based loss to narrow down the gap between training objective and evaluation metrics (F1 score). Experimental results show that the proposed loss function help  Chinese Onto4.0 English QuoRef  = 0 . to achieve significant performance boost without changing model architectures. We thank all anonymous reviewers, as well as Qinghong Han, Wei Wu and Jiawei Wu for their comments and suggestions. The work is supported by the National Natural Science Foundation of China (NSFC No. 61625107 and 61751209).
abstain abstain abstain objective abstain objective abstain objective result other other abstain abstain other abstain abstain abstain abstain other objective abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain result method objective abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method other other method abstain other abstain abstain abstain other other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain method other abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective abstain abstain other other