Recent studies in deep learning have shown significant progress in named entity recognition (NER). Most existing works assume clean data annotation, yet a fundamental challenge in real-world scenarios is the large amount of noise from a variety of sources (e.g., pseudo, weak, or distant annotations). This work studies NER under a noisy labeled setting with calibrated confidence estimation. Based on empirical observations of different training dynamics of noisy and clean labels, we propose strategies for estimating confidence scores based on local and global independence assumptions. We partially marginalize out labels of low confidence with a CRF model. We further propose a calibration method for confidence scores based on the structure of entity labels. We integrate our approach into a self-training framework for boosting performance. Experiments in general noisy settings with four languages and distantly labeled settings demonstrate the effectiveness of our method 1 . Recent progress in deep learning has significantly advanced NER performances (Lample et al., 2016; Devlin et al., 2018). While most existing works assume clean data annotation, real-world data inevitably involve different levels of noise (e.g., distant supervision from the dictionary (Peng et al., 2019), or weak supervision from the web Vrandecic and Krtzsch, 2014; Cao et al., 2019a). Figure 1 gives an example of such noisy labels. To train robust models with high performance, it is fundamentally critical to tackle the challenges associated with noisy data annotation. In this work, we propose a confidence estimation approach for NER with noisy labels. We motivate  Equal Contribution. our approach with important empirical observations of the training dynamics of clean and noisy labels: usually, clean data are easier to fit with faster convergence and smaller loss values (Jiang et al., 2018; Han et al., 2018a; Arazo et al., 2019). Consequently, loss values (probabilities or scores of labels) can serve as strong indicators for the existence of noise, which we utilize to build our confidence estimation. The key contribution of this work is a confidence estimation method with calibration. We use probabilities of labels as confidence scores and apply two estimation strategies based on global or local normalization that assume different dependency structures about how the noisy labels are generated. We further calibrate the confidence score for positive labels (labels representing entity parts, e.g., B-LOC ) based on the structure of these labels: we separately estimate scores for the position part (e.g., B in B-LOC ) and the type part (e.g., LOC in B-LOC ). Such fine-grained calibration leads to a more accurate estimation and better performance in our experiments. We apply our method in a CRF model (Bellare and McCallum, 2007; Yang et al., 2018), marginalize out labels we do not trust, and maximize the likelihood of trusted labels. We use a self-training approach (Jie et al., 2019) that iteratively estimates confidence scores in multiple training iterations and re-annotates the data at each iteration. Experiments show that our approach outperforms baselines on a general noisy-labeled setting with datasets in four languages and shows promising results on a distantly-labeled setting with four datasets. Given a sentence x = [ x 1 , ..., x n ] and its tag sequence  y 1 , ...,  y n , n is the sentence length. We model the conditional probability of y with a bidirectional LSTM-CRF (Huang et al., 2015): h = BiLSTM ( x )  i = Linear ( h i ) (1) p ( y | x ) = ( y ) /Z , Z = Forward () (2) Where h denotes LSTM states, Linear (  ) denotes a linear layer, ( y ) denotes the potential (weight) evaluated for tag sequence y , Z denotes the partition function,  denotes the forward variables, and Forward (  ) denotes the Forward algorithm (Sutton and McCallum, 2006). The advantage of the CRF model is that it gives us a probabilistically uniform way to handle labels we do or do not trust by partial marginalization, which we discuss later. Our confidence estimation model reuses the base LSTM-CRF architecture and assigns a confidence score s i for each  y i . A natural choice is to use the CRF marginal probability: s i = p ( y i | x ) p ( y i | x ) =  i  i /Z (3) where  is the backward variable and can be computed with the Backward algorithm (Sutton and McCallum, 2006). This strategy infers s i based on global-normalization and assumes strong dependency between consecutive labels. The intuition is that annotators are more likely to make mistakes on a label if they have already made mistakes on previous labels. Our second strategy makes a stronger local independence assumption and considers a noisy label at step i only relies on the word context, not the label context. To this end, we use a simple categorical distribution parameterized by a Softmax: s i = p ( y i | x ) p ( y i | x ) = Softmax ( i ) (4) Here we reuse the factor  i as the logits of the Softmax because in the CRF context it also means how likely a label y i may be observed given the input h i . Intuitively, this strategy assumes that annotators make mistakes solely based on words, no matter whether they have already made mistakes previously. We use s i to decide if we want to trust a label  y i and marginalize out labels we do not trust. Our marginalization relies on a threshold to determine the portion of trusted labels and the noise ratio that we believe the data contain. Given a batch of ( x,  y ) pairs, after confidence estimation, we collect all word-label-confidence triples into a set D = { x j ,  y j , s j } Nj =1 , N denotes total number of the triples. We further separate the estimation for positive labels (entities) and negative labels (i.e., the O label) because we empirically observe that their probabilities are consistently different. To this end, we divide D into positive and negative groups D p = { ( x j ,  y j , s j ) ,  y j  Y p } and D n = { ( x j ,  y j , s j ) ,  y j  Y n } , Y p and Y n denotes sets of positive and negative labels. We rank triples in D l ( l  {p, n}) according to confidence scores and retain the most confident r l ( e )  |D l | triples at epoch e as clean for which we do maximum likelihood. We view the remaining triples as noisy and marginalize them out. We update the keep ratio r l ( e ) at each epoch following Han et al. (2018b): r l ( e ) = 1  min (cid:110) e K  l ,  l (cid:111) , l  { p , n } (5) where  l is the ratio of noise that we believe in the training data. Basically this says we gradually decrease the epoch-wise keep ratio r l ( e ) to the full ratio 1   l after K epochs. We grid-search  l heuristically in experiments (results in Figure 3(b)). For positive cases in D p viewed as noisy according to the previous procedure, we do a further confidence calibration. Noting that a y i always take the form y pi y ti (position-type) (e.g. if y i = B-LOC , Method General Noise Distant Supervision En Sp Ge Du CoNLL Tweet Webpage Wikigold 1. BiLSTM-CRF 73.3 61.9 57.7 58.3 59.5 21.8 43.3 42.9 2. BiLSTM-CRF (clean data upper bound) 90.3 85.2 77.3 81.1 91.2 52.2 52.3 54.9 3. RoBERTa (clean data upper bound) --90.1 52.2 72.4 86.4 Proposed for General Noise Setting 4. NA (Hedderich and Klakow, 2018) 61.5 57.3 46.1 41.5 --5. CBL (Mayhew et al., 2019) 82.6 76.1 65.6 68.5 75.4 18.2 31.7 42.6 6. Self-training (Jie et al., 2019) 84.0 71.4 66.5 59.6 77.8 42.3 49.6 51.3 Proposed for Distant Supervision Setting 7. AutoNER (Shang et al., 2018) --67.0 26.1 51.4 47.5 8. LRNT (Cao et al., 2019a) --69.7 23.8 47.7 46.2 9. BOND (RoBERTa Liang et al., 2020) --81.5 48.0 65.7 60.1 Ours, best configurations 10. Ours (local,   ) 87.0 78.8 68.3 69.1 79.4 43.6 51.8 54.0 11. Ours (global,   ) 86.4 79.0 69.2 71.2 79.2 43.1 50.0 53.0 Ours, other possible configurations 12. Ours (local,  (cid:63) ) 86.2 79.2 68.2 67.2 --13. Ours (global,  (cid:63) ) 85.4 75.4 68.4 69.0 --14. Ours (local,   , w/o. calibration) 85.8 77.3 67.2 68.0 79.9 40.8 46.9 50.0 Ours with pretrained LM 15. Ours (local,   , BERT) --77.2 46.7 59.3 57.3 16. Ours (global,   , BERT) --78.9 47.3 61.9 57.7 Table 1: Results (F1%) on artificially perturbed datasets and distantly supervised datasets. then y pi = B and y ti = LOC ), an important assumption is that annotators are unlikely to mistake both parts  mistakes usually happen on only one of them. So we calculate two calibrated confidence scores s pi and s ti for  y pi and  y ti : s pi = 1 | Y ( y pi ) | (cid:88) y i p ( y i | x ) where y pi =  y pi (6) s ti = 1 | Y ( y ti ) | (cid:88) y i p ( y i | x ) where y ti =  y ti (7) where Y ( y ti ) denotes the set of labels sharing the same  y ti part, and Y ( y pi ) is defined similarily. If s pi > s ti , we trust the  y pi (position) part of the label and marginalize out all labels with different positions except for the O label. For example, in Figure 2, for the word Brooklyn we trust the all labels with the position B ( B-PER and B-LOC ) and the O label, sum over the tag sequences passing these labels, and reject other labels. Similar operation applies for cases where s pi < s t i (E.g., the word York ). For labels we do not trust in the negative group D n , we simply marginalize all labels out (E.g., the word New ). We maximize the partially marginalized probability (Bellare and McCallum, 2007):  p ( y | x ) = (cid:88) y   Y ( y ) /Z (8) where  Y denotes the set of tag sequences compatible with  y after confidence estimation. A concrete example is given in Figure 2. The summation in equation 8 can be calculated exactly with Forward-styled dynamic programming (Sasada et al., 2016). We integrate our approach into a self-training framework proposed by Jie et al. (2019). At each round, the training set is randomly divided into two parts for cross-validation. We iteratively reannotate half of the training set with a model trained on the other half. After a round, we use the updated training set to train the next round. General Noise. Following Mayhew et al. (2019), we first consider general noise by artificially perturbing the CoNLL dataset (Sang and De Meulder, 2003) on four languages including English, Spanish, German, and Dutch. Gold annotations are per-85 86 87 88 89 90 91 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 F 1 Noise rate Negative noise rate (local) Positive noise rate (local) Negative noise rate (global) Positive noise rate (global) Self-training (Jie et al., 2019) 00.10.20.30.40.50.60.70.80.9 0123456789 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 F 1 L o ss / NLL Epoch Negative NLL Positive NLL Training loss Negative noise detection F1 Positive noise detection F1 Dev F1 Oracle  ! turbed by: (a) tagging some entities to O to lower the recall to 0.5; (b) introducing some random positive tags to lower the precision to 0.9. We compare our methods with Noise Adaption (NA, Hedderich and Klakow, 2018), Self Training (Jie et al., 2019), and CBL (Mayhew et al., 2019). This setting is for testing our approach in a controlled environment. Distant Supervision. We consider four datasets including CoNLL03 (Sang and De Meulder, 2003), Tweet (Godin et al., 2015), Webpage (Ratinov and Roth, 2009), and Wikigold (Balasuriya et al., 2009). In this setting, the distantly supervised tags are generated by the dictionary following BOND (Liang et al., 2020). We compare our methods with AutoNER (Shang et al., 2018), LRNT (Cao et al., 2019a), and BOND. This setting aims to test our approach in a more realistic environment. Table 1 shows our primary results. We use local and global to denote locally / globally normalized confidence estimation strategies. We use oracle (unavailable in real settings) / searched  to denote how we obtain the prior noise ratio  . We note that the Self-training baseline (Jie et al., 2019, line 6) is the most comparable baseline since our confidence estimation is directly integrated into it. We primarily compare this baseline with our best configurations (line 10 and 11). We focus on the shaded results as they are the most informative for demonstrating our method. General Noise. Our methods (both local and global) outperforms the state-of-the-art method (Jie et al., 2019) by a large margin in three datasets (En, Sp, Du, line 10 and 11 v.s. 6), showing the effectiveness of our approach. We observe the oracle  does not necessarily give the best performance and an over estimate of confidence could leave a better performance. Ablation results without calibration further show the effectiveness of our calibration methods (line 10 v.s. 14). We note that the CoNLL dataset is an exception where the calibration slightly hurts performance. Otherwise the improvements with calibration is clear in the other 7 datasets. Distant Supervision. Our method outperforms AutoNER and LRNT without pre-trained language models. Reasons that we are worse than BOND (line 16 v.s. 6) are: (a) many implementation aspects are different, and it is (currently) challenging to transplant their settings to ours; (b) they use multiple tailored techniques for distantly-labeled data (e.g., the adversarial training), while ours is more general-purpose. Though our method does not outperform BOND, it still outperforms AutoNER and LRNT (under the setting all without pretrained model, line 10 and 11 v.s. 7 and 8) and shows promising gain. We conduct more detailed experiments on the general noise setting for more in-depth understanding. Training Dynamics (Figure 3(a)). As the model converges, as clean data converge faster, the confidence gap between the clean and the noisy is larger, thus the two are more confidently separated, so both noise detection F1 and dev F1 increase. Noise Rate Search (Figure 3(b)). Our method consistently outperforms baseline without confidence estimation. Lines tend to be higher at the right side of the figure, showing an over-estimate of noise tends to give better performance. Level of Noise (Figure 3(c)). In many real-world scenarios, the noise w.r.t. precision is more con-stant and it is the recall that varies. So we simulate the level of noise with different recall (lower recall = larger noise ratio). Our method outperforms baselines in all ratios and is particularly effective under a large noise ratio. Case Studies (Figure 4). The top three cases give examples of how our method detects: (1) false negative noise when an entity is not annotated, (2) entities with wrong boundaries and (3) wrong entity types. The last example (case 4) gives a failure case when the model treats some correct tags as noise due to our over-estimate of noise (for better end performance). State-of-the-art NER models (Ma and Hovy, 2016; Lample et al., 2016; Devlin et al., 2018) are all under the traditional assumption of clean data annotation. The key motivation of this work is the intrinsic gap between the clean data assumption and noisy real-world scenarios. We believe that the noisy label setting is fundamentally challenging in NER and all related supervised learning tasks. Previous works on NER with noise could be organized into two threads: (a) some works treat this task as learning with missing labels. Bellare and McCallum (2007) propose a missing label CRF to deal with partial annotation. Jie et al. (2019) propose a self-training framework with marginal CRF to re-annotate the missing labels. (b) other works treat missing labels as noise and try to avoid them in the training process. For example, Mayhew et al. (2019) train a binary classifier supervised by entity ratio to classify tokens into entities and nonentities. A widely-used way to collect NER annotations is distant supervision, which consequently becomes an important source of noise. Peng et al. (2019) formulate this task as the positive-unlabeled (PU) learning to avoid using noisy negatives. AutoNER (Shang et al., 2018) trains the model by assigning ambiguous tokens with all possible labels and then maximizing the overall likelihood using a fuzzy LSTM-CRF model. Cao et al. (2019b) and Yang et al. (2018) try to select high-quality sentences with less annotation errors for sequential model. Liang et al. (2020) leverage pre-trained language models to improve the prediction performance of NER models under a self-training framework. Our inspiration of confidence estimation comes from the so-called memorization effect observed in the computer vision (Jiang et al., 2018; Han et al., 2018a; Arazo et al., 2019). It observes that neural networks usually take precedence over noisy data to fit clean data, which indicates that noisy data are more likely to have larger loss values in the early training epochs (Arpit et al., 2017). In this work, we leverage it to estimate the confidence scores of labels. In this work, we propose a calibrated confidence estimation approach for noisy-labeled NER. We integrate our method in an LSTM-CRF model under a self-training framework. Extensive experiments demonstrate the effectiveness of our approach. Our method outperforms strong baseline models in a general noise setting (especially for larger noise ratios), and shows promising results in a distant supervision setting. We thank all anonymous reviewers for their helpful comments. This work is supported by Alibaba Group through Alibaba Research Intern Program and AZFT Joint Lab for Knowledge Engine.
abstain abstain method objective abstain objective method objective abstain abstain abstain abstain objective objective abstain method objective method abstain abstain method method result abstain method method method abstain abstain abstain method method abstain method method method method method method method method method method abstain method other abstain method method method abstain method method abstain method abstain method method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result abstain result result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain other objective method other other other other other other other other other other other abstain other abstain objective abstain objective result other other