Fine-grained Entity Typing (FET) has made great progress based on distant supervision but still suffers from label noise. Existing FET noise learning methods rely on prediction distributions in an instance-independent manner, which causes the problem of confirmation bias. In this work, we propose a clustering-based loss correction framework named Feature Cluster Loss Correction (FCLC), to address these two problems. FCLC first train a coarse backbone model as a feature extractor and noise estimator. Loss correction is then applied to each feature cluster, learning directly from the noisy labels. Experimental results on three public datasets show that FCLC achieves the best performance over existing competitive systems. Auxiliary experiments further demonstrate that FCLC is stable to hyperparameters and it does help mitigate confirmation bias. We also find that in the extreme case of no clean data, the FCLC framework still achieves competitive performance. Fine-grained entity typing (FET) is the task of classifying named entity mentions in a sentence over the given class set (typically a hierarchical class structure as shown in Fig. 1. FET serves as an important component in many down-stream NLP applications, e.g., relation extraction (Liu et al., 2014), entity linking (Raiman and Raiman, 2018) and question answering (Dong et al., 2015). FET task has a more wide range of entity types (usu-ally over 100 classes) compared to entity typing, and hence neural-based FET systems require large-scale annotated training corpus. as the ground-truth labels. Although large-scale annotated data is provided, it brings about label noises in training. To overcome the problem of noisy label, some works directly pruned noisy instances (Gillick et al., 2014; Onoe and Durrett, 2019a). The others retain noisy training data but further improve by choosing (Ren et al., 2016a; Xu and Barbosa, 2018), weighting (Wu et al., 2019), and relabeling (Zhang et al., 2020) noisy labels using the prediction distribution. However, these noise combating methods have two major limitations. 1) They rely on the prediction distribution. As a result, they ought to cope with instance-agnostic noise better. The previous works expirically show (Zheng and Yang, 2021) that the prediction distribution is more likely to be affected by noisy instances and suffer from confirmation bias . This bias problem is also verified in our Sec. 3.5. The limitation leads to the intriguing question: Besides prediction distribution and entropy, what other information can we use to model label noise? 2) They mostly aim to modify each instance isolatedly and only use instance-level information. Meanwhile, typical anti-noise machine learning (Patrini et al., 2017; Hendrycks et al., 2018) uses instance-agnostic global statistics. The latter is 1997 more robust to noise but might be too general. Local information is potentially more informative. For example, when the distant supervision introduces similar noise in some instances, these noises form a locality in feature space. The noisy instances are near to each other and are separate from instances with the same but true labels. Our experiment result is similar to Fig. 1, even when the feature extractor is trained to fit noisy labels, they are still easily separable due to underlying semantic differences. These two limitations are inter-related, causing noise-learning-based FET methods to still suffer from distantly supervised noise. To alleviate the label noise and avert these limitations, we propose a novel framework FCLC for noisy label learning inspired by weighted training and loss correction (Hendrycks et al., 2018) in machine learning. Our method utilizes feature representations from the model and learns global (local) information, i.e. a cluster-level label confusion matrix. Firstly, we use a backbone learner on noisy data. It serves as a feature extractor and a noise estimator. Secondly, all training data, including noisy data and a small portion of clean data are clustered. The clean data serve as anchors in the feature space to estimate label corruption and sample quality of each cluster. Finally, label corruption and sample quality are used for label correction. Our main contributions are three-fold: ( i ) This study provides fresh insight into instance dependent label noise in FET. We pointed out a novel training method to further exploit feature space and global information. ( ii ) We designed a framework with feature clustering, estimating cluster-level confusion matrix, and loss correction. ( iii ) We experimented the proposed method on three datasets. Results show that we made significant improvements over previous state-of-the-art, thus proving the effectiveness of our model. Ablation studies further prove the robustness and wide applicability of our framework. Given a finite set of types, T = { t 1 , t 2 , ..., t | T | } , where | T | denotes the number of candidate types. The task is to assign appropriate types to each mention under context. Formally, an instance is a triplet, ( m, c, y ) . c = { w 1 , w 2 , ..., w n } is the context of m , usually the original sentence. m = { w p 1 , ..., w p l } is the mention. obviously, m is a continuous subsequence of c . Y  T denotes appropriate types for ( m, c ) . For convenience, denote Y 's vector form y  { 0 , 1 } | T | , y j = 1 means t j  Y . When the instance is produced with crowd-sourcing or distant supervision, annotated labels might contain so-called noise. We denote labels with noise  y . The instance is thus ( m, c,  y ) . Denote the corpus with noisy instances  D , the corpus with trusted instances D t . * The two corpus form the whole training corpus D . As shown in Fig. 2, the FCLC framework consists of the following steps : Step 1. (Phase 1) Train the backbone model with noisy data  D for e 1 epochs and get M 1 . It serves as a feature extractor and a noise estimator. (Sec. 2.3) Step 2. Cluster all training samples D with the feature extracted by E 1 , and estimate confusion matrix for each cluster with predictions of M 1 . (Sec. 2.4) Step 3. (Phase 2) The calculated clustering-aware confusion matrix and FCLC loss are used to continue training the backbone model. (Sec. 2.5) 2.3 Backbone For fair comparison, the backbone of our model has the same structure as NFETC (Xu and Barbosa, 2018). For an instance ( m, c, y ) , for each word w i in c , word embedding is e wi  R d w looked up in word embedding matrix W  R d w | V | . A position embedding e pi  R d p is used to model the context word position i and mention position ( p 1 , p l ) by looking up relative position in position embedding matrix P  R d p  2 N . The final embedding is the concatenation e i = [ e wi , e pi ] . Context Representation A Bi-LSTM (Hochre-iter and Schmidhuber, 1997) is used to model the context representation. Feeding the embedding of c i.e. { e 1 , e 2 , ..., e n } into BiLSTM gets the two directional hidden states  h i and  h i for each word w i . Word level attention weighted sum following (Zhou et al., 2016) is applied on h i = [  h i   h i ] , resulting in the final context representation r c  R d c , * Normally |D t |  |  D| , as in all the datasets we reported in this paper. where  means element-wise sum and d c is the hidden size of the BiLSTM and the dimension of the context embedding. Mention Representation The average encoder of a mention takes word embeddings of the mention { e p 1 , e p 2 , ..., e p l } and takes the average: r w = 1 l (cid:80) lk =1 e p k . The LSTM encoder of a mention takes an extended mention with one more token before and after the original mention and produces hidden state features { h p 1  1 , ..., h p l +1 } . Take the last output h p l +1 as r l . The final representation of the mention is r m = [ r w , r l ] Classification Softmax classifier and cross-entropy are used based on the feature r m,c = [ r c , r m ] of x : s ( x ) = W r m,c + b (1)  p ( y | x ) = softmax( s ( x )) (2)  ( x, y ;  ) =  log ( p ( y | x )) (3) With a given dataset D , the model is trained with all samples ( x, y ) in D . We make the assumption that the noise ( y ,  y ) forms locality in the feature space, especially when the feature is calculated from the original mention and context ( m, c ) , ( m, c ) determines y , and the feature is trained with  y . We adopt clustering to utilize local statistics as smaller-grained feature information. To be specific, we perform k-means with r m,c on the whole training set D , and separate D into K clusters. Denote the k -th cluster  C k , C t  k =  C k  D t ,  C k =  C k   D . We mainly utilize the two following statistics:  k = |C t  k | |D k | (5)  k estimates the quality of the cluster k . It acts as a soft cluster sieving. where A ik = { ( x, y ) | ( x, y )  C t  k and y i = 1 } , (cid:98) C ijk estimates the probability in cluster k to annotate noise j for true label i . The idea of forward loss correction is proposed by Patrini et al. (2017). The basic idea is to modify the loss with the noise transition matrix T . Such that the minimizer under the new loss with noisy labels is the same as the minimizer of the original loss under clean labels. The modification relies on the assumption that the label noise is independent from instances, i.e.  y  x | y . Hendrycks et al. (2018) proposed to estimate T with a small set of clean labels, under the assumption that  y  y | x . While these assumptions do not hold globally for distantly 1999 supervised FET, they hold better in clusters. We introduce the cluster-wise loss correction in the following sections. Transition Matrix Estimation Assuming the backbone model is well trained, i.e.  p ( y j = 1 | x ) is close enough to p ( y j = 1 | x ) . We use the predicted probability on trusted instances in clusterk to estimate the transition probability. C ijk = p ( y j = 1 | y i = 1 , x   C k )  p ( y j = 1 | y i = 1 , x  C t  k )  1 | A ik | (cid:88) ( x,y )  A ik  p ( y j = 1 | x ) = (cid:98) C ijk (7) Forward Loss Correction Cross-entropy is composite (Reid and Williamson, 2010),denote it as   , its inverse link function   1 is softmax . Notice C ijk can bridge the loss with noisy label  y , ( x   C k ,  y i = 1) , to predictions for the true label:  log ( p ( y | x ))   log c (cid:88) j =1 C jik  p ( y j = 1 | x ) (8) Let T k = C  k , define the forward loss as:    ( s ( x )) =   ( T  k s ( x )) (9) The property holds on each cluster similar as in (Patrini et al., 2017), with all x   C k , training with noisy label  y on    is the same as with true label y on the original loss   : argmin s E x ,  y    ( s ( x )) = argmin s E x , y   ( s ( x )) (10) Different from global forward loss correction, the parameters that minimize the loss in each cluster are not the same. We balance the clusters with  k . The trusted samples ( x, y )  D t are also used. The loss of the full model is: LFCLC = (cid:80) ( x,y ) D t   ( s ( x )) +  (cid:80) Kk =1  k (cid:80) ( x,  y )   C k    ( s ( x ))) +(1   ) (cid:80) Kk =1  k (cid:80) ( x,  y )   C k   ( s ( x ))) (11) Where  is the hyperparameter to balance FCLC loss and the original loss. Our introduced framework has several advantages: 1) Lightweight . This method does not include extra trainable parameters to the backbone model. 2) Stable . The framework involves two hyperparameters,  and phase-1 train epochs e 1 and we empirically find them stable. 3) Flexibility . Our improvement is orthogonal to the backbone model. It only requires that the backbone model is sufficiently expressive and uses an appropriate composite loss (Reid and Williamson, 2010). Thus, it is pluggable to a large number of FET models. We evaluate the proposed model on three different FET datasets and compare it to several state-of-the-art models. In addition, to support our claims we also conduct several subsidiary experiments to analyze the impacts of our proposed module in detail. The datasets are described below, we use exactly the same train/dev/test split with previous works (Ren et al., 2016a; Chen et al., 2019). Detailed statistics of the three datasets are also shown in Table 1. BBN It contains sentences extracted from the Wall Street Journal and distantly labeled by DBpedia Spotlight (Weischedel and Brunstein, 2005). OntoNotes It was constructed using sentences in the OntoNotes corpus and distantly supervised by DBpedia Spotlight (Weischedel et al., 2013). Wiki/FIGER It was derived from Wikipedia articles and news reports, entities of the training samples are distantly annotated using Freebase (Ling and Weld, 2012). We follow prior work and use the strict accuracy (Acc), Macro F1 (Ma-F1), and Micro F1 (Mi-F1) scores. During the experiment, all these metrics are calculated by running the model five times and computing the mean and standard deviation values. We consider the following competitive FET systems as our baselines: (1) AFET (Ren et al., 2016a); (2) Attentive (Shimaoka et al., 2016); (3) NFETC/NFETC hier (Xu and Barbosa, 2018); (4) CLSC/CLSC hier (Chen et al., 2019); (5) NFETC-AR/NFETC-AR hier (Zhang et al., 2020); (6) NFETC-VAT/CLSC-VAT (Shi et al., 2020); (7) Multi Level Learning to Rank (ML-L2R) (Chen et al., 2020); (8) Box (Onoe et al., 2021). These baselines are compared with several variants of our proposed model: (1) FCLC : proposed model without the hierarchical loss; (2) FCLC hier proposed model with the hierarchical loss; (3) FCLC (without  k ) our proposed model trained without cluster quality estimation, i.e.  = 1 for all clusters; (4) FCLC (without loss correction) our proposed model without loss correction, only cluster quality estimation working; (5) FCLC (without cluster) our proposed model without clustering, i.e. calculated a globally-uniform confusion matrix; (6) FCLC (with reinit): our proposed model with fresh parameters before the start of step 3 as suggested by Patrini et al. (2017). (3)-(6) are implemented based on and should be compared with the best configuration between FCLC and FCLC hier on each dataset, that is, compared with FCLC on BBN and compared with FCLC hier on Wiki and OntoNotes. To make an equal comparison, following (Xu and Barbosa, 2018; Chen et al., 2019; Zhang et al., 2020), we use exactly the same pre-trained 300-dimensional GloVe word embeddings (Pennington et al., 2014) and fix the embedding vectors during training. The model parameters are optimized using the Adam (Kingma and Ba, 2014) optimizer. All of our models are implemented in Tensorflow. The implementation of our model can be cound at https://github.com/Los-Phoenix/NFETC-FCLC. As NFETC and NFETC hier are our backbone models, we follow the hyper-parameters of the backbone except for our introduced hyper-parameters  and e 1 . The detailed hyper-parameter settings on the three datasets are shown in Table 2, we also report hyper-parameter impact curves in Fig. 3. 3.5 Results and Analysis Main Result Table 3 shows the results of our proposed approach ( FCLC ) and several competitive FET systems. We highlight the statistically significant best scores of each metric in bold. According to the experimental results, we make two main observations: (1) The performances of our proposed model surpass the backbone NFETC model by a remarkable large margin (improving Micro F1 by 2.1%, 3.8%, and 7.8% separately), demonstrating the benefits of the proposed two-phase FCLC module. The relative performance improvements are consistent with or without the hierarchy loss (compared FCLC and FCLC hier to the corresponding baselines). (2) Compared to other noisy learning methods such as CLSC, NFETC-AR, and VAT, our model still achieves considerable improvements under most metrics when using the same backbone and very similar hyper-parameter settings. For example, compared to NFETC-AR, our model improves Micro-F1 by 1.25% to 6.38% on three datasets. It indicates that, by utilizing both the feature space representations and the global and local statistical information, the model can reduce the impact of noisy labels more effectively. Ablation Study To study the detail of our models, we explore the performances of three main model variants, shown in the last several rows of Table 3. We find that the cluster quality  k , the loss correction module and the feature cluster process are all critical to model performances in some situations. Specifically, as shown in FCLC (without cluster), feature clustering has minor impacts on Wiki and Ontonotes. This is probably because the noisy distribution on these two datasets is relatively simple and the global confusion matrix is sufficient. Moreover, we observe that the re-initialization before Step 3 has a great impact on all metrics. Staring Step 3 with a fresh re-initialized FET model degrades the accuracy by 3.2% on Ontonotes. It denotes that the learner trained in the first phase is beneficial for the noisy robust learning process, by providing optimal parameters initialization. Sensitivity of the introduced hyper-parameters Using the same setting for model training, Fig. 3 analyses the sensitivity of FCLC to the introduced hyper-parameters: the FCLC objective weight  , the Step-1 training epochs e 1 . Fig. 3(a, b) shows the performance trend on the Ontonotes and BBN datasets when changing  . While selecting a proper ratio between loss-correction loss and the original loss is important, the performance near optimum  is stable and steadily outperforms the baseline. Fig. 3(c, d) analyses the sensitivity with respect to e 1 . the Micro-F1 improves as e 1 increases but stops improving and become unstable when e 1 is large enough, since the model starts to overfit noise. It is also reasonable that the optimal range of  and e 1 in BBN and Ontonotes are different as they have different training set sizes and different distance supervision noise distribution. Will cluster number affect performance? We investigate how much the FCLC model benefits from different values of feature cluster number k . Fig. 5 demonstrates that under a reasonable feature cluster range (near | T | ), the model can achieve competitive and similar performances. How many trusted instances does the model need? We examine the robustness of the model to the amount of clean data by comparing the performances with 5% to 100% trusted instances. Refer to Fig. 4, we observe that due to the differences of the training set, our model achieves comparable accuracy with 30%, 40%, and 70% D t samples on 2002 -10 -5 0 5 10 15 20 25 30 Wiki, Ontonotes, and BBN separately. With only a very small size of trusted instances, e.g. 20% BBN trusted set, or 128 samples, the model begins to improve significantly. What if we did not have any trusted instances? Although a small number of clean samples is always practical to obtain or relabel with an expert, we push the limit to no trusted instances at all. What performance can our model achieve in such a situation? We performed the "no clean training set" experiment to test the robustness of our model. In Table 4, FCLC (w/o D t ) indicates for the variant that the trusted instances are not used for phase 2 training but only in feature clustering and confusion matrix calculation. In that situation, our approach still has similar performances with previous SOTA models on most metrics  . FCLC (w/ pl) variant means that, during the clustering process, instead of using the trusted instance set D t split from the training set, we introduce a simple and classic pseudo labeling method (Lee et al., 2013) to generate the labels needed by clustering and training. We find that compared to the baseline method, FCLC with pseudo labeling still achieves much better performances. It is worth pointing out that it means our model is trained with fewer instances than previous SOTA, since D t is a part from the training set they use. Visualization of the representations We analyze the role of FCLC module by visualizing the feature vectors. Fig. 6 illustrates samples in a cluster (circled in all 4 sub-figures). From Fig. 6(a), we observe that the backbone model fails to distinguish some samples of class A (/ORGANIZATION/GOVERN-MENT, red) and class B (/GPE/COUNTRY, blue), due to noisy labels. Fig. 6(b) shows that our model learns to correct these instances. With FCLC the classifier is corrected to predict the right label. Meanwhile, in feature space, the boundary between these samples and the confusing class is also clearer, which means FCLC also helps to refine feature extraction with loss correction. Fig. 6(e) shows the row of '/GPE/COUNTRY'. Managing to notice the confusion from '/GPE/COUNTRY' to '/ORGANIZATION/GOVERNMENT' enables our model to perform the appropriate correction. Due to this, FCLC are resistant to the noisy labels. further verify our claim that our model can alleviate the confirmation bias in the noisy FET task, we analyze the prediction confidence on test set samples, as shown in Fig. 7. The average confidence of correct and wrong test samples is calculated after each training epoch. The results show that, on the Wiki dataset, after phase one the wrong sample average confidence is 0.700 but the backbone model reached 0.833 at the end of the training (with early stopping). Also, after phase two FCLC improves the correct sample confidence from back-bone's 0.939 to 0.950 on Wiki. The usage of datasets collected with distant supervision often results in so-called noisy labels. Several studies have investigated deep learning approaches with noise. Existing noisy learning methods include designing robust loss functions (Wang et al., 2019), designing robust architectures by adding noise adaptation layers (Chen and Gupta, 2015; Goldberger and Ben-Reuven, 2017), selecting samples (Onoe and Durrett, 2019b), and adding noise-robust regularization (Shi et al., 2020). Among them, Patrini et al. (2017) and Hendrycks et al. (2018) proposed forward loss correction. It avoided explicit relabeling and matrix inversion. These noisy learning methods are mostly restricted to the 2003 noise that is conditionally independent of the data features (Frnay and Verleysen, 2014). However, in real-world applications such as FET, noise distributions are more complex and instance-dependent, requiring more powerful noisy learning methods. FET is studied based on the distant supervision training data (Mintz et al., 2009; Ling and Weld, 2012). Various features (Yogatama et al., 2015; Xu and Barbosa, 2018), network structures (Dong et al., 2015; Shimaoka et al., 2016), and feature space (Ali et al., 2021; Onoe et al., 2021)are explored to refine the mention and type representation. Label inter-dependency (Lin and Ji, 2019) and type hierarchy (Chen et al., 2020) are often used, added by relations among instances and labels (Ali et al., 2020; Li et al., 2021; Liu et al., 2021). Label noise is the main problem brought by distance supervision. Besides common noisy learning methods discussed in Sec. 4.1 (Onoe and Durrett, 2019b; Shi et al., 2020; Wu et al., 2019), FET-specific noise combat methods are proposed. Ren et al. (2016a,b) utilized partial-label embedding. Xu and Barbosa (2018) modified hierarchical loss to cope with overly-specific noise. Zhang et al. (2020) automatically generated pseudo-truth label distribution for each sample. Additional resource also help to improve the performance. The resource include external knowledge base (Xin et al., 2018; Dai et al., 2019), and with BERT-like pipeline (Pa-tel and Ferraro, 2020; Ding et al., 2021). Choi et al. (2018) proposed a way to utilize more distance supervision and crowd source, followed by Onoe and Durrett (2019b). Apart from the above, (Chen et al., 2019) and (Ali et al., 2020) are the closest to our proposed method. They both select some instances by feature distance to modify labels or refine mention representation for noisy instances. However, their refinement is still explicit and isolated to each instance. Thus the quality relies on the instances they retrieve for label propagation/men-tion reference. Different from these studies, we do not rely on any of these external resources and aim to impose label noise with only the original data without explicit sieving or label changing. In this work, in order to tackle the instance-dependent label noise in fine-grained entity typing tasks, we present a neural FET noisy learning framework that utilizes the feature space information and global information jointly. Experimental results on three publicly available datasets demonstrate that our proposed model achieves the best performance compared with competitive existing FET systems. Furthermore, based on extensive auxiliary experiments, we study the impact of our proposed noisy learning framework in-depth with qualitative and quantitative analysis. In the future, the proposed approach can motivate the need for further understanding of the relationships between dataset noise distribution estimation and the instance features. More work can be done towards this direction. In addition, performances of the proposed framework under different backbone models can be dug to validate the flexibility of the framework. This work was supported by the National Key Research and Development Project of China (No. 2021ZD0110700).
abstain abstain objective abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result method abstain abstain abstain abstain abstain abstain result abstain abstain method method abstain abstain abstain abstain objective objective abstain method abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method other other abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain other abstain result abstain result abstain abstain result result abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain other other other other other other other other other other other other other other other other other other other abstain other other other objective method abstain objective objective abstain abstain abstain other