Recently, distant supervision has gained great success on Fine-grained Entity Typing (FET). Despite its efficiency in reducing manual labeling efforts, it also brings the challenge of dealing with false entity type labels, as distant supervision assigns labels in a context-agnostic manner. Existing works alleviated this issue with partial-label loss, but usually suffer from confirmation bias, which means the classifier fit a pseudo data distribution given by itself. In this work, we propose to regularize distantly supervised models with Compact Latent Space Clustering (CLSC) to bypass this problem and effectively utilize noisy data yet. Our proposed method first dynamically constructs a similarity graph of different entity mentions; infer the labels of noisy instances via label propagation. Based on the inferred labels, mention embeddings are updated accordingly to encourage entity mentions with close semantics to form a compact cluster in the embedding space, thus leading to better classification performance. Extensive experiments on standard benchmarks show that our CLSC model consistently outperforms state-of-the-art distantly supervised entity typing systems by a significant margin. Recent years have seen a surge of interests in fine-grained entity typing (FET) as it serves as an important cornerstone of several nature language processing tasks including relation extraction (Mintz et al., 2009), entity linking (Raiman and Raiman, 2018), and knowledge base completion (Dong et al., 2014). To reduce manual efforts in labelling training data, distant supervision (Mintz et al., 2009) has been widely adopted by recent FET systems. With the help of an external knowledge base (KB), an entity mention is first  Corresponding Author. linked to an existing entity in KB, and then labeled with all possible types of the KB entity as supervision. However, despite its efficiency, distant supervision also brings the challenge of out-of-context noise , as it assigns labels in a context agnostic manner. Early works usually ignore such noise in supervision (Ling and Weld, 2012; Shi-maoka et al., 2016), which dampens the performance of distantly supervised models. Towards overcoming out-of-context noise, two lines of work have been proposed to distantly supervised FET. The first kind of work try to fil-ter out noisy labels using heuristic rules (Gillick et al., 2014). However, such heuristic pruning sig-nificantly reduces the amount of training data, and thus cannot make full use of distantly annotated data. In contrast, the other thread of works try to incorporate such imperfect annotation by partial-label loss ( PLL ). The basic assumption is that, for a noisy mention, the maximum score associated with its candidate types should be greater than the scores associated with any other non-candidate types (Ren et al., 2016a; Abhishek et al., 2017; Xu and Barbosa, 2018). Despite their success, PLL based models still suffer from Confirmation Bias by taking its own prediction as optimization objective in the next step. Specifically, given an entity mention, if the typing system selected a wrong CLSC Classifier KL Regularization Label Clean Data FeatureExtractorFeatureExtractor Noise Data FeatureExtractorFeatureExtractor Distant Supervision Training Data person artist root location athlete ... legal director ... music author actor Knowledge Base S1: by Defense Secretary William Cohen and Joint Chiefs chairman General Hugh Shelton Labeled Corpus Unlabeled Corpus sup L sup L sup L S2: Defense Secretary William Cohen says if a formal investigation shows it was a Maine Republican William Cohen said the plan might violate the assassination ban Candidate Type (cid:966) /person/artist/actor /person/political_figure/person/legal clsc L clsc L Figure 2: The overall framework of CLSC. type with the maximum score among all candidates, it will try to further maximize the score of the wrong type in following optimization epoches (in order to minimize PLL ), thus amplifying the confirmation bias. Such bias starts from the early stage of training, when the typing model is still very suboptimal, and can accumulate in training process. Related discussion can be also found in the setting of semi-supervised learning (Lee et al., 2006; Laine and Aila, 2017; Tarvainen and Valpola, 2017). In this paper, we propose a new method for distantly supervised fine-grained entity typing. Enlightened by (Kamnitsas et al., 2018), we propose to effectively utilize imperfect annotation as model regularization via C ompact L atent S pace C lustering (CLSC) . More specifically, our model encourages the feature extractor to group mentions of the same type as a compact cluster (dense region) in the representation space, which leads to better classification performance. For training data with noisy labels, instead of generating pseudo supervision by the typing model itself, we dynamically construct a similarity-weighted graph between clean and noisy mentions, and apply label propagation on the graph to help the formation of compact clusters. Figure 1 demonstrates the effectiveness of our method in clustering mentions of different types into dense regions. In contrast to PLL -based models, we do not force the model to fit pseudo supervision generated by itself, but only use noisy data as part of regularization for our feature extractor layer, thus avoiding bias accumulation. Extensive experiments on standard benchmarks show that our method consistently outperforms state-of-the-art models. Further study reveals that, the advantage of our model over the competitors gets even more significant as the portion of noisy data rises. Fine-grained entity typing takes a corpus and an external knowledge base (KB) with a type hierarchy Y as input. Given an entity mention (i.e., a sequence of token spans representing an entity) in the corpus, our task is to uncover its corresponding type-path in Y based on the context. By applying distant supervision, each mention is first linked to an existing entity in KB, and then labeled with all its possible types. Formally, a labeled corpus can be represented as triples D = { ( m i , c i , Y i ) } ni =1 , where m i is the i -th mention, c i is the context of m i , Y i is the set of candidate types of m i . Note that types in Y i can form one or more type paths. In addition, we denote all terminal (leaf) types of each type path in Y i as the target type set Y ti ( e.g. , for Y i = { artist, teacher, person } , Y ti = { artist, teacher } ). This setting is also adopted by (Xu and Barbosa, 2018). As each entity in KB can have several type paths, out-of-context noise may exist when Y i contains type paths that are irrelevant to m i in context c i . In this work, we argue triples where Y i contains only one type path (i.e., |Y ti | = 1 ) as clean data . Other triples are treated as noisy data , where Y i contains both the true type path and irrel-formal investigation Defense Secretary [William Cohen] says if Bi-LSTM Word-levelAttention Secretary [William Cohen] says LSTM Encoder Average Encoder Feature Representation [William Cohen] Embedding Context Encoder ( ) Figure 3: The architecture of feature extractor z (( m i , c i );  z ) evant type paths. Noisy data usually takes a considerable portion of the entire dataset. The major challenge for distantly supervised typing systems is to incorporate both clean and noisy data to train high-quality type classifiers. Overview. The basic assumptions of our idea are: (1) all mentions belong to the same type should be close to each other in the representation space because they should have similar context, (2) similar contexts lead to the same type. For clean data, we compact the representation space of the same type to comply (1). For noisy data, given assumption (2), we infer the their type distributions via label propagation and candidate types constrain. Figure 2 shows the overall framework of the proposed method. Clean data is used to train classifier and feature extractor end-to-endly, while noisy data is only used in CLSC regularization. Formally, given a batch of samples { ( m i , c i , Y ti ) } Bi =1 , we first convert each sample ( m i , c i ) into a real-valued vector z i via a feature extractor z (( m i , c i );  z ) parameterized by  z . Then a type classifier g ( z i ;  g ) parameterized by  g gives the posterior P ( y | z i ;  g ) . By incorporating CLSC regularization in the objective function, we encourage the feature extractor z to group mentions of the same type into a compact cluster, which facilitates classification as is shown in Figure 1. Noisy data enhances the formation of compact clusters with the help of label propagation. Figure 3 illustrates our feature extractor. For fair comparison, we adopt the same feature extraction pipeline as used in (Xu and Barbosa, 2018). The feature extractor is composed of an embedding layer and two encoders which encode mentions and contexts respectively. Embedding Layer: The output of this layer is a concatenation of word embedding and word position embedding. We use the popular 300-dimensional word embedding supplied by (Pen-nington et al., 2014) to capture the semantic information and random initialized position embedding (Zeng et al., 2014) to acquire information about the relation between words and the mentions. Formally, Given a word embedding matrix W word of shape d w  | V | , where V is the vocabulary and d w is the size of word embedding, each column of W word represents a specific word w in V . We map each word w j in ( m i , c i ) to a word embedding w dj  R d w . Analogously, we get the word position embedding w pj  R d p of each word according to the relative distance between the word and the mention, we only use a fixed length context here. The final embedding of the j-th word is w Ej = [ w dj , w pj ] . Mention Encoder: To capture lexical level information of mentions, an averaging mention encoder and a LSTM mention encoder (Hochreiter and Schmidhuber, 1997) is applied to encode mentions. Given m i = ( w s , w s +1 ,    , w e ) , the averaging mention representation r a i  R d w is : r a i = 1 e  s + 1 e (cid:88) j = s w dj (1) By applying a LSTM over an extended mention ( w s  1 , w s , w s +1 ,    , w e , w e +1 ) , we get a sequence ( h s  1 , h s , h s +1 ,    , h e , h e +1 ) . We use h e +1 as LSTM mention representation r l i  R d l . The final mention representation is r m i = [ r a i , r l i ]  R d w + d l . Es  W Es  W + 1 Ee + W  h j = LST M (  h j  1 , w Ej  1 )  h j = LST M (  h j  1 , w Ej  1 ) h j =[  h j   h j ] (2) where  denotes element-wise plus. Then, the word-level attention mechanism computes a score  i,j over different word j in the context c i to get the final context representation r c i :  j = w T tanh ( h j )  i,j = exp (  j ) (cid:80) k exp (  k ) r c i = (cid:88) j  i,j h i,j (3) We use r i = [ r m i , r c i ]  R d z = R d w + d l + d l as the feature representation of ( m i , c i ) and use a Neural Networks q over r i to get the feature vector z i . q has n layers with h n hidden units and use ReLu activation. The overview of CLSC regularization is exhibited in Figure 4, which includes three steps: dynamic graph construction (Figure 4c), label propagation (Figure 4d, e) and Markov chains (Fig-ure 4g). The idea of compact clustering for semi-supervised learning is first proposed by (Kamnit-sas et al., 2018). The basic idea is to encourage mentions of the same type to be clustered into a dense region in the embedding space. We introduce more details of CLSC for distantly supervised FET in following sections. Dynamic Graph Construction: We start by creating a fully connected graph G over the batch of samples Z = { z i } Bi =1 , as shown in Figure 4c 1 . Each node of G is a feature representation z i , while the distance between nodes is defined by a scaled dot-product distance function (Vaswani et al., 2017): A ij = exp ( z Ti z j  d z ) ,  z i , z j  Z A = exp ( ZTZ  d z ) (4) Each entry A ij measures the similarity between z i and z j , A  RB  B can be viewed as the weighted adjacency matrix of G . Label Propagation: The end goal of CLSC is to cluster mentions of the same type to a dense region. For mentions which have more than one labeled types, we apply label propagation ( LP ) on G to estimate their type distribution. Formally, we denote   RB  K as the label propagation posterior of a training batch. The original label propagation proposed by (Zhu and Ghahramani, 2002) uses a transition matrix H to model the probability of a node i propagating its type posterior  i = P ( y i | x i )  RK to the other nodes. Each entry of the transition matrix H  RB  B is defined as: H ij = A ij / (cid:88) b A ib (5) The original label propagation algorithm is defined as: 1. Propagate the label by transition matrix H ,  ( t +1) = H  ( t ) 2. Clamp the labeled data to their true labels. Repeat from step 1 until  converges In this work  (0) is randomly initialized 2 . Unlike unlabeled data in semi-supervised learning, distantly labeled mentions in FET have a limited set of candidate types. Based on this observation, We assume that ( m i , c i ) can only transmit and receive probability of types in Y ti no matter it is noisy data or clean data. Formally, define a B  K indicator matrix M  RB  K , where M ij = 1 if type j in Y ti otherwise 0 , where B is the batch size and K 1 Z = { z i } B i =1 is a small subsample of the entire data, we didn't observe significant performance gain when the batch size increases. 2 We also explored other initialization (e.g. uniform ini-tialization), but found no essential performance difference between different initialization setups. ( t +1) ij   ( t +1) ij M ij / (cid:88) k  ( t +1) ik M ik (6) For convenience, we iterate through these two steps S lp times, S lp is a hyperparameter. Based on this assumption, the desirable transition matrix T  RB  B is defined as: T ij = K (cid:88) k =1  ik  jk m k , m k = B (cid:88) b =1  bk (7) m k is a normalization term for class k . Thus we minimize the cross entropy between T and H : L 1  step =  1 B 2 B (cid:88) i =1 B (cid:88) j =1 T ij log ( H ij ) (8) For instance, if T ij is close to 1, H ij needs to be bigger, which results in the growth of A ij and finally optimize  z (Eq.4). Compact Clustering: The LP posterior  =  ( S lp +1) is used to judge the label agreement between samples. In the desired optimal state, transition probabilities between samples should be uniform inside the same class, while be zero between different classes. Transition matrix H derived from z (( m i , c i );  z ) should be in keeping with T . The loss L 1  step has largely described the regularization we use in z (( m i , c i );  z ) for compression clustering. In order to keep the structure of existing clusters, (Kamnitsas et al., 2018) proposed an extension of L 1  step to the case of Markov chains with multiple transitions between samples, which should remain within a single class. The extension maximizes probability of paths that only traverse among samples belong to one class. Define E  RB  B as: E =  T  (9) E ij measures the label similarities between z i and z j , which is used to mask the transition between different clusters. The extension is given by: H (1) = HH ( s ) =( H (cid:12) E ) ( s  1) H =( H (cid:12) E ) H ( s  1) , (10) where (cid:12) is Hadamard Product, and H ( s ) ij is the probability of a Markov process to transit from node i to node j after s  1 steps within the same class. The extended loss function models paths of different length s between samples on the graph: L clsc =  1 S m 1 B 2 S m (cid:88) s =1 B (cid:88) i =1 B (cid:88) j =1 T ij log ( H ( s ) ij ) . For S m = 1 , L clsc = L 1  step . By minimizing the cross entropy between T and H ( s ) (Eq.11), L clsc compact paths of different length between samples within the same class. Here, S m is a hyper-parameter to control the maximum length of Markov chain. Given the representation of a mention, the type posterior is given by a standard softmax classifier parameterized by  g : where W c  RK  d z is a parameter matrix, b  RK is the bias vector, where K is the number of types. The predicted type is then given by  t i = argmax y i P (  y i | z i ;  g ) . Here B c is the number of clean data in a training batch, K is the number of target types. The regularization term is given by L clsc . Hence, the overall loss function is: L final = L sup +  clsc  L clsc (14)  clsc is a hyper parameter to control the influence of CLSC. We evaluate our method on two standard benchmarks: OntoNotes and BBN: OntoNotes: The OntoNotes dataset is composed of sentences from the Newswire part of OntoNotes corpus (Weischedel et al., 2013). (Gillick et al., 2014) annotated the training part with the aid of DBpedia spotlight (Daiber et al., 2013), while the test data is manually annotated. BBN: The BBN dataset is composed of sentences from Wall Street Journal articles and is manually annotated by (Weischedel and Brunstein, 2005). (Ren et al., 2016a) regenerated the training corpus via distant supervision. In this work we use the preprocessed datasets provided by (Abhishek et al., 2017; Xu and Barbosa, 2018). Table 2 shows detailed statistics of the datasets. We compare the proposed method with several state-of-the-art FET systems 3 : Attentive (Shimaoka et al., 2016) uses an attention based feature extractor and doesn't distinguish clean from noisy data;  AFET (Ren et al., 2016a) trains label embedding with partial label loss;  AAA (Abhishek et al., 2017) learns joint representation of mentions and type labels;  PLE+HYENA/FIGER (Ren et al., 2016b) proposes heterogeneous partial-label embedding for label noise reduction to boost typing systems. We compare two PLE models with HYENA (Yogatama et al., 2015) and FIGER (Ling and Weld, 2012) as the base typing system respectively;  NFETC (Xu and Barbosa, 2018) trains neural fine-grained typing system with hierarchy-aware loss. We compare the performance of the NFETC model with two different loss functions: partial-label loss and PLL +hierarchical loss. We denote the two variants as NFETC and NFETC hier respectively;  NFETC-CLSC is the proposed model in this work. We use the NFETC model as our base model, based on which we apply Compact Latent Space Clustering Regularization as described in Section 3.2; Similarly, we report results produced by using both KL-divergense-based loss ( NFETCCLSC ) and KL +hierarchical loss ( NFETCCLSC hier ). For evaluation metrics, we adopt strict accuracy, loose macro, and loose micro F-scores widely used in the FET task (Ling and Weld, 2012). To fine tuning the hyper-parameters, we randomly sampled 10% of the test set as a development set for both datasets. With the fine-tuned hyper-parameter as mentioned in 4.4, we run the model five times and report the average strict accuracy, macro F1 and micro F1 on the test set. 3 The baselines result are reported on (Abhishek et al., 2017; Xu and Barbosa, 2018) in addition to performance of NFETC on BBN, we search the hyper parameters for it. (Xu and Barbosa, 2018) didn't report the results on BBN Method OntoNotes BBN Strict Acc. We search the hyper parameter of Ontonotes and BBN respectively via Hyperopt proposed by (Bergstra et al., 2013). Hyper parameters are shown in Appendix A . We optimize the model via Adam Optimizer. The full hyper parameters includes the learning rate lr , the dimension d p of word position embedding, the dimension d l of the mention encoder's output (equal to the dimension of the context encoder's ourput), the input dropout keep probability p i and output dropout keep probability p o for LSTM layers (in context encoder and LSTM mention encoder), the L2 regularization parameter  , the factor of hierarchical loss normalization  (  > 0 means use the normalization), BN (whether using Batch normalization), the max step S lp of the label propagation, the max length S m of Markov chain, the influence parameter  clsc of CLSC, the batch size B , the number n of hidden layers in q and the number h n of hidden units of the hidden layers. We implement all models using Tensorflow 4 . Table 1 shows performance comparison between the proposed CLSC model and state-of-the-art FET systems. On both benchmarks, the CLSC model achieves the best performance in all three metrics. When focusing on the comparison between NFETC and CLSC, we have following observation:  Compact Latent Space Clustering shows its effectiveness on both clean data and noisy data. By applying CLSC regularization on the basic NFETC model, we observe consistent and significant performance boost;  Hierarchical-aware loss shows significant advantage on the OntoNotes dataset, while showing insignificant performance boost on the BBN dataset. This is due to different distribution of labels on the test set. The proportion of terminal types of the test set is 69% for the BBN dataset, while is only 33% on the OntoNotes dataset. Thus, applying hierarchical-aware loss on the BBN dataset brings little improvement;  Both algorithms are able to utilize noisy data to improve performance, so we would like to further study their performance in different noisy scenarios in following discussions. 4 The code for experiments is available at https://github. com/herbertchen1/NFETC-CLSC 4.6 How robust are the methods to the proportion of noisy data? By principle, with sufficient amount of clean training data, most typing systems can achieve satisfying performance. To further study the robustness of the methods to label noise, we compare their performance with the presence of 25% , 20% , 15% , 10% and 5% clean training data and all noisy training data. Figure 5 shows the performance curves as the proportion of clean data drops. As it reveals, the CLSC model consistently wins in the comparison. The advantage is especially clear on the BBN dataset, which offers less amount of training data. Note that, with only 27 . 9% of training data (when only leaving 5% clean data) on the BBN dataset, the CLSC model yield a comparable result with the NFETC model trained on full data. This comparison clearly shows the superiority of our approach in the effectiveness of utilizing noisy data. Table 3 shows the performance of CLSC with one-step transition ( L 1  step ) and with Markov Chains ( L clsc ) as described in Section 3.2. Results show that the use of Markov Chains does bring improvement to the overall performance, which is consistent with the model intuition. Named entity Recognition (NER) has been excavated for a long time (Collins and Singer, 1999; Manning et al., 2014), which classifies coarse-grained types (e.g. person, location). Recently, (Nagesh and Surdeanu, 2018a,b) applied ladder network (Rasmus et al., 2015) to coarse-grained entity classification in a semi-supervised learning fashion. (Ling and Weld, 2012) proposed Fine-Grained Entity Recognition (FET). They used distant supervision to get training corpus for FET. Embedding techniques was applied to learn feature representations since (Yogatama et al., 2015; Dong et al., 2015). (Shimaoka et al., 2016) introduced attention mechanism for FET to capture informative words. (Xin et al., 2018a) used the TransE entity embeddings (Bordes et al., 2013) as the query vector of attention. Early works ignore the out-of-context noise, (Gillick et al., 2014) proposed context dependent FET and use three heuristics to clean the noisy labels with the side effect of losing training data. To utilize noisy data, (Ren et al., 2016a) distinguished the loss function of noisy data from clean data via partial label loss ( PLL ). (Abhishek et al., 2017; Xu and Barbosa, 2018) proposed variants of PLL , which still suffer from confirmation bias. (Xu and Barbosa, 2018) proposed hierarchical loss to handle over-specific noise. On top of AFET , (Ren et al., 2016b) proposed a method PLE to reduce the label noise, which lead to a great success in FET. Because label noise reduction is separated from the learning of FET, there might be error propagation problem. Recently, (Xin et al., 2018b) proposed utilizing a pretrained language model measures the compatibility between context and type names, and use it to repel the interference of noisy labels. However, the compatibility got by language model may not be right and type information is defined by corpus and annotation guidelines rather than type names as is mentioned in (Azad et al., 2018). In addition, there are some work about entity-level typing which aim to figure out the types of entities in KB (Yaghoobzadeh and Schutze, 2015; Jin et al., 2018). In this paper, we propose a new method for distantly supervised fine-grained entity typing, which leverages imperfect annotations as model regularization via Compact Latent Space Clustering (CLSC). Experiments on two standard benchmarks demonstrate that our method consistently outperforms state-of-the-art models. Further study reveals our method is more robust than the former state-of-the-art approach as the portion of noisy data rises. The proposed method is general for other tasks with imperfect annotation. As a part of future investigation, we plan to apply the approach to other distantly supervised tasks, such as relation extraction. This work has been supported in part by NSFC (No.61751209, U1611461), Zhejiang University-iFLYTEK Joint Research Center, Chinese Knowledge Center of Engineering Science and Technology (CKCEST), Engineering Research Center of Digital Library, Ministry of Education. Xiang Ren's research has been supported in part by National Science Foundation SMA 18-29268.
abstain abstain abstain objective objective abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain objective abstain method method objective objective result result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method method method abstain abstain method abstain method method method abstain abstain method abstain method method abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain method method other abstain method method abstain abstain method abstain method method method method abstain method abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other method method method result abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain result result abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other other other other other other other other other other other other other other other other objective objective result abstain method other other