Distant supervision has obtained great progress on relation classification task. However, it still suffers from noisy labeling problem. Different from previous works that underutilize noisy data which inherently characterize the property of classification, in this paper, we propose RCEND, a novel framework to enhance Relation Classification by Exploiting Noisy Data. First, an instance discriminator with reinforcement learning is designed to split the noisy data into correctly labeled data and incorrectly labeled data. Second, we learn a robust relation classifier in semi-supervised learning way, whereby the correctly and incorrectly labeled data are treated as labeled and unlabeled data respectively. The experimental results show that our method outperforms the state-of-the-art models. Relation classification plays a crucial role in natural language processing (NLP) tasks, such as question answering and knowledge base completion (Xu et al., 2016; Han et al., 2018a). The goal of relation classification is to predict relations of the target entity pair given a plain text. Traditional supervised learning methods (Zelenko et al., 2002; Bunescu and Mooney, 2005; Zhou et al., 2005) heavily rely on large scale annotated data which is time and labor consuming. Mintz et al. (2009) proposed distant supervision (DS) to automatically generate training data for relation classification based on the assumption that if two target entities have a relation in knowledge base (KB), sentences containing this entity pair might express the relation. For example, if a relational fact < Apple, founder , Steve Jobs > exists in KB, distant supervision will assign founder as the label of all sentences that contain Apple and Steve Jobs together. However, it suffers from noisy labeling problem due to the irrelevance of aligned text and incompleteness of KB, which consists of false positives and false negatives . The false positives means that not all sentences containing two entities mention the relation in KB, such as S1 and S2 in Table 1. And the false negatives are sentences are mislabeled as no relation ( NA ) due to the absence of relational fact in KB even though they express the target relation, such as S3 in Table 1. In order to reduce the impact of noisy data, previous works (Riedel et al., 2010; Hoffmann et al., 2011; Surdeanu et al., 2012; Zeng et al., 2015; Lin et al., 2016; Han et al., 2018b) adopt Multi-Instance Learning (MIL) for relation classification. Recent studies (Feng et al., 2018; Qin et al., 2018b,a) introduce reinforcement learning (RL) and adversarial learning to filter out incorrectly labeled sentences and achieve significant improvements. However, there are two remaining challenges of noisy labeling problem. tives. As illustrated in Figure 1, they concentrate on discovering the false positive instances 1 which are suppressed or removed at last and obtain a better decision boundary (green dashed line) than without consideration of false positive instances. Nevertheless, there are still a lot of false negative instances expressing similar semantic information with positive data. These instances also provide evidence for the target relation. The incorrect labels will weaken the discriminative capability of available features and confuse the model if they stay the same. However, when we remedy the label correctly, we indeed possess the optimal decision boundary (red solid line). There lacks an effective method to fully utilize noisy data of distant supervision. (Xu et al., 2013; Liu et al., 2017) apply methods such as pseudo-labels to directly correct the label of noisy data and Luo et al. (2017) design a dynamic transition matrix to model noise patterns. They still suffer from the drawback of error propagation during training. To tackle the above challenges, we propose a novel framework exploiting noisy data to enhance distant supervision relation classification. We design an instance discriminator with reinforcement learning to recognize both false positive and false negative instances simultaneously, and further split the noisy dataset into two sets, representing correctly labeled and incorrectly labeled data respectively. Additionally, we learn a robust relation classifier applying a semi-supervised learning method, whereby the correctly and incorrectly labeled data are regarded as labeled and unlabeled data. On the one hand, we mitigate the 1 In this paper, instance is the same as sentence side effect of incorrectly labeled data by recognizing them and treating them as unlabeled data. On the other hand, taking full advantage of the incorrectly labeled data in semi-supervised learning way facilitates robust property of model and improves generalization performance. Our contributions in this work are three-fold:  We propose a deep reinforcement learning framework to discriminate both false-positive and false-negative instances simultaneously. We introduce a semi-supervised learning method to fully exploit the noisy data in distant supervision relation classification. We conduct experiments on a widely used benchmark dataset and the results show that our method achieves significant improvements as compared with strong baselines. Many efforts based on supervised learning (Ze-lenko et al., 2002; Bunescu and Mooney, 2005; Zhou et al., 2005) have been devoted to relation classification. As is well-known, achieving a good performance while applying supervised learning paradigm requires a large amount of high-quality annotated data. To address the issue of data sparsity, Mintz et al. (2009) propose distant supervision to automatically annotate large scale training data, which inevitably results in noisy labeling problem. To tolerate noisy instances in positive examples, most early approaches employ multi-instance learning framework, including multi-instance single-label learning (Riedel et al., 2010) and multi-instance multi-label learning (Hoff-mann et al., 2011; Surdeanu et al., 2012). Recently, deep learning has also been introduced to propose an end-to-end convolutional neural network for relation classification (Zeng et al., 2014). In the sentences bag of one entity pair, Zeng et al. (2015) select the most reliable sentence, and Lin et al. (2016) propose attention schemes to de-emphasize unreliable sentences. Han et al. (2018b) incorporate hierarchical information of relations to enhance the attention scheme. But they fail to handle the issue where all sentences in one bag are mislabeled. Feng et al. (2018); Qin et al. (2018b,a) further achieve improvement by using reinforcement Instance Discriminator with Reinforcement Learning Relation Classifier with Semi-Supervised Learning PosAgent FN FPTP NegAgent DLDNADPOSTN DU Instance State Agent Classifier x u Encoder z Decoder Encoder Decoder U loss y u x l y l z x u ' x l ' y pred L loss C loss reward Figure 2: The framework of train process. learning and adversarial learning to explicitly remove incorrectly labeled sentences. However, they neglect the useful inherent information of those sentences which should be replaced correctly. In other words, they remove the noise rather than utilize it in the right way. Furthermore, Xu et al. (2013) correct false negative instances by using pseudo-relevance feedback to expand the origin knowledge base. Liu et al. (2017) apply a dynamic soft-label instead of the immutable hard label produced by DS during the training process. Luo et al. (2017) design a transition matrix which characterizes the underlying noise pattern to correct noisy labels. They utilize the noisy data and address the false negative problem to some extent, but they still suffer from the drawback that errors may be propagated because the model is unable to correct its own mistakes. In this work, we propose a unified framework for learning a discriminator to recognize both false-positive and false-negative instances with reinforcement learning, and utilizing the incorrectly labeled data as unlabeled data in semi-supervised learning way. In this section, we introduce our framework and the details of instance discriminator and relation classifier as follows. In MIL paradigm, the entire instances are split into multiple entity-pair bags { B h i ,t i } ki =1 . The sentences in B h,t mention both head entity h and tail entity t . Here we denote dataset as D = { ( x i , y i ) } ni =1 , where x i is a sentence associated with the corresponding entity pair, y i is a noisy relation label produced by distant supervision and n is the total number of sentences contained in each bag. As mentioned above, NA is a special relation which indicates the sentence does not express any relations in the KB. We define other relations in the KB as positive relations. Accordingly, we split the dataset into DPOS and DNA . In the scenario of distant supervision, an ideal model is not only capable of capturing valid supervision information about correctly labeled data with less noise, but also leveraging information contained in incorrectly labeled data by correcting the noisy label implicitly. As a result, we solve the task of distant supervision relation classification in two steps. As depicted in Figure 2, we design an instance discriminator to heuristically recognize false positive and false negative instances from the noisy distantly-supervised dataset with reinforcement learning. The correctly labeled instances discovered by the discriminator are split into labeled data while the incorrectly labeled ones are split into unlabeled data. The details of the instance discriminator will be introduced in Section 3.2. After scanning the entire noisy dataset, we train a robust classifier with semi-supervised learning utilizing above labeled data and unlabeled data. The details of the relation classifier will be introduced in Section 3.3. Meanwhile, the relation classifier provides rewards to the instance discriminator for updating parameters of its policy function. We regard recognizing incorrectly labeled instances as a reinforcement learning problem. The instance discriminator acts as an agent interacting with the environment that consists of a noisy dataset and a relation classifier. The agent is parameterized with a policy network  ( a | s ;  ) which gives the probability distribution of action a at each state s and receives reward r from the relation classifier to update parameters  . Note that NA indicates that there is no relation between two entities or that the relation is of no interest. The relation NA is very ambiguous since instances have no unified pattern. Thus we cannot decide whether a sentence belongs to NA only by the fact that it does not express any other positive relation. Under this consideration, we adopt two agents, PosAgent and NegAgent, to recognize false positive and false negative instances respectively. The definitions of the components in RL are introduced as follows. State The state includes the semantic and syntactic information of current sentence and the relation label given by DS. We use a piecewise convolutional neural network (PCNN) (Zeng et al., 2015) to convert each sentence into real-valued vector x and build a class representation matrix M to represent each relation type. As we decide whether the current sentence is correctly labeled according to the similarity between the semantic meanings of sentence and relation, we only take the current sentence into consideration without sentences in early states. For PosAgent, state s p is the concatenation of the current sentence vector x and corresponding relation embedding. For NegAgent, we represent state s n by the vector of relational scores based on the representation of the current sentence x . where y is relation label of the current sentence. b  R n r is a bias vector and n r is the number of class. Action We desire the agent to distinguish whether the current sentence is mislabeled or not. Therefore, the action of our agent is defined as a i  { 0 , 1 } , where 0 indicates the sentence is incorrectly labeled and 1 indicates the sentence is correctly labeled. Reward The reward function can reflect the advantage of redistributing the noisy data. As previously mentioned, the actions of our agent redistribute noisy data into labeled data and unlabeled data, corresponding to correctly labeled and incorrectly labeled instances. Therefore, the average likelihood of labeled data will be larger than unlabeled data when the agent makes correct actions. We define the difference of likelihood between them as the reward to evaluate the performance of our policy. Consequently, the reward is defined as follows: r =  ( 1 | L | (cid:88) x  L p  ( y | x )  1 | U | (cid:88) x  U p  ( y | x )) (2) where L and U is the subset of labeled data and unlabeled data respectively, and y is the relation label given by DS. p  ( y | x ) is calculated by the relation classifier from the semi-supervised learning framework. is used to scale the difference to a rational numeric range. The objective of the agent is to maximize the expected reward of the actions sampled from the probability distribution. Given a mini-batch data B , following the policy, our agent produces a set of probability distributions of actions  ( a i | s i ;  ) . Based on the actions, the agent achieves a performance-driven reward r . We use a policy gradient strategy to compute the gradient and update our agent referring to the policy gradient theorem (Sutton et al., 1999) and the REINFORCE algorithm (Williams, 1992). The parameters of the policy network are updated according to the following gradient:    +  | B | (cid:88) i =1 (cid:53)  r log  ( a i | s i ;  ) (3) As the goal of our agent is to determine whether an annotated sentence expresses target relation with weak supervision, we need a relation classifier to compute the reward for updating the policy network. We first pre-train our classifier on the entire dataset with supervised learning until rough convergence. Then we pre-train the policy network by receiving rewards from the pre-trained classifier with the parameters frozen. The pre-training strategy is necessary as it saves time that would otherwise be spent training the model by trial and error. It is also widely used by other related works (Silver et al., 2016; Bahdanau et al., 2016). The training procedure for instance discriminator is summarized in Algorithm 1. 3.3 Relation Classifier In order to reach the maximum utilization of noisy data, we train our relation classifier with semi-supervised learning. Below, we introduce PCNN and SemiVAE, the method we adopt for semi-supervised learning. We take the widely used CNN architecture (PCNN) (Zeng et al., 2015; Lin et al., 2016) to encode input sentences into low-dimensional vectors and predict their corresponding relation labels. Given a sentence containing an entity pair, we represent the i -th word as v i by concatenating its word embedding w i and position embedding p i which encodes the relative distances from it to two entities ( v i  R d , w i  R d w , p i  R d p , d = d w + d p ). Afterward, the convolution layer applies a kernel of the window size l to slide over the input sequence { v 1 , v 2 , ..., v m } and output the d h dimensional hidden embeddings h , where h  R m  d c and d c is the number of feature maps. Then, piecewise max-pooling is used to divide the hidden embeddings into three parts { h 1 , h 2 , h 3 } by the position of head and tail entities. We perform max-pooling on each part respectively and get final embedding x by concatenating the pooling results, where x  R d s ( d s = d c  3) . Finally, we formalize the probability of predicting y given sentence x as follows: SemiVAE, a semi-supervised learning method based on variational inference, is introduced and developed by (Kingma et al., 2014; Xu et al., 2017). The inference model consists of three components as follows. An encoder network p  ( z | x, y ) encodes data x and label y into a latent variable z . The decoder network p  ( x | z, y ) is used to estimate the probability of generating x given z and categorical label y . Finally, classifier p  ( y | x ) predict the corresponding label y of x . We model both encoder and decoder by multilayer perceptron (MLP) and employ the PCNN model as the classifier in SemiVAE. where first term represent the expectation of the conditional log-likelihood on latent variable z and the last term is Kullabck-Leibler divergence between the prior distribution p ( z ) and the latent posterior p  ( z | x l , y l ) . For the case of unlabeled data x u , the unobserved label y u is obtained from the classifier in the inference model. The variational lower bound is: log p  ( x u )  (cid:88) y p  ( y u | x u )( L ( x u , y u ))+ H ( p  ( y u | x u )) = U ( x u ) (7) where H denotes the entropy of p  ( y u | x u ) . Since the classifier p  ( y | x ) is unable to learn directly from labeled data, a classification loss is introduced as: C = E ( x,y )  D l [  log p  ( y | x )] (8) To maximize the evidence lower bound of both labeled data and unlabeled data and minimize the classification loss, the objective is defined as: J = (cid:88) ( x,y ) D l L ( x, y ) + (cid:88) x D u U ( x ) +  C (9) where D l and D u are labeled and unlabeled data respectively,  is a factor used to scale the classification loss of labeled data. Algorithm 1 Reinforcement Learning Algorithm for Instance Discriminator. Algorithm 2 Semi-supervised Learning Algorithm Input: Labeled data D l , unlabeled data D u . 1: Initialize parameters of relation classifier as  . 2: for epoch i = 1  N do 3: Sample m data pair ( x l , y l ) from D l 4: Sample m data x u from D u and predict their corresponding unobserved label y u via p  ( y | x ) 5: Update  by Eq.(9) 6: end for After our reinforcement learning process, we obtain an instance discriminator which possesses the capability of recognizing incorrectly labeled instances from the noisy dataset. Additionally, the entire DS dataset D is split into labeled data D l and unlabeled data D u . Therefore, we utilize the above data to train SemiVAE model and obtain a robust relation classifier which explicitly learns from correctly labeled data and correct incorrectly labeled data implicitly. The training procedure for relation classifier is summarized in Algorithm 2. 4 Experiment 4.1 Datasets and Evaluation We evaluate our model on a widely used dataset that is generated by aligning entity pairs from Batch size b s 160 Word Dimension d w 50 Position Dimension d p 5  2 Convolution Filter Dimension d c 230 Convolution Window Size l 3 Latent Variable Dimension d z 100 Dropout p 0.5 Regulator  ,  100, 2 Table 2: Hyperparameter settings Freebase with New York Times corpus(NYT) 2 and developed by (Riedel et al., 2010). Entity mentions are recognized by the Stanford named entity recognizer (Finkel et al., 2005). The relation facts in Freebase are divided into two parts for training and testing respectively. The sentences from the corpus of the years 2005-2006 are used as the training instances, and sentences from 2007 are used as the testing instances. There are 52 positive relations and a special relation NA . Following previous works, we evaluate our model on the held-out evaluation, which compares relation facts extracted from the test corpus with those in Freebase. We adopt aggregated preci-sion/recall curves and precision@N (P@N) to illustrate the performance of our model. We adopt the Adam (Kingma and Ba, 2014) optimizer to optimize our instance discriminator and relation classifier with learning rate 0.0001 and 0.001 respectively. We also apply dropout to prevent overfitting. More detailed hyperparameter settings are presented in Table 2. 4.3 Overall Evaluations Results We adopt the following baselines with which we compare our model:  Mintz (Mintz et al., 2009) is the original distantly supervised model. MultiR (Hoffmann et al., 2011) and MIML (Surdeanu et al., 2012) handle overlapping relation problem with graphical model in multi-instance and multi-instance multi-label framework. All above models are based on handcrafted feature. PCNN+ONE (Zeng et al., 2015) and PCNN+ATT (Lin et al., 2016) are both robust models to solve noisy labeling problem 2 http://iesl.cs.umass.edu/riedel/ecml/ 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Recall 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 P r e c i s i o n RCENDPCNN+HATTPCNN+ATT+SLPCNN+ONE+SLPCNN+ATTPCNN+ONEMIMLREMultiRMintz Figure 3: Precision-recall curves of our model and baselines. based on the at-least-one assumption and selective attention. PCNN+HATT (Han et al., 2018b) is a attention-based method which employs hierarchical attention to exploit correlations among relations. PCNN+ONE+SL and PCNN+ATT+SL (Liu et al., 2017) use a soft-label method to alleviate the negative impact of the noisy labeling problem. We compare our model with aforementioned baselines and the results are shown in Figure 3. From the overall result we can see that: (1) All feature-based models preform poorly as their features are derived from NLP tools, which will generate errors that propagate through in model. (2) PCNN+ONE and PCNN+ATT boost the performance because they reduce noise in the bag of entity pair by selecting the most confident sentence or de-emphasize the incorrectly labeled sentences with an attention mechanism. (3) When PCNN+ONE and PCNN+ATT use soft labels, they achieve an improvement. This indicates correcting the noisy label is helpful to relation classification in MIL scheme. (4) PCNN+HATT further enhances the performance as it incorporates hierarchical information of relations to improve the attention mechanism. (5) Our method RCEND achieves the best precision over the entire recall 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Recall 0.4 0.5 0.6 0.7 0.8 0.9 1.0 P r e c i s i o n RCENDRCEND w/o Semi PCNN+HATT Figure 4: Precision-recall curves of our model with different settings. range compared with all baselines. The performance achieves further improvement when we regard the incorrectly labeled sentences as unlabeled data and adopt a semi-supervised learning method to train our model. It shows that exploiting noisy data with our method is beneficial to promote distant supervision relation classification. We also report the result of Precisions@N (100, 200, 300) in Table 3. We can see that our method outperforms the baselines on the precision values of top N triples extracted. To further verify the impact of the unlabeled data, we conduct experiments with both utilization and non-utilization of unlabeled data. The results are presented in Figure 4. Note that, the method RCEND w/o Semi is similar to the method proposed by (Feng et al., 2018), which only removes the incorrectly labeled sentences but does not fully utilize them. We can see that it achieves higher precision over the entire level of recall compared to PCNN+HATT, the best noise-tolerate method in MIL scheme, which shows that removing noise is better than dealing with them with soft attention weights. However, it is still unable to surpass our method. In Table 4, our method also shows notable improvement over RCEND w/o Semi. This demonstrates that fully utilizing noisy data is more advantageous than reducing them. This can be partially explained due to the label rectification of the incorrectly labeled data during semi-supervised learning with correctly labeled data which improves the generalization performance. The goal of this experiment is to inspect whether the relation classifier is enhanced more through the utilization of false negatives or through the utilization of false positives. As depicted in Figure 5, RCEND(P) only recognizes the false positive sentences in DPOS by PosAgent and regards them as unlabeled data. Likewise, RCEND(N) only discovers and utilizes false negative sentences. RCEND(P) and RCEND(N) behave similarly and achieve further improvement when utilizing both false-positive and false-negative sentences, which implies that both of them are important and promote the ability of our relation classifier. And the results in Table 4 also show utilizing false negative data performs slightly better than false positives since false negative data might be predicted as positive relation and increase samples of the target relation to learn a more accurate decision boundary. We sample some examples of incorrectly labeled data which are regarded as unlabeled data during training. In Table 5, it can be seen that our discriminator recognizes both false positive and false negative instances. For example, though the fact (John Allison, EmployedBy , Opera) is absent in the KB due to the incompleteness of the KB, C2 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Recall 0.4 0.5 0.6 0.7 0.8 0.9 1.0 P r e c i s i o n RCENDRCEND(P)RCEND(N)PCNN+HATT Figure 5: Precision-recall curves of our model with different settings. expresses EmployedBy relation and provides evidence of target relation. Additionally, C4 is mislabeled as BornIn due to the relational fact (Bill Cosby, BornIn , Philadelphia), even though it mentions LivedIn relation. Further more, they are all predicted correctly by our relation classifier in the end which shows that our model indeed captures the valid information of noisy data and exploits them to enhance its ability. In this paper, we proposed RCEND to fully exploit valid information of the noisy data in distant supervision relation classification. The instance discriminator is trained with reinforcement learning, which aims to recognize the instances mislabeled by distant supervision. We treat the correctly labeled instances as labeled data and incorrectly labeled ones as unlabeled data. Afterward, we adopt a semi-supervised learning method to learn a robust relation classifier to utilize the data. In this way, not only can our model reduce the side effect of noisy labels, but also adequately take advantage of valid information contained in noisy data. Experiments demonstrate that our model outperforms state-of-the-art baselines. We would like to express gratitude to Robert Ridley and the anonymous reviewers for their valuable feedback on the paper. This work is supported by the National Natural Science Foundation of China (No. 61672277, U1836221), the Jiangsu Provincial Research Foundation for Basic Research (No. BK20170074).
abstain abstain objective abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective objective abstain objective objective objective objective method method method abstain objective abstain result other other other other other other other other other other other other other other other other objective method abstain abstain method abstain method method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain method other abstain method abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method method method method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective abstain abstain method method objective other other