This paper presents a neural relation extraction method to deal with the noisy training data generated by distant supervision. Previous studies mainly focus on sentence-level de-noising by designing neural networks with intra-bag attentions. In this paper, both intra-bag and inter-bag attentions are considered in order to deal with the noise at sentence-level and bag-level respectively. First, relation-aware bag representations are calculated by weighting sentence embeddings using intra-bag attentions. Here, each possible relation is utilized as the query for attention calculation instead of only using the target relation in conventional methods. Furthermore, the representation of a group of bags in the training set which share the same relation label is calculated by weighting bag representations using a similarity-based inter-bag attention module. Finally, a bag group is utilized as a training sample when building our relation extractor. Experimental results on the New York Times dataset demonstrate the effectiveness of our proposed intra-bag and inter-bag attention modules. Our method also achieves better relation extraction accuracy than state-of-the-art methods on this dataset 1 . Relation Extraction is a fundamental task in natural language processing (NLP), which aims to extract semantic relations between entities. For example, sentence [ Barack Obama ] e 1 was born in [ Hawaii ] e 2  expresses the relation BornIn between entity pair Barack Obama and Hawaii . Conventional relation extraction methods, such as (Zelenko et al., 2002; Culotta and Sorensen, 2004; Mooney and Bunescu, 2006), adopted supervised training and suffered from the lack of 1 The code is available at https://github.com/ZhixiuYe/Intra-Bag-and-Inter-Bag-Attentions . large-scale manually labeled data. To address this issue, the distant supervision method (Mintz et al., 2009) was proposed, which generated the data for training relation extraction models automatically. The distant supervision assumption says that if two entities participate in a relation, all sentences that mention these two entities express that relation. It is inevitable that there exists noise in the data labeled by distant supervision. For example, the precision of aligning the relations in Freebase to the New York Times corpus was only about 70% (Riedel et al., 2010). Thus, the relation extraction method proposed in (Riedel et al., 2010) argued that the distant supervision assumption was too strong and relaxed it to expressed-at-least-once assumption. This assumption says that if two entities participate in a relation, at least one sentence that mentions these two entities might express that relation. An example is shown by sentences S1 and S2 in Table 1. This relation extraction method first divided the training data given by distant supervision into bags where each bag was a set of sentences containing the same entity pair. Then, bag representations were derived by weighting sentences within each bag. It was expected that the weights of the sentences with incorrect labels were reduced and the bag representations were calculated mainly using the sentences with correct labels. Finally, bags were utilized as the samples for training relation extraction models instead of sentences. In recent years, many relation extraction methods using neural networks with attention mechanism (Lin et al., 2016; Ji et al., 2017; Jat et al., 2018) have been proposed to alleviate the influence of noisy training data under the expressed-at-least-once assumption. However, these methods still have two deficiencies. First, only the target relation of each bag is used to calculate the attention weights for deriving bag representations from sentence embeddings at training stage. Here we argue that the bag representations should be calculated in a relation-aware way. For example, the bag B1 in Table 1 contains two sentences S1 and S2. When this bag is classified to relation BornIn , the sentence S1 should have higher weight than S2, but when classified to relation PresidentOf , the weight of S2 should be higher. Second, the expressed-at-least-once assumption ignores the noisy bag problem which means that all sentences in one bag are incorrectly labeled. An example is shown by bag B2 in Table 1. In order to deal with these two deficiencies of previous methods, this paper proposes a neural network with multi-level attentions for distant supervision relation extraction. At the instance/sentence-level, i.e., intra-bag level, all possible relations are employed as queries to calculate the relation-aware bag representations instead of only using the target relation of each bag. To address the noisy bag problem, a bag group is adopted as a training sample instead of a single bag. Here, a bag group is composed of bags in the training set which share the same relation label. The representation of a bag group is calculated by weighting bag representations using a similarity-based inter-bag attention module. The contributions of this paper are threefold. First, an improved intra-bag attention mechanism is proposed to derive relation-aware bag representations for relation extraction. Second, an inter-bag attention module is introduced to deal with the noisy bag problem which is ignored by the expressed-at-least-once assumption. Third, our methods achieve better extraction accuracy than state-of-the-art models on the widely used New York Times (NYT) dataset (Riedel et al., 2010). Some previous work (Zelenko et al., 2002; Mooney and Bunescu, 2006) treated relation extraction as a supervised learning task and designed hand-crafted features to train kernel-based models. Due to the lack of large-scale manually labeled data for supervised training, the distant supervision approach (Mintz et al., 2009) was proposed, which aligned raw texts toward knowledge bases automatically to generate relation labels for entity pairs. However, this approach suffered from the issue of noisy labels. Therefore, some subsequent studies (Riedel et al., 2010; Hoffmann et al., 2011; Surdeanu et al., 2012) considered distant supervision relation extraction as a multi-instance learning problem, which extracted relation from a bag of sentences instead of a single sentence. With the development of deep learning techniques (LeCun et al., 2015), many neural-network-based models have been developed for distant supervision relation extraction. Zeng et al. (2015) proposed piecewise convolutional neural networks (PCNNs) to model sentence representations and chose the most reliable sentence as the bag representation. Lin et al. (2016) employed PCNNs as sentence encoders and proposed an intra-bag attention mechanism to compute the bag representation via a weighted sum of all sentence representations in the bag. Ji et al. (2017) adopted a similar attention strategy and combined entity descriptions to calculate the weights. Liu et al. (2017) proposed a soft-label method to reduce the influence of noisy instances. All these methods represented a bag with a weighted sum of sentence embeddings, and calculated the probability of the bag being classified into each relation using the same bag representation at training stage. In our proposed method, intra-bag attentions are computed in a relation-aware way, which means that different bag representations are utilized to calculate the probabilities for different relation types. Besides, these existing methods focused on intra-bag attentions and ignored the noisy bag problem. Some data filtering strategies for robust distant supervision relation extraction have also been proposed. Feng et al. (2018) and Qin et al. (2018b) both employed reinforcement learning to train instance selector and to filter out the samples with wrong labels. Their rewards were calcu-intra-bag attention   11  21   1 1  11   1 1 SE SE SER  1 inter-bag attention  input sentence sentence encoder sentence representation bag representation intra-bag attention inter-bag attention representation of a group of bags  21 intra-bag attention   12  22  12   2 2 SE SE SER  2  22 intra-bag attention   1  2      1     SE SE SER    2   matrix vector module   2 2 Figure 1: The framework of our proposed neural network with intra-bag and inter-bag attentions for relation extraction. lated from the prediction probabilities and the performance change of the relation classifier respectively. Qin et al. (2018a) designed an adversarial learning process to build a sentence-level generator via policy-gradient-based reinforcement learning. These methods were proposed to filter out the noisy data at sentence-level and also failed to deal with the noisy bag problem explicitly. In this section, we introduce a neural network with intra-bag and inter-bag attentions for distant supervision relation extraction. Let g = { b 1 , b 2 , ..., b n } denote a group of bags which have the same relation label given by distant supervision, and n is the number of bags within this group. Let b i = { x i 1 , x i 2 , ..., x im i } denote all sentences in bag b i , and m i is the number of sentences in bag b i . Let x i j = { w i j 1 , w i j 2 , ..., w i jl ij } denote the j -th sentence in the i -th bag and l ij is its length (i.e., number of words). The framework of our model is shown in Fig. 1, which has three main modules. Sentence Encoder Given a sentence x ij and the positions of two entities within this sentence, CNNs or PCNNs (Zeng et al., 2015) are adopted to derive the sentence representation s i j . Intra-Bag Attention Given the sentence representations of all sentences within a bag b i and a relation embedding matrix R , attention weight vectors  ik and bag representations b ik are calculated for all relations, where k is the relation index. Inter-Bag Attention Given the representations of all bags with the group g , a weight matrix  is further calculated via similarity-based attention mechanism to obtain the representation of the bag group. Each word w ijk within the sentence x ij is first mapped into a d w -dimensional word embedding. To describe the position information of two entities, the position features (PFs) proposed in (Zeng et al., 2014) are also adopted in our work. For each word, the PFs describe the relative distances between current word and the two entities and are further mapped into two vectors p ijk and q ijk of d p dimensions. Finally, these three vectors are concatenated to get the word representation w ijk = [ e ijk ; p ijk ; q ijk ] of d w + 2 d p dimensions. For sentence x ij , the matrix of word representations W i j  R l ij  ( d w +2 d p ) is first input into a CNN with d c filters. Then, piecewise max pooling (Zeng et al., 2015) is employed to extract features from the three segments of CNN outputs, and the segment boundaries are determined by the positions of the two entities. Let S i  R m i  3 d c represent the representations of all sentences within bag b i , and R  R h  3 d c denote a relation embedding matrix where h is the number of relations. Different from conventional methods (Lin et al., 2016; Ji et al., 2017) where a unified bag representation was derived for relation classification, our method calculates bag representations b ik for bag b i on the condition of all possible relations as b ik = m i (cid:88) j =1  ikj s ij , (1) where k  { 1 , 2 , ..., h } is the relation index and  ikj is the attention weight between the k -th relation and the j -th sentence in bag b i . ikj can be further defined as  ikj = exp ( e ikj ) (cid:80) m i j (cid:48) =1 exp ( e ikj (cid:48) ) , (2) where e ikj is the matching degree between the k -th relation query and the j -th sentence in bag b i . In our implementation, a simple dot product between vectors is adopted to calculate the matching degree as e ikj = r k s i (cid:62) j , (3) where r k is the k -th row of the relation embedding matrix R 2 . Finally, the representations of bag b i compose the matrix B i  R h  3 d c in Fig. 1, where each row corresponds to a possible relation type of this bag. In order to deal with the noisy bag problem, a similarity-based inter-bag attention module is designed to reduce the weights of noisy bags dynamically. Intuitively, if two bags b i 1 and b i 2 are both labeled as relation k , their representations b i 1 k and b i 2 k should be close to each other. Given a group of bags with the same relation label, we assign higher weights to those bags which are close to other bags 2 We also tried r k As i (cid:62) j , where A was a diagonal matrix, in experiments and achieved similar performance. in this group. As a result, the representation of bag group g can be formulated as g k = n (cid:88) i =1  ik b ik , (4) where g k is the k -th row of the matrix G  R h  3 d c in Fig. 1, k is the relation index and  ik composes the attention weight matrix   R n  h . Each  ik is defined as  ik = exp (  ik ) (cid:80) ni =1 exp (  ik ) , (5) where  ik describes the confidence of labeling bag b i with the k -th relation. Inspired by the self-attention algorithm (Vaswani et al., 2017) which calculates the attention weights for a group of vectors using the vectors themselves, we calculate the weights of bags according to their own representations. Mathematically,  ik is defined as  ik = (cid:88) i (cid:48) =1 ,...,n,i (cid:48) (cid:54) = i similarity( b ik , b i (cid:48) k ) , (6) where the function similarity is a simple dot product in our implementation as similarity( b ik , b i (cid:48) k ) = b ik b i (cid:48) (cid:62) k . And also, in order to prevent the influence of vector length, all bag representations b ik are normalized to unit length as b ik = b ik / || b ik || 2 before calculating Eq.(4)-(7). Then, the score o k of classifying bag group g into relation k is calculated via g k and relation embedding r k as o k = r k g (cid:62) k + d k , (8) where d k is a bias term. It should be noticed that the same relation embedding matrix R is used for calculating Eq.(3) and Eq.(8). Similar to Lin et al. (2016), the dropout strategy (Srivastava et al., 2014) is applied to bag representation B i to prevent overfitting. First of all, all sentences in the training set that contain the same two entities are accumulated into one bag. Then, we tie up every n bags that share the same relation label into a group. It should be noticed that a bag group is one training sample in our method. Therefore, the model can also be trained in mini-batch mode by packing multiple bag groups into one batch. In our implementation, the objective function is defined as where T is the set of all training samples and  is the set of model parameters, including word embedding matrix, position feature embedding matrix, CNN weight matrix and relation embedding matrix. The model parameters are estimated by minimizing the objective function J(  ) through mini-batch stochastic gradient descent (SGD). As introduced above, at the training phase of our proposed method, n bags which have the same relation label are accumulated into one bag group and the weighted sum of bag representations is calculated to obtain the representation G of the bag group. Due to the fact that the label of each bag is unknown at test stage, each single bag is treated as a bag group (i.e., n =1) when processing the test set. And also, similar to (Qin et al., 2018b), we only apply inter-bag attentions to positive samples, i.e., the bags whose relation label is not NA ( NoRelation ). The reason is that the representations of the bags that express no relations are always diverse and it's difficult to calculate suitable weights for them. In our implementation, a pre-training strategy is adopted. We first train the model with only intra-bag attentions until convergence. Then, the inter-bag attention module is added and the model parameters are further updated until convergence again. Preliminary experimental results showed Component Parameter Value word embedding dimension 50 position feature max relative distance  30 dimension 5 CNN window size 3 filter number 230 dropout dropout rate 0.5 optimization strategy SGD learning rate 0.1 batch size N p 50 batch size N t 10 group size n 5 gradient clip 5.0 Table 2: Hyper-parameters of the models built in our experiments. that this strategy can lead to better model performance than considering inter-bag attentions from the very beginning. The New York Times (NYT) dataset was adopted in our experiments. This dataset was first released by (Riedel et al., 2010) and has been widely used by previous research on distant supervision relation extraction (Liu et al., 2017; Jat et al., 2018; Qin et al., 2018a,b). This dataset was generated by aligning Freebase with the New York Times (NYT) corpus automatically. There were 52 actual relations and a special relation NA which indicated there was no relation between two entities. Following previous studies (Mintz et al., 2009; Liu et al., 2017), we evaluated our models on the held-out test set of the NYT dataset. Precision-recall (PR) curves, area under curve (AUC) values and Precision@N (P@N) values (Lin et al., 2016) were adopted as evaluation metrics in our experiments. All of the numerical results given by our experiments were the mean values of 10 repetitive trainings, and the PR curves were randomly selected from the repetitions because there was no significant visual difference among them. All of the hyperparameters used in our experiments are listed in Table 2. Most of them followed the hyperparameter settings in (Lin et al., 2016). The 50-dimensional word embeddings released by (Lin et al., 2016) 3 were also adopted for initialization. The vocabulary contained the words which appeared more than 100 times in the NYT corpus. 3 https://github.com/thunlp/NRE . Two different batch sizes N p and N t were used for pre-training and training respectively. In our experiments, a grid search is employed using training set to determine the optimal values of n , N p and N t among n  { 3 , 4 , ..., 10 } , N p  { 10 , 20 , 50 , 100 , 200 } and N t  { 5 , 10 , 20 , 50 } . Note that increasing the bag group size n may boost the effect of inter-bag attentions but lead to less training samples. The effects of inter-bag attentions would be lost when n =1. For optimization, we employed mini-batch SGD with the initial learning rate of 0.1. The learning rate was decayed to one tenth every 100,000 steps. The pre-trained model with only intra-bag attentions converged within 300,000 steps in our experiments. Thus, the initial learning rate for training the model with inter-bag attentions was set as 0.001. Eight models were implemented for comparison. The names of these models are listed in Table 3, where CNN and PCNN denote using CNNs or piecewise CNNs in sentence encoders respectively, ATT BL means the baseline intra-bag attention method proposed by (Lin et al., 2016), ATT RA means our proposed relation-aware intra-bag attention method, and BAG ATT means our proposed inter-bag attention method. At the training stage of the ATT BL method, the relation query vector for attention weight calculation was fixed as the embedding vector associated with the distant supervision label for each bag. At the test stage, all relation query vectors were applied to calculate the posterior probabilities of relations respectively and the relation with the highest prob-0.0 0.1 0.2 0.3 0.4 0.5 Recall 0.4 0.5 0.6 0.7 0.8 0.9 1.0 P r e c i s i o n PCNN+ATT_BL PCNN+ATT_BL+BAG_ATT PCNN+ATT_RA PCNN+ATT_RA+BAG_ATT Figure 3: PR curves of different models using PCNN sentence encoders. ability was chosen as the classification result (Lin et al., 2016). The means and standard deviations of the AUC values given by the whole PR curves of these models are shown in Table 3 for a quantitative comparison. Following (Lin et al., 2016), we also plotted the PR curves of these models in Fig. 2 and 3 with recall smaller than 0 . 5 for a visualized comparison. From Table 3, Fig. 2 and Fig. 3, we have the following observations. (1) Similar to the results of previous work (Zeng et al., 2015), PCNNs worked better than CNNs as sentence encoders. (2) When using either CNN or PCNN sentence encoders, ATT RA outperformed ATT BL . It can be attributed to that the ATT BL method only considered the target relation when deriving bag representations at training time, while the ATT RA method calculated intra-bag attention weights using all relation embeddings as queries, which improved the flexibility of bag representations. (3) For both sentence encoders and both intra-bag attention methods, the models with BAGATT always achieved better performances than the ones without BAG ATT . This result verified the effectiveness of our proposed inter-bag attention # of Test Sentences one two all P@N(%) 100 200 300 mean 100 200 300 mean 100 200 300 mean (Lin et al., 2016) 73.3 69.2 60.8 67.8 77.2 71.6 66.1 71.6 76.2 73.1 47.4 72.2 (Liu et al., 2017) 84.0 75.5 68.3 75.9 86.0 77.0 73.3 78.8 87.0 84.5 77.0 82.8 CNN+ATT BL 74.2 68.9 65.3 69.5 77.8 71.5 68.1 72.5 79.2 74.9 70.3 74.8 CNN+ATT RA 76.8 72.7 67.9 72.5 79.6 73.9 70.7 74.7 81.4 76.3 72.5 76.8 CNN+ATT BL+BAG ATT 78.6 74.2 69.7 74.2 82.4 76.2 72.1 76.9 83.0 78.0 74.0 78.3 CNN+ATT RA+BAG ATT 79.8 75.3 71.0 75.4 83.2 76.5 72.1 77.3 87.2 78.7 74.9 80.3 PCNN+ATT BL 78.6 73.5 68.1 73.4 77.8 75.1 70.3 74.4 80.8 77.5 72.3 76.9 PCNN+ATT RA 79.4 73.9 69.6 74.3 82.2 77.6 72.4 77.4 84.2 79.9 73.0 79.0 PCNN+ATT BL+BAG ATT 85.2 78.2 71.3 78.2 84.8 80.0 74.3 79.7 88.8 83.7 77.4 83.9 PCNN+ATT RA+BAG ATT 86.8 77.6 73.9 79.4 91.2 79.2 75.4 81.9 91.8 84.0 78.7 84.8 Table 4: P@N values of the entity pairs with different number of test sentences. method for distant supervision relation extraction. (4) The best AUC performance was achieved by combining PCNN sentence encoders with the intra-bag and inter-bag attentions proposed in this paper. The PR curves of several models in previous work and our best model PCNN+ATT RA+BAG ATT are compared in Fig. 4, where Mintz (Mintz et al., 2009), MultiR (Hoffmann et al., 2011) and MIMLRE (Surdeanu et al., 2012) are conventional feature-based methods, and (Lin et al., 2016) and (Liu et al., 2017) are PCNN-based ones 4 . For a fair comparison with (Lin et al., 2016) and (Liu et al., 2017), we also plotted the curves with only the top 2000 points. We can see that our model achieved better PR performance than all the other models. ATT BL+DSGAN (Qin et al., 2018a) and ATT BL+RL (Qin et al., 2018b) are two recent studies on distant supervision relation extraction with reinforcement learning for data filtering, which reported the AUC values of PR curves composed by the top 2000 points. Table 5 compares the AUC values reported in these two papers and the results of our proposed models. We can see that introducing the proposed ATT RA and BAG ATT methods to baseline models achieved larger improvement than using the methods proposed in (Qin et al., 2018a,b). Following (Lin et al., 2016), we evaluated our models on the entity pairs with more than one training sentence. One, two and all sentences for each test entity pair were randomly selected to construct three new test sets. The P@100, P@200, P@300 values and their means given by our proposed models on these three test sets are reported in Table 4 together with the best results of (Lin et al., 2016) and (Liu et al., 2017). Here, P@N bag sentence correct? means the precision of the relation classification results with the top N highest probabilities in the test set. We can see our proposed methods achieved higher P@N values than previous work. Furthermore, no matter whether PCNN or BAG ATT were adopted, the ATT RA method outperformed the ATT BL method on the test set with only one sentence for each entity pair. Note that the decoding procedures of ATT BL and ATT RA were equivalent when there was only one sentence in a bag. Therefore, the improvements from ATT BL to ATT RA can be attributed to that ATT RA calculated intra-bag attention weights in a relation-aware way at the training stage. We divided the training set into 5 parts according to the number of sentences in each bag. For each bag, the inter-bag attention weights given by the PCNN+ATT RA+BAG ATT model were recorded. Then, the mean and standard deviation of inter-bag attention weights for each part of the training set were calculated and are shown in Table 7. From this table, we can see that the bag with smaller number of training sentences were usually assigned with lower inter-bag attention weights. This result was consistent with the finding in (Qin et al., 2018b) that the entity pairs with fewer training sentences were more likely to have incorrect relation labels. A test set example of relation /location/location/contains is shown in Table 6. The bag group contained 3 bags, which consisted of 2, 1, and 2 sentences respectively. We calculated the intra-bag and inter-bag attentions for this bag group using our PCNN+ATT RA+BAG ATT model and the weights of the target relation are also shown in Table 6. In this example, the second bag was a noisy bag because the only sentence in this bag didn't express the relation /location/location/contains between the two entities Naugatuck and Connecticut . In conventional methods, these three bags were treated equally for model training. After introducing inter-bag attention mechanism, the weight of this noisy bag was reduced significantly as shown in the last column of Table 6. In this paper, we have proposed a neural network with intra-bag and inter-bag attentions to cope with the noisy sentence and noisy bag problems in distant supervision relation extraction. First, relation-aware bag representations are calculated by a weighted sum of sentence embeddings where the noisy sentences are expected to have smaller weights. Further, an inter-bag attention module is designed to deal with the noisy bag problem by calculating the bag-level attention weights dynamically during model training. Experimental results on New York Times dataset show that our models achieved significant and consistent improvements compared with the models using only conventional intra-bag attentions. To deal with the multi-label problem of relation extraction and to integrate external knowledge into our model will be the tasks of our future work. We thank the anonymous reviewers for their valuable comments.
method abstain method abstain abstain abstain method objective result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain objective abstain abstain result other other other other other other other other other other objective other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain objective abstain abstain result abstain other