We study the problem of textual relation embedding with distant supervision. To combat the wrong labeling problem of distant supervision, we propose to embed textual relations with global statistics of relations, i.e., the co-occurrence statistics of textual and knowledge base relations collected from the entire corpus. This approach turns out to be more robust to the training noise introduced by distant supervision. On a popular relation extraction dataset, we show that the learned textual relation embedding can be used to augment existing relation extraction models and significantly improve their performance. Most remarkably, for the top 1,000 relational facts discovered by the best existing model, the precision can be improved from 83.9% to 89.3%. Relation extraction requires deep understanding of the relation between entities. Early studies mainly use hand-crafted features (Kambhatla, 2004; Zhou et al., 2005), and later kernel methods are introduced to automatically generate features (Zelenko et al., 2003; Culotta and Sorensen, 2004; Bunescu and Mooney, 2005; Zhang et al., 2006). Recently neural network models have been introduced to embed words, relations, and sentences into continuous feature space, and have shown a remarkable success in relation extraction (Socher et al., 2012; Zeng et al., 2014; Xu et al., 2015b; Zeng et al., 2015; Lin et al., 2016). In this work, we study the problem of embedding textual relations , defined as the shortest dependency path 1 between two entities in the dependency graph of a sentence, to improve relation extraction. relation extraction models (Bunescu and Mooney, 2005). A number of recent studies have explored textual relation embedding under the supervised setting (Xu et al., 2015a,b, 2016; Liu et al., 2016), but the reliance on supervised training data limits their scalability. In contrast, we embed textual relations with distant supervision (Mintz et al., 2009), which provides much larger-scale training data without the need of manual annotation. However, the assertion of distant supervision,  any sentence containing a pair of entities that participate in a knowledge base (KB) relation is likely to express the relation, can be violated more often than not, resulting in many wrongly labeled training examples. A representative example is shown in Figure 1. Embedding quality is thus compromised by the noise in training data. Our main contribution is a novel way to combat the wrong labeling problem of distant supervision. Traditional embedding methods (Xu et al., 2015a,b, 2016; Liu et al., 2016) are based on local statistics , i.e., individual textual-KB relation pairs like in Figure 1 (Left). Our key hypothesis is that global statistics is more robust to noise than local statistics . For individual examples, the relation label from distant supervision may be wrong from time to time. But when we zoom out to consider the entire corpus, and collect the global co-occurrence statistics of textual and KB relations, we will have a more comprehensive view of relation semantics: The semantics of a textual relation can then be represented by its co-occurrence distribution of KB relations. For example, the distribution in Figure 1 (Right) indicates that the textual relation SUBJECT nsubjpass  born nmod:in  OBJECT mostly means place of birth , and is also a good indicator of nationality , but not place of death . Although it is still wrongly labeled with place of death a number of times, the negative impact becomes negligible. Similarly, 820 Michael_Jackson was born in the US nsubjpass nmod:in Michael_Jackson US place_of_birth place_of_death Text Corpus Knowledge Base Michael_Jackson died in the US nsubj nmod:in Michael_Jackson US nsubjpass  born nmod:in  nsubj  died nmod:in  place of birth 1868 14 nationality 389 20 place of death 37 352 ... ... ... we can confidently believe that SUBJECT nsubj  died nmod:in  OBJECT means place of death in spite of the noise. Textual relation embedding learned on such global statistics is thus more robust to the noise introduced by the wrong labeling problem. We augment existing relation extractions using the learned textual relation embedding. On a popular dataset introduced by Riedel et al. (2010), we show that a number of recent relation extraction models, which are based on local statistics, can be greatly improved using our textual relation embedding. Most remarkably, a new best performance is achieved when augmenting the previous best model with our relation embedding: The precision of the top 1,000 relational facts discovered by the model is improved from 83.9% to 89.3%, a 33.5% decrease in error rate. The results suggest that relation embedding with global statistics can capture complementary information to existing local statistics based models. The rest of the paper is organized as follows. In Section 2 we discuss related work. For the modeling part, we first describe how to collect global co-occurrence statistics of relations in Section 3, then introduce a neural network based embedding model in Section 4, and finally discuss how to combine the learned textual relation embedding with existing relation extraction models in Section 5. We empirically evaluate the proposed method in Section 6, and conclude in Section 7. Relation extraction is an important task in information extraction. Early relation extraction methods are mainly feature-based (Kambhatla, 2004; Zhou et al., 2005), where features in various levels, including POS tags, syntactic and dependency parses, are integrated in a max entropy model. With the popularity of kernel methods, a large number of kernel-based relation extraction methods have been proposed (Zelenko et al., 2003; Culotta and Sorensen, 2004; Bunescu and Mooney, 2005; Zhang et al., 2006). The most related work to ours is by Bunescu and Mooney (Bunescu and Mooney, 2005), where the importance of shortest dependency path for relation extraction is first validated. More recently, relation extraction research has been revolving around neural network models, which can alleviate the problem of exact feature matching of previous methods and have shown a remarkable success (e.g., (Socher et al., 2012; Zeng et al., 2014)). Among those, the most related are the ones embedding shortest dependency paths with neural networks (Xu et al., 2015a,b, 2016; Liu et al., 2016). For example, Xu et al. (2015b) use a RNN with LSTM units to embed shortest dependency paths without typed dependency relations, while a convolutional neural network is used in (Xu et al., 2015a). However, they are all based on the supervised setting with a limited scale. In contrast, we embed textual relations with distant supervision (Mintz et al., 2009), which provides much larger-scale training data at a low cost. Various efforts have been made to combat the long-criticized wrong labeling problem of distant supervision. Riedel et al. (2010), Hoffmann et al. (2011), and Surdeanu et al. (2012) have attempted a multi-instance learning (Dietterich et al., 1997) framework to soften the assumption of distant supervision, but their models are still feature-based. Zeng et al. (2015) combine multi-instance learning with neural networks, with the assumption that at least one of the contextual sentences of an entity pair is expressing the target relation, but this will lose useful information in the neglected sentences. Instead, Lin et al. (2016) use all the contextual sentences, and introduce an attention mechanism to weight the contextual sentences. Li et al. (2017) also use an attention 821 mechanism to weight contextual sentences, and incorporate additional entity description information from knowledge bases. Luo et al. (2017) manage to alleviate the negative impact of noise by modeling and learning noise transition patterns from data. Liu et al. (2017) propose to infer the true label of a context sentence using a truth discovery approach (Li et al., 2016). Wu et al. (2017) incorporate adversarial training, i.e., injecting random perturbations in training, to improve the robustness of relation extraction. Using PCNN+ATT (Lin et al., 2016) as base model, they show that adversarial training can improve its performance by a good margin. However, the base model implementation used by them performed inferior to the one in the original paper and in ours, and therefore the results are not directly comparable. No prior study has exploited global statistics to combat the wrong labeling problem of distant supervision. Another unique aspect of this work is that we focus on com-pact textual relations, while previous studies along this line have focused on whole sentences. In universal schema (Riedel et al., 2013) for KB completion and relation extraction as well as its extensions (Toutanova et al., 2015; Verga et al., 2016), a binary matrix is constructed from the entire corpus, with entity pairs as rows and tex-tual/KB relations as columns. A matrix entry is 1 if the relational fact is observed in training, and 0 otherwise. Embeddings of entity pairs and relations, either directly or via neural networks, are then learned on the matrix entries, which are still individual relational facts, and the wrong labeling problem remains. Global co-occurrence frequencies (see Figure 1 (Right)) are not taken into account, which is the focus of this study. Another distinction is that our method directly models the association between textual and KB relations, while universal schema learns embedding for shared entity pairs and use that as a bridge between the two types of relations. It is an interesting venue for future research to comprehensively compare these two modeling approaches. When using a corpus to train statistical models, there are two levels of statistics to exploit: local and global . Take word embedding as an example. The skip-gram model (Mikolov et al., 2013) is based on local statistics: During training, we sweep through the corpus and slightly tune the nsubjpass SUBJECT born nmod:in OBJECT nsubj SUBJECT died nmod:in OBJECT place_of_birth place_of_death ... ... 0.73 0.89 Figure 2: Relation graph. embedding model in each local window (e.g., 10 consecutive words). In contrast, in global statistics based methods, exemplified by latent semantic analysis (Deerwester et al., 1990) and GloVe (Pen-nington et al., 2014), we process the entire corpus to collect global statistics like word-word co-occurrence counts, normalize the raw statistics, and train an embedding model directly on the normalized global statistics. Most existing studies on relation extraction are based on local statistics of relations, i.e., models are trained on individual relation examples. In this section, we describe how we collect global co-occurrence statistics of textual and KB relations, and how to normalize the raw statistics. By the end of this section a bipartite relation graph like Figure 2 will be constructed, with one node set being textual relations T , and the other being KB relations R . The edges are weighted by the normalized co-occurrence statistics of relations. Given a corpus and a KB, we first do entity linking on each sentence, and do dependency parsing if at least two entities are identified 2 . For each entity pair ( e, e 0 ) in the sentence, we extract the fully lexicalized shortest dependency path as a textual relation t , forming a relational fact ( e, t, e 0 ) . There are two outcomes from this step: a set of textual relations T = { t i } , and the support S ( t i ) for each t i . The support of a textual relation is a multiset containing the entity pairs of the textual relation. The multiplicity of an entity pair, m S ( t i ) ( e, e 0 ) , is the number of occurrences of the corresponding relational fact ( e, t i , e 0 ) in 2 In the experiments entity linking is assumed given, and dependency parsing is done using Stanford Parser (Chen and Manning, 2014) with universal dependencies. the corpus. For example, if the support of t i is S ( t i ) = { ( e 1 , e 0 1 ) , ( e 1 , e 0 1 ) , ( e 2 , e 0 2 ) , . . . } , entity pair ( e 1 , e 0 1 ) has a multiplicity of 2 because the relational fact ( e 1 , t i , e 0 1 ) occur in two sentences. We also get a set of KB relations R = { r j } , and the support S ( r j ) of a KB relation r j is the set of entity pairs having this relation in the KB, i.e., there is a relational fact ( e, r j , e 0 ) in the KB. The number of co-occurrences of a textural relation t i and a KB relation r j is n ij = X ( e,e 0 )  S ( r j ) m S ( t i ) ( e, e 0 ) , (1) i.e., every occurrence of relational fact ( e, t i , e 0 ) is counted as a co-occurrence of t i and r j if ( e, e 0 )  S ( r j ) . A bipartite relation graph can then be constructed, with T and R as the node sets, and the edge between t i and r j has weight n ij (no edge if n ij = 0 ), which will be normalized later. The raw co-occurrence counts have a heavily skewed distribution that spans several orders of magnitude: A small portion of relation pairs co-occur highly frequently, while most relation pairs co-occur only a few times. For example, a textual relation, SUBJECT nsubjpass  born nmod:in  OBJECT , may co-occur with the KB relation place of birth thousands of times (e.g.,  Michelle Obama was born in Chicago ), while a synonymous but slightly more compositional textual relation, SUBJECT nsubjpass  born nmod:in  city nmod:of  OBJECT , may only co-occur with the same KB relation a few times in the entire corpus (e.g.,  Michelle Obama was born in the city of Chicago ). Learning directly on the raw co-occurrence counts, an embedding model may put a disproportionate amount of weight on the most frequent relations, and may not learn well on the majority of rarer relations. Proper normalization is therefore necessary, which will encourage the embedding model to learn good embedding not only for the most frequent relations, but also for the rarer relations. A number of normalization strategies have been proposed in the context of word embedding, including correlationand entropy-based normalization (Rohde et al., 2005), positive pointwise mutual information (PPMI) (Bullinaria and Levy, 2007), and some square root type transformation (Lebret and Collobert, 2014). A shared goal is to reduce the impact of the most frequent words, e.g., the and is, which tend to be less informative for the purpose of embedding. We have experimented with a number of normalization strategies and found that the following strategy works best for textual relation embedding: For each textual relation, we normalize its co-occurrence counts to form a probability distribution over KB relations. The new edge weights of the relation graph thus become w ij =  p ( r j | t i ) = n ij / P j 0 n ij 0 . Every textual relation is now associated with a set of edges whose weights sum up to 1. We also experimented with PPMI and smoothed PPMI with  = 0 . 75 (Levy et al., 2015) that are commonly used in word embedding. However, the learned textual relation embedding turned out to be not very helpful for relation extraction. One possible reason is that PPMI (even the smoothed version) gives inappropriately large weights to rare relations (Levy et al., 2015). There are many textual relations that correspond to none of the target KB relations but are falsely labeled with some KB relations a few times by distant supervision. PPMI gives large weights to such falsely labeled cases because it thinks these events have a chance significantly higher than random. Next we discuss how to learn embedding of textual relations based on the constructed relation graph. We call our approach Glo bal R elation E mbedding (GloRE) in light of global statistics of relations. Given the relation graph, a straightforward way of relation embedding is matrix factorization, similar to latent semantic analysis (Deerwester et al., 1990) for word embedding. However, textual relations are different from words in that they are sequences composed of words and typed dependency relations. Therefore, we use recurrent neural networks (RNNs) for embedding, which respect the compositionality of textual relations and can learn the shared sub-structures of different textual relations (Toutanova et al., 2015). For the examples in Figure 1, an RNN can learn, from both textual relations, that the shared dependency relation  nmod:in  is indicative of location modifiers. It is worth noting that other models like convolutional neural networks can also be used, but it is not the focus of this paper to compare all the alternative embedding models; rather, we aim to show 823 -nsubjpass born nmod:in -nsubjpass born nmod:in <GO> place_of_birth : 0.73 Figure 3: Embedding model. the effectiveness of global statistics with a reasonable embedding model. For a textual relation, we first decompose it into a sequence of tokens { x 1 , ..., x m } , which includes lexical words and directional dependency relations. For example, the textual relation SUBJECT nsubjpass  born nmod:in  OBJECT is decomposed to a sequence of three tokens {  nsubjpass, born, nmod:in } , where    represents a left arrow. Note that we include directional dependency relations, because both the relation type and the direction are critical in determining the meaning of a textual relation. For example, the dependency relation  nmod:in  often indicates a location modifier and is thus strongly associated with location-related KB relations like place of birth . The direction also plays an important role. Without knowing the direction of the dependency relations, it is impossible to distinguish child of and parent of . An RNN with gated recurrent units (GRUs) (Cho et al., 2014) is then applied to consecutively process the sequence as shown in Figure 3. We have also explored more advanced constructs like attention, but the results are similar, so we opt for a vanilla RNN in consideration of model simplicity. Let  denote the function that maps a token x l to a fixed-dimensional vector, the hidden state vectors of the RNN are calculated recursively: h l = GRU (cid:0)  ( x l ) , h l  1 (cid:1) . We use global statistics in the relation graph to train the embedding model. Specifically, we model the semantics of a textual relation as its co-occurrence distribution of KB relations, and learn textual relation embedding to reconstruct the corresponding co-occurrence distributions. We use a separate GRU cell followed by softmax to map a textual relation embedding to a distribution over KB relations; the full model thus resembles the sequence-to-sequence architecture (Sutskever et al., 2014). Given a textual relation t i and its embedding h m , the predicted conditional probability of a KB relation r j is thus: p ( r j | t i ) = softmax ( GRU (  ( < GO > ) , h m )) j , (3) where () j denotes the j -th element of a vector, and < GO > is a special token indicating the start of decoding. The training objective is to minimize  = 1 |E| X i,j : p ( r j | t i ) > 0 (log p ( r j | t i )  log  p ( r j | t i )) 2 , (4) where E is the edge set of the relation graph. It is modeled as a regression problem, similar to GloVe (Pennington et al., 2014). Baseline. We also define a baseline approach where the unnormalized co-occurrence counts are directly used. The objective is to maximize:  0 = 1 P i,j n ij X i,j : n ij > 0 n ij log p ( r j | t i ) . (5) It also corresponds to local statistics based embedding, i.e., when the embedding model is trained on individual occurrences of relational facts with distant supervision. Therefore, we call it Lo cal R elation E mbedding (LoRE). Learned from global co-occurrence statistics of relations, our approach provides semantic matching information of textual and KB relations, which is often complementary to the information captured by existing relation extraction models. In this section we discuss how to combine them together to achieve better relation extraction performance. We follow the setting of distantly supervised relation extraction. Given a text corpus and a KB with relation set R , the goal is to find new relational facts from the text corpus that are not already contained in the KB. More formally, for each entity pair ( e, e 0 ) and a set of contextual sentences C containing this entity pair, a relation extraction model assigns a score E ( z | C ) to each candidate relational fact z = ( e, r, e 0 ) , r  R . On the 824 other hand, our textual relation embedding model works on the sentence level. It assign a score G ( z | s ) to each contextual sentence s in C as for how well the textual relation t between the entity pair in the sentence matches the KB relation r , i.e., G ( z | s ) = p ( r | t ) . It poses a challenge to aggregate the sentence-level scores to get a set-level score G ( z | C ) , which can be used to combine with the original score E ( z | C ) to get a better evaluation of the candidate relational fact. One straightforward aggregation is max pooling, i.e., only using the largest score max s  CG ( z | s ) , similar to the at-least-one strategy used by Zeng et al. (2015). But it will lose the useful signals from those neglected sentences (Lin et al., 2016). Because of the wrong labeling problem, mean pooling is problematic as well. The wrongly labeled contextual sentences tend to make the aggregate scores more evenly distributed and therefore become less informative. The number of contextual sentences positively supporting a relational fact is also an important signal, but is lost in mean pooling. Instead, we use summation with a trainable cap : G ( z | C ) = min ( cap, X s  CG ( z | s )) , (6) In other words, we additively aggregate the signals from all the contextual sentences, but only to a bounded degree. We simply use a weighted sum to combine E ( z | C ) and G ( z | C ) , where the trainable weights will also handle the possibly different scale of scores generated by different models:  E ( z | C ) = w 1 E ( z | C ) + w 2 G ( z | C ) . The original score E ( z | C ) is then replaced by the new score  E ( z | C ) . To find the optimal values for w 1 , w 2 and cap , we define a hinge loss:  Merge = 1 KKX k =1 max (cid:8) 0 , 1 +  E ( z  k )   E ( z + k ) (cid:9) , (8) where { z + k } K k =1 are the true relational facts from the KB, and { z  k } Kk =1 are false relational facts generated by replacing the KB relation in true relational facts with incorrect KB relations. In this experimental study, we show that GloRE can greatly improve the performance of several recent relation extraction models, including the previous best model on a standard dataset. Dataset. Following the literature (Hoffmann et al., 2011; Surdeanu et al., 2012; Zeng et al., 2015; Lin et al., 2016), we use the relation extraction dataset introduced in (Riedel et al., 2010), which was generated by aligning New York Times (NYT) articles with Freebase (Bollacker et al., 2008). Articles from year 2005-2006 are used as training, and articles from 2007 are used as testing. Some statistics are listed in Table 1. There are 53 target KB relations, including a special relation NA indicating that there is no target relation between entities. We follow the approach described in Section 3 to construct the relation graph from the NYT training data. The constructed relation graph contains 321,447 edges with non-zero weight. We further obtain a training set and a validation set from the edges of the relation graph. We have observed that using a validation set totally disjoint from the training set leads to unstable validation loss, so we randomly sample 300K edges as the training set, and another 60K as the validation set. The two sets can have some overlap. For the merging model (Eq. 8), 10% of the edges are reserved as the validation set. Relation extraction models. We evaluate with four recent relation extraction models whose source code is publicly available 3 . We use the optimized parameters provided by the authors. CNN+ONE and PCNN+ONE (Zeng et al., 2015): A convolutional neural network (CNN) is used to embed contextual sentences for relation classification. Multi-instance learning with at-least-one (ONE) assumption is used to combat the wrong labeling problem. In PCNN, piecewise max pooling is 3 https://github.com/thunlp/NRE 825 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Recall 0.4 0.5 0.6 0.7 0.8 0.9 1.0 P r e c i s i o n CNN+ATT CNN+ATT+GloRE 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Recall 0.4 0.5 0.6 0.7 0.8 0.9 1.0 P r e c i s i o n CNN+ONE CNN+ONE+GloRE 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Recall 0.4 0.5 0.6 0.7 0.8 0.9 1.0 P r e c i s i o n PCNN+ONE PCNN+ONE+GloRE Figure 4: Held-out evaluation: other base relation extraction models and the improved versions when augmented with GloRE. CNN+ATT and PCNN+ATT (Lin et al., 2016): Different from the at-least-one assumption which loses information in the neglected sentences, these models learn soft attention weights (ATT) over contextual sentences and thus can use the information of all the contextual sentences. PCNN+ATT is the best-performing model on the NYT dataset . Evaluation settings and metrics. Similar to previous work (Riedel et al., 2010; Zeng et al., 2015), we use two settings for evaluation: (1) Held-out evaluation, where a subset of relational facts in KB is held out from training (Table 1), and is later used to compare against newly discovered rela-0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Recall 0.4 0.5 0.6 0.7 0.8 0.9 1.0 P r e c i s i o n BASE BASE+(CNN+ONE) BASE+(CNN+ATT) BASE+(PCNN+ONE) BASE+GloRE Figure 6: Held-out evaluation: GloRE brings the largest improvement to BASE (PCNN+ATT), which further shows that GloRE captures useful information for relation extraction that is complementary to existing models. tional facts. This setting avoids human labor but can introduce some false negatives because of the incompleteness of the KB. (2) Manual evaluation, where the discovered relational facts are manually judged by human experts. For held-out evaluation, we report the precision-recall curve. For manual evaluation, we report P recision @ N , i.e., the precision of the top N discovered relational facts. Implementation. Hyper-parameters of our model are selected based on the validation set. For the embedding model, the mini-batch size is set to 128, and the state size of the GRU cells is 300. For the merging model, the mini-batch size is set to 1024. We use Adam with parameters recommended by the authors for optimization. Word embeddings are initialized with the 300-dimensional word2vec vectors pre-trained on the Google News corpus 4 . Early stopping based on the validation set is employed. Our model is implemented using Tensorflow (Abadi et al., 2016), and the source code is available at https://github.com/ ppuliu/GloRE . Existing Models + GloRE. We first show that our approach, GloRE, can improve the performance of the previous best-performing model, PCNN+ATT, leading to a new state of the art on the NYT dataset. As shown in Figure 5, when PCNN+ATT is augmented with GloRE, a consistent improvement along the precision-recall curve is observed. It is worth noting that although PCNN+ATT+GloRE seems to be inferior to PCNN+ATT when recall < 0 . 05 , as we will show via manual evaluation, it is actually due to false negatives. We also show in Figure 4 that the improvement brought by GloRE is general and not specific to PCNN+ATT; the other models also get a consistent improvement when augmented with GloRE. To investigate whether the improvement brought by GloRE is simply from ensemble, we also augment PCNN+ATT with the other three base models in the same way as described in Section 5. The results in Figure 6 show that pairwise ensemble of existing relation extraction models does not yield much improvement, and GloRE brings much larger improvement than the other models. In summary, the held-out evaluation results suggest that GloRE captures useful information for relation extraction that is not captured by these local statistics based models. LoRE v.s. GloRE. We compare GloRE with the baseline approach LoRE (Section 4) to show the advantage of normalization on global statistics. We use PCNN+ATT as the base relation extraction model. As shown in Figure 7, GloRE consistently outperforms LoRE. It is worth noting that LoRE can still improve the base relation extraction model when recall > 0 . 15 , further confirming Precision@ N 100 300 500 700 900 1000 PCNN+ATT 97.0 93.7 92.8 89.1 85.2 83.9 PCNN+ATT+LoRE 97.0 95.0 94.2 91.6 89.6 87.0 PCNN+ATT+GloRE 97.0 97.3 94.6 93.3 90.1 89.3 Table 2: Manual evaluation: false negatives from held-out evaluation are manually corrected by human experts. the usefulness of directly embedding textual relations in addition to sentences. Due to the incompleteness of the knowledge base, held-out evaluation introduces some false negatives. The precision from held-out evaluation is therefore a lower bound of the true precision. To get a more accurate evaluation of model performance, we have human experts to manually check the false relational facts judged by held-out evaluation in the top 1,000 predictions of three models, PCNN+ATT, PCNN+ATT+LoRE and PCNN+ATT+GloRE, and report the corrected results in Table 2. Each prediction is examined by two human experts who reach agreement with discussion. To ensure fair comparison, the experts are not aware of the provenance of the predictions. Under manual evaluation, PCNN+ATT+GloRE achieves the best performance in the full range of N . In particular, for the top 1,000 predictions, GloRE improves the precision of the previous best model PCNN+ATT from 83.9% to 89.3%. The manual evaluation results reinforce the previous observations from held-out evaluation. Table 3 shows two examples. For better illustration, we choose entity pairs that have only one contextual sentence. For the first example, PCNN+ATT predicts that most likely there is no KB relation between the entity pair, while both LoRE and GloRE identify the correct relation with high confidence. The textual relation clearly indicates that the head entity is ( appos ) a criminologist at ( nmod:at ) the tail entity. For the second example, there is no KB relation between the entity pair, and PCNN+ATT is indeed able to rank NA at the top. However, it is still quite confused by nationality , probably because it has learned that sentences about a person and a country with many words about profession (poet, playwright, and novelist) 827 Contextual Sentence Textual Relation PCNN+ATT Predictions LoRE Predictions GloRE Predictions [ Alfred Blumstein ] head , a criminologist at [ Carnegie Mellon University ] tail , called ... appos  criminologist nmod:at  NA (0.63) employee of (1.00) employee of (0.96) employee of (0.36) NA (0.00) NA (0.02) founder of (0.00) founder of (0.00) founder of (0.02) [ Langston Hughes ] head , the American poet, playwright and novelist, came to [ Spain ] tail to ... -nsubj  came to  NA (0.58) place of death (0.35) NA (0.73) nationality (0.38) NA (0.33) contain location (0.07) place lived (0.01) nationality (0.21) employee of (0.06) Table 3: Case studies. likely express the person's nationality. As a result, its prediction on NA is not very confident. On the other hand, GloRE learns that if a person came to a place, likely it is not his/her birthplace. In the training data, due to the wrong labeling problem of distant supervision, the textual relation is wrongly labeled with place of death and nationality a couple of times, and both PCNN+ATT and LoRE suffer from the training noise. Taking advantage of global statistics, GloRE is more robust to such noise introduced by the wrong labeling problem. Our results show that textual relation embedding trained on global co-occurrence statistics captures useful relational information that is often complementary to existing methods. As a result, it can greatly improve existing relation extraction models. Large-scale training data of embedding can be easily solicited from distant supervision, and the global statistics of relations provide a natural way to combat the wrong labeling problem of distant supervision. The idea of relation embedding based on global statistics can be further expanded along several directions. In this work we have focused on embedding textual relations, but it is in principle bene-ficial to jointly embed knowledge base relations and optionally entities. Recently a joint embedding approach has been attempted in the context of knowledge base completion (Toutanova et al., 2015), but it is still based on local statistics, i.e., individual relational facts. Joint embedding with global statistics remains an open problem. Compared with the size of the training corpora for word embedding (up to hundred of billions of tokens), the NYT dataset is quite small in scale. Another interesting venue for future research is to construct much larger-scale distant supervision datasets to train general-purpose textual relation embedding that can help a wide range of downstream relational tasks such as question answering and textual entailment. The authors would like to thank the anonymous reviewers for their thoughtful comments. This research was sponsored in part by the Army Research Laboratory under cooperative agreements W911NF09-2-0053 and NSF IIS 1528175. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.
method objective abstain objective abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method abstain abstain abstain method abstain objective result objective abstain abstain abstain objective objective other other other other other other other other abstain other other other other other other other other other other other method other other other abstain method other abstain abstain abstain abstain abstain abstain method abstain abstain objective method abstain abstain abstain abstain other abstain other abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method other abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain other result abstain method abstain abstain abstain method abstain method method abstain abstain abstain method method abstain abstain other abstain abstain abstain method abstain abstain abstain method method abstain method abstain abstain method abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain abstain other other other other