Knowledge Bases (KBs) require constant updating to reflect changes to the world they represent. For general purpose KBs, this is often done through Relation Extraction (RE), the task of predicting KB relations expressed in text mentioning entities known to the KB. One way to improve RE is to use KB Embeddings (KBE) for link prediction. However, despite clear connections between RE and KBE, lit-tle has been done toward properly unifying these models systematically. We help close the gap with a framework that unifies the learning of RE and KBE models leading to significant improvements over the state-of-the-art in RE. The code is available at https://github. com/billy-inn/HRERE . Knowledge Bases (KBs) contain structured information about the world and are used in support of many natural language processing applications such as semantic search and question answering. Building KBs is a never-ending challenge because, as the world changes, new knowledge needs to be harvested while old knowledge needs to be revised. This motivates the work on the Relation Extraction (RE) task, whose goal is to assign a KB relation to a phrase connecting a pair of entities, which in turn can be used for updating the KB. The state-of-the-art in RE builds on neural models using distant (a.k.a. weak) supervision (Mintz et al., 2009) on large-scale corpora for training. A task related to RE is that of Knowledge Base Embedding (KBE), which is concerned with representing KB entities and relations in a vector space for predicting missing links in the graph. Aiming to leverage the similarities between these tasks, Weston et al. (2013) were the first to show that combining predictions from RE and KBE models was beneficial for RE. However, the way in which they combine RE and KBE predictions is rather naive (namely, by adding those scores). To the best of our knowledge, there have been no systematic attempts to further unify RE and KBE, particularly during model training . We seek to close this gap with HRERE (Het-erogeneous REpresentations for neural Relation Extraction), a novel neural RE framework that learns language and knowledge representations jointly . Figure 1 gives an overview. HRERE 's backbone is a bi-directional long short term memory (LSTM) network with multiple levels of attention to learn representations of text expressing relations. The knowledge representation machinery, borrowed from ComplEx (Trouillon et al., 2016), nudges the language model to agree with facts in the KB. Joint learning is guided by three loss functions: one for the language representation, another for the knowledge representation, and a third one to ensure these representations do not diverge. In effect, this contributes to HRERE 's generalization power by preventing over-fitting by either model. We build on state-of-the-art methods for learning the separate RE and KBE representations and on learning tools that allow us to scale to a moderately large training corpus. (We use a subset of Freebase with 3M entities as our KB.) We validate our approach on an established benchmark against state-of-the-art methods for RE, observing not only that our base model significantly outperforms previous methods, but also the fact that jointly learning the heterogeneous representations consistently brings in improvements. To the best of our knowledge, ours is the first principled framework to combine and jointly learn heterogeneous representations from both language and knowledge for the RE task. Contributions. This paper describes and evaluates a novel neural framework for jointly learning Figure 1: Workflow of the proposed framework. representations for RE and KBE tasks that uses a cross-entropy loss function to ensure both representations are learned together, resulting in significant improvements over the current state-of-the-art for the RE task. Recent neural models have been shown superior to approaches using hand-crafted features for the RE task. Among the pioneers, Zeng et al. (2015) proposed a piecewise convolutional network with multi-instance learning to handle weakly labeled text mentions. Recurrent neural networks (RNN) are another popular architecture (Wu et al., 2017). Similar fast progress has been seen for the KBE task for representing entities and relations in KBs with vectors or matrices. Bordes et al. (2013) introduced the influential translation-based embeddings (TransE), while Yang et al. (2014) leveraged latent matrix factorization in their DistMult method. We build on ComplEx (Trouillon et al., 2016), which extends DistMult into the complex space and has been shown significantly better on several benchmarks. Weston et al. (2013) were the first to connect RE and KBE models for the RE task. Their simple idea was to train the two models independently and only combine them at inference time. While they showed that combining the two models is better than using the RE model alone, newer and better models since then have obviated the net gains of such a simple strategy (Xu and Barbosa, 2018). We propose a much tighter integration of RE and KBE models: we not only use them for prediction, but also train them together, thus mutually reinforcing one another. Recently, many methods have been proposed to use information from KBs to facilitate relation extraction. Sorokin and Gurevych (2017) considered other relations in the sentential context while predicting the target relation. Vashishth et al. (2018) utilized additional side information from KBs for improved RE. However, these methods didn't leverage KBE method to unify RE and KBE in a principled way. Han et al. (2018) used a mutual attention between KBs and text to perform better on both RE and KBE, but their method was still based on TransE (Bordes et al., 2013) which can not fully exploit the advantage of the information from KBs. The goal in the task of Relation Extraction is to predict a KB relation that holds for a pair of entities given a set of sentences mentioning them (or NA if no such relation exists). The input is a KB  with relation set R  , a set of relations of interest R , R  R  , and an automatically labelled training dataset D obtained via distant supervision. Given a sentence mentioning entities h, t , the output is a relation r  R that holds for h, t or the catch-all relation NA if no such r exists. customary, we denote a KB  with relation scheme R  as a set of triples T  = { ( h, r, t )  E   R   E  } , where E  is the set of entities of interest. Distant supervision exploits the KB to automatically annotate sentences in a corpus containing mentions of entities with the relations they participate in. Formally, a labeled dataset for relation extraction consists of fact triples { ( h i , r i , t i ) } Ni =1 and a multi-set of extracted sentences for each triple {S i } Ni =1 , such that each sentence s  S i mentions both the head entity h i and the tail entity t i . task is to estimate the probability of each relation in R  { NA } . Formally, for each relation r , we want to predict P ( r | h, t, S ) . In practice, the input set of sentences S can have arbitrary size. For the sake of computational effi-ciency, we normalize the set size to a fixed number T by splitting large sets and oversampling small ones. We also restrict the length of each sentence in the set by a constant L by truncating long sentences and padding short ones. We now go over the details of our framework outlined in Figure 1 for unifying the learning of the language and the knowledge representations used for relation extraction. In a nutshell, we use LSTM with attention mechanisms for language representation and we follow the approach of Trouillon et al. (2016) for KB embedding. Input Representation. For each word token, we use pretrained word embeddings and randomly initialized position embeddings (Zeng et al., 2014) to project it into ( d w + d p ) -dimensional space, where d w is the size of word embedding and d p is the size of position embedding. Sentence Encoder. For each sentence s i , we apply a non-linear transformation to the vector representation of s i to derive a feature vector z i = f ( s i ;  ) given a set of parameters  . In this paper, we adopt bidirectional LSTM with d s hidden units as f ( s i ;  ) (Zhou et al., 2016). Multi-level Attention Mechanisms. We employ attention mechanisms at both word-level and sentence-level to allow the model to softly select the most informative words and sentences during training (Zhou et al., 2016; Lin et al., 2016). With the learned language representation s L , the conditional probability p ( r |S ;  ( L ) ) is computed through a softmax layer, where  ( L ) is the parameters of the model to learn language representation. Following the score function  and training procedure of Trouillon et al. (2016), we can get the knowledge representations e h , w r , e t  C d k . With the knowledge representations and the scoring function, we can obtain the conditional probability p ( r | ( h, t );  ( G ) ) for each relation r : p ( r | ( h, t );  ( G ) ) = e  ( e h ,w r ,e t ) (cid:80) r (cid:48) R{ NA } e  ( e h ,w r (cid:48) ,e t ) where  ( G ) corresponds to the knowledge representations e h , w r , e t  C d k . Since NA /  R  , we use a randomized complex vector as w NA . As stated, this paper seeks an elegant way of connecting language and knowledge representations for the RE task. In order to achieve that, we use separate loss functions (recall Figure 1) to guide the language and knowledge representation learning and a third loss function that ties the predictions of these models thus nudging the parameters towards agreement. The cross-entropy losses based on the language and knowledge representations are defined as: JL =  1 NN (cid:88) i =1 log p ( r i |S i ;  ( L ) ) (1) JG =  1 NN (cid:88) i =1 log p ( r i | ( h i , t i );  ( G ) ) (2) where N denotes the size of the training set. Finally, we use a cross-entropy loss to measure the dissimilarity between two distributions, thus connecting them, and formulate model learning as minimizing JD : JD =  1 NN (cid:88) i =1 log p ( r  i |S i ;  ( L ) ) (3) where r  i = arg max r R{ NA } p ( r | ( h i , t i );  ( G ) ) . Based on Eq. 1, 2, 3, we form the joint optimization problem for model parameters as min  J = JL + JG + JD +  (cid:107)  (cid:107) 22 (4) where  =  ( L )   ( G ) . The knowledge representations are first trained on the whole KB independently and then used as the initialization for the joint learning. We adopt the stochastic gradient descent with mini-batches and Adam (Kingma and Ba, 2014) to update  , employing different learning rates lr 1 and lr 2 on  ( L ) and  ( G ) respectively 4.5 Relation Inference In order to get the conditional probability p ( r | ( h, t ) , S ; ) , we use the weighed average to combine the two distribution p ( r |S ;  ( L ) ) and p ( r | ( h, t );  ( G ) ) : p ( r | ( h, t ) , S ; ) =   p ( r |S ;  ( L ) ) +(1   )  p ( r | ( h, t );  ( G ) ) . where  is the combining weight of the weighted average. Then, the predicted relation  r is  r = argmax r R{ NA } p ( r | ( h, t ) , S ; ) . Datasets. We evaluate our model on the widely used NYT dataset (Riedel et al., 2010) by aligning Freebase relations mentioned in the New York Times Corpus. Articles from years 2005-2006 are used for training while articles from 2007 are used for testing. As our KB, we used a Freebase subset with the 3M entities with highest degree (i.e., participating in most relations). Moreover, to prevent the knowledge representation from memorizing the true relations for entity pairs in the test set, we removed all entity pairs present in the NYT. Evaluation Protocol: Following previous work (Mintz et al., 2009), we evaluate our model using held-out evaluation which approximately measures the precision without time-consuming manual evaluation. We report both Precision/Recall curves and Precision@N (P@N) in our experiments, ignoring the probability predicted for the NA relation. Moreover, to evaluate each sentence in the test set as in previous methods, we append T copies of each sentence into S for each testing sample. Word Embeddings: In this paper, we used the freely available 300-dimensional pre-trained word embeddings distributed by Pennington et al. (2014) to help the model generalize to words not appearing in the training set. Hyperparameter Settings: For hyperparameter tuning, we randonly sampled 10% of the training set as a development set. All the hyperparameters were obtained by evaluating the model on the development set. With the well-tuned hyperparameter setting, we run each model five times on the whole training set and report the average P@N. For Precision/Recall curves, we just select the results from the first run of each model. For training, learning rate on  ( L ) lr 1 5  10  4 learning rate on  ( K ) lr 2 1  10  5 size of word position embedding d p 25 state size for LSTM layers d s 320 input dropout keep probability p i 0 . Methods Evaluated. We study three variants of our framework: (1) HRERE -base : basic neural model with local loss JL only; (2) HRERE -naive : neural model with both local loss JL and global loss JG but without the dissimilarities JD ; (3) HRERE -full : neural model with both local and global loss along with their dissimilarities. We compare against two previous state-of-the-art neural models, CNN+ATT and PCNN+ATT (Lin et al., 2016). We also implement a baseline Weston based on the strategy following Weston et al. (2013), namely to combine the scores computed Relation Textual Mention base naive full contains Much of the middle east tension stems from the sense that shiite power is growing, led by Iran . with the methods stated in this paper directly without joint learning. Analysis. Figure 2 shows the Precision/Recall curves for all the above methods. As one can see, HRERE -base significantly outperforms previous state-of-the-art neural models and Weston over the entire range of recall. However, HRERE base performs worst compared to all other variants, while HRERE -full always performs best as shown in Figure 2 and Table 2. This suggests that introducing knowledge representation consistently results in improvements, which validates our motivating hypothesis. HRERE -naive simply optimizes both local and global loss at the same time without attempting to connect them. We can see that HRERE -full is not only consistently superior but also more stable than HRERE -naive when the recall is less than 0.1. One possible reason for the instability is that the results may be dominated by one of the representations and biased toward it. This suggests that (1) jointly learning the heterogeneous representations bring mutual benefits which are out of reach of previous methods that learn each independently; (2) connecting heterogeneous representations can increase the robustness of the framework. Case Study. Table 3 shows two examples in the testing data. For each example, we show the relation, the sentence along with entity mentions and the corresponding probabilities predicted by HRERE -base and HRERE -full . The entity pairs in the sentence are highlighted with bold formatting. From the table, we have the following observations: (1) The predicted probabilities of three variants of our model in the table match the observations and corroborate our analysis. (2) From the text of the two sentences, we can easily infer that middle east contains Iran and Henry Fonda was born in Omaha . However, HRERE -base fails to detect these relations, suggesting that it is hard for models based on language representations alone to detect implicit relations, which is reasonable to expect. With the help of KBE, the model can effectively identify implicit relations present in the text. (3) It may happen that the relation cannot be inferred by the text as shown in the last example. It's a common wrong labeled case caused by distant supervision. It is a case of an incorrectly labeled instance, a typical occurrence in distant supervision. However, the fact is obviously true in the KBs. As a result, HRERE -full gives the underlying relation according to the KBs. This observation may point to one direction of de-noising weakly labeled textual mentions generated by distant supervision. This paper describes an elegant neural framework for jointly learning heterogeneous representations from text and from facts in an existing knowledge base. Contrary to previous work that learn the two disparate representations independently and use simple schemes to integrate predictions from each model, we introduce a novel framework using an elegant loss function that allows the proper connection between the the heterogeneous representations to be learned seamlessly during training. Experimental results demonstrate that the proposed framework outperforms previous strategies to combine heterogeneous representations and the state-of-the-art for the RE task. A closer inspection of our results show that our framework enables both independent models to enhance each other. This work was supported in part by grants from the Natural Sciences and Engineering Research Council of Canada and a gift from Diffbot Inc.
abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain method method result objective abstain objective abstain other other other other other abstain other other other objective other other other other other other other other abstain abstain abstain abstain method abstain method method method method abstain method abstain method method abstain method abstain method other method abstain abstain other method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result other