Understanding the dynamics of international politics is important yet challenging for civilians. In this work, we explore unsupervised neural models to infer relations between nations from news articles. We extend existing models by incorporating shallow linguistics information and propose a new automatic evaluation metric that aligns relationship dynamics with manually annotated key events. As understanding international relations requires carefully analyzing complex relationships, we conduct in-person human evaluations with three groups of participants. Overall, humans prefer the outputs of our model and give insightful feedback that suggests future directions for human-centered models. Furthermore, our model reveals interesting regional differences in news coverage. For instance, with respect to US-China relations, Singaporean media focus more on strength-ening and purchasing, while US media focus more on criticizing and denouncing. In the context of growing globalization (Baylis et al., 2017), understanding complex international relations is increasingly relevant to our daily life. Yet this is a challenging task due to the inherently dynamic nature of international relations. As Kissinger famously said, America has no permanent friends or enemies, only interests. Staying informed becomes even harder in the continuous streams of information from news outlets and social media. This very availability of such information, however, opens up exciting opportunities for natural language processing to support individuals in understanding international relations. Supervised extraction has been incredibly useful at identifying pre-defined relations and events (Dodding-ton et al., 2004; Mintz et al., 2009) but fails to capture emerging or complex information needs. Topic models and neural models have been proposed to explore relations between entities without supervision (O'Connor et al., 2013; Chaney et al., 2016; Iyyer et al., 2016). In particular, Iyyer et al. (2016) introduces an unsupervised neural model for tracking relations between fictional characters, and this approach outperforms baselines from topic models and hidden Markov models. In this work, we incorporate linguistic insights into this model to track relation dynamics between nations from news articles. Our model reconstructs textual information in the embedding space using relation embeddings, as proposed in Iyyer et al. (2016). We integrate simple yet effective linguistic insights: verbal predicates often describe the relationship between entities, 1 while nouns and proper nouns provide the context of this relationship. For example, in U.S. denounces Russia for its interference in the 2016 election, denounce describes the relation, and election and interference provide the context. We show that this intuition leads the model to discover relation descriptors that are easier to interpret and less noisy. Evaluating these exploratory models for subjective tasks poses a challenge as there are no gold labels. Along with the model, we propose new approaches for evaluation. We introduce a quantitative metric which aligns pre-annotated key events with the temporal trends of relationships produced by the models. Since this task requires careful analysis of complex international relations, we conduct in-person user studies with NLP researchers and undergraduate students recruited from political science and linguistics courses. Both quantitative evaluation and human evaluation indicate that our model better rep-1 We use entities, countries, and nations interchangeably in this work. resents the dynamic relationships between nations than the prior model (Iyyer et al., 2016): 75.9% of participants preferred our model for finding natural language words describing international relations and 85.5% preferred temporal trends generated by our model. Finally, we qualitatively explore the context of relations provided by an attention-based mechanism and demonstrate a practical application of our model by studying regional differences in news coverage of relationships between two countries. We conclude with discussions on future directions for buildings models that can support individuals in navigating a large collection of news articles. Our code is available at https:// github.com/BoulderDS/LARN . We start by introducing our dataset of news articles, the shallow linguistic information that we extract, and our annotation of key events. News article collection. Our dataset is derived from the NOW corpus, the largest corpus of news articles that is available in full-text format. 2 The NOW corpus collects the news articles that are available on Google News in 20 English-speaking countries and thus include news articles that span a wide variety of topics, ranging from politics, to sports, to celebrity news from 23K media outlets. In this work, we consider the news articles in recent years, i.e., from January 2016 to June 2018, to facilitate human evaluation. We consider 12 nations (U.S., Russia, China, UK, Germany, Canada, France, India, Japan, Iran, Israel, and Syria) and the 66 nation pairs between them. To identify mentions of each nation in news articles, we manually construct a set of aliases for each nation to cover common abbreviations and the names of political leaders (e.g., Trump, Putin). On average, each nation has 3.5 aliases. We then use these aliases to find sentences that contain a pair of nations under consideration and obtain 1.2M sentences associated with 634K articles. Adding Shallow Linguistic Information. To incorporate shallow linguistic knowledge, we process the news article collection for each nation pairs to extract (1) verbal predicates and (2) nouns 2 The dataset can be obtained from https://www. and proper nouns from each sentence. Specifi-cally, we use a dependency parser to detect verbal predicates and their subjects and objects, and only include predicates for which both subjects and objects were found. For sentences with such predicates, we find nouns and proper nouns using part of speech tags. All data processing was done in spaCy (Honnibal and Montani, 2017) and full details can be found in the appendix. Key events annotation. The main goal of our model is to support the exploration of international relations, which is very challenging to evaluate. To derive quantitative evaluation measures and provide the basic context of international relations, we manually identify key events over the 30 months for 8 most frequently mentioned nation pairs (i.e. US-[China, Russia, UK, India, Canada, Japan, Syria], China-India) by reading through the top Google search results for each two countries and each month. 3 We identified roughly five key events per nation pair. For example, one key event for US-China relation is Chairman Xi's visit to the US in April 2017. A complete list of key events is shown in the appendix. Table 1 summarizes the data statistics. In this section, we formally introduce our model that builds on Relation Modeling Network ( RMN ) (Iyyer et al., 2016). Our main contribution is to integrate shallow linguistic information (i.e., verbal predicates and nouns/proper nouns) and identify the context of relations. The intuition behind our model follows RMN, i.e., inferring relation embeddings by reconstruct-3 reconstruct-3 This annotation is done by the first author and is thus inherently subjective. We add a robustness check in the appendix based on another set of independently annotated key events by the third author. Automatic evaluation shows a similar trend holds for the two sets of annotations. ing textual information in the embedding space. Specifically, we learn a fixed set of relation embeddings and use a convex combination of these relation embeddings to reconstruct information in sentences that mention both entities. Our main hypothesis is that relation information is often encoded in verbal predicates and we can obtain more interpretable and robust relations if we focus on predicates. For each pair of entities, we extract information from the predicates in sentences with both entities and reconstruct these predicates using shared relation embeddings. In addition, we use nouns to provide the context for the relations. We refer to our model as Linguistically Aware Relationship Networks ( LARN ). We now formally define our problem. The input of our model is a collection of news articles. For each entity pair e i , e j , we obtain a set of articles, A e i ,e j , containing at least one sentence mentioning both entities. We identify sentences where both e i and e j occur based on any alias associated with an entity (nation). We extract all the verbal predicates from these sentences in article a  A e i ,e j as { p a,e i ,e j 1 , . . . , p a,e i ,e j N } , as well as the proper nouns or nouns as { n a,e i ,e j 1 , . . . , n a,e i ,e j M } . Preprocessing details could be found in the appendix. We then use GloVe embeddings to represent these words, i.e., v p a,ei,ejk for p a,e i ,e j k and v n a,ei,ejk for n a,e i ,e j k (Pennington et al., 2014). These word embeddings are static in the entire learning process. Our model learns relation embeddings R  RK  d , where K is a hyperparameter for the number of relations and d is the dimension of the relation embedding as well as the word embedding. Following Iyyer et al. (2016), we obtain a list of natural language descriptors for each relation using the nearest neighbors of the relation embedding within the 500 most common predicates. 4 The model also provides (1) a probability distribution over relations for each article, (2) a probability distribution over nouns for each relation between each entity pair, and (3) an embedding for each entity. Figure 1 describes the overall architecture. We will describe the construction of v a label in  3.2, a 4 Refer to the appendix for a version that include all words. The relation descriptors from RMN required a manual filtering step in Iyyer et al. (2016), and become unintelligible without the 500 common words constraint. Note that our model produces intelligible descriptors even without the 500 most common predicates constraint. We compute the representation for each article to be reconstructed as the sum of bag-of-words embeddings. While Iyyer et al. (2016) considers all the words in a window in which both entities occur, we only consider predicates in the sentences where both entities show up: v a,e i ,e j label = N (cid:88) k =1 v p a,ei,ejk . v a,e i ,e j label depends on both the news article and the entity pair. In the rest of the paper, We omit e i , e j in the superscript here for simplicity, i.e., v a label . We represent each article as a weighted sum of relation embeddings. Assuming d a represents a weight vector over the K relation embeddings that sums to 1 for an article a with respect to a pair of entities ( e i , e j ) , we obtain r a as follows: r a = R (cid:62) d a . d a can also be thought of as a distribution over relations. This distribution over relations depends on entity pair ( v ae ), predicate information ( v ap ), and noun information ( v an ). While RMN simply takes the average of all words in the sentence, LARN focuses on verbal predicates and nouns to capture our intuition that predicates describe the main relations, whereas nouns provide background information to explain the relations. We now describe these three components in details. Representing predicates and entity pair. We follow RMN to construct embedding for each entity pair and for the predicates. The entity pair vector, v a e , simply adds the embedding of the two entities. The predicate vector, v ap , is equivalent to v a label except for word dropout during training, i.e., setting b ak to be 0 or 1 with a probability of 0.5. Representing context with nouns. To understand relations between an entity pair in a sentence, nouns should be considered in addition to predicates. For example, tariff is indicative of the relation between US and China in Originally, Trump favoured the simple imposition of a tariff on products from selected countries, especially China and Mexico, despite the seemingly positive predicate favour . As nouns are much more common than predicates (see Table 1) and not all of them are meaningful for understanding international relations, we employ a weighted sum of noun vectors. We use an attention mechanism (Conneau et al., 2017; Bahdanau et al., 2015) and consider each entity pair as a unique key to compute the attention weights, since the same noun can be interpreted differently across different entity pairs. To this end, we train an attention query embedding q e i ,e j for each entity pair separately. We further encode the temporal information by concatenating a one-hot vector t a that indicates the month when the article was published with the noun representation v n ak . This allows us to capture the shifts in a word's meaning over time. h n ak = tanh ( W proj  [ v n ak ; t a ]) ,  n ak = exp ( h n ak  q e i ,e j ) (cid:80) Mk (cid:48) =1 exp ( h n ak (cid:48)  q e i ,e j ) , v an = M (cid:88) k =1  n ak h n ak . Finally, we concatenate the three representations as the input to a feedforward network and pass to a softmax layer to create the weight vector d a . This d a is multiplied with the descriptor matrix R to get the final representation r a . Different from RMN , we do not consider temporal dependencies between time steps in our model because it is important to understand sudden shifts in international relations rather than assuming that the relations slowly evolve. We also found the temporal dependencies were not helpful empirically in our do-main but rather computationally expensive. The reconstruction objective pushes r a to resemble v alabel . Our formulation is identical to RMN : the loss function consists of a contrastive max-margin loss term, J , and an auxiliary loss term, X , to encourage unique relation embeddings. L (  ) = J (  ) + X (  ) , J (  ) = (cid:88) a A ei,ej (cid:88) v a label (cid:48) N max(0 , 1  r a  v a label || r a ||  || v a label || + r a  v a label (cid:48) || r a ||  || v a label (cid:48) || ) , X (  ) = (cid:107) RR (cid:62)  I (cid:107) , where v (cid:48) label is a randomly sampled negative example, N is a collection of them, and  is a hyperparameter for balancing two loss terms. In this section, we compare our model to RMN . For both models, we fixed the number of descriptors to 30 following Iyyer et al. (2016). As tracking dynamic international relations requires careful analysis, we hosted onsite user studies for quality control and in-person feedback. We first describe the model outputs, and then present both quantitative and qualitative evaluation results. Given a set of time-stamped news articles and a list of nations of interest, both models provide a set of relation descriptors, where each one defines a type of relation and a temporal trend analysis of these descriptors that shows how the relation evolves . Relation descriptors. Table 2 shows the top five and bottom five descriptors from LARN and RMN sorted by the average weights over all news articles related to the most frequently mentioned eight nation pairs. 5 By using predicates to describe relations, our descriptors seem to contain more semantically meaningful words. For instance, the top relation in RMN consists of exclusively noncontent words. Another interesting advantage of our model is that the five relations with the lowest 5 We focus on the top eight nation pairs to be consistent with human evaluation. weight have much higher weights in LARN than in RMN . This suggests that RMN tends to generate useless relations that do not show up in the data, while even bottom relations in LARN remain useful for describing the data. Temporal trends. We visualize the temporal trends of the most prominent relations between nation pairs. We further provide our annotated key events as the context to interpret these temporal trends. Figure 2 gives an example for US and China. The top three relations based on LARN are denounce, strengthen, and leave, 6 while the top three based on RMN are seem, range, and negotiate. We find our model generally aligns better with the key events: for instance, the denounce relation peaked around the time that Trump started issuing a series of tariffs based on 6 For space reasons, we only include the top word in the relation descriptor. We leverage our manually annotated key events to develop a novel automatic metric to evaluate how well the temporal trends from our model are aligned with key events. We define a change rate at each month  t as the weighted average of changes in relation weights in the top three relations:  t = (cid:88) i  top 3 relations w t,i  | d t,i  d t prev ,i | d t prev ,i , where d t,i is the average weight for the i -th relation in all the articles published at t , d t prev ,i is the average weight before t in a window of W preceding months ( 1 W (cid:80) Ww =1 d ( t  w ) ,i ), and weight w t,i is the normalized weight for top three descriptors ( d t,i (cid:80) j  top3relations d t,j ). Our results are robust to the choices of W , and we set it to 6 for presentation. We expect change rates to be greater when sig-nificant events happen in international politics. Figure 3 compares the change rate at months where key events occurred with other months for eight nation pairs for which we annotated key events. Both models present more abrupt changes when key events occurred. Unlike RMN , our model does not have temporal dependencies between relation distributions over time, 7 and thus has a higher discontinuity in general. However, even in relative terms, our model fluctuated more substantially than RMN when key events occurred. We also did a robustness check with another set of independently annotated key events and the results can be found in the appendix. This measure captures whether the model can detect the change points, but does not measure whether the model correctly captures the semantics of the key events, i.e., did a negative relation increase after hostile events such as war? To this end, we performed human evaluations. We hosted three human evaluations with participants from different demographics: undergraduate students from political science classes, graduate students from a computer science department (mostly in NLP), and undergraduate students taking a linguistics class. The total number of participants was 29, roughly equally divided among the three groups. The participants were shown outputs from RMN and LARN , and asked to choose the output that better aligns with their intuitions. Each participant answered about 10 questions and provided justification for their answers to each question, taking roughly 30 minutes to an hour. Table 3 summarizes the results of our human evaluations. Relation descriptor evaluation. The participants were shown a list of top five descriptors (as in Table 2) from two models, and prompted to select a set which adequately covers possible relationships that can occur between countries. 75.9% of participants preferred our model. Temporal trends evaluation. We showed temporal trends between nation pairs annotated with key events, one from RMN and the other from LARN (as in Figure 2). We asked them to evaluate whether the temporal trends accurately reflect the dynamics in nation-to-nation relationships. Each participant evaluated four randomly chosen nation pairs. The temporal trend from our model was preferred more frequently (85.5% of total responses). Nation pair matching. We designed a novel task where we showed the participants the other four temporal trends without annotated key events and asked them to match each trend with the corresponding nation pair from the four candidates, based on their world knowledge about nation-nation relations. Each participant did the matching twice, once for RMN and once for LARN . The participants found correct temporal trends for 45.2% of entity pairs for RMN , and 38.0% of entity pairs for LARN , when random pairing would yield 25.0%. The difference between two models here is not statistically significant. Most participants found this task very challenging, as they did not know much about the relationship between certain entity pairs (e.g., a participant said As an American, there's no way to know the relation between China and India.). Even political science students do not perform better than the other two groups. Discussion. Overall the output from our model is preferred by the participants. We found that political science students paid more attention to detail, took a longer time to finish, and were more ambivalent between the performance of the two models. For example, for temporal trends, they preferred our model for 71.4% of the examples, compared to other groups which preferred our model for 90% of the examples. They also preferred RMN 's relation descriptors slightly (42.9% selected our model) and commented that a few concepts from RMN , like infrastructure, supply, and value, are more concrete (e.g., a participant said I chose the left one ( RMN ), because it is easy to determine and remember the positive of items such as infrastructure, value, and supply. Those have more positive undertones, while it is easy to gauge negative sentiments with terrorism,' con-demn' and those.). As LARN encodes the background context specific to each nation pair separately, our relation descriptors do not contain such concrete concepts. In the next section, we will discuss contexts for relations, where these concepts appear in LARN . exFigure 4: Top contextual words for US-China's denounce relation derived from three approaches. Top : Showing word's avg. attention score in all denounce articles each month. Middle : Showing word's frequency in all denounce articles each month. Bottom : Showing word's average attention score multiplied by log( appearance ) in each month. All figures are normalized by the global maximum score in the figure. amine the context (nouns) associated with a relation between two nations based on an attention-based mechanism, which RMN does not handle. Second, we perform an in-depth analysis to show how our model can reveal regional differences in news coverage on the same topic. To help users better understand the inferred relations, we offer specific contexts that relate to the inferred relations based on the attention-based mechanism introduced in 3. For each relation, we find articles that place the most weight in that relation, and rank the nouns and proper nouns in those articles by their average attention score (i.e.,  n ai in  3.3). The top part of Figure 4 shows the top 10 words for the denounce relation in Figure 2a, the temporal trend for US-China relations. Since RMN does not support such mechanism, we show the most frequent nouns that occurred in the documents that mention both entities as a baseline (middle part). 8 We find that the attended nouns from our model are more informative than frequent nouns: tariff is the most attended word; words such as sanction, treaty, and pressure also show up, while the frequency baseline centers around words like China and president. As we noticed that the frequency baseline can capture alignment with key events, we incorporate attention score and frequency in the bottom part of Figure 4. This augmented version captures informative words (e.g., tariff, sanction, and missile) 9 and closely aligns with the key events. To further demonstrate the utility of our model, we explore regional differences in news coverage, as it is possible to build real knowledge by comparing perspectives from different social con-texts. 10 This also relates to the longstanding literature on framing in news coverage, i.e., se-lecting some aspects of a perceived reality and make them more salient to promote problem def-inition/interpretation (Entman, 1993; Chong and Druckman, 2007). We picked two countries, Singapore and US, to study US-China relations. 11 Using country source of media outlets in the NOW corpus, we found 10K articles from Singaporean media and 5.7K articles from US media on US and China. Table 4 shows the top five relations sorted by their absolute weight differences between US media outlets and Singaporean media outlets. Singaporean media more frequently use positive descriptors such as strengthen and purchase, whereas US media report negative relations such as denounce and criticize more frequently. Table 5 shows two example sentences from articles with the most weight in the denounce relation. Even though two media sources are focusing on events leading to the same type of relation, a reader who mainly consume news articles in Singapore would get a clearly different impression of US-China relations from those who read US news. 8 We had a comparison between the top figure and the middle figure in the human evaluation, but we found an error in visualization and thus focus on qualitative comparisons. 9 missile points us to another event related to US deploying missile defense system in South Korea, which also impacts US-China relations. 10 https://www.publicbooks.org/why-an-age-of-machine-learning-needs-the-humanities/ 11 China is not an English speaking country and is thus not in the NOW corpus. Singapore contained the most articles containing both US and China. US media: President Donald Trump is preparing to impose a package of $60 billion in annual tariffs against Chinese products, following through on a longtime threat that he says will punish China for intellectual property theft and create more American jobs. Singaporean media: It does not look like just a trade war , but rather the US is trying to bully China and the rest of the world in order to make China concede economic resources and development opportunities to the US and make the US forever big and strong. Prior work (Chambers et al., 2015; Choi et al., 2016; Rashkin et al., 2017) studied entity-entity relations in terms of positive and negative sentiments between them. Similarly, literature on relation extraction (Riedel et al., 2010; Gardner and Krishnamurthy, 2017; Elson et al., 2010; Srivas-tava et al., 2016) focused on pre-defined relations between a pair of entities in the database schema. In comparison, our work discovers descriptors for relations between entity pairs instead of finding entity pairs matching pre-defined relation schema. Topic modeling has been an important method to grasp important concepts from a large collection of documents in an unsupervised fashion (Blei et al., 2003; Das et al., 2015; Chang et al., 2009; Schein et al., 2015). Similar to our work, O'Connor et al. (2013) incorporates linguistic insights with topic models to identify event classes and detect conflicts. Our work additionally models the context of relations through nouns and focuses on exploring the potential of neural models. Most relevant to our work is Iyyer et al. (2016), which suggests RMN better capture dynamic relationships in literature than hidden Markov model (Gruber et al., 2007) and LDA (Blei et al., 2003). Recent work extended and applied RMN to other settings such as studying user roles in online communities (Wang et al., 2016; Frermann and Szarvas, 2017). Notably, Chaturvedi et al. (2017) suggests HMM with shallow linguistic features (i.e., frame net parses) and global constraints can outperform RMN for modeling relations in literature. In this work, we incorporate linguistic insights with RMN and apply it to news domain. Last but not least, researchers have studied the dynamics of media coverage from a wide range of perspectives, ranging from framing (Card et al., 2015; Field et al., 2018), to relationship between ideas (Tan et al., 2017), to quotes of politicians (Niculae et al., 2015; Tan et al., 2018; Leskovec et al., 2009). There is also signifi-cant effort for building event databases in political science (Leetaru and Schrodt, 2013), and assisting journalists with tools (Handler and O'Connor, 2017), and dating historical text (Niculae et al., 2014). We investigate the promise of unsupervised neural models for automatically inferring relations between nations. We find that incorporating shallow linguistic information is a simple yet effective strategy for deriving robust and interpretable relations. We develop a novel quantitative evaluation metric for understanding international relations and in-depth human evaluation also confirms the effectiveness of our model. We further show that our models can provide the background of relations using attention score and reveal regional differences for future studies on media framing. Meanwhile, our work suggests important future directions for using NLP technologies to support individuals in navigating a large collection of news articles. Our participants often find it challenging to infer information simply from temporal dynamics of our inferred relations based on natural language descriptors. It is thus important to incorporate human cognitive preferences in developing such models and provide narratives beyond words such as key events. Furthermore, different populations pay attention to different parts of information. We need to understand the diversity when developing NLP technologies for end users and provide helpful personalized hints to lower the barrier of benefiting from model outputs. We thank the anonymous reviewers, Dallas Card, and Mohit Iyyer for helpful feedback and discussion. We thank all the participants in our human evaluation from LING 3813 at Georgia Tech by Lelia Glass, students at X-lab at University of Washington, and undergraduate students at University of Colorado Boulder. We also thank Andy Baker and Sven Steinmo for recruiting students in their classes. Choi is supported by a Facebook Fellowship.
abstain objective objective method abstain result method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain result abstain objective method method method result objective abstain other method abstain abstain abstain method method method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain method abstain abstain method abstain method method abstain method abstain abstain method method abstain method abstain method abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain method abstain abstain method other method method method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain method abstain method abstain abstain other abstain method abstain other other abstain other method objective abstain other other method other other objective result objective result method result abstain abstain objective other other other other