Event-based, Recursive Neural Networks for the Extraction and Aggregation of International Alliance Relations
In this article, we explore how text mining can help producing reliable, high-level numerical data for multi-document relation extraction applications. We combine neural network techniques, information aggregation and visualization methods to extract event-based relations with a good tolerance to noise. We describe a search engine based on these methods, identifying the evolution of alliance and opposition relations between countries.
1 Introduction
Information and communication technologies have provided tools and methods to make the production of information more democratic. As a result, a vast amount of content is available, which arguably creates more noise than knowledge at the end of the day. Without hierarchical organization and contextualization, users may lack perspective to understand and assimilate the multiplicity of events that they come accross every day, and to link them to related events in the past.
In this context, we need tools for extracting, aggregating and visualizing knowledge in order to facilitate information analysis and access a variety of points of view. In this article, we take the example of the geopolitical alliances and their evolution; we extract relations from the texts and aggregate them in order to turn a set of very focused pieces of information into a broader knowledge and a bigger picture of a given situation.
The main contributions of this paper are:
• Relation extraction: we present an eventbased, recursive neural network approach for identifying alliance and opposition relations between countries, at sentence level, in English. We show that adapting the models
to the eventive nature of extracted relations helps existing recursive neural models. We use a precision-oriented cost function for a better later aggregation.
• Relation aggregation: we aggregate the sentence-level relations in a multi-document environment, in order to obtain a picture of the geopolitical situation and evolution on a specific subject defined by a user query (e.g., situation in Syria, nuclear proliferation, North Pole ownership). We cope with the inevitable amount of noise to produce reliable numerical data from texts.
2 System Overview and Dataset
2.1 Relation Extraction
We extract opposition (NEG), alliance (POS) or neutral (NEU) relations between two countries, explicitly expressed in news titles. Such relations are illustrated by the following examples:
(1) Japan and China agree to reduce tensions over Senkaku islands. → POS(Japan, China)
(2) Obama, Merkel warn Russia against intervention. → NEG(U.S.A., Russia), NEG(Germany, Russia), POS(U.S.A., Germany)
(3) Serbia prepares hero’s welcome for Putin. → POS(Serbia, Russia)
(4) Russia backs Ukraine rebel vote. → NEG(Russia, Ukraine)
(5) British teens detained en route to Syria, police say. → NEU(Great Britain, Syria)
Countries (arguments) can be designated by their actual names, the name of their capital or a representative person. Unlike for many knowledge extraction approaches, these relations are very changing, making the problem more difficult.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
2.2 Relation Aggregation
We then index these relations together with the text and the date of the corresponding articles. At query time, we retrieve all titles relevant to the query, extract the candidate entities and their relations. For each day, POS and NEG relations are aggregated and a tendency of this day is obtained, as well as a trend over time. A graphical visualization of this trend is then proposed to the user.
We chose to use only titles because we can consider that a title describes only one event, and that this event can be time-stamped by the document creation time, which is not true in article content. However, this comes with a drawback : titles in English often follow some different syntactic rules than traditional sentences, as in examples 2 (coordination expressed with only a comma) and 5 (contracted reported speech). Most parsers are not trained on title sentences and will then experience a higher error rate.
2.3 Dataset
Country, nationality, capital name and main leader lists were extracted from the linked data repository of the CIA World Factbook, and linguistic variations of these names were collected from DBPedia (Lehmann et al., 2013). Major unions of countries (United Nations, European Union, ASEAN, African Union...) and important groups (ISIS...) are also considered. All these elements were considered as entities potentially parts of an alliance/opposition relation.
We used a corpus of 3.3 millions of English news titles from about 20 different web news sites, between 2013 and early 2016 (3,000 documents/day in average). From this corpus, we randomly selected titles containing at least two entities. Each pair of entities is an instance of the classifier, which means that one sentence can lead to several instances. Furthermore, we added the heads of the nominal phrase containing these entities, as well as “person triggers” (man, minister, etc.) in order to refine the training. For example:
(6) EU, U.N. condemn Ukraine rebels’ election plans. → NEG(EU, rebels), NEG(UN, rebels), POS(EU, Ukraine), POS(UN, Ukraine), POS(EU,UN)
the word rebels is added to EU, U.N. and Ukraine, leading to five possible pairs (entities
from the same chunks are not considered).1 These extra entities are expected to help the model use all possible information.
Among all these automatically collected entities, we manually annotated 5172 pairs with polarities NEU (2837 instances), NEG (1473 instances) and POS (862 instances), coming from 2166 different titles (between 1 and 15 pairs in a title). Only explicit relations were annotated POS or NEG, and no geopolitical knowledge was used.
3 Relation Extraction at Sentence Level with Recursive Neural Models
We describe now the models used for classifying relations at sentence level, as described in Sec. 2.1.
3.1 Recursive Neural Networks
Recursive neural networks (Socher et al., 2011, 2013) have proved useful for tasks involving longdistance relations, such as semantic relation extraction (Hashimoto et al., 2013; Li et al., 2015).
As illustrated by Figure 1 (from Socher et al. (2013)), an n-gram is parsed into a binary tree and each word is represented as a K-dimension vector. The representation of each parent node is computed based on its immediate children, in a bottom-up fashion, using the following equations:
p1 = f(WL · b+WR · c); p2 = f(WL · a+WR · p1)
where f(·) is the function tanh, and WL and WR are K × K matrices. Vector p1 has the same dimension as word vectors and can then be used for computing its own parent node.
The root representation will then be used as features to classify each phrase with a softmax layer.
3.2 Our Models
Our full-tree model (FT) is based on this exact principle.2 For example, in Figure 2.a, the
1Note that we consider the mention of the country as representing its government position.
2Note that we do not report results with the tensor version of recursive NNs since they systematically decreased results.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
a. Full-Tree (FT) model b. Shortest-Path (SP) model
c. Initial dependency tree d. Dependency name conversion e. Event-Centered (EC) model
Figure 2: Different representations of the relation “Kurds agree oil deal with Iraq president”.
Figure 3: Other examples of conversion from dependency graph to EC trees.
parse tree obtained with Stanford parser (Klein and Manning, 2003) is binarized, and the vector for parent p is calculated from K-dimension word embeddings wwith and wEND:
p = f(WLEFT · wwith +WRIGHT · wEND + b)
where b is the 1×K bias vector. In a ShortestPath (SP) model, we use the same representation but we keep only nodes and branches that are part of the shortest dependency path (see Figure 2.c then 2.b, where the “oil deal” is pruned but “president” is added), as experimented by Xu et al. (2015) for LSTM networks.
Dependency tree recursive neural networks (Socher et al., 2014) generalizes to non-binary trees by considering as many composition matrices as necessary (until one matrix per dependency type). This will allow us to manipulate the tree as desired in our Event-Centered (EC) model described in the following section.
As we do not seek for general knowledge but for changing relationships, the entities themselves are replaced by generic *START* and *END* words, as well as proper nouns that are replaced by *PN*. This guarantees that the model will really learn the relational structure rather than the specific connection between two countries. This also gives us a way to represent the entities in the structure, without having to stick to the exact path between them (as in Socher et al. (2012)) or to add positional features (as in Zeng et al. (2014)).
3.3 Event-Centered Modeling
A known issue of recursive models is that their deep tree structure leads to a bias towards the topmost nodes (Kalchbrenner et al., 2014). We turn this problem into a desirable feature by pushing the main event governing the relation to the top of the structure.3 This section details this operation, illustrated by Figures 2.c, 2.d and 2.e.
At step ¬, dependencies from Stanford parser are renamed by the position of the dependent w.r.t the governor (LEFT is the dependent if before the governor in the sentence, RIGHT otherwise). Only the path between entities is kept. Then (step ), each branch node (agree, with and president) is extracted and converted into a leaf node, with an edge labeled MAIN going from this leaf node to the former position of the word. This label represents the fact that the word is the governor of its tree sibling in a dependency.
3Therefore, we do not use any node weighting nor normalization by the number of words underneath them, unlike in previous work (Socher et al., 2011, 2014).
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
We obtain a structure that we call EventCentered (EC), that can be processed by a dependency tree recursive model. For example, in Figure 2.e, the root vector r is calculated as follows:
r = f(WLEFT ·p1+WMAIN ·wagree+WRIGHT ·p2+b)
The top node of EC trees can have three children, all the others have only two. Only three matrices (WMAIN,WLEFT,WRIGHT) are created, keeping the number of parameters reasonable. Figure 3 illustrates two more examples of conversion.
As our relations are triggered by events, we expect at least one event word to be part of the dependency path between the two entities (we call an event word either a non-state verb or a noun tagged as act or event in WordNet (Miller, 1995)). It may not be the case, either because of a specific structure of the sentence or because of a parser error, mainly due to the speficities of titles or to traditional bad prepositional attachments. We now describe how we force the path to contain an event word.
If the dependency path from E1 to E2 does not contain any event word, we extend this path by collecting common ancestors until we meet one. For example, in the following sentence:
(7) Japan and China agree to reduce tensions over Senkaku islands
the path is turned from Japan–conj–China to agree—Japan—China.4
If no event word is found, then the closest event word to E1 and E2 (or as a last resort, the root node) is attached to the entities. This is the case in
(8) ISIS militants attack Syrian town near Turkey border
with the (erroneous) initial dependency graph
where attack is not in the path, but is added because attack is an event word. The graph is then centered on this event so that we obtain a tree as described before:
Using positional relations instead of real dependency names allows to twist the graph as desired to save what can be saved from an erroneous graph.
4Note that we used the uncollapsed version of Stanford dependency graphs in order to keep word vectors for prepositions. Not doing so decreases dramatically the results.
3.4 Precision-Oriented Cost Function
Our final application aggregates the sentencebased relations and uses a large corpus where the same relation on the same day will be expressed many times in different manners. This is why, faced with the traditional trade-off dilemma between precision and recall, we want to favor precision, for two reasons:
• Aggregating results with poor precision tends to lead to wrong aggregated values.
• The corpus redundancy will help to gather useful information even with a classifier with lower recall.
We then bias the classical cross-entropy error function by adding a penalty to false positives (i.e. POS or NEG erroneous predictions). For each relation r represented by the parameter vector θ, the error function is then
E(r; θ) = −λFP
( 3∑
k=1
tk log dk(r; θ)
)
where k represents each of the three classes POS, NEG and NEU, tk is the kth element of the multinomial target label distribution t for a relation, dk(r; θ) the 3-dimensional output of the softmax layer. Finally, λFP = 1 if the prediction is correct or false negative (NEU instead of POS or NEG) and λFP = α (with α > 1) if the prediction is a false positive, so that it is more costly. This can also be seen as an extra regularization parameter on the sum of all false positive predictions. In our experiments, we choose α such that it guarantees a precision higher than 80% on dev POS and NEG classes and maximizes the overall accuracy.
3.5 Extra Features
In addition to the top vector of the recursive model, we add the following features to the softmax classifier: average of the three words before E1 (resp. after E2); concatenation of the three words after E1 (resp. before E2); average of all event words (as defined above) between E1 and E2, as well as in the sentence. This leads to a total of 10×K extra parameters.
4 Classification Experiments and Results
4.1 Baselines
We evaluate this sentence-level extraction step and we compare our different models with:
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
SVM. A SVM two-step model using a sentiment lexicon and hand-crafted resources for alliance/opposition triggers, together with traditional positional, lexical and syntactic features. The first step classifies between neutral and nonneutral relations, while the second step distinguishes between positive and negative relations.
LSTM. A Long Short Term Memory (LSTM) model (Hochreiter and Schmidhuber, 1997) with softmax classifier.
TABARI. A rule engine (Schrodt and Yonamine, 2012) and a set of rules oriented toward inter-state behavior (Schrodt et al., 2008), composed of 284 relation labels that we mapped to either positive, negative and neutral relations. As the rules date back to 2008, we enriched the actor list by the entities from our corpus.
4.2 Implementation
We report results for two different configurations:
• Models FT, SP and EC with a softmax layer on the top node of the relation representation. The parameters are then the word embeddings (V × K parameters), the two or three K × K composition matrices, the softmax matrix (K × 3) and bias vectors (K each).
• The same models with the extra features described in Section 3.5 (10 × K parameters). These are called respectively FT+F, SP+F and EC+F, with or without precision-oriented cost function.
We pre-train word vectors with word2vec (Mikolov et al., 2013) applied to our entire article corpus. For all models, we use a development set for early stopping and hyper-parameter tuning (regularization weights, word vector size – set to K = 50 – and parameter α). We failed at finding an α meeting the requirements for the FT model. All these models are implemented with Theano (Bastien et al., 2012) and L-BFGS-B optimization, and tested on a test set of 453 relations.
4.3 Classification Results
Table 1 reports the results obtained by the described classifiers. Precision and recall values are
based on NEG and POS classes5, accuracy is on the three classes.
SP and EC models do not improve the results when only the top node of the structure is used as a feature for the softmax classifier. This can be explained by the fact that more sentence words are not represented at all in SP and EC structure, while FT contains all words between entities. However, with the extra content features, the EC accuracy is much higher, showing that a more focused tree structure can make the difference when the vocabulary gap is filled by representing missing words with simple average or concatenation.
Even the unbiased cost function leads to a precision much higher than recall. Our precisionoriented cost function with a tuned α value leads to a small precision gain with a small recall loss.
The system TABARI does not perform well on this dataset. This can be due to the fact that the dataset addresses a very large variety of subjects and geographic areas; TABARI gets better performances on specific topics with dedicated rules.
In what follows, we use the model EC+Fα=2.3.
5 Time-Aware Aggregation and Visualization
We describe here how we aggregate the sentencelevel relations. A Solr search engine (Kuć, 2013)
5NEU is much more represented and useless for aggregation; integrating it to prec. and rec. values would increase them artificially, making a NEU-only classifier the best one.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
is built from time-stamped alliance relations. Each sentence containing at least one relation is indexed with its stemmed content, the countries involved in relations, the relations themselves and the document creation date. A typical query is then composed of a few keywords representing the topic, a temporal interval (minimum of maximum dates) and zero or more country names on which the user wants to restrict relation extraction.
5.1 Aggregating Information
All sentences returned by Solr, without any number limit, are aggregated. For all pairs of considered countries, inside the same day d, if the numbers of POS and NEG relations between the two countries are respectively P (d) andN(d), then the weight for the pair and the day d is:
w(d) = log
( 1 + P (d)
1 +N(d) ) which leads to a number between−∞ and +∞, where w = 0 means that the relation is neutral, w < 0 that the relation is an opposition, andw > 0 that the relation is an alliance.
The noise is then reduced by a weighted mean smoothing over a temporal window of W days:
sw(d) =
∑n j=−n
w(d+j) abs(j)+1∑n
j=−n 1
abs(j)+1
where n is a half-window (n = floor(W2 )) For example, for a 5-day window:
sw(d) = w(d−2) 3 + w(d−1) 2 + w(d) + w(d+1) 2 + w(d+2)
3 1 3 + 1 2 + 1 + 1 2 + 1 3
5.2 Visualization
5.2.1 Time-Series Plot
For bilateral relations (field countries containing two items), we provide the users with a timeseries plot representing sw(d), and show them on demand the sentences which led to this value. Figure 4 shows an example of results concerning the relations between United States and Russia (“countries:United States AND Russia”) concerning the situation in Syria (“keywords:syria”).
5.2.2 Force-directed Graph
When zero or one country is specified by the user, we generate a graph of countries, where the distance between vertices (i.e., countries) reflects the opposition between the countries in a given time
span (generally, larger that a single day). The problem here is that our classifier provides a relation between two countries independently of the others; it is therefore very unlikely that these values can be used as distance values in the whole graph (especially, triangle inequality will not be respected). For this reason, we use the Barnes-Hut force-directed layout algorithm (Barnes and Hut, 1986), where a value between two vertices is considered as a repulsive force. The algorithm tries to minimize the energy spent to keep the whole graph together, and then to adapt the distances between vertices so that the rendering on a 2D plan is possible.
For this algorithm, we need to transform our weights sw(d) into positive numbers (repulsive forces). We also need to damp the noise and the variations of the weights that are introduced by the volume of data. This problem is illustrated by Table 2, where three countries A1, A2 and A3 are allies against another one B on a given topic and a given time span. The weights sw(d) of the three relations between allies are positive but vary between 1 and 3.5. As a result, A1 is finally closer to B (difference of 2) than to A2 (difference of 2.5), which is not desirable.
This kind of effects can be corrected by a “Sshaped”, logistic function, that models a level of saturation after an approximately exponential growth (or, in our case, decrease):
sw′(d) = 1− 1 1 + e−sw(d)
(9)
This function levels off high weights (both negative and positive), increases differences between positive and negative values and thus helps reducing noise without having to discretize values arbitrarily. Resulting numbers are all positive, between 0 (strong alliance, attraction in the graph) and 1 (strong opposition, repulsion in the graph).
Figure 5 shows an example of such graphs obtained by the force-directed algorithm, with the keyword syria.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Figure 4: Example of plot produced by the system for bilateral relations between United States and Russia on the query “Syria”, on a daily scale. The right frame shows sentences corresponding to the user-selected date (Oct. 2, 2015). Circled numbers have been manually added to the screenshot to make some peaks clearer. ¬ Russian military involvement grows in Syria (bad relation, sw(d) < 0);  US and Russian defence chiefs start discussing (better relation, sw(d) > 0); ® Russian airstrikes in Syria; ¯ Optimism after U.S. Secretary of State’s visit to Moscow.
Figure 5: Example of graph produced by the system for relations between different states on the query “syria”, for the year 2015. Edge colors indicate the kind of relation (from dark red for strong alliance to dark blue for strong opposition), and vertice colors reflects proximity of countries with each other.
5.3 Evaluation
Evaluating the relevance of trends suggested by a plot or a dynamic map is very subjective and would require a high level of expertise in each domain concerned by the tested queries. That is why we opted for a protocol that is at the same time more objective and easier to conduct:
1. We chose 14 queries with the following information: names of two countries (or unions of countries) involved in the relation, and an optional keyword-based thematic restriction. We selected queries having potentially a high density of extracted relations (e.g. North Korea vs. South Korea, or Russia vs. United Nations on “Syria”), as well as sparser topics (China vs. Japan on maritime affairs).
2. On the resulting plot, we selected up to 5 strong peaks — abs(sw(d)) > 1 — and 5 weak peaks — abs(sw(d)) ≤ 1. This led to a total of 97 relations to evaluate.
3. For each of these peaks, we estimated whether the polarity of the peak was relevant or not, from a different news article collection. This is still a heavy task, which explains the low number of tested instances.
The accuracy of strong peaks is 0.93, making them highly reliable. Accuracy of weak peaks is 0.752.
6 Related Work
TDT and Opinion Mining. The idea of automatically finding topically related material in streams of newswire data goes back to Topic Detection and Tracking evaluation campaigns (Allan, 2002). These campaigns and the follow-up research aim at developing algorithms for identifying and organizing events described in documents.
Our work contributes to this global research effort and explores the quantitative aspects of knowledge that can be extracted from textual documents. On this point, as well as on the topic
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
of alliance and opposition relations, this can be connected to sentiment analysis and opinion mining (Liu, 2012). Different opinions must be aggregated, not only for getting more information, as in multidocument information extraction, but also for balancing positive and negative statements and conclude about the average opinion.
Fewer research works focused on extracting opinion relations between entities, including Gottipati et al. (2013); Yang and Cardie (2013); Johansson and Moschitti (2013), that can be related to our work.
International Relations. TABARI (a.k.a. KEDS) is an open-source program extracting events concerning international relations from news texts (Schrodt and Yonamine, 2012). This rule-based system makes extensive use of handcrafted dictionaries and patterns. It has been widely used in the field of political science, for example for modeling dependencies in international relations networks (Hoff and Ward, 2004), as well as in the GDELT Project.6 Several versions have been proposed for studying different aspects of international relations (Levant, Turkey, Balkans, Political Instability Task Force, etc.) and a more general-purpose set of rules, CAMEO, has also been developed (Schrodt et al., 2008). We compare it to our approach, after a few necessary adaptations (Section 4.1). More recently, (Boschee et al., 2013) presented a proprietary system and compare to TABARI as well.
With very different approaches, O’Connor et al. (2013) describe an unsupervised model for extracting events between political actors, and Gerrish (2013) presents a time-series topic model of foreign affairs. Similar aspects with our work are the detection of conflicts, as well as the interest in the temporal evolution of these conflicts and a general approach for studying specific situations. However, the purpose and output (distributions of topical terms) are different.
Finally, Chambers et al. (2015) describe an identification of political sentiment between nations at the citizen level, by using a large Twitter dataset. Their work is more directly related to traditional sentiment analysis and does not rely specifically on geopolitical events, even if the observed trends can follow these events.
6http://gdeltproject.org
Neural Networks for Relation Extraction. Several neural network models have been used for sentiment analysis and relation extraction. Examples are convolutional networks (Zeng et al., 2014; Nogueira dos Santos et al., 2015), recurrent networks such as LSTM (Xu et al., 2015) or recursive networks (Socher et al., 2011, 2012, 2013). Li et al. (2015) shows that using tree structures, as do recursive models, helps in tasks where longdistance relations are involved, such as semantic relation extraction. The results described in this paper confirm this conclusion.
Our precision-oriented approach relies on linguistic variation and redundancy in a large amount of documents to ensure a good coverage (Yates, 2007). It is related to some works in web questionanswering (Dumais et al., 2002; M.Banko et al., 2002; Magnini et al., 2002), temporal information aggregation (Kessler et al., 2012) or opinion mining (Turney, 2002; Hu and Liu, 2004).
7 Conclusion & Perspectives
The advances in information extraction techniques and the increase of computing power make possible quite deep analysis of very large collections of texts. In this paper, we propose to take advantage of this progress to aggregate a large volume of extracted relations and to turn this knowledge into numerical data. We also propose a tool identifying the evolution of alliance and opposition relations between countries, on a specific subject.
Most information conveyed every day remains textual and unstructured. With such applications, a wide new range of knowledge can become available, in many different fields, which would bring high added value to the final users, in terms of aggregation, contextualization and hierarchical organization of information. We see our international relations example as a use case study of this effort.
We now intend to focus on reducing the necessary amount of supervision by using distant supervision, so that different subjects could be treated with less effort. Also, such study on country relationships should be multilingual in order to consider the cultural dimension as well.
This article is accompanied with supplementary material: the annotated corpus, the FT, SP and EC versions of all relations (JSON format), the script converting text into each of these versions, and the source for our recursive models.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
- Strengths: Useful modeling contribution, and potentially useful annotated data, for an important problem -- event extraction for the relationships between countries as expressed in news text.  - Weaknesses: Many points are not explained well in the paper.   - General Discussion:  This work tackles an important and interesting event extraction problem -- identifying positive and negative interactions between pairs of countries in the world (or rather, between actors affiliated with countries).  The primary contribution is an application of supervised, structured neural network models for sentence-level event relation extraction.  While previous work has examined tasks in the overall area, to my knowledge there has not been any publicly availble sentence-level annotated data for the problem -- the authors here make a contribution as well by annotating some data included with the submission; if it is released, it could be useful for future researchers in this area.  The proposed models -- which seem to be an application of various tree-structured recursive neural network models -- demonstrate a nice performance increase compared to a fairly convincing, broad set of baselines (if we are able to trust them; see below).  The paper also presents a manual evaluation of the inferred time series from a news corpus which is nice to see.  I'm torn about this paper.  The problem is a terrific one and the application of the recursive models seems like a contribution to this problem.  Unfortunately, many aspects of the models, experimentation, and evaluation are not explained very well.  The same work, with a more carefully written paper, could be really great.  Some notes:  - Baselines need more explanation.  For example, the sentiment lexicon is not explained for the SVM.                    The LSTM classifier is left highly unspecified (L407-409) -- there are multiple different architectures to use an LSTM for classification.  How was it trained?  Is there a reference for the approach?  Are the authors using off-the-shelf code (in which case, please refer and cite, which would also make it easier for the reader to understand and replicate if necessary)?  It would be impossible to replicate based on the two-line explanation here.    - (The supplied code does not seem to include the baselines, just the recursive NN models.  It's great the authors supplied code for part of the system so I don't want to penalize them for missing it -- but this is relevant since the paper itself has so few details on the baselines that they could not really be replicated based on the explanation in the paper.)  - How were the recursive NN models trained?  - The visualization section is only a minor contribution; there isn't really any innovation or findings about what works or doesn't work here.  Line by line:  L97-99: Unclear. Why is this problem difficult?  Compared to what? (also the sentence is somewhat ungrammatical...)  L231 - the trees are binarized, but how?  Footnote 2 --  the tensor version  - needs citation to explain what's being referred to.  L314: How are non-state verbs defined?                    Does the definition of  event word s here come from any particular previous work that motivates it?                    Please refer to something appropriate or related.  Footnote 4: of course the collapsed form doesn't work, because the authors aren't using dependency labels -- the point of stanford collapsed form is to remove prepositions from the dependeny path and instead incorporate them into the labels.  L414: How are the CAMEO TABARI categories mapped to positive and negative entries?  Is performance sensitive to this mapping?  It seems like a hard task (there are hundreds of those CAMEO categories....) Did the authors consider using the Goldstein scaling, which has been used in political science, as well as the cited work by O'Connor et al.?  Or is it bad to use for some reason?  L400-401: what is the sentiment lexicon and why is it appropriate for the task?  L439-440: Not clear.   We failed at finding an alpha meeting the requirements for the FT model.   What does that mean? What are the requirements? What did the authors do in their attempt to find it?  L447,L470:  precision and recall values are based on NEG and POS classes .  What does this mean?  So there's a 3x3 contingency table of gold and predicted (POS, NEU, NEG) classes, but this sentence leaves ambiguous how precision and recall are calculated from this information.  5.1 aggregations: this seems fine though fairly ad-hoc.  Is this temporal smoothing function a standard one?  There's not much justification for it, especially given something simpler like a fixed window average could have been used.  5.2 visualizations: this seems pretty ad-hoc without much justification for the choices.  The graph visualization shown does not seem to illustrate much.  Should also discuss related work in 2d spatial visualization of country-country relationships by Peter Hoff and Michael Ward.  5.3 L638-639:  unions of countries  isn't a well defined concept.  mMybe the authors mean  international organizations ?  L646-648: how were these 5 strong and 5 weak peaks selected?  In particular, how were they chosen if there were more than 5 such peaks?  L680-683: This needs more examples or explanation of what it means to judge the polarity of a peak.  What does it look like if the algorithm is wrong?                    How hard was this to assess?  What was agreement rate if that can be judged?  L738-740: The authors claim Gerrish and O'Connor et al. have a different  purpose and outputs  than the authors' work.  That's not right.  Both those works try to do both (1) extract time series or other statistical information about the polarity of the relationships between countries, and *also* (2) extract topical keywords to explain aspects of the relationships.  The paper here is only concerned with #1 and less concerned with #2, but certainly the previous work addresses #1.  It's fine to not address #2 but this last sentence seems like a pretty odd statement.  That raises the question -- Gerrish and O'Connor both conduct evaluations with an external database of country relations developed in political science ( MID , military interstate disputes).              Why don't the authors of this work do this evaluation as well?  There are various weaknesses of the MID data, but the evaluation approach needs to be discussed or justified.