Evaluating Creative Language Generation: The Case of Rap Lyric Ghostwriting
Language generation tasks that seek to mimic human ability to use language creatively are difficult to evaluate, since one must consider creativity, style, and other non-trivial aspects of the generated text. The goal of this paper is to develop evaluation methods for one such task, ghostwriting of rap lyrics, and to provide an explicit, quantifiable foundation for the goals and future directions of this task. Ghostwriting must produce text that is similar in style to the emulated artist, yet distinct in content. We develop a novel evaluation methodology that addresses several complementary aspects of this task, and illustrate how such evaluation can be used to meaningfully analyze system performance. We provide a corpus of lyrics for 13 rap artists, annotated for stylistic similarity, which allows us to assess the feasibility of manual evaluation for generated verse.
1 Introduction
Language generation tasks are often among the most difficult to evaluate. Evaluating machine translation, image captioning, summarization, and other similar tasks is typically done via comparison with existing human-generated “references”. However, human beings also use language creatively, and for the language generation tasks that seek to mimic this ability, determining how accurately the generated text represents its target is insufficient, as one also needs to evaluate creativity and style. We believe that one of the reasons such tasks receive little attention is the lack of sound evaluation methodology, without which no task is well-defined, and no progress can be made. The goal of this paper is to develop an evaluation
methodology for one such task, ghostwriting, or more specifically, ghostwriting of rap lyrics.
Ghostwriting is ubiquitous in politics, literature, and music. As such, it introduces a distinction between the performer/presenter of text, lyrics, etc, and the creator of text/lyrics. The goal of ghostwriting is to present something in a style that is believable enough to be credited to the performer. In the domain of rap specifically, rappers sometimes function as ghostwriters early on before embarking on their own public careers, and there are even businesses that provide written lyrics as a service1. The goal of automatic ghostwriting is therefore to create a system that can take as input a given artist’s work and generate similar yet unique lyrics.
Our objective in this work is to provide a quantifiable direction and foundation for the task of rap lyric generation and similar tasks through (1) developing an evaluation methodology for such models, and (2) illustrating how such evaluation can be used to analyze system performance, including advantages and limitations of a specific language model developed for this task. As an illustration case, we use the ghostwriter model previously proposed in exploratory work by Potash et al. (2015), which uses a recurrent neural network (RNN) with Long Short-Term Memory (LSTM) for rap lyric generation.
The following are the main contributions of this paper. We present a comprehensive manual evaluation methodology of the generated verses along three key aspects: fluency, coherence, and style matching. We introduce an improvement to the semi-automatic methodology used by Potash et al. (2015) that automatically penalizes repetitive text, which removes the need for manual intervention
1http://www.rap-rebirth.com/, http://www.precisionwrittens.com/ rap-ghostwriters-for-hire/
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
and enables a large-scale analysis. Finally, we build a corpus of lyrics for 13 rap artists, each with his own unique style, and conduct a comprehensive evaluation of the LSTM model performance using the new evaluation methodology. The corpus includes style matching annotation for select verses in dataset, which can form a gold standard for future work on automatic representation of similarity between artists’ styles. The resulting rap lyric dataset is publicly available from the authors’ website.
Additionally, we believe that the annotation method we propose for manual style evaluation can be used for other similar generation tasks. One example is ’Deep Art’ work in the computer vision community that seeks to apply the style of a particular painting to other images (Gatys et al., 2015; Li and Wand, 2016). One of the drawbacks of such work is a lack of systematic evaluation. For example, Li and Wand (2016) compared the results of the model with previous work by doing a manual inspection during an informal user study. The presence of a systematic formal evaluation process would lead to a clearer comparison between models and facilitate progress in this area of research. With this in mind, we make the interface used for style evaluation in this work available for public use.
Our evaluation results highlight the truly multifaceted nature of the ghostwriting task. While having a single measure of success is clearly desirable, our analysis shows the need for complementary metrics that evaluate different components of the overall task. Indeed, despite the fact that our test-case LSTM model outperforms a baseline model across numerous artists based on automated evaluation, the full set of evaluation metrics is able to showcase the LSTM model’s strengths and weaknesses. The coherence evaluation demonstrates the difficulty of incorporating large amounts of training data into the LSTM model, which intuitively would be desirable to create a flexible ghostwriting model. The style matching experiments suggest that the LSTM is effective at capturing an artist’s general style. However, this may indicate that it tends to form ‘average’ verses, which are then more likely to be matched with existing verses from an artist rather than another random verse from the same artist. Overall, the evaluation methodology we present provides an explicit, quantifiable foundation for
the ghostwriting task, allowing for a deeper understanding of the task’s goals and future research directions.
2 Related Work
In the past few years there has been a significant amount of work dedicated to the evaluation of natural language generation (Hastie and Belz, 2014), dealing with different aspects of evaluation methodology. However, most of this work focuses on simple tasks, such as referring expressions generation. For example, Belz and Kow (2011) investigated the impact of continuous and discrete scales for generated weather descriptions, as well as and simple image descriptions that typically consist of a few words (e.g., ”the small blue fan”). Perplexity is arguably the most popular automated metric for language models. Unfortunately, work such as Vougiouklis et al. (2016) shows that perplexity can sometimes be negatively correlated with human evaluation.
Previous work that explores text generation for artistic purposes, such as poetry and lyrics, generally uses either automated or manual evaluation. In terms of manual evaluation, Barbieri et al. (2012) have a set of annotators evaluate generated lyrics along two separate dimensions: grammar and semantic relatedness to song title. The annotators rated the dimensions with scores 1-3. A similar strategy was used by Gervás (2000), where the author had annotators evaluate generated verses with regard to syntactic correctness and overall aesthetic value, providing scores in the range 1- 5. Wu et al. (2013) had annotators determine the effectiveness of various systems based on fluency as well as rhyming.
Some heuristic-based automated approaches have also been used. For example, Oliveira et al. (2014) use a simple automatic heuristic that awards lines for ending in a termination previously used in the generated stanza. Malmi et al. (2015) evaluate their generated lyrics based on the verses’ rhyme density, on the assumption that a higher rhyme density means better lyrics.
Note that none of the work cited above provide a comprehensive evaluation methodology, but rather focus on certain specific aspects of generated verses, such as rhyme density or syntactic correctness. Moreover, the methodology for generating lyrics, proposed by the various authors, in-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
fluences the evaluation process. For instance, Barbieri et al. (2012) did not evaluate the presence of rhymes because the model was constrained to produce only rhyming verses. Furthermore, none of the aforementioned works implement models that generate complete verses at the token level (including verse structure), which is the goal of the models we aim to evaluate. In contrast to previous approaches that evaluate whole verses, our evaluation methodology uses a fine-grained, lineby-line scheme, which makes it easier for human annotators, as they no longer need to evaluate the whole verse at once. In addition, despite the fact the each line is annotated using a discrete scale, our methodology produces a continuous numeric score for the whole verse, enabling better comparison.
3 Dataset
For our evaluation experiments, we selected the following list of artists in four different categories: three top-selling rap artists(Eminem, Jay Z, Tupac), artists with the largest (Aesop Rock, GZA, Sage Francis) and the smallest (DMX, Drake) vocabulary, and best classified artists from Hirjee and Brown (2010b) (Fabolous, Nototious B.I.G., Lil’ Wayne).
We collected all available songs from the above artists from the site The Original Hip-Hop (Rap) Lyrics Archive - OHHLA.com - Hip-Hop Since 19922. We removed the metadata, line repetiton markup, and chorus lines, and tokenized the lyrics using the NLTK library (Bird et al., 2009). Since the preprocessing was done heuristically, the resulting dataset may still contain some text that is not actual verse, but rather dialogue or chorus lines. We therefore filter out all verses that are shorter than 20 tokens. Statistics of our dataset are shown in Table 1. We have made the dataset publicly available3.
4 Evaluation Methodology
We believe that adequate evaluation for the ghostwriting task requires both manual and automatic approaches. The automated evaluation methodology enables large-scale analysis of the generated verse. However, given the nature of the task, the automated evaluation is not able to assess certain critical aspects of fluency and style, such as the
2http://www.ohhla.com/ 3Left blank for review.
vocabulary, tone, and themes preferred by a particular artist. In this section, we present a manual methodology for evaluating these aspects of the generated verse, as well as an improvement to the automatic methodology proposed by Potash et al. (2015).
4.1 Manual Evaluation
We have designed two annotation tasks for manual evaluation. The first task is to determine how fluent and coherent the generated verses are. The second task is to evaluate manually how well the generated verses match the style of the target artist.
Fluency/Coherence Evaluation Given a generated verse, we ask annotators to determine the fluency and coherence of the lyrics. Even though our evaluation is for systems that produce entire verses, we follow the work of Wu (2014) and annotate fluency, as well as coherence, at the line level. To assess fluency, we ask to what extent a given line can be considered a valid English utterance. Since a language model may produce highly disjointed verses as it progresses through the training process, we offer the annotator three options for grading fluency: strongly fluent, weakly fluent, and not fluent. If a line is disjointed, i.e., it is only fluent in specific segments of the line, the annotators are instructed to mark it as weakly fluent. The grade of not fluent is reserved for highly unintelligible text.
To assess coherence, we ask the annotator how well a given line matches the preceding line. That is, how believable is it that these two lines would follow each other in a rap verse. We offer the annotators the same choices as in the fluency evaluation: strongly coherent, weakly coherent, and not coherent. During the training process, a language model may output the same line repeatedly. We account for this in our coherence evaluation by defining the consecutive repetition of a line as not coherent. This is important to define because the line on its own may be strongly fluent, however, a coherent verse cannot consist of a single fluent line repeated indefinitely.
Style Matching The goal of the style matching annotation is to determine how well a given verse captures the style of the target artist. In this annotation task, a user is presented with an evaluation verse and asked to compare it against four other verses. The goal is to pick the verse that is written in a similar style. One of the four choices
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Artist Verses Unique Vocab Vocab Richness Avg Len Stdev Len Max Len Tupac 660 5776 7.1 117 83 423 Aesop Rock 549 11815 14.8 140 139 1039 DMX 819 5593 5.3 125 82 552 Drake 665 6064 7.0 128 112 1057 Eminem 1429 12393 6.2 136 105 931 Fabolous 892 8304 7.4 122 91 662 GZA 287 6845 15.9 145 102 586 Jay Z 1245 9596 6.7 111 81 842 Lil’ Wayne 1564 10848 5.5 124 101 977 Nototious B.I.G. 426 5465 10.2 120 88 557 Sage Francis 570 8082 11.9 114 112 645 Kanye West 851 7007 7.6 105 109 2264 Kool Keith 1471 13280 7.4 118 85 626 Too Short 1259 7396 4.3 134 123 1411
Table 1: Rap lyrics dataset statistics. Vocabulary richness measures how varied an artist’s vocabulary is, computed as the total number of words divided by vocabulary size.
is always a verse from the same artist that was used to generate the verse being evaluated. The other three verses are chosen from the remaining artists in our dataset. Each verse is evaluated in this manner four times, each time against different verses, so that it has the chance to get matched with a verse from each of the remaining twelve artists. The generated verse is considered stylistically consistent if the annotators tend to select the verse that belongs to the target artist. To evaluate the difficulty of this task, we also perform style matching annotation for authentic verse, in which the evaluated verse is not generated, but rather is an actual existing verse from the target artist. 4
4.2 Automated Evaluation
The automated evaluation we describe below attempts to capture computationally the dual aspects of “unique yet similar” in a manner originally proposed by Potash et al. (2015).
Uniqueness of Generated Lyrics We use a modified tf-idf representation for verses, and calculate cosine similarity between generated verses and the verses from the training data to determine novelty (or lack thereof). In order to more directly penalize generated verses that are primarily the reproduction of a single verse from the training set, we calculate the maximum similarity score across all training verses. That is, we do not want generated verses that contain text from a single training verse, which in turn rewards generated verses that draw from numerous training verses.
4We have made the annotation interface available on (https://github.com/placeholder).
Stylistic Similarity via Rhyme Density of Lyrics We use the rhyme density method proposed by Hirjee and Brown (2010a) to evaluate how well the generated verse models an artist’s style. The point of an effective system is not to produce arbitrary rhymes: it is to produce rhyme types and rhyme frequency similar to the target artist. Distilling the various rhyme metrics offered by the tool, we focus on rhyme density, which is defined as the number of rhymed syllables divided by the total number of syllables (Hirjee and Brown, 2010a). Certain artists distinguish themselves by having more complicated rhyme schemes, such as the use of internal5 or polysyllabic rhymes6. Rhyme density is able to capture this in a single metric, since the tool we use is able to detect these various forms of rhymes. Moreover, as was shown in Wu (2014), the inter-annotator agreement (IAA) for manual rhyme detection is low (the highest IAA was only 0.283 using a two-scale annotation scheme), which is expected due to the subjective nature of the task. Therefore, an objective automatic methodology is desirable. Since this tool is trained on a distinct corpus of lyrics, it can provide a ”uniform” experience and give an impartial and objective score.
However, the rhyme detection tool is not designed to deal with highly repetitive text, which the LSTM model produces often in the early stages of training. Since the same phoneme is repeated (because the same word is repeated), the
5e.g. “New York City gritty committee pity the fool” and “How I made it you salivated over my calibrated”
6e.g. “But it was your op to shop stolen art/Catch a swollen heart form not rolling smart”.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
rhyme detection tool generates a false positive. Potash et al. (2015) deal with this by manually inspecting the rhyme densities of verses generated in the early stages of training to determine if a generated verse should be kept for the evaluation procedure. Unfortunately, This approach is clearly not scalable, as it is not fully automatic.
In order to fully automate this method, we propose to handle highly repetitive text by weighting the rhyme density of a given verse by its entropy. More specifically, for a given verse, we calculate entropy at the token level and divide by total number of tokens in that verse. Verses with highly repetitive text will have a low entropy, which results in down-weighting the rhyme density of verses that produce false positive rhymes due to their repetitive text.
Merging Uniqueness and Similarity Since ghostwriting is a balancing act of the two opposing forces of textual uniqueness and stylistic similarity, we want a low correlation between rhyme density (stylistic similarity) and maximum verse similarity (lack of textual uniqueness). However, our goal is not to have a high rhyme density, but rather to have a rhyme density similar to the target artist, while simultaneously keeping the maximum similarity score low. As the model overfits the training data, both the value of maximum similarity and the rhyme density will increase, until the model generates the original verse directly. Therefore, our goal is to evaluate the value of the maximum similarity at the point where the rhyme density has the value of the target artist. In order to accomplish this, we follow Potash et al. (2015) and plot the values of rhyme density and maximum similarity obtained at different points during model training. We use regression lines for these points to identify the value of the maximum similarity line at the point where the rhyme density line has the value of the target artist. We give more detail below.
5 Lyric Generation Experiments
The main generative model we use in our evaluation experiments is an LSTM. Similar to Potash et al. (2015), we use an n-gram model as a baseline system for automated evaluation. We refer the reader to the original work for a detailed description. After every 100 iterations of training7 the
7Training is done in batches with two verses per iteration. Therefore, since each artist has a different training set, the number of iterations that constitutes a full epoch also varies.
LSTM model generates a verse. For the baseline model, we generate five verses at values 1-9 for n. We see a correspondence between higher n and higher iteration: as both increase, the models become more ‘fit’ to the training data.
For the baseline model, we use the verses generated at different n-gram lengths (n ∈ {1, ..., 9}) to obtain the values for regression. At every value of n, we take the average rhyme density and maximum similarity score of the five verses that we generate to create a single data point for rhyme density and maximum similarity score, respectively.
To enable comparison, we also create nine data points from the verses generated by the LSTM as follows. A separate model for each artist is trained for a minimum of 16,400 iterations. We take the verses generated every 2,000 iterations, from 0 to 16,000 iterations, giving us nine points. The averages for each point are obtained by using the verses generated in iterations ±x, x ∈ {100, 200, 300, 400} for each interval of 2,000.
6 Results
6.1 Fluency/Coherence
In order to fairly compare the fluency/coherence of verses across artists, we use the verses generated by each artist’s model at 16,000 iterations. We apply the fluency/coherence annotation methodology from Section 4.1. Each line is annotated by two annotators. Annotation results are shown in Figure 1 and Figure 2. For each annotated verse, we report the percentage of lines annotated as strongly fluent, weakly fluent, and not fluent, as well as the corresponding percentages for coherence. We convert the raw annotation results into a single score for each verse by treating the labels “strongly fluent”, “weakly fluent”, and “not fluent” as numeric values 1, 0.5, and 0, respectively. Treating each annotation on a given line separately, we calculate the average numeric rating for a given verse: Fluency = #sf+0.5#wf#a where #sf is the number of times any line is labeled strongly fluent, #wf is the number of times any line is labeled weakly fluent, and #a is the total annotations provided for a verse, which is equal to the number of lines × 2. Coherence is calculated in a similar manner. Raw inter-annotator agreement (IAA) for fluency annotation was 0.67. For coherence annotation, the IAA was 0.43. We believe coherence has a lower agreement because it is more
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
587
588
589
590
591
592
593
594
595
596
597
598
599
semantic, as opposed to syntactic, in nature, causing it to be more subjective. Note that while the agreement is relatively low, it is expected, given the subjective nature of the task. For example, Wu (2014) report similar agreement values for the fluency annotation they perform.
Fluency evaluation
Figure 1: Percentage of lines annotated as strongly fluent, weakly fluent, and not fluent. The numbers above the bars reflect the total score of the artist (higher is better). The resulting mean is 0.723 and the standard deviation is 0.178.
Coherence evaluation
Figure 2: Percentage of lines annotated as strongly coherent, weakly coherent, and not coherent. The numbers above the bars reflect the total score of the artist (higher is better). The resulting mean is 0.713 and the standard deviation is 0.256.
6.2 Style Matching
We performed style-matching annotation for the verses generated at iterations 16,000–16,400 for each artist. For the experiment with authentic verses, we randomly chose five verses from each artist, with a verse length of at least 40 tokens. Each page was annotated twice, by native Englishspeaking rap fans. The results of our style matching annotations are shown in Table 2. We present two different views of the results. First, each annotation for a page is considered separately and
we calculate: Match% = #m#a , where #m is the number of times, on a given page, the chosen verse actually came from the target artist, and #a is the total number of annotations done. For a given artist, five verses were evaluated, each verse appeared on four separate pages, and each page is annotated twice, so #a is equal to 40. Since in each case (i.e., page) the classes are different, we cannot use Fleiss’s kappa directly. Raw agreement for style annotation, which corresponds to the percentage of times annotators picked the same verse (whether or not they are correct) is shown in the column ’Raw agreement %’ in Table 2.
We also report annotators’ joint ability to guess the target artist correctly, which we compute as follows: MatchA% =
#mA #sA , where #sA is the number of times the annotators agreed on a verse on the same page, and #mA is the number of times that the agreed upon verse is from the target artist.
6.2.1 Artist Confusion
The results of style-matching annotation also provides us with an interesting insight into the similarity between two artists’ styles. This is captured by the confusion between two artists during the annotation of the pages with authentic verses, which is computed as follows:
Confusion(a, b) = #c(a, b) + #c(b, a)
#p(a, b) + #p(b, a) (1)
where #p(a, b) is the number of times a verse from artist a is presented for evaluation and a verse
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Artist Authentic Generated Match% MatchA% Raw agreement % Match% MatchA% Raw agreement % Tupac 35.0 50.0 40.0 45.0 57.1 35.0 Aesop Rock 30.0 25.0 40.0 37.5 100.0 10.0 DMX 40.0 71.4 35.0 27.5 30.0 50.0 Drake 32.5 44.4 45.0 37.5 40.0 25.0 Eminem 12.5 00.0 50.0 35.0 50.0 30.0 Fabolous 25.0 12.5 40.0 45.0 50.0 40.0 GZA 52.5 72.7 55.0 32.5 22.2 45.0 Jay Z 35.0 42.9 35.0 22.5 22.2 45.0 Lil’ Wayne 27.5 22.2 45.0 37.5 57.1 35.0 Notorious B.I.G. 25.0 0.00 35.0 27.5 33.3 30.0 Sage Francis 52.5 66.7 45.0 22.5 16.7 30.0 Average 33.4 37.1 42.3 33.6 43.5 34.1
Table 2: The percentage of correct matches and the inter-annotator agreement in style matching evaluation
from artist b is shown as one of four choices; #c(a, b) is the number of times the verse from artist b was chosen as the matching verse. The resulting confusion matrix is presented in Figure 3. We intend for this data to provide a gold standard for future experiments that would attempt to encode the similarity of artists’ styles.
6.3 Automated Evaluation
The results of our automated evaluation are shown in Table 3. For each artist, we calculate their average rhyme density across all verses. We then use this value to determine at which iteration this rhyme density is achieved during generation (using the regression line for rhyme density). Next, we use the maximum similarity regression line to determine the maximum similarity score at that iteration. Low maximum similarity score indicates that we have maintained stylistic similarity while producing new, previously unseen lyrics.
Note the presence of negative numbers in Table 3. The reason is that in the beginning of training (in the LSTM’s case) and at a low ngram length (for the baseline model), the models actually achieved a rhyme density that exceeded the artist’s average rhyme density. As a result, the rhyme density regression line hits the average rhyme density on a negative iteration.
7 Discussion
In order to better understand the interaction between the four metrics we have introduced in this paper, we examined correlations between different measures of quality for generated verse (see Table 4a). The lack of strong correlation supports the notion that different aspects of verse quality should be addressed separately. Moreover, the metrics
are in fact complementary. Even the measures of fluency and coherence, despite sharing a similar goal, have a relatively low correlation of 0.4. Such low correlations emphasize our contribution, since other works (Barbieri et al., 2012; Wu, 2014; Malmi et al., 2015) do not provide a comprehensive evaluation methodology, and evaluate just one or two particular aspects. For example, Wu (2014) evaluated only fluency and rhyming, and Barbieri et al. (2012) evaluated only syntactic correctness and semantic relatedness to the title, whereas we present complementary approaches for evaluating different aspects of the generated verses.
Interestingly, the number of verses a rapper has in our dataset has a strong negative correlation with coherence score (cf. Table 4b). This can be explained by the following consideration: on iteration 16,000, the model for the authors with the smaller number of verses has seen the same verses more times than the model trained on a larger number of verses. Therefore, it is easier for the former to produce more coherent lyrics since it saw more of the same patterns. As a result, models trained on a larger number of verses have a lower coherence score. For example, Lil’ Wayne has the most verses in our data, and correspondingly, the model for his verse has the worst coherence score. Note that the fluency score does not have this negative correlation with the number of verses. Based on our evaluation, 16,000 iterations is enough to learn a language model for any artist that produces fluent lines, regardless of training set size. However, these lines will not necessarily form a coherent verse if the artist has a large number of verses.
As can be seen from Table 2, the Match% score suggests that the LSTM-generated verses are able to capture the style of the artist as well as the orig-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Artist Avg Rhyme Density Baseline LSTMSimilarity N-gram Similarity iteration Tupac 0.302 0.024 −2 0.065 −3168 Aesop Rock 0.349 0.745 7 0.460 12 470 DMX 0.341 0.663 6 0.431 8271 Drake 0.341 0.586 5 0.519 9949 Eminem 0.325 0.337 3 0.302 8855 Fabolous 0.360 1.353 14 0.569 14 972 GZA 0.280 0.520 4 0.616 14 939 Jay Z 0.365 0.499 5 0.463 15 147 Lil’ Wayne 0.362 0.619 6 0.406 9249 Notorious B.I.G. 0.383 0.701 7 0.428 3723 Sage Francis 0.415 0.764 8 0.241 −187 Average - 0.619 - 0.409 -
Table 3: The results of the automated evaluation. The bold indicates the system with a lower similarity at the target rhyme density.
Coherence Fluency Similarity Matching Coherence 1.000 0.398 0.102 -0.285 Fluency 0.398 1.000 0.137 -0.276 Similarity 0.102 0.137 1.000 0.092 Matching -0.285 -0.276 0.092 1.000
(a) Correlation between the four metrics we have developed: Coherence, Fluency, similarity score based on automated evaluation (Similarity), and Style Matching (Matching).
Coherence Fluency Similarity Matches Verses -0.509 -0.084 0.133 0.111 Tokens -0.463 -0.229 -0.012 0.507 Vocab Richness 0.214 0.116 -0.263 0.107
(b) Correlation between the number of verses/tokens and average coherence, fluency, and similarity scores, as well as MatchA% at 16000 iterations.
Table 4: Correlations
inal verses. Furthermore, MatchA% is significantly higher for the LSTM model, which means that the annotators agreed on matching verses more frequently. We believe this means that the LSTM model, trained on all verses from a given artist, is able to capture the artist’s “average” style, whereas authentic verses represent a random selection that are less likely, statistically speaking, to be similar to another random verse. Note that, as we expect, there is a strong correlation between the number of tokens in the artist’s data and the frequency of agreed-upon correct style matches (cf. Table 4b). Since verses vary in length, this correlation is not observed for verses. Finally, the lack of strong correlation with vocabulary richness suggests that token uniqueness is not as important as the sheer volume.
Lastly, we note that the fully automated methodology we propose is able to replicate the results of the previously available semi-automatic method for the rapper Fabolous, which was the
only artist evaluated by Potash et al. (2015). Furthermore, the results of automated evaluation for the 11 artists confirm that the LSTM model generalizes better than the baseline model.
8 Conclusion
In this paper, we have presented a comprehensive evaluation methodology for the task of ghostwriting rap lyrics, which captures complementary aspects of this task and its goals. We developed a manual evaluation method that assesses several key properties of generated verse, and created a data set of authentic verse, manually annotated for style matching. A previously proposed semiautomatic evaluation method has now been fully automated, and shown to replicate results of the original method. We have illustrated how the proposed evaluation methodology can be used to inspect an LSTM-based ghostwriter model. We believe our evaluation experiments also clearly demonstrate that complementary evaluation methods are required to capture different aspects of the ghostwriting task.
Lastly, our evaluation provides key insights into future directions for generative models. For example, the automated evaluation shows how the LSTM’s inability to integrate new vocabulary makes it difficult to achieve truly desirable similarity scores; future models can draw on the work of Graves (2013) and Bowman et al. (2015) in an attempt to leverage other artists’ lyrics.

This paper proposes to present a more comprehensive evaluation methodology for the assessment of automatically generated rap lyrics (as being similar to a target artist).  While the assessment of the generation of creative work is very challenging and of great interest to the community, this effort falls short of its claims of a comprehensive solution to this problem.  All assessment of this nature ultimately falls to a subjective measure -- can the generated sample convince an expert that the generated sample was produced by the true artist rather than an automated preocess?  This is essentially a more specific version of a Turing Test.   The effort to automate some parts of the evaluation to aid in optimization and to understand how humans assess artistic similarity is valuable.  However, the specific findings reported in this work do not encourage a belief that these have been reliably identified.  Specifically -- Consider the central question: Was a sample generated by a target artist?        The human annotators who were asked this were not able to consistently respond to this question.        This means either 1) the annotators did not have sufficient expertise to perform the task, or 2) the task was too challenging, or some combination of the two.    The proposed automatic measures also failed to show a reliable agreement to human raters performing the same task.        This dramatically limits their efficacy in providing a proxy for human assessment.   The low interannotator agreement may be  expected  because the task is subjective, but the idea of decomposing the evaluation into fluency and coherence components is meant to make it more tractable, and thereby improve the consistency of rater scores.  A low IAA for an evaluation metric is a cause for concern and limits its viability as a general purpose tool.    Specific questions comments:  * Why is a line-by-line level evaluation prefered to a verse level analysis.  Specifically for  coherence , a line by line analysis limits the scope of coherence to consequtive lines.  * Style matching -- This term assumes that these 13 artists each have a distinct style, and always operate in that style. I would argue that some of these artists (kanye west, eminem, jay z, drake, tupac and notorious big) have produced work in multiple styles.  A more accurate term for this might be  artist matching .  * In Section 4.2 The central automated component of the evaluation is low tf*idf with existing verses, and similar rhyme density.  Given the limitations of rhyme density -- how well does this work.  Even with the manual intervention described?  * In Section 6.2 -- This description should include how many judges were used in this study? In how many cases did the judges already know the verse they were judging?  In this case the test will not assess how easy it is to match style, but rather, the judges recall and rap knowledge.


Event-based, Recursive Neural Networks for the Extraction and Aggregation of International Alliance Relations
In this article, we explore how text mining can help producing reliable, high-level numerical data for multi-document relation extraction applications. We combine neural network techniques, information aggregation and visualization methods to extract event-based relations with a good tolerance to noise. We describe a search engine based on these methods, identifying the evolution of alliance and opposition relations between countries.
1 Introduction
Information and communication technologies have provided tools and methods to make the production of information more democratic. As a result, a vast amount of content is available, which arguably creates more noise than knowledge at the end of the day. Without hierarchical organization and contextualization, users may lack perspective to understand and assimilate the multiplicity of events that they come accross every day, and to link them to related events in the past.
In this context, we need tools for extracting, aggregating and visualizing knowledge in order to facilitate information analysis and access a variety of points of view. In this article, we take the example of the geopolitical alliances and their evolution; we extract relations from the texts and aggregate them in order to turn a set of very focused pieces of information into a broader knowledge and a bigger picture of a given situation.
The main contributions of this paper are:
• Relation extraction: we present an eventbased, recursive neural network approach for identifying alliance and opposition relations between countries, at sentence level, in English. We show that adapting the models
to the eventive nature of extracted relations helps existing recursive neural models. We use a precision-oriented cost function for a better later aggregation.
• Relation aggregation: we aggregate the sentence-level relations in a multi-document environment, in order to obtain a picture of the geopolitical situation and evolution on a specific subject defined by a user query (e.g., situation in Syria, nuclear proliferation, North Pole ownership). We cope with the inevitable amount of noise to produce reliable numerical data from texts.
2 System Overview and Dataset
2.1 Relation Extraction
We extract opposition (NEG), alliance (POS) or neutral (NEU) relations between two countries, explicitly expressed in news titles. Such relations are illustrated by the following examples:
(1) Japan and China agree to reduce tensions over Senkaku islands. → POS(Japan, China)
(2) Obama, Merkel warn Russia against intervention. → NEG(U.S.A., Russia), NEG(Germany, Russia), POS(U.S.A., Germany)
(3) Serbia prepares hero’s welcome for Putin. → POS(Serbia, Russia)
(4) Russia backs Ukraine rebel vote. → NEG(Russia, Ukraine)
(5) British teens detained en route to Syria, police say. → NEU(Great Britain, Syria)
Countries (arguments) can be designated by their actual names, the name of their capital or a representative person. Unlike for many knowledge extraction approaches, these relations are very changing, making the problem more difficult.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
2.2 Relation Aggregation
We then index these relations together with the text and the date of the corresponding articles. At query time, we retrieve all titles relevant to the query, extract the candidate entities and their relations. For each day, POS and NEG relations are aggregated and a tendency of this day is obtained, as well as a trend over time. A graphical visualization of this trend is then proposed to the user.
We chose to use only titles because we can consider that a title describes only one event, and that this event can be time-stamped by the document creation time, which is not true in article content. However, this comes with a drawback : titles in English often follow some different syntactic rules than traditional sentences, as in examples 2 (coordination expressed with only a comma) and 5 (contracted reported speech). Most parsers are not trained on title sentences and will then experience a higher error rate.
2.3 Dataset
Country, nationality, capital name and main leader lists were extracted from the linked data repository of the CIA World Factbook, and linguistic variations of these names were collected from DBPedia (Lehmann et al., 2013). Major unions of countries (United Nations, European Union, ASEAN, African Union...) and important groups (ISIS...) are also considered. All these elements were considered as entities potentially parts of an alliance/opposition relation.
We used a corpus of 3.3 millions of English news titles from about 20 different web news sites, between 2013 and early 2016 (3,000 documents/day in average). From this corpus, we randomly selected titles containing at least two entities. Each pair of entities is an instance of the classifier, which means that one sentence can lead to several instances. Furthermore, we added the heads of the nominal phrase containing these entities, as well as “person triggers” (man, minister, etc.) in order to refine the training. For example:
(6) EU, U.N. condemn Ukraine rebels’ election plans. → NEG(EU, rebels), NEG(UN, rebels), POS(EU, Ukraine), POS(UN, Ukraine), POS(EU,UN)
the word rebels is added to EU, U.N. and Ukraine, leading to five possible pairs (entities
from the same chunks are not considered).1 These extra entities are expected to help the model use all possible information.
Among all these automatically collected entities, we manually annotated 5172 pairs with polarities NEU (2837 instances), NEG (1473 instances) and POS (862 instances), coming from 2166 different titles (between 1 and 15 pairs in a title). Only explicit relations were annotated POS or NEG, and no geopolitical knowledge was used.
3 Relation Extraction at Sentence Level with Recursive Neural Models
We describe now the models used for classifying relations at sentence level, as described in Sec. 2.1.
3.1 Recursive Neural Networks
Recursive neural networks (Socher et al., 2011, 2013) have proved useful for tasks involving longdistance relations, such as semantic relation extraction (Hashimoto et al., 2013; Li et al., 2015).
As illustrated by Figure 1 (from Socher et al. (2013)), an n-gram is parsed into a binary tree and each word is represented as a K-dimension vector. The representation of each parent node is computed based on its immediate children, in a bottom-up fashion, using the following equations:
p1 = f(WL · b+WR · c); p2 = f(WL · a+WR · p1)
where f(·) is the function tanh, and WL and WR are K × K matrices. Vector p1 has the same dimension as word vectors and can then be used for computing its own parent node.
The root representation will then be used as features to classify each phrase with a softmax layer.
3.2 Our Models
Our full-tree model (FT) is based on this exact principle.2 For example, in Figure 2.a, the
1Note that we consider the mention of the country as representing its government position.
2Note that we do not report results with the tensor version of recursive NNs since they systematically decreased results.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
a. Full-Tree (FT) model b. Shortest-Path (SP) model
c. Initial dependency tree d. Dependency name conversion e. Event-Centered (EC) model
Figure 2: Different representations of the relation “Kurds agree oil deal with Iraq president”.
Figure 3: Other examples of conversion from dependency graph to EC trees.
parse tree obtained with Stanford parser (Klein and Manning, 2003) is binarized, and the vector for parent p is calculated from K-dimension word embeddings wwith and wEND:
p = f(WLEFT · wwith +WRIGHT · wEND + b)
where b is the 1×K bias vector. In a ShortestPath (SP) model, we use the same representation but we keep only nodes and branches that are part of the shortest dependency path (see Figure 2.c then 2.b, where the “oil deal” is pruned but “president” is added), as experimented by Xu et al. (2015) for LSTM networks.
Dependency tree recursive neural networks (Socher et al., 2014) generalizes to non-binary trees by considering as many composition matrices as necessary (until one matrix per dependency type). This will allow us to manipulate the tree as desired in our Event-Centered (EC) model described in the following section.
As we do not seek for general knowledge but for changing relationships, the entities themselves are replaced by generic *START* and *END* words, as well as proper nouns that are replaced by *PN*. This guarantees that the model will really learn the relational structure rather than the specific connection between two countries. This also gives us a way to represent the entities in the structure, without having to stick to the exact path between them (as in Socher et al. (2012)) or to add positional features (as in Zeng et al. (2014)).
3.3 Event-Centered Modeling
A known issue of recursive models is that their deep tree structure leads to a bias towards the topmost nodes (Kalchbrenner et al., 2014). We turn this problem into a desirable feature by pushing the main event governing the relation to the top of the structure.3 This section details this operation, illustrated by Figures 2.c, 2.d and 2.e.
At step ¬, dependencies from Stanford parser are renamed by the position of the dependent w.r.t the governor (LEFT is the dependent if before the governor in the sentence, RIGHT otherwise). Only the path between entities is kept. Then (step ), each branch node (agree, with and president) is extracted and converted into a leaf node, with an edge labeled MAIN going from this leaf node to the former position of the word. This label represents the fact that the word is the governor of its tree sibling in a dependency.
3Therefore, we do not use any node weighting nor normalization by the number of words underneath them, unlike in previous work (Socher et al., 2011, 2014).
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
We obtain a structure that we call EventCentered (EC), that can be processed by a dependency tree recursive model. For example, in Figure 2.e, the root vector r is calculated as follows:
r = f(WLEFT ·p1+WMAIN ·wagree+WRIGHT ·p2+b)
The top node of EC trees can have three children, all the others have only two. Only three matrices (WMAIN,WLEFT,WRIGHT) are created, keeping the number of parameters reasonable. Figure 3 illustrates two more examples of conversion.
As our relations are triggered by events, we expect at least one event word to be part of the dependency path between the two entities (we call an event word either a non-state verb or a noun tagged as act or event in WordNet (Miller, 1995)). It may not be the case, either because of a specific structure of the sentence or because of a parser error, mainly due to the speficities of titles or to traditional bad prepositional attachments. We now describe how we force the path to contain an event word.
If the dependency path from E1 to E2 does not contain any event word, we extend this path by collecting common ancestors until we meet one. For example, in the following sentence:
(7) Japan and China agree to reduce tensions over Senkaku islands
the path is turned from Japan–conj–China to agree—Japan—China.4
If no event word is found, then the closest event word to E1 and E2 (or as a last resort, the root node) is attached to the entities. This is the case in
(8) ISIS militants attack Syrian town near Turkey border
with the (erroneous) initial dependency graph
where attack is not in the path, but is added because attack is an event word. The graph is then centered on this event so that we obtain a tree as described before:
Using positional relations instead of real dependency names allows to twist the graph as desired to save what can be saved from an erroneous graph.
4Note that we used the uncollapsed version of Stanford dependency graphs in order to keep word vectors for prepositions. Not doing so decreases dramatically the results.
3.4 Precision-Oriented Cost Function
Our final application aggregates the sentencebased relations and uses a large corpus where the same relation on the same day will be expressed many times in different manners. This is why, faced with the traditional trade-off dilemma between precision and recall, we want to favor precision, for two reasons:
• Aggregating results with poor precision tends to lead to wrong aggregated values.
• The corpus redundancy will help to gather useful information even with a classifier with lower recall.
We then bias the classical cross-entropy error function by adding a penalty to false positives (i.e. POS or NEG erroneous predictions). For each relation r represented by the parameter vector θ, the error function is then
E(r; θ) = −λFP
( 3∑
k=1
tk log dk(r; θ)
)
where k represents each of the three classes POS, NEG and NEU, tk is the kth element of the multinomial target label distribution t for a relation, dk(r; θ) the 3-dimensional output of the softmax layer. Finally, λFP = 1 if the prediction is correct or false negative (NEU instead of POS or NEG) and λFP = α (with α > 1) if the prediction is a false positive, so that it is more costly. This can also be seen as an extra regularization parameter on the sum of all false positive predictions. In our experiments, we choose α such that it guarantees a precision higher than 80% on dev POS and NEG classes and maximizes the overall accuracy.
3.5 Extra Features
In addition to the top vector of the recursive model, we add the following features to the softmax classifier: average of the three words before E1 (resp. after E2); concatenation of the three words after E1 (resp. before E2); average of all event words (as defined above) between E1 and E2, as well as in the sentence. This leads to a total of 10×K extra parameters.
4 Classification Experiments and Results
4.1 Baselines
We evaluate this sentence-level extraction step and we compare our different models with:
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
SVM. A SVM two-step model using a sentiment lexicon and hand-crafted resources for alliance/opposition triggers, together with traditional positional, lexical and syntactic features. The first step classifies between neutral and nonneutral relations, while the second step distinguishes between positive and negative relations.
LSTM. A Long Short Term Memory (LSTM) model (Hochreiter and Schmidhuber, 1997) with softmax classifier.
TABARI. A rule engine (Schrodt and Yonamine, 2012) and a set of rules oriented toward inter-state behavior (Schrodt et al., 2008), composed of 284 relation labels that we mapped to either positive, negative and neutral relations. As the rules date back to 2008, we enriched the actor list by the entities from our corpus.
4.2 Implementation
We report results for two different configurations:
• Models FT, SP and EC with a softmax layer on the top node of the relation representation. The parameters are then the word embeddings (V × K parameters), the two or three K × K composition matrices, the softmax matrix (K × 3) and bias vectors (K each).
• The same models with the extra features described in Section 3.5 (10 × K parameters). These are called respectively FT+F, SP+F and EC+F, with or without precision-oriented cost function.
We pre-train word vectors with word2vec (Mikolov et al., 2013) applied to our entire article corpus. For all models, we use a development set for early stopping and hyper-parameter tuning (regularization weights, word vector size – set to K = 50 – and parameter α). We failed at finding an α meeting the requirements for the FT model. All these models are implemented with Theano (Bastien et al., 2012) and L-BFGS-B optimization, and tested on a test set of 453 relations.
4.3 Classification Results
Table 1 reports the results obtained by the described classifiers. Precision and recall values are
based on NEG and POS classes5, accuracy is on the three classes.
SP and EC models do not improve the results when only the top node of the structure is used as a feature for the softmax classifier. This can be explained by the fact that more sentence words are not represented at all in SP and EC structure, while FT contains all words between entities. However, with the extra content features, the EC accuracy is much higher, showing that a more focused tree structure can make the difference when the vocabulary gap is filled by representing missing words with simple average or concatenation.
Even the unbiased cost function leads to a precision much higher than recall. Our precisionoriented cost function with a tuned α value leads to a small precision gain with a small recall loss.
The system TABARI does not perform well on this dataset. This can be due to the fact that the dataset addresses a very large variety of subjects and geographic areas; TABARI gets better performances on specific topics with dedicated rules.
In what follows, we use the model EC+Fα=2.3.
5 Time-Aware Aggregation and Visualization
We describe here how we aggregate the sentencelevel relations. A Solr search engine (Kuć, 2013)
5NEU is much more represented and useless for aggregation; integrating it to prec. and rec. values would increase them artificially, making a NEU-only classifier the best one.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
is built from time-stamped alliance relations. Each sentence containing at least one relation is indexed with its stemmed content, the countries involved in relations, the relations themselves and the document creation date. A typical query is then composed of a few keywords representing the topic, a temporal interval (minimum of maximum dates) and zero or more country names on which the user wants to restrict relation extraction.
5.1 Aggregating Information
All sentences returned by Solr, without any number limit, are aggregated. For all pairs of considered countries, inside the same day d, if the numbers of POS and NEG relations between the two countries are respectively P (d) andN(d), then the weight for the pair and the day d is:
w(d) = log
( 1 + P (d)
1 +N(d) ) which leads to a number between−∞ and +∞, where w = 0 means that the relation is neutral, w < 0 that the relation is an opposition, andw > 0 that the relation is an alliance.
The noise is then reduced by a weighted mean smoothing over a temporal window of W days:
sw(d) =
∑n j=−n
w(d+j) abs(j)+1∑n
j=−n 1
abs(j)+1
where n is a half-window (n = floor(W2 )) For example, for a 5-day window:
sw(d) = w(d−2) 3 + w(d−1) 2 + w(d) + w(d+1) 2 + w(d+2)
3 1 3 + 1 2 + 1 + 1 2 + 1 3
5.2 Visualization
5.2.1 Time-Series Plot
For bilateral relations (field countries containing two items), we provide the users with a timeseries plot representing sw(d), and show them on demand the sentences which led to this value. Figure 4 shows an example of results concerning the relations between United States and Russia (“countries:United States AND Russia”) concerning the situation in Syria (“keywords:syria”).
5.2.2 Force-directed Graph
When zero or one country is specified by the user, we generate a graph of countries, where the distance between vertices (i.e., countries) reflects the opposition between the countries in a given time
span (generally, larger that a single day). The problem here is that our classifier provides a relation between two countries independently of the others; it is therefore very unlikely that these values can be used as distance values in the whole graph (especially, triangle inequality will not be respected). For this reason, we use the Barnes-Hut force-directed layout algorithm (Barnes and Hut, 1986), where a value between two vertices is considered as a repulsive force. The algorithm tries to minimize the energy spent to keep the whole graph together, and then to adapt the distances between vertices so that the rendering on a 2D plan is possible.
For this algorithm, we need to transform our weights sw(d) into positive numbers (repulsive forces). We also need to damp the noise and the variations of the weights that are introduced by the volume of data. This problem is illustrated by Table 2, where three countries A1, A2 and A3 are allies against another one B on a given topic and a given time span. The weights sw(d) of the three relations between allies are positive but vary between 1 and 3.5. As a result, A1 is finally closer to B (difference of 2) than to A2 (difference of 2.5), which is not desirable.
This kind of effects can be corrected by a “Sshaped”, logistic function, that models a level of saturation after an approximately exponential growth (or, in our case, decrease):
sw′(d) = 1− 1 1 + e−sw(d)
(9)
This function levels off high weights (both negative and positive), increases differences between positive and negative values and thus helps reducing noise without having to discretize values arbitrarily. Resulting numbers are all positive, between 0 (strong alliance, attraction in the graph) and 1 (strong opposition, repulsion in the graph).
Figure 5 shows an example of such graphs obtained by the force-directed algorithm, with the keyword syria.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Figure 4: Example of plot produced by the system for bilateral relations between United States and Russia on the query “Syria”, on a daily scale. The right frame shows sentences corresponding to the user-selected date (Oct. 2, 2015). Circled numbers have been manually added to the screenshot to make some peaks clearer. ¬ Russian military involvement grows in Syria (bad relation, sw(d) < 0);  US and Russian defence chiefs start discussing (better relation, sw(d) > 0); ® Russian airstrikes in Syria; ¯ Optimism after U.S. Secretary of State’s visit to Moscow.
Figure 5: Example of graph produced by the system for relations between different states on the query “syria”, for the year 2015. Edge colors indicate the kind of relation (from dark red for strong alliance to dark blue for strong opposition), and vertice colors reflects proximity of countries with each other.
5.3 Evaluation
Evaluating the relevance of trends suggested by a plot or a dynamic map is very subjective and would require a high level of expertise in each domain concerned by the tested queries. That is why we opted for a protocol that is at the same time more objective and easier to conduct:
1. We chose 14 queries with the following information: names of two countries (or unions of countries) involved in the relation, and an optional keyword-based thematic restriction. We selected queries having potentially a high density of extracted relations (e.g. North Korea vs. South Korea, or Russia vs. United Nations on “Syria”), as well as sparser topics (China vs. Japan on maritime affairs).
2. On the resulting plot, we selected up to 5 strong peaks — abs(sw(d)) > 1 — and 5 weak peaks — abs(sw(d)) ≤ 1. This led to a total of 97 relations to evaluate.
3. For each of these peaks, we estimated whether the polarity of the peak was relevant or not, from a different news article collection. This is still a heavy task, which explains the low number of tested instances.
The accuracy of strong peaks is 0.93, making them highly reliable. Accuracy of weak peaks is 0.752.
6 Related Work
TDT and Opinion Mining. The idea of automatically finding topically related material in streams of newswire data goes back to Topic Detection and Tracking evaluation campaigns (Allan, 2002). These campaigns and the follow-up research aim at developing algorithms for identifying and organizing events described in documents.
Our work contributes to this global research effort and explores the quantitative aspects of knowledge that can be extracted from textual documents. On this point, as well as on the topic
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
of alliance and opposition relations, this can be connected to sentiment analysis and opinion mining (Liu, 2012). Different opinions must be aggregated, not only for getting more information, as in multidocument information extraction, but also for balancing positive and negative statements and conclude about the average opinion.
Fewer research works focused on extracting opinion relations between entities, including Gottipati et al. (2013); Yang and Cardie (2013); Johansson and Moschitti (2013), that can be related to our work.
International Relations. TABARI (a.k.a. KEDS) is an open-source program extracting events concerning international relations from news texts (Schrodt and Yonamine, 2012). This rule-based system makes extensive use of handcrafted dictionaries and patterns. It has been widely used in the field of political science, for example for modeling dependencies in international relations networks (Hoff and Ward, 2004), as well as in the GDELT Project.6 Several versions have been proposed for studying different aspects of international relations (Levant, Turkey, Balkans, Political Instability Task Force, etc.) and a more general-purpose set of rules, CAMEO, has also been developed (Schrodt et al., 2008). We compare it to our approach, after a few necessary adaptations (Section 4.1). More recently, (Boschee et al., 2013) presented a proprietary system and compare to TABARI as well.
With very different approaches, O’Connor et al. (2013) describe an unsupervised model for extracting events between political actors, and Gerrish (2013) presents a time-series topic model of foreign affairs. Similar aspects with our work are the detection of conflicts, as well as the interest in the temporal evolution of these conflicts and a general approach for studying specific situations. However, the purpose and output (distributions of topical terms) are different.
Finally, Chambers et al. (2015) describe an identification of political sentiment between nations at the citizen level, by using a large Twitter dataset. Their work is more directly related to traditional sentiment analysis and does not rely specifically on geopolitical events, even if the observed trends can follow these events.
6http://gdeltproject.org
Neural Networks for Relation Extraction. Several neural network models have been used for sentiment analysis and relation extraction. Examples are convolutional networks (Zeng et al., 2014; Nogueira dos Santos et al., 2015), recurrent networks such as LSTM (Xu et al., 2015) or recursive networks (Socher et al., 2011, 2012, 2013). Li et al. (2015) shows that using tree structures, as do recursive models, helps in tasks where longdistance relations are involved, such as semantic relation extraction. The results described in this paper confirm this conclusion.
Our precision-oriented approach relies on linguistic variation and redundancy in a large amount of documents to ensure a good coverage (Yates, 2007). It is related to some works in web questionanswering (Dumais et al., 2002; M.Banko et al., 2002; Magnini et al., 2002), temporal information aggregation (Kessler et al., 2012) or opinion mining (Turney, 2002; Hu and Liu, 2004).
7 Conclusion & Perspectives
The advances in information extraction techniques and the increase of computing power make possible quite deep analysis of very large collections of texts. In this paper, we propose to take advantage of this progress to aggregate a large volume of extracted relations and to turn this knowledge into numerical data. We also propose a tool identifying the evolution of alliance and opposition relations between countries, on a specific subject.
Most information conveyed every day remains textual and unstructured. With such applications, a wide new range of knowledge can become available, in many different fields, which would bring high added value to the final users, in terms of aggregation, contextualization and hierarchical organization of information. We see our international relations example as a use case study of this effort.
We now intend to focus on reducing the necessary amount of supervision by using distant supervision, so that different subjects could be treated with less effort. Also, such study on country relationships should be multilingual in order to consider the cultural dimension as well.
This article is accompanied with supplementary material: the annotated corpus, the FT, SP and EC versions of all relations (JSON format), the script converting text into each of these versions, and the source for our recursive models.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: Useful modeling contribution, and potentially useful annotated data, for an important problem -- event extraction for the relationships between countries as expressed in news text.  - Weaknesses: Many points are not explained well in the paper.   - General Discussion:  This work tackles an important and interesting event extraction problem -- identifying positive and negative interactions between pairs of countries in the world (or rather, between actors affiliated with countries).  The primary contribution is an application of supervised, structured neural network models for sentence-level event relation extraction.  While previous work has examined tasks in the overall area, to my knowledge there has not been any publicly availble sentence-level annotated data for the problem -- the authors here make a contribution as well by annotating some data included with the submission; if it is released, it could be useful for future researchers in this area.  The proposed models -- which seem to be an application of various tree-structured recursive neural network models -- demonstrate a nice performance increase compared to a fairly convincing, broad set of baselines (if we are able to trust them; see below).  The paper also presents a manual evaluation of the inferred time series from a news corpus which is nice to see.  I'm torn about this paper.  The problem is a terrific one and the application of the recursive models seems like a contribution to this problem.  Unfortunately, many aspects of the models, experimentation, and evaluation are not explained very well.  The same work, with a more carefully written paper, could be really great.  Some notes:  - Baselines need more explanation.  For example, the sentiment lexicon is not explained for the SVM.                    The LSTM classifier is left highly unspecified (L407-409) -- there are multiple different architectures to use an LSTM for classification.  How was it trained?  Is there a reference for the approach?  Are the authors using off-the-shelf code (in which case, please refer and cite, which would also make it easier for the reader to understand and replicate if necessary)?  It would be impossible to replicate based on the two-line explanation here.    - (The supplied code does not seem to include the baselines, just the recursive NN models.  It's great the authors supplied code for part of the system so I don't want to penalize them for missing it -- but this is relevant since the paper itself has so few details on the baselines that they could not really be replicated based on the explanation in the paper.)  - How were the recursive NN models trained?  - The visualization section is only a minor contribution; there isn't really any innovation or findings about what works or doesn't work here.  Line by line:  L97-99: Unclear. Why is this problem difficult?  Compared to what? (also the sentence is somewhat ungrammatical...)  L231 - the trees are binarized, but how?  Footnote 2 --  the tensor version  - needs citation to explain what's being referred to.  L314: How are non-state verbs defined?                    Does the definition of  event word s here come from any particular previous work that motivates it?                    Please refer to something appropriate or related.  Footnote 4: of course the collapsed form doesn't work, because the authors aren't using dependency labels -- the point of stanford collapsed form is to remove prepositions from the dependeny path and instead incorporate them into the labels.  L414: How are the CAMEO TABARI categories mapped to positive and negative entries?  Is performance sensitive to this mapping?  It seems like a hard task (there are hundreds of those CAMEO categories....) Did the authors consider using the Goldstein scaling, which has been used in political science, as well as the cited work by O'Connor et al.?  Or is it bad to use for some reason?  L400-401: what is the sentiment lexicon and why is it appropriate for the task?  L439-440: Not clear.   We failed at finding an alpha meeting the requirements for the FT model.   What does that mean? What are the requirements? What did the authors do in their attempt to find it?  L447,L470:  precision and recall values are based on NEG and POS classes .  What does this mean?  So there's a 3x3 contingency table of gold and predicted (POS, NEU, NEG) classes, but this sentence leaves ambiguous how precision and recall are calculated from this information.  5.1 aggregations: this seems fine though fairly ad-hoc.  Is this temporal smoothing function a standard one?  There's not much justification for it, especially given something simpler like a fixed window average could have been used.  5.2 visualizations: this seems pretty ad-hoc without much justification for the choices.  The graph visualization shown does not seem to illustrate much.  Should also discuss related work in 2d spatial visualization of country-country relationships by Peter Hoff and Michael Ward.  5.3 L638-639:  unions of countries  isn't a well defined concept.  mMybe the authors mean  international organizations ?  L646-648: how were these 5 strong and 5 weak peaks selected?  In particular, how were they chosen if there were more than 5 such peaks?  L680-683: This needs more examples or explanation of what it means to judge the polarity of a peak.  What does it look like if the algorithm is wrong?                    How hard was this to assess?  What was agreement rate if that can be judged?  L738-740: The authors claim Gerrish and O'Connor et al. have a different  purpose and outputs  than the authors' work.  That's not right.  Both those works try to do both (1) extract time series or other statistical information about the polarity of the relationships between countries, and *also* (2) extract topical keywords to explain aspects of the relationships.  The paper here is only concerned with #1 and less concerned with #2, but certainly the previous work addresses #1.  It's fine to not address #2 but this last sentence seems like a pretty odd statement.  That raises the question -- Gerrish and O'Connor both conduct evaluations with an external database of country relations developed in political science ( MID , military interstate disputes).              Why don't the authors of this work do this evaluation as well?  There are various weaknesses of the MID data, but the evaluation approach needs to be discussed or justified.