Exploring Vector Spaces for Semantic Relations
Word embeddings are used with success for a variety of tasks involving lexical semantic similarities between individual words. Using unsupervised methods and just cosine similarity, encouraging results were obtained for analogical similarities. In this paper, we explore the potential of pre-trained word embeddings to identify generic types of semantic relations in an unsupervised experiment. We propose a new relational similarity measure based on the combination of word2vec’s CBOW input and output vectors which outperforms concurrent vector representations, when used for unsupervised clustering on SemEval 2010 Relation Classification data.
1 Introduction
1.1 Vector space semantics Vector space word representations or word embeddings, both ’count’ models (Turney and Pantel, 2010) and learned vectors (Mikolov et al., 2013a; Pennington et al., 2014), were proven useful for a variety of semantic tasks (Mikolov et al., 2013b; Baroni et al., 2014). Word vectors are used with success because they capture a notion of semantics directly extracted from corpora. Distributional representations allow to compute a functional or topical semantic similarity between two words or, more recently, bigger text units (Le and Mikolov, 2014). The closer two entities are in the vector space (quantified usually, but not necessarily in terms of cosine similarity), the more similar they are semantically. This similarity can be
exploited for lexical substitution, synonym detection, subcategorization learning etc. Recent studies suggest that neural word embeddings show higher performance than count models (Baroni et al., 2014; Krebs and Paperno, 2016) for most semantic tasks, although Levy et al. (2015a) argue that this is only due to some specific hyperparameters that can be adapted to count vectors. In what follows, we will concentrate on exploring whether and how pre-trained, general-purpose word embeddings encode relational similarities.
1.2 Relational analogies as vector offsets
Relation extraction and classification deal with identifying the semantic relation linking two entities or concepts based on different kinds of information, such as their respective contexts, their co-occurrences in a corpus and their position in an ontology or other kind of semantic hierarchy. Whether the vector spaces of pretrained word embeddings are appropriate for discovering or identifying relational similarities remains to be seen. Mikolov et al. (2013b) claimed that the embeddings created by a recursive neural network indeed encode a specific kind of relational similarities, i.e. analogies between pairs of words. He found that by using simple vector arithmetic, analogy questions in the form of "a1 is to a2 as b1 is to b2" (man ~king :: woman ~queen) could be solved. Relationships are assumed to be present as vector offsets, so that in the embedding space, all pairs of words sharing a particular relation are related by the same constant offset. Vector arithmetics give us the vector which fills the analogy, and we can search for the word b2 whose embedding vector has the greatest simi-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
larity to it:
argmaxb2 = sim(b2, (b1 − a1 + a2)) (1)
Levy et al. (2015a) suggested that instead of a vector offset method, this calculation can also be considered as a combination of similarities. Using cosine similarity for sim, equation 1 can be written as a combination of similarities (Levy et al., 2015a) as
argmaxb2 = sim(b2, b1)− sim(b2, a1)+ + sim(b2, a2) (2)
Analogy pairs, however, are a special case of relational similarity because not only a1 (man) relates to a2 (king) the same way that b1 (woman) relates to b2 (queen); the relation between a1 (man) and b1 (woman) is also parallel to the relation between a2 (king) and b2 (queen.) When it comes to different types of semantic relations, their instances may or may not be analogical: e.g., while we can say that a (claw) is a component of an (owl) just like (walls) are a component of a (hospital), no meaningful relational similarity is shared between the pairs (claw)→ (walls) and (owl) → (hospital) (see Figure 1).
Figure 1: Analogical vs non-analogical semantic relation
1.3 Criticism of the vector offset method
As precise as neural word embeddings combined with cosine similarity may be for calculating semantic proximity between individual words, recent results seem to suggest that their value in identifying relational analogies using vector arithmetics is limited. In fact, a big part of their merits is likely to come from the precise calculation of individual similarities instead of relational similarities. Hence, they can be approximated using relation-independent baselines. Linzen (2016) remarks that currently
used analogy tasks evaluate not only the consistency of the offsets a1 − a2 and b1 − b2, but also the neighborhood structure of the words in the vector space. Concretely, "if a1 and a2 are very similar to each other (...) the nearest word to b2 may simply be the nearest neighbor of b1 (...) regardless of offset consistency" (Linzen, 2016). Moreover, some of the success obtained by the vector offset method on analogies can also be obtained by baselines that ignore a2, or even both a1 and a2.
Levy et al. (2015b) point out similar limitations of this line of work. Different word embedding combinations in supervised learning of taxonomical relations do not seem to learn the relations themselves, but individual properties of words. They tested previously suggested vector compositions for supervised learning of inference relations: concatenation, difference, comparing only the first or only the second element of the pairs. The study concludes that the classifiers only learn individual properties (e.g. a "category" type word is a good hyponym candidate), but not semantic relations between words. These studies suggest that the semantic information obtained from word embeddings is correct for identifying similar or related units, but is already self-contained and difficult to enrich in order to retrieve more specific semantic contents such as relational similarities or specific relations.
In this paper, we aim to challenge this conclusion within a large scale semantic relation classification experiment, and show that it is possible to achieve improvements compared to baselines and current methods. We apply known vector composition methods, and propose a new one, to unsupervised large-scale clustering of entity pairs categorized according to their semantic relation. While large scale semantic relation classification is a very difficult task and the state of the art does not perform close to a human level, we expect that the experiment provides information to compare the semantic potential of different vector/similarity combinations.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
2 Semantic Relations in Vector
Spaces
2.1 Related work Relation classification includes the task of finding the instances of the semantic relations, i.e. the entity tuples, and categorizing their relation according to an existing typology. In an unsupervised framework, relation types are inferred directly from the data. Supervised systems rely on a list of pre-defined relations and categorized examples, as described in the shared tasks of MUC, ACE or SemEval campaigns (Hobbs and Riloff, 2010; Jurgens et al., 2012; Hendrickx et al., 2010). Competing systems extract different kinds of features eventually combined with external knowledge sources, and build classifiers to categorize new relationship mentions (Zhou et al., 2005). A commonly used method, initiated by Turney (2005; 2006), is to represent entity couples by a pair-pattern matrix and calculate similarities over the distribution of the couples. Another way of constructing a distributional vector space to represent quantifiable context features for relation extraction is to combine the vectors of the two entities. Different combination methods were proposed to represent compositional meaning (Mitchell and Lapata, 2010; Baroni and Zamparelli, 2010; Baroni et al., 2012). Popular methods include addition (Mitchell and Lapata, 2010), concatenating the two vectors (Baroni et al., 2012) or taking their difference (Weeds et al., 2014). As of now, these vector combinations had two types of applications in semantic relation classification. The first one aims to find specific types of semantic or functional analogies (Herdaǧdelen and Baroni, 2009; Makrai et al., 2013; Levy et al., 2015a). The second one tries to infer taxonomical relations in supervised experiments (Weeds et al., 2014; Turney and Mohammad, 2014). Recently, Turney (2012) suggested a dual distributional feature space for supervised classification. Herdaǧdelen and Baroni (2009) combine individual entity vectors with co-occurrence contexts in their vector space. These experiments either aim to identify very specific relation types (typically taxonomical relations) with a mixture of features and a supervised classifier, or target analogy pairs: a task in which, as we have seen, relation-unaware
baselines approximate relation-aware representations.
2.2 Task definition Whether we use the vector offset method or any pairwise similarity combination, finding the missing word in an analogy depends on two factors:
1. Vector quality (do semantically close elements have a higher cosine similarity?)
2. Density and structure of the vector space.
If we adapt 1) above to the more generic relational similarity task, the question can be formulated as follows:
3. How much information about the semantic relation is actually in the text and how fit is the vector combination method to encode this information?
In accordance with Linzen (2016) and Levy et al. (2015b), we also think that analogy test sets are not optimal to answer this question. We propose to study relational similarity using a more generic and large scale relation classification task (Hendrickx et al., 2010), and clustering pairs according to semantic relations, instead of finding the one missing word in an analogy. This way, we rely less on the neighborhood structure and more on actual "linguistic regularities". We evaluate different vector combination methods, and propose a new one, for calculating relational similarities. The evaluation concentrates on the aspects above. We test whether cosine similarity over these vector spaces is adapted for discovering groups and classifying individual instances. We report clustering results and compare the vector combinations by their performance.
2.3 Lexical vs contextual relations The semantic relation classification task, supervised or not, is a difficult one with a strong upper bound: not every piece of relational information is explicit in the text. Some relations are more lexical by nature: relations such as dog is an animal; a teacher works at a school; a car is kept at a parking lot, can be interpreted independently of the context.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
On the other hand, many relation instances are contextual. Contextual relations (e.g. "the accident was caused by the woman") tend to be expressed explicitly, but rarely, in a corpus. They can be handled by pattern-based approaches rather than distributional representation combinations. We expect to be able to identify prototypical lexical relation instances with vector combination methods. In the scope of the current experiment, we do not intend to combine contextual information with lexical semantic representations, since our primary goal is to argue that vector combinations may encode lexical relational similarities in themselves. If a representation is more capable than others to group together word pairs according to relational similarities, this potential can further be exploited in unsupervised as well as in supervised experiments. On the other hand, well-known outliers (contextual relations and less typical examples) will require a complementary approach. This task is difficult and requires a change of perspective: when we look for missing elements in an analogy we know the word exists and we presume to know where it will be in the vector space, while in unsupervised clustering, our aim is to infer a global structure from the data.
2.4 Semantic relation data The SemEval 2010 Task 8 data we used (Hendrickx et al., 2010) contains examples of relation instances for 9 relations with sufficiently broad coverage to be of general and practical interest (Table 1). There is no overlap between classes, but there are two groups of strongly related relations to assess models’ ability to make fine-grained distinctions (CONTENTCONTAINER, COMPONENT-WHOLE, MEMBER-COLLECTION and ENTITYORIGIN, ENTITY-DESTINATION). This data set is very challenging, not only because of the fine semantic distinctions, but also because semantic relations were annotated in context and contain many less typical relation instances. In the current experiment, the goal we set for ourselves is to explore models’ abilities to capture the structure of the data, rather then in achieving a classification precision close to that of humans. We used 6637 pairs of single word instances
from the training data. Contexts in the training data were discarded. Class bias is present: the most frequent relation has 979 instances, the least frequent has 486.
3 Vector combination methods
If a1, a2, b1, b2 are entities (nouns or nominal compositions) from a corpus, each of them assigned a pre-trained word embedding, we would like to classify entity couples a = (a1, a2) and b = (b1, b2) according to their semantic relation. This means that we are looking for an efficient combination of a1, a2 and b1, b2 vectors that encode their relational attributes. We aim to find effective methods to calculate a relational similarity sim(a, b) by combining entity vectors a1, a2 and b1, b2.
Pairwise similarities build on the idea that if a1 is semantically similar to b1 and a2 is similar to b2, the relation between a1 and a2 is similar to the relation between b1 and b2. The recall of this approach is expected to be limited: the same relation can hold between different types of entities. Analogical similarities presume that b1− b2 shares the direction with a1 − a2, ignoring the pairwise similarities. We adapt this measure, while aware that analogy pairs are a specific case of relational similarity in that analogies work both ways (man ~king :: woman ~queen and also man ~woman :: king ~queen). IN-OUT similarities: a new combination that builds on the integration of second order similarities. Only a1 : In this baseline solution, the similarity between two pairs is calculated as the similarity between the first entity of each pair, the other pair being ignored.
sim(a, b) = sim(a1, b1) (3)
3.1 Pairwise similarities Different combinations proposed in the literature were compared.
• concatenative : one vector for each entity couple is defined as the concatenation of the vectors of the two entities.
sim(a, b) = sim((a1 ⊕ a2), (b1 ⊕ b2)) (4)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Relation Instances in Typical examples Atypical examples training data
Cause-Effect 979 suicide − death, injury − discomfort women − accident Component-Whole 978 claw − owl, walls − hospital image − photos Entity-Destination 789 solvent − flask, hay − barn chair − corporation Product-Producer 775 industry − models, artist − design officer − oath Entity-Origin 762 relics − culture, plane − runway error − definition Member-Collection 729 stable − hounds, ensemble − ladies mission − monkeys Message-Topic 622 pages − scene, speech − measures exhibition − glamour Instrument-Agency 517 user − console, eye − telescope companies − governments Content-Container 486 document − folder, pictures − box message − paper
Table 1: Semantic Relation Classification data
• pairwise addition Pairwise similarities between respective entities are added up. If we use cosine similarity, this is only slightly different from the concatenative method. Vector addition proved to work well as a compositional representation (Mitchell and Lapata, 2010), despite the fact that word order is ignored.
sim(a, b) = sim(a1, b1)+sim(a2, b2) (5)
A potential problem with this addition objective is that different properties of words are expressed on a different scale and, as a consequence, terms sharing these properties have a higher cosine similarity than terms that are similar with respect to a flatter property. It can be overcome by using multiplication instead of addition (Levy and Goldberg, 2014):
• pairwise multiplication
sim(a, b) = sim(a1, b1)×sim(a2, b2) (6)
3.2 Analogies This is an adaptation of the measure proposed for queen = king - man + woman (Mikolov et al., 2013b). Vector arithmetics give us the vector which fills the analogy, and we can search for the word b2 whose embedding vector has the greatest similarity to it:
argmaxb2 = sim(b2, (b1 − a1 + a2)) (7)
which, using cosine similarity for sim, can be written as a combination of similarities (Levy et al., 2015a) as
argmaxb2 = sim(b2, b1)−sim(b2, a1)+sim(b2, a2) (8)
Mikolov (2013b) notes that this measure is qualitatively similar to the relational similarity model in (Turney, 2012), which predicts similarity between members of the word pairs (xb, xd), (xc, xd) and dissimilarity for (xa, xd). In the current context, we do not look for the missing b2 which maximizes the equation. Instead, we have different couples a and b, and we aim to calculate sim(a, b) to quantify how much the analogy queen - woman = king - man holds.
• difference Focuses on the similarity of b1, b2 and a1, a2, but does not take into account the pairwise distances between the individual entity vectors.
sim(a, b) = sim((a1 − a2), (b1 − b2)) (9)
Levy et al. (2015a) propose a multiplicative version of the analogy formula. We tried to adapt it; however, this measure is not symmetrical (conceived to find b2 which maximizes the form) and the adaptation gave bad results.
3.3 IN-OUT similarities This metric is a combination of first order and second order similarities between the two entity pairs, adapted to relational similarity : a and b are similar if a1 is similar to b1 and also similar to the contexts of b2, the opposite entity in b. In the current experiment, second order similarities are estimated using both input and output vectors generated by word2vec’s CBOW model. In this model, the IN vectors of words get closer to the OUT vectors of other words that they co-occur with. Words with a high input-output similarity tend to appear in the context of each other. This similarity combination was shown to improve information retrieval (Nalisnick et al., 2016).
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
587
588
589
590
591
592
593
594
595
596
597
598
599
Also, Pennington et al. (2014) use second-order similarity to improve similarity calculation between words. Their proposed formula combines first and second order similarity, normalized by the reflective second order similarity of the words with themselves. This is based on the observation that "words are similar if they tend to appear in similar contexts, or if they tend to appear in the contexts of each other (and preferably both)." (Note that they use ’first order’ for word-context (IN-OUT) similarity and ’second order’ for word-word (IN-IN) similarity.)
sim(x, y) = sim2(x, y) + sim1(x, y) 2 √ sim1(x, x) + 1 √
sim1(y, y) + 1 (10)
In our experiment, second order similarities are used in a different way and with a different purpose. Second-order similarities are calculated between opposite elements of the entity couples. We combine those similarities by taking the in-in similarity between a1 and b1, and the in-out similarities between a1 and b2, and between a2 and b1.
• additive in-out
sim(a, b) = sim(a1, b1) + sim(a2, b2) + sim2(a1, b2) + sim2(a2, b1) (11)
where sim2 designates the second order similarity and is calculated as follows:
sim2(x1, y2) = sim(xin1 , yout2 ) + sim(xout1 , yin2 ) (12)
• multiplicative in-out: The same as above, but addition is replaced by multiplication in sim and sim2.
sim(a, b) = sim(a1, b1) ∗ sim(a2, b2) ∗ sim2 (a1, b2) ∗ sim2(a2, b1)
where sim2(x1, y2) = sim(xin1 , yout2 ) ∗ sim(xout1 , yin2 )
4 Clustering Experiments
We trained a word2vec CBOW model (Mikolov et al., 2013a) with negative sampling and a window size of 10 words on the ukWaC corpus (Baroni et al., 2009), and extracted both input
and output vectors of size = 400 to build the vector combinations above. An adjacency matrix was constructed for each vector/similarity combination using cosine similarity. Clustering was implemented with Cluto’s (Zhao et al., 2005) clustering function which takes the adjacency matrix as input. We used a hierarchical agglomerative clustering with the unweighted average distance (UPGMA) criterion function1.
4.1 Evaluation as classification At first, we ran the clustering with 9 clusters (the number of classes in the standard) and tried to make one-to-one correspondences between the standard and the output. Every cluster is mapped to the standard class that shares the more elements with. We then calculate precision and recall for each standard class (zero if the class doesn’t show up as a majority class in any cluster). Average class-based precision and recall is reported, as well as the number of classes in the standard that could be assigned. These scores were published for the SemEval task participants, but ours are not comparable because we only consider one cluster for each class, and because we did the clustering on the training data.
4.2 Evaluation as clustering While the scores above can be indicative of the potential of different representations, they do not provide information on other aspects as cluster stability, purity, the amount of postprocessing needed. Above all, in a completely unsupervised setting, the number of classes in
1We observed that these settings are sensitive to the chaining effect and there is probably room for improvement by experimenting with different task-specific clustering parameters.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
the standard is not known and cluster quality (precision) plays an important role with respect to interpretability: it is easier unify two homogeneous clusters than to separate a noisy one. We ran complementary experiments with different numbers of clusters. Table 3 indicates results for 20 and 30 clusters. The input-output combination method still has an advantage, and concatenation and multiplication also perform well. However, the advantages over the baseline are less significant than when the number of clusters was identical to the standard.
INPUT #clust P R F
a1(base) 20 0.3429 0.1642 0.2221 add 20 0.2434 0.1843 0.2098 conc 20 0.2718 0.2116 0.2380 in-out.add 20 0.2947 0.2076 0.2436 mult 20 0.3405 0.1886 0.2428 in-out.mult 20 0.2711 0.1432 0.1874 diff 20 0.2997 0.1161 0.1674
a1(base) 30 0.3855 0.1712 0.2371
add 30 0.2714 0.1726 0.2110 conc 30 0.3331 0.1862 0.2389 in-out.add 30 0.3548 0.1947 0.2514 mult 30 0.3037 0.1995 0.2408 in-out.mult 30 0.3916 0.1304 0.1957 diff 30 0.3770 0.1318 0.1953
Table 3: Class-based results for 20 and 30 clusters
In the next runs, we measure how stable the different clustering solutions are with settings that are structurally very different from the standard, i.e. have significantly more clusters. Class-based precision and recall are less relevant measures in this setting, since they take the average over the nine standard classes and not over the produced clusters. We therefore decided to use modified purity (Korhonen et al., 2008), adapted for structurally different clustering solution. Modified purity gives the proportion of word couples belonging to the majority class c in their cluster k:
PUR = ∑|K|
i=1 maxj |w in ki ∩ w in cj |∑|K| i=1 w in ki (13)
Modified purity is indicative of the quality and interpretability of the clusters. It favorizes small clusters, but singleton clusters were discarded. This measure corresponds to predic-
tion accuracy in classification if we assign the majority label to clusters. Two series of runs were evaluated: for 10, 20... up to 50, and for 60, 70... up to 100 clusters. Average results are reported. These scores indicate the average purity of clusters over different runs.
INPUT PUR
a1(baseline) 0.2940
add 0.3059 conc 0.3107 in-out.add 0.3235 mult 0.2575 in-out.mult 0.2119 diff 0.2297
5 Discussion
The additive input-output combination shows promising results, especially when it comes to capturing the structure: in the clustering setting with 9 clusters, it identifies 8 classes out of 9 in the standard. This indicates a good potential in differentiating between relation types, especially because the standard is conceived in a way that it contains strongly related classes. It outperforms every other measure until the number of clusters grows significantly above those in the standard (Table 5), when the concatenative measure catches up. The baseline performs well, but additive methods all beat it, while difference is especially weak. Pairwise multiplication is good at recognizing the structure (7 classes out of 9), but not good at assigning elements. Multiplicative methods show a fluctuating performance, especially the multiplicative inputoutput combination. This is due to the higher variance in similarities obtained by multiplica-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
tion (in the case of input-output combination, 6 operands are multiplied), combined with the agglomerative clustering, which is sensitive to chaining. The very high precision of the baseline method with a large number of clusters (Table 3) is noteworthy but not unexpected. Individual similarities have a strong precision for the easily identifiable clusters, while additional relational information is mostly expected to improve recall.
6 Conclusion and Future Work
We presented an experiment to identify relational similarities in word embedding compositions at a large scale, using an unsupervised approach. On the one hand, our results confirm the recent finding that many of the success attributed to vector arithmetics for analogies come from similarities between individual elements. On the other hand, taking second order similarity into account, we can improve relational similarities and take a step toward a meaningful representation for entity couples in a semantic relation.
The baseline performs well and is difficult to enrich with relation-aware information. The results indicate that the vector offset method for analogies, which replaces the pairwise similarity, is the least efficient in capturing generic semantic relations at a large scale. The vector difference representation does not conserve pairwise similarities and the offsets do not prove to be constant enough for unsupervised clustering. Multiplicative methods do not scale up either, although to a lesser extent: they capture some of the relational information, but this happens at the expense of losing precision from individual similarities. Pairwise similarities can be better exploited in an additive or concatenative setting. Moreover, they can be meaningfully complemented by including second order similarities without losing too much information for precise classification. The input-output combination measure coherently outperformed the other combinations in almost every setting, indicating a better potential for unsupervised experiments.
Unsupervised relation classification is a very challenging task for several reasons. Some relation instances are lexical by nature and, there-
fore, can be expected to show up in the same cluster based on distributional cues. On the other hand, contextual relation instances tend to have relation-specific indicators when they co-occur, but their individual vectors will not reveal this information (unless they co-occur very often). Moreover, semantic relations differ with respect to the semantic constraints they put on their arguments. For instance, the second argument of the Content-Container relation tend to belong to a specific semantic class in the standard (bag, box, trunk, case, drawer...), while both arguments of the CauseEffect relation are much freer (gas, prices, pain, acts, species and pyrolysis, collapse, contraction, society, noise). Any future development towards an automated unsupervised classification needs to take these aspects into account and work towards a hybrid solution by separating relations with semantically constrained arguments from free ones, as well as adapting the clustering method to handle outliers.

This paper investigates the application of distributional vectors of meaning in tasks that involve the identification of semantic relations, similar to the analogical reasoning task of Mikolov et al. (2013): Given an expression of the form  X is for France what London is for the UK d, X can be approximated by the simple vector arithmetic operation London-UK+France. The authors argue that this simple method can only capture very specific forms of analogies, and they present a measure that aims at identifying a wider range of relations in a more effective way.  I admit I find the idea of a single vector space model being able to capture a number of semantic relationships and analogies rather radical and infeasible. As the authors mention in the paper, a number of studies already suggest for the opposite. The reason is quite simple: behind all these models lies (some form of) the distributional hypothesis (words in similar contexts have similar meanings), and this poses certain limitations in their expressive abilities; for example, words like  big d and  small d will always be considered as semantically similar from a vector perspective (although they express opposite meanings), since they occur in similar contexts. So I cannot see how the example given in Figure 1 is relevant to the very nature of vector spaces (or to any other semantic model for that matter!): there is a certain analogy between  man-king d, and  woman-queen d, but asking from a word space to capture  has-a d relationships of the form  owl-has-claws d, hence  hospital-has-walls d, doesn t make much sense to me.  The motivation behind the main proposal of the paper (a similarity measure that involves a form of cross-comparison between vectors of words and vectors representing the contexts of the words) is not clearly explained. Further, the measure is tested on the relation categories of the SemEval 2010 task with rather unsatisfactory results; in almost all cases, a simple baseline that takes into account only partial similarities between the tested word pairs present very high performance, with a difference from the best-performing model which seems to me statistically insignificant. So from both a methodological and an experimental perspective, the paper has weaknesses, and in its current form seems to describe work in progress;  as such I am inclined against its presentation in ACL.  (Formatting issue: The authors use the LaTeX styles for ACL 2016   this should be fixed in case the paper is accepted).  AUTHORS RESPONSE ================ Thank you for the clarifications. I am still not comfortable with the idea of a metric or a vector space that tries to capture both semantic and relational similarity, and I think you don't present enough experimental evidence that your method works. I have to agree with one of the other reviewers that a more appropriate format for this work would be a short paper.


Creating Training Corpora for NLG Micro-Planning
In this paper, we focus on how to create data-to-text corpora which can support the learning of wide-coverage microplanners i.e., generation systems that handle lexicalisation, aggregation, surface realisation, sentence segmentation and referring expression generation. We start by reviewing common practice in designing training benchmarks for Natural Language Generation. We then present a novel framework for semi-automatically creating linguistically challenging NLG corpora from existing Knowledge Bases. We apply our framework to DBpedia data and compare the resulting dataset with (Wen et al., 2016)’s dataset. We show that while (Wen et al., 2016)’s dataset is more than twice larger than ours, it is less diverse both in terms of input and in terms of text. We thus propose our corpus generation framework as a novel method for creating challenging data sets from which NLG models can be learned which are capable of generating text from KB data.
1 Introduction
To train Natural Language Generation (NLG) systems, various input-text corpora have been developed which associate (numerical, formal, linguistic) input with text. As discussed in detail in Section 2, these corpora can be classified into three main types namely, (i) domain specific corpora, (ii) benchmarks constructed from “Expert” Linguistic Annotations and (iii) crowdsourced benchmarks1.
1We ignore here (Lebret et al., 2016)’s dataset which was created fully automatically from Wikipedia by associating infoboxes with text because this dataset fails to ensure an
In this paper, we focus on how to create datato-text corpora which can support the learning of wide-coverage micro-planners i.e., generation systems that handles such NLG subtasks as lexicalisation (mapping data to words), aggregation (exploiting linguistic constructs such as ellipsis and coordination to avoid repetition), surface realisation (using the appropriate syntactic constructs to build sentences), sentence segmentation and referring expression generation.
We start by reviewing the main existing types of NLG benchmarks and we argue for a crowdsourcing approach where data units are automatically built from an existing knowledge base and where text is crowdsourced from the data (Section 2).
We then propose a generic framework for semiautomatically creating training corpora for NLG (Section 3) from existing Knowledge Bases. In Section 4, we apply this framework to DBpedia data and we compare the resulting dataset with (Wen et al., 2016)’s using various metrics to evaluate the linguistic and computational adequacy of both datasets. By applying these metrics, we show that while (Wen et al., 2016)’s dataset is more than twice larger than ours, it is less diverse both in terms of input and in terms of text. We also compare the performance of a sequence-to-sequence model (Vinyals et al., 2015) on both datasets to estimate the complexity of the learning task induced by each dataset. We show that the performance of this neural model is much lower on the new data set than on the existing ones. We thus propose our corpus generation framework as a novel method for creating challenging data sets from which NLG
adequate match between data and text. We manually examined 50 input/output pairs randomly extracted from this dataset and did not find a single example where data and text matched. As such, this dataset is ill-suited for training microplanners. Moreover, since its texts contain both missing and additional information, it cannot be used to train joint models for content selection and micro-planning either.
models can be learned which are capable of generating complex texts from KB data.
2 NLG Benchmarks
Domain Specific Benchmarks. Several domain specific data-text corpora have been built by researchers to train and evaluate NLG systems. In the sports domain, Chen and Mooney (2008) constructed a dataset mapping soccer games events to text which consists of 1,539 data-text pairs and a vocabulary of 214 words. For weather forecast generation, (Liang et al., 2009)’s dataset includes 29,528 data-text pairs with a vocabulary of 345 words. For the air travel domain, Ratnaparkhi (2000) created a dataset consisting of 5,426 datatext pairs with a richer vocabulary (927 words) and in the biology domain, the KBGen shared task (Banik et al., 2013) made available 284 data-text pairs where the data was extracted from an existing knowledge base and the text was authored by biology experts.
An important limitation of these datasets is that, because they are domain specific, systems learned from them are restricted to generating domain specific, often strongly stereotyped text (e.g., weather forecast or soccer game commentator reports). Arguably, training corpora for NLG should support the learning of wide-coverage generators. By nature however, domain specific corpora restrict the lexical and often the syntactic coverage of the texts to be produced and thereby indirectly limit the expressivity of the generators trained on them.
Benchmarks Constructed from “Expert” Linguistic Annotations. NLG benchmarks have also been proposed where the input data is either derived from dependency parse trees (SR’11 task, (Belz et al., 2011)) or constructed through manual annotation (AMR Corpus (Banarescu et al., 2012)). Contrary to the domain-specific data sets just mentioned, these corpora have a wider coverage and are large enough for training systems that can generate linguistically sophisticated text.
One main drawback of these benchmarks however is that their construction required massive manual annotation of text with complex linguistic structures (parse trees for the SR task and Abstract Meaning Representation for the AMR corpus). Moreover because these structures are complex, the annotation must be done by experts. It cannot be delegated to the crowd. In short, the creation of such benchmark is costly both in terms
of time and in terms of expertise. Another drawback is that, because the input representation derived from a text is relatively close to its surface form2, the NLG task is mostly restricted to surface realisation (mapping input to sentences). That is, these benchmarks give very limited support for learning models that can handle micro-planning NLG subtasks such as lexicalisation, aggregation, sentence segmentation and referring expression generation.
Crowdsourced Benchmarks. More recently, data-to-text benchmarks have also been created by associating data units with text using crowdsourcing.
Wen et al. (2016) first created data by enumerating all possible combinations of 14 dialog act types (e.g., request, inform) and attribute-value pairs present in four small-size, hand-written ontologies about TVs, laptops, restaurants and hotels. They then use crowdsourcing to associate each data unit with a text. The resulting dataset is both large and varied (4 domains) and was successfully exploited to train neural and imitation learning data-to-text generator (Wen et al., 2016; Lampouras and Vlachos, 2016). Similarly, Novikova and Rieser (2016) described a framework for collecting data-text pairs using automatic quality control measures and evaluating how the type of the input representations (text vs pictures) impacts the quality of crowdsourced text.
The crowdsourcing approach to creating inputtext corpora has several advantages.
First, it is low cost in that the data is produced automatically and the text is authored by a crowdworker. This is in stark contrast with the previous approach where expert linguists are required to align text with data.
Second, because the text is crowd-sourced from the data (rather than the other way round), there is an adequate match between text and data both semantically (the text expresses the information contained in the data) and computationally (the data is sufficiently different from the text to require the learning of complex generation operations such as sentence segmentation, aggregation and referring expression generation).
2For instance, the input structures made available by the shallow track of the SR task contain all the lemmas present in the corresponding text. In this case, the generation task is limited to determining (i) the linear ordering and (ii) the full form of the word in the input.
Third, by exploiting small hand-written ontologies to quickly construct meaningful artificial data, the crowdsourcing approach allows for the easy creation of a large dataset with data units of various size and bearing on different domains. This, in turn, allows for better linguistic coverage and for NLG tasks of various complexity since typically, inputs of larger size increases the need for complex microplanning operations.
3 A Framework for Creating Data-to-Text, Micro-Planning Benchmarks
While as just noted, the crowdsourcing approach presented in (Wen et al., 2016) has several advantages, it also has a number of shortcomings.
One important drawback is that it builds on artificial rather than “real” data i.e., data that would be extracted from an existing knowledge base. As a result, the training corpora built using this method cannot be used to train KB verbalisers i.e., generation systems that can verbalise KB fragments.
Another limitation concerns the shape of the input data. (Wen et al., 2016)’s data can be viewed as trees of depth one (a set of attributes-value pairs describing a single entity e.g., a restaurant or a laptop). As illustrated in Figure 1 however, there is a strong correlation between the shape of the input and the syntactic structure of the corresponding sentence. The path structure T1 where B is shared by two predicates (mission and operator) will favour the use of a participial or a passive subject relative clause. In contrast, the branching structure T2 will favour the use of a new clause with a pronominal subject or a coordinated VP. More generally, allowing for trees of deeper depth is necessary to indirectly promote the introduction in the benchmark of a more varied set of syntactic constructs to be learned by generators.
To address these issues, we introduce a novel method for creating data-to-text corpora from large knowledge bases such as DBPedia. Our method combines (i) a content selection module designed to extract varied, relevant and coherent data units from DBPedia with (ii) a crowdsourcing process for associating data units with human authored texts that correctly capture their meaning. Example 1 shows a data/text unit created by our method using DBPedia as input KB.
(1) a. (John E Blaha birthDate 1942 08 26) (John E Blaha birthPlace San Antonio) (John E Blaha occupation Fighter pilot)
b. John E Blaha, born in San Antonio on 1942-08-26, worked as a fighter pilot
Our method has the following features. First, it can be used to create a data-to-text corpus from any knowledge base where entities are categorised and there is a large number of entities belonging to the same category. As noted above, this means that the resulting corpus can be used to train KB verbalisers i.e., generators that are able to verbalise fragments of existing knowledge bases. It could be used for instance, to verbalise fragments of e.g., MusicBrainz3, FOAF4 or LinkedGeoData5.
Second, as crowdworkers are required to enter text that matches the data and a majority vote validation process is used to eliminate mis-matched pairs, there is a direct match between text and data. This allows for a clear focus on the non content selection part of generation known as microplanning.
Third, because data of increasing size is matched with texts ranging from simple clauses to short texts consisting of several sentences, the resulting benchmark is appropriate for exercising the main subtasks of microplanning. For instance, in Example (1) above, given the input shown in (1a), generating (1b) involves lexicalising the occupation property as the phrase worked as (lexicalisation); using PP coordination (born in San Antonio on 1942-08-26) to avoid repeating the word born (aggregation); and verbalising the three triples using a single complex sentence including an apposition, a PP coordination and a transitive verb construction (sentence segmentation and surface realisation).
3https://musicbrainz.org/ 4http://www.foaf-project.org/ 5http://linkedgeodata.org/
3.1 DBPedia
To illustrate the functioning of our benchmark creation framework, we apply it to DBPedia. DBPedia is a multilingual knowledge base that was built from various kinds of structured information contained in Wikipedia (Mendes et al., 2012). This data is stored as RDF (Resource Description Format) triples of the form (subject, property, object) where the subject is a URI (Uniform Resource Identifier), the property is a binary relation and the object is either a URI or a literal value such as a string, a date or a number. We use an English version of the DBPedia knowledge base which encompasses 6.2M entities, 739 classes, 1,099 properties with reference values and 1,596 properties with typed literal values.6
3.2 Selecting Content
To create data units, we follow the procedure outlined in (Perez-Beltrachini et al., 2016) and sketched in Figure 2. This method can be summarised as follows.
First, DBPedia category graphs are extracted from DBPedia by retrieving up to 500 entity graphs for entities of the same category7. For example, we build a category graph for the Astronaut category by collecting, graphs of depth five for 500 entities of types astronaut.
Next, category graphs are used to learn bigram models of DBPedia properties which specify the probability of two properties co-occuring together. Three types of bi-gram models are extracted from category graphs using the SRILM toolkit: one model (S-Model) for bigrams occurring in sibling triples (triples with a shared subject); one model (C-Model) for bigrams occurring in chained triples (the object of one triple is the subject of the other); and one model (M-Model) which is a linear interpolation of the sibling and the chain model. The intuition is that these sibling and chain models capture different types of coherence, namely, topic-based coherence for the S-Model and discourse-based coherence for the CModel.
Finally, the content selection task is formulated as an Integer Linear Programming (ILP) problem to select, for a given entity of category C and its
6http://wiki.dbpedia.org/ dbpedia-dataset-version-2015-10
7An entity graph for some entity e is a graph obtained by traversing the DBPedia graph starting in e and stopping at depth five.
entity graph Ge, subtrees of Ge with maximal bigram probability and varying size (between 1 and 7 RDF triples).
We applied this content selection procedure to the DBPedia categories Astronaut (A), Building (B), Monument (M), University (U), Sports team (S) and Written work (W), using the three bi-gram models (S-Model, C-Model, M-Model) and making the number of triples required by the ILP constraint to occur in the output solutions vary between 1 and 7. The results are shown in Table 1. An input is a set of triples produced by the content selection module. The number of input is thus the number of distinct sets of triples produced by this module. In contrast, input patterns are inputs where subject and object have been abstracted over. That is, the number of input patterns is the number of distinct sets of properties present in the set of inputs. The number of properties is the number of distinct RDF properties occurring in the dataset. Similarly, the number of entities is the number of distinct RDF subjects and objects occurring in each given dataset.
3.3 Associating Content with Text
We associate data with text using the Crowdflower platform8. We do this in four main steps as follows.
1. Clarifying Properties. One difficulty when collecting texts verbalising sets of DBPedia triples
8http://www.crowdflower.com
is that the meaning of DBPedia properties may be unclear. We therefore first manually clarified for each category being worked on, those properties which have no obvious lexicalisations (e.g., crew1up was replaced by commander).
2. Getting Verbalisations for Single Triples. Next, we collected three verbalisations for data units of size one, i.e. single triples consisting of a subject, a property and an object. For each such input, crowdworkers were asked to produce a sentence verbalising its content. We used both a priori automatic checks to prevent spamming and a posteriori manual checks to remove incorrect verbalisations. We also monitored crowdworkers as they entered their input and banned those who tried to circumvent our instructions and validators. The automatic checks comprise 12 custom javascript validators implemented in the CrowdFlower platform to block contributor answers which fail to meet requirements such as the minimal time a contributor should stay on page, the minimal length of the text produced, the minimal match of tokens between a triple and its verbalisation and various format restrictions used to detect invalid input. The exact match between a triple and its verbalisation was also prohibited. In addition, after data collection was completed, we manually checked each data-text pair and eliminated from the data set any pair where the text either did not match the information conveyed by the triple or was not a well-formed English sentence.
3. Getting Verbalisations for Input containing more than one Triple. The verbalisations collected for single triples were used to construct input with bigger size. Thus, for input with a number of triples more than one, the crowd was asked to merge the sentences corresponding to each triple (obtained in step 2) into a natural sounding text. In such a way, we diminish the risk of having misinterpretations of the original semantics of a data unit. Contributors were also encouraged to change the order, and the wording of sentences, while writing their texts. For each data unit, we collected three verbalisations.
4. Verifying the Quality of the Collected Texts. The verbalisations obtained in Step 3 were verified through crowdsourcing. Each verbalisation collected in Step 3 was displayed to CrowdFlower contributors together with the corresponding set of triples. Then the crowd was asked to assess its
fluency, semantic adequacy, and grammaticality. Those criteria were checked by asking the following three questions: Does the text sound fluent and natural?, Does the text contain all and only the information from the data?, Is the text good English (no spelling or grammatical mistakes)?. We collected five answers per verbalisation. A verbalisation was considered as bad, if it received three negative answers in at least one criterion. After the verification step, the total corpus loss was of 8.7%.
Table 2 shows some statistics about the text obtained using our crowdsourcing procedure.
4 Comparing Benchmarks
We now compare a dataset created using our dataset creation framework (henceforth DBPNLG) with (Wen et al., 2016)’s dataset9 (henceforth, RNNLG). Example 2 shows a sample data-text pair taken from the RNNLG dataset. The DBPNLG dataset has been uploaded with this submission. (2) Dialog Moves
recommend(name=caerus 33;type=television; screensizerange=medium;family=t5;hasusbport=true) The caerus 33 is a medium television in the T5 family that’s USB-enabled
As should be clear from the discussion in Section 2 and 3, both datasets are similar in that, in both cases, data is built from ontological information and text is crowdsourced from the data. An important difference between the two datasets is that, while the RNNLG data was constructed by enumerating possible combinations of dialog act types and attribute-value pairs, the DBPNLG data is created using a sophisticated content selection procedure geared at producing sets of data units that are relevant for a given ontological category and that are varied in terms of size, shape and content (Perez-Beltrachini et al., 2016). We now investigate the impact of this difference on the two datasets (DBPNLG and RNNLG). To assess the degree to which both datasets support the generation of linguistically varied text requiring complex microplanning operations, we examine a number of data and text related metrics. We also compare the results of an out-of-the-box sequence-to-sequence model as a way to estimate the complexity of the learning task induced by each dataset.
4.1 Data Comparison
Terminology. The attributes in (Wen et al., 2016)’s dataset can be viewed as binary relations between
9https://github.com/shawnwun/RNNLG
the object talked about (a restaurant, a laptop, a TV or a hotel) and a value. Similarly, in DBPNLG, DBpedia RDF properties relate a subject entity to an object which can be either an entity or a datatype value. In what follows, we refer to both as attributes.
Table 3 shows several statistics which indicate that, while the RNNLG dataset is larger than DBPNLG, DBPNLG is much more diverse in terms of attributes, input patterns and input shapes.
Number of attributes. As illustrated in Example (3) below, different attributes can be lexicalised using different parts of speech. A dataset with a larger number of attributes is therefore more likely to induce texts with greater syntactic variety.
(3) Verb: X title Y / X served as Y Relational noun: X nationality Y / X’s nationality is Y Preposition: X country Y / X is in Y Adjective: X nationality USA / X is American
As shown in Table 3, DBPNLG has a more diverse attribute set than RNNLG both in absolute (172 attributes in DBPNLG against 108 in RNNLG) and in relative terms (RNNLG is a little more than twice as large as DBPNLG).
Number of Input Patterns. Since attributes may give rise to lexicalisation with different parts of speech, the sets of attributes present in an input (input pattern10) indirectly determine the syntactic realisation of the corresponding text. Hence a higher number of input patterns will favour a higher number of syntactic realisations. This is exemplified in Example (4) where two inputs with the same number of attributes give rise to texts with different syntactic forms. While in Example (4a), the attribute set { country, location, startDate } is realised by a passive (is located), an apposition (Australia) and a deverbal nominal (its construction), in Example (4b), the attribute set { almaMater, birthPlace, selection } induced a passive (was born) and two VP coordinations (graduated and joined).
10Recall from section 3 that input patterns are inputs where subjects and objects have been remove thus, in essence, an input pattern is the set of all the attributes occurring in a given input.
(4) a. (‘108 St Georges Terrace location Perth’, ‘Perth country Australia’, ‘108 St Georges Terrace startDate 1981’) country, location, startDate 108 St. Georges Terrace is located in Perth, Australia. Its construction began in 1981. passive, apposition, deverbal nominal
b. (‘William Anders selection 1963’, ‘William Anders birthPlace British Hong Kong’, ‘William Anders almaMater ”AFIT, M.S. 1962”’) almaMater, birthPlace, selection William Anders was born in British Hong Kong, graduated from AFIT in 1962, and joined NASA in 1963. passive, VP coordination, VP coordination
Again, despite the much larger size of the RNNLG dataset, the number of input patterns in both datasets is almost the same. That is, the relative variety in input patterns is higher in DBPNLG.
Number of Input / Number of Input Patterns. The ratio between number of inputs and the number of input patterns has an important impact both in terms of linguistic diversity and in terms of learning complexity. A large ratio indicates a “repetitive dataset” where the same pattern is instantiated a high number of times. While this facilitates learning, this also reduces linguistic coverage (less combinations of structures can be learned) and may induce overfitting. Note that because datasets are typically delexicalised when training NLG models (cf. e.g., (Wen et al., 2015; Lampouras and Vlachos, 2016)) , at training time, different instantiations of the same input pattern reduce to identical input.
The two datasets markedly differ on this ratio which is five times lower in DBPNLG. While in DBPNLG, the same pattern is instantiated in average 2.40 times, it is instantiated 10.31 times in average in RNNLG. From a learning perspective, this means that the RNNLG dataset facilitates learning but also makes it harder to assess how well systems trained on it can generalise to handle unseen input.
Input Shape. As mentioned in Section 3, in the RNNLG dataset, all inputs can be viewed as trees of depth one while in the DBPNLG dataset, input may have various shapes. As a result, RNNLG texts will be restricted to syntactic forms which permit expressing such multiple predications of the same
entity e.g., subject relative clause, VP and sentence coordination etc. In contrast, the trees extracted by the DBPNLG content selection procedure may be of depth five and therefore allow for further syntactic constructs such as object relative clause and passive participials (cf. Figure 1).
We can show this empirically as well that DBPNLG is far more diverse than RNNLG in terms of input shapes. The RNNLG dataset has only 6 distinct shapes and all of them are of depth 1, i.e., all (attribute, value) pairs in an input are siblings to each other. In contrast, the DBPNLG dataset has 58 distinct shapes, out of which only 7 shapes are with depth 1, all others have depth more than 1 and they cover 49.6% of all inputs.
4.2 Text Comparison
Table 4 gives some statistics about the texts contained in each dataset.
(5) a. (Alan Bean birthDate “1932-03-15”) Alan Bean was born on March 15, 1932
(6) a. (‘Alan Bean nationality United States’, ‘Alan Bean birthDate “1932-03-15”’, ‘Alan Bean almaMater “UT Austin, B.S. 1955”’, ‘Alan Bean birthPlace Wheeler, Texas’, ‘Alan Bean selection 1963’) Alan Bean was an American astronaut, born on March 15, 1932 in Wheeler, Texas. He received a Bachelor of Science degree at the University of Texas at Austin in 1955 and was chosen by NASA in 1963.
As illustrated by the contrast between example 5 and 6 above, text length (number of tokens per text) and the number of sentences per text are strong indicators of the complexity of the generation task. We use the Stanford Part-Of-Speech Tagger and Parser version 3.5.2 (date 2015-04-20) to tokenize and to perform sentence segmentation on text. As shown in Table 4, DBPNLG’s texts are longer both in terms of tokens and in terms of number of sentences per text. Another difference between the two datasets is that DBPNLG contains a
higher number of text per input thereby providing a better basis for learning paraphrases.
The size and the content of the vocabulary is another important factor in ensuring the learning of wide coverage generators. While a large vocabulary makes the learning problem harder, it also allows for larger coverage. DBPNLG exhibits a higher corrected type-token ratio (CTTR), which indicates greater lexical variety, and higher lexical sophistication (LS). Lexical sophistication measures the proportion of relatively unusual or advanced word types in the text. In practice, LS is the proportion of lexical word types (lemma) which are not in the list of 2,000 most frequent words generated from the British National Corpus11. Typetoken ratio (TTR) is a measure of diversity defined as the ratio of the number of word types to the number of words in a text. To address the fact that this ratio tends to decrease with the size of the corpus, corrected TTR can be used to control for corpus size. It is defined as T/ √ 2N , where T is the number of types and N the number of tokens. Overall, the results shown in Table 4 indicate that DBPNLG texts are both lexically more diverse (higher corrected type/token ratio) and more sophisticated (higher proportion of unfrequent words) than RNNLG’s. They also show a proportionately larger vocabulary for DBPNLG (2992 types for 290479 tokens in DBPNLG against 3524 types for 531871 tokens in RNNLG).
4.3 Neural Generation
Richer and more varied datasets are harder to learn from. As a proof-of-concept study of the comparative difficulty of the two datasets with respect to machine learning, we compare the performance of a sequence-to-sequence model for generation on both datasets. In particular, we use a multilayered sequence-to-sequence model with an at-
11We compute LS and CTTR using the Lexical Complexity Analyzer developed by Lu (2012).
tention mechanism (Vinyals et al., 2015).12 The model was trained with 3 layers of 512 units each. To allow for a fair comparison, we use a similar amount of data (13K data-text pairs) for both datasets. As RNNLG is bigger in size than DBPNLG, we constructed a balanced sample of RNNLG which included equal number of instances per category (tv, laptop, etc). We use a 3:1:1 ratio for training, developement and testing. The training was done in two delexicalisation modes: fully and name only. In case of fully delexicalisation, all entities were replaced by their generic terms, whereas in name only mode only subjects were modified in that way. For instance, the triple FC Köln manager Peter Stöger was delexicalised as SportsTeam manager Manager in the first mode, and as SportsTeam manager Peter Stöger in the second mode. The delexicalisation in sentences was done using the exact match between entities and tokens.
Table 5 shows the perplexity results. In both modes, RNNLG yielded lower scores than DBPNLG. This is inline with the observations made above concerning the higher data diversity, larger vocabulary and more complex texts of DBPNLG. Similary, the BLEU score of the generated sentences (Papineni et al., 2002) is lower for DBPNLG suggesting again a dataset that is more complex and therefore more difficult to learn from.
5 Conclusion
We presented a framework for building NLG datato-text training corpora from existing knowledge bases.
One feature of our framework is that datasets created using this framework can be used for training and testing KB verbalisers an in particular,
12We used the TensorFlow code available at https://github.com/tensorflow/models/ tree/master/tutorials/rnn/translate. Alternatively, we could have used (Wen et al., 2016)’s implementation which is optimised for generation. However the code is geared toward dialog acts and modifying it to handle RDF triples is non trivial. Since the comparison aims at examining the relative performance of the same neural network on the two datasets, we used the tensor flow implementation instead.
verbalisers for RDF knowledge bases. Following the development of the semantic web, many large scale datasets are encoded in the RDF language (e.g., MusicBrainz, FOAF, LinkedGeoData) and official institutions13 increasingly publish their data in this format. In this context, our framework is useful both for creating training data from RDF KB verbalisers and to increase the number of datasets available for training and testing NLG.
Another important feature of our framework is that it permits creating semantically and linguistically diverse datasets which should support the learning of lexically and syntactically, wide coverage micro-planners. We applied our framework to DBpedia data and showed that although twice smaller than the largest corpora currently available for training data-to-text microplanners, the resulting dataset is more semantically and linguistically diverse. Despite the disparity in size, the number of attributes is comparable in the two datasets. The ratio between input and input patterns is five times lower in our dataset thereby making learning harder but also diminishing the risk of overfitting and providing for wider linguistic coverage. Conversely, the ratio of text per input is twice higher thereby providing better support for learning paraphrases.
We are currently working on further extending the DBPNLG dataset and once completed, will make it available as part of a shared task for evaluating data-to-text micro-planners. While we only report on a dataset developed using 6 DBpedia categories, we have collected content for 14 further categories using the content selection procedure described in Section 3 and will collect the corresponding texts using our selective crowdsourcing procedure.
Recently, several sequence-to-sequence models have been proposed for generation. Our experiments suggest that these are not optimal when it comes to generate linguistically complex texts from rich data. More generally, they indicate that the data-to-text corpora built by our framework are challenging for such models. We hope that the DBPNLG dataset which we will make available in the shared task will drive the deep learning community to take up this new challenge and work on the development of neural generators that can handle the generation of linguistically rich texts.
13See http://museum-api.pbworks.com for examples.

- Strengths:  This paper presents a step in the direction of developing more challenging corpora for training sentence planners in data-to-text NLG, which is an important and timely direction.   - Weaknesses:  It is unclear whether the work reported in this paper represents a substantial advance over Perez-Beltrachini et al.'s (2016) method for selecting content.  The authors do not directly compare the present paper to that one. It appears that the main novelty of this paper is the additional analysis, which is however rather superficial.  It is good that the authors report a comparison of how an NNLG baseline fares on this corpus in comparison to that of Wen et al. (2016).  However, the BLEU scores in Wen et al.'s paper appear to be much much higher, suggesting that this NNLG baseline is not sufficient for an informative comparison.  - General Discussion:  The authors need to more clearly articulate why this paper should count as a substantial advance over what has been published already by Perez-Beltrachini et al, and why the NNLG baseline should be taken seriously.  In contrast to LREC, it is not so common for ACL to publish a main session paper on a corpus development methodology in the absence of some new results of a system making use of the corpus.  The paper would also be stronger if it included an analysis of the syntactic constructions in the two corpora, thereby more directly bolstering the case that the new corpus is more complex.  The exact details of how the number of different path shapes are determined should also be included, and ideally associated with the syntactic constructions.  Finally, the authors should note the limitation that their method does nothing to include richer discourse relations such as Contrast, Consequence, Background, etc., which have long been central to NLG. In this respect, the corpora described by Walker et al. JAIR-2007 and Isard LREC-2016 are more interesting and should be discussed in comparison to the method here.  References  Marilyn Walker, Amanda Stent, Fran u00e7ois Mairesse, and Rashmi Prasad. 2007. Individual and domain adaptation in sentence planning for dialogue. Journal of Artificial Intelligence Research (JAIR), 30:413 456.  Amy Isard, 2016.  The Methodius Corpus of Rhetorical Discourse Structures and Generated Texts d , Proceedings of the Tenth Conference on Language Resources and Evaluation (LREC 2016), Portoro u017e, Slovenia, May 2016.  --- Addendum following author response:  Thank you for the informative response.  As the response offers crucial clarifications, I have raised my overall rating.  Re the comparison to Perez-Beltrachini et al.: While this is perhaps more important to the PC than to the eventual readers of the paper, it still seems to this reviewer that the advance over this paper could've been made much clearer.  While it is true that Perez-Beltrachini et al.  just  cover content selection, this is the key to how this dataset differs from that of Wen et al.  There doesn't really seem to be much to the  complete methodology  of constructing the data-to-text dataset beyond obvious crowd-sourcing steps; to the extent these steps are innovative or especially crucial, this should be highlighted.  Here it is interesting that 8.7% of the crowd-sourced texts were rejected during the verification step; related to Reviewer 1's concerns, it would be interesting to see some examples of what was rejected, and to what extent this indicates higher-quality texts than those in Wen et al.'s dataset.  Beyond that, the main point is really that collecting the crowd-sourced texts makes it possible to make the comparisons with the Wen et al. corpus at both the data and text levels (which this reviewer can see is crucial to the whole picture).  Re the NNLG baseline, the issue is that the relative difference between the performance of this baseline on the two corpora could disappear if Wen et al.'s substantially higher-scoring method were employed.  The assumption that this relative difference would remain even with fancier methods should be made explicit, e.g. by acknowledging the issue in a footnote.  Even with this limitation, the comparison does still strike this reviewer as a useful component of the overall comparison between the datasets.  Re whether a paper about dataset creation should be able to get into ACL without system results:  though this indeed not unprecedented, the key issue is perhaps how novel and important the dataset is likely to be, and here this reviewer acknowledges the importance of the dataset in comparison to existing ones (even if the key advance is in the already published content selection work).  Finally, this reviewer concurs with Reviewer 1 about the need to clarify the role of domain dependence and what it means to be  wide coverage  in the final version of the paper, if accepted.