Evaluation Metrics for Reading Comprehension: Prerequisite Skills and Readability
Knowing the quality of reading comprehension (RC) datasets is important for the development of natural language understanding systems. In this study, two classes of metrics were adopted for evaluating RC datasets: prerequisite skills and readability. We applied these classes to six existing datasets, including MCTest and SQuAD, and demonstrated the characteristics of the datasets according to each metric and the correlation between the two classes. Our dataset analysis suggested that the readability of RC datasets does not directly affect the question difficulty and that it is possible to create an RC dataset that is easy-to-read but difficult-to-answer.
1 Introduction
A major goal of natural language processing (NLP) is to develop agents that can understand natural language. Such an ability can be tested with a reading comprehension (RC) task that requires the agent to read open-domain documents and answer questions about them. Building the RC ability is challenging because RC comprises multiple processes including parsing, understanding cohesion, and inference with linguistic and general knowledge.
Clarifying what a system achieves is important to the development of RC systems. To achieve robust improvement, systems need to be measured according to various metrics, not just simple accuracy. However, a current problem is that most RC datasets are presented only with superficial categories, such as question types (e.g., what, where, and who) and answer types (e.g., numeric, location, and person). In addition, Chen et al. (2016) revealed that some questions in datasets may not
have the quality to test RC systems. In these situations, it is difficult to assess systems accurately.
As Norvig (1989) stated, questions easy for humans often turn out to be difficult for systems. For example, see the two RC questions in Figure 1. In the first example from SQuAD (Rajpurkar et al., 2016), although the document is taken from a Wikipedia article and is thus written for adults, the question is easy to solve simply look at one sentence and select an entity. On the other hand, in the second example from MCTest (Richardson et al., 2013), the document is written for children and is easy to read, but the question requires reading multiple sentences and a combination of several skills, such as understanding of causal relations (Sara wanted... → they went to...), coreference resolution (Sara and Her Dad = they), and complementing ellipsis (baseball team = team). These two examples show that the readability of the text does not necessarily correlate with the difficulty of the questions. Nevertheless, the accompanying categories of existing RC datasets cannot
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
provide any way to analyze this issue.
In this study, our goal was to investigate how the two difficulties of “answering questions” and “reading text” relate in RC. Therefore, we adopted two classes of evaluation metrics for RC datasets to analyze both the quality of datasets and the performance of systems. Our paper is divided into the following sections. First, we adopt the two classes: prerequisite skills and readability (Section 3). We then specify evaluation metrics for each (Sections 3.1 and 3.2). Next, we annotate six existing RC datasets with these metrics (Section 4). Finally, we present our annotation results (Section 5) and discuss them (Section 6).
Our two classes of metrics are based on McNamara and Magliano (2009)’s analysis of human text comprehension in psychology. The first class defines the difficulty of comprehending the context to answer questions. We adopted prerequisite skills proposed in a previous study by Sugawara et al. (2017). That study presented an important observation of the relation between the difficulty of an RC task and prerequisite skills: the more skills that are required to answer a question, the more difficult the question is. From this observation, we assume that the number of required skills corresponds to the difficulty of a question. This is because each skill corresponds to a function of a system, which has to be equipped with the system. However, a problem in previous studies, including that of Sugawara and Aizawa (2016), is that they analyzed only two datasets and that their categorization of knowledge reasoning is provisional with a weak theoretical background.
Therefore, we reorganized the category of knowledge reasoning in terms of textual entailment and human text comprehension. In research on textual entailment, several methodologies have been proposed for precise analysis of entailment phenomena (Dagan et al., 2013; LoBue and Yates, 2011). In psychology research, Kintsch (1993) proposed dichotomies for the classification of human inferences: retrieved versus generated. In addition, McNamara and Magliano (2009) proposed a similar distinction for inferences: bridging versus elaboration. We utilized these insights in order to develop a comprehensive but not overly specific classification of knowledge reasoning.
The second class defines the difficulty of reading contents, readability, in documents considering syntactic and lexical complexity. We leverage
a wide range of linguistic features proposed by Vajjala and Meurers (2012).
In the annotation, annotators selected sentences needed for answering and then annotated them with prerequisite skills under the same condition for RC datasets with different task formulations. Therefore, our annotation was equivalent in that the datasets were annotated from the point of view of whether a context entails a hypothesis that can be made from a question and its answer. This means our methodology could not evaluate the competence of looking for sentences that need to be read and answer candidates from the context. Therefore, our methodology was used to evaluate the understanding of contextual entailments in a broader sense for RC.
The contributions of this paper are as follows:
1. We adopt two classes of evaluation metrics to show the qualitative features of RC datasets. Through analyses of RC datasets, we demonstrate that there is only a weak correlation between the difficulty of questions and the readability of context texts in RC datasets.
2. We revise the previous classification of prerequisite skills for RC. Specifically, skills of knowledge reasoning are organized in terms of entailment phenomena and human text comprehension in psychology.
3. We annotate six existing RC datasets with our organized metrics for the comparison and make the results publicly available.
We believe that our annotation results will help researchers to develop a method for the stepby-step construction of better RC datasets and a method to improve RC systems.
2 Related Work
2.1 Reading Comprehension Datasets
In this section, we present a short chronicle of RC datasets. To our knowledge, Hirschman et al. (1999) were the first to use NLP methods for RC. Their dataset consisted of reading materials for grades 3–6 with simple 5W (wh-) questions. Afterwards, investigations into natural language understanding questions mainly focused on other formulations, such as question answering (Yang et al., 2015; Wang et al., 2007; Voorhees et al., 1999) and textual entailment (Bentivogli et al., 2010; Sammons et al., 2010; Dagan et al., 2006).
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
One of the RC tasks following it was QA4MRE (Sutcliffe et al., 2013). The top accuracy of this task remained 59% at that time, and the size of the dataset was very limited: there were only 224 gold-standard questions, which is not enough for machine learning methods.
Thus, an important issue for designing RC datasets is their scalability. Richardson et al. (2013) presented MCTest, which is an opendomain narrative dataset for gauging comprehension at a childs level. This dataset was created by crowdsourcing and is based on scalable methodology. Since then, more large-scale datasets have been proposed with the development of machine learning. For example, the CNN/Daily Mail dataset (Hermann et al., 2015) and CBTest (Hill et al., 2016) have approximately 1.4M and 688K passages, respectively. These context texts and questions were automatically curated and generated from large corpora. However, Chen et al. (2016) indicated that approximately 25% of the questions in the CNN/Daily Mail dataset are unsolvable or nonsense. This has highlighted the demand for more stable and robust sourcing methods regarding dataset quality.
In addition to this, several RC datasets were presented in the last half of 2016 with large documents and sensible queries that were guaranteed by crowdsourcing or human testing. We explain those datasets in Section 4.2. They were aimed at achieving large and good-quality content for machine learning models. Nonetheless, as shown in the examples of Figure 1, there is room for improvement, and there is still no methodology available for evaluating the quality.
2.2 Reading Comprehension in Psychology
In psychology, there is a rich tradition of research on human text comprehension. The construction– integration (C–I) model (Kintsch, 1988) is one of the most basic and influential theories. This model assumes connectional and computational architecture for text comprehension. It assumes that comprehension is the processing of information based on the following steps:1
1. Construction: Read sentences or clauses as inputs; form and elaborate concepts and propositions corresponding to the inputs.
2. Integration: Associate the contents to consistently understand them (e.g., coreference,
1Note that this is only a very simplified overview.
discourse, and coherence).
During these steps, three levels of representation are constructed (Van Dijk and Kintsch, 1983): the surface code (i.e., wording and syntax), the textbase (i.e., text propositions with cohesion), and the situation model (i.e., mental representation). Based on the above assumptions, McNamara and Magliano (2009) proposed two aspects of text comprehension: “strategic/skilled comprehension” and “text ease of processing.” We employed these assumptions as the basis of two classes of evaluation metrics (Section 3).
On the other hand, Kintsch (1993) proposed two dichotomies for the classification of human inferences, including the knowledge-based ones that are performed in the C–I model. The first is whether inferences are automatic or controlled. However, Graesser et al. (1994) indicated that this distinction is ambiguous because there is a continuum between the two states that depends on individuals. Therefore, this dichotomy is not suited for empirical evaluation, which we are working on attempting. The second is whether inferences are retrieved or generated. Retrieved means that the information used for inference is retrieved from context. In contrast, when inferences are generated, the reader uses external knowledge going beyond the context.
A similar distinction was proposed by McNamara and Magliano (2009): bridging and elaboration. Bridging inference connects current information to other information that was previously encountered. Elaboration connects current information to external knowledge that is not in a context. We use these two types of inferences in the classification of knowledge reasoning.
3 Evaluation Metrics for Datasets
Based on McNamara and Magliano (2009)’s depiction of text comprehension, we adopted two classes for the evaluation of RC datasets: prerequisite skills and readability.
We refined the prerequisite skills (Section 3.1) for RC that were proposed by Sugawara et al. (2017) and Sugawara and Aizawa (2016) using the insights mentioned in the previous section. This class covers the textbase and situation model, or understanding each fact and associating multiple facts in a text: relations of events, characters, topic of story, and so on. This class also includes knowledge reasoning; this is divided into several met-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
rics according to the distinctions of human inferences, as discussed by Kintsch (1993) and McNamara and Magliano (2009), and according to the classification of entailment phenomena by Dagan et al. (2013) and LoBue and Yates (2011).
Readability metrics (Section 3.2) intuitively describe the difficulty of reading: vocabulary and the complexity of texts, or the surface code.
3.1 Prerequisite Skills
By refining Sugawara et al. (2017)’s ten reading comprehension skills, we reorganized thirteen prerequisite skills in total, which are presented below. Skills that have been modified/elaborated from the original definition are appended with an asterisk (∗), and new skills unique to this study are appended with a dagger (†).
1. Object tracking∗: Jointly tracking or grasping of multiple objects, including set or membership (Clark, 1975). This skill is a renamed version of list/enumeration used in the original classification to emphasize its scope for multiple objects.
2. Mathematical reasoning∗: We merged statistical and quantitative reasoning with mathematical reasoning. This skill is a renamed version of mathematical operations.
3. Coreference resolution∗: This skill has a small modification: it includes one anaphora (Dagan et al., 2013). This skill is similar to direct reference (Clark, 1975).
4. Logical reasoning∗: We reorganized this skill as understanding predicate logic, e.g., conditionals, quantifiers, negation, and transitivity. Note that mathematical reasoning and this skill are intended to align with offline skills mentioned by Graesser et al. (1994).
5. Analogy∗: Understanding of metaphors including metonymy and synecdoche. See LoBue and Yates (2011) for examples of synecdoche.
6. Causal relation: Understanding of causality that is represented by explicit expressions of why, because, the reason... (only if they exist).
7. Spatiotemporal relation: Understanding of spatial and/or temporal relationships between multiple entities, events, and states.
For commonsense reasoning in the original classification, we defined the following four categories, which can be jointly required.
8. Ellipsis†: Recognizing implicit/omitted information (argument, predicate, quantifier, time, place). This skill is inspired by Dagan et al. (2013)
and the discussion of Sugawara et al. (2017). 9. Bridging†: Inferences between two facts supported by grammatical and lexical knowledge (e.g., synonymy, hypernymy, thematic role, part of events, idioms, and apposition). This skill is inspired by the concept of indirect reference in the literature (Clark, 1975). Note that we excluded direct reference because it is coreference resolution (pronominalization) or elaboration (epithets).
10. Elaboration†: Inference using known facts, general knowledge (e.g., kinship, exchange, typical event sequence, and naming), and implicit relations (e.g., noun-compound and possessive) (see Dagan et al. (2013) for details). Bridging and elaboration are distinguished by whether knowledge used in inferences is grammatical/lexical versus general/commonsense.
11. Meta-knowledge†: Using knowledge including a reader, writer, and text genre (e.g., narratives and expository documents) from metaviewpoints (e.g., Who are the principal characters of the story? and What is the main subject of this article?). Although this skill can be regarded as part of elaboration, we defined an independent skill because this knowledge is specific to RC.
The last two skills are intended to be performed on a single sentence.
12. Schematic clause relation: Understanding of complex sentences that have coordination or subordination, including relative clauses.
13. Punctuation∗: Understanding of punctuation marks (e.g., parenthesis, dash, quotation, colon, and semicolon). This skill is a renamed version of special sentence structure. Concerning the original definition, we regarded “scheme” in figures of speech as ambiguous and excluded it (ellipsis was defined as a skill, and apposition was merged into bridging). We did the same with understanding of constructions, which was merged into idioms in bridging.
Note that we did not construct this classification to be dependent on RC models. This is because our methodology is intended to be general and applicable to many kinds of architectures.
3.2 Readability Metrics
In this study, we evaluated the readability of texts based on metrics in NLP. Several studies have examined readability for various applications, such
4http://en.wikipedia.org/wiki/ Academic_Word_List
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
- Ave. num. of characters per word (NumChar) - Ave. num. of syllables per word (NumSyll) - Ave. sentence length in words (MLS) - Proportion of words in AWL (AWL) - Modifier variation (ModVar) - Num. of coordinate phrases per sentence (CoOrd) - Coleman–Liau index (Coleman) - Dependent clause to clause ratio (DC/C) - Complex nominals per clause (CN/C) - Adverb variation (AdvVar)
Figure 2: Readability metrics. AWL refers to the Academic Word List.4
as second language learning (Razon and Barnden, 2015) and text simplification (Aluisio et al., 2010), and various aspects, such as development measures of second language acquisition (Vajjala and Meurers, 2012) and discourse relations (Pitler and Nenkova, 2008).
Of these, we used the classification of linguistic features proposed by Vajjala and Meurers (2012). This is because they presented a comparison of a wide range of linguistic features focusing on second language acquisition and their method can be applied to plain text.2
We list the readability metrics in Figure 2, which were reported by Vajjala and Meurers (2012) as the top ten features that affect human readability. To classify these metrics, we can place three classes: lexical features (NumChar, NumSyll, AWL, AdvVar, and ModVar), syntactic features (MLS, CoOrd, DC/C, and CN/C), and traditional features (Coleman). We applied these metrics only to sentences that needed to be read to answer questions. Nonetheless, because these metrics were proposed for human readability, they do not necessarily correlate with those for RC systems. Therefore, in a system analysis, we ideally have to consult various features.3
4 Annotation of Reading Comprehension Datasets
We annotated six existing RC datasets with the prerequisite skills. We explain the annotation procedure in Section 4.1 and the specifications of the annotated RC datasets in Section 4.2.
2Pitler and Nenkova (2008)’s work is more suitable for measuring text quality. However, we could not use their results because we did not have discourse annotations.
3We will make available the analysis of RC datasets on basic features as much as possible.
4.1 Annotation Procedure
We asked three annotators to annotate questions of RC datasets with the prerequisite skills that are required to answer each question. We allowed multiple labeling. For each task that was curated from the datasets, the annotators saw the context, question, and its answer jointly. When a dataset consisted of multiple choice questions, we showed all candidates and labeled the correct one with an asterisk. The annotators then selected sentences that needed to be read to answer the question and decided if each prerequisite skill was required. The annotators were allowed to select nonsense for an unsolvable question to distinguish it from a solvable question that required no skills.
4.2 Dataset Specifications
For the annotation, we randomly selected 100 questions from each dataset in Table 1. This amount of questions was considered to be sufficient for the analysis of RC datasets as performed by Chen et al. (2016). The questions were sampled from the gold-standard dataset of QA4MRE and the development sets of the other RC datasets. We explain the method of choosing questions for the annotation in Appendix A.
There were other datasets we did not annotate in this study. We decided not to annotate those datasets because of the following reasons. CNN/Daily Mail (Hermann et al., 2015) is anonymized and contains some errors (Chen et al., 2016), so it did not seem to be suitable for annotation. We considered CBTest (Hill et al., 2016) to be for language modeling tasks rather than RC task and excluded it. LAMBADA (Paperno et al., 2016)’s texts are formatted for machine reading,
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Skills QA4MRE MCTest SQuAD WDW MARCO NewsQA
1. Tracking 11.0 6.0 3.0 8.0 6.0 2.0 2. Math. 4.0 4.0 0.0 3.0 0.0 1.0 3. Coref. resol. 32.0 49.0 13.0 19.0 15.0 24.0 4. Logical rsng. 15.0 2.0 0.0 8.0 1.0 2.0 5. Analogy 7.0 0.0 0.0 7.0 0.0 3.0 6. Causal rel. 1.0 6.0 0.0 2.0 0.0 4.0 7. Sptemp rel. 26.0 9.0 2.0 2.0 0.0 3.0 8. Ellipsis 13.0 4.0 3.0 16.0 2.0 15.0 9. Bridging 69.0 26.0 42.0 59.0 36.0 50.0 10. Elaboration 60.0 8.0 13.0 57.0 18.0 36.0 11. Meta 1.0 1.0 0.0 0.0 0.0 0.0 12. Clause rel. 52.0 40.0 28.0 42.0 27.0 34.0 13. Punctuation 34.0 1.0 24.0 20.0 14.0 25.0
Nonsense 10.0 1.0 3.0 27.0 14.0 1.0
Table 2: Frequencies (%) of prerequisite skills needed for the RC datasets.
#Skills QA4MRE MCTest SQuAD WDW MARCO NewsQA
0 2.0 18.0 27.0 2.0 15.0 13.0 1 13.0 36.0 33.0 5.0 35.0 26.0 2 13.0 24.0 24.0 14.0 29.0 23.0 3 20.0 15.0 6.0 22.0 6.0 25.0 4 14.0 4.0 6.0 16.0 2.0 9.0 5 13.0 1.0 1.0 6.0 0.0 2.0 6 10.0 1.0 0.0 6.0 0.0 1.0 7 1.0 0.0 0.0 2.0 0.0 0.0 8 1.0 0.0 0.0 0.0 0.0 0.0 9 0.0 0.0 0.0 0.0 0.0 0.0 10 3.0 0.0 0.0 0.0 0.0 0.0
Ave. 3.25 1.56 1.28 2.43 1.19 1.99
Table 3: Frequencies (%) of the number of required prerequisite skills for the RC datasets.
and all tokens are lowercased, which seemingly prevents inferences based on proper nouns. Thus, we decided that its texts are not suitable for human reading and annotation.
5 Results of the Dataset Analysis
We present the results from evaluating the RC datasets according to the two classes of metrics. The inter-annotator agreement was 90.1% for 62 randomly sampled questions. The evaluation was performed according to the following four points of view (i)–(iv).
(i) Frequencies of prerequisite skills (Table 2): QA4MRE had the highest scores for frequencies among the datasets. This seems to reflect the fact that QA4MRE has technical documents that contain a wide range of knowledge (bridging and elaboration), multiple clauses, and punctuation and that the questions are devised by experts.
MCTest achieved high scores in several skills (first in causal relation and meta-knowledge and second in coreference resolution and spatiotemporal relation) and lower score in punctuation. These scores seem to be because the MCTest dataset consists of narratives.
Another dataset that achieved remarkable scores is Who-did-What. This dataset achieved the highest score for ellipsis. This is because the questions of Who-did-What are automatically generated from articles not used as context. This methodology can avoid textual overlap between a question and its context; therefore, the skills of ellipsis, bridging, and elaboration are frequently required.
With regard to nonsense, MS MARCO and Who-did-What received relatively high scores. This appears to have been caused by the automated curation, which may generate separation between the contents of the context and question (i.e., web segments and a search query in MS MARCO, and a context article and question article in Who-didWhat). In stark contrast, NewsQA had no nonsense questions. Although this result was affected by our filtering described in Appendix A, it is important to note that the NewsQA dataset includes annotations of meta information whether or not a question makes sense (is question bad).
(ii) Number of required prerequisite skills (Table 3): QA4MRE had the highest score; on average, each question required 4.6 skills. There were few questions in QA4MRE that required zero or one skill, whereas the other datasets contained some questions that required zero or one skill. Table 3 also indicates that more than 90% of the MS MARCO questions required fewer than three skills, at least according to the annotation.
(iii) Readability metrics for each dataset (Table 4): SQuAD and QA4MRE achieved the highest scores in most metrics; this reflects the fact that
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
1.0 1.5 2.0 2.5 3.0 3.5 Average number of required skills 2
4
6
8
10
12
14
16
FK
g ra
de le
ve l QA4MRE
MCTest
SQuAD WDW
MARCO NewsQA
Figure 3: Flesch–Kincaid grade levels and average number of required prerequisite skills for the RC datasets.
0 2 4 6 8 10 Number of required skills 10
0
10
20
30
40
50
FK
g ra
de le
ve l
QA4MRE MCTest SQuAD
Figure 4: Flesch–Kincaid grade levels and number of required prerequisite skills for all questions of the selected RC datasets.
Wikipedia articles and technical documents generally require a high grade level to understand. In contrast, MCTest had the lowest scores; its dataset consist of narratives for children.
(iv) Correlation between numbers of required prerequisite skills and readability metrics (Figure 3, Figure 4, and Table 5): Our main interest was in the correlation between prerequisite skills and readability. To investigate this, we examined the relation between the number of required prerequisite skills and readability metrics (represented by the Flesch–Kincaid grade level), as shown in Figure 3 for each dataset and in Figure 4 for each question. The first figure shows the trends of the datasets. QA4MRE was relatively difficult both to read and to answer, and SQuAD was difficult to read but easy to answer. For further investigation, we selected three datasets (QA4MRE, MCTest, and SQuAD) and plotted all of their questions in the second figure. Three separate domains can be seen.
Table 5 presents Pearson’s correlation coeffi-
cients between the number of required prerequisite skills and each readability metric for all questions of the RC datasets. Although there are weak correlations from 0.025 to 0.416, the results highlight that there is not necessarily a strong correlation between the two values. This leads to the following two insights. First, the readability of RC datasets does not directly affect the difficulty of their questions. That is, RC datasets that are difficult to read are not necessary difficult to answer. Second, it is possible to create difficult questions from context that is easy to read. MCTest is a good example. The context texts of MCTest dataset are easy to read, but the difficulty of the questions is comparable to that of the other datasets.
To summarize our results in terms of each RC dataset, we present the following observations:
- QA4MRE seemed to be the most difficult dataset among the RC datasets we analyzed, whereas MS MARCO seemed to be the easiest.
- MCTest is a good example of an RC dataset that is easy to read but difficult to answer. The corpus’ genre (i.e., narrative) seems to reflect the trend of required skills for the questions.
- SQuAD was difficult to read along with QA4MRE but relatively easy to answer compared to other datasets.
- Who-did-What performed well in terms of its query sourcing method. Although its questions are automatically created, they are sophisticated in terms of knowledge reasoning. However, an issue with the automated sourcing method is excluding nonsense questions.
- MS MARCO was a relatively easy dataset in terms of prerequisite skills. A problem is that the dataset contained nonsense questions.
- NewsQA is advantageous in that it provides meta information on the reliability of the ques-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
tions. Such information enabled us to avoid using nonsense questions, such as in the training of machine learning models.
6 Discussion
In this section, we discuss several matters regarding the construction of RC datasets and the development of RC systems using our methodology.
How to utilize the two classes of metrics for system development: One example for the development of an RC system is that it should be built to solve an easy-to-read and easy-to-answer dataset. The next step is to improve the system so that it can solve an easy-to-read and difficult-toanswer dataset. Finally, only after it can solve such dataset should the system be applied to a difficultto-read and difficult-to-answer dataset. Appropriate datasets can be prepared for every step by measuring their properties using the metrics of this study. Such datasets can be placed in a continuum based on the grades of the metrics and applied to each step of the development, like in curriculum learning (Bengio et al., 2009) and transfer learning (Pan and Yang, 2010).
Corpus genre: Attention should be paid to the genre of corpus used to construct a dataset. Expository documents like news articles tend to require factorial understanding. Most existing RC datasets use such texts because of their availability. On the other hand, narrative texts have a close correspondence to our everyday experience, such as the emotion and intention of characters (Graesser et al., 1994). If we want to build agents that work in the real world, RC datasets may have to be constructed from narratives.
Question type: In contrast to factorial understanding, comprehensive understanding of natural language texts needs a better grasp of the global coherence (e.g., the main point or moral of the text, goal of a story, and intention of characters) from the broad context (Graesser et al., 1994). Most questions that are prevalent now require only local coherence (e.g., referential relations and thematic roles) with a narrow context. Such questions based on global coherence may be generated by techniques of abstractive text summarization (Rush et al., 2015; Ganesan et al., 2010).
Annotation issues: There were questions for which there were disagreements regarding the decision of nonsense. For example, some questions can be solved by external knowledge without seeing their context. Thus, we should clarify what a
Sentence QA4MRE MCTest SQuAD WDW MARCO NewsQA
Number 1.120 1.180 1.040 1.110 1.080 1.170 Distance 1.880 0.930 0.090 0.730 0.280 0.540
Table 6: Average number and distance of sentences that need to be read to answer a question in the RC datasets.
“solvable” or “reasonable” question is in RC. In addition, annotators reported that the prerequisite skills did not easily treat questions whose answer was “none of the above” in QA4MRE. We considered those “no answer” questions difficult in another sense, so our methodology failed to specify them.
Competence of selecting necessary sentences: As mentioned in Section 1, our methodology cannot evaluate the competence of selecting sentences that need to be read to answer questions. As a brief analysis, we further investigated sentences in the context of the datasets that were highlighted in the annotation. Analyses were performed in two ways: for each question, we counted the number of required sentences and their distance (see Appendix B for the calculation method). The values of the first row in Table 6 show the average number of required sentences per question for each RC dataset. Although the scores seemed to be approximately level, MCTest required multiple sentences the most frequently. The second row presents the average distance of required sentences. QA4MRE demonstrated the longest distance because readers had to look for clues in long context texts of the dataset. In contrast, SQuAD and MS MARCO showed lower scores: most of their questions seem only to require reading a single sentence to answer. Of course, the scores of distances should depend on the length of the context texts.
7 Conclusion
In this study, we adopted evaluation metrics to analyze both the performance of a system and the quality of RC datasets. We assumed two classes— refined prerequisite skills and readability—and defined evaluation metrics for each. Next, we annotated six existing RC datasets with those defined metrics. Our annotation highlights the characteristics of the datasets and provides a valuable guide for the construction of new datasets and the development of RC systems. For future work, we plan to use the analysis in the present study to construct a system that can be applied to multiple datasets.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Sampling Methods for Questions
In this appendix, we explain the method of choosing questions for the annotation.
QA4MRE (Sutcliffe et al., 2013): The goldstandard dataset consists of four different topics and four documents for each topic. We randomly selected 100 main and auxiliary questions so that at least one question of each document was included.
MCTest (Richardson et al., 2013): This dataset consists of two sets: MC160 and MC500. Their development sets have 80 tasks in total; each includes context texts and four questions. We randomly chose 25 tasks (100 questions) from the development sets.
SQuAD (Rajpurkar et al., 2016): This dataset includes some Wikipedia articles from various topics, and those articles are divided into paragraphs. We randomly chose 100 paragraphs from
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
15 articles and used only one question of each paragraph for the annotation.
Who-did-What (WDW) (Onishi et al., 2016): This dataset is constructed from the English Gigaword newswire corpus (v5). Its questions are automatically created from an article that differs from one used for context. In addition, questions that can be solved by a simple baseline method are excluded from the dataset.
MS MARCO (MARCO) (Nguyen et al., 2016): A task of this dataset comprises several segments, one question, and its answer. We randomly chose 100 tasks (100 questions) and only used segments whose attribute was is selected = 1 as context.
NewsQA (Trischler et al., 2016): we randomly chose questions that satisfied the following conditions: is answer absent = 0, is question bad = 0, and validated answers do not include bad question or none.
B Calculation of Sentence Distance
As mentioned in Section 6, the distance of sentences was calculated as follows. If a question required only one sentence to be read, its distance was zero. If a question required two adjacent sentences to be read, its distance was one. If a question required more than two sentences to be read, its distance was the sum of distances of any two sentences.

- Strengths:  - this article puts two fields together: text readability for humans and machine comprehension of texts  - Weaknesses:  - The goal of your paper is not entirely clear. I had to read the paper 4 times and I still do not understand what you are talking about! - The article is highly ambiguous what it talks about - machine comprehension or text readability for humans - you miss important work in the readability field - Section 2.2. has completely unrelated discussion of theoretical topics. - I have the feeling that this paper is trying to answer too many questions in the same time, by this making itself quite weak. Questions such as  does text readability have impact on RC datasets d should be analyzed separately from all these prerequisite skills.  - General Discussion:  - The title is a bit ambiguous, it would be good to clarify that you are referring to machine comprehension of text, and not human reading comprehension, because  reading comprehension d and  readability d usually mean that. - You say that your  dataset analysis suggested that the readability of RC datasets does not directly affect the question difficulty d, but this depends on the method features used for answer detection, e.g. if you use POS dependency parse features. - You need to proofread the English of your paper, there are some important omissions, like  the question is easy to solve simply look.. d on page 1. - How do you annotate datasets with  metrics d?? - Here you are mixing machine reading comprehension of texts and human reading comprehension of texts, which, although somewhat similar, are also quite different, and also large areas. -  readability of text d is not  difficulty of reading contents d. Check this: DuBay, W.H. 2004. The Principles of Readability. Costa Mesa, CA: Impact information.  - it would be good if you put more pointers distinguishing your work from readability of questions for humans, because this article is highly ambiguous. E.g. on page 1  These two examples show that the readability of the text does not necessarily correlate with the difficulty of the questions d you should add  for machine comprehension d - Section 3.1. - Again: are you referring to such skills for humans or for machines? If for machines, why are you citing papers for humans, and how sure are you they are referring to machines too? - How many questions the annotators had to annotate? Were the annotators clear they annotate the questions keeping in mind machines and not people?


The paper presents a procedure of building an evaluation dataset for validation of compositional distributional semantics models estimated for languages other than English. The procedure generally builds on steps designed to assemble the SICK corpus which contains pairs of English sentences annotated for semantic relatedness and entailment, as we aim at building a comparable dataset. However, the implementation of particular building steps significantly differs from the original SICK design assumptions, which is caused by both lack of necessary extraneous resources for an investigated language and the demand for language-specific transformation rules. The designed procedure is verified on Polish, a fusional language with a relatively free word order, and contributes to building a Polish evaluation dataset. The resource consists of 10K sentence pairs which are human-annotated for semantic relatedness and entailment. The dataset may be used for evaluation of compositional distributional semantics models of Polish.
The paper presents a procedure of building an evaluation dataset for validation of compositional distributional semantics models estimated for languages other than English. The procedure generally builds on steps designed to assemble the SICK corpus which contains pairs of English sentences annotated for semantic relatedness and entailment, as we aim at building a comparable dataset. However, the implementation of particular building steps significantly differs from the original SICK design assumptions, which is caused by both lack of necessary extraneous resources for an investigated language and the demand for language-specific transformation rules. The designed procedure is verified on Polish, a fusional language with a relatively free word order, and contributes to building a Polish evaluation dataset. The resource consists of 10K sentence pairs which are human-annotated for semantic relatedness and entailment. The dataset may be used for evaluation of compositional distributional semantics models of Polish.
1 Introduction and related work
Can you imagine a straightforward answer to the question How to automatically analyse semantics of a natural language and to represent the meaning of phrases (or even sentences) in this language in an accessible way? A few years ago most of us would probably answer I don’t know! And now, in the era of high-speed and high-performance computing, everybody seems to know it – with distributional semantics models.
1.1 Distributional semantics
The basic idea of distributional semantics, i.e. determining the meaning of a word based on its co-occurrence with other words, is derived from the empiricists – Harris (1954) and Firth (1957). John R. Firth drew attention to the contextdependent nature of meaning especially with his famous maxim “You shall know a word by the company it keeps” (Firth, 1957, p. 11).
Nowadays, distributional semantics models are estimated with various methods, e.g. word embedding techniques (Bengio et al., 2003, 2006; Mikolov et al., 2013). To ascertain the purport of a word, e.g. bath, you can use the context of other words that surround it. If we assume that the meaning of this word expressed by its lexical context is associated with a distributional vector, the distance between distributional vectors of two semantically similar words, e.g bath and shower, should be smaller than between vectors representing semantically distinct words, e.g. bath and tree.
1.2 Compositional distributional semantics
Based on empirical observations that distributional vectors encode certain aspects of the word meaning, it is expected that similar aspects of the meaning of phrases and sentences can also be represented with vectors obtained via composition of distributional word vectors. The idea of semantic composition is not new. It is well known as principle of compositionality:1 “The meaning of a compound expression is a function of the meaning of its parts and of the way they are syntactically combined.” (Janssen, 2012, p. 19).
Modelling the meaning of textual units larger than words using compositional and distributional information is the main subject of compositional
1As the principle of compositionality is attributed to Gottlob Frege, it is often called Frege’s principle.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
distributional semantics (Mitchell and Lapata, 2010; Baroni and Zamparelli, 2010; Grefenstette and Sadrzadeh, 2011; Socher et al., 2012, to name a few studies). Fundamental principles of compositional distributional semantics, henceforth referred to as CDS, are mainly propagated with papers written on the topic. Apart from papers it was the SemEval-2014 Shared Task 1 (Marelli et al., 2014) which essentially contributed to the expansion of CDS and increased an interest in this domain. The goal of the task was to evaluate CDS models of English in terms of semantic relatedness and entailment on proper sentences from the SICK corpus.
1.3 The SICK corpus
The SICK corpus (Bentivogli et al., 2014) consists of 10K pairs of English sentences containing multiple lexical, syntactic, and semantic phenomena. It builds on two external data sources – the 8K ImageFlickr dataset (Rashtchian et al., 2010) and SemEval-2012 Semantic Textual Similarity dataset (Agirre et al., 2012). Each sentence pair is human-annotated for relatedness in meaning and entailment.
The relatedness score corresponds to the degree of semantic relatedness between two sentences and is calculated as the average of ten human ratings collected for this sentence pair on the 5-point Likert scale. This score indicates the extent to which the meanings of two sentences are related.
The entailment relation between two sentences, in turn, is labelled with entailment, contradiction, or neutral. According to the SICK guidelines, the label assigned by the majority of human annotators is selected as the valid entailment label.
1.4 Motivation and organisation of the paper
Studying approaches to various natural language processing (henceforth NLP) problems, we have observed that the availability of language resources (e.g. training or testing data) stimulates development of NLP tools and estimation of NLP models. Hence, we aim at building datasets for evaluation of CDS models in languages other than English, which are often under-resourced. We therefore assume that availability of test data will encourage development of CDS models in these languages.
We start with a high-quality dataset for Polish, which is a completely different language than English in at least two dimensions. First, it is
a rather under-resourced language in contrast to the resource-rich English. Second, it is a fusional language with a relatively free word order in contrast to the isolated English with a relatively fixed word order.
The procedure of building an evaluation dataset for validating compositional distributional semantics models of Polish generally builds on steps designed to assemble the SICK corpus (described in Section 1.3), as we aim at building an evaluation dataset which is comparable to the SICK corpus. However, the implementation of particular building steps significantly differs from the original SICK design assumptions, which is caused by both lack of necessary extraneous resources for Polish (see Section 2.1) and demand for Polishspecific transformation rules (see Section 2.2). Furthermore, rules of arranging sentences into pairs (see Section 2.3) are defined anew taking into account the characteristic of data and bidirectional entailment annotations, since an entailment relation between two sentences must not be symmetric. Even if our assumptions of annotating sentence pairs coincide with the SICK principles to a certain extent (see Section 3.1), the annotation process differs from the SICK procedure, in particular by introducing an element of human verification of correctness of automatically transformed sentences (see Section 3.2) and some additional post-corrections (see Section 3.3). Finally, a summary of the dataset is provided in Section 4.1 and the dataset evaluation is given in Section 4.2.
2 Procedure of collecting data
2.1 Selection and description of images
The first step of building the SICK corpus consisted in the random selection of English sentence pairs from existing datasets (Rashtchian et al., 2010; Agirre et al., 2012). Since we are not aware of accessibility of analogous resources for Polish, we have to select images first and then describe the selected images.
Images are selected from the 8K ImageFlickr dataset (Rashtchian et al., 2010). At first we wanted to take only these images the descriptions of which were selected for the SICK corpus. However, a cursory check shows that these images are quite homogeneous, with a predominant number of dogs depictions. Therefore, we independently extract 1K images and split them into 46 thematic groups (e.g. children, musical instruments, motor-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
bikes, football, dogs). Numbers of images within individual thematic groups vary from 6 images in the volleyball and telephoning groups to 94 images in the various people group. The second largest groups are children and dogs with 50 images each.
The chosen images are given to two authors who independently of each other formulate their descriptions based on a short instruction. The authors are instructed to write one single sentence (with a sentence predicate) describing the action on a displayed image. They should not describe an imaginable context or an interpretation of what may lie behind the scene on the picture. If some details on the picture are not obvious, they should not be described too. Furthermore, the authors should avoid multiword expressions, such as idioms, metaphors, and named entities, as those are not compositional linguistic phenomena. Finally, descriptions should contain Polish diacritics and proper punctuation.
2.2 Transformation of descriptions
The second step of building the SICK corpus consisted in pre-processing extracted sentences, i.e. normalisation and expansion (Bentivogli et al., 2014, p. 3–4). Since the authors of Polish descriptions are asked to follow the guidelines (presented in Section 2.1), the normalisation step is not essential for our data. The expansion step, in turn, is implemented and the sentences provided by the authors are lexically and syntactically transformed in order to obtain derivative sentences with similar, contrastive, or neutral meanings. The following transformations are implemented:
1. dropping conjunction concerns sentences with coordinated predicates sharing a subject, e.g. Rowerzysta odpoczywa i obserwuje morze. (Eng. ‘A cyclist is resting and watching the sea.’). The finite form of one of the coordinated predicates is transformed into:
• an active adjectival participle, e.g. Odpoczywający rowerzysta obserwuje morze. (Eng. ‘A resting cyclist is watching the sea.’) or Obserwujący morze rowerzysta odpoczywa. (Eng. ‘A cyclist, who is watching the sea, is resting.’), • a contemporary adverbial participle,
e.g. Rowerzysta, odpoczywając, obserwuje morze. (Eng. ‘A cyclist is watching the sea, while resting.’) or Rowerzysta odpoczywa, obserwując morze.
(Eng. ‘A cyclist is resting, while watching the sea.’).
2. removing conjunct in adjuncts, i.e. the deletion of one of coordinated elements of an adjunct, e.g. Mały, ale zwinny kot miauczy. (Eng. ‘A small but agile cat miaows.’) can be changed into either Mały kot miauczy. (Eng. ‘A small cat miaows.’) or Zwinny kot miauczy. (Eng. ‘An agile cat miaows.’).
3. passivisation, e.g. Człowiek ujeżdża byka. (Eng. ‘A man is breaking a bull in.’) can be transformed into Byk jest ujeżdżany przez człowieka. (Eng. ‘A bull is being broken in by a man.’).
4. removing adjuncts, e.g. Dwa białe króliki siedzą na trawie. (Eng. ‘Two small rabbits are sitting on the grass.’) can be changed into Króliki siedzą. (Eng. ‘The rabbits are sitting.’).
5. swapping relative clause for participles, i.e. a relative clause swaps with a participle (and vice versa), e.g. Kobieta przytula psa, którego trzyma na smyczy. (Eng. ‘A woman hugs a dog which she keeps on a leash.’). The relative clause is interchanged for a participle construction, e.g. Kobieta przytula trzymanego na smyczy psa. (Eng. ‘A woman hugs a dog kept on a leash.’).
6. negation, e.g. Mężczyźni w turbanach na głowie siedzą na słoniach. (Eng. ‘Men in turbans on their heads are sitting on elephants.’) can be transformed into Nikt nie siedzi na słoniach. (Eng. ‘Nobody is sitting on elephants.’), Żadni mężczyźni w turbanach na głowie nie siedzą na słoniach. (Eng. ‘No men in turbans on their heads are sitting on elephants.’), and Mężczyźni w turbanach na głowie nie siedzą na słoniach. (Eng. ‘Men in turbans on their heads are not sitting on elephants.’).
7. constrained mixing of dependents from various sentences, e.g. Dwoje dzieci siedzi na wielbłądach w pobliżu wysokich gór. (Eng. ‘Two children are sitting on camels near high mountains.’) can be changed into Dwoje dzieci siedzi przy zastawionym stole w pobliżu wysokich gór. (Eng. ‘Two children
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
are sitting at the table laden with food near high mountains.’).
The first five transformations are designed to produce sentences with a similar meaning, the sixth transformation outputs sentences with a contradictory meaning, and the seventh transformation should generate sentences with a neutral (or unrelated) meaning. All transformations are performed on dependency structures of input sentences (Wróblewska, 2014).
Some of the transformations are very productive (e.g. mixing dependents). Other, in turn, are sparsely represented in the output (e.g. dropping conjunction). The number of transformed sentences randomly selected to build the dataset is in the second column of Table 1.
transformation selected dropping conjunction 139 2.0% removing conjunct in adjunct 485 6.9% passivisation 893 12.8% removing adjuncts 1013 14.5% swapping rc↔ptcp 1291 18.4% negation 1304 18.6% mixing dependents 1878 26.8%
Table 1: Numbers of transformed sentences selected for annotation.
2.3 Data ensemble
The final step of building the SICK corpus consisted in arranging normalised and expanded sentences into pairs. As our data diverges from SICK data, the process of arranging Polish sentences into pairs also differs from pairing in the SICK corpus. Apart from pairs connecting two sentences originally written by humans (as described in Section 2.1), there are also pairs in which an original sentence is connected with a transformed sentence. For each of the 1K images, the following 10 pairs are constructed (for A being the set of all sentences originally written by the first author, B being the set of all sentences originally written by the second author, a ∈ A and b ∈ B being the original descriptions of the picture):
1. (a,b),
2. (a,a1), where a1 ∈ t(a), for t(a) being the set of all transformations of the sentence a,
3. (b,b1), where b1 ∈ t(b),
4. (a,b2), where b2 ∈ t(b),
5. (b,a2), where a2 ∈ t(a),
6. (a,a3), where a3 ∈ t(a′),a′ ∈ A, T (a′) = T (a),a′ 6= a, for T (a) being the thematic group of a,
7. (b,b3), where b3 ∈ t(b′),b′ ∈ B, T (b′) = T (b),b′ 6= b,
8. (a,a4), where a4 ∈ A, T (a4) 6= T (a),
9. (b,b4), where b4 ∈ B, T (b4) 6= T (b),
10. (a,a5), where a5 ∈ t(a),a5 6= a1 for 50% images, (b,b5) (analogously) for other 50%.
For each sentence pair (a,b) created according to this procedure, its reverse (b,a) is also included in our corpus. As a result, the working set consists of 20K sentence pairs.
3 Corpus annotation
3.1 Annotation assumptions
The degree of semantic relatedness between two sentences is calculated as the average of all human ratings on the Likert scale with the range from 0 to 5. Since we do not want to excessively influence the annotations, the guidelines given to annotators are mainly example-based:
• 5 (very related): Kot siedzi na płocie. (Eng. ‘A cat is sitting on the fence.’) vs. Na płocie jest duży kot. (Eng. ‘There is a large cat on the fence.’),
• 1–4 (more or less related): Kot siedzi na płocie. (Eng. ‘A cat is sitting on the fence.’) vs. Kot nie siedzi na płocie. (Eng. ‘A cat is not sitting on the fence.’); Kot siedzi na płocie. (Eng. ‘A cat is sitting on the fence.’) vs. Właściciel dał kotu chrupki. (Eng. ‘The owner gave kibble to his cat.’); Kot siedzi na płocie. (Eng. ‘A cat is sitting on the fence.’) vs. Kot miauczy pod płotem. (Eng. ‘A cat miaows under the fence.’).
• 0 (unrelated): Kot siedzi na płocie. (Eng. ‘A cat is sitting on the fence.’) vs. Zaczął padać deszcz. (Eng. ‘It starts raining.’).
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Apart from these examples, there is a note in the annotation guidelines indicating that the degree of semantic relatedness is not the same thing as the degree of semantic similarity. Since semantic similarity is only a special case of semantic relatedness, semantic relatedness is thus a more general term than the other one.
Polish entailment labels correspond directly to the SICK labels (i.e. entailment, contradiction, neutral). The entailment label assigned by the majority of human judges is selected as the gold label. The entailment labels are defined as follows:
• a wynika z b (b entails a) – if a situation or an event described by the sentence b occurs, it is recognised that a situation or an event described by a occurs as well, i.e. a and b refer to the same event or the same situation,
• a jest zaprzeczeniem b (a is the negation of b) – if a situation or an event described by b occurs, it is recognised that a situation or an event described by a may not occur at the same time,
• a jest neutralne wobec b (a is neutral to b) – the truth of a situation described by a cannot be determined on the basis of b.
3.2 Annotation procedure
Similarly as in the SICK corpus, each Polish sentence pair is human annotated for semantic relatedness and entailment by 3 human judges experienced in Polish linguistics.2 Since for each annotated pair (a,b), its reverse (b,a) is also subject to annotation, the entailment relation is in practice determined ‘in both directions’ for 10K sentence pairs. For the task of relatedness annotation, the order of sentences within pairs seems to be irrelevant, we can thus assume to obtain 6 relatedness scores for 10K unique pairs.
Since the transformation process is fully automatic and to a certain extent based on imperfect dependency parsing, we cannot ignore errors in the transformed sentences. In order to avoid annotating erroneous sentences, the annotation process is divided into two stages:
1. a sentence pair is sent to a judge with the leader role, who is expected to edit and
2Our annotators have relatively strong linguistic background. Five of them have PhD in linguistics, five are PhD students, one is a graduate, and one is an undergraduate.
to correct the transformed sentence from this pair before annotation, if necessary,
2. the verified and possibly enhanced sentence pair is sent to the other two judges, who can only annotate it.
The leader judges should correct incomprehensible and ungrammatical sentences with a minimal number of necessary changes. Unusual sentences which are generally accepted by Polish speakers should not be modified. Moreover, the modified sentence may not be identical with the other sentence in the pair.
3.3 Impromptu post-corrections
During the annotation process it came out that sentences accepted by some human annotators are unacceptable for other annotators. We thus decided to garner annotators’ comments and suggestions for improving sentences. After validation of these suggestions by an experienced linguist, it turns out that most of these proposals concern punctuation errors and typos in 228 distinct sentences. These errors are fixed directly in the corpus, as they should not impact annotations of sentence pairs. The other suggestions concern more significant changes in 141 distinct sentences. Annotations of pairs with modified sentences are resent to the annotators so that they can verify and update them.
4 Corpus summary and evaluation
4.1 Corpus statistics
Tables 2 and 3 summarise the annotations of the resulting 10K sentence pairs corpus. Table 2 aggregates the occurrences of 6 possible relatedness scores, calculated as the mean of all 6 individual annotations, rounded to an integer.
relatedness # of pairs 0 1977 1 1430 2 1081 3 2165 4 2384 5 963
Table 2: Final relatedness scores rounded to integers (total: 10K pairs).
Table 3 shows the number of the particular entailment labels in the corpus. Since each sentence
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
pair is annotated for entailment in both directions, the final entailment label is actually a pair of two labels:
• entailment+neutral points to ‘one-way’ entailment,
• contradiction+neutral points to ‘one-way’ contradiction,
• entailment+entailment, contradiction+contradiction, and neutral+neutral point to equivalence.
While the actual corpus labels are ordered in the sense that there is a difference between, e.g. entailment+neutral and neutral+entailment (the entailment occurs in different directions), we treat all labels as unordered for the purpose of this summary (e.g. entailment+neutral covers neutral+entailment as well, representing the same type of relation between two sentences). The X label corresponds to pairs for which a majority label could not be established.
entailment # of pairs neutral+neutral 6477 entailment+neutral 1741 entailment+entailment 932 contradiction+contradiction 717 contradiction+neutral 113 X 20
Table 3: Final entailment labels (total: 10K pairs).
4.2 Inter-annotator agreement
The standard measure of inter-annotator agreement in various natural language labelling tasks is Cohen’s kappa (Cohen, 1960). However, this coefficient is designed to measure agreement between two annotators only. As there are three annotators of each pair of ordered sentences, we decided to apply Fleiss’ kappa3 (Fleiss, 1971) designed for measuring agreement between multiple raters who give categorical ratings to a fixed number of items. An additional advantage of this measure is that different items can be rated by different human judges, which doesn’t impact measurement. The normalised Fleiss’ measure of inter-annotator agreement is:
3As Fleiss’ kappa is actually generalisation of Scott’s π (Scott, 1955), it is sometimes referred to as Fleiss’ multi-π, cf. Artstein and Poesio (2008).
κ = P̄ − P̄e 1− P̄e
where the quantity P̄ − P̄e measures the degree of agreement actually attained in excess of chance, while “[t]he quantity 1 − P̄e measures the degree of agreement attainable over and above what would be predicted by chance” (Fleiss, 1971, p. 379).
We recognise Fleiss’ kappa as particularly useful for measuring inter-annotator agreement with respect to entailment labelling in our evaluation dataset. First, there are more then two raters. Second, entailment labels are categorial. Measured with Fleiss’ kappa, there is inter-annotator agreement of κ = 0.732 for entailment labels in Polish evaluation dataset, which is quite satisfactory as for a semantic labelling task.
Relative to semantic relatedness, the distinction in meaning of two sentences made by human judges is often very subtle. This is also reflected in the inter-annotator agreement scores measured with Fleiss’ kappa. Inter-annotator agreement measured for six semantic relatedness groups corresponding to points on the Likert scale is quite low: κ = 0.336. If we measure interannotator agreement for three classes corresponding to the three relatedness groups from the annotation guidelines (see Section 3.1), i.e. <0>, <1, 2, 3, 4>, and <5>, the Fleiss’ score is significantly higher: κ = 0.543. Hence, we conclude that Fleiss’ kappa is not a reliable measure of inter-annotator agreement in relation to relatedness scores. Therefore, we decided to use Krippendorff’s α instead.
Krippendorff’s α (Krippendorff, 1980, 2013) is a coefficient appropriate for measuring interannotator agreement of a dataset which is annotated with multiple judges and characterised by different magnitudes of disagreement and missing values. Krippendorff (1980; 2013) offers distance metrics suitable for various scales: binary, nominal, interval, ordinal, and ratio. In ordinal measurement4 the attributes can be rank-ordered, but
4Nominal measurement is useless for measuring agreement between relatedness scores (α = 0.336 is the identical value as Fleiss’ kappa, since all disagreements are considered equal). We also test interval measurement, in which the distance between the attributes does have meaning and an average of an interval variable is computed. The interval score measured for relatedness annotations is quite high α = 0.785, but we doubt whether the distance between relatedness scores is meaningful in this case.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
distances between them do not have any meaning. Measured with Krippendorff’s α, there is interannotator agreement of α = 0.780 for relatedness scores in the Polish evaluation dataset, which is quite satisfactory as well. Hence, we conclude that our dataset is a reliable resource for purpose of evaluating compositional distributional semantics model of Polish.
5 Conclusions
The goal of this paper is to present the procedure of building the Polish evaluation dataset for validation of compositional distributional semantics models. As we aim at building an evaluation dataset which is comparable to the SICK corpus, the general assumptions of our procedure correspond to the design principles of the SICK corpus. However, the procedure of building the SICK corpus cannot be adapted without modifications. First, the Polish seed-sentences have to be written based on the images which are selected from 8K ImageFlickr dataset and split into thematic groups, since usable datasets are not publicly available. Second, as the process of transforming sentences seems to be language-specific, the linguistic transformation rules appropriate for Polish have to be defined from scratch. Third, the process of arranging Polish sentences into pairs is defined anew taking into account the data characteristic and bidirectional entailment annotations. The discrepancies relative to the SICK procedure also concern the annotation process itself. Since an entailment relation between two sentences must not be symmetric, each sentence pair is annotated for entailment in both directions. Furthermore, we introduce an element of human verification of correctness of automatically transformed sentences and some additional post-corrections.
The presented procedure results in building the Polish test corpus of relatively high quality. There are inter-annotator agreement on entailment labels of κ = 0.732 measured with Fleiss’ kappa and inter-annotator agreement on relatedness scores of α = 0.78 measured with Krippendorff’s alpha. The evaluation dataset will be made publicly available upon publication of this paper.

- Strengths:  1) This paper proposed a semi-automated framework (human generation to auto expansion to human post-editing) to construct a compositional semantic similarity evaluation data set.  2) The proposed framework is used to create a Polish compositional semantic similarity evaluation data set which is useful for future work in developing Polish compositional semantic models.  - Weaknesses:  1) The proposed framework has only been tested on one language. It is not clear whether the framework is portable to other languages. For example, the proposed framework relies on a dependency parser which may not be available in some languages or in poor performance in some other languages.  2) The number of sentence pairs edited by leader judges is not reported so the correctness and efficiency of the automatic expansion framework can not be evaluated. The fact that more than 3% (369 out of 10k) of the post-edited pairs need further post-editing is worrying.   3) There are quite a number of grammatical mistakes. Here are some examples but not the complete and exhaustive list:  line 210, 212, 213:  on a displayed image picture  to  in a displayed image picture   line 428:  Similarly as in  to  Similar to   A proofread pass on the paper is needed.  - General Discussion: