A Corpus of Annotated Revisions for Studying Argumentative Writing
This paper presents a new corpus of annotated revisions of argumentative essays. This corpus analyzes between-draft revisions in the context of the full essay. The writer’s intention for each revision is labeled with categories analogous to those used for argument mining and discourse analysis. The corpus enables more advanced research in writing comparison and revision analysis. Applications of the corpus are demonstrated via a study on student revision behaviors and a study on automatic revision intention prediction.
1 Introduction
Much of the writing-related NLP research focuses on the analysis of single drafts. Examples include document-level quality assessment (Attali and Burstein, 2006; Burstein and Chodorow, 1999), discourse-level analysis and mining (Brown and Yule, 1983; Burstein et al., 2003; Falakmasir et al., 2014; Persing and Ng, 2016), as well as fine-grained phrasal-level error detection (Leacock et al., 2010; Grammarly, 2016). Less studied is the analysis of between drafts – a comparison of revisions and the properties of the differences. Research on the topic allows a variety of applications: revision analysis (Zhang and Litman, 2015), paraphrase (Malakasiotis and Androutsopoulos, 2011) and correction detection (Swanson and Yamangil, 2012; Xue and Hwa, 2014).
Although there are some corpora resources for NLP research on writing comparisons, most tend to be between individual sentences/phrases for tasks such as paraphrase comparison (Dolan and Brockett, 2005; Tan and Lee, 2014) or grammar error correction (Dahlmeier et al., 2013; Yannakoudakis et al., 2011). In terms of revision anal-
ysis, the most relevant work are on Wikipedia revisions (Daxenberger and Gurevych, 2013; Bronner and Monz, 2012); however, the domain of Wikipedia is so specialized that the properties of their revisions do not correspond well with other kinds of texts.
This work presents an annotated corpus to facilitate revision analysis for argumentative essays. The corpus consists of a collection of three drafts of essays written by college students; the drafts are manually aligned at the sentence level, and the purpose of each revision is manually coded, using a revision schema closely related to argument mining/discourse analysis. Within the domain of argumentative essays, the corpus may be used for research and application of argument mining techniques and argumentative revision analysis. Outside of the domain, the corpus may also be of interest to research on paraphrase comparisons, grammar error correction, and computational stylistics (Popescu and Dinu, 2008; Flekova et al., 2016). We expect the corpus to be useful for advanced writing comparison study (discourse level) that connects to the mainstream research on writing analysis (e.g., argument mining). In this paper, we present two applications of our corpus: 1) Rewriting behavior data analysis 2) Automatic revision purpose classification.
2 Corpus Design Decisions
Consider this scenario: Alice begins her social science argumentative essay with the sentence: “Electronic communication allows people to make connections beyond physical limits.”
An analytical system might (rightly) identify the sentence as the thesis of her essay, and an evaluative system might give the essay a lower score due to this sentence’s vagueness and lack of evidence (though Alice may not know why she re-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
ceived that score). Now suppose in a revised draft, Alice expanded the sentence: “Electronic communication allows people to make connections beyond physical limits location and enriches connections that would have been impossible to make otherwise.”
An analytical system would still identify the sentence as the thesis, and an evaluative system might raise the overall score a little higher. Alice may become satisfied with the increase and move on. However, there is an opportunity lost – neither systems addressed the quality of her revision.
A revision analysis system might be helpful for Alice because it would link “limits” to “location and ...” and identify the reason why she made the change – perhaps adding precision. If Alice had intended her change as a way to add evidential support for her thesis, she would see that her attempt was not as successful as she hoped.
The above scenario highlights the application of a revision analysis system. This paper is about creating a corpus to enable the development of such systems. Because this is a relatively new problem, there are many possible ways for us to design the corpus. Here, we discuss some of our decisions.
First, we need to define the unit of revision. In the example above illustrates a phrase-aligned revision. While this offers a more precise definition of the scope of a revision, it may be difficult to achieve consistent annotations. For example, the changes may not adhere to any syntactic linguistic unit. For this first corpus, we define our unit of revision to be at the sentence level. In other words, even if a pair of sentences contain multiple edits, the entire sentence pair will be annotated as one sentence revision.
Second, we need to define the quality we want to observe about the revision sentence pair. For this first corpus, we focus on recognizing the purpose of the revision, as in the example above. It is a useful property, and it has previously been studied by others in the literature. People have considered both binary purpose categories such as Content vs. Surface (Faigley and Witte, 1981) or Factual vs. Fluency (Bronner and Monz, 2012); and more fine-grained categories (Pfeil et al., 2006; Jones, 2008; Liu and Ram, 2009; Daxenberger and Gurevych, 2012; Zhang and Litman, 2015). Our corpus follows the two-tiered schema used by (Zhang and Litman, 2015) (see Section 3.2).
Third, we not only have to decide on the an-
notation format, we also need to decide on how to obtain the raw text: argumentative essays with multiple drafts. We decided to sample from a population of predominantly college students, inclusive of both native and proficient non-native (aka L2) speakers. Comparing to high school students, college students are expected to have a better organization of the argument elements. Including native and L2 speakers allows for the exploration of possible rewriting differences between writers of varying backgrounds. We decided to give all subjects the same writing prompt and collect three drafts. The identical prompt minimizes the impact of topic difference for argumentation-related study. The collection of three drafts allows for a comparison of revision differences at different stages of rewriting.
Finally, we need a method of eliciting two revised drafts from each writer. Ideally, an instructor would give formative feedback after each draft for each student, but we do not have the resources to carry out such an expensive project. We decided to simulate instructor feedback by asking students to add more examples after the first draft. To elicit a second revised draft, we tried two strategies: 1) show students a character-based comparison between their first and second draft,1; and 2) show students what an idealized revision analysis system might tell them.
3 The Argumentative Revision Corpus
Based on the above design decisions, we have developed a corpus of argumentative essays with three drafts and detailed annotations for sentencealigned revisions between each consecutive pair of drafts. The main corpus has five elements; the relationships between them are shown in Figure 1;
1Code derived from https://code.google.com/ p/google-diff-match-patch/ which implements Myers’ algorithm (Myers, 1986).
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
(a) Interface A. (b) Interface B.
Figure 2: Screenshot of the interfaces. (a) Interface A with the annotated revision purposes, (b) Interface B with a streamlined character-based diff.
Section 3.1 describes the procedure of obtaining them. Section 3.2 briefly describes the revision schema we used, and reports the inter-annotator agreement. Additionally, we have collected metadata from the participants who contributed to the corpus (discussed in Section 3.3); these data may be useful for user behavior analysis.
3.1 Corpus Development Procedure
We have recruited 60 participants aged 18 years and older, among whom 40 were English native speakers and 20 were non-native speakers with sufficient English proficiency.2 The study is carried out in three 40-60 minute sessions over the duration of two weeks.
Draft1 Each participant begins by completing a pre-study questionnaire (Section 3.3) and writing a short essay online. Participants are instructed to keep the essay around 400 words, making a single main point with two supporting examples. They are given the following prompt:
“Suppose you’ve been asked to contribute a short op-ed piece for The New York Times. Argue whether the proliferation of electronic communications (e.g., email, text or other social media) enriches or hinders the development of interpersonal relationships.”
Draft2 A few days after, participants are asked to revise their first draft online based on the following feedback: Strengthen the essay by adding one more example or reasoning for the claim; then
2i.e., with a TOEFL score higher than 100.
add a rebuttal to an opposing idea; keep the essay at 400 words.
Annotated Revisions I (Rev12) The two drafts are semi-manually aligned at the sentence level.3 Then, the purpose of each pair of sentence revision is manually coded by a trained annotator, following the annotation guideline (see Section 3.2).
Draft3 Participants perform their third revision in a lab environment. This time, they are not given additional instructional feedback. Instead, participants are shown a computer interface that highlights the differences between their first and second draft. They are asked to revise the third draft to improve the general quality of their essay. We experimented with two variations of elicitation. Chosen at random, half of the participants are shown Interface A, an interface that highlights the annotated differences between the drafts (Figure 2(a)); half of the participants are shown Interface B, a streamlined character-based diff (Figure 2(b)). Both groups are asked to read a tutorial about their respective interfaces before beginning to revise. Additionally, participants in group A are also asked to verify the manually annotated revision purposes between their first and second draft. After completing the final revision, all participants are given a post-study survey about their experiences (Section 3.3). Additionally, participants in group A are asked to verify the automatically predicted revision purposes between their second and
3Sentences are first automatically aligned using the algorithm in Zhang and Litman (2014) and then manually corrected by human.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Draft1 Revision Purpose
Draft2 Revision Purpose
Draft3
This world has no restriction on who one can talk to. Conventions/ Grammar/ Spelling This world has no restrictions on whom one can talk to.
This world has no restrictions on whom one can talk to.
Rebuttal/ Reservation
Unfortunately, the younger users of digital communication cannot be entirely protected from the rhetoric of any outsider.
Warrant/ Reasoning/ Backing Modern society is now faced with the issue of cyber bullying as a result.
The only aspects of communication that this new development improves are internet navigation and faux internet relatability. WordUsage/ Clarity
The only aspects of digital communication that this new development improves are internet navigation and faux internet relatability. WordUsage/ Clarity
The only aspects of digital communication that this new development improves are internet navigation and faux internet relationships.
Claims/ Ideas
Being immersed in the sphere of new technologies can allow for complete isolation from the active, non-digital world.
Being immersed in the sphere of new technologies can allow for complete isolation from the active, non-digital world.
Table 1: An example of the annotated corpus. The sentences were aligned across the drafts and the revision purposes were labeled on the aligned sentence pairs. From Draft1 to Draft2, there are two Modify revisions (Spelling and Clarity) and one Add revision. From Draft2 to Draft3, there are two Add revisions (Rebuttal and Reasoning) and one Modify revision (Clarity)
third draft (Section 4.2).
Annotated Revisions II (Rev23) Regardless of which interface the participant used, the second and third draft are compared and annotated by the trained annotator in the same process as before.
3.2 Revision Annotation Guideline
Following prior codings, sentence revisions are first coarsely categorized as surface or contentbased changes (Faigley and Witte, 1981), depending on whether any informational content was modified; within each, we distinguish between several finer categories based on the argumentative and discourse writing literature (Kneupper, 1978; Faigley and Witte, 1981; Burstein et al., 2003). Our adapted schema has three Surface categories: Organization, Conventions/Grammar/Spelling, and Word-usage/Clarity; and five Content-based categories: Claim/Ideas, Warrant/Reasoning/Backing, Evidence, Rebuttal/Reservation, and General Content Development. Table 1 shows examples of the aligned sentences in the three collected drafts and their annotated revision categories.
Two annotators (one is experienced, and the other is newly trained) annotated the files separately and discussed on the disagreed annotations to remove possible annotation errors due to misun-
derstanding. After clearing the possible errors, the Kappa on the held-out data is 0.84 on two highlevel categories of Surface vs. Content-based; and 0.71 on their eight low-level categories. The files labeled by the trained annotator were used as the gold standard annotation after the corrections.
3.3 Meta-Data
In addition to the raw text and the annotations, this corpus release also includes meta-data collected about the participants. This includes: a pre-study survey, a post-study survey, and demographic statistics.
Pre-Study Survey The pre-study survey contains participant demographic information as well as their self-reported writing background, such as their confidence in their writing ability, the number of drafts they typically make, etc.
Post-Study Survey The post-study survey contains questions about the participants’ in-lab revision experience, such as whether they found the computer interface helpful. All questions are answered on a scale of 5, ranging from “strongly disagree” to “strongly agree”.
Demographic Statistics Among the 40 native speakers, there were 29 (72.5%) undergraduates, 6 (15%) graduate students, and 5 (12.5%) non-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
students. Among the 20 L2 speakers, there were 4 (20%) undergraduates, and 16 (80%) graduate students; there were 9 Chinese, 2 Bengali, 2 Marathi, 2 Persian, 1 Arabic, 1 Korean, 1 Portuguese, 1 Spanish, and 1 Tamil. In terms of their disciplines, 33 participants (55.9%) were from the natural sciences, 24 (40.7%) from the social sciences, and 2 (3.4%) from the humanities.
3.4 Public Release
This corpus will be freely available for research purposes, with the first release coordinated with the publication of this paper. This version release will include: the raw text plus revision annotations, and the meta-data. The revision annotations are stored as .xlsx files. There are 60 spreadsheet files for revisions from Draft1 to Draft2 and 60 more spreadsheet files for revisions from Draft2 to Draft3. Each spreadsheet file contains two sheets: Old Draft and New Draft. Each row in the sheet represents one sentence in the corresponding draft. The index of the aligned sentence row in the other draft and the type of the revision on the sentence are recorded. The meta-data are in .log text files. Information in the text files are stored as the JSON data format.
4 Applications of the Corpus
While the development of a full fledged revision analysis system is outside the scope of this work, we demonstrate the potential applications of our corpus with two examples. The first performs statistical analyses on the collected revision data and meta-data to understand aspects of participant behaviors. The second uses the corpus to train a supervised classifier to automatically predict revision purposes.
4.1 Application: Student Revision Behavior Analysis
While it is well-established that thoughtful revisions improve one’s writing, and while many college-level writing courses require students to submit multiple drafts on their writing assignments (Addison and McGee, 2010), instructors rarely monitor and provide feedback to students while they revise, partly due to time constraints, partly due to their uncertainty about how to support students revisions (Cole, 2014; Melzer, 2014). There is much we do not know about how to stimulate students to self-reflect and revise.
4.1.1 Hypotheses
Using the Argumentative Revision Corpus, we can begin to ask and address some questions about student revision habits and behaviors. Our first question is: How do different types of revision feedback impact student revision? And relatedly: Does student background (native vs. L2) make a difference?
We mine the corpus to test the following hypotheses:
H1. There is a difference in participants’ revising behaviors depending on which interface is used to elicit the third draft.
H2. For participants who used Interface A, if the recognized revision purpose differs from the participants’ revision intention, participants will further modify their revision.
H3. L2 and native speakers have different behaviors in making revisions.
H1 and H2 address the first question; H3 addresses the second.
4.1.2 Methodology
To test the hypotheses, we will use both subjective and objective measures. Subjective measures are based on participant post-study survey answers. Ideally, objective measures should be based on an assessment of improvements in the revised drafts; since we do not have evaluative data at this time, we approximate the degree of improvement with the number of revisions, since these two quantities were demonstrated to be positively correlated (Zhang and Litman, 2015). The objective measures are computed from Tables 2 and 3.
To compare differences between specific subgroups on the subjective and objective measures, we conduct ANOVA tests with two factors. For example, one factor is the native language of the participant, and another is the interface used. To
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Revision Purpose Draft1 to Draft2 Draft2 to Draft3#Add #Delete #Modify #Add #Delete #Modify Content 294 179 33 320 27 16 Claims/Ideas 25 8 4 5 0 0 Warrant/Reasoning/Backing 166 83 7 191 13 3 Rebuttal/Reservation 23 1 0 13 0 0 General Content 50 80 18 86 13 13 Evidence 30 7 4 25 1 0 Surface 0 0 466 0 0 422 Word Usage/Clarity 0 0 362 0 0 357 Conventions/Grammar/Spelling 0 0 75 0 0 52 Organization 0 0 29 0 0 13
Table 3: Number of revisions, by fine-grain revision purposes and edit types (add, delete, modify).
determine correlation between quantitative measures, we conduct Pearson and Spearman correlation tests.
4.1.3 Results and Discussions
Testing for H1 Comparing Group A and Group B participants, we observe some differences. First, we detect that Group A agrees with the statement “The system helps me to recognize the weakness of my essay” more so than Group B (Group A has a mean ratings of 3.97 (“Agree”) while Group B’s is 3.17 (“Neutral”), p < .003). Second, in Group A, there is a trending positive correlation between the number of revisions from Draft2 to Draft3 and the ratings for the statement “The system encourages me to make more revisions than I usually make” (ρ=.33 and p < .07); whereas there is no such correlation for Group B. Additional information about revision purposes may elicit a stronger self-reflection response in Group A participants. In contrast, in Group B, there is a significant negative correlation between the number of revisions from Draft1 to Draft2 and ratings for the statement “it is convenient to view my previous revisions with the system” (ρ=-.36 and p < .05). This suggests that the character-based interface is ineffective when participants have to reflect on many changes.
On the other hand, when comparing the number of revisions made by Group A and Group B on Rev23 (controlling for their Rev12 numbers), we did not find a significant difference.
With no main effect for interface group we cannot verify that H1 is true; possibly a larger pool of participants is needed; or possibly the assignment is not extensive enough (in length and in the number of drafts).
Testing for H2 Focusing on the 30 participants from Group A, we check the impact of the feed-
back Rev12 on how they subsequently revise (Rev23). We counted the number of times in which the participant disagrees with the revision purpose assigned by the annotator in Rev12. Of those, we then count the number of times the corresponding sentences were further revised (i.e., in Rev23). Of the 53 sentences where the participants disagreed with the annotator, 45 were further revised in the third draft. The ratio is 0.849, much higher than the overall ratio of general Rev12 revisions being further revised in Rev23 (161/394 = 0.408) and the ratio of the agreed Rev12 revisions being revised in Rev23 (67/341 = 0.196). In further analysis, a Pearson correlation test is conducted to check the correlation between the number of Rev23 and the number of disagreements for different types of agreement/disagreements, controlling for the number of Rev12. We find a negative correlation between Rev23 and the number of cases in which the revisions annotated as Content are verified by the participants; we also find a positive correlation between Rev23 and the number of cases in which the revisions annotated as Surface are intended to be Content revisions by the participants. Both findings are consistent with H2, which suggests that participants will revise further if they perceive that their intended revisions were not recognized.
Testing for H3 We observe that native and L2 speakers exhibit different behaviors. First, we detect a significant difference in the number of Content and Surface revisions made by L2 and native speakers (p < .02 and p < .03). More specifically, native speakers tend to make more Content changes while the L2 speakers are likely to make more Surface changes. Second, there is also a significant interaction effect among two factors of Group and users’ native language (p < .021) on their ratings for the statement “the system helps
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
me to recognize the weakness of my essay”. Third, we observe a significant positive correlation in the native group between the number of content revisions in Rev23 and the ratings of the statement “the system encourages me to make more revisions than I usually make” (ρ=.4 and p < .009). This suggests that giving feedback (from either interface) encourages native speakers to make more content revisions. Finally, in the L2 group, there is a significant negative correlation between the number of surface revisions in Rev12 and the ratings for the statement “the system helps me to recognize the weakness of my essay” (ρ=-.57 and p < .008). This shows that giving feedback to L2 speakers is less helpful when they make more surface revisions. These results are consistent with H3.
Summary Experimental findings over the three hypotheses suggest that feedback on revisions do impact how students review and rewrite their drafts. However, there are many factors at play, including the interface design and the students’ linguistic backgrounds.
4.2 Application: Automatic Revision Identification
Another application of the corpus is to serve as the gold standard for training and evaluation of an automatic revision analysis system. One subtask of such a system is to predict the intention of the revision purpose. This task was previously investigated by Zhang and Litman (2015). They developed and reported the performance of a binary classifier for each individual revision category using features from the argument mining and discourse analysis researches. The availability of our corpus makes it possible for researchers to replicate their findings and conduct further studies.
4.2.1 Hypotheses
In this paper, we repeat the experiment of Zhang and Litman (2015) under different settings to investigate three new hypotheses that can now be investigated given the features of our corpus:
H4. The method used in Zhang and Litman (2015) for high school writings is also useful for the writings of college students.
H5. The same revision classification method works differently for first revision attempts and second revision attempts.
H6. The revision classification model trained
on L2 essays has a different preference from the model trained on native essays.
4.2.2 Methodology
We followed the work in Zhang and Litman (2015), where the unigram feature was used as the baseline and the SVM classifier was used as the classifier. Besides the unigram feature, three groups of features that were used in argument mining and discourse analysis researches were extracted (Location, Textual and Language) (Burstein et al., 2001; Falakmasir et al., 2014). Unweighted average F-score for each category is reported.
For H4. 10-fold (participant) cross-validation is conducted on all the essays in the corpus. We compared the results using unigram features and the results using all the features. If H4 is true, we should expect a similar improvement over the unigram baseline using our corpus.
For H5. 10-fold cross-validation was conducted for the revisions from Draft1 to Draft2 and revisions from Draft2 to Draft3 separately. We compared the improvement ratio brought by the advanced features over the unigram baseline.
For H6. In this experiment we trained two classifiers separately with L2 and native essays with all the features. 20 native participants were first randomly selected as the test data. Afterwards classifiers were trained separately using the 20 L2 participants’ essays and the remaining 20 native participants’ essays. We would expect that the performance of the two trained classifiers is different on the same test data.
4.2.3 Results and Discussion
The first two rows of Table 4 support H4. We observe that the method used in Zhang and Litman (2015) significantly improves performance (compared to a unigram baseline) for half of the classification tasks, which is similar to Zhang and Litman’s results on high school (primarily L1) writing. In our corpus, performance on Claim, Evidence, Rebuttal and Organization was not significantly better than the baseline, possibly due to the limited number of training samples for these categories (Table 3).
For H5, the four rows in the middle of Table 4 show the difference of the cross-validation results on first attempt revisions and second attempt revisions. The earlier results using all the revisions, versus now just using only Rev12 or Rev23 re-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Experiments Text-based SurfaceClaim Warrant General Evidence Rebuttal Org. Word Conv 10fold + All Revs + Unigram 0.49 0.58 0.48 0.49 0.49 0.49 0.73 0.49 10fold + All Revs + All features 0.49 0.77∗ 0.55∗ 0.50 0.49 0.49 0.86∗ 0.62∗ 10fold + Rev12 + Unigram 0.50 0.58 0.47 0.50 0.50 0.50 0.57 0.62 10fold + Rev12 + All features 0.50 0.77∗ 0.56∗ 0.50 0.50 0.50 0.72∗ 0.72∗ 10fold + Rev23 + Unigram 0.50 0.46 0.53 0.49 0.50 0.50 0.58 0.46 10fold + Rev23 + All features 0.50 0.60∗ 0.65∗ 0.49 0.50 0.50 0.78∗ 0.50 20 L2 (train) + 20 Native (test) 0.50 0.72 0.48 0.49 0.50 0.50 0.83 0.63 20 Native (train) + 20 Native (test) 0.50 0.76 0.52 0.49 0.50 0.50 0.89 0.54
Table 4: Average unweighted F-score for each binary classification task; The first 6 rows shows the average value of 10-fold cross-validation. ∗ indicates significantly better than unigram baseline (p < .05); The last 2 rows shows the F-value for training on L2/Native data and testing on Native data. Bold indicates larger than the number in the other row.
visions are similar, which provides little support for H5. With one exception, the features proposed in (Zhang and Litman, 2015) could again significantly improve the performance over the unigram baseline, for the same set of categories as when using all the revisions. However, for the Conventions/Grammar/Spelling category, we did not observe a significant improvement for revisions from Draft2 and Draft3. A possible explanation is that there is a bigger difference in the writers’ rewriting behavior from Draft2 to Draft3, which increases the difficulty of prediction.
The last two rows of Table 4 support H6. Interestingly, we observe a better performance on Warrant, General and Word Usage/Clarity with a classifier trained and tested using native essays. Perhaps essays of native speakers are more similar to each other when revised along these dimensions. For Conventions/Grammar/Spelling, in contrast, the classifier trained on L2 data yields a better performance on the same native test set. This may be because the L2 revisions usually include more spelling/grammar corrections.
5 Conclusion and Future Works
In this paper we present a new corpus for writing comparison research. Currently the corpus focuses on revision analysis over essay revisions made by both native and L2 college students. In addition to three drafts of essays, we have compared and analyzed the drafts to align semantically identical sentences and assigned revision purposes for each aligned revision sentence pairs. We have also conducted two separate studies to demonstrate the application of the corpus for revision behavior analysis and for automatic revision purpose classification.
We plan to further augment the corpus to advance research on revision analysis in the future. Some potential augmentations include: more finegrained revision categories, other properties of the revisions, such as statement strength (Tan and Lee, 2014), sub-sentential scopes, and evaluative measures on the revisions.

This paper presents a corpus of annotated essay revisions.   It includes two examples of application for the corpus:  1) Student Revision Behavior Analysis and 2) Automatic Revision Identification  The latter is essentially a text classification task using an SVM classifier and a variety of features. The authors state that the corpus will be freely available for research purposes.  The paper is well-written and clear. A detailed annotation scheme was used by two annotators to annotate the corpus which added value to it. I believe the resource might be interesting to researcher working on writing process research and related topics. I also liked that you provided two very clear usage scenarios for the corpus.   I have two major criticisms. The first could be easily corrected in case the paper is accepted, but the second requires more work.  1) There are no statistics about the corpus in this paper. This is absolutely paramount. When you describe a corpus, there are some information that should be there.   I am talking about number of documents (I assume the corpus has 180 documents (60 essays x 3 drafts), is that correct?), number of tokens (around 400 words each essay?), number of sentences, etc.   I assume we are talking about 60 unique essays x 400 words, so about 24,000 words in total. Is that correct? If we take the 3 drafts we end up with about 72,000 words but probably with substantial overlap between drafts.  A table with this information should be included in the paper.  2) If the aforementioned figures are correct, we are talking about a very small corpus. I understand the difficulty of producing hand-annotated data, and I think this is one of the strengths of your work, but I am not sure about how helpful this resource is for the NLP community as a whole. Perhaps such a resource would be better presented in a specialised workshop such as BEA or a specialised conference on language resources like LREC instead of a general NLP conference like ACL.  You mentioned in the last paragraph that you would like to augment the corpus with more annotation. Are you also willing to include more essays?  Comments Minor:  - As you have essays by native and non-native speakers, one further potential application of this corpus is native language identification (NLI).  - p. 7:  where the unigram feature was used as the baseline  -  word unigram . Be more specific.  - p. 7:  and the SVM classifier was used as the classifier.  - redundant.


Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
We introduce a new multi-modal task for computer systems, posed as a combined vision-language comprehension challenge: identifying the most suitable text describing a scene, given several similar options. Accomplishing the task entails demonstrating comprehension beyond just recognizing “keywords” (or key-phrases) and their corresponding visual concepts. Instead, it requires an alignment between the representations of the two modalities that achieves a visually-grounded “understanding” of various linguistic elements and their dependencies. This new task also admits an easy-to-compute and wellstudied metric: the accuracy in detecting the true target among the decoys. The paper makes several contributions: an effective and extensible mechanism for generating decoys from (human-created) image captions; an instance of applying this mechanism, yielding a large-scale machine comprehension dataset (based on the COCO images and captions) that we make publicly available; human evaluation results on this dataset, informing a performance upper-bound; and several baseline and competitive learning approaches that illustrate the utility of the proposed task and dataset in advancing both image and language comprehension. We also show that, in a multi-task learning setting, the performance on the proposed task is positively correlated with the end-to-end task of image captioning.
We introduce a new multi-modal task for computer systems, posed as a combined vision-language comprehension challenge: identifying the most suitable text describing a scene, given several similar options. Accomplishing the task entails demonstrating comprehension beyond just recognizing “keywords” (or key-phrases) and their corresponding visual concepts. Instead, it requires an alignment between the representations of the two modalities that achieves a visually-grounded “understanding” of various linguistic elements and their dependencies. This new task also admits an easy-to-compute and wellstudied metric: the accuracy in detecting the true target among the decoys.
The paper makes several contributions: an effective and extensible mechanism for generating decoys from (human-created) image captions; an instance of applying this mechanism, yielding a large-scale machine comprehension dataset (based on the COCO images and captions) that we make publicly available; human evaluation results on this dataset, informing a performance upper-bound; and several baseline and competitive learning approaches that illustrate the utility of the proposed task and dataset in advancing both image and language comprehension. We also show that, in a multi-task learning setting, the performance on the proposed task is positively correlated with the end-to-end task of image captioning.
1 Introduction
There has been a great deal of interest in multimodal artificial intelligence research recently, bringing together the fields of Computer Vision and Natural Language Processing. This interest has been fueled in part by the availability of many large-scale image datasets with textual annotations. Several vision+language tasks have been proposed around these datasets (Hodosh et al., 2013; Karpathy and Fei-Fei, 2015; Lin et al., 2014; Antol et al., 2015). Image Captioning (Hodosh et al., 2013; Donahue et al., 2014; Karpathy and Fei-Fei, 2015; Fang et al., 2015; Kiros et al., 2015; Vinyals et al., 2015; Mao et al., 2015; Xu et al., 2015b) and Visual Question Answering (Malinowski and Fritz, 2014; Malinowski et al., 2015; Tu et al., 2014; Antol et al., 2015; Yu et al., 2015; Wu et al., 2016b; Ren et al., 2015; Gao et al., 2015; Yang et al., 2016; Zhu et al., 2016; Lin and Parikh, 2016) have in particular attracted a lot of attention. The performances on these tasks have been steadily improving, owing much to the wide use of deep learning architectures (Bengio, 2009).
A central theme underlying these efforts is the use of natural language to identify how much visual information is perceived and understood by a computer system. Presumably, a system that understands a visual scene well enough ought to be able to describe what the scene is about (thus “captioning”) or provide correct and visuallygrounded answers when queried (thus “questionanswering”).
In this paper, we argue for directly measuring how well the semantic representations of the visual and linguistic modalities align (in some abstract semantic space). For instance, given an image and two captions – a correct one and an incorrect yet-cunningly-similar one – can we both qualitatively and quantitatively measure the ex-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
tent to which humans can dismiss the incorrect one but computer systems blunder? Arguably, the degree of the modal alignment is a strong indicator of task-specific performance on any vision+language task. Consequentially, computer systems that can learn to maximize and exploit such alignment should outperform those that do not.
We take a two-pronged approach for addressing this issue. First, we introduce a new and challenging Dual Machine Comprehension (DMC) task, in which a computer system must identify the most suitable textual description from several options: one being the target and the others being “adversarialy”-chosen decoys. All options are free-form, coherent, and fluent sentences with high degrees of semantic similarity (hence, they are “cunningly similar”). A successful computer system has to demonstrate comprehension beyond just recognizing “keywords” (or key phrases) and their corresponding visual concepts; they must arrive at a coinciding and visually-grounded understanding of various linguistic elements and their dependencies. What makes the DMC task even more appealing is that it admits an easy-tocompute and well-studied performance metric: the accuracy in detecting the true target among the decoys. Second, we illustrate how solving the DMC task benefits related vision+language tasks. To this end, we render the DMC task as a classification problem, and incorporate it in a multitask learning framework for end-to-end training of joint objectives.
Our work makes the following contributions: (1) an effective and extensible algorithm for generating decoys from human-created image captions (Section 3.2); (2) an instantiation of applying this algorithm to the COCO dataset (Lin et al., 2014), resulting in a large-scale dual machinecomprehension dataset that we make publicly available (Section 3.3); (3) a human evaluation on this dataset, which provides an upper-bound on performance (Section 3.4); (4) a benchmark study of baseline and competitive learning approaches (Section 5), which underperform humans by a substantial gap (about 20%); and (5) a multi-task learning model that simultaneously learns to solve the DMC task and the Image Captioning task (Section 4).
Our empirical study shows that performance on the DMC task positively correlates with per-
formance on the Image Captioning task. Therefore, besides acting as a standalone benchmark, the new DMC task can be useful in improving other complex vision+language tasks. Both suggest the DMC task as a fruitful direction for future research.
2 Related work
Image understanding is a long-standing challenge in computer vision. There has recently been a great deal of interest in bringing together vision and language understanding. Particularly relevant to our work are image captioning (IC) and visual question-answering (VQA). Both have instigated a large body of publications, a detailed exposition of which is beyond the scope of this paper. Interested readers should refer to two recent surveys (Bernardi et al., 2016; Wu et al., 2016a).
In IC tasks, systems attempt to generate a fluent and correct sentence describing an input image. IC systems are usually evaluated on how well the generated descriptions align with human-created captions (ground-truth). The language generation model of an IC system plays a crucial role; it is often trained such that the probabilities of the ground-truth captions are maximized (MLE training), though more advanced methods based on techniques borrowed from Reinforcement Learning have been proposed (Ranzato et al., 2015). To provide visual grounding, image features are extracted and injected into the language model. Note that language generation models need to both decipher the information encoded in the visual features, and model natural language generation.
In VQA tasks, the aim is to answer an input question correctly with respect to a given input image. In many variations of this task, answers are limited to single words or a binary response (“yes” or “no”) (Antol et al., 2015). The Visual7W dataset (Zhu et al., 2016) contains anaswers in a richer format such as phrases, but limits questions to “wh-”style (what, where, who, etc). The Visual Genome dataset (Krishna et al., 2016), on the other hand, can potentially define more complex questions and answers due to its extensive textual annotations.
Our DMC task is related but significantly different. In our task, systems attempt to discriminate the best caption for an input image from a set of captions — all but one are decoys. Arguably, it is a form of VQA task, where the same default
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
(thus uninformative) question is asked: Which of the following sentences best describes this image? However, unlike current VQA tasks, choosing the correct answer in our task entails a deeper “understanding” of the available answers. Thus, to perform well, a computer system needs to understand both complex scenes (visual understanding) and complex sentences (language understanding), and be able to reconcile them.
The DMC task admits a simple classificationbased evaluation metric: the accuracy of selecting the true target. This is a clear advantage over the IC tasks, which often rely on imperfect metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin and Och, 2004), METEOR (Banerjee and Lavie, 2005), CIDEr (Vedantam et al., 2015), or SPICE (Anderson et al., 2016).
Related to our proposal is the work in (Hodosh et al., 2013), which frames image captioning as a ranking problem. While both share the idea of selecting captions from a large set, our framework has some important and distinctive components. First, we devise an algorithm for smart selection of candidate decoys, with the goal of selecting those that are sufficiently similar to the true targets to be challenging, and yet still be reliably identifiable by human raters. Second, we have conducted a thorough human evaluation in order to establish a performance ceiling, while also quantifying the level to which current learning systems underperform. Lastly, we show that there exists a positive correlation between the performance on the DMC task and the performance on related vision+language tasks by proposing and experimenting with a multi-task learning model. Our work is also substantially different from their more recent work (Hodosh and Hockenmaier, 2016), where only one decoy is considered and its generation is either random, or focusing on visual concept similarity (“switching people or scenes”) instead of our focus on both linguistic surface and paragraph vector embedding similarity.
3 The Dual Machine Comprehension Task
3.1 Design overview
We propose a new multi-modal machine comprehension task to examine how well visual and textual semantic understanding are aligned. Given an image, human evaluators or machines must accurately identify the best sentence describing the
scene from several decoy sentences. Accuracy on this task is defined as the percentage that the true targets are identified.
It seems straightforward to construct a dataset for this task, as there are several existing datasets which are composed of images and their (multiple) ground-truth captions, including the popular COCO dataset (Lin et al., 2014). Thus, for any given image, it appears that one just needs to use the captions corresponding to other images as decoys. However, this naı̈ve approach could be overly simplistic as it is provides no control over the properties of the decoys.
Specifically, our desideratum is to recruit challenging decoys that are sufficiently similar to the targets. However, for a small number of decoys, e.g. 4-5, randomly selected captions could be significantly different from the target. The resulting dataset would be too “easy” to shed any insight on the task. Since we are also interested in human performance on this task, it is thus impractical to increase the number of decoys to raise the difficulty level of the task at the expense of demanding humans to examine tediously and unreliably a large number of decoys. In short, we need an automatic procedure to reliably create difficult sets of decoy captions that are sufficiently similar to the targets.
We describe such a procedure in the following. While it focuses on identifying decoy captions, the main idea is potentially adaptable to other settings. The algorithm is flexible in that the “difficulty” of the dataset can be controlled to some extent through the algorithm’s parameters.
3.2 Algorithm to create an MC-IC dataset
The main idea behind our algorithm is to carefully define a “good decoy”. The algorithm exploits recent advances in paragraph vector (PV) models (Le and Mikolov, 2014), while also using linguistic surface analysis to define similarity between two sentences. Due to space limits, we omit a detailed introduction of the PV model. It suffices to note that the model outputs a continuouslyvalued embedding for a sentence, a paragraph, or even a document.
The pseudo-code for the algorithm is in the Listing 1 (the name MC-IC stands for “MachineComprehension for Image-Captions”). As input, the algorithm takes a large set C of
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Algorithm 1: MC-IC(C, N , Score) Result: Dataset MCIC PV← OPTIMIZE-PV(C) λ← OPTIMIZE-SCORE(PV, C, Score) MCIC ← ∅ nr decoys = 4 for 〈ii, ci〉 ∈ C do
A← [] Tci ← PV(ci)[1..N ] for cd ∈ Tci do
score← Score(PV, λ, cd, ci) if score > 0 then
A.append(〈score, cd〉) end
end if |A|≥ nr decoys then
R← descending-sort(A) for l ∈ [1..nr decoys] do 〈score, cd〉 ← R[l] MCIC ← MCIC ∪{(〈ii, cd〉, false)} end MCIC ← MCIC ∪{(〈ii, ci〉, true)}
end end
〈image, {caption(s)} 〉 pairs∗, as those extracted from a variety of publicly-available corpora, including the COCO dataset (Lin et al., 2014). The output of the algorithm is the dataset MCIC which is used for the DMC task.
Concretely, the MC-IC Algorithm has three main arguments: a dataset C = {〈ii, ci〉|1 ≤ i ≤ m} where ii is an image and ci is its ground-truth caption. For an image with multiple groundtruth captions, we split it to multiple instances with the same image and one unique groundtruth caption per instance. An integerN which controls the size of ci’s neighborhood in the embedding space defined by the paragraph vector model PV; and a function Score which is used to score theN items in each such neighborhood.
The first two steps of the algorithm tune several hyperparameters. The first step finds optimal settings for the PV model given the dataset C. The second finds a weight parameter λ given PV, dataset C, and the Score function. These hyperparameters are dataset-specific. Details are discussed in the next section.
The main body of the algorithm, the outer for loop, generates a set of nr decoys (4 here) decoys for each ground-truth caption. It accomplishes this by first extracting N candidates from the PV neighborhood of the ground-truth caption, excluding those that belong to the same image.
∗On the order of at least hundreds of thousands of examples; smaller sets result in less challenging datasets.
In the inner for loop, it computes the similarity of each candidate to the ground-truth and stores them in a list A. If enough candidates are generated, the list is sorted in descending order of score. The top nr decoys captions are marked as “decoys” (i.e.false), while the ground-truth caption is marked as “target” (i.e.true).
The score function Score(PV, λ, c′, c) is a crucial component of the decoy selection mechanism. Its definition leverages our linguistic intuition by combining linguistic surface similarity, simSURF(c′, c), with the similarity suggested by the embedding model, simPV(c′, c):
Score= {
0 if simSURF≥L λ simPV+(1−λ) simSURF otherwise (1)
where the common argument (c′, c) is omitted. The higher the similarity score, the more likely that c′ is a good decoy for c. Note that if the surface similarity is above the threshold L, the function returns 0, flagging that the two captions are too similar to be used as a pair of target and decoy.
In this work, simSURF is computed as the BLEU score between the inputs (Papineni et al., 2002) (with the brevity penalty set to 1). The embedding similarity, simPV, is computed as the cosine similarity between the two in the PV embedding space.
3.3 The MCIC dataset
We applied the MC-IC Algorithm to the COCO dataset (Lin et al., 2014) to generate a dataset for the visual-language dual machine comprehension task. The dataset, called MCIC , is made publicly available (address anonymized). We describe the details of this dataset below.
We set the neighborhood size at N = 500, and the threshold atL = 0.5 (see Eq. 1). As the COCO dataset has a large body of images (thus captions) focusing on a few categories (such as sports activities), this threshold is important in discarding significantly similar captions to be decoys – otherwise, even human annotators will experience difficulty in selecting the ground-truth captions.
The hyperparameters of the PV model, dim (embedding dimension) and epochs (number of training epochs), are optimized in the OPTIMIZE-PV step of the MC-IC Algorithm. The main idea is to learn embeddings such that ground-truth captions from the same image have similar embeddings. Details are in the Suppl. Ma-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Split dev test train total #unique˙images 2,000 2,000 110,800 114,800
# instances 9,999 10,253 554,063 574,315
Table 1: MCIC dataset descriptive statistics
terial. The optimal settings are dim=1024 and epochs=5.
Likewise, the λ in the Score function is optimized by the OPTIMIZE-SCORE step, such that ground-truth captions associated with the same image have similar scores. Details are in the Suppl. Material. The optimal λ is 0.3.
The resulting MCIC dataset has 574,315 instances that are in the format of {i : (〈ii, cji 〉, label j i ), j = 1 . . . 5} where label j i ∈ {true, false}. For each such instance, there is one and only one j such that the label is true. We have created a train/dev/test split such that all of the instances for the same image occur in the same split. Table 1 reports the basic statistics for the dataset. Figure 1 shows several instances from the MCIC dataset.
3.4 Human performance on MCIC
To measure how well humans can perform on the DMC task, we randomly drew 1,000 instances from the MCIC dev set and submitted those instances to human “raters”† via a crowdsourcing platform.
Three independent responses from 3 different raters were gathered for each instance, for a total of 3,000 responses. To ensure diversity, raters were prohibited from evaluating more than six instances or from responding to the same task instance twice. In total, 807 distinct raters were employed.
Raters were shown one instance at a time. They were shown the image and the five caption choices (ground-truth and four decoys) and were instructed to choose the best caption for the image. To supplement the instructions, raters were initially shown a few examples from the training set with the ground-truth caption highlighted, to illustrate how to discern the most appropriate caption for the image (see the Suppl. Material for details).
We assessed human performance using two metrics: (1) Percentage of correct rater responses (1-human system): 81.1% (2432 out of 3000); (2) †Raters are vetted, screened and tested before working on any tasks; requirements include native-language proficiency level.
Percentage of instances with at least 50% (i.e.2) correct responses (3-human system): 82.8% (828 out of 1000). Due to space limitation, more discussions about the human raters disagreement is available in the Suppl. Material.
4 Learning Methods
We describe several learning methods for the dual machine comprehension (DMC) task with the MCIC dataset.
Regression. To examine how well the two embeddings are aligned in “semantic understanding space”, a simple approach is to assume that the learners do not have access to the decoys. Instead, by accessing the ground-truth captions only,
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
the models learn a linear regressor from the image embeddings to the target captions’ embeddings (“forward regression”), or from the captions to the images (“backward regression”). With the former approach, referred as Baseline-I2C, we check whether the predicted caption for any given image is closest to its true caption. With the latter, referred as Baseline-C2I, we check whether the predicted image embedding by the ground-truth caption is the closest among predicted ones by decoy captions to the real image embeddings.
Linear classifier. Our next approach BaselineLinM is a linear classifier learned to discriminate true targets from the decoys. Specifically, we learn a linear discriminant function f(i, c;θ) = i> θ c where θ is a matrix measuring the compatibility between two types of embeddings, cf. (Frome et al., 2013). The loss function is then given by
L(θ) = ∑ i [max j 6=j∗ f(ii, c j i ;θ)− f(ii, c j∗ i ;θ)]+ (2)
where [ ]+ is the hinge function and j indexes over all the available decoys and i indexes over all training instances. The optimization tries to increase the gap between the target cj ∗
i and the worst “offending” decoy. We use stochastic (sub)gradient methods to optimize θ, and select the best model in terms of accuracy on the MCIC dev set.
FFNN Model. To present our neural-network– based models, we use the following notations. Each training instance pair is a tuple 〈ii, cji 〉, where i denotes the image, and cji denotes the caption options, which can either be the target or the decoys. We use a binary variable yijk ∈ {0, 1} to denote whether j-th caption of the instance i is labeled as k, and ∑ k yijk = 1.
We first employ the standard feedforward neural-network models to solve the MCIC task. For each instance pair 〈ii, cji 〉, the input to the neural network is an embedding tuple 〈DNN(ii; Γ),Emb(cji ; Ω)〉, where Γ denotes the parameters of a deep convolutional neural network DNN. DNN takes an image and outputs an image embedding vector. Ω is the embedding matrix, and Emb(.) denotes the mapping from a list of word IDs to a list of embedding vectors using Ω. The loss function for our FFNN is given by: L(Γ,Ω,u) = ∑ i,j,k yijk log FNk(DNN(ii; Γ),Emb(cji ; Ω);u)
(3)
where FNk denotes the k-th output of a feedforward neural network, and ∑ k FNk(.) = 1. Our architecture uses a two hidden-layer fully connected network with Rectified Linear hidden units, and a softmax layer on top.
The formula in Eq. 3 is generic with respect to the number of classes. In particular, we consider a 2-class–classifier (k ∈ {0, 1}, 1 for ’yes’, this is a correct answer; 0 for ’no’, this is an incorrect answer), applied independently on all the 〈ii, cji 〉 pairs and apply one FFNN-based binary classifier for each; the final prediction is the caption with the highest ’yes’ probability among all instance pairs belonging to instance i.
Vec2seq + FFNN Model. We describe here a hybrid neural-network model that combines a recurrent neural-network with a feedforward one. We encode the image into a single-cell RNN encoder, and the caption into an RNN decoder. Because the first sequence only contains one cell, we call this model a vector-to-sequence (Vec2seq) model as a special case of Seq2seq model as in (Sutskever et al., 2014; Bahdanau et al., 2015). The output of each unit cell of a Vec2seq model (both on the encoding side and the decoding side) can be fed into an FFNN architecture for binary classification (see the Suppl. Material for an architecture illustration).
In addition to the classification loss (Eq. 3), we also include a loss for generating an output sequence cji based on an input ii image. We define a binary variable zijlv ∈ {0, 1} to indicate whether the lth word of cji is equal to word v. O d ijl denotes the l-th output of the decoder of instance pair 〈ii, cji 〉, O e ij denotes the output of the encoder, and Odij: denotes the concatenation of decoder outputs. With these definitions, the loss function for the Vec2seq + FFNN model is:
L(θ,w,u) = ∑ i,j,k yijk log FNk(Oeij(ii, c j i ;θ),O d ij:(ii, c j i ;θ);u)
+λgen ∑ i,j,l,v yij1zijlv log softmaxv(Odijl(ii, c j i ;θ);w)
(4) where ∑
v softmaxv(.) = 1; θ are the parameters of the Vec2seq model, which include the parameters within each unit cell, as well as the elements in the embedding matrices for images and target sequences; w are the output projection parameters that transform the output space of the decoder to the vocabulary space. u are the parameters of the
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
FFNN model (Eq. 3); λgen is the weight assigned to the sequence-to-sequence generation loss. Only the true target candidates (the ones with yij1 = 1) are included in this loss, as we do not want the decoy target options to affect this computation.
The Vec2seq model we use here is an instantiation of the attention-enhanced models proposed in (Bahdanau et al., 2015; Chen et al., 2016). However, our current model does not support location-wise attention, as in the Show-Attendand-Tell (Xu et al., 2015a) model. In this sense, our model is an extension of the Show-and-Tell model with a single attention state representing the entire image, used as image memory representation for all decoder decisions. We apply Gated Recurrent Unit (GRU) as the unit cell (Cho et al., 2014). We also compare the influence on performance of the λgen parameter.
5 Experiments
5.1 Experimental Setup
Baseline models For the baseline models, we use the 2048-dimensional outputs of GoogleInception-v3 (Szegedy et al., 2015) (pre-trained on ImageNet ILSSVR 2012) to represent the images, and 1024-dimensional paragraph-vector embeddings (section 3.2) to represent captions. To reduce computation time, both are reduced to 256- dimensional vectors using random projections. Neural-nets based models The experiments with these models are done using the Tensorflow package (Abadi et al., 2015). The hyper-parameter choices are decided using the hold-out development portion of the MCIC set. For modeling the input tokens, we use a vocabulary size of 8,855 types, selected as the most frequent tokens over the captions from the COCO training set (words occurring at least 5 times). The models are optimized using ADAGRAD with an initial learning rate of 0.01, and clipped gradients (maximum norm 4). We run the training procedures for 3, 000, 000 steps, with a mini-batch size of 20. We use 40 workers for computing the updates, and 10 parameter servers for model storing and (asynchronous and distributed) updating.
We use the following notations to refer to the neural network models: FFNNargmax 1..52-class refers to the version of feedforward neural network architecture with a 2-class–classifier (’yes’ or ’no’ for answer correctness), over which an argmax function computes a 5-way decision (i.e., the choice
with the highest ’yes’ probability); we henceforth refer to this model simply as FFNN.
The Vec2seq+FFNN refers to the hybrid model combining Vec2seq and FFNNargmax 1..52-class . The RNN part of the model uses a two-hidden–layer GRU unit-cell (Cho et al., 2014) configuration, while the FFNN part uses a two-hidden–layer architecture. The λgen hyper-parameter from the loss-function L(θ,w,u) (Eq. 4) is by default set to 1.0 (except for Section 5.3 where we directly measure its effect on performance).
We also include the result of a Vec2Seq model that is trained for caption generation only. To use the model for classification, we feed both image and each of its caption candidates to the Vec2Seq model, and then pick the caption candidate which has the lowest perplexity. Evaluation metrics The metrics we use to measure performance come in two flavors. First, the accuracy in detecting (the index of) the true target among the decoys provides a direct way of measuring the performance level on the comprehension task. We use this metric as the main indicator of comprehension performance. Second, because our Vec2seq+FFNN models are multitask models, they can also generate new captions given the input image. The performance level for the generation task is measured using the standard scripts measuring ROUGE-L (Lin and Och, 2004) and CIDEr (Vedantam et al., 2015), using as reference the available captions from the COCO data (around 5 for most of the images). Code for these metrics is available as part of the COCO evaluation toolkit ‡. As usual, both the hypothesis strings and the reference strings are preprocessed: remove all the non-alphabetic characters; transform all letters to lowercase, and tokenize using white space; replace all words occurring less than 5 times with an unknown token 〈UNK〉 (total vocabulary of 8,855 types); truncate to the first 30 tokens.
5.2 Results
Table 2 summarizes our main results on the comprehension task. We report the accuracies (and their standard deviations) for random choice, baselines, and neural network-based models.
Interestingly, the Baseline-I2C model performs at the level of random choice, and much worse than the Baseline-C2I model. This discrepancy reflects the inherent difficulty in vision-
‡https://github.com/tylin/coco-caption
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Model dim Dev Test Baseline-I2C 256 19.6 ±0.4 19.3±0.4 Baseline-C2I 256 32.8 ±0.5 32.0±0.5 Baseline-LinM 256 44.6 ±0.5 44.5±0.5 FFNN 256 56.3 ±0.5 55.1±0.5
Vec2seq+FFNN 256 60.5 ±0.5 59.0±0.5 Vec2Seq 256 15.6 ± 0.4 16.0 ± 0.4
Table 2: Performance on the DMC Task, in accuracies (and standard deviations) on MCIC for baselines and NN models.
Language tasks: for each image, there are several possible equally good descriptions, thus a linear mapping from the image embeddings to the captions might not be enough – statistically, the linear model will just predict the mean of those captions. However, for the reverse direction where the captions are the independent variables, the learned model does not have to capture the variability in image embeddings corresponding to the different but equally good captions – there is only one such image embedding.
Nonlinear neural networks overcome these modeling limitations. The results clearly indicate their superiority over the baselines. The Vec2seq+FFNN model obtains the best results, with accuracies of 60.5% (dev) and 59.0% (test); the accuracy numbers indicate that the Vec2seq+FFNN architecture is superior to the non-recursive fully-connected FFNN architecture (at 55.1% accuracy on test). In the Suppl. Material, we show the impact on performance of the mebedding dimension and neural-network sizes.
Last but not least, the Vec2Seq model performs the worst in the classification task. This result indicates that a caption generation model alone is unable to discriminate among captions that are close in a pretrained embedding space, even when it has a good caption generation performance (CIDEr 0.983 on dev and 0.927 on test).
5.3 DMC and Image Captioning
In this section, we compare models with different values of λgen in Eq. 4. This parameter allows for a natural progression from learning for the DMC task only (λgen = 0) to focusing more on the image captioning loss (λgen = 16).
The results in Table 3 illustrate one of the main points of this paper. That is, the ability to perform the comprehension task (as measured by the accuracy metric) positively correlates with the ability to perform other tasks that require machine comprehension, such as caption generation. At
λgen = 4, the Vec2seq+FFNN model not only has a high accuracy of detecting the ground-truth option, but it also generates its own captions given the input image, with an accuracy measured on MCIC at 0.9890 (dev) and 0.9380 (test) CIDEr scores. On the other hand, at an accuracy level of about 59% (on test, at λgen = 0.1), the generation performance is at only 0.9010 (dev) and 0.8650 (test) CIDEr scores.
We note that there is an inherent trade-off between prediction accuracy and generation performance, as seen for λgen values above 4.0. This agrees with the intuition that training a Vec2seq+FFNN model using a loss L(θ,w,u) with a larger λgen means that the ground-truth detection loss (the first term of the loss in Eq.4) may get overwhelmed by the word-generation loss (the second term). However, our empirical results suggest that there is value in training models with a multi-task setup, in which both the comprehension side as well as the generation side are carefully tuned to maximize performance.
6 Discussion
We have proposed and described in detail a new multi-modal machine comprehension task (DMC). The underlying hypothesis is that computer systems that can be shown to perform increasingly well on this task will do so by constructing a visually-grounded understanding of various linguistic elements and their dependencies.
The Vec2seq+FFNN architecture can be trained end-to-end to display both the ability to choose the most likely text associated with an image, as well as the ability to generate a complex description of that image. The empirical results validate that improvements in comprehension and generation happen in tandem.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  Authors generate a dataset of  rephrased d captions and are planning to make this dataset publicly available.  The way authors approached DMC task has an advantage over VQA or caption generation in terms of metrics. It is easier and more straightforward to evaluate problem of choosing the best caption. Authors use accuracy metric. While for instance caption generation requires metrics like BLUE or Meteor which are limited in handling semantic similarity.  Authors propose an interesting approach to  rephrasing d, e.g. selecting decoys. They draw decoys form image-caption dataset. E.g. decoys for a single image come from captions for other images. These decoys however are similar to each other both in terms of surface (bleu score) and semantics (PV similarity). Authors use lambda factor to decide on the balance between these two components of the similarity score. I think it would be interesting to employ these for paraphrasing.  Authors support their motivation for the task with evaluation results. They show that a system trained with the focus on differentiating between similar captions performs better than a system that is trained to generate captions only. These are, however, showing that system that is tuned for a particular task performs better on this task.  - Weaknesses:   It is not clear why image caption task is not suitable for comprehension task and why author  system is better for this. In order to argue that system can comprehend image and sentence semantics better one should apply learned representation, e.g. embeddings. E.g. apply representations learned by different systems on the same task for comparison.  My main worry about the paper is that essentially authors converge to using existing caption generation techniques, e.g. Bahdanau et al., Chen et al.  They way formula (4) is presented is a bit confusing. From formula it seems that both decoy and true captions are employed for both loss terms. However, as it makes sense, authors mention that they do not use decoy for the second term. That would hurt mode performance as model would learn to generate decoys as well. The way it is written in the text is ambiguous, so I would make it more clear either in the formula itself or in the text. Otherwise it makes sense for the model to learn to generate only true captions while learning to distinguish between true caption and a decoy.  - General Discussion:  Authors formulate a task of Dual Machine Comprehension. They aim to accomplish the task by challenging computer system to solve a problem of choosing between two very similar captions for a given image. Authors argue that a system that is able to solve this problem has to  understand d the image and captions beyond just keywords but also capture semantics of captions and their alignment with image semantics.  I think paper need to make more focus on why chosen approach is better than just caption generation and why in their opinion caption generation is less challenging for learning image and text representation and their alignment.  For formula (4). I wonder if in the future it is possible to make model to learn  not to generate d decoys by adjusting second loss term to include decoys but with a negative sign. Did authors try something similar?