In this paper, we aim to understand whether current language and vision (LaVi) models truly grasp the interaction between the two modalities. To this end, we propose an extension of the MS-COCO dataset, FOIL-COCO, which associates images with both correct and ‘foil’ captions, that is, descriptions of the image that are highly similar to the original ones, but contain one single mistake (“foil word”). We show that current LaVi models fall into the traps of this data and perform badly on three tasks: a) caption classification (correct vs. foil); b) foil word detection; c) foil word correction. Humans, in contrast, have near-perfect performance on those tasks. We demonstrate that using language cues only is not enough to deal with FOILCOCO and that it challenges the state-ofthe-art by requiring a fine-grained understanding of the relation between text and image.
1 Introduction
Most human language understanding is grounded in perception. There is thus growing interest in combining information from language and vision in the NLP and AI communities.
So far, the primary testbeds of Language and Vision (LaVi) models have been ‘Visual Question Answering’ (VQA) (e.g. Antol et al. (2015); Malinowski and Fritz (2014); Malinowski et al. (2015); Gao et al. (2015); Ren et al. (2015)) and ‘Image Captioning’ (IC) (e.g. Hodosh et al. (2013); Fang et al. (2015); Chen and Lawrence Zitnick (2015); Donahue et al. (2015); Karpathy and Fei-Fei (2015); Vinyals et al. (2015)). Whilst some models have seemed extremely successful
on those tasks, it remains unclear how the reported results should be interpreted and what those models are actually learning. There is an emerging feeling in the LaVi community that the current VQA task should be revised, especially as it has been shown that it can be handled well by ‘blind’ models which use only language input or by simple concatenation of language and vision features (Agarwal et al., 2016; Jabri et al., 2016; Zhang et al., 2016; Goyal et al., 2016). In IC too, Hodosh and Hockenmaier (2016) showed that, contrarily to what prior research had suggested, the task is far from been solved, since IC models are not able to distinguish between a correct and incorrect caption.
Such results indicate that in current datasets, language provides priors that make it possible for a LaVi model to be successful without truly understanding and integrating language and vision. But problems do not stop at biasing. Johnson et al. (2016) highlight a second weakness of such datasets, pointing out that “they conflate multiple sources of error, making it hard to pinpoint model weaknesses”. The authors thus suggest the need for a diagnostic dataset. Thirdly, it has been observed that existing IC evaluation metrics are sen-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
sitive to n-gram overlap and that there is a need for measures that better substitute human judgments (Hodosh et al., 2013; Elliott and Keller, 2014; Anderson et al., 2016).
Our paper contributes to investigate these three weaknesses of current VQA and IC research by proposing an automatic method for creating a big dataset of real images with minimal language bias and some diagnostic abilities. Our dataset, called FOIL (Find One mismatch between Image and Language caption), consists of images associated with incorrect captions (Figure 1). We only introduce one error (or ‘foil’) per caption (i.e. the caption is ‘nearly’ correct). Specifically, we replace one word in the original caption with an incorrect one. This allows us to produce challenging error-detection/correction tasks while simultaneously providing ground truth (we know where the error is) that can be used to objectively measure the performance of current models. Given this data, we propose three tasks based on widely accepted evaluation measures. We evaluate a) the ability of the system to compute whether a caption is compatible (correct) or incompatible (foil) with the image (T1); b) when it is incompatible, what makes it so: the system has to highlight the mismatch in the caption (T2); c) which word should be used to correct the caption by replacing the foil word (T3). T1 is related to IC, but instead of generating a caption from scratch, we investigate the ability of the system to understand that the caption does not describe the image correctly. T2 and T3 are more related to VQA, however require a deep understanding of an image, as the aim is to spot a single mistake in an otherwise correct caption, with foils potentially close to the correct words.
The dataset presented in this paper (Section 3) is built on top of MS-COCO (Lin et al., 2014), and contains 297,268 datapoints and 97,847 images. We will refer to it as FOIL-COCO. We evaluate two state-of-the-art models: the popular VQA system proposed by Antol et al. (2015); Lu et al. (2015), and the attention-based model by Lu et al. (2016). We show that those models perform close to chance level, while humans are capable of doing the task accurately (Section 4). Section 5 provides an analysis of our results, allowing us to diagnose three failures of LaVi models. First, their coarse representations of language and visual input do not encode suitably structured information to spot mismatches between an utterance and the
corresponding scene (tested by T1). Second, their language representation is not fine-grained enough to identify the part of an utterance that causes a mismatch with the image as it is (T2). Third, their visual representation is also too poor to spot and name the visual area that corresponds to a marked mistake in the text (T3).
2 Related Work
Image captioning (IC) and visual question answering (VQA) tasks are the most relevant to our work. In IC (Fang et al., 2015; Chen and Lawrence Zitnick, 2015; Donahue et al., 2015; Karpathy and Fei-Fei, 2015; Vinyals et al., 2015), the goal is to generate a caption for a given image, such that it is both semantically and syntactically correct, and properly describes the content of that image. In VQA (Antol et al., 2015; Malinowski and Fritz, 2014; Malinowski et al., 2015; Gao et al., 2015; Ren et al., 2015), the system attempts to answer open-ended questions related to the content of a given image. There is a wealth of literature on both tasks, but we only discuss here the ones most related to our work and refer the reader to the recent surveys by (Bernardi et al., 2016; Wu et al., 2016).
Despite the successes of the state-of-the-art LaVi models, it is unclear whether they capture vision and language in a truly integrative fashion. We could identify three types of arguments surrounding the high performance of LaVi models:
(i) Triviality of the LaVi tasks: Recent work has shown that the LaVi models are mainly based on language priors (Ren et al., 2015; Agarwal et al., 2016; Kafle and Kanan, 2016) and even simple correlation and memorization (Zhou et al., 2015; Jabri et al., 2016; Hodosh and Hockenmaier, 2016) can result in a good superficial performance, without the underlying models truly understanding the visual content. Zhang et al. (2016) first unveiled that there exists a huge dataset bias in the popular VQA dataset by Antol et al. (2015): they showed that almost half of all the questions in this dataset can be answered correctly by using the question alone and ignoring the image completely. In the same vein, Zhou et al. (2015) proposed a simple baseline for the task of VQA. This baseline simply concatenates the Bag of Words (BoW) features from the question and Convolutional Neural Networks (CNN) features from the image to predict the answer. They showed that such a simple
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
method can achieve comparable performance to complex and deep architectures. Jabri et al. (2016) proposed a similar model for the task of multiple choice VQA, and suggested a cross-dataset generalization scheme as an evaluation criterion for VQA systems. We complement this research by introducing three new tasks with different levels of difficulty, on which LaVi models can be evaluated sequentially.
(ii) Need for diagnostics: To overcome the bias uncovered in previous datasets, several research groups have started proposing tasks which involve distinguishing distractors from a groundtruth caption for an image. Zhang et al. (2016) introduced a binary VQA task along with a dataset composed of sets of similar artificial images, allowing for more precise diagnostics of a system’s errors. Goyal et al. (2016a) balanced the dataset of Antol et al. (2015), collecting a new set of complementary natural images which are similar to existing items in the original dataset, but result in different answers to a common question. Hodosh and Hockenmaier (2016) also proposed to evaluate a number of state-of-the-art LaVi algorithms in the presence of distractors. Their evaluation was however limited to a small dataset (namely, Flickr30K (Young et al., 2014)) and the caption generation was based on a hand-crafted scheme using only inter-dataset distractors.
Most related to our paper is the work by Ding et al. (2016).1 Like us, they propose to extend the MS-COCO dataset by generating decoys from human-created image captions. They also suggest an evaluation apparently similar to our T1, requiring the LaVi system to detect the true target caption amongst the decoys. Our efforts, however, differ in some substantial ways. First, their technique to create incorrect captions (using BLEU to set an upper similarity threshold) is so that many of those captions will differ from the gold description in more than one respect. For instance, the caption two elephants standing next to each other in a grass field is associated with the decoy a herd of giraffes standing next to each other in a dirt field (errors: herd, giraffe, dirt) or with animals are gathering next to each other in a dirt field (error: dirt; infelicities: animals and gathering, which are both pragmatically odd). Clearly, the more the caption changes in the decoy, the eas-
1Please note that this paper was only published on arxiv a few weeks ago, at the end of December 2016. We have not yet evaluated their system against our data.
ier the task becomes. In contrast, the foil captions we propose only differ from the gold description by one word and are thus more challenging. Secondly, their automatic caption generation means that ‘correct’ descriptions can be produced, resulting in some confusion in human responses to the task. We made sure to prevent such cases, and human performance on our dataset is thus close to 100%. We note as well that our task does not require any complex instructions for the annotation, indicating that it is intuitive to human beings (see §4). Thirdly, their evaluation is a multiple-choice task, where the system has to compare all captions to understand which one is closest to the image. This is arguably a simpler task than the one we propose, where a caption is given and the system is asked to classify it as correct or foil: as we show in §4, detecting a correct caption is much easier than detecting foils. So evaluating precision on both gold and foil items is crucial.
Finally, (Johnson et al., 2016) proposed CLEVR, a dataset to aid in diagnostic evaluation of VQA systems. This dataset has been designed with the explicit goal of enabling detailed analysis of different aspects of visual reasoning, by minimizing dataset biases and providing rich groundtruth representations for both images and questions.
(iii) Lack of objective evaluation metrics: The evaluation of Natural Language Generation (NLG) systems is known to be a hard problem. It is further unclear whether the quality of LaVi models should be measured using the same metrics designed for language-only tasks. Elliott and Keller (2014) performed a sentence-level correlation analysis of NLG evaluation measures against expert human judgements in the context of IC. Their study revealed that most of those metrics were only weakly correlated with human judgements. In the same line of research, Anderson et al. (2016) showed that the most widely-used metrics for IC fail to capture semantic propositional content, which is an essential component of human caption evaluation. They proposed a semantic evaluation metric called SPICE, that measures how effectively image captions recover objects, attributes and the relations between them. In this paper, we tackle this problem by proposing tasks which can be evaluated based on objective metrics for classification/detection error.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
3 Dataset
In this section, we describe how we automatically generate FOIL-COCO datapoints, viz., image, original and foil caption triples. We used the training and validation Microsoft’s Common Objects in Context (MS-COCO) dataset (Lin et al., 2014) (2014 version) as our starting point. In MS-COCO, each image is described by at least five descriptions written by humans via Amazon Mechanical Turk (AMT). The images contains 91 common object categories (e.g. dog, elephant, bird, . . . and car, bicycle, ariplane, . . . ), from 11 supercategories (Animal, Vehicle, resp.), with 82 of them having more than 5K labeled instances. In total there are 123,287 images for captions (82,783 training and 40,504 validation sets).2
Our data generation process consists of four main steps: 1. Generation of candidate word-pairs (target::foil) replacement lists; 2. Splitting of replacement word pairs into training and testing pairs; 3. Generation of foil captions; 4. Mining of the hardest foil caption for each image-original caption pair. The last two steps are illustrated in
Figure 2.3
Generation of replacement word pairs (step 1) We want to replace one word in the original caption with a wrong word, we refer to them as target and foil word, respectively. We take target and foil words to be nouns, specifically the labels of MSCOCO categories, and we couple together words belonging to the same supercategory (e.g., bicycle::motorcycle, bicycle::car, bird::dog). In this step, we have used as our vocabulary 73 out of the 91 MS-COCO categories by leaving out those categories that were names with multi-word expressions (e.g. traffic light). We have obtained 472 target::foil word pairs.
Splitting of replacement word pairs into training and testing (step 2) To avoid the models learning trivial correlations due to replacement frequency, we randomly split, within each supercategory, the candidate target::foil pairs which are used to generate the captions of the training vs. test sets. We have obtained 256 pairs, built out of 72 target and 70 foil words, to generate the captions for the training set and 216 pairs, containing 73 target and 71 foil words, to generate the captions
2MS-COCO test set is not available for downloading. 3We will make our dataset available with also a validation
set.
for the test set.
Generation of foil captions (step 3) We would like to generate foil captions by replacing only target words which refer to visually salient objects. To this end, given an image, we replaced only those target words that occur in more than one MS-COCO caption associated with that image. Moreover, we want to use foils which are visually not present, viz. that refer to visual content not present in the image. Hence, given an image, an original caption and a target word, we replace the latter only with foils that are not among the labels (objects) annotated in MS-COCO for that image. We have used the images from MS-COCO training and validation sets to generate our training and test sets, respectively. We obtained 2,229,899 and 1,097,012 captions for our training and test sets, respectively.
Mining of the hardest foil caption for each image-original caption pair (step 4) To eliminate possible visual-language dataset bias, for each image and original caption, out of all foil captions generated in step 3, we selected only the hardest one. For this purpose, we need to model the visual-language bias of the dataset. To this end, we use Neuraltalk4 proposed in (Fei-Fei, 2015), one of the state-of-the-art image captioning systems, pre-trained on MS-COCO. Neuraltalk is based on an LSTM which takes as input an image and generates a sentence describing its content. We obtain a neural network N that implicitly represents the visual-language bias in its weights. We use N to approximate the conditional probability of a caption C given a dataset T and and an image I (P (C|I, T )). This is obtained simply using the loss l(C,N (I)) i.e., the error obtained by comparing the pseudo-ground truth C with the sentence predicted by N : P (C|I, T ) = 1 − l(C,N (I)). We refer to (Fei-Fei, 2015) for more details on how l() is computed. Finally, P (C|I, T ) is used to select, for an image and an original caption, the hardest foil among all the possible foil captions, viz., the one with the highest probability according to the dataset bias learned by N . Through this process, we obtained 197,788 and 99,480 original::foil caption pairs for the training and test sets, respectively. None of the target::foil word pairs has been filtered out by this mining process.
4https://github.com/karpathy/ neuraltalk
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
nr. of datapoints nr. unique images nr. of tot. captions nr. target::foil pairs Train 197,788 65,697 395,576 256 Test 99,480 32,150 198,960 216
Table 1: FOIL-COCO: a dataset consisting for each image of both a correct (original MS-COCO) caption and a wrong caption. The latter is created by replacing one of the nouns in the original caption (target word) with a foil noun.
Figure 2: The main aspects of the foil caption generation process. Left column: some of the original COCO captions associated with an image. In bold we highlight one of the target words (bicycle), chosen because it is mentioned by more than one annotator. Middle column: For each original caption and each chosen target word, different foil captions are generated by replacing the target word with all possible candidate foil replacements. Right column: A single caption is selected amongst all foil candidates. We select the ”hardest” caption, according to Neuraltalk model, trained using only the original captions.
The final FOIL-COCO dataset consists of 297,268 datapoints (197,788 in training and 99,480 in test set). All the 11 MS-COCO supercategories are represented in our dataset and contain 73 categories from the 91 MS-COCO ones (4.8 categories per supercategory on average.) Further details are reported in Table 1.
4 Experiments and Results
We have conducted three tasks aiming at understanding whether LaVi models can spot if there are mismatches between their coarse representations of language vs. visual inputs (T1); if their language representation is fine-grained enough to detect where the mismatch is (T2); if their visual representation is fine-grained enough so to be able to extract from it the information necessary to correct the foil (T3).
Task 1 (T1): Correct vs. foil classification Given an image and a caption, the model is asked to mark whether the caption is correct or wrong.
Task 2 (T2): Foil word detection Given an image and a foil caption, the model has to detect the foil word. In order to systematically check the system’s performance with different prior information, for this task, we have tested two different settings (a) the foil has to be selected among only the nouns in the caption, or (b) among all the content words in it.
Task 3 (T3): Foil word correction Given an image, a foil caption and the foil word, the model has to detect the foil one and provide its correction. For computational reasons, we have instantiated this task by asking models to correct the foil word by selecting the correct word from the target words, instead of from the whole dataset vocabulary (viz. more that 10K words).
4.1 Models
There are two possible approaches to solve the proposed tasks: by using IC models and classify the top ranked captions in the generation process;
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
or by using VQA models. State-of-the-art IC models are trained to generate captions using Recursive Neural Network (RNN), this make them not suitable for our tasks since our foil captions contain only one wrong word. These models are instead useful for selecting over n-captions the best one (Hodosh and Hockenmaier, 2016; Ding et al., 2016). Our classification task could be seen as a multiple choice question, where there are only two choices “correct” and “foil”. Therefore we have evaluated VQA models. In particular, we have used two of the three models evaluated in (Goyal et al., 2016) against a balanced VQA dataset.5
LSTM + norm I: We use the VQA best performing model in (Antol et al., 2015), namely deeper LSTM + norm I (Lu et al., 2015). This model uses a two stack Long-Short Term Memory (LSTM) to encode the questions and the last fully connected layer of VGGNet to encode the images. Both image embedding and caption embedding are projected into a 1024-dimensional feature space. Following (Goyal et al., 2016), we have normalized the image feature before projecting it into a new feature space. The combination of these two projected embeddings is performed by a point-wise multiplication. The multi-model representation so obtained is used for the classification, which is performed by multi-layer perceptron (MLP) classifier. The original VQA model is trained on 1000 most frequent answers, while in our case it is trained on only 2 possible answers i.e. “correct” or “foil” captions.
HieCoAtt We use the Hierarchical Co-Attention model proposed by (Lu et al., 2016) that coattends to both the image and the question to solve the task. In particular, we evaluated the alternate version, viz. the model that sequentially alternates between generating image and question attention, and does so in a hierarchical way by starting from the word-level, then going to the phrase and then to the entire question-level. These levels are combined recursively to produce the distribution over the foil vs. correct captions.
For the foil word detection task, we have applied the occlusion method to the models above. Following (Goyal et al., 2016b), we systematically
5We have not evaluated the Multimodal Compact Bilinear Pooling (Fukui et al., 2016) against our tasks, however in (Goyal et al., 2016) this model accuracy is higher than the HieCoAtt of 4.3% – which in the context of our results is a rather little difference.
occlude subsets of the language input, forward propagate the masked input through the model, and compute the change in the probability of the answer predicted with the unmasked original input. For the error correction task, we applied the linear regression method over all the target words and selected the target word which has the highest probability of making that wrong caption correct with respect to the given image.
Baselines We compare the SoA models above against the following baselines. For the classification task, we used a Blind model, viz. MLP+BoW (Jabri et al., 2016). This model only accepts captions as input to predict the answer. Differently from (Jabri et al., 2016) instead of concatenating BoW representations of caption and corresponding answer as input to MLP, we only input BoW representation of caption to the MLP network and use it as classifier. Apart from that, we have used the same number of parameters, viz. MLP has 2 hidden layers having 8,192 hidden units and dropout is used after the first layer. For the error detection and error classification tasks, we compare the models’ results against chance.6
Upper-bound Using Crowdflower, we collected human answers from 256 subjects for 352 image and caption tuples randomly selected from the test set. We collected 1056 judgements (av. 3.9 per raters) and for each tuple, we took the answer provided by at least 2/3 annotators. Subjects were given an image and a caption and had to decide whether it was correct or wrong (T1) and if they thought it was wrong, they had to write which was the foil word (T2). Annotators are nearly perfect in classifying captions and detecting foil words. Hence, though, we have collected human answers only on a rather small subset of the test set, we believe their results are representative of how easy are the tasks for humans.
4.2 Results
As shown in Table 2, FOIL-COCO dataset is challenging. The ‘blind’, language-only model does badly on T1 with an accuracy of 43.95% (19.53% on foil captions), demonstrating that language bias is minimal. The two state-of-the-art systems do significantly worse than humans on both T1 and
6The average number of nouns per caption is 4.3 and average number of content words (viz., after removing the stop words) is 6.3. Similary for T3, there are 72 possible target words for a given foil word.
7 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649
658
659
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
ACL 2017 Submission ***. Confidential Review Copy. DO NOT DISTRIBUTE.
T2. Systems show a strong bias towards correct captions and poor overall performance. They respectively only identify 34.51% (LSTM +norm I) and 36.38% (HieCoAtt) of the incorrect captions (T1). On the foil word detection task, they only reach 24.25% and 33.69% accuracy (T2). Their accuracy on foil word correction (T3) is extremely low, at 4.7% and 4.21% respectively. The result on T3 makes it clear that systems are unable to extract from the image representation the information needed to correct the foil: despite being told which element in the caption is wrong, they are not able to zoom into the correct part of the image to correct the foil word, or if they are, cannot name the object in that region.
5 Analysis
We performed a mixed-effect logistic regression analysis in order to check whether the performance of the models in T1 can be predicted by various linguistic variables (see results in Figure 3). We included: 1) semantic similarity between the original word and the foil (computed as the cosine between the two corresponding Word2Vec embeddings); 2) frequency of original word in FOILCOCO captions; 3) frequency of the foil word in FOIL-COCO captions; 4) length of the caption (number of words). The mixed-effect model was performed to get rid of possible effects due to either object supercategory (indoor, food, vehicle, etc.) or target::foil pair (e.g., zebra::giraffe, boat::airplane, etc.). For both LSTM + norm I and HieCoAtt, Word2Vec similarity, frequency of the original word, and frequency of the foil word turned out to be highly reliable predictors of the model’s response. The higher the values of these variables, the more the models tend to provide the wrong output. That is, when the foil word (e.g. cat) is semantically very similar to the original one (e.g. dog), the models tend to wrongly classify the caption as ‘correct’. The same holds for frequency values. In particular, the higher the frequency of both the original word and the foil one, the more the models fail. This indicates that systems find it difficult to distinguish related concepts at the textvision interface, and also that they may tend to be biased towards frequently occurring concepts, ‘seeing them everywhere’ even when they are not present in the image. Caption length turned out to be only a partially reliable predictor in the LSTM + norm I model, whereas it is a reliable predictor
600 601 602 603 604 605 606 607 608 609 610 611 612 613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650 651 652 653 654 655 656 657 658 659 660 661 662 663
664
665
672
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
95
696
697
698
699
ACL 2017 Submission ***. Confidential Review Copy. DO NOT DISTRIBUTE. T1: Classification task Overall Correct Foil Blind 43.95 68.36 19.53 LSTM + norm I 63.26 92.02 34.51 HieCoAtt 64.14 91.89 36.38 Human 91.48 91.67 91.28 T2: Error detection task only nouns all content words Chance 23.25 15.87 LSTM + norm I 26.32 24.25 HieCoAtt 38.79 33.69 Human 100 T3: Error correction task all target words Chance 1.38 LSTM + norm I 4.7 HieCoAtt 4.21
Table 2: T1: Accuracy results on the classification task, relativelly to all the datapoints (overall) and by type of caption (correct vs. foil); T2: Accuracy results on the error detection task, when the foil is known to be among the nouns only or when it is know to be among all the content words;7 T3: Accuracy results
on the error correction task when the correct word has to be choosen among any of the target words
object super-categ ry (indoor, food, vehicle, etc.) or original word/foil word. For both LSTM Q + norm I and HieCoAtt, Word2Vec similarity, frequency of the original word, a d frequ ncy of the foil word turned out to be highly reliable predictors of the model’s res onse. The higher the values of these variables, the more the models tend to provide the wrong output. That is, when the foil word (e.g. cat) is semantically very similar to the original one (e.g. dog), the models tend to wrongly classify the caption as ‘correct’. The same holds for frequency values. In particular, the higher the frequency of both the original word and the foil one, the more the odels fail. This indicates that systems find it difficult to distinguish related concepts at the text-vision interface, and also that they may tend to be biased towards frequently occurring concepts, ‘seeing them everywhere’ even when they are not present in th image.
Caption length turned out to be only a partially reliable predictor in the LSTM + norm I mod l, whereas it is a reliable predictor in HieCoAtt. In particular, the longer the foil caption, the higher the pr bability that the model will wrongly label the caption as correct. Intuitively, the longer the caption, the harder for the model to spot that there is a foil word that makes the caption wrong.
We also checked the average precision of several object d t ction models on the categories in our dataset. For each category (e.g. ‘dog’), we calculated how likely the model was to localize that object in the images that contained it. We obtained between 24% and 36% average precision
AH: Correct numbers, showing that detection on our dataset is not an easy task. However, we cannot conclude that the systems performed badly for that reason only. Indeed, there is low correlation between object detection precision and model accuracy (Pearson 0.0363 for the LSTM + norm I, 0.179 for HieCoAtt on Task 1). While this is only an indicative result (because the models used for object detection are different from the endto-end VQA models tested here), we can assume that other challenges lower the performance of the tested systems.
Finally, we checked whether there was any correlation between results and the position of the foil in the sentence, to ensure the models did not profit from any artefact of the data. We did not find any such correlation.
RB: add comments on the example with accuracy per pairs.
6 Conclusions
To trust students truely understand a task, teachers check whether they can correct mistakes. This is what we have done with the FOIL-COCO dataset. The mistakes have been carefully thought, but au-
in HieCoAtt. In particular, the longer the foil caption, the higher the probability that the model will wrongly label the caption as correct. Intuitively, the longer the caption, the harder for the model to spot th th re is a foil word hat makes th caption wrong. As revealed by the fairly high variance explained by the random effect related to target::foil pairs in the regression analysis, both models perform very well on some target::foil pairs, but fail on some others. (See Table 3 for same examples of easy/hard target::foil pairs.)
We also checked the average precision of several object detection models on the categories in our dataset. For each category (e.g. ‘dog’), we calculated how likely the model was to localize that object in th images that contained it. W obtained between 25.12% (Accessory) and 57.58% (Animal) average precision, showing that detection on our dataset is not an easy task. However, we cannot conclude that the systems performed badly for that reason only. Indeed, there is low correlation between object detection precision and model accuracy (Pearson −0.0363 for the LSTM + norm I, 0.179 for HieCoAtt on T1). While this is only an indicative result (because the models used for object detection are different from the endto-end VQA models tested here), we can assume that other challenges lower the performance of the tested syst ms.
Finally, we checked whether there was any correlation between results and the position of the foil in the sentence, to ensure the models did not profit from any artefact of the data. We did not find any such correlation.
6 Conclusion
We have introduced FOIL-COCO, a large dataset of images associated with both correct and foil captions. The error production is automatically
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
T1: Classification task Overall Correct Foil
Blind 43.95 68.36 19.53 LSTM + norm I 63.26 92.02 34.51 HieCoAtt 64.14 91.89 36.38 Human 91.48 91.67 91.28
T2: Foil word detection task only nouns all content words
Chance 23.25 15.87 LSTM + norm I 26.32 24.25 HieCoAtt 38.79 33.69 Human 100
T3: Foil word correction task all target words
Chance 1.38 LSTM + norm I 4.7 HieCoAtt 4.21
Top-5 Bottom-5 T1: LSTM + norm I
racket::glove 100 motorcycle::ariplane 0 racket::kite 97.29 bicycle::ariplane 0 couch::toilet 97.11 drier::scissors 0 racket::skis 95.23 bus::ariplane 0.35 giraffe::sheep 95.09 zebra::giraffe 0.43 T1: HieCoAtt tie::handbag 100 drier::scissor 0 snowboard::glove 100 fork::glass 0 racket::skis 100 handbag::tie 0 racket::glove 100 motorcycle::airplane 0 backpack::handbag 100 train::airplane 0
Top-5 Bottom-5 T2: LSTM + norm I
drier::scissor 100 glove::skis 0 zebra::giraffe 88.98 snowboard::racket 0 boat::airplane 87.87 donut::apple 0 truck::airplane 85.71 glove::surfboard 0 train::airplane 81.93 spoon::bottle 0 T2: HieCoAtt zebra::elephant 94.92 direr::scissors 0 backpack::handbag 94.44 handbag::tie 0 cow::zebra 93.33 broccolli:orange 1.47 bird::sheep 93.11 zebra::giraffe 1.96 orange::carrot 92.37 boat::airplane 2.09
Table 3: Easiest and hardest target::foil pairs: T1 (caption classification) and T2 (foil word detection).
generated, but carefully thought out, making the task of spotting foils particularly challenging. By associating the dataset with a series of tasks, we allow for diagnosing various failures of current LaVi systems, from their coarse understanding of the correspondences between text and vision to their grasp of language and image structure.
Our hypothesis is that systems which, like humans, deeply integrate the language and vision modalities, should spot foil captions quite easily. The state-of-the art LaVi models we have tested fall through that test, implying that they fail to integrate the two modalities. To complete the analysis of these results, we plan to carry out a further task, namely ask the system to detect in the image the area that produces the mismatch with the foil word (the red box around the bird in Figure 1.) This extra step would allow us to fully diagnose the failure of the tested systems and confirm what is implicit in our results from task 3:
that the algorithms are unable to map particular elements of the text to their visual counterparts. We note that the addition of this extra step will move this work closer to the textual/visual explanation research (e.g., (Park et al., 2016; Selvaraju et al., 2016)). We will then have a pipeline able to not only test whether a mistake can be detected, but also whether the system can explain its decision: ‘the wrong word is dog because the cyclists are in fact approaching a bird, there, in the image’.
LaVi models are a great success of recent research, and we are impressed by the amount of ideas, data and models produced in this stimulating and attractive area. With our work, we would like to push the community to think of ways the models can better merge language and vision rather than merely use one as a supplement to the other.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

In this work, the authors extend MS-COCO by adding an incorrect caption to each existing caption, with only one word of difference. The authors demonstrate that two state-of-the-art methods (one for VQA and one for captioning) perform extremely poorly at a) determining if a caption is fake, b) determining which word in a fake caption is wrong, and c) selecting a replacement word for a given fake word.  This work builds upon a wealth of literature regarding the underperformance of vision language models relative to their apparent capacities. I think this work makes concrete some of the big, fundamental questions in this area: are vision language models doing  interesting  things, or not? The authors consider a nice mix of tasks and models to shed light on the  broken-ness  of these settings, and perform some insightful analyses of factors associated with model failure (e.g., Figure 3).  My biggest concerns with the paper are similarity to Ding et al. That being said, I do think the authors make some really good points; Ding et al. generate similar captions, but the ones here differ by only one word and *still* break the models -- I think that's a justifiably fundamental difference. That observation demonstrates that Ding et al.'s engineering is not a requirement, as this simple approach still breaks things catastrophically.  Another concern is the use of NeuralTalk to select the  hardest  foils.              While a clever idea, I am worried that the use of this model creates a risk of self-reinforcement bias, i.e., NeuralTalk's biases are now fundamentally  baked-in  to FOIL-COCO.   I think the results section could be a bit longer, relative to the rest of the paper (e.g. I would've liked more than one paragraph -- I liked this part!)  Overall, I do like this paper, as it nicely builds upon some results that highlight defficiencies in vision language integration. In the end, the Ding et al. similarity is not a  game-breaker,  I think -- if anything, this work shows that vision language models are so easy to fool, Ding et al.'s method is not even required.  Small things:  I would've liked to have seen another baseline that simply concatenates BoW + extracted CNN features and trains a softmax classifier over them. The  blind  model is a nice touch, but what about a  dumb  vision+langauge baseline? I bet that would do close to as well as the LSTM Co-attention. That could've made the point of the paper even stronger.  330: What is a supercategory? Is this from WordNet? Is this from COCO? I understand the idea, but not the specifics.  397: has been to were  494: that to than  693: artefact to undesirable artifacts (?)  701: I would have included a chance model in T1's table -- is 19.53% [Line 592] a constant-prediction baseline? Is it 50% (if so, can't we flip all of the  blind  predictions to get a better baseline?) I am not entirely clear, and I think a  chance  line here would fix a lot of this confusion.  719: ariplane  ~~ After reading the author response...  I think this author response is spot-on. Both my concerns of NeuralTalk biases and additional baselines were addressed, and I am confident that these can be addressed in the final version, so I will keep my score as-is.


Morphology Generation for Statistical Machine Translation using Deep Learning Techniques
Morphology in unbalanced languages remains a big challenge in the context of machine translation. In this paper, we propose to de-couple machine translation from morphology generation in order to better deal with the problem. We investigate the morphology simplification with a reasonable trade-off between expected gain and generation complexity. For the Chinese-Spanish task, optimum morphological simplification is in gender and number. For this purpose, we design a new classification architecture which, compared to other standard machine learning techniques, obtains the best results. This proposed neural-based architecture consists of several layers: an embedding, a convolutional followed by a recurrent neural network and, finally, ends with sigmoid and softmax layers. We obtain classification results over 98% accuracy in gender classification, over 93% in number classification, and an overall translation improvement of 0.7 METEOR.
1 Introduction
Machine Translation (MT) is evolving from different perspectives. One of the most popular paradigms is still Statistical Machine Translation (SMT), which consists in finding the most probable target sentence given the source sentence using probabilistic models based on co-ocurrences. Recently, deep learning techniques applied to natural language processing, speech recognition and image processing and even in MT have reached quite successful results. Early stages of deep learning applied to MT include using neural language modeling for rescoring (Schwenk et al., 2007). Later,
deep learning has been integrated in MT in many different steps (Zhand and Zong, 2015). Nowadays, deep learning has allowed to develop an entire new paradigm, which within one-year of development has achieved state-of-the-art results (Jean et al., 2015) for some language pairs.
In this paper, we are focusing on a challenging translation task, which is Chinese-to-Spanish. This translation task has the characteristic that we are going from an isolated language in terms of morphology (Chinese) to a fusional language (Spanish). This means that for a simple word in Chinese (e.g. 鼓励 ), the corresponding translation has many different morphology inflexions (e.g. alentar, alienta, alentamos, alientan ...), which depend on the context. It is still difficult for MT in general (no matter which paradigm) to extract information from the source context to give the correct translation.
We propose to divide the problem of translation into translation and then a postprocessing of morphology generation. This has been done before, e.g. (Toutanova et al., 2008; Formiga et al., 2013), as we will review in the next section. However, the main contribution of our work is that we are using deep learning techniques in morphology generation. This gives us significant improvements in translation quality.
The rest of the paper is organised as follows. Section 2 describes the related work both in morphology generation approaches and in ChineseSpanish translation. Section 3 overviews the phrase-based MT approach together with an explanation of the divide and conquer approach of translating and generating morphology. Section 4 details the architecture of the morphology generation module and it reports the main classification techniques that are used for morphology generation. Section 5 describes the experimental framework. Section 6 reports and discusses both clas-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
sification and translation results, which show significant improvements. Finally, section 7 summarises the main conclusions and further work.
2 Related Work
In this section we are reviewing the previous related works on morphology generation for MT and on Chinese-Spanish MT approaches.
Morphology generation There have been many works in morphological generation and some of them are in the context of the application of MT. In this cases, MT is faced in two-steps: first step where the source is translated to a simplified target text that has less morphology variation than the original target; and then, second step, a postprocessing module (morphology generator) adds the proper inflections. To name a few of these works, for example, (Toutanova et al., 2008) build maximum entropy markov models for inflection prediction of stems; (Clifton and Sarkar, 2011) and (Kholy and Habash, 2012) use conditional random fields (CFR) to predict one or more morphological features; and (Formiga et al., 2013) use Support Vector Machines (SVMs) to predict verb inflections. Other related works are in the context of Part-of-Speech (PoS) tagging generation such as (Giménez and Màrquez, 2004) in which a model is trained to predict each individual fragment of a PoS tag by means of machine learning algorithms. The main difference is that in PoS tagging the word itself has information about morphological inflection, whereas in our task, we do not have this information.
In this paper, we use deep learning techniques to morphology generation or classification. Based on the fact that Chinese does not have number and gender inflections and Spanish does, (Costajussà, 2015) show that simplification in gender and number has the best trade-off between improving translation and keeping the morphology generation complexity at a low level.
Chinese-Spanish There are few works in Chinese-Spanish MT despite being two of the most spoken languages in the world. Most of these works are based on comparing different pivot strategies like standard cascade or pseudo-corpus (Costa-jussà et al., 2012). Also it is important to mention that, in 2008, there were two tasks organised by the popular IWSLT
evaluation campaign1 (International Workshop on Spoken Language Translation) between these two languages (Paul, 2008). The first task was based on a direct translation for Chinese-Spanish. The second task provided corpus in Chinese-English and English-Spanish and asked participants to provide Chinese-Spanish translation through pivot techniques. The second task obtained better results than direct translation because of the larger corpus provided. Differently, (Costa-jussà and Centelles, 2016) present the first rule-based MT system for Chinese to Spanish. Authors describe a hybrid method for constructing this system taking advantage of available resources such as parallel corpora that are used to extract dictionaries and lexical and structural transfer rules. Finally, it is worth mentioning that novel successful neural approximations (Jean et al., 2015), already mentioned in the introduction, have not yet achieved state-of-the-art results for this language pair (Costa-jussà et al., 2017).
3 Machine Translation Architecture
In this section, we review the baseline system which is a standard phrase-based MT system and explain the architecture that we are using by dividing the problem of translation into: morphologically simplified translation and morphology generation.
3.1 Phrase-based MT baseline system
The popular phrase-based MT system (Koehn et al., 2003) focuses on finding the most probable target text given a source text. In the last 20 years, the phrase-based system has dramatically evolved introducing new techniques and modifying the architecture; for example, replacing the noisychannel for the log-linear model which combines a set of feature functions in the decoder, including the translation and language model, the reordering model and the lexical models. There is a widely used open-source software, Moses (Koehn et al., 2007), which englobes a large community that helps in the progress of the system. As a consequence, phrase-based MT is a commoditized technology used at the academic and commercial level. However, there are still many challenges to solve, such as morphology generation.
1http://iwslt2010.fbk.eu
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
3.2 Divide and conquer MT architecture: simplified translation and morphology generation
Morphology generation is not always achieved in the standard phrase-based system. This may be due to the fact that phrase-based MT uses a limited source context information to translate. Therefore, we are proposing to follow a similar strategy to previous works (Toutanova et al., 2008; Formiga et al., 2013), where authors do a first translation from source to a morphology-based simplified target and then, use the morphology generation module that transforms the simplified translation into the full form output.
4 Morphological Generation Module
In order to design the morphology generation module, we have to decide the morphology simplification we are applying to the translation. Since we are focusing on Chinese-to-Spanish task and based on (Costa-jussà, 2015), the simplification which achieves the best trade-off among highest translation gain and lowest complexity of morphological generation is the simplification in number and gender. Table 1 shows examples of this simplification. The main challenge of this task is that number and gender are generated from words where this inflection information (both number and gender) has been removed beforehand.
With these results at hand, we propose an architecture of the morphology generation module, which is language independent and it is easily generalizable to other simplification schemes.
The morphology generation procedure is summarised as follows and further detailed in the next subsections.
• Feature selection. We have investigated different set of features including information from both source and target languages.
• Classification. We propose a new deep learning classification architecture composed of different layers.
• Rescoring and rules. We generate different alternatives of the classification output and rerank them using a language model. After, we use hand-crafted rules that allow to solve some specific problems.
This procedure is depicted in Figure 1, in which we can see that each of the above processes gener-
Figure 1: Block Diagram for Morphology Generation
ates the needed input for the next step. Figure also shows in red the main subprocesses that have been developed on this work.
4.1 Feature selection
We propose to compare several features for recovering morphological information. Given that both Chinese and simplified Spanish languages do not contain explicit morphology information, we start by simply using windows of words as source of information. We follow Collobert’s approach in which each word is represented by a fixed size window of words in which the central element is the one to classify (Collobert et al., 2011).
In our case, we experiment with three different inputs: (1) Chinese window; (2) simplified Spanish window; (3) Spanish window adding information about its correspondant word in Chinese, i.e. information about pronouns and the number of characters in the word. The main advantage of the second one is that it is not dependant on the alignment file generated during translation.
Our classifiers did not have to train all types of words. Some types of words, such as prepositions (a, ante, cabo, de...), do not have gender or number. Therefore our system was trained using only determiners, adjectives, verbs, pronouns and nouns which are the ones that present morphology
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Esnum decidir[VMIP3N0] examinar[VMN0000] el[DA0MN0] c uestión[NCFN000] en[SPS00] el[DA0MN0] perı́odo[NCM N000] de[SPS00] sesión[NCFN000] el[DA0MN0] tema[NCMN000] titular [AQ0MN0] “[Fp] cuestión[NCFN000] relativo[AQ0FN0] a[SPS00] el[DA0MN0] derecho[NCMN000] humano[AQ0MN0] “[Fp] .[Fp] Esgen decidir[VMIP3S0] examinar[VMN0000] el[DA0GS0] cuestión [NCGS000] en[SPS00] el[DA0GS0] perı́odo[NCGS000] de [SPS00] sesión[NCGS000] el[DA0GS0] tema[NCGS000] titular [AQ0GS0] “[Fp] cuestión[NCGS000] relativo[AQ0GS0] a[SPS00] el[DA0GS0] derecho[NCGS000] humano[AQ0GS0] “[Fp] .[Fp] Esnumgen decidir[VMIP3N0] examinar[VMN0000] el[DA0GN0 ] cuestión[NCGN000] en[SPS00] el[DA0GN0] perı́odo[NCGN000] de[SPS00] sesión[NCGN000] el[DA0GN0] tema[NCGN000] titular [AQ0GN0] “[Fp] cuestión[NCGN000] relativo[AQ0GN00 ] a[SPS00] el[DA0GN0] derecho[NCGN000] humano[AQ0GN0] “[Fp] .[Fp] Es Decide examinar la cuestión en el perı́odo de sesiones el tema titulad o “ Cuestiones relativas a los derechos humanos ” .
Table 1: Example of Spanish simplification into number, gender and both
.
variations in gender or number. However, note that all types of words are used in the windows.
4.2 Classification architecture
Description We propose to train two different models: one to retrieve gender and another to retrieve number. Each model decides among three different classes. Classes for gender classifier are masculine (M ), femenine (F ) and none (N ); and classes for number classifier are singular (S), plural (P ) and none (N ) 2. Again, we inspire our architecture in previous Collobert’s proposal and we modify it by adding a recurrent neural network. This recurrent neural network is relevant because it keeps information about previous elements in a sequence and, in our classification problem, context words are very relevant. As a recurrent neural network, we use a Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) that is proven efficient to deal with sequence NLP challenges (Sutskever et al., 2014). This kind of recurrent neural network is able to maintain information for several elements in the sequence and to forget it when needed. Figure 2 shows an overview of the different layers involved in the final classification architecture, which are detailed as follows:
Embedding. We represent each word as its index in the vocabulary, i.e. every word is represented as one discrete value: E(w) = W,W ∈ Rd, w being the index of the word in the sorted vocabulary and d the user chosen size of the array. Then, each word is represented as a numeric array and each window is a matrix.
Convolutional. We add a convolutional neural network. This step allows the system to detect some common patterns between the different
2None means that there is no need to specify gender or number information because the word is invariant in these terms. This happens for types of words (determiners, nouns, verbs, pronouns and adjectives) that can have gender and number in other cases.
words. This layer’s input consists in W l matrix of multidimensional arrays of size n · d, where n is the window length (in words) and d is the size of the array created by the previous embedding layer. This layer’s output is a matrix of the same size as the input.
Max Pooling. This layer allows to extract most relevant features from the input data and reduces feature vectors to half.
LSTM. Each feature array is treated individually, generating a fixed size representation hi of the ith word using information of all the previous words (in the sequence). This layer’s output, h, is the result of the last element of the sequence using information from all previous words.
Sigmoid. This layer smoothes results obtained by previous layer and compresses results to the interval [−1, 1]. This layer’s input is a fixed size vector of shape 1 ·n where n is the number of neurons in the previous LSTM layer. This layer’s output is a vector of length c equal to the number of classes to predict.
Softmax. This layer allows to show results as probabilities by ensuring that the returned value of each class belongs to the [0, 1) interval and all classes add up 1.
Motivation Our input data is PoS tagged and morphogically simplified before the classification architecture which largely reduces the information that can be extracted from individual words in the vocabulary. In addition, we can encounter out-ofvocabulary words for which no morphological information can be extracted.
The main source of information available is the context in which the word can be found in the sentence. Considering the window as a sequence enforces the behaviour a human would have while reading the sentence. The information of a word consists in itself and the words that surround it. Sometimes information preceeds the word and sometimes information is after the word. Words
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Figure 2: Neural network overview.
(like adjetives), which are modifying or complementing another word, generally take information from preceeding words. For example, in the sequence casa blanca, the word blanca could also be blanco, blancos or blancas but because noun and adjective are required to have gender and number agreement, the femenine word casa forces the femenine for blanca. While, for example, determiners usually take information from posterior words. This fact motivates that the word to classify has to be placed at the center of the window.
Finally, given that we rely only on the context information since words themselves may not have any information, makes the recurrent neural network a key element in our architecture. The output h of the layer can be considered a context vector of the whole window maintaining information of all the previously encountered words (in the same window).
4.3 Rescoring and rules
At this point in the pipeline, we have two models (gender and number) that allow us to generate the full Part-of-Speech (PoS) tag by combining the best results of both classifiers.
However, in order to improve the overall performance, we add a rescoring step followed by some hand-crafted rules. To generate the different alternatives, we represent every sentence as a graph (see Figure 3), with the following properties:
• Each word is represented as a layer of the graph and each node represents a class of the
classification model.
• A node only has edges with all the nodes of the next layer. This way we force the original sentence order.
• An edge’s weight is the probability given by the classification model.
• Each accepted path in the graph starts in the first layer and ends in the last one. This acyclic structure finds the best path in linear time, due to the fact that it goes through all layers and it picks the node with the greatest weight. One layer can have either 1 element (the word does not need to be classified, e.g. prepositions) or 3 elements (the word needs to be classified among the three number or gender categories).
• Add the weight of a previously trained target language model.
We used Yen’s algorithm (Yen, 1971) to find the best path, which has an associated cost of O(KN2logN), being K the number of paths to find.
See pseudo code in Algorithm 1, where A is the set paths chosen in the graph. The algorithm ends when this A set contains K paths or no further paths available to explore. B contains all suboptimal paths that can be elected in future iterations.
There are two special cases that the models were not able to treat and we apply specific rules: (1) conjunctions y and o are replaced by e and u if they are placed in front of vowels. This could not be generated during translation because both words share the same tag and lemma; (2) verbs with a pronoun as a suffix, producirse, second to
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
last syllabe stretched (palabras llanas) and ending in a vowel are not accentuated. However, after adding the suffix, these words should be accentuated because they become palabras esdrújulas, which happen to be always accentuated.
Data: G Graph of the sentence, K Result: best k paths in G initialization; A[0] = bestPath(G,0,final); B = [] ; i = 0; for i <K do
for i in range(0, len(A[K-1])-1) do spurNode = A[K-1][i]; root = A[K-1][0;i]; for path in A do
if root = path[0:i] then remove edge(i-1,i) from G;
end end for node in root and node != spurNode do
removes node node from G; end spurPath = bestPath(G,spurnode, final) totalPath = root + spurPath; B.append(totalPath); restore edges from G; restore nodes from G; if B is empty then
break; end B.sort(); A.append(B[0]); B.pop();
end end
Algorithm 1: Pseudo-code for k-best paths generation.
5 Experimental framework
In this section, we describe the data used for experimentation together with the corresponding preprocessing. In addition, we detail chosen parameters for the MT system and the classification algorithm.
5.1 Data and preprocessing
One of the main contributions of this work is using the Chinese-Spanish language pair. In the last years, there has appeared more and more resources for this language pair available in (Ziemski et al.,
L Set S W V ES Train Small 58.6K 2.3M 22.5K Large 3.0M 51.7M 207.5K
Development 990 43.4K 5.4k Test 1K 44.2K 5.5K
ZH Train Small 58.6K 1.6M 17.8K Large 3.0M 43.9M 373.5K
Development 990 33K 3.7K Test 1K 33.7K 3.8K
Table 2: Corpus Statistics. Number of sentences (S),words (W), vocabulary (V). M stands for millions and K stands for thousands.
2016) or from TAUS corporation3. Therefore, differently from previous works on this language pair, we can test our approach in both a small and large data sets.
• A small training corpus by using the United Nations Corpus (UN) (Rafalovitch and Dale, 2009).
• A large training corpus by using, in addition to the UN corpus, the TAUS corpus, the Bible corpus (Chew et al., 2006) and the BTEC (Basic Traveller Expressions Corpus) (Takezawa, 2006). The TAUS corpus is around 2,890,000 sentences, the Bible corpus about 30,000 sentences and the BTEC corpus about 20,000 sentences.
Corpus statistics are shown in Table 2. Development and test sets are taken from UN corpus.
Corpus preprocessing consisted in tokenization, filtering empty sentences and longer than 50 words, Chinese segmentation by means of the ZhSeg (Dyer, 2016), Spanish lowercasing, filtering pairs of sentences with more than 10% of nonChinese characters in the Chinese side and more than 10% of non-Spanish characters in the Spanish side. Spanish PoS tagging was done using Freeling (Padró and Stanilovsky, 2012). All chunking or name entity recognition was disabled to preserve the original number of words.
5.2 MT Baseline
Moses has been trained using default parameters, which include: grow-diag-final word alignment symmetrization, lexicalized reordering, relative frequencies (conditional and posterior probabilities) with phrase discounting, lexical weights, phrase bonus, accepting phrases up to length 10, 5-gram language model with kneser-ney smoothing, word bonus and MERT optimisation.
5.3 Classification parameters
To generate the classification architecture we used the library keras (Chollet, 2015) for creating and ensambling the different layers. Using NVIDIA GTX Titan X GPUs with 12GB of memory and 3072 CUDA Cores, each classifier is trained on aproximately 1h and 12h for the small and large corpus, respectively.
3http://www.taus.net
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Regarding classification parameters, experimentation has shown that number and gender classification tasks have different requirements. Table 3 summarizes these parameters. The best window size is 9 and 7 words for number and gender, respectively. In both cases increasing this size lowers the accuracy of the system. The vocabulary size is fixed as a trade-off between giving enough information to the system to perform the classification while removing enough words to train the classifier for unknown words. The embedding size of 128 results in stable training, while further increasing this value augmented the training time and hardware cost. The filter size in the convolutional layer reached best results when it was slightly smaller than the window size, being 7 and 5 the best values for number and gender classification, respectively. Finally, increasing LSTM nodes up to 70 improved significantly for both classifiers.
Table 3: Values of the different parameters of the classifiers
Parameter Small Large Num Gen Num Gen Window size 9 7 9 7 Vocabulary size 7000 9000 15000 15000 Embedding 128 128 128 128 Filter size 7 5 7 5 LSTM nodes 70 70 70 70
For windows, we only used the simplified Spanish translation. In Table 4 we can see that testing different sources of information with the classifier of number for the small corpus. Adding Chinese has a negative effect in the classifier accuracy.
5.4 Rescoring and Full form generation
As a rescoring tool, we use the one available in Moses 4. We trained a standard n-gram language model with the SRILM toolkit (Stolcke, 2002).
In order to generate the final full form we use the full PoS tag, generated from the postprocessing step, and the lemma, taken from the morphology-simplified translation output. Then, we use the vocabulary and conjugations rules pro-
4https://github.com/mosessmt/mosesdecoder/tree/master/scripts/nbest-rescore
Table 4: Accuracy of the classifier of number using different sources of information.
Features Accuracy (%) Chinese window 72 Spanish window 93,4 Chinese + Spanish window 86
vided by Freeling. Freeling’s coverage raises the 99%. When a word is not found in the dictionary, we test all gender and/or number inflections in descendant order of probability until a match is found. If none matched, the lemma is used as translation, which usually happens only in the case of cities or demonyms.
6 Evaluation
In this section we discuss the results obtained both in classification and in the final translation task.
Table 5 shows results for the classification task both number and gender and with the different corpus sets. We have contrasted our proposed classification architecture based on neural networks with standard machine learning techniques such as linear, cuadratic and sigmoid kernels SVMs (Cortes and Vapnik, 1995), random forests (Breiman, 2001), convolutional(Fukushima, 1980) and LSTM(Hochreiter and Schmidhuber, 1997) neural networks (NN). All algorithms were tested using features and parameters described in previous sections with the exception of random forests in which we added the one hot encoding representation of the words to the features.
We observe that our proposed architecture achieves by large the best results in all tasks. It is also remarkable that the accuracy is lower using the bigger corpus, this is due to the fact that the small set consisted in texts of the same domain and the vocabulary had a better representation of specific words such as country names.
Table 5: Classification results. In bold, best results. Num stands for Number and Gen, for Gender
Algorithm Small Large Num Gen Num Gen Naive Bayes 61.3 53.5 61.3 53.5 Lineal kernel SVM 68.1 71.7 65.8 69.3 Cuadratic kernel SVM 77.8 81.3 77.6 82.7 Sigmoid kernel SVM 83,1 87.4 81.5 84.2 Random Forest 81.6 91.8 77.8 88.1 Convolutional NN 81.3 93.9 83.9 94.2 LSTM NN 68.1 73.3 70.8 71.4 CNN + LSTM 93.7 98.4 88.9 96.1
Tabla 6 shows translation results. We show both the Oracle and the result in terms of METEOR (Banerjee and Lavie, 2005). We observe improvement in most cases (when classifying number, gender, both and rescoring), but best results are obtained when classifying number and gender and rescoring number in the large corpus, obtaining a
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Table 6: METEOR results. In bold, best results. Num stands for Number and Gen, for Gender
Set System UN Oracle Result Small Baseline - 55.29 +Num 55.60 55.35 +Gen 55.45 55.39 +Num&Gen 56.81 55.48 +Num&Gen +Rescoring(Num&Gen) - 54.91 +Num&Gen +Rescoring(Num) - 55.56 Large Baseline - 56.98 +Num 58.87 57.51 +Gen 57.56 57.32 +Num&Gen 62.41 57.13 +Num&Gen Rescoring - 57.74
gain up to +0.7 METEOR. Rescoring step improves final results. Note that rescoring was only applied to number classification because gender classification model has a low classification error (bellow 2%) which makes it harder to further decrease it. Additionally, gender and number classification scores are not be comparable and not easily integrated in Yen’s algorithm.
7 Conclusions
Chinese-to-Spanish translation task is challenging, specially because of Spanish being morphologically rich compared to Chinese. Main contributions of this paper include correctly de-coupling the translation and morphological generation tasks and proposing a new classification architecture, based on deep learning, for number and gender.
Standard phrase-based MT procedure is changed to first translating into a morphologically simplified target (in terms of number and gender); then, introducing the classification algorithm, based on a new proposed neural network-based architecture, that retrieves the simplified morphology; and composing the final full form by using the standard Freeling dictionary.
Results of the proposed neural-network architecture in the classification task compared to standard algorithms (SVM or random forests) are significantly better and results in the translation task achieve up to 0.7 METEOR improvement. As further work, we intend to further simplify morphology and extend the scope of the classification.
Acknowledgements

This paper presents a method for generating morphology, focusing on gender and number, using deep learning techniques. From a morphologically simplified Spanish text, the proposed approach uses a classifier to reassign the gender and number for each token, when necessary. The authors compared their approach with other learning algorithms, and evaluated it in machine translation on the Chinese-to-Spanish (ZhtoEs) translation direction.  Recently, the task of generating gender and number has been rarely tackled, morphology generation methods usually target, and are evaluated on, morphologically-rich languages like German or Finnish. However, calling the work presented in this paper  morphology generation  u00a0is a bit overselling as the proposed method clearly deals only with gender and number. And given the fact that some rules are handcrafted for this specific task, I do not think this method can be straightforwardly applied to do more complex morphology generation for morphologically-rich languages.  This paper is relatively clear in the sections presenting the proposed method. A lot of work has been done to design the method and I think it can have some interesting impact on various NLP tasks. However the evaluation part of this work is barely understandable as many details of what is done, or why it is done, are missing. From this evaluation, we cannot know if the proposed method brings improvements over state-of-the-art methods while the experiments cannot be replicated. Furthermore, no analysis of the results obtained is provided. Since half a page is still available, there was the possibility to provide more information to make more clear the evaluation. This work lacks of motivation. Why do you think deep learning can especially improve gender and number generation over state-of-the-art methods?  In your paper, the word  contribution  should be used more wisely, as it is now in the paper, it is not obvious what are the real contributions (more details below).   abstract: what do you mean by unbalanced languages?  section 1: You claim that your main contribution is the use of deep learning. Just the use of deep learning in some NLP task is not a contribution.  section 2: You claim that neural machine translation (NMT), mentioned as  neural approximations ,  does not achieve state-of-the-art results for ZhtoEs. I recommend to remove this claim from the paper, or to discuss it more, since Junczys-Dowmunt et al. (2016), during the last IWSLT, presented some results for ZhtoEs with the UN corpus, showing that NMT outperforms SMT by around 10 BLEU points.  section 5.1: You wrote that using the ZhtoEs language pair is one of your main contributions. Just using a language pair is not a contribution. Nonetheless, I think it is nice to see a paper on machine translation that does not focus of improving machine translation for English. The numbers provided in Table 2 were computed before or after preprocessing? Why did you remove the sentences longer than 50 tokens? Precise how did you obtain development and test sets, or provide them. Your experiments are currently no replicable especially because of that.  section 5.2: You wrote that you used Moses and its default parameters, but the default parameters of Moses are not the same depending on the version, so you should provide the number of the version used.  section 5.3: What do you mean by  hardware cost ? Table 3: more details should be provided regarding how did you obtain these values. You chose these values given the classifier accuracy, but how precisely and on what data did you train and test the classifiers? On the same data used in section 6? If I understood the experiments properly, you used simplified Spanish. But I cannot find in the text how do you simplify Spanish. And how do you use it to train the classifier and the SMT system?   section 6: Your method is better than other classification algorithms, but it says nothing about how it performs compared to the state-of-the-art methods. You should at least precise why you chose these classifications algorithms for comparison. Furthermore, how your rules impact these results? And more generally, how do you explain such a high accuracy for you method? Did you implement all these classification algorithms by yourselves? If not, you must provide the URL or cite the framework you used. For the SMT experiments, I guess you trained your phrase table on simplified Spanish. You must precise it. You chose METEOR over other metrics like BLEU to evaluate your results. You must provide some explanation for this choice. I particularly appreciate when I see a MT paper that does not use BLEU for evaluation, but if you use METEOR, you must mention which version you used. METEOR has largely changed since 2005. You cited the paper of 2005, did you use the 2005 version? Or did you use the last one with paraphrases?  Are your METEOR scores statistically significant?  section 7: As future work you mentioned  further simplify morphology . In this paper, you do not present any simplification of morphology, so I think that choosing the word  further  is misleading.  some typos: femenine ensambling cuadratic  style: plain text citations should be rewritten like this:  (Toutanova et al, 2008) built u00a0  should be  Toutanova et al. (2008) built   place the caption of your tables below the table and not above, and with more space between the table and its caption. You used the ACL 2016 template. You must use the new one prepared for ACL 2017. More generally, I suggest that you read again the FAQ and the submission instructions provided on the ACL 2017 website. It will greatly help you to improve the paper. There are also important information regarding references: you must provide DOI or URL of all ACL papers in your references.  -----------------------  After authors response:  Thank you for your response.  You wrote that rules are added just as post-processing, but does it mean that you do not apply them to compute your classification results? Or if you do apply them before computing these results, I'm still wondering about their impact on these results.  You wrote that Spanish is simplified as shown in Table 1, but it does not answer my question: how did you obtain these simplifications exactly? (rules? software? etc.) The reader need to now that to reproduce your approach.  The classification algorithms presented in Table 5 are not state-of-the-art, or if they are you need to cite some paper. Furthermore, this table only tells that deep learning gives the best results for classification, but it does not tell at all if your approach is better than state-of-the-art approach for machine translation. You need to compare your approach with other state-of-the-art morphology generation approaches (described in related work) designed for machine translation. If you do that your paper will be much more convincing in my opinion.