Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation
Sarcasm is a form of speech in which speakers say the opposite of what they truly mean in order to convey a strong sentiment. In other words, ”Sarcasm is the giant chasm between what I say, and the person who doesn’t get it.”. In this paper we present the novel task of sarcasm interpretation, defined as the generation of a non-sarcastic utterance conveying the same message as the original sarcastic one. We introduce a novel dataset of 3000 sarcastic tweets, each interpreted by five human judges. Addressing the task as monolingual machine translation (MT), we experiment with MT algorithms and evaluation measures. We then present SIGN: an MT based sarcasm interpretation algorithm that targets sentiment words, a defining element of textual sarcasm. We show that while the scores of n-gram based automatic measures are similar for all interpretation models, SIGN’s interpretations are scored higher by humans for adequacy and sentiment polarity. We conclude with a discussion on future research directions for our new task.1
1 Introduction
Sarcasm is a sophisticated form of communication in which speakers convey their message in an indirect way. It is defined in the MerriamWebster dictionary (Merriam-Webster, 1983) as the use of words that mean the opposite of what one would really want to say in order to insult someone, to show irritation, or to be funny. Considering this definition, it is not surprising to find frequent use of sarcastic language in opinionated
1Our dataset and code will be made publicly available.
user generated content, in environments such as Twitter, Facebook, Reddit and many more.
In textual communication, knowledge about the speaker’s intent is necessary in order to fully understand and interpret sarcasm. Consider, for example, the sentence ”what a wonderful day”. A literal analysis of this sentence demonstrates a positive experience, due to the use of the word wonderful. However, if we knew that the sentence was meant sarcastically, wonderful would turn into a word of a strong negative sentiment. In spoken language, sarcastic utterances are often accompanied by a certain tone of voice which points out the intent of the speaker, whereas in textual communication, sarcasm is inherently ambiguous, and its identification and interpretation may be challenging even for humans.
In this paper we present the novel task of interpretation of sarcastic utterances. We define the purpose of the interpretation task as the capability to generate a non-sarcastic utterance that captures the meaning behind the original sarcastic text.
Our work currently targets the Twitter domain since it is a medium in which sarcasm is prevalent, and it allows us to focus on the interpretation of tweets marked with the content tag #sarcasm. And so, for example, given the tweet ”how I love Mondays. #sarcasm” we would like our system to generate interpretations such as ”how I hate Mondays” or ”I really hate Mondays”. In order to learn such interpretations, we constructed a parallel corpus of 3000 sarcastic tweets, each of which has five non-sarcastic interpretations (Section 3).
Our task is complex since sarcasm can be expressed in many forms, it is ambiguous in nature and its understanding may require world knowledge. Following are several examples taken from our corpus:
1. loving life so much right now. #sarcasm 2. Way to go California! #sarcasm
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
3. Great, a choice between two excellent candidates, Donald Trump or Hillary Clinton. #sarcasm
In example (1) it is quite straightforward to see the exaggerated positive sentiment used in order to convey strong negative feelings. Examples (2) and (3), however, do not contain any excessive sentiment. Instead, previous knowledge is required if one wishes to fully understand and interpret what went wrong with California, or who Hillary Clinton and Donald Trump are.
Since sarcasm is a refined and indirect form of speech, its interpretation may be challenging for certain populations. For example, studies show that children with deafness, autism or Asperger’s Syndrome struggle with non literal communication such as sarcastic language (Peterson et al., 2012; Kimhi, 2014). Moreover, since sarcasm transforms the polarity of an apparently positive or negative expression into its opposite, it poses a challenge for automatic systems for opinion mining, sentiment analysis and extractive summarization (Popescu et al., 2005; Pang and Lee, 2008; Wiebe et al., 2004). Extracting the honest meaning behind the sarcasm may alleviate such issues.
In order to design an automatic sarcasm interpretation system, we first rely on previous work in established similar tasks (section 2), particularly machine translation (MT), borrowing algorithms as well as evaluation measures. In section 4 we discuss the automatic evaluation measures we apply in our work and present human based measures for: (a) the fluency of a generated nonsarcastic utterance, (b) its adequacy as interpretation of the original sarcastic tweet’s meaning, and (c) whether or not it captures the sentiment of the original tweet. Then, in section 5, we explore the performance of prominent phrase-based and neural MT systems on our task in development data experiments. We next present the Sarcasm SIGN (Sarcasm Sentimental Interpretation GeNerator, section 6), our novel MT based algorithm which puts a special emphasis on sentiment words. Lastly, in Section 7 we assess the performance of the various algorithms and show that while they perform similarly in terms of automatic MT evaluation, SIGN is superior according to the human measures. We conclude with a discussion on future research directions for our task, regarding both algorithms and evaluation.
2 Related Work
The use of irony and sarcasm has been well studied in the linguistics (Muecke, 1982; Stingfellow, 1994; Gibbs and Colston, 2007) and the psychology (Shamay-Tsoory et al., 2005; Peterson et al., 2012) literature. In computational work, the interest in sarcasm has dramatically increased over the past few years. This is probably due to factors such as the rapid growth in user generated content on the web, in which sarcasm is used excessively (Maynard et al., 2012; Kaplan and Haenlein, 2011; Bamman and Smith, 2015; Wang, 2013) and the challenge that sarcasm poses for opinion mining and sentiment analysis systems (Pang and Lee, 2008; Maynard and Greenwood, 2014). Despite this rising interest, and despite many works that deal with sarcasm identification (Tsur et al., 2010; Davidov et al., 2010; González-Ibánez et al., 2011; Riloff et al., 2013; Barbieri et al., 2014), to the best of our knowledge, generation of sarcasm interpretations has not been previously attempted.
Therefore, the following sections are dedicated to previous work from neighboring NLP fields which are relevant to our work: sarcasm detection, MT, paraphrasing and text summarization.
Sarcasm Detection Recent computational work on sarcasm revolves mainly around detection. Due to the large volume of detection work, we survey only several representative examples.
Tsur et al. (2010) and Davidov et al. (2010) presented a semi-supervised approach for detecting irony and sarcasm in product-reviews and tweets, where features are based on ironic speech patterns extracted from a labeled dataset. González-Ibánez et al. (2011) used lexical and pragmatic features, e.g. emojis and whether the utterance is a comment to another person, in order to train a classifier that distinguishes sarcastic utterances from tweets of positive and negative sentiment.
Riloff et al. (2013) observed that a certain type of sarcasm is characterized by a contrast between a positive sentiment and a negative situation. Consequently, they described a bootstrapping algorithm that learns distinctive phrases connected to negative situations along with a positive sentiment and used these phrases to train their classifier. Barbieri et al. (2014) avoided using word patterns and instead employed features such as the length and sentiment of the tweet, and the use of rare words.
Despite the differences between detection and
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
interpretation, this line of work is highly relevant to ours in terms of feature design. Moreover, it presents fundamental notions, such as the sentiment polarity of the sarcastic utterance and of its interpretation, that we adopt. Finally, when utterances are not marked for sarcasm as in the Twitter domain, or when these labels are not reliable, detection is a necessary step before interpretation.
Machine Translation We approach our task as one of monolingual MT, where we translate sarcastic English into non-sarcastic English. Therefore, our starting point is the application of MT techniques and evaluation measures. The three major approaches to MT are phrase based (Koehn et al., 2007), syntax based (Koehn et al., 2003) and the recent neural approach. For automatic MT evaluation, often an n-gram co-occurrence based scoring is performed in order to measure the lexical closeness between a candidate and a reference translations. Example measures are NIST (Doddington, 2002), METEOR (Denkowski and Lavie, 2011), and the widely used BLEU (Papineni et al., 2002), which represents precision: the fraction of n-grams from the machine generated translation that also appear in the human reference.
Here we employ the phrase based Moses system (Koehn et al., 2007) and an RNN-encoder-decoder architecture, based on Cho et al. (2014). Later we will show that these algorithms can be further improved and will explore the quality of the MT evaluation measures in the context of our task.
Paraphrasing and Summarization Tasks such as paraphrasing and summarization are often addressed as monolingual MT, and so they are close in nature to our task. Quirk et al. (2004) proposed a model of paraphrasing based on monolingual MT, and utilized alignment models used in the Moses translation system (Koehn et al., 2007; Wubben et al., 2010; Bannard and Callison-Burch, 2005). Xu et al. (2015) presented the task of paraphrase generation while targeting a particular writing style, specifically paraphrasing modern English into Shakespearean English, and approached it with phrase based MT.
Work on paraphrasing and summarization is often evaluated using MT evaluation measures such as BLEU. As BLEU is precision-oriented, complementary recall-oriented measures are often used as well. A prominent example is ROUGE (Lin, 2004), a family of measures used mostly for
evaluation in automatic summarization: candidate summaries are scored according to the fraction of n-grams from the human references they contain.
We also utilize PINC (Chen and Dolan, 2011), a measure which rewards paraphrases for being different from their source, by introducing new n-grams. PINC is often combined with BLEU due to their complementary nature: while PINC rewards n-gram novelty, BLEU rewards similarity to the reference. The highest correlation with human judgments is achieved by the product of PINC with a sigmoid function of BLEU (Chen and Dolan, 2011).
3 A Parallel Sarcastic Tweets Corpus
To properly investigate our task, we collected a dataset, first of its kind, of sarcastic tweets and their non-sarcastic (honest) interpretations. This data, as well as the instructions provided for our human judges, will be made publicly available and will hopefully provide a basis for future work regarding sarcasm on Twitter. Despite the focus of the current work on the Twitter domain, we consider our task as a more general one, and hope that our discussion, observations and algorithms will be beneficial for other domains as well.
Using the Twitter API2, we collected tweets marked with the content tag #sarcasm, posted between Januray and June of 2016. Following Tsur et al. (2010), González-Ibánez et al. (2011) and Bamman and Smith (2015), we address the problem of noisy tweets with automatic filtering: we remove all tweets not written in English, discard retweets (tweets that have been forwarded or shared) and remove tweets containing URLs or images, so that the sarcasm in the tweet regards to the text only and not to an image or a link. This results in 3000 sarcastic tweets containing text only.
In order to obtain honest interpretations for our sarcastic tweets, we used Fiverr3 – a platform for selling and purchasing services from independent suppliers (also referred to as workers). We employed ten Fiverr workers, half of them from the field of comedy writing, and half from the field of literature paraphrasing. The chosen workers were made sure to have an active Twitter account, in order to ensure their acquaintance with social networks and with Twitter’s colorful language (hashtags, common acronyms such as LOL, etc.).
2http://apiwiki.twitter.com 3https://www.fiverr.com
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Sarcastic Tweets Honest Interpretations
What a great way to end my night. #sarcasm 1. Such a bad ending to my night 2. Oh what a great way to ruin my night 3. What a horrible way to end a night 4. Not a good way to end the night 5. Well that wasn’t the night I was hoping for
Staying up till 2:30 was a brilliant idea, very productive #sarcasm 1. Bad idea staying up late, not very productive 2. It was not smart or productive for me to stay up so late 3. Staying up till 2:30 was not a brilliant idea, very non-productive 4. I need to go to bed on time 5. Staying up till 2:30 was completely useless
Table 1: Examples from our parallel sarcastic tweet corpus.
We then randomly divided our tweet corpus to two batches of size 1500 each, and randomly assigned five workers to each batch. We instructed the workers to translate each sarcastic tweet into a non sarcastic utterance, while maintaining the original meaning. We encouraged the workers to use external knowledge sources (such as Google) if they came across a subject they were not familiar with, or if the sarcasm was unclear to them.
Table 1 presents two examples from our corpus. The table demonstrates the tendency of the workers to generally agree on the core meaning of the sarcastic tweets. Yet, since sarcasm is inherently vague, it is not surprising that the interpretations differ from one worker to another. For example, some workers change only one or two words from the original sarcastic tweet, while others rephrase the entire utterance. We regard this as beneficial, since it brings a natural, human variance into the task. This variance makes the evaluation of automatic sarcasm interpretation algorithms challenging, as we further discuss in the next section.
4 Evaluation Measures
As mentioned above, in certain cases world knowledge is mandatory in order to correctly evaluate sarcasm interpretations. For example, in the case of the second sarcastic tweet in table 1, we need to know that 2:30 is considered a late hour so that staying up till 2:30 and staying up late would be considered equivalent despite the lexical difference. Furthermore, we notice that transforming a sarcastic utterance into a non sarcastic one often requires to change a small number of words. For example, a single word change in the sarcastic tweet ”How I love Mondays. #sarcasm” leads to the non-sarcastic utterance How I hate Mondays.
This is not typical for MT, where usually the entire source sentence is translated to a new sentence
in the target language and we would expect lexical similarity between the machine generated translation and the human reference it is compared to. This raises a doubt as to whether n-gram based MT evaluation measures such as the aforementioned are suitable for our task. We hence asses the quality of an interpretation using automatic evaluation measures from the tasks of MT, paraphrasing, and summarization (Section 2), and compare these measures to human-based measures. Automatic Measures We use BLEU and ROUGE as measures of n-gram precision and recall, respectively. We report scores of ROUGE-1, ROUGE-2 and ROUGE-L (recall based on unigrams, bigrams and longest common subsequence between candidate and reference, respectively). In order to asses the n-gram novelty of interpretations (i.e, difference from the source), we report PINC and PINC∗sigmoid(BLEU) (see Section 2). Human judgments We employed an additional group of five Fiverr workers and asked them to score each generated interpretations with two scores on a 1-7 scale, 7 being the best. The scores are: adequacy: the degree to which the interpretation captures the meaning of the original tweet; and fluency: how readable the interpretation is. In addition, reasoning that a high quality interpretation is one that captures the true intent of the sarcastic utterance by using words suitable to its sentiment, we ask the workers to assign the interpretation with a binary score indicating whether the sentiment presented in the interpretation agrees with the sentiment of the original sarcastic tweet.4
The human measures enjoy high agreement levels between the human judges. The averaged root mean squared error calculated on the test set
4For example, we consider ”Best day ever #sarcasm” and its interpretation ”Worst day ever” to agree on the sentiment, despite the use of opposite sentiment words.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Sarcastic Tweet Moses Interpretation Neural Interpretation Boy , am I glad the rain’s here #sarcasm Boy, I’m so annoyed that the rain is here I’m not glad to go today Another night of work, Oh, the joy #sarcasm Another night of work, Ugh, unbearable Another night, I don’t like it Being stuck in an airport is fun #sarcasm Be stuck in an airport is not fun Yay, stuck at the office again You’re the best. #sarcasm You’re the best You’re my best friend
Table 2: Sarcasm interpretations generated by Moses and by the RNN.
Evaluation Measure Moses RNN Precision Oriented BLEU 62.91 41.05 Novelty Oriented PINC 51.81 76.45 PINC∗sigmoid(BLEU) 33.79 45.96
Recall Oriented
ROUGE-1 66.44 42.20 ROUGE-2 41.03 29.97 ROUGE-l 65.31 40.87
Human Judgments Fluency 6.46 5.12 Adequacy 2.54 2.08 % correct sentiment 28.84 17.93
Table 3: Development data results for MT models.
across all pairs of judges and across the various algorithms we experiment with are: 1.44 for fluency and 1.15 for adequacy. For sentiment scores the averaged agreement at the same setup is 93.2%.
5 Sarcasm Interpretations as MT
As our task is about the generation of one English sentence given another, a natural starting point is treating it as monolingual MT. We hence begin with utilizing two widely used MT systems, representing two different approaches: Phrase Based MT vs. Neural MT. We then analyze the performance of these two systems, and based on our conclusions we design our SIGN model.
Phrase Based MT We employ Moses5, using word alignments extracted by GIZA++ (Och and Ney, 2003) and symmetrized with the grow-diagfinal strategy. We use phrases of up to 8 words to build our phrase table, and do not filter sentences according to length since tweets contain at most 140 characters. We employ the KenLM algorithm (Heafield, 2011) for language modeling, and train it on the non-sarcastic tweet interpretations (the target side of the parallel corpus).
Neural Machine Translation We use GroundHog, a publicly available implementation of an RNN encoder-decoder, with LSTM hidden states.6 Our encoder and decoder contain 250 hidden units each. We use the minibatch stochastic gradient
5http://www.statmt.org/moses 6https://github.com/lisa-groundhog/GroundHog
descent (SGD) algorithm together with Adadelta (Zeiler, 2012) to train each model, where each SGD update is computed using a minibatch of 16 utterances. Following Sutskever et al. (2014), we use beam search for test time decoding. Henceforth we refer to this system as RNN. Performance Analysis We divide our corpus into training, development and test sets of sizes 2400, 300 and 300 respectively. We train Moses and the RNN on the training set and tune their parameters on the development set. Table 3 presents development data results, as these are preliminary experiments that aim to asses the compatibility of MT algorithms to our task.
Moses scores much higher in terms of BLEU and ROUGE, meaning that compared to the RNN its interpretations capture more n-grams appearing in the human references while maintaining high precision. The RNN outscores Moses in terms of PINC and PINC∗sigmoid(BLEU), meaning that its interpretations are more novel, in terms of ngrams. This alone might not be a negative trait; However, according to human judgments Moses performs better in terms of fluency, adequacy and sentiment, and so the novelty of the RNN’s interpretations does not necessarily contribute to their quality, and even possibly reduces it.
Table 2 illustrates several examples of the interpretations generated by both Moses and the RNN. While the interpretations generated by the RNN are readable, they generally do not maintain the meaning of the original tweet. We believe that this is the result of the neural network overfitting the training set, despite regularization and dropout layers, probably due to the relatively small training set size. In light of these results when we experiment with the SIGN algorithm (Section 7), we employ Moses as its MT component.
The final example of Table 2 is representative of cases where both Moses and the RNN fail to capture the sarcastic sense of the tweet, incorrectly interpreting it or leaving it unchanged. In order to deal with such cases, we wish to utilize a property typical of sarcastic language. Sarcasm is mostly
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
“How I love Mondays # sarcasm
“How I cluster-i Mondays # sarcasm MOSES
love like ...
cluster-i
“How I hate Mondays
“How I cluster-j Mondays # sarcasm
hate despise ...
cluster-j de-clusteringclustering
Positive Clusters merit, wonder, props, praise, congratulations..
patience, dignity, truth, chivalry, rationality...
Negative Clusters hideous, horrible, nasty, obnoxious, scary, pathetic... shame, sadness, sorrow, fear, disappointment, regret, danger...
Table 4: Examples of two positive and two negative clusters created by the SIGN algorithm.
used to convey a certain emotion by using strong sentiment words that express the exact opposite of their literal meaning. Hence, many sarcastic utterances can be correctly interpreted by keeping most of their words, replacing only sentiment words with expressions of the opposite sentiment. For example, the sarcasm in the utterance ”You’re the best. #sarcasm” is hidden in best, a word of a strong positive sentiment. If we transform this word into a word of the opposite sentiment, such as worst, then we get a non-sarcastic utterance with the correct sentiment.
We next present the Sarcasm SIGN (Sarcasm Sentimental Interpretation GeNerator), an algorithm which capitalizes on sentiment words in order to produce accurate interpretations.
6 The Sarcasm SIGN Algorithm
SIGN (Figure 1) targets sentiment words in sarcastic utterances. First, it clusters sentiment words according to semantic relatedness. Then, each sentiment word is replaced with its cluster 7 and the transformed data is fed into an MT system (Moses in this work), at both its training and test phases. Consequently, at test time the MT system outputs non-sarcastic utterances with clusters replacing sentiment words. Finally, SIGN performs a declustering process on these MT outputs, replacing sentiment clusters with suitable words.
In order to detect the sentiment of words, we 7This means that we replace a word with cluster-j where j
is the number of the cluster to which the word belongs.
turn to SentiWordNet (Esuli and Sebastiani, 2006), a lexical resource based on WordNet (Miller et al., 1990; Kilgarriff and Fellbaum, 2000). Using SentiWordNet’s positivity and negativity scores, we collect from our training data a set of distinctly positive words (∼ 70) and a set of distinctly negative words (∼ 160).8 We then utilize the pre-trained dependency-based word embeddings of Levy and Goldberg (2014)9 and cluster each set using the k-means algorithm with L2 distance. We aim to have ten words on average in each cluster, and so the positive set is clustered into 7 clusters, and the negative set into 16 clusters. Table 4 presents examples from our clusters.
Upon receiving a sarcastic tweet, at both training and test, SIGN searches it for sentiment words according to the positive and negative sets. If such a word is found, it is replaced with its cluster. For example, given the sentence ”How I love Mondays. #sarcasm”, love will be recognized as a positive sentiment word, and the sarcastic tweet will become: ”How I cluster-i Mondays. #sarcasm” where i is the cluster number of the word love.
During training, this process is also applied to the non-sarcastic references. And so, if one such reference is ”I dislike Mondays.”, then dislike will be identified and the reference will become ”I cluster-j Mondays.”, where j is the cluster number of the word dislike. Moses is then trained on these new representations of the corpus, using the exact same setup as before. This training process produces a mapping between positive and negative clusters, and outputs sarcastic interpretations with clustered sentiment words (e.g, ”I cluster-j Mondays.”). At test time, after Moses generates an utterance containing clusters, a de-clustering pro-
8The scores are in the [0,1] range. We set the threshold of 0.6 for both distinctly positive and distinctly negative words.
9https://levyomer.wordpress.com/2014/ 04/25/dependency-based-word-embeddings/. We choose these embeddings since they are believed to better capture the relations between a word and its context, having been trained on dependency-parsed sentences.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Evaluation Measure Moses SIGN-centroid SIGN-context SIGN-oracle Precision Oriented BLEU 65.24 63.52 66.96 67.49 Novelty Oriented PINC 45.92 47.11 46.65 46.10 PINC∗sigmoid(BLEU) 30.21 30.79 31.13 30.54 Recall Oriented ROUGE-1 70.26 68.43 69.67 70.34 ROUGE-2 42.18 40.34 40.96 42.81 ROUGE-l 69.82 68.24 69.98 70.01
Table 5: Test data results with automatic evaluation measures.
cess takes place: the clusters are replaced with the appropriate sentiment words.
We experiment with several de-clustering approaches: (1) SIGN-centroid: the chosen sentiment word will be the one closest to the centroid of cluster j. For example in the tweet ”I cluster-j Mondays.”, the sentiment word closest to the centroid of cluster j will be chosen; (2) SIGNcontext: the cluster is replaced with its word that has the highest average Pointwise Mutual Information (PMI) with the words in a symmetric context window of size 3 around the cluster’s location in the output. For example, for ”I cluster-j Mondays.”, the sentiment word from cluster j which has the highest average PMI with the words in {’I’,’Mondays’} will be chosen. The PMI values are computed on the training data; and (3) SIGNOracle: an upper bound where a person manually chooses the most suitable word from the cluster.
We expect this process to improve the quality of sarcasm interpretations in two aspects. First, as mentioned earlier, sarcastic tweets often differ from their non sarcastic interpretations in as little as one sentiment word. SIGN should help highlight the sentiment words most in need of interpretation. Second, under the pre-processing SIGN performs to the input examples of Moses, the latter is inclined to learn a mapping from positive to negative clusters, and vice versa. This is likely to encourage the Moses output to generate outputs of the same sentiment as the original sarcastic tweet, but with honest sentiment words. For example, if the sarcastic tweet expresses a negative sentiment with strong positive words, the non-sarcastic interpretation will express this negative sentiment with negative words, thus stripping away the sarcasm.
7 Experiments and Results
We experiment with SIGN and the Moses and RNN baselines at the same setup of section 5. We report test set results for automatic and human measures, in Tables 5 and 6 respectively. As in
Fluency Adequacy % correct sentiment % changed
Moses 6.67 2.55 25.7 42.3 SIGN-Centroid 6.38 3.23* 42.2* 67.4 SIGN-Context 6.66 3.61* 46.2* 68.5 SIGN-Oracle 6.69 3.67* 46.8* 68.8
Table 6: Test set results with human measures. %changed provides the fraction of tweets that were changed during interpretation (i.e. the tweet and its interpretation are not identical). In cases where one of our models presents significant improvement over Moses, the results are decorated with a star. Statistical significance is tested with the paired t-test for fluency and adequacy, and with the McNemar paired test for labeling disagreements (Gillick and Cox, 1989) for % correct sentiment, in both cases with p < 0.05.
the development data experiments (Table 3), the RNN presents critically low adequacy scores of 2.11 across the entire test set and of 1.89 in cases where the interpretation and the tweet differ. This, along with its low fluency scores (5.74 and 5.43 respectively) and its very low BLEU and ROUGE scores make us deem this model immature for our task and dataset, hence we exclude it from this section’s tables and do not discuss it further.
In terms of automatic evaluation (Table 5), SIGN and Moses do not perform significantly different. When it comes to human evaluation (Table 6) however, SIGN-context presents substantial gains. While for fluency Moses and SIGN-context perform similarly, SIGN-context performs much better in terms of adequacy and the percentage of tweets with the correct sentiment. The differences are substantial as well as statistically significant: adequacy of 3.61 for SIGN-context compared to 2.55 of Moses, and correct sentiment for 46.2% of the SIGN interpretations, compared to only 25.7% of the Moses interpretations.
Table 6 further provides an initial explanation to the improvement of SIGN over Moses: Moses tends to keep interpretations identical to the origi-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
nal sarcastic tweet, altering them in only 42.3% of the cases, 10 while SIGN-context’s interpretations differ from the original sarcastic tweet in 68.5% of the cases, which comes closer to the 73.8% in the gold standard human interpretations. If for each of the algorithms we only regard to interpretations that differ from the original sarcastic tweet, the differences between the models are less substantial. Nonetheless, SIGN-context still presents improvement by correctly changing sentiment in 67.5% of the cases compared to 60.8% for Moses.
Both tables consistently show that the contextbased selection strategy of SIGN outperforms the centroid alternative. This makes sense as, being context-ignorant, SIGN-centroid might produce non-fluent or inadequate interpretations for a given context. For example, the tweet ”Also gotta move a piano as well. joy #sarcasm” is changed to ”Also gotta move a piano as well. bummer” by SIGN-context, while SIGN-centroid changes it to the less appropriate ”Also gotta move a piano as well. boring”. Nonetheless, even this naive de-clustering approach substantially improves adequacy and sentiment accuracy over Moses.
Finally, comparison to SIGN-oracle reveals that the context selection strategy is not far from human performance with respect to both automatic and human evaluation measures. Still, some gain can be achieved, especially for the human measures on tweets that were changed at interpretation. This indicates that SIGN can improve mostly through a better clustering of sentiment words, rather than through a better selection strategy.
8 Discussion and Future Work
The performance gap between Moses and SIGN may stem from the difference in their optimization criteria. Moses aims to optimize the BLEU score and given the overall lexical similarity between the original tweets and their interpretations, it therefore tends to keep them identical. SIGN, in contrast, targets sentiment words and changes them frequently. Consequently, we do not observe substantial differences between the algorithms in the automatic measures that are mostly based on n-gram differences between the source and the interpretation. Likewise, the human fluency measure that accounts for the readability of the interpretation is not seriously affected by the translation process. When it comes to the human adequacy and
10We elaborate on this in section 8.
sentiment measures, which account for the understanding of the tweet’s meaning, SIGN reveals its power and demonstrates much better performance compared to Moses.
To further understand the relationship between the automatic and the human based measures we computed the Pearson correlations for each pair of (automatic, human) measures. We observe that all correlation values are low (up to 0.12 for fluency, 0.13-0.18 for sentiment and 0.19-0.24 for adequacy). Moreover, for fluency the correlation values are insignificant (using a correlation significance t-test with p = 0.05). We believe this indicates that these automatic measures do not provide appropriate evaluation for our task. Designing automatic measures is hence left for future research.
A qualitative analysis reveals that as expected, SIGN-context performs well on sarcastic tweets with clear sentiment words, transforming expressions such as ”Audits are a blast to do #sarcasm” and ”Being stuck in an airport is fun #sarcasm” into ”Audits are a bummer to do” and ”Being stuck in an airport is boring”, respectively. Even when there are several sentiment words and not all of them require a change, e.g. in ”Constantly being irritated, anxious and depressed is a great feeling! #sarcasm”, SIGN-context produces the adequate interpretation: ”Constantly being irritated, anxious and depressed is a terrible feeling”.
In some cases, however, SIGN struggles with producing correct interpretations. For example, the tweet ”Can you imagine if Lebron had help? #sarcasm”, was left unchanged by all SIGN models (including the upper bound oracle). Notice that the sarcasm in this tweet is not expressed by specific sentiment words. Moreover, world knowledge (who Lebron is, what kind of help he requires) is crucial in order to comprehend and interpret this sarcastic utterance. In addition, phrases that express sentiment without explicit sentiment words, for example ”can’t wait” in the tweet ”Can’t wait until tomorrow #sarcasm”, diminish SIGN’s advantage. Further Improving SIGN so that it properly interprets such tweets is a major future direction.

This paper focuses on interpreting sarcasm written in Twitter identifying sentiment words and then using a machine translation engine to find an equivalent not sarcastic tweet.   EDIT: Thank you for your answers, I appreaciate it. I added one line commenting about it.  - Strengths:  Among the positive aspects of your work, I would like to mention the parallel corpus you presented. I think it will be very useful for other researchers in the area for identifying and interpreting sarcasm in social media. An important contribution is also the attempt to evaluate the parallel corpora using existing measures such as the ones used in MT tasks. But also because you used human judgements to evaluate the corpora in 3 aspects: fluency, adequacy and equivalent sentiment.  - Room for improvement:  Tackling the problem of interpretation as a monolingual machine translations task is interesting, while I do appreciate the intent to compare the MT with two architectures, I think that due the relatively small dataset (needed for RNN) used it was predictable that the  Neural interpretation d is performing worse than  moses interpretation d. You came to the same conclusion after seeing the results in Table3. In addition to comparing with this architecture, I would've liked to see other configuration of the MT used with moses. Or at least, you should provide some explanation of why you use the configuration described in lines 433 through 442; to me this choice is not justified.    - thank you for your response, I understand it is difficult to write down all the details but I hope you include a line with some of your answer in the paper, I believe this could add valuable information.  When you presented SING, it is clear that you evaluate some of its components beforehand, i.e. the MT. But other important components are not evaluated, particularly, the clustering you used of positive and negative words. While you did said you used k-means as a clustering algorithm it is not clear to me why you wanted to create clusters with 10 words. Why not test with other number of k, instead of 7 and 16, for positive and negative words respectively. Also you could try another algorithm beside kmeans, for instance, the star clustering algorithm (Aslam et al. 2004), that do not require a k parameter.     - thanks for clarifying.  You say that SIGN searches the tweet for sentiment words if it found one it changes it for the cluster ID that contain that word. I am assuming that there is not a limit for the number of sentiment words found, and the MT decides by itself how many sentiment words to change. For example, for the tweet provided in Section 9:  Constantly being irritated, anxious and depressed is a great feeling d the clustering stage of SIGN should do something like  Constantly being cluster-i, cluster-j and cluster-k is a cluster-h feeling d, Is that correct? If not, please explain what SIGN do.     - Thanks for clarifying  - Minor comments:  In line 704, section 7, you said:  SIGN-context  interpretations differ from the original sarcastic tweet in 68.5% of the cases, which come closer to the 73.8% in the gold standard human interpretations. d This means that 25% of the human interpretations are the same as the original tweet? Do you have any idea why is that?  In section 6, line 539 you could eliminate the footnote 7 by adding  its cluster ID d or  its cluster number d.  References: Aslam, Javed A., Pelekhov, Ekaterina, and Rus, Daniela.  The star clustering algorithm for static and dynamic information organization..  Journal of Graph Algorithms and Applications 8.1 (2004): 95-129. <http:  eudml.org doc 51529>.


We have developed an automated Japanese short-answer scoring and support machine for new National Center written test exams. Our approach is based on the fact that recognizing textual entailment and/or synonymy has been almost impossible for several years. The system generates automated scores on the basis of evaluation criteria or rubrics, and human raters revise them. The system determines semantic similarity between the model answers and the actual written answers as well as a certain degree of semantic identity and implication. Owing to the need for the scoring results to be classified at multiple levels, we use random forests to utilize many predictors effectively rather than use support vector machines. An experimental prototype operates as a web system on a Linux computer. We compared human scores with the automated scores for a case in which 3–6 allotment points were placed in 8 categories of a social studies test as a trial examination. The differences between the scores were within one point for 70–90 percent of the data when high semantic judgment was not needed.
1 Introduction
Educational advisory body to the Japanese government has decided that writing tests will be introduced into the new national center test for university entrance examinations, as announced in a final report (MEXT, 2016) at the high school and university articulation meeting by the Ministry of Education, Culture, Sports, Science and Technology. The use of AI-based computers was proposed to stabilize the test scores efficiently. The required
type of writing test is a short-answer test, where a correct answer is expected to exist. Therefore, the test is scored by judging agreement of the meaning with the correct answer. Another type of writing test is essay writing, where a correct answer does not exist. The written answers are evaluated based on the rhetoric, the connection expressions, and the content.
Because short-answer scoring involves technical difficulty, the number of characters is restricted to 80 characters at most from dozens of characters. Two characters in Japanese are generally equivalent to one word in English. A short-answer test is widely considered to be more authentic and reliable for measuring ability compared with a multiple-choice test. If technical problems related to the short-answer test are solved, the potential demand for its use, as well as that for the national center test, will be enormous. Many systems for evaluating essays have been developed and offered in the United States (Shermis and Burstein, 2013). The authors’ group also developed the first and most well-known Japanese automated essay scoring system named Jess (Ishioka and Kameda, 2006), and it is in practical use now.
While a short-answer scoring system has been developed because of its importance, various technical problems remain unsolved. New York University (NYU) and the Educational Testing Service (ETS) developed the first automated scoring tools in this field; they evaluated the NYU online program (Vigilante, 1999). Leacock and Chodorow (2003) reported the latest specifications of the c-rater developed by ETS. Pulman and Sukkarieh (2005) tried to generate several sentences having the same meaning as the correct answer sentence using the natural language technique of information extraction. However, the concordance rate with human examiners was found to be small and impractical.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
The key technology to solve the short answer scoring seems to be recognizing textual entailment. The National Institute of Informatics (NII) in Japan has promoted the “Todai robot project” to solve multiple choice tests of the Japanese national center test for university entrance examinations (NCTUEE) using this technology with knowledge resources from many textbooks and Wikipedia. They found that perfect recognition of textural entailment or correct understanding of the meaning is difficult at present (NII, 2013). To combine several methods and patch them ad hoc is the most that can be done if people feigned to answer. We have also tried to acquire the basic technology and to produce an automated written short-answer system based on our scientific research conducted in 2014–2016, but we reached the limit of the system.
Therefore, we thought of a support system for short written tests where a human rater can correct the automated score by referring to the original scores. When the human rater agrees with the result of the automated score, he/she can just approve the score indicated by default and can produce the corresponding mark. We chose to leave room for human raters to overwrite it without making it a perfect automated scoring system.
The new examination, which was created by the NCTUEE, will utilize the written test for Japanese literature, whose scoring seems to be more difficult than the examinations of science and social studies, which are prepared basically using the facts written in their respective textbooks. The written test for Japanese literature needs reading skills rather than skills of information processing and pattern matching. In this test in particular, we have to detect the semantic difference in the compared written answers with the model sentences, though almost no difference exists in the vocabulary used. Therefore, we tried to tackle the scoring of short sentences in social studies, where precise judgments are less needed.
In section 2, we indicate the test items and the model answers used in a trial examination for university entrance examinations. In section 3, we show the specifications of our proposed system. In section 4, we present our evaluation of the performance on eight tests of social studies. Section 5 concludes with a summary.
2 Test items used in a trial examination
We assigned a theme in three subjects of world history, Japanese history, and geography of the “Gakken nation-wide trial examination” in fiscal year 2015. The world history test set includes four written test items and two test items each for geography and Japanese history; the total is 8 test items.
Table 1 shows the “content” asked and the “correct answer,” which are given to test examinees in a distributed booklet of “test answers and explanations.”
Herein, I comment a little about the correct answer of World history B2 #3. “Jizya” was the capitation imposed on non-Muslims as compensation required for their beliefs in the Islamic world; when indigenous people in a place of conquest converted to Islam, they were not taxed originally. However, “kharaj” was a land tax, and it was imposed on those who possessed land, regardless of whether or not they were Muslim.
However, even if indigenous people converted to Islam in the time of Umayyad, “Jizya” was still imposed, and even if land was possessed, they were exempted from “kharaj.” The Abbasid dynasty returned it to the original state (Table 2). This test item asks about the content, and the recognizing correct meaning is necessary for getting scores.
Table 2: Tax system changes from Umayyad to Abbasid dynasty
Muslims non-Muslims Jizya tax → tax exempt tax kharaj tax tax exempt → tax
3 Specifications of the scoring support system
3.1 Outline
Our system is for automated scoring and for supporting human raters. The approach functions as follows.
1. A system automatically judges each answer posed on whether or not its prepared key phrases agree with those of the model answer using the “scoring criteria” from a surfacelike point of view.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Table 1: The content and the correct answer examples of written test items
Subject Test Item #
(Allotment) Content and the correct answer
World history B2
#1 (3 pt.) [content] Ancient Greece: Solon’s politics of assets [correct answer] A citizen’s right to vote was set according to oneEfs political classes, which depended on his or her ownership of properties. (17 characters in Japanese)
#3 (5 pt.) [content] Islam: Tax system in the Abbasid dynasty [correct answer] When indigenous people in a place of conquest were Muslim, they were exempted from “Jizya.” When Arabs had land in a place of conquest, “kharaj” was imposed. (60 chars.)
Japanese history B1
#2 (3 pt.) [content] Peace treaty of the Russo-Japanese War; Change in the territory [correct answer] The south in Sakhalin was ceded to Japan from Russia. (22 chars) #4 (5 pt.) [content] Dissolution of financial giants after World War II [correct answer] A holding company, a cartel, and a trust were prohibited by the Antimonopoly Act, and a huge monopoly was divided by the excessive economic power decentralization law. (59 chars)
Japanese history B2
#1 (3pt.) [content] The Emperor Genmei: Tax burden in the ordinance system [correct answer] A certain amount of cloth is offered instead of labor for capital[m9]. (20 chars) #3 (6 pt.) [content] Kamakura era second half: Commercial activities in the city [correct answer] The carrier, called a ”Toi,” and the usury person, called a ”Kariage,” appeared, and an exchange settled in the bill was started instead of sending money. (59 chars)
Geometry B #1 (3 pt.) [content] The genesis of the Namib desert [correct answer] The cold current that flows through an offing[m10] and a medium latitude high-pressure area (19 chars)
#4 (6 pt.) [content] The population of the world: A comparison in India and China [correct answer] The birthrate in India is higher than that of China, and the population growth is remarkable because India failed in its attempts at strict birth control, whereas China conducted a one-child policy. (59 chars)
c⃝Gakken Holdings
2. The system gives not only a temporary score based on the criterion-based judgment but also a prediction score offered by machine learning based on the understanding of other human raters or supervised data. A certain degree of semantic meaning is also used.
3. A human rater can certify the prediction score by which a system presents this information as reference. He or she can correct this and overwrite based on his/her judgment.
To reduce the time and effort, the system precision should possess a certain degree of fitness with human ratings; more than 80% of the precision is desirable for tentative targets.
The flowchart of our system is as shown in figure 1.
(a) Before scoring, we collected a lot of score data from various human raters and performed a machine learning of “Random Forests” (Breiman, 2001). The degree of fitness with the scoring guideline is also necessary. On the basis of these learning results, we set up a scoring engine to return the scores for new answers.
(b) The system generates a scoring screen written in the Hyper Text Markup Language.
(c) A user or human rater opens a scoring screen of (b) using a web browser on his/her terminal machine. Then, a CGI program is activated. The recommended value as a result of the scoring engine of (a) is indicated here. The scoring result is stocked in a file or a database. The user repeats this mark operation.
3.2 Scoring Screen
Figure 2 shows a screen shot of our prototype system. “The answer sentence that should be scored” (in red ink) is located in the upper part of the system; the middle part has some scoring criteria such as “synonyms and permitted different transcriptions,” “model or correct answers that warrant a full mark,” “partial phrases that warrant partial scores,” and “mandatory phrases.” For the “model answer” and “partial correct phrases,” the system judges the degree of fitness with the answer sentence to be scored; the system also judges whether or not the answer sentence includes “mandatory phrases,” whether or not it is meaningfully composed, and whether or not it exceeds the character limit; if the answer should be written as a noun or noun phrase, the system judges whether or not it matches the specified “type” format. These judg-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Written answers
(c)
Terminal
Scoring
Results Display the Score
RandomForests Machine Learning
Scoring Criteria
Scoring Screen （HTML）
Scoring Engine
Web
CGI
(a)
(b)
Figure 1: Flowchart of the system
ments are either yes or no, and toggle buttons are used. A human rater reviews these judgments and revises them if necessary.
Tentative scores located in the lower part are based on the aforementioned alternative judgment. The right-hand window is to determine the final score. The initial mark is settled by which predictive probability based on the past learned results gives the maximum. The probability values are also indicated.
When no learning data exist, that is to say, when no pre-scored data about the relevant test item exist, the message to that effect is shown in the top windows: no probability and no initial mark are naturally determined.
3.3 Difference between the tentative score and the mechanical prediction score
The example indicated by figure 2 is a case of World history B2 #3. The answer indicated here is as follows: Indigenous people in a place of conquest originally had both “Jizya” and “kharaj” imposed on them, but only “kharaj,” like it was with Arabs, later came to be imposed on them. Compared with the model answer of “Even when indigenous people in a place of conquest were Muslim, they were exempted from Jizya. When an Arab had land in a place of conquest, kharaj was imposed,” we found that the appearance of written words was similar, but the apparent meaning of the sentences was quite different. Therefore,
the system gives a score of 4 points of the tentative score (5 points of allotment) determined by buttons checked based on agreement of surfacelike words and phrases that appeared; but the mechanical prediction score is 0 points, and it takes account of other elements besides the surface-like side. The prediction probability of the recommended score is 0.79. It shows that the effect of the machine learning is functioning appropriately.
3.4 Automatic screen creation from a scoring criterion file
Our system is a Web application. Thus, the screen indicated by figure 2 is generated by HyperText Markup Language. We built the mechanism to make this HTML file automatically from a plain scoring criterion file that a computer beginner can handle.
Figure 3 is a plain original file that makes a screen like the one in figure 2. Two or three elements are set for criteria. In order, the label, allotment of points, and correspondence are located. The tab is the delimiter.
Synonyms and different transcriptions are recorded in “syno,” which appeared in “gold” as a model answer and in “part” as a partially correct phrase. “Syno” is not always limited to a definite lexical meaning. When it has semantically the same meaning, it is also permitted. “Part” includes two types; one is possible to add to a partial point, and the other is for which a maximum is taken. If there are multiple same labels (for example, part1), we use the maximum of the points; different labels (for example, part 1 and part 2) can add the allotted points. “Lack” is a mandatory phrase; if no phrases exist, the point is deducted. A comma can be used for the meaning of “both.” “Vol” shows the number of characters available. “None” shows a nonsense sentence, and “goji” shows a wrong word such as kanji that does not exist. Minus points indicate points to be deducted.
We use “fitness” as the degree of the relationship between the written answer and “model answer” designated in “gold” or “partial correct phrases” in “part.” We define this as the harmonic mean of two kinds of relationships: one is the degree of the reference during the sentence keywords from the viewpoint of a written answer; the other is that from a model answer. These relationships are just like precision and recall often used in in-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Figure 2: Short-answer scoring and support system screen (In case of world history B2 #3)
syno jizya jizyah syno Muslim Islam syno "Indigenous people" "Different ethnic groups" gold 5 "Even when indigenous people in a place of conquest were Muslim, they were exempted from Jizya. Even when an Arab had land in a place of conquest, kharaj was imposed." part1 2 "Even when indigenous people in a place of conquest were Muslim, they were exempted from Jizya." part2 2 "Even when an Arab had land in a place of conquest, kharaj was imposed." lack1 -1 "had land" lack2 -2 jizya lack2 -2 kharaj lack2 -5 jizya,kharaj vol -5 60-40 nons -5 goji -1
Figure 3: Scoring criterion file (labels, allotment of points, and correspondences are tab delimited.)
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
formation retrieval, e.g., a Google search. This harmonic mean or “fitness” is called an F-measure taking a float number from 0 to 1. Our system rounds this to either 0 or 1 as a toggle button occurrence, and it shows a non-rounded value as a reference for the user.
4 Performance Evaluation
4.1 Evaluation of the classification
We built a prediction model using machine learning of random forests (RFs) V4.1 (Liaw and Wiener, 2015; Breiman and Cutler, 2004). The predictors include not only the degree of fitness indicated in Figure 3 but also semantic (cosine) similarity between the answer, model answer, and test item sentences. The reasons for using the RFs in the methods of many machine learning techniques are as follows:
1. When using many predictor variables, the classification often functions effectively.
2. The degree of contributions can be estimated to determine effective predictor variables quantitatively in the classification.
3. RFs are suitable for this because test scoring requires multiple classifications with values of 0–3 or 0–6.
For eight test items indicated in table 1, we compared human ratings with the estimate based prediction model. The best rate that that a prediction and a correct answer were identical is 78% in the cases of Geometry B #4. The worst rate is 43% in the cases of Japanese History B1 #2, which was surprisingly low. This is because about 80% of examinees got zero scores; thus, machine learning does not work well. We omitted cross matrices between human ratings and the estimate because of space limitations.
Table 3 shows the probability in which the differences between the scores were within one point. These values consist of 71–95% removing the case of world history B2 #3, which is necessary for correct understanding of the meaning. It shows the performance of the classification was in the level available.
RFs do not need cross validation to calculate the error rate. That is, a separated test set is not necessary to determine it (Breiman, 2001). The error rate can be estimated internally during a run. In RFs, each tree is constructed using a different
Table 3: Probability in which the estimates differed from the human ratings within one point
Item # Prob. Item # Prob. World History B2 #1 0.75 Jpn History B2 #1 0.86 World History B2 #3 0.48 Jpn History B2 #3 0.71 Jpn History B1 #2 0.76 Geo B #1 0.91 Jpn History B1 #4 0.88 Geo B #4 0.95
bootstrap sample from two-thirds of the original data. The remaining one-third of cases is used for the test data.
The default procedure runs 500 times and forms the final classification tree using the most votes, when using the RandomForest package implemented in R(Liaw and Wiener, 2015). The error rate for the test set can be obtained at the same time. The numerical values shown in Table 3 were obtained using this procedure. The sample sizes ranged in 70–120, depending on the subjects.
If we compare the original data with the estimates using the final RF model, the concordance rate will be near 100% because too many predictive variables were compared with the sample size.
4.2 Variables that contribute to the classification
RFs evaluate the importance of variables in distinction using an index of the Gini coefficient. The bigger the coefficient, the more the classification is affected. Figure 4 shows variables arranged in descending order of the Gini coefficient; the horizontal axis shows their values. Due to space limitations, we show three cases of world history B2 #3, Japanese history B1 #2, and geometry B #4.
The three dominant contributions in case of world history B2 #3 are as follows:
QA sim: The cosine similarity between test item sentences and the answer.
SA sim 1: The cosine similarity between model answer #1 and the answer.
sa jpkwrel Fvl gold std01: The F-measure in keywords agreement between model answer #1 and the answer.
A variable name that starts with a capital letter implies linguistic semantic meaning built by the vocabulary used in Japanese Wikipedia. The typical examples are cosine similarity, precision, recall, and F-measure in the semantic space. Variations that multiply the allotment are also included.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
The variable names that start with a capital letter are an index showing surface-like lexical relationships. The total number of used variables is from 40 to 70, depending on the size of the scoring criteria.
When the indexes indicated in figure 4 are compared by each test item, the variables that contribute to distinction are not fixed. However, the Fmeasure and cosine similarity between the model (correct) answer and the answer in the semantic space often appear in a higher position. Generally speaking, semantic variables are dominant compared with surficial lexical variables.
4.3 Inspection using agreement of case elements
The case grammar proposed by Charles J. Fillmore (Fillmore, 1968) is part of a theory that tries to understand sentence meanings by putting a verb (the word with declined or conjugated endings) at the center of the understanding and that analyzes using a combination with deep cases such as the Agent and Object. Deep cases are often not determined easily, so we use surface cases instead of them. We tried a check on the agreement with case elements of a “model answer” and those of a “written answer.” The mechanism for the inspection had been arranged.
However, almost no agreement of case elements happened; the degree of agreement does not become one of the effective predictors of the classification. The ways in which sentences can be filled out to indicate the same meaning are numerous. Therefore, we did not add case grammar variables on the agreement of the case elements in this experiment.
5 Conclusion
Recognizing textual entailment between a model (correct) answer and a written answer is still difficult technically because complicated collation needs to be determined under contractual and semantic levels. Our technique is based on the collation between the keywords in two answers, and it uses both predictors considering superficial and semantic aspects. Therefore, it can be judged as sufficiently realistic for an approach of the first step. A form that entrusts the last judgment to the human is most suitable.
In this study, we investigated cases of scoring in social studies (geography and world/Japanese
sa_jpkwrel_prcxpt_part1_std04 sa_jpkwrel_Fvlxpt_part2_std05 SA_f1_part3 sa_jpkwrel_prcxpt_part2_std05 sa_jpkwrel_prc_part1_std03 sa_jpkwrel_prc_part2_std05 sa_jpkwrel_Fvlxpt_part1_std02 sa_jpkwrel_Fvl_part2_std05 sa_jpkwrel_prcxpt_gold__std01 sa_jpkwrel_Fvl_part1_std02 SA_pr_1 sa_jpkwrel_prcxpt_part1_std03 sa_jpkwrel_rcl_gold__std01 sa_jpkwrel_prc_lack2_std08 sa_jpkwrel_mcc_gold__std01 sa_jpkwrel_rclxpt_gold__std01 sa_jpkwrel_prc_gold__std01 sa_jpkwrel_Fvlxpt_part1_std03 SA_prXpt_1 sa_jpkwrel_Fvl_part1_std04 sa_jpwc_morps sa_jpkwrel_Fvl_part1_std03 sa_jpkwrel_Fvlxpt_part1_std04 SA_f1_1 sa_jpkwrel_Fvlxpt_gold__std01 sa_jpwc_chars sa_jpwc_words sa_jpkwrel_Fvl_gold__std01 SA_sim_1 QA_sim
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.5 1.0 1.5
WrdHistoryB2_3
MeanDecreaseGini
sa_jpkwrel_prcxpt_lack__std05 sa_jpkwrel_rclxpt_gold__std02 sa_jpkwrel_prc_lack__std05 SA_prXpt_2 sa_jpkwrel_rclxpt_lack__std05 sa_jpkwrel_prcxpt_gold__std01 sa_jpkwrel_Fvl_gold__std01 sa_jpkwrel_mcc_gold__std02 SA_pr_2 sa_jpkwrel_prcxpt_gold__std02 sa_jpkwrel_Fvl_part__std03 sa_jpkwrel_mcc_gold__std01 SA_f1_2 sa_jpkwrel_Fvlxpt_lack__std05 sa_jpkwrel_Fvlxpt_gold__std01 sa_jpkwrel_Fvlxpt_gold__std02 SA_prXpt_1 sa_jpkwrel_prc_part__std03 sa_jpkwrel_prcxpt_part__std03 sa_jpwc_chars sa_jpkwrel_prc_gold__std02 sa_jpkwrel_Fvlxpt_part__std03 sa_jpkwrel_Fvl_lack__std05 QA_sim SA_f1_1 sa_jpkwrel_Fvl_gold__std02 sa_jpwc_words sa_jpwc_morps SA_sim_2 SA_sim_1
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.1 0.2 0.3 0.4 0.5 0.6
JpnHistoryB1_2
MeanDecreaseGini
sa_jpkwrel_prc_part2_std08 SA_f1_part6 sa_jpkwrel_Fvl_part2_std07 sa_jpkwrel_rcl_gold__std01 sa_jpkwrel_prcxpt_part2_std04 sa_jpkwrel_Fvl_part2_std04 sa_jpkwrel_rclxpt_part1_std02 sa_jpkwrel_Fvlxpt_gold__std01 sa_jpkwrel_prc_part2_std04 sa_jpwc_chars sa_jpkwrel_Fvlxpt_part2_std07 SA_f1_part1 sa_jpkwrel_Fvl_part2_std08 sa_jpkwrel_rcl_part1_std02 SA_f1_part7 SA_prXpt_part1 sa_jpkwrel_Fvlxpt_part2_std08 sa_jpkwrel_Fvlxpt_part2_std04 sa_jpwc_morps sa_jpkwrel_Fvl_gold__std01 sa_jpkwrel_prcxpt_gold__std01 SA_pr_part1 sa_jpwc_bnsts QA_sim sa_jpkwrel_prc_gold__std01 SA_sim_1 sa_jpkwrel_prcxpt_part1_std02 sa_jpkwrel_Fvl_part1_std02 sa_jpkwrel_Fvlxpt_part1_std02 sa_jpkwrel_prc_part1_std02
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0 1.2
GeoB_4
MeanDecreaseGini
Figure 4: Variables that contribute to distinction (In the case of world history B2 #3, Japanese history B1 #2, and geometry B #4)
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
history). Our procedure will be applied to other subjects, such as Japanese literature, when many different transcriptions of a correct answer are prepared. Many different expressions in the same sense can be allowed, especially in Japanese. Our system has a mechanism to choose the biggest score among the same labels, so this can be prepared using the specifications of the current state.
Moreover, when there is a content-like important word or phrase such as “father’s feeling,” our system can register this type of words as “mandatory” in scoring criteria and can decrease the score by a suitable allotment when it is lacking. This function has already been implemented and will be helpful to understand the semantics.
However, a sufficiently large number of human scores cannot be provided for supervised learning. Indeed, actual written answer scores are often zero because they are illogical or are off-topic. Obtaining suitable and well-balanced score data will be necessary to ensure proper estimation.
Because a short written test has been proposed as a new common test for entering Japanese universities, our new scoring and support system is now being considered. We hope this system will provide researchers who study this field and practical businesspeople with useful information and suggestions.
Acknowledgments
The authors would like to thank everybody in the Gakken group for offering the test sets, scoring criteria, and human scores. In particular, Mr. Tsugunao Matsuoka and Ms. Naoko Saigo gave us valuable comments. This project was supported by JSPS KAKENHI Grant Number JP:26350357.

This paper describes a system to assist written test scoring.  - Strengths: The paper represents an application of an interesting NLP problem -- recognizing textual entailment -- to an important task -- written test scoring.  - Weaknesses: There isn't anything novel in the paper. It consist of an application of an existing technology to a known problem.  The approach described in the paper is not autonomous -- it still needs a human to do the actual scoring. The paper lacks any quantitative or qualitative evaluation of how useful such system is. That is, is it making the job of the scorer easier? Is the scorer more effective as compared to not having automatic score?  The system contains multiple components and it is unclear how the quality of each one of them contributes to the overall experience.  The paper needs more work with the writing. Language and style is rough in several places.  The paper also contains several detailed examples, which don't necessarily add a lot of value to the discussion.   For the evaluation of classification, what is the baseline of predicting the most frequent class?  - General Discussion: I find this paper not very inspiring. I don't see the message in the paper apart from announcing having build such a system