The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task
A writer’s style depends not just on personal traits but also on her intent and mental state. In this paper, we show how variants of the same writing task can lead to measurable differences in writing style. We present a case study based on the story cloze task (Mostafazadeh et al., 2016a), where annotators were assigned similar writing tasks with different constraints: (1) writing an entire story, (2) adding a story ending for a given story context, and (3) adding an incoherent ending to a story. We show that a simple linear classifier informed with stylistic features is able to successfully distinguish among the three cases, without even looking at the story context. In addition, our style-based classifier establishes a new state-of-the-art result on the story cloze challenge, substantially higher than previous results based on deep learning models. Our results demonstrate that different task framings can dramatically affect the way people write.
1 Introduction
Writing style is expressed through a range of linguistic elements such as words, sentence structure, and rhetorical devices. It is influenced by personal factors such as age and gender (Schler et al., 2006), by personality traits such as agreeableness and openness (Ireland and Mehl, 2014), as well as by mental states such as sentiment (Davidov et al., 2010), sarcasm (Tsur et al., 2010), and deception (Feng et al., 2012). In this paper, we study the extent to which writing style is affected by the nature of the writing task the writer was asked to perform, since different tasks likely engage different cognitive processes (Campbell and Pennebaker, 2003;
Banerjee et al., 2014).
We show that similar writing tasks with different constraints on the author can lead to measurable differences in her writing style. As a case study, we present experiments based on the recently introduced ROC story cloze task (Mostafazadeh et al., 2016a). In this task, authors were asked to write five-sentence self-contained stories, henceforth original stories. Then, each original story was given to a different author, who was shown only the first four sentences as a story context, and asked to write two contrasting story endings: a right (coherent) ending, and a wrong
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
(incoherent) ending. Framed as a story cloze task, the goal of this dataset is to serve as a commonsense challenge for NLP and AI research. Table 1 shows an example of an original story, a coherent story, and an incoherent story.
While the story cloze task was originally designed to be a story understanding challenge, its annotation process introduced three variants of the same writing task: writing an original, right, or wrong ending to a short story. In this paper, we show that a linear classifier informed by stylistic features can distinguish among the different endings to a large degree, even without looking at the story context (64.5–75.6% binary classification results).
Our results allow us to make a few key observations. First, people adopt a different writing style when asked to write coherent vs. incoherent story endings. Second, people change their writing style when writing the entire story on their own compared to writing only the final sentence for a given story context written by someone else.
In order to further validate our method, we also directly tackle the story cloze task. Adapting our classifier to the task, we obtain 72.4% accuracy, a 12.5% increase over the previously reported stateof-the-art (Salle et al., 2016). We also show that the style differences captured by our model can be combined with neural language models to make a better use of the story context. Our final model that combines context with stylistic features achieves 75.2%—an additional 2.8% gain, 15.3% better than the best published result.
The contributions of our study are threefold. First, findings from our study can potentially shed light on how different kinds of cognitive load influence the style of written language. Second, our results indicate that when designing new NLP tasks, special attention needs to be payed to the instructions given to authors. Third, we establish a new state-of-the-art result on the commonsense story cloze challenge.
2 Background: The Story Cloze Task
To understand how different writing tasks affect writing style, we focus on the story cloze task (Mostafazadeh et al., 2016a). While this task was developed to facilitate representation and learning of commonsense story understanding, its design included a few key choices which make it ideal for our study. We describe the task below.
ROC stories. The ROC story corpus consists of 49,255 five-sentence commonsense stories, collected on Amazon Mechanical Turk (AMT).1 Workers were instructed to write a coherent selfcontained story, which has a clear beginning and end. To collect a broad spectrum of commonsense knowledge, there was no imposed subject for the stories, which resulted in a wide range of different topics.
Story cloze task. After compiling the story corpus, the story cloze task—a task based on the corpus—was introduced. A subset of the stories was selected, and only the first four sentences of each story were presented to AMT workers. Workers were asked to write a pair of new story endings for each story context: one right and one wrong. Both endings are required to complete the story using one of the characters in the story context. Additionally, the ending is required to be “realistic and sensible” (Mostafazadeh et al., 2016a) when read out of context.
The resulting stories, both right and wrong, were then individually rated for coherence and meaningfulness by additional AMT workers. Only stories rated as simultaneously coherent with a right ending and neutral with a wrong ending were selected for the task. It is worth noting that workers rated the stories as a whole, not only the endings.
Based on the new stories, Mostafazadeh et al. (2016a) proposed the story cloze task. The task is simple: given a pair of stories that differ only in their endings, the system decides which ending is right and which is wrong. The official training data contains only the original stories (without alternative endings), while development and test data consist of the revised stories with alternative endings (for a different set of original stories that are not included in the training set). The task was suggested as an extensive evaluation framework: as a commonsense story understanding task, as the shared task for the Linking Models of Lexical, Sentential and Discourse-level Semantics workshop (LSDSem 2017), and as a testbed for vectorspace evaluation (Mostafazadeh et al., 2016b).
Interestingly, at the time of this submission, 10 months after the task was first introduced, the published benchmark on this task is still below
1Recently, an additional 53K stories were released, which results in roughly 100K stories.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
60% (Salle et al., 2016).2 This comes in contrast to other recent similar machine reading tasks such as CNN/DailyMail (Hermann et al., 2015), SNLI (Bowman et al., 2015), LAMBADA (Paperno et al., 2016) and SQuAD (Rajpurkar et al., 2016), for which results improved dramatically over a similar or shorter period of time. This suggests that this task is challenging and that high performance is hard to achieve.
In addition, Mostafazadeh et al. (2016a) made substantial efforts to ensure the quality of this dataset. First, each pair of endings was written by the same author, which ensured that style differences between authors could not be used to solve the task. Furthermore, Mostafazadeh et al. implemented nine baselines for the task, using surface level features as well as narrative-informed ones, and showed that each of them reached roughly chance-level. These results suggest that real understanding of text is required in order to solve the task.
Different writing tasks in the story cloze task. Several key design decisions make the task an interesting testbed for the purpose of this study. First, the training set for the task (ROC Stories corpus) is not a training sample in the usual sense,3 as it contains only positive (right) samples, and not negative (wrong) ones.
On top of that, the original endings, which serve as positive training samples, were generated differently from the right samples, which serve as the positive samples in the development and test sets. While the former are part of a single coherent story written by the same author, the latter were generated by letting an author read four sentences, and then asking her to generate a fifth right ending.
Finally, although the right and wrong sentences were generated by the same author, the tasks for generating them were quite different: in one case, the author was asked to write a right ending, which would create a coherent five-sentence story along with the other four sentences. In the other case, the author was asked to write a wrong ending, which would result in an incoherent five-sentence story.
2The LSDSem 2017 shared task website (https: //competitions.codalab.org/competitions/ 15333) does report higher results, which are still unpublished along with the underlying methodology.
3I.e., the training instances are not drawn from a population similar to the one that future testing instances will be drawn from.
3 Surface Analysis of the Story Cloze Task
We begin by computing several characteristics of the three types of endings: original endings (from the ROC story corpus training set), right endings and wrong endings (both from the story cloze task development set). Our analysis reveals several style differences between different groups. First, original endings are on average longer (11 words per sentence) than right endings (8.75 words), which are in turn slightly longer than wrong ones (8.47 words). Previous work has shown that sentence length is also indicative of whether a text was deceptive (Yancheva and Rudzicz, 2013; Qin et al., 2004). Although writing wrong sentences is not the same as lying, it is not entirely surprising to observe similar trends in both tasks.
Second, Figure 1a shows the distribution of five frequent POS tags in all three groups. The figure shows that both original and right endings use pronouns more frequently than wrong endings. Once again, deceptive text is also characterized by fewer pronouns compared to truthful text (Newman et al., 2003).
Finally, Figure 1b presents the distribution of five frequent words across the different groups. The figure shows that original endings use coordinations (“and”) more than right endings, and substantially more than wrong ones. Furthermore, original and right endings seem to prefer enthusiastic language (e.g., “!”), while wrong endings tend to use more negative language (“hates”), similarly as deceptive text (Newman et al., 2003). Next we show that these style differences are not anecdotal, but can be used to distinguish among the different types of story endings.
4 Model
The goal of this paper is to determine the extent to which different writing constraints lead the authors to adopt different writing styles. In order to answer these questions, we use simple methods that have been shown to be very effective for recognizing style (see Section 9). We describe our model below.
We train a logistic regression classifier to categorize an ending, either as right vs. wrong or as original vs. new (right). Each feature vector is computed using the words in one ending, without considering earlier parts of the story. We use the following style features.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
original right wrong
5
10 15 Fr eq ue nc y in C
or pu
s (%
) NN VBD PRP DT NNP
(a) POS tags
original right wrong 0
1
2
3
to and I hates !
(b) Words
Figure 1: The distribution of five frequent POS tags (1a) and words (1b) across original endings (story cloze training set), and right and wrong endings (from the story cloze task).
• Length. The number of words in the sentence.
• Word n-grams. We use sequences of 1-5 words. Following Tsur et al. (2010) and Schwartz et al. (2013b), we distinguish between high frequency and low frequency words. Specifically, we replace content words (nouns, verbs, adjectives, and adverbs), which are often low frequency, with their part-of-speech tags.
• Character n-grams. Character n-grams are one of the most useful features in identifying author style (Stamatatos, 2009). We use character 4-grams.
5 Experiments
We design two experiments to answer our research questions. The first is an attempt to distinguish between right and wrong endings, the second between original endings and new (right) endings. We describe both experiments below.
Experiment 1: right/wrong endings. The goal of this experiment is to measure the extent to which style features capture differences between the right and wrong endings. As the story cloze task doesn’t have a training corpus for the right and wrong endings (see Section 2), we use the development set as our training set, holding out 10% for development (3,366 training endings, 374 for development). We keep the story cloze test set as is (3,742 endings).
It is worth noting that our classification task is slightly different from the story cloze task. Instead of classifying pairs of endings, one which is right and another which is wrong, our classifier decides about each ending individually, whether it
is right (positive instance) or wrong (negative instance). By ignoring the coupling between right and wrong pairs, we are able to decrease the impact of author-specific style differences, and focus on the difference between the styles accompanied with right and wrong writings.
Experiment 2: original/new endings. Here the goal is to measure whether writing the ending as part of a story imposes different style compared to writing a new (right) ending to an existing story. We use the endings of the ROC stories as our original samples and right endings from the story cloze task as new samples. As there are far more original instances than new instances, we randomly select the same number of original instances as we have new instances (3,366 training endings, 374 development endings, and 3,742 test endings). We randomly sample 5 original sets and repeat the classification experiments. We report the average classification result.
Experimental setup. In both experiments, we add a START symbol at the beginning of each sentence.4 For computing our features, we keep ngram (character or word) features that occur at least five times in the training set. All feature values are normalized to [0, 1]. For the POS features, we tag all endings with the Spacy POS tagger.5 We use Python’s sklearn logistic regression implementation (Pedregosa et al., 2011) with L2 regularization, performing grid search on the development set to tune a single hyperparameter—the regularization parameter.
4Virtually all sentences end with a period or an exclamation mark, so we do not add a STOP symbol.
5http://spacy.io/
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Experiment Accuracy right vs. wrong 0.645 original vs. right 0.685 original vs. wrong 0.756
Table 2: Results of experiments 1 (right vs. wrong) and 2 (original vs. right (new)). The bottom row shows an additional experiment which classifies original vs. wrong (new) endings. In all cases, our setup implies a 50% random baseline.
6 Results
Table 2 shows our results. In both experiments, our model achieves performance well above what would be expected under chance (50% by design). Noting again that our model ignores the story context (the preceding four sentences), our model is unable to capture any notion of coherence. This finding provides strong evidence that the authors’ style was affected by the writing task they were given to perform. We further applied the model to a third task, original vs. wrong (new) endings, and even stronger differences were apparent to the model, allowing over 75% accuracy.
Story cloze task. The results of Experiment 1 indicate that right and wrong endings are characterized by different styles. In order to further estimate the quality of our classification results, we tackle the story cloze task using our classifier. This classification task is more constrained than Experiment 1, as two endings are given and the question is which is right and which is wrong. We apply the classifier from Experiment 1 as follows: if it assigns different labels to the two given endings, we keep them. Otherwise, the label whose posterior probability is lower is reversed.
Table 3 shows our results on the story cloze test set. Our classifier obtains 72.4% accuracy, 12.5% (absolute) higher than the published state-of-theart result on the task (Salle et al., 2016). Importantly, unlike previous approaches, our classifier does not require the story corpus training data, and in fact doesn’t even consider the first four sentences of the story in question. These numbers further support the claim that the styles of right and wrong endings are indeed very different.
Combination with a neural language model. We investigate whether our model can benefit from state-of-the-art text comprehension models,
Model Acc. †DSSM (Mostafazadeh et al., 2016a) 0.585 †LexVec (Salle et al., 2016) 0.599 †RNN 0.677 Ours 0.724 †Combined (ours + RNN) 0.752 †Human judgment 1.000
Table 3: Results on the test set of the story cloze task. The upper block shows published results, the middle block are our results. LexVec results are taken from (Speer et al., 2016). Human judgement scores are taken from (Mostafazadeh et al., 2016a). Methods marked with (†) use the story context in order to make a prediction.
for which this task was designed. Specifically, we experiment with an LSTM-based (Hochreiter and Schmidhuber, 1997) recurrent neural network language model (RNNLM; Mikolov et al., 2010). Unlike the model in this paper, which only considers the story endings, this language model follows the protocol suggested by the story cloze task designers, and harnesses their ROC Stories training set, which consists of single-ending stories, as well as the story context for each pair of endings. We show that adding our features to this powerful language model gives improvements over our classifier as well as the language model.
We train the RNNLM using a single-layer LSTM of hidden dimension 512. We use the ROC stories for training,6 setting aside 10% for validation of the language model. We replace all words occurring less than 3 times with a special out-ofvocabulary character, yielding a vocabulary size of 21,582. Only during training, we apply a dropout rate of 60% while running the LSTM over all 5 sentences of the stories. Using the Adam optimizer (Kingma and Ba, 2015) and a learning rate of η = .001, we train to minimize cross-entropy.
To apply the language model to the classification problem, we select as right the ending with the higher value of
pθ(ending | story) pθ(ending)
(1)
The intuition is that a right ending should be unsurprising (to the model) given the four preceding
6We use the extended, 100K stories corpus (see Section 2).
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Feature Type Accuracy Word n-grams 0.612
Character n-grams 0.639 Full model 0.645
Table 4: Results on Experiment 1 with different subsets of features.
sentences of the story (the numerator), controlling for the inherent surprisingness of the words in that ending (the denominator).
On its own, our neural language model performs moderately well on the story cloze test. Selecting endings based on pθ(ending | story) (i.e., the numerator of Equation 1), we obtained only 55% accuracy. The ratio in Equation 1 achieves 67.7% (see Table 3).7
We combine our linear model with the RNNLM by adding three features to our classifier: the numerator, denominator, and ratio in Equation 1, all in log space. We retrain our linear model with the new feature set, and gain 2.8% absolute, reaching 75.2% (15.3% better than the published state-ofthe-art result). These results indicate that contextignorant style features can be used to obtain high accuracy on the task, adding value even when context and a large training dataset are used.
7 Further Analysis
7.1 Most Discriminative Feature Types
A natural question that follows this study is which style features are most helpful in detecting the underlying task an author was asked to perform. To answer this question, we re-ran Experiment 1 with different sub-groups of features. Table 4 shows our results. Results show that character n-grams are the most effective style predictors, reaching within 0.6% of the full model, but that word ngrams also capture much of the signal, yielding 61.2%, which is only 3.3% worse than the full model. These findings are in line with previous work that used character n-grams along with other types of features to predict writing style (Schwartz et al., 2013b).
7.2 Most Salient Features
A follow-up question is which individual features contribute most to the classification process, as
7Note that taking the logarithm of the expression in Equation 1 gives the pointwise mutual information between the story and the ending, under the language model.
these could shed light on the stylistic differences imposed by each of the writing tasks.
In order to answer this question, we consider the highest absolute positive and negative coefficients in the logistic regression classifier in both Experiments 1 and 2, an approach widely used as a method of extracting the most salient features (Nguyen et al., 2013; Burke et al., 2013; Brooks et al., 2013). It is worth noting that its reliability is not entirely clear, since linear models like logistic regression can assign large coefficients to rare features (Yano et al., 2012). To mitigate this concern, we consider only features appearing in at least 5% of the endings in our training set.
Experiment 1. Table 5 shows the most salient features for right (coherent) and wrong (incoherent) endings in Experiment 1, along with their corpus frequency. The table shows a few interesting trends. First, authors tend to structure their sentences differently when writing coherent vs. incoherent endings. For instance, incoherent endings are more likely to start with a proper noun and end with a common noun, while coherent endings have a greater tendency to end with a past tense verb.
Second, right endings will make wider use of coordination structures, as well as adjectives. The latter might indicate that writing coherent stories inspires the authors to write more descriptive text compared to incoherent ones, as is the case in truthful vs. deceptive text (Ott et al., 2011). Finally, we notice a few syntactic differences: right endings will more often use infinite verb structure, while wrong endings prefer gerunds (VBG).
Experiment 2. Table 6 shows the same analysis for Experiment 2. As noted in Section 2, original endings tend to be much longer, which is indeed the most salient feature for them. An interesting observation is that exclamation marks are a strong indication for an original ending. This suggests
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Original Freq. New Freq. length .100.0% ‘.’ 93.0% ‘!’ 006.1% START NNP 39.2% NN 0078.9% START NNP VBD 29.0% RB 044.7% NN . 42.3% ‘,’ 012.7% the NN . 10.6%
Table 6: The top 5 most heavily weighted features for predicting original vs. new (right) endings. length is the sentence length feature (see Section 4).
that authors are more likely to show or evoke enthusiasm when writing their own text compared to ending an existing task.
Finally, when comparing the two groups of salient features from both experiments, we find an interesting trend. Several features, such as “START NNP” and “NN .”, which indicate wrong sentences in Experiment 1, are used to predict new (i.e., right) endings in Experiment 2. This indicates that, for instance, incoherent endings have a stronger tendency to begin with a proper noun compared to coherent endings, which in turn are more likely to do so than original endings. This partially explains why distinguishing between original and wrong endings is an easier task compared to the other pairs (Section 6).
8 Discussion
The effect of writing tasks on mental states. In this paper we have shown that giving a writer different writing tasks affects her writing style in easily detected ways. Our results indicate that when authors are asked to write the last sentence of a five-sentence story, they will use different style to write a right ending compared to a wrong ending. We have also shown that writing the ending as part of one’s own five-sentence story is very different than reading four sentences and then writing the fifth. Our findings hint that the nature of the writing task imposes a different mental state on the author, which is expressed in ways that can be observed using extremely simple automatic tools.
Previous work has shown that a writing task can affect mental state. For instance, writing deceptive text leads to a significant cognitive burden accompanied by a writing style that is different from truthful text (Newman et al., 2003; Banerjee et al., 2014). Writing tasks can even have a long-term
effect, as writing emotional texts was observed to benefit both physical and mental health (Lepore and Smyth, 2002; Frattaroli, 2006). Campbell and Pennebaker (2003) also showed that the health benefits of writing emotional text are accompanied by changes in writing style, mostly in the use of pronouns.
Another line of work has shown that writing style is affected by mental state. First, an author’s personality traits (e.g., depression, neuroticism, narcissism) affect her writing style (Schwartz et al., 2013a; Ireland and Mehl, 2014). Second, temporary changes, such as a romantic relationship (Ireland et al., 2011; Bowen et al., 2016), work collaboration (Tausczik, 2009; Gonzales et al., 2009), or negotiation (Ireland and Henderson, 2014) may also affect writing style. Finally, writing style can also change from one sentence to another, for instance between positive and negative text (Davidov et al., 2010) or when writing sarcastic text (Tsur et al., 2010).
This large body of work indicates a tight connection between writing tasks, mental states, and variation in writing style. This connection hints that the link discovered in this paper, between different writing tasks and resulting variation in writing style, involves differences in mental state. Further investigation is required in order to further validate this hypothesis.
Design of NLP tasks. Our study also provides important insights for the future design of NLP tasks. The story cloze task was very carefully designed. Many factors, such as topic diversity and temporal and causal relation diversity, were controlled for (Mostafazadeh et al., 2016a). The authors also made sure each pair of endings was written by the same author, partly in order to avoid author-specific style effects. Nonetheless, despite these efforts, several significant style differences can be found between the training and the test set, as well as between the positive and negative labels.
Our findings suggest that careful attention must be paid to instructions given to authors, especially in unnatural tasks such as writing a wrong ending. One way to avoid such problems is by using shorter text spans, such as the ones used in the Winograd schema (Levesque, 2011). A different approach is to use naturally occurring text, as used in recent machine reading tasks (see Section 9). One way to avoid the inherent biases people have when writing is to have them rate sentences from
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
naturally occurring text by parameters such as coherence (or, very differently, the level of surprise), rather than asking them to generate new text.
9 Related Work
Writing style. Writing style has been an active topic of research for decades. The models used to characterize style are often linear classifiers with style features such as character and word n-grams (Stamatatos, 2009; Koppel et al., 2009). Previous work has shown that different authors can be grouped by their writing style, according to factors such as age (Pennebaker and Stone, 2003; Argamon et al., 2003; Schler et al., 2006; Rosenthal and McKeown, 2011; Nguyen et al., 2011), gender (Argamon et al., 2003; Schler et al., 2006; Bamman et al., 2014), and native language (Koppel et al., 2005; Tsur and Rappoport, 2007; Bergsma et al., 2012). At the extreme case, each individual author adopts a unique writing style (Mosteller and Wallace, 1963; Pennebaker and King, 1999; Schwartz et al., 2013b).
The line of work that most resembles our work is the detection of deceptive text. Several researchers have used stylometric features to predict deception (Newman et al., 2003; Hancock et al., 2007; Ott et al., 2011; Feng et al., 2012). Some works even showed that gender affects a person’s writing style when lying (Pérez-Rosas and Mihalcea, 2014a,b). In this work, we have shown that an even more subtle writing task—writing coherent and incoherent story endings—imposes different styles on the author.
Machine reading. The story cloze task, which is the focus of this paper, is part of a wide set of machine reading/comprehension challenges published in the last few years. These include datasets like bAbI (Weston et al., 2016), SNLI (Bowman et al., 2015), CNN/DailyMail (Hermann et al., 2015), LAMBADA (Paperno et al., 2016) and SQuAD (Rajpurkar et al., 2016). While these works have presented resources for researchers, it is often the case that these datasets suffer from methodological problems caused by applying noisy automatic tools to generate them (Chen et al., 2016). In this paper, we have pointed to another methodological challenge in designing machine reading tasks: different writing tasks used to generated the data affect writing style, confounding classification problems.
10 Conclusion
Different writing tasks assigned to an author result in different writing styles for that author. We experimented with the story cloze task, which introduces two interesting comparison points: the difference between writing a story on one’s own and continuing someone else’s story, and the difference between writing a coherent and an incoherent story ending. In both cases, a simple linear model reveals measurable differences in writing styles, which in turn allows our final model to achieve state-of-the-art results on the story cloze task.
The findings presented in this paper have cognitive implications, as they motivate further research on the effects that a writing prompt has on an author’s mental state, and also her concrete response. They also provide valuable lessons for designing new NLP datasets.

The paper analyzes the story endings (last sentence of a 5-sentence story) in the corpus built for the story cloze task (Mostafazadeh et al. 2016), and proposes a model based on character and word n-grams to classify story endings. The paper also shows better performance on the story cloze task proper (distinguishing between  right  and  wrong  endings) than prior work.  Whereas style analysis is an interesting area and you show better results than prior work on the story cloze task, there are several issues with the paper. First, how do you define  style ? Also, the paper needs to be restructured (for instance, your section  Results  actually mixes some results and new experiments) and clarified (see below for questions comments): right now, it is quite difficult for the reader to follow what data is used for the different experiments, and what data the discussion refers to.  (1) More details about the data used is necessary in order to assess the claim that  subtle writing task [...] imposes different styles on the author  (lines 729-732). How many stories are you looking at, written by how many different persons? And how many stories are there per person? From your description of the post-analysis of coherence, only pairs of stories written by the same person in which one was judged as  coherent  and the other one as  neutral  are chosen. Can you confirm that this is the case? So perhaps your claim is justified for your  Experiment 1 . However my understanding is that in experiment 2 where you compare  original  vs.  right  or  original  vs.  wrong , we do not have the same writers. So I am not convinced lines 370-373 are correct.  (2) A lot in the paper is simply stated without any justifications. For instance how are the  five frequent  POS and words chosen? Are they the most frequent words POS? (Also theses tables are puzzling: why two bars in the legend for each category?). Why character *4*-grams? Did you tune that on the development set? If these were not the most frequent features, but some that you chose among frequent POS and words, you need to justify this choice and especially link the choice to  style . How are these features reflecting  style ?  (3) I don't understand how the section  Design of NLP tasks  connects to the rest of the paper, and to your results. But perhaps this is because I am lost in what  training  and  test  sets refer to here.  (4) It is difficult to understand how your model differs from previous work. How do we reconcile lines 217-219 ( These results suggest that real understanding of text is required in order to solve the task ) with your approach?  (5) The terminology of  right  and  wrong  endings is coming from Mostafazadeh et al., but this is a very bad choice of terms. What exactly does a  right  or  wrong  ending mean ( right  as in  coherent  or  right  as in  morally good )? I took a quick look, but couldn't find the exact prompts given to the Turkers. I think this needs to be clarified: as it is, the first paragraph of your section  Story cloze task  (lines 159-177) is not understandable.  Other questions comments:  Table 1. Why does the  original  story differ from the coherent and incoherent one? From your description of the corpus, it seems that one Turker saw the first 4 sentences of the original story and was then ask to write one sentence ending the story in a  right  way (or did they ask to provide a  coherent  ending?) and one sentence ending the story in a  wrong  way (or did they ask to provide an  incoherent  ending)? I don't find the last sentence of the  incoherent  story that incoherent... If the only shoes that Kathy finds great are $300, I can see how Kathy doesn't like buying shoes ;-) This led me to wonder how many Turkers judged the coherence of the story ending and how variable the judgements were. What criterion was used to judge a story coherent or incoherent? Also does one Turker judge the coherence of both the  right  and  wrong  endings, making it a relative judgement? Or was this an absolute judgement? This would have huge implications on the ratings.  Lines 380-383: What does  We randomly sample 5 original sets  mean?  Line 398:  Virtually all sentences ? Can you quantify this?  Table 5: Could we see the weights of the features?   Line 614:  compared to ending an existing task : the Turkers are not ending a  task   Line 684-686:  made sure each pair of endings was written by the same author  to this is true for the  right   wrong  pairs, but not for the  original - new  pairs, according to your description.  Line 694:  shorter text spans : text about what? This is unclear.  Lines 873-875: where is this published?


Learning attention for historical text normalization by learning to pronounce
Automated processing of historical texts often relies on pre-normalization to modern word forms. Training encoder-decoder architectures to solve such problems typically requires a lot of training data, which is not available for the named task. We address this problem by using several novel encoder-decoder architectures, including a multi-task learning (MTL) architecture using a grapheme-to-phoneme dictionary as auxiliary data, pushing the state-of-theart by an absolute 2% increase in performance. We analyze the induced models across 44 different texts from Early New High German. Interestingly, we observe that, as previously conjectured, multi-task learning can learn to focus attention during decoding, in ways remarkably similar to recently proposed attention mechanisms. This, we believe, is an important step toward understanding how MTL works.
1 Introduction
There is a growing interest in automated processing of historical documents, as evidenced by the growing field of digital humanities and the increasing number of digitally available collections of historical documents. A common approach to deal with the high amount of variance often found in this type of data is to perform spelling normalization (Piotrowski, 2012), which is the mapping of historical spelling variants to standardized/modernized forms (e.g. vnd→ und ‘and’).
Training data for supervised learning of historical text normalization is typically scarce. On the other hand, neural networks are said to work best when trained on large amounts of data. It is therefore not clear whether neural networks are a good
choice for this particular task. We explore framing the spelling normalization task as a character-based sequence-to-sequence transduction problem, and use encoder–decoder recurrent neural networks (RNNs) to induce our transduction models. This is similar to models that have been proposed for neural machine translation (e.g., Cho et al. (2014)), so essentially, our approach could also be considered a form of character-based neural machine translation.
By basing our model on individual characters as input, we keep the vocabulary size small, which in turn reduces the model’s complexity and the amount of data required to train it effectively. Using an encoder–decoder architecture removes the need for an explicit character alignment between historical and modern wordforms. Furthermore, we explore using an auxiliary task for which data is more readily available, namely grapheme-tophoneme mapping (word pronunciation), to regularize the induction of the normalization models.
We propose several architectures, including multi-task learning architectures taking advantage of the auxiliary data, and evaluate them across 44 small datasets from Early New High German.
Contributions Our contributions are as follows:
• We are, to the best of our knowledge, the first to propose and evaluate encoder-decoder architectures for historical text normalization.
• We evaluate several such architectures across 44 datasets of Early New High German.
• We show that such architectures benefit from bidirectional encoding, beam search, and attention.
• We also show that MTL with pronunciation as an auxiliary task improves the performance of architectures without attention.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
• We analyze the above architectures and show that the MTL architecture learns attention from the auxiliary task, making the attention mechanism largely redundant.
• We make our implementation publicly available at anonymized.
In sum, we both push the state-of-the-art in historical text normalization and present an analysis that, we believe, brings us a step further in understanding the benefits of multi-task learning.
2 Datasets
Normalization For the normalization task, we use a total of 44 texts from the Anselm corpus (Dipper and Schultz-Balluff, 2013) of Early New High German.1 The corpus is a collection of manuscripts and prints of the same core text, a religious treatise. Although the texts are semi-parallel and share some vocabulary, they were written in different time periods (between the 14th and 16th century) as well as different dialectal regions, and show quite diverse spelling characteristics. For example, the modern German word Frau ‘woman’ can be spelled as fraw/vraw (Me), frawe (N2), frauwe (St), fraüwe (B2), frow (Stu), vrowe (Ka), vorwe (Sa), or vrouwe (B), among others.2
All texts in the Anselm corpus are manually annotated with gold-standard normalizations following guidelines described in Krasselt et al. (2015). For our experiments, we excluded texts from the corpus that are shorter than 4,000 tokens, as well as a few for which annotations were not yet available at the time of writing (mostly Low German and Dutch versions). Nonetheless, the remaining 44 texts are still quite short for machine-learning standards, ranging from about 4,200 to 13,200 tokens, with an average length of 7,350 tokens.
For all texts, we removed tokens that consisted solely of punctuation characters. We also lowercase all characters, since it helps keep the size of the vocabulary low, and uppercasing of words is usually not very consistent in historical texts. Tokenization was not an issue for pre-processing these texts, since modern token boundaries have already been marked by the transcribers.
1https://www.linguistics.rub.de/ anselm/
2We refer to individual texts using the same internal IDs that are found in the Anselm corpus (cf. the website).
Grapheme-to-phoneme mappings We use learning to pronounce as our auxiliary task. This task consists of learning mappings from sequences of graphemes to the corresponding sequences of phonemes. We use the German part of the CELEX lexical database (Baayen et al., 1995), particularly the database of phonetic transcriptions of German wordforms. The database contains a total of 365,530 wordforms with transcriptions in DISC format, which assigns one character to each distinct phonological segment (including affricates and diphthongs). For example, the word Jungfrau ‘virgin’ is represented as ’jUN-frB.
3 Model
3.1 Base model
We propose several architectures that are extensions of a base neural network architecture, closely following the sequence-to-sequence model proposed by Sutskever et al. (2014). It consists of the following:
• an embedding layer that maps one-hot input vectors to dense vectors;
• an encoder RNN that transforms the input sequence to an intermediate vector of fixed dimensionality;
• a decoder RNN whose hidden state is initialized with the intermediate vector, and which is fed the output prediction of one timestep as the input for the next one; and
• a final dense layer with a softmax activation which takes the decoder’s output and generates a probability distribution over the output classes at each timestep.
For the encoder/decoder RNNs, we use long short-term memory units (LSTM) (Hochreiter and Schmidhuber, 1997). LSTMs are designed to allow recurrent networks to better learn long-term dependencies, and have proven advantageous to standard RNNs on many tasks. We found no significant advantage from stacking multiple LSTM layers for our task, so we use the simplest competitive model with only a single LSTM unit for both encoder and decoder.
By using this encoder–decoder model, we avoid the need to generate explicit alignments between the input and output sequences, which would bring up the question of how to deal with input/output
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
v r o w e (START) f r a u
f r a u (END)
Figure 1: Flow diagram of the base model; left side is the encoder, right side the decoder, the latter of which has an additional prediction layer on top. Multi-task learning variants use two separate prediction layers for main/auxiliary tasks, while sharing the rest of the model. Embedding layers for the inputs are not explicitly shown.
pairs of different lengths. Another important property is that the model does not start to generate any output until it has seen the full input sequence, which in theory allows it to learn from any part of the input, without being restricted to fixed context windows. An example illustration of the unrolled network is shown in Fig. 1.
3.2 Training
During training, the encoder inputs are the historical wordforms, while the decoder inputs correspond to the correct modern target wordforms. We then train each model by minimizing the crossentropy loss across all output characters; i.e., if y = (y1, ..., yn) is the correct output word (as a list of one-hot vectors of output characters) and ŷ = (ŷ1, ..., ŷn) is the model’s output, we minimize the mean loss− ∑n i=1 yi log ŷi over all training samples. For the optimization, we use the Adam algorithm (Kingma and Ba, 2015) with a learning rate of 0.003.
To reduce computational complexity, we also set a maximum word length of 14, and filter all training samples where either the input or output word is longer than 14 characters. This only affects 172 samples across the whole dataset, and is only done during training. In other words, we evaluate our models across all the test examples.
3.3 Decoding
For prediction, our base model generates output character sequences in a greedy fashion, selecting the character with the highest probability at each timestep. This works fairly well, but the greedy approach can yield suboptimal global picks, in which each individual character is sensibly derived from the input, but the overall word is non-
sensical. We therefore also experiment with beam search decoding, setting the beam size to 5.
Finally, we also experiment with using a lexical filter during the decoding step. Here, before picking the next 5 most likely characters during beam search, we remove all characters that would lead to a string not covered by the lexicon. This is again intended to reduce the occurrence of nonsensical outputs. For the lexicon, we use all word forms from CELEX (cf. Sec. 2) plus the target word forms from the training set.3
3.4 Attention
In our base architecture, we assume that we can decode from a single vector encoding of the input sequence. This is a strong assumption, especially with long input sequences. Attention mechanisms give us more flexibility. The idea is that instead of encoding the entire input sequence into a fixedlength vector, we allow the decoder to “attend” to different parts of the input character sequence at each time step of the output generation. Importantly, we let the model learn what to attend to based on the input sequence and what it has produced so far.
Our implementation is identical to the decoder with soft attention described by Xu et al. (2015): we calculate a context vector from the encoder’s output and the decoder’s hidden state, and factor this vector into the hidden state of the decoder for the next timestep. Additionally, we use a bidirectional encoder in this scenario, comprised of one LSTM layer that reads the input sequence normally and another LSTM layer that reads it back-
3We observe that due to this filtering, we cannot reach 2.25% of the targets in our test set, most of which are Latin word forms.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
wards, and attend over the concatenated outputs of these two layers.
While a precise alignment of input and output sequences is sometimes difficult, most of the time the sequences align in a sequential order, which can be exploited by an attentional component.
3.5 Multi-task learning
Finally, we introduce a variant of the base architecture, with or without beam search, that does multi-task learning (Caruana, 1993). The multitask architecture only differs from the base architecture in having two classifier functions at the outer layer, one for each of our two tasks. Our auxiliary task is to predict a sequence of phonemes as the correct pronunciation of an input sequence of graphemes. This choice is motivated by the relationship between phonology and orthography, in particular the observation that spelling variation often stems from phonological variation.
We train our multi-task learning architecture by alternating between the two tasks, sampling one instance of the auxiliary task for each training sample of the main task. We use the encoderdecoder to generate a corresponding output sequence, whether a modern word form or a pronunciation. Doing so, we suffer a loss with respect to the true output sequence and update the model parameters. The update for a sample from a specific task affects the parameters of corresponding classifier function, as well as all the parameters of the shared hidden layers.
3.6 Hyperparameters
We used a single manuscript (B) for manually evaluating and setting the hyperparameters. This manuscript is left out of the averages reported below. We believe that using a single manuscript for development, and using the same hyperparameters across all manuscripts, is more realistic, as we often do not have enough data in historical text normalization to reliably tune hyperparameters.
For the final evaluation, we set the size of the embedding and the recurrent LSTM layers to 128, applied a dropout of 0.3 to the input of each recurrent layer, and trained the model on mini-batches with 50 samples each for a total of 50 epochs (in the multi-task learning setup, mini-batches contain 50 samples of each task, and epochs are counted by the size of the training set for the main task only). All these parameters were set on the B manuscript alone.
3.7 Implementation
We implemented all of the models in Keras (Chollet, 2015). Any parameters not explicitly described here were left at their default values in Keras v1.0.8.
4 Evaluation
We split up each text into three parts, using 1,000 tokens each for a test set and a development set (that is not currently used), and the remainder of the text (between 2,000 and 11,000 tokens) for training. We then train and evaluate on each of the 43 texts (excluding the B text that was used for hyper-parameter tuning) individually.
Baselines We compare our architectures to several competitive baselines. Our first baseline is an averaged perceptron model trained to predict output character n-grams for each input character, after using Levenshtein alignment with generated segment distances (Wieling et al., 2009, Sec. 3.3) to align input and output characters. Our second baseline uses the same alignment, but trains a deep bi-LSTM sequential tagger, following Bollmann and Søgaard (2016). We evaluate this tagger using both standard and multi-task learning. Finally, we compare our model to the rule-based and Levenshtein-based algorithms provided by the Norma tool (Bollmann, 2012).4
4.1 Word accuracy
We use word-level accuracy as our evaluation metric. While we also measure character-level metrics, minor differences on character level can cause large differences in downstream applications, so we believe that perfectly matching the output sequences is more useful. Average scores across all 43 texts are presented in Table 1 (see Appendix A for individual scores).
We first see that almost all our encoder-decoder architectures perform significantly better than the four state-of-the-art baselines. All our architectures perform better than Norma and the averaged perceptron, and all the MTL architectures outperform Bollmann and Søgaard (2016).
We also see that beam search, filtering, and attention lead to cumulative gains in the context of the single-task architecture – with the best architecture outperforming the state-of-the-art by almost 3% in absolute terms.
4https://github.com/comphist/norma
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Average Std. Dev.
Norma 77.89% 2.99 Averaged perceptron 75.72% 3.19 Bi-LSTM tagger 79.91% 2.68 MTL bi-LSTM tagger 79.56% 2.71
Base model
GREEDY 78.91% 2.85 BEAM 79.27% 2.80 BEAM+FILTER 80.46% 2.61 BEAM+FILTER+ATTENTION 82.72% 2.24
MTL model GREEDY 80.64% 2.45 BEAM 81.13% 2.47 BEAM+FILTER 82.76% 2.22 BEAM+FILTER+ATTENTION 82.02% 2.41
Table 1: Average word accuracy across 43 texts from the Anselm dataset, evaluated on the first 1,000 tokens of each text. Evaluation on the base encoder-decoder model (Sec. 3.1) with greedy search, beam search (k = 5) and/or lexical filtering (Sec. 3.3), with attentional decoder (Sec. 3.4), and the multi-task learning (MTL) model using grapheme-to-phoneme mappings (Sec. 3.5).
For our multi-task architecture, we also observe gains when we add beam search and filtering, but importantly, adding attention does not help. In fact, attention hurts the performance of our multitask architecture quite significantly. Also note that the multi-task architecture without attention performs on-par with the single-task architecture with attention.
We hypothesize that the reason for this pattern, which is not only observed in the average scores in Table 1, but also quite consistent across the individual results in Appendix A, is that our multi-task learning already learns how to focus attention.
This is the hypothesis that we will try to validate in Sec. 5: That multi-task learning can induce strategies for focusing attention comparable to attention strategies for recurrent neural networks.
Sample predictions A small selection of predictions from our models is shown in Table 2. They serve to illustrate the effects of the various settings; e.g., the base model with greedy search tends to produce more nonsense words (ters, ünsget) than the others. Using a lexical filter helps the most in this regard: the base model with filtering correctly normalizes ergieng to erging ‘(he) fared’, while decoding without a filter produces the non-word erbiggen. Even for herczenlichen (modern herzlichen ‘heartfelt’), where no model finds the correct target form, only the model with filtering produces a somewhat reasonable alterna-
tive (herzgeliebtes ‘heartily loved’). In some cases (such as gewarnet ‘warned’), only the models with attention or multi-task learning produce the correct normalization, but even when they are wrong, they often agree on the prediction (e.g. dicke, herzel). We will investigate this property further in Sec. 5.
4.2 Learned vector representations
To gain further insights into our model, we created t-SNE projections (Maaten and Hinton, 2008) of vector representations learned on the M4 text.
Fig. 2 shows the learned character embeddings. In the representations from the base model (Fig. 2a), characters that are often normalized to the same target character are indeed grouped closely together: e.g., historical <v> and <u> (and, to a smaller extent, <f>) are often used interchangeably in the M4 text. Note the wide separation of <n> and <m>, which is a feature of M4 that does not hold true for all of the texts, as these do not always display a clear distinction between nasals. On the other hand, the MTL model shows a better generalization of the training data (Fig. 2b): here, <u> is grouped closer to other vowel characters and far away from <v>/<f>. Also, <n> and <m> are now in close proximity.
We can also visualize the internal word representations that are produced by the encoder (Fig. 3). Here, we chose words that demonstrate the interchangeable use of <u> and <v>. Histor-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Input Target Base model MTL model
GREEDY BEAM B+F B+F+ATT B+F
ergieng erging erbiggen erbiggen erging erging erging herczenlichen herzlichen herrgelichen herzgelichen herzgeliebtes herzel herzel tewr teuer ters terter terme teurer der iüngst jüngst ünsget pingst fingst fingst jüngst gewarnet gewarnt prandet prandert pranget gewarnt gewarnt dick oft oft oft oft dicke dicke
ical vnd, vns, vmb become modern und, uns, um, changing the <v> to <u>. However, the representation of vmb learned by the base model is closer to forms like von, vor, uor, all starting with <v> in the target normalization. In the MTL model, however, these examples are indeed clustered together.
5 Analysis: Multi-task learning helps focus attention
Table 1 shows that models which employ either an attention mechanism or multi-task learning obtain similar improvements in word accuracy. However, we observe a decline in word accuracy for models that combine multi-task learning with attention.
A possible interpretation of this counterintuitive pattern might be that attention and MTL, to some degree, learn similar functions of the input data, a conjecture by Caruana (1998). We put this hypothesis to the test by closely investigating properties of the individual models below.
5.1 Model parameters
First, we are interested in the weight parameters of the final layer that transforms the decoder output to class probabilities. We consider these parameters for our standard encoder-decoder model and compare them to the weights that are learned by the attention and multi-task models, respectively.5
We observe that the weight differences between the standard and the attention model correlate with the differences between the standard and multi-task model by a Pearson’s r of 0.346 (averaged across datasets, with a standard deviation of 0.315). Figure 4 illustrates these highly parallel weight changes for the different models when trained on the N4 dataset.
5For the multi-task models, this analysis disregards those dimensions that do not correspond to classes in the main task.
5.2 Final output
Next, we compare the effect that employing either an attention mechanism or multi-task learning has on the actual output of our system. We find that out of the 210.9 word errors that the base model produces on average across all test sets (comprising 1,000 tokens each), attention resolves 47.7, while multi-task learning resolves an average of 45.4 errors. Crucially, the errors that both of these approaches resolve independently of each other amount to 27.7 on average.
Attention and multi-task also introduce new errors compared to the base model (26.6 and 29.5 per test set, respectively), and again we can observe a relatively high agreement of the models (11.8 word errors are introduced by both models).
Finally, the attention and multi-task models display a word-level agreement of κ=0.834 (Fleiss).
5.3 Saliency analysis
Our last analysis regards the saliency of the input timesteps with respect to the predictions of our models. We follow Li et al. (2016) in calculating first-derivative saliency for given input/output pairs and compare the scores from the different models. The higher the saliency of an input timestep, the more important it is in determining the model’s prediction at a given output timestep. Therefore, if two models produce similar saliency matrices for a given input/output pair, they have learned to focus on similar parts of the input during the prediction. Our hypothesis is that the attentional and the multi-task learning model should be more similar in terms of saliency scores than either of them compared to the base model.
Figure 5 shows a plot of the saliency matrices generated from the word pair czeychen – zeichen ‘sign’. Here, the scores for the attentional and
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
(a) Base model (b) Multi-task learning model
Figure 2: t-SNE projections (with perplexity 7) of character embeddings from models trained on M4
(a) Base model (b) Multi-task learning model
Figure 3: t-SNE projections (with perplexity 5) of the intermediate vectors produced by the encoder (“historical word embeddings”), from models trained on M4
Figure 4: Heat map of parameter differences in the final dense layer between (a) the plain and the attention model as well as (b) the plain and the multi-task model, when trained on the N4 manuscript. The changes correlate by ρ = 0.959.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Figure 5: First-derivative saliency w.r.t. the input sequence, as calculated from the base model (left), the attentional model (center), and the MTL model (right). The scores for the attentional and the multi-task model correlate by ρ = 0.615, while the correlation of either one with the base model is |ρ| < 0.12.
the MTL model indeed correlate by ρ = 0.615, while those for the base model do not correlate with either of them. A systematic analysis across 19,000 word pairs (where all models agree on the output) shows that this effect only holds for longer input sequences (≥ 7 characters), with a mean ρ = 0.303 (±0.177) for attentional vs. MTL model, while the base model correlates with either of them by ρ < 0.21.
6 Related Work
Many traditional approaches to spelling normalization of historical texts use edit distances or some form of character-level rewrite rules, handcrafted (Baron and Rayson, 2008) or learned automatically (Bollmann, 2013; Porta et al., 2013).
A more recent approach is based on characterbased statistical machine translation applied to historical text (Pettersson et al., 2013; SánchezMartínez et al., 2013; Scherrer and Erjavec, 2013; Ljubešić et al., 2016) or dialectal data (Scherrer and Ljubešić, 2016). This is conceptually very similar to our approach, except that we substitute the classical SMT algorithms for neural networks. Indeed, our models can be seen as a form of character-based neural MT (Cho et al., 2014).
Neural networks have rarely been applied to historical spelling normalization so far; one example is Azawi et al. (2013), who use bi-directional LSTMs, along with a layer that performs alignment between input and output wordforms, to normalize old Bible text.
7 Conclusion and Future Work
We presented an approach to historical spelling normalization using neural networks with an
encoder-decoder architecture, and showed that it consistently outperforms several existing baselines. Encouragingly, our work proves to be fully competitive with the sequence-labeling approach by Bollmann and Søgaard (2016), without requiring a prior character alignment.
Specifically, we demonstrated the aptitude of multi-task learning to mitigate the shortage of training data for the named task. We included a multifaceted analysis of the effects that MTL introduces to our models and the resemblance that it bears to attention mechanisms. We believe that this analysis is a valuable contribution to the understanding of MTL approaches also beyond spelling normalization, and we are confident that our observations will stimulate further research into the relationship between MTL and attention.
Finally, many improvements to the presented approach are conceivable, most notably introducing some form of token context to the model. Currently, we only consider word forms in isolation, which is problematic for ambiguous cases (such as jn, which can normalize to in ‘in’ or ihn ‘him’) and conceivably makes the task harder for others. Reranking the predictions with a language model could be one possible way to improve on this. Ljubešić et al. (2016), for example, experiment with segment-based normalization, using segments (essentially, token n-grams) instead of single tokens as the input for their character-based SMT model, which also introduces context. Such an approach could also deal with the issue of tokenization differences between the historical and the modern text, which is another challenge often found in datasets of historical text.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Supplementary Material
For interested parties, we provide our full evaluation results for each single text in our dataset. Table 3 shows token counts, a rough classification of each text’s dialectal region, and the results for the baseline methods. Table 4 presents the full results for our encoder-decoder models.
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
ID Region Tokens Norma Avg. Perc. Bi-LSTM Tagger
BASE MTL
B East Central 4,718 79.60% 76.30% 79.20% 78.82% D3 East Central 5,704 79.70% 77.20% 80.10% 81.62% H East Central 8,427 83.00% 78.60% 85.00% 84.32%
B2 West Central 9,145 76.20% 74.60% 82.00% 80.12% KÄ1492 West Central 7,332 78.40% 74.80% 81.60% 80.82% KJ1499 West Central 7,330 77.00% 73.50% 84.50% 80.22% N1500 West Central 7,272 77.60% 72.70% 79.00% 78.52% N1509 West Central 7,418 78.40% 74.30% 80.80% 80.02% N1514 West Central 7,412 78.50% 72.20% 79.00% 79.62% St West Central 7,407 73.30% 70.30% 75.50% 73.03%
D4 Upper/Central 5,806 76.10% 72.40% 76.50% 76.62% N4 Upper 8,593 79.30% 80.00% 81.80% 82.52% s1496/97 Upper 5,840 81.20% 77.70% 83.00% 82.62%
B3 East Upper 6,222 82.30% 79.50% 81.50% 83.02% Hk East Upper 8,690 79.10% 78.20% 80.90% 79.52% M East Upper 8,700 75.20% 72.80% 83.90% 82.72% M2 East Upper 8,729 76.30% 75.10% 76.70% 79.32% M3 East Upper 7,929 79.20% 77.30% 80.40% 81.52% M5 East Upper 4,705 81.60% 76.40% 77.70% 76.92% M6 East Upper 4,632 74.90% 73.70% 75.20% 75.72% M9 East Upper 4,739 81.00% 79.00% 80.40% 79.32% M10 East Upper 4,379 77.20% 76.00% 75.10% 75.92% Me East Upper 4,560 80.20% 76.90% 80.30% 79.12% Sb East Upper 7,218 79.60% 75.70% 80.00% 80.12% T East Upper 8,678 76.00% 73.40% 75.80% 73.43% W East Upper 8,217 77.60% 78.20% 81.40% 80.72% We East Upper 6,661 82.70% 78.60% 81.50% 82.22%
Ba North Upper 5,934 79.10% 80.20% 80.70% 80.02% Ba2 North Upper 5,953 80.70% 78.10% 82.50% 82.12% M4 North Upper 8,574 76.70% 75.70% 79.40% 79.32% M7 North Upper 4,638 78.60% 75.60% 78.20% 77.42% M8 North Upper 8,275 79.30% 78.20% 81.10% 80.02% n North Upper 9,191 79.80% 81.90% 84.40% 84.62% N North Upper 13,285 74.00% 71.70% 79.00% 79.42% N2 North Upper 7,058 82.80% 80.30% 84.30% 81.72% N3 North Upper 4,192 78.10% 76.40% 77.60% 77.12%
Be West Upper 8,203 74.90% 75.30% 78.80% 77.52% Ka West Upper 12,641 72.80% 75.40% 80.10% 81.62% SG West Upper 7,838 79.70% 78.00% 81.70% 81.12% Sa West Upper 8,668 71.50% 71.90% 76.10% 74.93% Sa2 West Upper 8,834 77.60% 73.50% 79.50% 79.72% St2 West Upper 8,686 72.80% 73.20% 78.20% 79.92% Stu West Upper 8,011 78.00% 76.50% 79.40% 79.62%
Le Dutch 7,087 71.30% 65.00% 75.60% 75.12%
Average (-B) 7,353 77.89% 76.30% 79.91% 79.56%
Table 3: Word accuracy on the Anselm dataset, evaluated on the first 1,000 tokens, using the baseline models (cf. Sec. 4): the Norma tool (Bollmann, 2012), an averaged perceptron model, and a deep biLSTM sequential tagger (Bollmann and Søgaard, 2016).
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
ID Base model Multi-task learning model
G B B+F B+F+A G B B+F B+F+A
B 76.90% 77.30% 78.40% 82.70% 77.70% 79.50% 81.70% 80.10% D3 81.50% 81.60% 82.70% 83.20% 81.10% 81.70% 82.90% 83.20% H 82.60% 82.90% 84.50% 87.40% 85.00% 85.80% 86.60% 85.20%
B2 81.00% 81.20% 82.40% 83.40% 80.00% 80.40% 82.70% 83.00% KÄ1492 83.00% 83.40% 83.60% 84.00% 83.40% 83.70% 85.10% 84.90% KJ1499 81.30% 81.30% 82.00% 84.60% 84.00% 84.00% 83.80% 82.50% N1500 79.50% 80.30% 81.30% 84.00% 82.20% 82.50% 83.60% 82.30% N1509 82.10% 82.40% 83.10% 85.00% 82.80% 83.50% 84.50% 82.80% N1514 80.40% 80.50% 81.10% 83.40% 82.30% 82.80% 84.20% 83.10% St 74.60% 74.60% 76.40% 79.70% 77.60% 77.80% 80.20% 77.70%
D4 77.90% 77.20% 79.00% 81.40% 77.00% 77.90% 81.50% 79.90% N4 82.10% 82.30% 82.90% 84.80% 83.10% 83.00% 84.40% 84.00% s1496/97 80.40% 80.10% 81.10% 82.10% 82.30% 82.50% 85.20% 83.90%
B3 80.80% 81.20% 82.20% 85.20% 82.70% 83.30% 84.80% 84.50% Hk 77.30% 79.00% 79.40% 82.90% 80.30% 80.40% 81.20% 83.70% M 81.40% 81.50% 82.60% 85.00% 82.90% 82.90% 82.70% 84.00% M2 79.90% 80.50% 81.30% 81.80% 78.80% 77.80% 79.60% 83.20% M3 81.00% 81.10% 82.00% 83.70% 82.80% 82.50% 83.50% 81.70% M5 76.60% 77.10% 79.00% 82.00% 78.20% 78.20% 80.90% 81.50% M6 72.70% 73.80% 75.20% 80.20% 77.30% 79.00% 80.30% 76.60% M9 78.20% 78.50% 79.70% 83.20% 80.70% 79.70% 83.20% 79.60% M10 72.00% 72.40% 73.20% 77.40% 75.70% 76.30% 77.90% 77.80% Me 76.90% 76.50% 78.50% 81.30% 77.30% 79.20% 81.00% 77.40% Sb 78.80% 79.10% 81.30% 81.40% 80.60% 81.00% 84.00% 82.90% T 75.60% 75.10% 77.40% 80.30% 76.90% 78.00% 80.10% 79.50% W 80.80% 81.20% 82.40% 81.90% 80.40% 81.60% 84.40% 84.40% We 77.70% 80.00% 81.80% 84.40% 83.00% 82.70% 83.80% 83.30%
Ba 81.00% 80.60% 80.90% 84.00% 80.40% 81.00% 82.60% 81.60% Ba2 79.70% 80.90% 82.00% 84.00% 82.60% 83.30% 85.40% 85.10% M4 78.40% 78.60% 79.90% 81.00% 82.10% 82.20% 82.60% 80.50% M7 74.70% 76.30% 78.60% 82.00% 79.60% 79.90% 82.30% 81.10% M8 80.80% 81.30% 82.50% 85.70% 82.00% 82.50% 84.00% 85.40% n 83.40% 83.40% 84.30% 86.00% 84.90% 86.30% 88.00% 85.50% N 77.40% 77.40% 79.40% 79.80% 80.00% 80.30% 81.50% 80.30% N2 82.00% 82.30% 83.80% 86.40% 82.40% 83.50% 86.60% 85.80% N3 73.60% 74.00% 75.10% 81.20% 76.00% 76.30% 80.30% 78.70%
Be 75.50% 75.40% 77.60% 78.10% 78.10% 78.40% 79.70% 80.20% Ka 81.20% 81.20% 81.80% 83.90% 81.20% 83.10% 83.40% 82.30% SG 81.10% 81.90% 83.40% 85.50% 82.60% 84.30% 84.90% 83.00% Sa 76.80% 77.20% 78.10% 80.60% 77.50% 78.00% 79.70% 79.90% Sa2 78.90% 79.70% 80.70% 81.30% 79.70% 81.00% 82.30% 82.30% St2 77.70% 78.10% 79.00% 81.60% 79.60% 79.70% 80.50% 80.60% Stu 77.40% 77.30% 78.30% 82.50% 82.00% 81.80% 83.10% 82.90%
Le 77.40% 78.10% 78.20% 79.60% 78.30% 78.60% 79.80% 78.90%
Average (-B) 78.91% 79.27% 80.46% 82.72% 80.64% 81.13% 82.76% 82.02%
Table 4: Word accuracy on the Anselm dataset, evaluated on the first 1,000 tokens, using our base encoder-decoder model (Sec. 3) and the multi-task model. G = greedy decoding, B = beam-search decoding (with beam size 5), F = lexical filter, A = attentional model. Best results (also taking into account the baseline results from Table 3) shown in bold.

[update after reading author response: the alignment of the hidden units does not match with my intuition and experience, but I'm willing to believe I'm wrong in this case.  Discussing the alignment in the paper is important (and maybe just sanity-checking that the alignment goes away if you initialize with a different seed).  If what you're saying about how the new model is very different but only a little better performing -- a 10% error reduction -- then I wonder about an ensemble of the new model and the old one.  Seems like ensembling would provide a nice boost if the failures across models are distinct, right?  Anyhow this is a solid paper and I appreciate the author response, I raise my review score to a 4.]  - Strengths:    1)  Evidence of the attention-MTL connection is interesting    2)  Methods are appropriate, models perform well relative to state-of-the-art  - Weaknesses:    1)  Critical detail is not provided in the paper    2)  Models are not particularly novel  - General Discussion:  This paper presents a new method for historical text normalization.  The model performs well, but the primary contribution of the paper ends up being a hypothesis that attention mechanisms in the task can be learned via multi-task learning, where the auxiliary task is a pronunciation task.  This connection between attention and MTL is interesting.  There are two major areas for improvement in this paper.  The first is that we are given almost no explanation as to why the pronunciation task would somehow require an attention mechanism similar to that used for the normalization task.  Why the two tasks (normalization and pronunciation) are related is mentioned in the paper: spelling variation often stems from variation in pronunciation.  But why would doing MTL on both tasks result in an implicit attention mechanism (and in fact, one that is then only hampered by the inclusion of an explicit attention mechanism?).                    This remains a mystery.  The paper can leave some questions unanswered, but at least a suggestion of an answer to this one would strengthen the paper.  The other concern is clarity.  While the writing in this paper is clear, a number of details are omitted.                    The most important one is the description of the attention mechanism itself.  Given the central role that method plays, it should be described in detail in the paper rather than referring to previous work.  I did not understand the paragraph about this in Sec 3.4.  Other questions included why you can compare the output vectors of two models (Figure 4), while the output dimensions are the same I don't understand why the hidden layer dimensions of two models would ever be comparable.  Usually how the hidden states are  organized  is completely different for every model, at the very least it is permuted.                    So I really did not understand Figure 4.  The Kappa statistic for attention vs. MTL needs to be compared to the same statistic for each of those models vs. the base model.  At the end of Sec 5, is that row < 0.21 an upper bound across all data sets?  Lastly, the paper's analysis (Sec 5) seems to imply that the attention and MTL approaches make large changes to the model (comparing e.g. Fig 5) but the experimental improvements in accuracy for either model are quite small (2%), which seems like a bit of a contradiction.