MORSE: Semantic-ally Drive-n MORpheme SEgment-er
We present in this paper a novel framework for morpheme segmentation which uses the morpho-syntactic regularities preserved by word representations, in addition to orthographic features, to segment words into morphemes. This framework is the first to consider vocabulary-wide syntactico-semantic information for this task. We also analyze the deficiencies of available benchmarking datasets and introduce our own dataset that was created on the basis of compositionality. We validate our algorithm across datasets and present state-of-the-art results.
1 Introduction
Morpheme segmentation is a core natural language processing (NLP) task used as an integral component in systems related to fields such as information retrieval (IR) (Zieman and Bleich, 1997; Kurimo et al., 2007), automatic speech recognition (ASR) (Bilmes and Kirchhoff, 2003; Kurimo et al., 2006), and machine translation (MT) (Lee, 2004; Virpioja et al., 2007). Most previous works relied solely on orthographic features (Harris, 1970; Goldsmith, 2000; Creutz and Lagus, 2002, 2005, 2007) and neglected underlying semantic information. This has led to an oversegmentation of words because a change of the surface form pattern is a necessary but insufficient indication of a morphological change. For example, although the surface form of “freshman”, hints that it should be segmented to “freshman”, “freshman” does not describe semantically the compositional meaning of “fresh” and “man”.
To compensate for this lack of semantic knowledge, previous works (Schone and Jurafsky, 2000; Baroni et al., 2002; Narasimhan et al., 2015)
have incorporated semantic knowledge locally by checking the semantic relatedness of possibly morphologically related pair of words. In (Narasimhan et al., 2015), they check for semantic relatedness using cosine similarity in word representations (Mikolov et al., 2013a; Pennington et al., 2014). A limitation of such approach is due to noise in word representations, even more so in the case of rare words. Moreover, limitation to local comparison enforces modeling morphological relations via semantic relatedness, although it has been shown that difference vectors model morphological relations more accurately (Mikolov et al., 2013b). To address this issue, we introduce a new framework (MORSE), the first to bring semantics into morpheme segmentation both on a local and a vocabulary-wide level. That is, when checking for the morphological relation between two words, we not only check for the semantic relatedness of the pair at hand (local), but also check if the difference vectors of pairs showing similar orthographic change are consistent (vocabulary-wide).
To summarize MORSE, it clusters pairs of words which only vary by an affix, for example pairs such as (“quick”, “quickly”) and (“hopeful”, “hopefully”) get clustered together. To verify the cluster of a specific affix from a semantic corpuswide standpoint, we check for the consistency of the difference vectors (Mikolov et al., 2013b). To evaluate it from an orthographic corpus-wide perspective, we check for the size of each cluster of an affix. To evaluate each pair in a cluster locally from a semantic standpoint, we check if a pair of words in a valid affix cluster are morphologically related by checking if its difference vector is consistent with other members in the cluster and if the words in the pair are semantically related (i.e. close in the vector space). The reason for local evaluations is examples like (“on”,“only”) which belong to the cluster of a valid affix (“ly”) although
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
they are not morphologically related. We would expect such a pair to fail the last two local evaluation methods.
Our proposed segmentation algorithm is evaluated using benchmarking datasets from the Morpho Challenge (MC) for multiple languages and a newly introduced dataset for English which compensates for lack of discriminating capabilities in the MC dataset. Experiments reveal that our proposed framework not only outperforms the widely used approach, but also performs better than published state-of-the-art results.
The central contribution of this work is a novel framework that performs morpheme segmentation resulting in new state-of-the-art results. To the best of our knowledge this is the first unsupervised approach to consider the vocabulary-wide semantic knowledge of words and their affixes in addition to relying on their surface forms. Moreover we point out the deficiencies in the MC datasets with respect to the compositionality of morphemes and introduce our own dataset free of these deficiencies.
2 Related Work
Extensive work has been done in morphology learning, with tasks such as morphological analysis (Baayen et al., 1993), morphological reinflection (Cotterell et al., 2016), and morpheme segmentation. Given the less complex nature of morpheme segmentation in comparison to the other tasks, most systems developed for morpheme segmentation have been unsupervised to minimally supervised with the minimal supervision mainly used for parameter tuning.
Unsupervised morpheme segmentation could be traced back to (Harris, 1970). His work falls under the framework of Letter Successor Variety (LSV) which builds on the hypothesis that predictability of successor letters is high within morphemes and low otherwise. The most dominant pieces of work on unsupervised morpheme segmentation, Morfessor (Creutz and Lagus, 2002, 2005, 2007) and Linguistica (Goldsmith, 2000) adopt the Minimum Description Length (MDL) principle (Rissanen, 1998). In other words, they aim to minimize describing the lexicon of morphs as well as minimizing the description of an input corpus. Morfessor has a widely used API and has inspired a large body of work (Kohonen et al., 2010; Grönroos et al., 2014).
The unsupervised original implementation was later adapted (Kohonen et al., 2010; Grönroos et al., 2014) to allow for minimal supervision. Another work on minimally supervised morpheme segmentation is (Sirts and Goldwater, 2013) which relies on Adaptor Grammars (AGs) (Johnson et al., 2006). AGs learn latent tree structures over an input corpus using a nonparametric bayesian model (Sirts and Goldwater, 2013).
(Lafferty et al., 2001) use Conditional Random Fields (CRF) for morpheme segmentation. In this supervised method, the morpheme segmentation task is modeled as a sequence-to-sequence learning problem, whereby the sequence of labels defines the boundaries of morphemes (Ruokolainen et al., 2013, 2014). In contrast to the previously mentioned generative approaches of MDL and AG, this method takes a discriminative approach and allows for the inclusion of a larger set of features. In this approach, CRF learns a conditional probability of a segmentation given a word (Ruokolainen et al., 2013, 2014).
All the previously mentioned morpheme segmenters rely solely on orthographic features of morphemes. Semantics were initially introduced to morpheme segmenters in the work of (Schone and Jurafsky, 2000). They use LSA to generate word representations and then evaluate if two words are morphologically related based on semantic relatedness, as well as deterministic orthographic methods. Similarly, (Baroni et al., 2002) use edit distance and mutual information as metrics for semantic and orthographic validity of a morphological relation between two words. Recent work in (Narasimhan et al., 2015), inspired by the log-linear model in (Poon et al., 2009) incorporates semantic relatedness into the model via word representations. Other systems such as (Üstün and Can, 2016) rely solely on evaluating two words from a semantic standpoint by the use of a twolayer neural network.
Similarly, MORSE introduces semantic information into its morpheme segmenters via distributed word representations while also relying on orthographic features. Inspired by the work of (Soricut and Och, 2015), instead of merely evaluating semantic relatedness, we are the first to evaluate the morphological relationship via the difference vector of morphologically related words. Comparing the difference vectors of multiple pairs across the corpus following the same morpho-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
logical relation, gives MORSE a vocabulary-wide evaluation of morphological relations learned.
3 System
The key limitation of previous frameworks that rely solely on orthographic features is the resulting over-segmentation. As an example, MDLbased frameworks segment “sing” to “s-ing” due to the high frequency of the morphemes: “s” and “ing”. Our framework combines semantic relatedness with orthographic relatedness to eliminate such error. For the example mentioned, MORSE validates morphemes such as “s” and “ing” from an orthographic perspective, yet invalidates the relation between “s” and “sing” from a local and vocabulary-wide semantic perspective. Hence, MORSE will segment “jumping” as “jump-ing”, and perform no segmentations on “sing”.
To bring in semantic understanding into MORSE, we rely on word representations (Mikolov et al., 2013a; Pennington et al., 2014). These word representations capture the semantics of the vocabulary through statistics over the context in which they appear. Moreover, morphosyntactic regularities have been shown over these word representations, whereby pairs of words sharing the same relationship exhibit equivalent difference vectors (Mikolov et al., 2013b). For example, it is expected in the vector space of word representations that ~wjumping ´ ~wjump « ~wplaying ´ ~wplay, but ~wsing ´ ~ws ff ~wplaying ´ ~wplay.
As a high level description, we first learn all possible affix transformations (morphological rules) in the language from pairs of words from an orthographic standpoint. For example, the pair (“jump”, “jumping”) corresponds to the valid affix transformation φ suffixÝÝÝÑ “ing”, and the pair (“slow”, “slogan”) corresponds to the invalid rule “w” suffixÝÝÝÑ “gan”. Then we invalidate the rules, such as “w” suffixÝÝÝÑ “gan”, that do not conform to the linear relation in the vector space. We also invalidate pairs of words which, due to randomness, are orthographically related via a valid rule although they are not morphologically related, such as (“on”, “only”).
Now we formalize the objects we learn in MORSE and the scores (orthographic and semantic) used for validation. This constitutes the training stage. Finally, we formalize the inference stage, where we use these objects and scores to perform morpheme segmentation.
3.1 Training Stage
Objects:
• Rule set R made of all possible affix transformations in a language. R is populated via the following definition: Rsuffix = {aff1
suffixÝÝÝÑ aff2: D (w1, w2) P V2, stem(w1) = stem(w2), w1 = stem(w1) + aff1, w2 = stem(w2) + aff2}, Rprefix is defined similarly for prefixes, and R = Rsuffix Y Rprefix. An example R would be equal to {φ suffixÝÝÝÑ “ly”, φ prefixÝÝÝÑ “un”, “ing” suffixÝÝÝÑ “ed”, ...}.
• Support set SSr for a rule r P R is made of all pairs of words related via r on a surface level. SSr is populated via the following definition: SSr = {(w1, w2): w1, w2 P V, w1
rÝÑ w2}. An example support set of the rule “ing” suffixÝÝÝÑ “ed” would be {(“playing”, “played”), (“crafting”, “crafted”), ...}.
Scores:
• scorer orth(r) is a vocabulary-wide orthographic confidence score for rule r P R. It reflects the validity of an affix transformation in a language from an orthographic perspective. This score is evaluated as scorer orth(r) = |SSr|. • scorer sem(r) is a vocabulary-wide seman-
tic confidence score for rule r P R. It reflects the validity of an affix transformation in a language from a semantic perspective. This score is evaluated as, scorer sem(r) = |clusterr|/|SSr|2 where clusterr = {((w1, w2), (w3, w4)): (w1, w2), (w3, w4) P SSr, ~w1 ´ ~w2 « ~w3 ´ ~w4 }. We consider ~w1 ´ ~w2 « ~w3 ´ ~w4 if cos(~w4, ~w2 ´ ~w1 ` ~w3) ą 0.1. • scorew sem((w1, w2) P SSr) is a vocabularywide semantic confidence score for a pair of words (w1, w2). This pair of words is related via r on an orthographic level, but this score reflects the validity of the morphological relation via r on a semantic level. This score is evaluated as, scorew sem((w1, w2) P SSr) = |{(w3, w4): (w3, w4) P SSr, ~w1 ´ ~w2 « ~w3´ ~w4}|/|SSr|. In other words, it is the fraction of pairs of words in the support set that exhibit a similar linear relation as (w1, w2) in the vector space.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
• scoreloc sem((w1, w2) P SSr) is a local semantic confidence score for a pair of words (w1, w2). This pair of words is related via r on an orthographic level, but this score reflects the semantic relatedness between the pair. This is evaluated as, scoreloc sem((w1, w2) P SSr) = cos(~w1, ~w2).
3.2 Inference Stage
In this stage we perform morpheme segmentation using the knowledge gained from the first stage. We first introduce some notation. Let Radd = {r : r P R, r = aff1
rÝÑ aff2, aff1 = φ, aff2 ‰ φ }, Rrep = {r : r P R, r = aff1
rÝÑ aff2, aff1 ‰ φ, aff2 ‰ φ }. In other words, we divide the rules to those where an affix is added (Radd) and to those where an affix is replaced (Rrep).
Now given a word w to segment, we search for r˚, the solution to the following optimization problem1. The search space is limited to the rules that include w in their support set, and thus it’s a limited search space and the problem is tractable:
max r
ÿ
t1
scoret1ppw1, wq P SSrq ` ÿ
t2
scoret2prq
s. t. r P Radd scorer semprq ą tr sem scorer orthprq ą tr orth scorew semppw1, wq P SSrq ą tw sem scoreloc semppw1, wq P SSrq ą tloc sem
Where t1 = {w sem, loc sem}, t2 = {r sem, r orth}, and tr sem, tr orth, tw sem, tloc sem are hyperparameters of the system. Now given r˚ = φ suffixÝÝÝÑ suf, w1 is defined as w1 r ˚ ÝÑ w. Thus the system segments wÑ w1-suf. We treat prefixes similarly. Then the system iterates over w1. Figure 1 shows the segmentation process of the word “unhealthy” based on the sequentially retrieved r*.
The reason we restrict our rule set to Radd in the optimization problem is to avoid rules such as “er” suffixÝÝÝÑ “ing” like in (“player”, “playing”) leading to false segmentations such as “playing” Ñ “playering”. Yet we cannot completely restrict our search to Radd due to rules such as “y” Ñ “ies” in words like (“sky”, “skies”). To be able to segment words such as “skies”, we’d have to consider rules in Rrep but only after searching in Radd. Thus if the first
1r and w uniquely identify w1, and thus the search space is defined only over r.
optimization problem was unfeasible, we repeat it while replacing Radd with Rrep. The program terminates when both optimization problems are unfeasible.
4 Experiments
We describe in this section the experiments done to assess the performance of MORSE. First, the performance is assessed intrinsically on the task of morpheme segmentation and against the most widely used morpheme segmenter: Morfessor. To evaluate the language agnosticity of the algorithm, we perform evaluation across three languages of varying morphology levels: English, Turkish, Finnish, with Finnish being the richest in morphology and English being the poorest. Second, we show the inadequacies of benchmarking gold datasets for this task and describe a new dataset that we create to address the inadequacy. Third, in order to highlight the effect of including semantic information, we compare MORSE against Morfessor on a set of words which should not be segmented from a semantic perspective although orthographically they seem to be segmentable (such as “freshman”).
In all of our experiments (unless specified otherwise), we report precision and recall (and consequently F1 scores) with locations of morpheme boundaries being considered positives and the rest of the locations considered negatives. It should be noted that we disregard starting and ending positions of words for being trivial boundaries (Virpioja et al., 2011).
4.1 Setup
Both systems, Morfessor and MORSE, were trained on the same monolingual corpus: Wikipedia2 (as of September 20, 2016) to control for affecting factors within the experiment. For each language considered, the respective Wikipedia dump was preprocessed using an available code3. We use Word2Vec (Mikolov et al., 2013a) to train word representations of
2https://dumps.wikimedia.org 3https://github.com/bwbaugh/wikipedia-extractor
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Dataset En Fi Tr Tuning Data 1000 1000 971
Test Data 686 760 809
Table 1: Morpho Challenge 2010 Dataset Sizes
300 dimensions and based on a context window of size 5. Also, for computational efficiency, MORSE was limited to a vocabulary of size 1M, a restriction not enforced on Morfessor.
MORSE’s hyperparameters are tuned based on a tuning set of gold morpheme segmentations. We release the source code of MORSE to the public4.
4.2 Morpho Challenge Dataset
For our first intrinsic experiment we resort to the Morpho Challenge (MC) gold segmentations available online5. For every language, two datasets are supplied: training and development. For the purpose of our experiments, all systems use the development dataset as a test dataset, and the training dataset is used for tuning MORSE’s hyperparameters. MC dataset sizes are reported in Table 1.
4.3 Semantically Driven Dataset
In this section we pinpoint the weaknesses of the MC dataset in assessing the relative performance of different morpheme segmenters. Consequently, we introduce a new semantically driven dataset (SD17) for morpheme segmentation along with the methodology used for creation. We make it publicly available in the canonical6 and noncanonical7 version (Cotterell and Vieira, 2016). Non-compositional segmentation: One of the key requirements of morpheme segmentation is the compositionality of the meaning of the word from the meaning of its morphemes. This requirement is violated on multiple occasions in the MC dataset. One example from Table 2 is segmenting the word “business” into “busi-ness”, which falsely assumes that “business” means the act of being busy. Such wrongly segmented words might have been truly derived from its components initially, but having undergone radical semantic change over time, they no longer semantically represent the compositionality of their components (Wijaya and Yeniterzi, 2011). Not only
4http://dropproxy.com/f/F94 5http://research.ics.aalto.fi/events/morphochallenge2010 6http://dropproxy.com/f/F92 7http://dropproxy.com/f/F93
does such a weakness contribute to false segmentations, but also favors segmentation methods following the MDL principle. Trivial instances: The second weakness in the MC dataset is due to abundance of trivial instances. These instances lack discriminating capability since all methods can easily predict them (Baker, 2001). These instances are comprised of genetive cases (such as teacher’s) as well as hyphenated words (such as turning-point). For genetive cases, segmenting at the apostrophe leads to perfect precision and recall, and thus such instances are deemed trivial. In the case of hyphenated words, segmenting at the hyphen is a correct segmentation with a very high probability. In the MC tuning dataset, in 43 times out of 46, the hyphen was a correct indication of segmentation. Other issues exist in the Morpho Challenge dataset although less abundant. There are instances of wrong segmentations possibly due to human error. One example of such instance is “turning-point” segmented to “turning - point” instead of “turn ing - point”. Another issue, which is hard to avoid, is ambiguity of segmentation boundaries. Take for example the word “strafed”, the segmentations “straf-ed” and “strafe-d” are equally justified. In such situations, the MC dataset favors complete affixes rather than complete lemmas. This also favors MDL-based segmenters. We note that the MC dataset also provides segmentations in a canonical version such as “strafe-ed”, yet for the sake of a fair comparison with Morfessor and all previously evaluated systems on the MC dataset, we consider only the former version of segmentations.
For the reasons mentioned above, we decide to create a new dataset for English gold morpheme segmentations with compositionality guiding the annotations. We select 2000 words randomly from the 10K most frequent words in the
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
English Turkish Finnish P R F1 P R F1 P R F1
Morfessor 74.46 56.66 64.35 40.81 25.00 31.01 43.09 28.16 34.06 MORSE 81.98 61.57 70.32 49.90 30.78 38.07 36.26 9.44 14.98
Table 3: Performance of MORSE on the MC dataset across three languages: English, Turkish, Finnish
English Wikipedia dump and have them annotated by two proficient English speakers. The segmentation criterion was to segment the word to the largest extent possible while preserving its compositionality from the segments. The inter-annotator agreement reached 91% on a word level. Based on post annotation discussions, annotators agreed on 99% of the words, and words not agreed on were eliminated along with words containing non-alpha characters to avoid trivial instances.
SD17 is used to evaluate the performance of Morfessor and MORSE. We claim that the performance on SD17 is a better indication of the performance of a morpheme segmenter. By the use of SD17 we expect to gain insights on the extent to which morpheme segmentation is a function of semantics in addition to orthography.
4.4 Handling Compositionality
One of the main claims of this paper is stating that following the MDL principle (such as Morfessor) will lead to over-segmentation. This oversegmentation happens specifically when the meaning of the word does not follow from the meaning of its morphemes. Examples include words such as “red head”, “duck face”, “how ever”, “s ing”. A subset of these words are defined by linguists as exocentric compounds (Bauer, 2008). MORSE does not suffer from this issue owing to its use of a semantic model.
We use a collection of 100 English words which appear to be segmentable but actually are not (example “however”). Such a collection will highlight a system’s capability of distinguishing frequent letter sequences from the semantic contribution of this letter sequence in a word. We make this collection publicly available8.
5 Results
We compare MORSE with Morfessor, and place the performance alongside the state-of-the-art published results.
8http://dropproxy.com/f/F95
En Tr Fi Candidate Rules 27.5M 14.9M 10.8M Candidate Rel. Pairs 53.3M 25.1M 18.6M
Table 4: Number of candidate rules and candidate related word pairs detected per language
5.1 Morpho Challenge Dataset
As demonstrated in Table 3, MORSE performs better than Mofessor on English and Turkish, and worse on Finnish. Considering English first, using MORSE instead of Morfessor, resulted in a 6% absolute increase in F1 scores. This supports our claim for the need of semantic cues in morpheme segmentation, and also validates the method used in this paper. With English being considered a less systematic language in terms of the orthographic structure of words, semantic cues are of larger need, and hence a system which relies on semantic cues is expected to perform better. Similarly, MORSE performs better on Turkish with a 7% absolute margin in terms of F1 score. On the other hand, Morfessor surpasses MORSE in performance on Finnish by a large margin as well, especially in terms of recall.
5.1.1 Discussion
We hypothesize that the richness of morphology in Finnish led to suboptimal performance of MORSE. This is because richness in morphology leads to word level sparsity which directly leads to: (1) Degradation of quality of word representations (2) Increased vocabulary size exacerbating the issue of limited vocabulary (recall MORSE was limited to a vocabulary of 1M). In a language with productive morphology, limiting its vocabulary results in a lower chance of finding morphologically related word pairs. This negatively impacts the training stage of MORSE which relies on the availability of such pairs. In order to detect the suffix “ly” from the word “cheerfully” MORSE needs to come across “cheerful” as well. Coming across “cheerful” is now a lower probability event
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
655
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
698
699
due to high sparsity. This is not as much of an issue for Morfessor under the MDL principle, since it might detect “ly” just by coming across multiple words ending with “ly” even without encountering the base forms of those words. We show how the detection of rules is affected by considering the number of candidate rules detected as well as the number of candidate morphologically related word pairs detected. As shown in Table 4, the number of detected candidate rules and candidate related words decreases with the increase in morphology in a language. This confirms our hypothesis.
This issue can be directly attributed to the limited vocabulary size in MORSE. With the increase in processing power, and thus larger vocabulary coverage, MORSE is expected to perform better.
5.2 Semantically Driven Dataset
The performance of MORSE and Morfessor on SD17 is shown in Table 5. The use of MC data (which does not adhere to the compositionality principle) to tune MORSE to be evaluated on SD17 (which does adhere to the compositionality principle) is not optimal. Thus, we evaluate MORSE on SD17 using 5-fold cross validation, where 80% of the dataset is used to tune and 20% is used to evaluate. Precision, Recall, and F1 scores are averaged and reported in Table 5 using code name MORSE-CV.
Based on the results in Table 5, we make the following observations. Comparing MORSE-CV to MORSE reflects the fundamental difference between SD17 and MC datasets. Knowing the basis of construction of SD17 and the fundamental weaknesses in MC datasets, we attribute the performance increase to the lack of compositionality in MC dataset. Comparing MORSE-CV to Morfessor, we observe a significant jump in performance (an increase of 24%). In comparison, the increase on the MC dataset (6%) shows that the Morpho Challenge dataset underestimates the performance gap between Morfessor and MORSE due its inherent weaknesses.
Since MORSE is equipped with the capability to retrieve full morphemes even when not present in full orthographically, a capability that Morfessor lacks, we evaluated both systems on the canonical version of SD17. The results are reported in Table 6. We notice that evaluating on the canonical form of SD17 gives a further edge for MORSE
over Morfessor. For evaluation on the canonical version of SD17, we switch to morpheme-level evaluation instead of boundary-level as a more suitable method for Morfessor.
We next compare MORSE against published state-of-the-art results . As one can see in Table 7 MORSE significantly performs better than published state-of-the-art results most notably (Narasimhan et al., 2015) referred to as LLSM in the Table. Comparison is also made against the top results in the latest Morpho Challenge: Morfessor S+W and Morfessor S+W+L (Kohonen et al., 2010), and Base Inference (Lignos, 2010).
5.3 Handling Compositionality
We compare the performance of MORSE and Morfessor on a set of words made up of morphemes which don’t compose the meaning of the word. Since all the boundaries in this dataset are negative, to evaluate both systems (with MORSE tuned on SD17), we only report the number of segments generated on this dataset. The more segments a system generates, the worse is its performance.
We find that MORSE generates 7 false morphemes whereas Morfessor generates 43 false
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Figure 2: Precision (left) and Recall (right) of MORSE as a function of the hyperparameters: tr sem, tw sem
morphemes. This shows MORSE’s robustness to such examples through its semantic knowledge and validates our claim that Morfessor oversegments on such examples.
6 Discussion
One of the benefits of MORSE against other frameworks such as MDL is its ability to identify the lemma within the segmentation. The lemma would be the last non-segmented word in the iterative process of segmentation. Hence, an advantage of our framework is its easy adaptability into a lemmatizer and even a stemmer.
Another key aspect which is not present in some of the competitive systems is the need for a small dataset for hyperparameter tuning. This is a point in favor of completely unsupervised systems such as Morfessor. On the other hand, these hyperparameters could allow for flexibility. Figure 2 shows how precision and recall changes as a function of the hyperparameter selection9. As one would expect, increasing the hyperparameters, in general, leads to a stricter search space and thus increases precision and decreases recall. Putting these results in perspective, the user of MORSE is given the capability of controlling for precision and recall based on the needs of the downstream task.
Moreover, to check for the level of dependency of MORSE on a set of gold morpheme segmentations for tuning, we check for the variation in performance with respect to size of tuning data. For the purpose of this experiment we take an 80- 20 split of SD17 and vary the size of the tuning set. We notice that the performance (81.90% F1)
9Only a subset of the hyperparameters is used for display purposes
reaches a steady state at 20% (« 300 gold segmentations) of the tuning data. This reflects the minimal dependency on gold morpheme segmentations.
As for the inference stage of MORSE, the greedy inference approach limits its performance. In other words, a wrong segmentation at the beginning will propagate and result in consequent wrong segmentations. Also, MORSE’s limitation to concatenative morphology decreases its efficacy on languages that include non-concatenative morphology. This opens the stage for further research on a more optimal inference stage and a more global modeling of orthographic morphological transformations.
7 Conclusions and Future Work
In this paper, we have presented MORSE, a first morpheme segmenter to consider semantic structure at this scale (local and vocabulary-wide). We show its superiority over state-of-the-art algorithms using intrinsic evaluation on a variety of languages. We also pinpointed the weaknesses in current benchmarking datasets, and presented a new dataset free of these weaknesses. With a relative increase in performance reaching 24% absolute increase over Morfessor, this work proves the significance of semantic cues as well as validates a new state-of-the-art morpheme segmenter. For future work, we plan to address the limitations of MORSE: minimal supervision, greedy inference, and concatenative orthographic model. Moreover, we plan to computationally optimize the training stage for the sake of wider adoption by the community.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper continues the line of work for applying word embeddings for the problem of unsupervised morphological segmentation (e.g. Soricut & Och, 2015;  u00dcst u00fcn & Can, 2016). The proposed method, MORSE, applies a local optimization for segmentation of each word, based on a set of orthographic and semantic rules and a few heuristic threshold values associated with them.  - Strengths:  The paper presents multiple ways to evaluate segmentation hypothesis on word embeddings, and these may be useful also in other type of methods. The results on English and Turkish data sets are convincing.  The paper is clearly written and organized, and the biliography is extensive.  The submission includes software for testing the English MORSE model and three small data sets used in the expriments.  - Weaknesses:  The ideas in the paper are quite incremental, based mostly on the work by Soricut & Och (2015). However, the main problems of the paper concern meaningful comparison to prior work and analysis of the method's limitations.  First, the proposed method does not provide any sensible way for segmenting compounds. Based on Section 5.3, the method does segment some of the compounds, but using the terminology of the method, it considers either of the constituents as an affix. Unsuprisingly, the limitation shows up especially in the results of a highly-compounding language, Finnish. While the limitation is indicated in the end of the discussion section, the introduction and experiments seem to assume otherwise.  In particular, the limitation on modeling compounds makes the evaluation of Section 4.4 5.3 quite unfair: Morfessor is especially good at segmenting compounds (Ruokolainen et al., 2014), while MORSE seems to segment them only  by accident . Thus it is no wonder that Morfessor segments much larger proportion of the semantically non-compositional compounds. A fair experiment would include an equal number of compounds that _should_ be segmented to their constituents.  Another problem in the evaluations (in 4.2 and 4.3) concerns hyperparameter tuning. The hyperparameters of MORSE are optimized on a tuning data, but apparently the hyperparameters of Morfessor are not. The recent versions of Morfessor (Kohonen et al. 2010, Gr u00f6nroos et al. 2014) have a single hyperparameter that can be used to balance precision and recall of the segmentation. Given that the MORSE outperforms Morfessor both in precision and recall in many cases, this does not affect the conclusions, but should at least be mentioned.  Some important details of the evaluations and results are missing: The  morpheme-level evaluation  method in 5.2 should be described or referred to. Moreover, Table 7 seems to compare results from different evaluation sets: the Morfessor and Base Inference methods seem to be from official Morpho Challenge evaluations, LLSM is from Narasimhan et al. (2015), who uses aggregated data from Morpho Challenges (probably including both development and training sets), and MORSE is evaluated Morpho Challenges 2010 development set. This might not affect the conclusions, as the differences in the scores are rather large, but it should definitely be mentioned.  The software package does not seem to support training, only testing an included model for English.  - General Discussion:  The paper puts a quite lot of focus on the issue of segmenting semantically non-compositional compounds. This is problematic in two ways: First, as mentioned above, the proposed method does not seem to provide sensible way of segmenting _any_ compound. Second, finding the level of lexicalized base forms (e.g. freshman) and the morphemes as smallest meaning-bearing units (fresh, man) are two different tasks with different use cases (for example, the former would be more sensible for phrase-based SMT and the latter for ASR). The unsupervised segmentation methods, such as Morfessor, typically target at the latter, and critizing the method for a different goal is confusing.  Finally, there is certainly a continuum on the (semantic) compositionality of the compound, and the decision is always somewhat arbitrary. (Unfortunately many gold standards, including the Morpho Challenge data sets, tend to be also inconsistent with their decisions.)  Sections 4.1 and 5.1 mention the computational efficiency and limitation to one million input word forms, but does not provide any details: What is the bottleneck here? Collecting the transformations, support sets, and clusters? Or the actual optimization problem? What were the computation times and how do these scale up?  The discussion mentions a few benefits of the MORSE approach: Adaptability as a stemmer, ability to control precision and recall, and need for only a small number of gold standard segmentations for tuning. As far as I can see, all or some of these are true also for many of the Morfessor variants (Creutz and Lagus, 2005; Kohonen et al., 2010; Gr u00f6nroos et al., 2014), so this is a bit misleading. It is true that Morfessor works usually fine as a completely unsupervised method, but the extensions provide at least as much flexibility as MORSE has.  (Ref: Mathias Creutz and Krista Lagus. 2005. Inducing the Morphological Lexicon of a Natural Language from Unannotated Text. In Proceedings of the International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning (AKRR'05), Espoo, Finland, June 15-17.)  - Miscellaneous:  Abstract should maybe mention that this is a minimally supervised method (unsupervised to the typical extent, i.e. excluding hyperparameter tuning).  In section 3, it should be mentioned somewhere that phi is an empty string.  In section 5, it should be mentioned what specific variant (and implementation) of Morfessor is applied in the experiments.  In the end of section 5.2, I doubt that increasing the size of the input vocabulary would alone improve the performance of the method for Finnish. For a language that is morphologically as complex, you never encounter even all the possible inflections of the word forms in the data, not to mention derivations and compounds.  I would encourage improving the format of the data sets (e.g.  using something similar to the MC data sets): For example using  aa  as a separator for multiple analyses is confusing and makes it impossible to use the format for other languages.  In the references, many proper nouns and abbreviations in titles are written in lowercase letters. Narasimhan et al. (2015) is missing all the publication details.


Semi-supervised Multitask Learning for Sequence Labeling
We propose a sequence labeling framework with a secondary training objective, learning to predict surrounding words for every word in the dataset. This language modeling objective incentivises the framework to learn general-purpose patterns of semantic and syntactic composition, which are also useful for improving accuracy on different sequence labeling tasks. The architecture was evaluated on 8 datasets, covering the tasks of error detection in learner texts, named entity recognition, chunking and POS-tagging. The novel language modeling objective provided consistent performance improvements on every benchmark, without requiring any additional annotated or unannotated data.
1 Introduction
Accurate and efficient sequence labeling models have a wide range of applications, including named entity recognition (NER), part-of-speech (POS) tagging, error detection and shallow parsing. Specialised approaches to sequence labeling often include extensive feature engineering, such as integrated gazetteers, capitalisation features, morphological information and POS tags. However, recent work has shown that neural network architectures are able to achieve comparable or improved performance, while automatically discovering useful features for a specific task and only requiring a sequence of tokens as input (Collobert et al., 2011; Irsoy and Cardie, 2014; Lample et al., 2016).
This feature discovery is usually driven by an objective function based on predicting the annotated labels for each word, without much incentive
to learn more general language features from the available text. In many sequence labeling tasks, the relevant labels in the dataset are very sparse and most of the words contribute very little to the training process. For example, in the CoNLL 2003 NER dataset (Tjong Kim Sang and De Meulder, 2003) only 17% of the tokens represent an entity. This ratio is even lower for error detection, with only 14% of all tokens being annotated as an error in the FCE dataset (Yannakoudakis et al., 2011). The sequence labeling models are able to learn this bias in the label distribution without obtaining much additional information from words that have the majority label (O for outside of an entity; C for correct word). Therefore, we propose an additional training objective which allows the models to make more extensive use of the available data.
The task of language modeling offers an easily accessible objective – learning to predict the next word in the sequence requires only plain text as input, without relying on any particular annotation. Neural language modeling architectures also have many similarities to common sequence labeling frameworks: words are first mapped to distributed embeddings, followed by a recurrent neural network (RNN) module for composing word sequences into an informative context representation (Mikolov et al., 2010; Graves et al., 2013; Chelba et al., 2013). Compared to any sequence labeling dataset, the task of language modeling has a considerably larger and more varied set of possible options to predict, making better use of each available word and encouraging the model to learn more general language features for accurate composition.
In this paper, we propose a neural sequence labeling architecture that is also optimised as a language model, predicting surrounding words in the dataset in addition to assigning labels to each token. Specific sections of the network are op-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
timised as a forward- or backward-moving language model, while the label predictions are performed using context from both directions. This secondary unsupervised objective encourages the framework to learn richer features for semantic composition without requiring additional training data. We evaluate the sequence labeling model on 8 datasets from the fields of NER, POS-tagging, chunking and error detection in learner texts. Our experiments show that by including the unsupervised objective into the training process, the sequence labeling model achieves consistent performance improvements on all the benchmarks. This multitask training framework gives the largest improvements on error detection datasets, where the system also achieves new state-of-the-art results.
2 Neural Sequence Labeling
We use the neural network model by Rei et al. (2016) as the baseline architecture for our sequence labeling experiments. The model takes as input one sentence, separated into tokens, and assigns a label to every token using a bidirectional LSTM.
The input tokens are first mapped to a sequence of distributed word embeddings [x1, x2, x3, ..., xT ]. Two LSTM (Hochreiter and Schmidhuber, 1997) components, moving in opposite directions through the sentence, are then used for constructing context-dependent representations for every word. Each LSTM takes as input the hidden state from the previous time step, along with the word embedding from the current step, and outputs a new hidden state. The hidden representations from both directions are concatenated, in order to obtain a context-specific representation for each word that is conditioned on the whole sentence in both directions:
−→ ht = LSTM(xt, −−→ ht−1) (1)
←− ht = LSTM(xt, ←−− ht+1) (2)
ht = [ −→ ht ; ←− ht ] (3)
Next, the concatenated representation is passed through a feedforward layer, mapping the components into a joint space and allowing the model to learn features based on both context directions:
dt = tanh(Wdht) (4)
where Wd is a weight matrix and tanh is used as the non-linear activation function.
In order to predict a label for each token, we use either a softmax or CRF output architecture. For softmax, the model directly predicts a normalised distribution over all possible labels for every word, conditioned on the vector dt:
P (yt|dt) = softmax(Wodt)
= eWo,kdt∑
k̃∈K e Wo,k̃dt
(5)
where K is the set of all possible labels, and Wo,k is the k-th row of output weight matrix Wo. The model is optimised by minimising categorical crossentropy, which is equivalent to minimising the negative log-probability of the correct labels:
E = − T∑ t=1 log(P (yt|dt)) (6)
While this architecture returns predictions based on all words in the input, the labels are still predicted independently. For some tasks, such as named entity recognition with a BIO1 scheme, there are strong dependencies between subsequent labels and it can be beneficial to explicitly model these connections. The output of the architecture can be modified to include a Conditional Random Field (CRF, Lafferty et al. (2001)), which allows the network to look for the most optimal path through all possible label sequences (Huang et al., 2015). The model is then optimised by maximising the score for the correct label sequence, while minimising the scores for all other sequences:
E = −s(y) + log ∑ ỹ∈Ỹ es(ỹ) (7)
where s(y) is the score for a given sequence y and Y is the set of all possible label sequences.
We also make use of the character-level component described by Rei et al. (2016), which builds an alternative representation for each word. The individual characters of a word are mapped to character embeddings and passed through a bidirectional LSTM. The last hidden states from both direction are concatenated and passed through a
1Each NER entity has sub-tags for Beginning, Inside and Outside
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
h2
x2
d2
o2
h2
proposes
m2
q2
m2
q2 OFischler measures
h3
x3
d3
o3
h3
measures
m3
q3
m3
q3 Oproposes </s>
h1
x1
d1
o1
h1
Fischler
m1
q1
m1
q1 PER<s> proposes
Figure 1: The unfolded network structure for a sequence labeling model with an additional language modeling objective, performing NER on the sentence ”Fischler proposes measures”. The input tokens are shown at the bottom, the expected output labels are at the top. Arrows above variables indicate the directionality of the component (forward or backward).
nonlinear layer. The resulting vector representation is combined with a regular word embedding using a dynamic weighting mechanism that adaptively controls the balance between word-level and character-level features. This framework allows the model to learn character-based patterns and handle previously unseen words, while still taking full advantage of the word embeddings.
3 Language Modeling Objective
The sequence labeling model in Section 2 is only optimised based on the correct labels. While each token in the input does have a desired label, many of these contribute very little to the training process. For example, in the CoNLL 2003 NER dataset (Tjong Kim Sang and De Meulder, 2003) there are only 8 possible labels and 83% of the tokens have the label O, indicating that no named entity is detected. This ratio is even higher for error detection, with 86% of all tokens containing no errors in the FCE dataset (Yannakoudakis et al., 2011). The sequence labeling models are able to learn this bias in the label distribution without obtaining much additional information from the majority labels. Therefore, we propose an additional objective which would allow the models to make full use of the training data.
In addition to learning to predict labels for each word, we propose optimising specific sections of the architecture as language models. The task of predicting the next word will require the model to learn more general patterns of semantic and syntactic composition, which can then be reused in order to predict individual labels more accurately. This objective is also generalisable to any
sequence labeling task and dataset, as it requires no additional annotated training data.
A straightforward modification of the sequence labeling model would add a second parallel output layer for each token, optimising it to predict the next word. However, the model has access to the full context on each side of the target token, and predicting information that is already explicit in the input would not incentivise the model to learn about composition and semantics. Therefore, we must design the loss objective so that only sections of the model that have not yet observed the next word are optimised to perform the prediction. We solve this by predicting the next word in the sequence only based on the hidden representation−→ ht from the forward-moving LSTM. Similarly, the previous word in the sequence is predicted based on ←− ht from the backward-moving LSTM. This architecture avoids the problem of giving the correct answer as an input to the language modeling component, while the full framework is still optimised to predict labels based on the whole sentence.
First, the hidden representations from forwardand backward-LSTMs are mapped to a new space using a non-linear layer:
−→mt = tanh( −→ Wm −→ ht) (8)
←−mt = tanh( ←− Wm ←− ht) (9)
where −→ Wm and ←− Wm are weight matrices. This separate transformation learns to extract features that are specific to language modeling, while the LSTM is optimised for both objectives. We also use the opportunity to map the representation to a smaller size – since language modeling is not the
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
main goal, we restrict the number of parameters available for this component, forcing the model to generalise more using fewer resources.
These representations are then used to predict the preceding and following word, by using a softmax layer with the size of the full vocabulary:
P (wt+1|−→mt) = softmax( −→ W q −→mt) (10)
P (wt−1|←−mt) = softmax( ←− W q ←−mt) (11)
The objective function for both components is then constructed as a regular language modeling objective, by calculating the negative loglikelihood of the next word in the sequence:
−→ E = − T−1∑ t=1 log(P (wt+1|−→mt)) (12)
←− E = − T∑ t=2 log(P (wt−1|←−mt)) (13)
Finally, these additional objectives are combined with the training objective E from either Equation 6 or 7, resulting in a new cost function Ẽ for the sequence labeling model:
Ẽ = E + γ( −→ E + ←− E ) (14)
where γ is a parameter that is used to control the importance of the language modeling objective in comparison to the sequence labeling objective.
Figure 1 shows a diagram of the unfolded neural architecture, when performing NER on a short sentence with 3 words. At each token position, the network is optimised to predict the previous word, the current label, and the next word in the sequence. The added language modeling objective encourages the system to learn richer feature representations that are then reused for sequence labeling. For example, −→ h1 is optimised to predict proposes as the next word, indicating that the current word is a subject, possibly a named entity. In addition, ←− h2 is optimised to predict Fischler as the previous word and these features are used as input to predict the PER tag at o1.
The proposed architecture introduces 4 additional parameter matrices that are optimised during training: −→ Wm, ←− Wm, −→ W q, and ←− W q. However, the computational complexity and resource
requirements of this model during sequence labeling are equal to the baseline from Section 2, since the language modeling components are ignored during testing and these additional weight matrices are not used. While our implementation uses a basic softmax as the output layer for the language modeling components, the efficiency during training could be further improved by integrating noise-contrastive estimation (NCE, Mnih and Teh (2012)) or hierarchical softmax (Morin and Bengio, 2005).
4 Evaluation Setup
The proposed architecture was evaluated on 8 different sequence labeling datasets, covering the tasks of error detection, NER, chunking, and POStagging. The word embeddings in the model were initialised with publicly available pretrained vectors, created using word2vec (Mikolov et al., 2013). For general-domain datasets we used 300-dimensional embeddings trained on Google News.2 For biomedical datasets, the word embeddings were initialised with 200-dimensional vectors trained on PubMed and PMC.3
The neural network framework was implemented using Theano (Al-Rfou et al., 2016) and we make the code publicly available online.4 For most of the hyperparameters, we follow the settings by Rei et al. (2016) in order to facilitate direct comparison with previous work. The LSTM hidden layers are set to size 200 in each direction for both word- and character-level components. All digits in the text were replaced with the character 0; any words that occurred only once in the training data were replaced by an OOV token. Sentences were grouped into batches of size 64 and parameters were optimised using AdaDelta (Zeiler, 2012) with default learning rate 1.0. Training was stopped when performance on the development set had not improved for 7 epochs. Performance on the development set was also used to select the best model, which was then evaluated on the test set. In order to avoid any outlier results due to randomness in the model initialisation, each configuration was trained with 10 different random seeds and the averaged results are presented in this paper. We use previously established splits for training, development and testing
2https://code.google.com/archive/p/word2vec/ 3http://bio.nlplab.org/ 4URL removed for anonymisation
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
FCE DEV FCE TEST CoNLL-14 TEST1 CoNLL-14 TEST2 F0.5 P R F0.5 P R F0.5 P R F0.5
Baseline 48.78 55.38 25.34 44.56 15.65 16.80 15.80 25.22 19.25 23.62 + dropout 48.68 54.11 23.33 42.65 14.29 17.13 14.71 22.79 19.42 21.91 + LMcost 53.17 58.88 28.92 48.48 17.68 19.07 17.86 27.62 21.18 25.88
Table 1: Precision, Recall and F0.5 score of alternative sequence labeling architectures on error detection datasets. Dropout and LMcost modifications are added incrementally to the baseline.
on each of these datasets. Based on development experiments, we found that error detection was the only task that did not benefit from having a CRF module at the output layer – since the labels are very sparse and the dataset contains only 2 possible labels, explicitly modeling state transitions does not improve performance on this task. Therefore, we use a softmax output for error detection experiments and CRF on all other datasets.
The publicly available sequence labeling system by Rei et al. (2016) is used as the baseline.5 During development we found that applying dropout (Srivastava et al., 2014) on word embeddings improves performance on nearly all datasets, compared to this baseline. Therefore, elementwise dropout was applied to each of the input embeddings with probability 0.5 during training, and the weights were multiplied by 0.5 during testing. In order to separate the effects of this modification from the newly proposed optimisation method, we report results for three different systems: 1) the publicly available baseline, 2) applying dropout on top of the baseline system, and 3) applying both dropout and the novel multitask objective from Section 3.
Based on development experiments we set the value of γ, which controls the importance of the language modeling objective, to 0.1 for all experiments throughout training. Since context prediction is not part of the main evaluation of sequence labeling systems, we expected the additional objective to mostly benefit early stages of training, whereas the model would later need to specialise only towards assigning labels. Therefore, we also performed experiments on the development data where the value of γ was gradually decreased, but found that a small static value performed comparably well or even better. These experiments indicate that the language modeling objective helps
5https://github.com/marekrei/sequence-labeler
the network learn general-purpose features that are useful for sequence labeling even in the later stages of training.
5 Error Detection
We first evaluate the sequence labeling architectures on the task of error detection – given a sentence written by a language learner, the system needs to detect which tokens have been manually tagged by examiners as being an error. As the first benchmark, we use the publicly released First Certificate in English (FCE) dataset, containing 33,673 manually annotated sentences. The texts were written by learners during language examinations in response to prompts eliciting freetext answers and assessing mastery of the upperintermediate proficiency level. In addition, we evaluate on the CoNLL 2014 shared task dataset (Ng et al., 2014), which has been converted to an error detection task. This contains 1,312 sentences, written by higher-proficiency learners on more technical topics. They have been manually corrected by two separate annotators, and we report results on each of these annotations. For both datasets we use the FCE training set for model optimisation and results on the CoNLL-14 dataset indicate out-of-domain performance. Rei and Yannakoudakis (2016) present results on these datasets using the same setup, along with evaluating the top shared task submissions on the task of error detection. As the main evaluation metric, we use the F0.5 measure, which is consistent with previous work and was also adopted by the CoNLL14 shared task.
Table 1 contains results for the three different sequence labeling architectures on the error detection datasets. The baseline results are comparable to the previous best results on each of these benchmarks. We found that including the dropout actually decreases performance in the setting of error detection, which is likely due to the rela-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
CoNLL-00 CoNLL-03 CHEMDNER JNLPBA DEV TEST DEV TEST DEV TEST DEV TEST
Baseline 92.92 92.67 90.85 85.63 83.63 84.51 77.13 72.79 + dropout 93.40 93.15 91.14 86.00 84.78 85.67 77.61 73.16 + LMcost 94.22 93.88 91.48 86.26 85.45 86.27 78.51 73.83
Table 2: Performance of alternative sequence labeling architectures on NER and chunking datasets, measured using CoNLL standard entity-level F1 score.
tively small amount of error examples available in the dataset – it is better for the model to memorise them without introducing additional noise in the form of dropout. However, we did verify that dropout indeed gives an improvement in combination with the novel language modeling objective. Because the model is receiving additional information at every token, dropout is no longer obscuring the limited training data but instead helps with generalisation.
The bottom row shows the performance of the language modeling objective when added on top of the baseline model, along with dropout on word embeddings. This architecture outperforms the baseline on all benchmarks, increasing both precision and recall, and giving a 3.9% absolute improvement on the FCE test set. These are also the new state-of-the-art results for error detection on both FCE and CoNLL-14 datasets. Error detection is the task where introducing the additional cost objective gave the largest improvement in performance, for a few reasons:
1. This task has very sparse labels in the datasets, with error tokens very infrequent and far apart. Without the language modeling objective, the network has very little use for all the available words that contain no errors.
2. There are only two possible labels, correct and incorrect, which likely makes it more difficult for the model to learn feature detectors for many different error types. Language modeling uses a much larger number of possible labels, giving a more varied training signal.
3. Finally, the task of error detection is directly related to language modeling. By learning a better model of the overall text in the training corpus, the system can more easily detect any irregularities.
We also analysed the performance of the different architectures during training. Figure 2 shows the F0.5 score on the development set for each model after every epoch over the training data. The baseline model peaks quickly, followed by a gradual drop in performance, which is likely due to overfitting on the available data. Dropout provides an effective regularisation method, slowing down the initial performance but preventing the model from overfitting. The added language modeling objective provides a substantial improvement – the system outperforms other configurations already in the early stages of training and the results are also sustained in the later epochs.
6 NER and Chunking
In the next experiments we evaluate the language modeling objective on named entity recognition and chunking. For general-domain NER, we use the English section of the CoNLL 2003 corpus (Tjong Kim Sang and De Meulder, 2003), containing news stories from the Reuters Corpus. We also report results on two biomedical NER datasets: The BioCreative IV Chemical and Drug corpus (CHEMDNER, Krallinger et al. (2015)) of 10,000 abstracts, annotated for mentions of chemical and drug names, and the JNLPBA corpus (Kim et al.,
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
2004) of 2,404 abstracts annotated for mentions of different cells and proteins. Finally, we use the CoNLL 2000 dataset (Tjong Kim Sang and Buchholz, 2000), created from the Wall Street Journal Sections 15-18 and 20 from the Penn Treebank, for evaluating sequence labeling on the task of chunking. The standard CoNLL entity-level F1 score is used as the main evaluation metric.
Compared to error detection corpora, the labels are more balanced in these datasets. However, majority labels still exist: roughly 83% of the tokens in the NER datasets are tagged as ”O”, indicating that the word is not an entity, and the NP label covers 53% of tokens in the chunking data.
Table 2 contains results for evaluating the different architectures on NER and chunking. On these tasks, the application of dropout provides a consistent improvement – applying some variance onto the input embeddings results in more robust models for NER and chunking. The addition of the language modeling objective consistently further improves performance on all benchmarks.
While these results are comparable to the respective state-of-the-art results on most datasets, we did not fine-tune hyperparameters for any specific task, instead providing a controlled analysis of the language modeling objective in different settings. For JNLPBA, the system achieves 73.83% compared to 72.55% by Zhou and Su (2004) and 72.70% by Rei et al. (2016). On CoNLL-03, Lample et al. (2016) achieve a considerably higher result of 90.94%, possibly due to their use of specialised word embeddings and a custom version of LSTM. However, our system does outperform a similar architecture by Huang et al. (2015), achieving 86.26% compared to 84.26% F1 score on the CoNLL-03 dataset.
Figure 3 shows F1 on the CHEMDNER development set after every training epoch. Without dropout, performance peaks quickly and then trails off as the system overfits on the training set. Using dropout, the best performance is sustained throughout training and even slightly improved. Finally, adding the language modeling objective on top of dropout allows the system to consistently outperform the other architectures.
7 POS tagging
We also evaluated the language modeling training objective on two POS-tagging datasets. The Penn Treebank POS-tag corpus (Marcus et al., 1993)
contains texts from the Wall Street Journal and has been annotated with 48 different part-of-speech tags. In addition, we use the POS-annotated subset of the GENIA corpus (Ohta et al., 2002) containing 2,000 biomedical PubMed abstracts. Following Tsuruoka et al. (2005), we use the same 210-document test set.
These datasets are somewhat more balanced in terms of label distributions, compared to error detection and NER, as no single label covers over 50% of the tokens. POS-tagging also offers a large variance of unique labels, with 48 labels in PTB and 42 in GENIA, and this can provide useful information to the models during training. The baseline performance on these datasets is also close to the upper bound, therefore we expect the language modeling objective to not provide much additional benefit.
The results of different sequence labeling architectures on POS-tagging can be seen in Table 3 and accuracy on the development set is shown in Figure 4. While the performance improvements are small, they are consistent across both domains and datasets. Application of dropout again provides a more robust model, and the language modeling cost improves the performance further. Even though the labels already offer a varied training objective, learning to predict the surrounding words at the same time has provided the model with additional general-purpose features.
8 Related Work
Our work builds on previous research exploring multi-task learning in the context of different sequence labeling tasks. The idea of multi-task learning was described by Caruana (1998) and has since been extended to many language process-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
GENIA-POS PTB-POS DEV TEST DEV TEST
Baseline 98.69 98.61 97.23 97.24 + dropout 98.79 98.71 97.36 97.30 + LMcost 98.89 98.81 97.48 97.43
Table 3: Accuracy of different sequence labeling architectures on POS-tagging datasets.
ing tasks using neural networks. For example, Collobert and Weston (2008) proposed a multitask framework using weight-sharing between networks that are optimised for different supervised tasks.
Cheng et al. (2015) described a system for detecting out-of-vocabulary names by also predicting the next word in the sequence. While they use a regular recurrent architecture, we propose a language modeling objective that can be integrated with a bidirectional network, making it applicable to existing state-of-the-art sequence labeling frameworks.
Plank et al. (2016) described a related architecture for POS-tagging, predicting the frequency of each word together with the part-of-speech, and showed that this can improve tagging accuracy on low-frequency words. While predicting word frequency is useful for POS-tagging, language modeling provides a more general training signal, allowing us to apply the model to many different sequence labeling tasks. Our framework also achieves higher accuracy on the PTB-POS dataset (97.43% vs 97.22%).
9 Conclusion
We proposed a novel sequence labeling framework with a secondary objective – learning to predict surrounding words for each word in the dataset. One half of a bidirectional LSTM is trained as a forward-moving language model, whereas the other half is trained as a backward-moving language model. At the same time, both of these are also combined, in order to predict the most probable label for each word. This modification can be applied to several common sequence labeling architectures and requires no additional annotated or unannotated data.
The objective of learning to predict surrounding words provides an additional source of information during training. The model is incentivised to
discover useful features in order to learn the language distribution and composition patterns in the training data. While language modeling is not the main goal of the system, this additional training objective leads to more accurate sequence labeling models on several different tasks.
The architecture was evaluated on 8 different datasets, covering the tasks of error detection in learner texts, named entity recognition, chunking and POS-tagging. We found that the additional language modeling objective provided consistent performance improvements on every benchmark. The largest benefit from the new architecture was observed on the task of error detection in learner writing. The label distribution in the original dataset is very sparse and unbalanced, making it a difficult task for the model to learn. The added language modeling objective allowed the system to take better advantage of the available training data, leading to 3.9% absolute improvement over the previous state-of-the-art. The language modeling objective also provided consistent improvements on other sequence labeling tasks, such as named entity recognition, chunking and POS-tagging.
Future work could investigate the extension of this architecture to additional unannotated resources. Learning generalisable language features from large amounts of unlabeled in-domain text could provide sequence labeling models with additional benefit. While it is common to pre-train word embeddings on large-scale unannotated corpora, only minimal work has looked at useful methods for pre-training or co-training more advanced compositional modules.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

The paper proposes an approach to sequence labeling with multitask learning, where language modeling is uses as the auxiliary objective. Thus, a bidirectional neural network architecture learns to predict the output labels as well as to predict the previous or next word in the sentence. The joint objectives lead to improvements over the baselines in grammatical error detection, chunking, NER, and POS tagging.  - Strengths:  The contribution is quite well-written and easy to follow for the most part. The model is exposed in sufficient detail, and the experiments are thorough within the defined framework. The benefits of introducing an auxiliary objective are nicely exposed.  - Weaknesses:  The paper shows very limited awareness of the related work, which is extensive across the tasks that the experiments highlight. Tables 1-3 only show the three systems proposed by the contribution (Baseline, +dropout, and +LMcost), while some very limited comparisons are sketched textually.  A contribution claiming novelty and advancements over the previous state of the art should document these improvements properly: at least by reporting the relevant scores together with the novel ones, and ideally through replication. The datasets used in the experiments are all freely available, the previous results well-documented, and the previous systems are for the most part publicly available.  In my view, for a long paper, it is a big flaw not to treat the previous work more carefully.  In that sense, I find this sentence particularly troublesome:  The baseline results are comparable to the previous best results on each of these benchmarks.  The reader is here led to believe that the baseline system somehow subsumes all the previous contributions, which is shady on first read, and factually incorrect after a quick lookup in related work.  The paper states  new state-of-the-art results for error detection on both FCE and CoNLL-14 datasets . Looking into the CoNLL 2014 shared task report, it is not straightforward to discern whether the latter part of the claim does holds true, also as per Rei and Yannakoudakis' (2016) paper. The paper should support the claim by inclusion replication of the related work.  - General Discussion:  The POS tagging is left as more of an afterthought. The comparison to Plank et al. (2016) is at least partly unfair as they test across multiple languages in the Universal Dependencies realm, showing top-level performance across language families, which I for one believe to be far more relevant than WSJ benchmarking. How does the proposed system scale up down to multiple languages, low-resource languages with limited training data, etc.? The paper leaves a lot to ask for in that dimension to further substantiate its claims.  I like the idea of including language modeling as an auxiliary task. I like the architecture, and sections 1-4 in general. In my view, there is a big gap between those sections and the ones describing the experiments (5-8).  I suggest that this nice idea should be further fleshed out before publication. The rework should include at least a more fair treatment of related work, if not replication, and at least a reflection on multilinguality. The data and the systems are all there, as signs of the field's growing maturity. The paper should in my view partake in reflecting this maturity, and not step away from it. In faith that these improvements can be implemented before the publication deadline, I vote borderline.