Morphological Inflection Generation with Hard Monotonic Attention
We present a neural model for morphological inflection generation which employs a hard attention mechanism, inspired by the nearly-monotonic alignment commonly found between the characters in a word and the characters in its inflection. We evaluate the model on three previously studied morphological inflection generation datasets and show that it provides state of the art results in various setups compared to previous neural and nonneural approaches. Finally we present an analysis of the continuous representations learned by both the hard and soft attention (Bahdanau et al., 2014) models for the task, shedding some light on the features such models extract.
We present a neural model for morphological inflection generation which employs a hard attention mechanism, inspired by the nearly-monotonic alignment commonly found between the characters in a word and the characters in its inflection. We evaluate the model on three previously studied morphological inflection generation datasets and show that it provides state of the art results in various setups compared to previous neural and nonneural approaches. Finally we present an analysis of the continuous representations learned by both the hard and soft attention (Bahdanau et al., 2014) models for the task, shedding some light on the features such models extract.
1 Introduction
Morphological inflection generation involves generating a target word (e.g. “härtestem”, the German word for “hardest”), given a source word (e.g. “hart”, the German word for “hard”) and the morpho-syntactic attributes of the target (POS=adjective, gender=masculine, type=superlative, etc.).
The task is important for many down-stream NLP tasks such as machine translation, especially for dealing with data sparsity in morphologically rich languages where a lemma can be inflected into many different word forms. Several studies have shown that translating into lemmas in the target language and then applying inflection generation as a post-processing step is beneficial for phrase-based machine translation (Minkov et al., 2007; Toutanova et al., 2008; Clifton and Sarkar, 2011; Fraser et al., 2012; Chahuneau et al., 2013)
and more recently for neural machine translation (Garcı́a-Martı́nez et al., 2016).
The task was traditionally tackled with hand engineered finite state transducers (FST) (Koskenniemi, 1983; Kaplan and Kay, 1994) which rely on expert knowledge, or using trainable weighted finite state transducers (Mohri et al., 1997; Eisner, 2002) which combine expert knowledge with datadriven parameter tuning. Many other machinelearning based methods (Yarowsky and Wicentowski, 2000; Dreyer and Eisner, 2011; Durrett and DeNero, 2013; Hulden et al., 2014; Ahlberg et al., 2015; Nicolai et al., 2015) were proposed for the task, although with specific assumptions about the set of possible processes that are needed to create the output sequence.
More recently, the task was tackled using neural sequence to sequence learning over character sequences with impressive results (Faruqui et al., 2016). The vanilla encoder-decoder models as used by Faruqui et al. compress the input sequence to a single, fixed-sized continuous representation. Instead, soft-attention based sequence to sequence learning paradigm (Bahdanau et al., 2014) allows directly conditioning on the entire input sequence, and was utilized for morphological inflection generation with great success (Kann and Schütze, 2016b,a).
However, the sequence-to-sequence models require large training sets in order to perform well: their performance on the relatively small CELEX dataset is inferior to the latent variable WFST model of Dreyer et al. (2008). Interestingly, the neural WFST model by Rastogi et al. (2016) also suffered from the same issue on the CELEX dataset, and surpassed the latent variable model only when given twice as much data to train on.
We propose a model which handles the above issues by directly modeling an almost monotonic alignment between the input and output character
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
sequences, which is commonly found in the morphological inflection generation task (e.g. in languages with suffixing morphology). The model consists of an encoder-decoder neural network with a dedicated control mechanism: in each step, the model attends to a single input state and either writes a symbol to the output sequence or advances the attention pointer to the next input state from the bi-directionally encoded sequence, as described visually in Figure 1.
This modeling suits the natural monotonic alignment between the input and output, as the network learns to attend to the relevant inputs before writing the output which they are aligned to. The encoder is a bi-directional RNN, where each character in the input word is represented using a concatenation of a forward RNN and a backward RNN states over the word’s characters. The combination of the bi-directional encoder and the controllable hard attention mechanism enables to condition the output on the entire input sequence. Moreover, since each character representation is aware of the neighboring characters, non-monotone relations are also captured, which is important in cases where segments in the output word are a result of long range dependencies in the input word. The recurrent nature of the decoder, together with a dedicated feedback connection that passes the last prediction to the next decoder step explicitly, enables the model to also condition the current output on all the previous outputs at each prediction step. The hard attention mechanism allows the network to jointly align and transduce while using a focused representation at each step, rather then the weighted sum of representations used in the soft attention model. In contrast to previous sequence-to-sequence work, we do not require the training procedure to also learn the alignment. Instead, we use a simple training procedure which relies on independently learned character-level alignments, from which we derive gold transduction+control sequences. The network can then be trained using straightforward cross-entropy loss.
To evaluate our model, we perform extensive experiments on three previously studied morphological inflection generation datasets: the CELEX dataset (Baayen et al., 1993), the Wiktionary dataset (Durrett and DeNero, 2013) and the SIGMORPHON2016 dataset (Cotterell et al., 2016). We show that while our model is on par with
or better than the previous neural and non-neural state-of-the-art approaches, it also performs significantly better with very small training sets, being the first neural model to surpass the performance of the weighted FST model with latent variables specifically tailored for the task by Dreyer et al. (2008). Finally, we analyze and compare our model and the soft attention model, showing how they function very similarly with respect to the alignments and representations they learn, in spite of our model being much simpler. This analysis also sheds light on the representations such models learn for the morphological inflection generation task, showing how they encode specific features like a symbol’s type and the symbol’s location in a sequence.
To summarize, our contributions in this paper are three-fold:
1. We present a hard attention model for nearlymonotonic sequence to sequence learning, as common in the morphological inflection setting.
2. We evaluate the model on the task of morphological inflection generation, establishing a new state of the art on three previouslystudied datasets for the task.
3. We perform an analysis and comparison of our model and the soft-attention model, shedding light on the features such models extract for the inflection generation task.
2 The Hard Attention Model
We would like to transduce an input sequence, x1:n ∈ Σ∗x into an output sequence, y1:m ∈ Σ∗y, where Σx and Σy are the input and output vocabularies, respectively. Imagine a machine with read-only random access to the encoding of the input sequence, and a single pointer that determines the current read location. We can then model sequence transduction as a series of pointer movement and write operations. If we assume the alignment is monotone, the machine can be simplified: the memory can be read in sequential order, where the pointer movement is controlled by a single “move forward” operation (step) which we add to the output vocabulary. We implement this behavior using an encoder-decoder neural network, with a control mechanism which determines in each step of the decoder whether to write an
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
output symbol or promote the attention pointer the next element of the encoded input.
2.1 Model Definition
In prediction time, we seek the output sequence y1:m ∈ Σ∗y, for which:
y1:m = arg max y′∈Σ∗y
p(y′|x1:n, f) (1)
Where x ∈ Σ∗x is the input sequence and f = {f1, ..., fl} is a set of features influencing the transduction task (in the inflection generation task these would be the desired morpho-syntactic features of the output sequence). Given a nearlymonotonic alignment between the input and the output, we replace the search for a sequence of letters with a sequence of write and step actions s1:q ∈ Σ∗s, where Σs = Σy ∪ {step}. This sequence is a series of step and write actions required to go from x1:n to y1:m according to the monotonic alignment between them. In this case we define: 1
s1:q = arg max s′∈Σ∗s
p(s′|x1:n, f)
= arg max s′∈Σ∗s ∏ s′i∈s′ p(s′i|s′1...s′i−1, x1:n, f) (2)
we can then estimate this using a neural network:
s1:q = arg max s′∈Σ∗s NN(x1:n, f,Θ) (3)
Where the network’s parameters Θ are learned using a set of training examples. We will now describe the network architecture.
2.2 Network Architecture
Notation We use bold letters for vectors and matrices. We treat LSTM as a parameterized function LSTMθ(x1...xn) mapping a sequence of input vectors x1...xn to a an output vector hn. The equations for the LSTM variant we use are detailed in the supplementary material of this paper.
Encoder For every element in the input sequence: x1:n = x1...xn, we take the corresponding embedding: ex1 ...exn , where: exi ∈ RE .
1We note that our model (Eq 2) solves a different objective than (Eq 1), as it searches for the best derivation and not the best sequence. In order to accurately solve (1) we would need to marginalize over the different derivations leading to the same sequence, which is computationally challenging. However, as we see in the experiments section, the best-derivation approximation is effective in practice.
These embeddings are parameters of the model which will be learned during training. We then feed the embeddings into a bi-directional LSTM encoder (Graves and Schmidhuber, 2005) which results in a sequence of vectors: x1:n = x1...xn, where each vector xi ∈ R2H is a concatenation of: LSTMforward(ex1 , ex2 , ...exi) and LSTMbackward(exn , exn−1 ...exi), the forward LSTM and the backward LSTM outputs when fed with exi .
Decoder Once the input sequence is encoded, we feed the decoder RNN, LSTMdec, with three inputs at each step:
1. The current attended input, xa ∈ R2H , initialized with the first element of the encoded sequence, x1.
2. A set of feature embeddings that influence the generation process, concatenated to a single vector: f = [f1...fl] ∈ RF ·l.
3. si−1 ∈ RE , which is an embedding for the predicted output symbol in the previous decoder step.
Those three inputs are concatenated into a single vector zi = [xa, f , si−1] ∈ R2H+F ·l+E , which is fed into the decoder, providing the decoder output vector: LSTMdec(z1...zi) ∈ RH . Finally, to model the distribution over the possible actions, we project the decoder output to a vector of |Σs| elements, followed by a softmax layer:
p(si = c)
= softmax c(W · LSTMdec(z1...zi) + b) (4)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Control Mechanism When the most probable action is step, the attention is promoted so xa contains the next encoded input representation to be used in the next step of the decoder. The process is demonstrated visually in Figure 1.
2.3 Training the Model
For every example: (x1:n, y1:m, f) in the training data, we should produce a sequence of step and write actions s1:q to be predicted by the decoder. The sequence is dependent on the alignment between the input and the output: ideally, the network will attend to all the input characters aligned to an output character before writing it. While recent work in sequence transduction advocate jointly training the alignment and the decoding mechanisms (Bahdanau et al., 2014; Yu et al., 2016), we instead show that in our case it is worthwhile to decouple these stages and learn a hard alignment beforehand, using it to guide the training of the encoder-decoder network and enabling the use of correct alignments for the attention mechanism from the beginning of the network training phase. For this purpose, we first run a character level alignment process on the training data. We use the character alignment model of Sudoh et al. (2013) which is based on a Chinese Restaurant Process which weights single alignments (character-to-character) in proportion to how many times such an alignment has been seen elsewhere out of all possible alignments. Once we have the character level alignment per input-output sequence pair in the training set, we deterministically infer the sequence of actions s1:q from it by requiring to read all the input elements aligned to an output element (using the step action) before writing it. We then train the network to predict this sequence of actions by using a conventional cross entropy loss function per example: L(x1:n, y1:m, f,Θ) = − ∑
sj∈s1:q
log softmax sj (d),
d = W · LSTMdec(z1...zi) + b (5)
An alternative view of our encoder-decoder architecture is that of a transition system with ADVANCE and WRITE(CH) actions, where the oracle is derived from a given character-level alignment, the input is encoded using a biRNN, and the next action is determined by an RNN over the previous state/action combinations.
3 Experiments
We perform extensive experiments with three previously studied morphological inflection generation datasets to evaluate our hard attention model in various settings. In all experiments we report the results of the best performing neural and nonneural baselines which were previously published on those datasets. The implementation details for our models are described in the supplementary material section of this paper. The source code for our models is available on github.2
CELEX Our first evaluation is on a very small dataset, to see if our model indeed avoids the tendency to overfit with small training sets. We report exact match accuracy on the German inflection generation dataset compiled by Dreyer et al. (2008) from the CELEX database (Baayen et al., 1993). The dataset includes only 500 training examples for each of the four inflection types: 13SIA→13SKE, 2PIE→13PKE, 2PKE→z, and rP→pA which we refer to as 13SIA, 2PIE, 2PKE and rP, respectively.3 We compare our model to three competitive models that reported results on this dataset: the Morphological Encoder-Decoder (MED) of Kann and Schütze (2016a) which is based on the soft-attention model of Bahdanau et al. (2014), the neural-weighted FST of Rastogi et al. (2016) which uses stacked bi-directional LSTM’s to weigh its arcs (NWFST), and the model of Dreyer et al. (2008) which uses a weighted FST with latent-variables structured particularly for morphological string transduction tasks (LAT). Following previous reports on this dataset, we use the same data splits as Dreyer et al. (2008), dividing the data for each inflection type into five folds, each consisting of 500 training, 1000 development and 1000 test examples. We train a separate model for each fold and report exact match accuracy, averaged over the five folds.
Wiktionary To neutralize the negative effect of very small training sets on the performance of the different learning approaches, we also evaluate our model on the dataset created by Durrett and DeN-
2the link is removed in this version to keep the submission anonymized
3The acronyms stand for: 13SIA=1st/3rd person, singular, indefinite, past;13SKE=1st/3rd person, subjunctive, present; 2PIE=2nd person, plural, indefinite, present;13PKE=1st/3rd person, plural, subjunctive, present; 2PKE=2nd person, plural, subjunctive, present; z=infinitive; rP=imperative, plural; pA=past participle.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
13SIA 2PIE 2PKE rP Avg. MED 83.9 95 87.6 84 87.62 NWFST 86.8 94.8 87.9 81.1 87.65 LAT 87.5 93.4 87.4 84.9 88.3 Hard 85.8 95.1 89.5 87.2 89.44
Table 1: Results on the CELEX dataset
DE-N DE-V ES-V FI-NA FI-V FR-V NL-V Avg. DDN13 88.31 94.76 99.61 92.14 97.23 98.80 90.50 94.47 NCK15 88.6 97.50 99.80 93.00 98.10 99.20 96.10 96.04 FTND16 88.12 97.72 99.81 95.44 97.81 98.82 96.71 96.34 YBB16 87.5 92.11 99.52 95.48 98.10 98.65 95.90 95.32 Hard 88.87 97.35 99.79 95.75 98.07 99.04 97.03 96.55
Table 2: Results on the Wiktionary datasets
ero (2013), which contains up to 360k training examples per language. It was built by extracting Finnish, German and Spanish inflection tables from Wiktionary, used in order to evaluate their system based on string alignments and a semiCRF sequence classifier with linguistically inspired features. We also used the expansion made by Nicolai et al. (2015) to include French and Dutch inflections as well. Their system also performs an align-and-transduce approach, extracting rules from the aligned training set and applying them in inference time with a proprietary character sequence classifier. In addition to those systems we also compare to the results of the recent neural approaches of Faruqui et al. (2016), which did not use an attention mechanism, and Yu et al. (2016), which coupled the alignment and transduction tasks.
SIGMORPHON As different languages show different morphological phenomena, we also experiment with how our model copes with this various phenomena using the morphological inflection dataset from the SIGMORPHON2016 shared task (Cotterell et al., 2016). Here the training data consists of ten languages, with five morphological system types (detailed in Table 3): Russian (RU), German (DE), Spanish (ES), Georgian (GE), Finnish (FI), Turkish (TU), Arabic (AR), Navajo (NA), Hungarian (HU) and Maltese (MA) with roughly 12,800 training and 1600 development examples per language. We compare our model to two soft attention baselines on this dataset: MED (Kann and Schütze, 2016b), which was the best participating system in the shared
task, and our implementation of the global (soft) attention model presented by Luong et al. (2015).
4 Results
On the low resource setting (CELEX), our model significantly outperforms both the recent neural models of Kann and Schütze (2016a) and Rastogi et al. (2016) and the morphologically aware latent variable model of Dreyer et al. (2008), as detailed in Table 1. It is also, to our knowledge, the first model that surpassed in overall accuracy the latent variable model on this dataset. We attribute our advantage over the soft attention model to the ability of the hard attention control mechanism to harness the monotonic alignments found in the data, while also conditioning on the entire output history which wasn’t available in the FST models. Figure 2 plots the train-set and dev-set accuracies of the soft and hard attention models as a function of the training epoch. While both models perform similarly on the train-set (with the soft attention model fitting it slightly faster), the hard attention model performs significantly better on the dev-set. This shows the soft attention model’s tendency to overfit on the small dataset, as it is not enforcing the monotonic assumption of the hard attention model.
On the large training set experiments (Wiktionary), our model is the best performing model on German verbs, Finnish nouns/adjectives and Dutch verbs, resulting in the highest reported average accuracy across all the inflection types when compared to the four previous neural and nonneural state of the art baselines, as detailed in Ta-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
595
596
597
598
599
ble 2. This shows the robustness of our model also with large amounts of training examples, and the advantage the hard attention mechanism provides over the encoder-decoder approach of Faruqui et al. (2016) which does not employ an attention mechanism. Our model is also significantly more accurate than the model of Yu et al. (2016), which shows the advantage of using independently learned alignments to guide the network’s attention from the beginning of the training process. As can be seen in Table 3, on the SIGMORPHON 2016 dataset our model performs better than both soft-attention baselines for the suffixing+stem-change languages (Russian, German and Spanish) and is slightly less accurate than our implementation of the soft attention model on the rest of the languages, which is now the best performing model on this dataset to our knowledge. We explain this by looking at the languages from a linguistic typology point of view, as detailed in Cotterell et al. (2016). Since Russian, German and Spanish employ a suffixing morphol-
ogy with internal stem changes, they are more suitable for monotonic alignment as the transformations they need to model are the addition of suffixes and changing characters in the stem. The rest of the languages in the dataset employ more context sensitive morphological phenomena like vowel harmony and consonant harmony, which require to model long range dependencies in the input sequence which better suits the soft attention mechanism. While our implementation of the soft attention model and MED are very similar modelwise, we hypothesize that our soft attention model results are better due to the fact that we trained the model for 100 epochs and picked the best performing model on the development set, while the MED system was trained for a fixed amount of 20 epochs (although trained on more data – both train and development sets).
5 Analysis
The Learned Alignments In order to see if the alignments predicted by our model fit the mono-
suffixing+stem changes circ. suffixing+agg.+v.h. c.h. templatic RU DE ES GE FI TU HU NA AR MA Avg.
MED 91.46 95.8 98.84 98.5 95.47 98.93 96.8 91.48 99.3 88.99 95.56 Soft 92.18 96.51 98.88 98.88 96.99 99.37 97.01 95.41 99.3 88.86 96.34 Hard 92.21 96.58 98.92 98.12 95.91 97.99 96.25 93.01 98.77 88.32 95.61
Table 3: Results on the SIGMORPHON 2016 morphological inflection dataset. The text above each language lists the morphological phenomena it includes: circ.=circumfixing, agg.=agglutinative, v.h.=vowel harmony, c.h.=consonant harmony
0 10 20 30 40
0
0.5
1
epoch
ac cu
ra cy
soft-train hard-train soft-dev hard-dev
Figure 2: Learning curves for the soft and hard attention models on the first fold of the CELEX dataset
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
tonic alignment structure found in the data, and whether are they more suitable for the task when compared to the alignments found by the soft attention model, we examined alignment predictions of the two models on examples from the development portion of the CELEX dataset, as depicted in Figure 3. First, we notice the alignments found by the soft attention model are also monotonic, supporting our modeling approach for the task. Figure 3 (bottom-right) also shows how the hardattention model performs deletion (legte→lege) by predicting a sequence of two step operations. Another notable morphological transformation is the one-to-many alignment, found in the top example: flog→fliege, where the model needs to transform a character in the input, o, to two characters in the output, ie. This is performed by two consecutive write operations after the step operation of the relevant character to be replaced. Notice that in this case, the soft attention model performs a different alignment by aligning the character i to o and the character g to the sequence eg, which
is not the expected alignment in this case from a linguistic point of view.
The Learned Representations How does the soft-attention model manage to learn monotonic alignments? Perhaps the the network learns to encode the sequential position as part of its encoding of an input element? More generally, what information is encoded by the soft and hard alignment encoders? We selected 500 random encoded characters-in-context from input words in the CELEX development set, where every encoded representation is a vector in R200. Since those vectors are outputs from the bi-LSTM encoders of the models, every vector of this form carries information of the specific character with its entire context. We project these encodings into 2-D using SVD and plot them twice, each time using a different coloring scheme. We first color each point according to the character it represents (Figures 4a, 4b). In the second coloring scheme (Figures 4c, 4d), each point is colored according
(a) Colors indicate which character is encoded.
(b) Colors indicate which character is encoded.
(c) Colors indicate the character’s position.
(d) Colors indicate the character’s position.
Figure 4: SVD dimension reduction to 2D of 500 character representations in context from the encoder, for both the soft attention (top) and hard attention (bottom) models.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
to the character’s sequential position in the word it came from, blue indicating positions near the beginning of the word, and red positions near its end.
While both models tend to cluster representations for similar characters together (Figures 4a, 4b), the hard attention model tends to have much more isolated character clusters. Figures 4c, 4d show that both models also tend to learn representations which are sensitive to the position of the character, although it seems that here the soft attention model is more sensitive to this information as its coloring forms a nearly-perfect red-to-blue transition on the X axis. This may be explained by the soft-attention mechanism encouraging the encoder to encode positional information in the input representations, which may help it to predict better attention scores, and to avoid collisions when computing the weighted sum of representations for the context vector. In contrast, our hardattention model has other means of obtaining the position information in the decoder using the step actions, and for that reason it does not encode it as strongly in the representations of the inputs. This behavior may allow it to perform well even with fewer examples, as the location information is represented more explicitly in the model using the step actions.
6 Related Work
Many previous works on inflection generation used machine learning methods (Yarowsky and Wicentowski, 2000; Dreyer and Eisner, 2011; Durrett and DeNero, 2013; Hulden et al., 2014; Ahlberg et al., 2015; Nicolai et al., 2015) with assumptions about the set of possible processes needed to create the output word. Our work was mainly inspired by Faruqui et al. (2016) which trained an independent encoder-decoder neural network for every inflection type in the training data, alleviating the need for feature engineering. Kann and Schütze (2016b,a) tackled the task with a single soft attention model (Bahdanau et al., 2014) for all inflection types, which resulted in the best submission at the SIGMORPHON 2016 shared task (Cotterell et al., 2016). Another closely related work (Rastogi et al., 2016) modeled the task with a WFST in which the arcs are learned by optimizing a global loss function over all the possible paths in the state graph while modeling contextual features with bi-directional LSTM’s. Another recent work (Kann et al., 2016)
proposed performing neural multi-source morphological reinflection, generating an inflection from several source forms of a word.
Previous works on neural sequence transduction include the RNN Transducer (Graves, 2012) which uses two independent RNN’s over monotonically aligned sequences to predict a distribution over the possible output symbols in each step, including a null symbol to model the alignment. Yu et al. (2016) improved this by replacing the null symbol with a dedicated learned transition probability. Both models are trained using a forwardbackward approach, marginalizing over all possible alignments. Our model differs from the above by learning the alignments independently, thus enabling a dependency between the encoder and decoder. While providing better results than Yu et al. (2016), this also simplifies the model training using a simple cross-entropy loss. Jaitly et al. (2015) proposed the Neural Transducer model, which is also trained on external alignments. They divide the input into blocks of a constant size and perform soft attention separately on each block. Lu et al. (2016) used a combination of an RNN encoder with a CRF layer to model the dependencies in the output sequence. A line of work on attention-based speech recognition (Chorowski et al., 2015; Bahdanau et al., 2016) proposed improvements to the attention mechanism: adding location awareness by using the previous attention weights when computing the next ones, and preventing the model from attending on too many or too few inputs using “sharpening” and “smoothing” techniques on the attention weight distributions.
7 Conclusion
We presented a hard attention model for morphological inflection generation. The model employs an explicit alignment which is used to train a neural network to perform transduction by decoding with a hard attention mechanism. Our model performs on par or better than previous neural and non-neural approaches on various morphological inflection generation datasets, while staying competitive with dedicated models even with very few training examples. Future work may include applying the model to other nearly-monotonic alignand-transduce tasks like abstractive summarization, transliteration or machine translation.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: A new encoder-decoder model is proposed that explicitly takes  into account monotonicity.  - Weaknesses: Maybe the model is just an ordinary BiRNN with alignments de-coupled. Only evaluated on morphology, no other monotone Seq2Seq tasks.  - General Discussion:  The authors propose a novel encoder-decoder neural network architecture with  hard monotonic attention . They evaluate it on three morphology datasets.  This paper is a tough one. One the one hand it is well-written, mostly very clear and also presents a novel idea, namely including monotonicity in morphology tasks.   The reason for including such monotonicity is pretty obvious: Unlike machine translation, many seq2seq tasks are monotone, and therefore general encoder-decoder models should not be used in the first place. That they still perform reasonably well should be considered a strong argument for neural techniques, in general. The idea of this paper is now to explicity enforce a monotonic output character generation. They do this by decoupling alignment and transduction and first aligning input-output sequences monotonically and then training to generate outputs in agreement with the monotone alignments. However, the authors are unclear on this point. I have a few questions:  1) How do your alignments look like? On the one hand, the alignments seem to be of the kind 1-to-many (as in the running example, Fig.1), that is, 1 input character can be aligned with zero, 1, or several output characters. However, this seems to contrast with the description given in lines 311-312 where the authors speak of several input characters aligned to 1 output character. That is, do you use 1-to-many, many-to-1 or many-to-many alignments?  2) Actually, there is a quite simple approach to monotone Seq2Seq. In a first stage, align input and output characters monotonically with a 1-to-many constraint (one can use any monotone aligner, such as the toolkit of Jiampojamarn and Kondrak). Then one trains a standard sequence tagger(!) to predict exactly these 1-to-many alignments. For example, flogtofliege (your example on l.613): First align as in  f-l-o-g   f-l-ie-ge . Now use any tagger (could use an LSTM, if you like) to predict  f-l-ie-ge  (sequence of length 4) from  f-l-o-g  (sequence of length 4). Such an approach may have been suggested in multiple papers, one reference could be [*, Section 4.2] below.  My two questions here are:   2a) How does your approach differ from this rather simple idea?  2b) Why did you not include it as a baseline?  Further issues:  3) It's really a pitty that you only tested on morphology, because there are many other interesting monotonic seq2seq tasks, and you could have shown your system's superiority by evaluating on these, given that you explicitly model monotonicity (cf. also [*]).  4) You perform  on par or better  (l.791). There seems to be a general cognitive bias among NLP researchers to map instances where they perform worse to  on par  and all the rest to  better . I think this wording should be corrected, but otherwise I'm fine with the experimental results.  5) You say little about your linguistic features: From Fig. 1, I infer that they include POS, etc.   5a) Where did you take these features from?  5b) Is it possible that these are responsible for your better performance in some cases, rather than the monotonicity constraints?  Minor points:  6) Equation (3): please re-write $NN$ as $  text{NN}$ or similar  7) l.231  Where  should be lower case  8) l.237 and many more: $x_1  ldots x_n$. As far as I know, the math community recommends to write $x_1,  ldots,x_n$ but $x_1  cdots x_n$. That is, dots should be on the same level as surrounding symbols.  9) Figure 1: is it really necessary to use cyrillic font? I can't even address your example here, because I don't have your fonts.  10) l.437: should be  these   [*]   @InProceedings{schnober-EtAl:2016:COLING,     author    = {Schnober, Carsten  and  Eger, Steffen  and  Do Dinh, Erik-L  ^{a}n  and  Gurevych, Iryna},   title     = {Still not there? Comparing Traditional Sequence-to-Sequence Models to Encoder-Decoder Neural Networks on Monotone String Translation Tasks},   booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},   month     = {December},   year                                                      = {2016},   address   = {Osaka, Japan},   publisher = {The COLING 2016 Organizing Committee},   pages     = {1703--1714},   url                                               = {http:  aclweb.org anthology C16-1160}  }  AFTER AUTHOR RESPONSE  Thanks for the clarifications. I think your alignments got mixed up in the response somehow (maybe a coding issue), but I think you're aligning 1-0, 0-1, 1-1, and later make many-to-many alignments from these.  I know that you compare to  Nicolai, Cherry and Kondrak (2015) but my question would have rather been: why not use 1-x (x in 0,1,2) alignments as in  Schnober et al. and then train a neural tagger on these (e.g. BiLSTM). I wonder how much your results would have differed from such a rather simple baseline. (A tagger is a monotone model to start with and given the monotone alignments, everything stays monotone. In contrast, you start out with a more general model and then put hard monotonicity constraints on this ...)  NOTES FROM AC  Also quite relevant is Cohn et al. (2016), http:  www.aclweb.org anthology N16-1102 .  Isn't your architecture also related to methods like the Stack LSTM, which similarly predicts a sequence of actions that modify or annotate an input?    Do you think you lose anything by using a greedy alignment, in contrast to Rastogi et al. (2016), which also has hard monotonic attention but sums over all alignments?


Fast and Accurate Sequence Labeling with Iterated Dilated Convolutions
Bi-directional LSTMs have emerged as a standard method for obtaining per-token vector representations serving as input to various token labeling tasks (whether followed by Viterbi prediction or independent classification). This paper proposes an alternative to Bi-LSTMs for this purpose: iterated dilated convolutional neural networks (ID-CNNs), which have better capacity than traditional CNNs for large context and structured prediction. We describe a distinct combination of network structure, parameter sharing and training procedures that is not only more accurate than Bi-LSTM-CRFs, but also 8x faster at test time on long sequences. Moreover, ID-CNNs with independent classification enable a dramatic 14x testtime speedup, while still attaining accuracy comparable to the Bi-LSTM-CRF. We further demonstrate the ability of IDCNNs to combine evidence over long sequences by demonstrating their improved accuracy on whole-document (rather than per-sentence) inference. Unlike LSTMs whose sequential processing on sentences of length N requires O(N) time even in the face of parallelism, IDCNNs permit fixed-depth convolutions to run in parallel across entire documents. Today when many companies run basic NLP on the entire web and large-volume traffic, faster methods are paramount to saving time and energy costs.
1 Introduction
In order to democratize large-scale NLP and information extraction, we require fast, resource-
efficient methods for sequence tagging tasks such as part-of-speech tagging and named entity recognition (NER). Speed is not sufficient of course: they must also be expressive enough to tolerate the tremendous lexical variation in input data.
The massively parallel computation facilitated by GPU hardware has led to a surge of successful neural network architectures for sequence labeling (Ling et al., 2015; Ma and Hovy, 2016; Chiu and Nichols, 2016; Lample et al., 2016). While these models are expressive and accurate, they fail to fully exploit the parallelism opportunities of a GPU, and thus their speed is limited. Specifically, they employ either recurrent neural networks (RNNs) for feature extraction, or Viterbi inference in a structured output model, both of which require sequential computation across the length of the input.
Instead, parallelized runtime independent of the length of the sequence saves time and energy costs, maximizing GPU resource usage and minimizing the amount of time it takes to train and evaluate models. Convolutional neural networks (CNNs) provide exactly this property (Kim, 2014; Kalchbrenner et al., 2014). Rather than composing representations incrementally over each token in a sequence, they apply filters in parallel across the entire sequence at once. Their computational cost grows with the number of layers, but not the input size, up to the memory and threading limitations of the hardware. This provides, for example, audio generation models that can be trained in parallel (van den Oord et al., 2016).
Despite the clear computational advantages of CNNs, RNNs have become the standard method for composing deep representations of text. This is because a token encoded by a bidirectional RNN will incorporate evidence from the entire input sequence, but the CNN’s representation is limited by the receptive field of the architecture. Specifi-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
cally, in a network composed of a series of stacked convolutional layers of convolution width w, the number r of context tokens incorporated into a token’s representation at a given layer l, is given by r = 2l(w 1)+1. The number of layers required to incorporate the entire input context grows linearly with the length of the sequence. To avoid this scaling, one could pool representations across the sequence, but this is not appropriate for sequence labeling, since it reduces the output resolution of the representation.
In response, this paper presents an application of dilated convolutions (Yu and Koltun, 2016) for sequence labeling (Figure 1). For dilated convolutions, the receptive field can grow exponentially with the depth, with no loss in resolution at each layer and with a modest number of parameters to estimate. Like typical CNN layers, dilated convolutions operate on a sliding window of context over the sequence, but unlike conventional convolutions, the context need not be consecutive; the dilated window skips over every dilation width d inputs. By stacking layers of dilated convolutions of exponentially increasing dilation width, we can expand the size of the receptive field to cover the entire length of most sequences using only a few layers: The size of the receptive field for a token at layer l is now given by 2l+1 1. More concretely, just four stacked dilated convolutions of width 3 produces token representations with a receptive field of 31 tokens – longer than the average sentence length (23) in the Penn TreeBank.
Our overall iterated dilated CNN architecture (ID-CNN) repeatedly applies the same block of dilated convolutions to token-wise representations. This parameter sharing prevents overfitting and also provides opportunities to inject supervision on intermediate activations of the network. Similar to models that use RNN features, the ID-CNN provides two methods for performing prediction: we can predict each token’s label independently, or by running Viterbi inference in a chain structured graphical model.
In experiments on CoNLL 2003 and Ontonotes 5.0 English NER, we demonstrate significant speed gains of our ID-CNNs over various recurrent models, while maintaining similar F1 performance. When performing prediction using independent classification, the ID-CNN consistently outperforms a bidirectional LSTM (Bi-LSTM), and performs on par with inference in a CRF
Figure 1: A dilated CNN block with maximum dilation width 4 and filter width 3. Neurons contributing to a single highlighted neuron in the last layer are also highlighted.
with features extracted by a Bi-LSTM (Bi-LSTMCRF). As a feature extractor for a CRF, our model out-performs the Bi-LSTM-CRF. We also apply ID-CNNs to entire documents, where independent token classification is more accurate than the Bi-LSTM-CRF while decoding almost 8⇥ faster. The clear accuracy gains resulting from incorporating broader context suggest that these models could similarly benefit many other contextsensitive NLP tasks which have until now been limited by the computational complexity of existing context-rich models.1
2 Background
2.1 Conditional Probability Models for Tagging
Let x = [x1, . . . , xT ] be our input text and y = [y1, . . . , yT ] be per-token output tags. Let D be the domain size of each y
i . We predict the most likely y, given a conditional model P (y|x).
This paper considers two factorizations of the conditional distribution. First, we have
P (y|x) = TY
t=1
P (y t
|F (x)), (1)
where the tags are conditionally independent given some features for x. Given these features, O(D) prediction is simple and parallelizable across the length of the sequence. However, feature extraction may not necessarily be parallelizable. For example, RNN-based features require iterative passes along the length of x.
We also consider a linear-chain CRF model that couples all of y together:
P (y|x) = 1 Z x
TY
t=1
t (y t |F (x)) p (y t , y t 1), (2)
1Our implementation in TensorFlow (Abadi et al., 2015) is available at: https://github.com/anonymized
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
where t is a local factor, p is a pairwise factor that scores consecutive tags, and Z
x is the partition function (Lafferty et al., 2001). To avoid overfitting,
p does not depend on the timestep t or the input x in our experiments. Prediction in this model requires global search using the O(D2T ) Viterbi algorithm.
CRF prediction explicitly reasons about interactions among neighboring output tags, whereas prediction in the first model compiles this reasoning into the feature extraction step (Liang et al., 2008). The suitability of such compilation depends on the properties and quantity of the data. While CRF prediction requires non-trivial search in output space, it can guarantee that certain output constraints, such as for IOB tagging (Ramshaw and Marcus, 1999), will always be satisfied. It may also have better sample complexity, as it imposes more prior knowledge about the structure of the interactions among the tags (London et al., 2016). However, it has worse computational complexity than independent prediction.
3 Dilated Convolutions
CNNs in NLP are typically one-dimensional, applied to a sequence of vectors representing tokens rather than to a two-dimensional grid of vectors representing pixels. In this setting, a convolutional neural network layer is equivalent to applying an affine transformation, W
c to a sliding window of width r tokens on either side of each token in the sequence. Here, and throughout the paper, we do not explicitly write the bias terms in affine transformations. The sliding-window representation c
t
for each token x t is:
c t =
rM
k=0
x t±k, (3)
where is vector concatenation. Dilated convolutions perform the same operation, except rather than transforming adjacent inputs, the convolution is defined over a wider receptive field by skipping over inputs at a time, where is the dilation width. We define the dilated convolution operator:
c t =
rM
k=0
x t±k . (4)
A dilated convolution of width 1 is equivalent to a simple convolution. Using the same number of
parameters as a simple convolution with the same radius, the > 1 dilated convolution incorporates broader context into the representation of a token than a simple convolution.
Finally, Lei et al. (2015) present a CNN variant where convolutions adaptively skip neighboring words. While the flexibility of this model is powerful, its adaptive behavior is not well-suited to GPU acceleration.
3.1 Multi-Scale Context Aggregation
We can leverage the ability of dilated convolutions to incorporate global context without losing important local information by stacking dilated convolutions of increasing width. First described for pixel classification in computer vision, Yu and Koltun (2016) achieve state-of-the-art results on image segmentation benchmarks by stacking dilated convolutions with exponentially increasing rates of dilation, a technique they refer to as multiscale context aggregation. By feeding the outputs of each dilated convolution as the input to the next, increasingly non-local information is incorporated into each pixel’s representation. Performing a dilation-1 convolution in the first layer ensures that no pixels within the receptive field of any pixel are excluded. By doubling the dilation width at each layer, the size of the receptive field grows exponentially while the number of parameters grows only linearly with the number of layers, so a pixel representation quickly incorporates rich global evidence from the entire image.
4 Iterated Dilated CNNs
Stacked dilated CNNs can easily incorporate global information from a whole sentence or document. For example, with a radius of 1 and 4 layers of dilated convolutions, the receptive field of each token is width 31, which exceeds the average sentence length (23) in the Penn TreeBank corpus. With a radius of size 2 and 8 layers of dilated convolutions, the receptive field exceeds 1,000 tokens, long enough to encode an many full documents.
Unfortunately, simply increasing the depth of stacked dilated CNNs causes considerable overfitting in our experiments. In response, we present Iterated Dilated CNNs (ID-CNNs), which instead iterate a small series of dilated convolutions. Repeatedly employing the same parameters in a recurrent fashion provides both broad receptive fields and desirable generalization capabil-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
ities. We also obtain significant accuracy gains with a training objective that strives for accurate labeling after each iterate, allowing follow-on iterates to observe and resolve dependency violations.
4.1 Model Architecture
ID-CNNs contain repeated blocks of several convolutional layers. The network takes as input a sequence of T vectors xt of dimension dw, and outputs a sequence of per-class scores ht, which serve either as the local conditional distributions of the model (1) or the local factors
t of model (2). The first layer in the network is a dilation-1 convolution D(1)1 that transforms the input to a representation it of dimension dc:
it = D (0) 1 xt (5)
Next, L c 1 layers of dilated convolutions of exponentially increasing dilation width are applied to it, folding in increasingly broader context into the embedded representation of xt at each layer, followed by a dilation-1 convolution. Let r() denote a ReLU activation function (Glorot et al., 2011). Beginning with ct(0) = it we define the stack of layers with the following recurrence:
ct (Lc 1) = r ⇣ D(1)
2Lc 2 ct
(Lc 2) ⌘
ct (Lc) = r ⇣ D(3)1 ct (Lc 1) ⌘
(6)
We refer to this stack of dilated convolutions as a block B, which has output resolution equal to the input resolution. To incorporate even broader context without over-fitting, we avoid making B deeper, and instead iteratively apply B L
b times, which introduces no extra parameters. Starting with bt(1) = B (it):
bt (Lb) = B ⇣ bt (Lb 1) ⌘
(7)
We apply a simple affine transformation W c to this final representation to obtain per-class scores for each token xt:
ht = Wcbt (Lb) (8)
4.2 Training
Our main focus is to apply the ID-CNN as feature extraction for the first conditional model described in Sec. 2.1, where tags are conditionally independent given deep features, since this will enable prediction that is parallelizable across the length
of the input sequence. Here, maximum likelihood training is straightforward because the likelihood decouples into the sum of the likelihoods of independent logistic regression problems for every tag, with natural parameters given by (8):
1
T
TX
t=1
logP (y t | ht) (9)
We can also use the ID-CNN as input features for the CRF model (2), where the partition function and its gradient are computed using the forward-backward algorithm.
We next present an alternative training method that helps bridge the gap between these two techniques. Sec. 2.1 identifies that the CRF has preferable sample complexity and accuracy since prediction directly reasons in the space of structured outputs. In response, we compile some of this reasoning in output space into ID-CNN feature extraction. Instead of explicit reasoning over output labels during inference, we train the network such that each block is predictive of output labels. Subsequent blocks learn to correct dependency violations of their predecessors, refining the final sequence prediction.
To do so, we first define predictions of the model after each of the L
b applications of the block. Let hbt be the result of applying the matrix W
c from (8) to the output of the block b, or the initial word embeddings in the case of b = 0. We minimize the average of the losses for each application of the block:
1
B
BX
b=0
1
T
TX
t=1
logP (y t | hbt ). (10)
By rewarding accurate predictions after each application of the block, we learn a model where later blocks are used to refine initial predictions. The loss also helps reduce the vanishing gradient problem (Hochreiter, 1998) for deep architectures. Such an approach has been applied in a variety of contexts for training very deep networks in computer vision (Romero et al., 2014; Szegedy et al., 2015; Lee et al., 2015; Gülçehre and Bengio, 2016), but not to our knowledge in NLP.
We apply dropout (Srivastava et al., 2014) to the raw inputs xt and to each block’s output bt(b) to help prevent overfitting. The version of dropout typically used in practice has the undesirable property that the randomized predictor used at train
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
time differs from the fixed one used at test time. Ma et al. (2017) present dropout with expectationlinear regularization, which explicitly regularizes these two predictors to behave similarly. All of our best reported results include such regularization. This is the first investigation of the technique’s effectiveness for NLP, including for RNNs. We encourage its further application.
5 Related work
The state-of-the art models for sequence labeling include an inference step that searches the space of possible output sequences of a chain-structured graphical model, or approximates this search with a beam (Collobert et al., 2011; Weiss et al., 2015; Lample et al., 2016; Ma and Hovy, 2016; Chiu and Nichols, 2016). These outperform similar systems that use the same features, but independent local predictions. On the other hand, the greedy sequential prediction (Daumé III et al., 2009) approach of Ratinov and Roth (2009), which employs lexicalized features, gazetteers, and word clusters, outperforms CRFs with similar features.
LSTMs (Hochreiter and Schmidhuber, 1997) were used for NER as early as the CoNLL shared task in 2003 (Hammerton, 2003; Tjong Kim Sang and De Meulder, 2003). More recently, a wide variety of neural network architectures for NER have been proposed. Collobert et al. (2011) employ a one-layer CNN with pre-trained word embeddings, capitalization and lexicon features, and CRF-based prediction. Huang et al. (2015) achieved state-of-the-art accuracy on partof-speech, chunking and NER using a Bi-LSTMCRF. Lample et al. (2016) proposed two models which incorporated Bi-LSTM-composed character embeddings alongside words: a Bi-LSTMCRF, and a greedy stack LSTM which uses a simple shift-reduce grammar to compose words into labeled entities. Their Bi-LSTM-CRF obtained the state-of-the-art on four languages without word shape or lexicon features. Ma and Hovy (2016) use CNNs rather than LSTMs to compose characters in a Bi-LSTM-CRF, achieving state-ofthe-art performance on part-of-speech tagging and CoNLL NER without lexicons. Chiu and Nichols (2016) evaluate a similar network but propose a novel method for encoding lexicon matches, presenting results on CoNLL and OntoNotes NER. Yang et al. (2016) use GRU-CRFs with GRUcomposed character embeddings of words to train
a single network on many tasks and languages. In general, distributed representations for text can provide useful generalization capabilities for NER systems, since they can leverage unsupervised pre-training of distributed word representations (Turian et al., 2010; Collobert et al., 2011; Passos et al., 2014). Though our models would also likely benefit from additional features such as character representations and and lexicons, we focus on simpler models which use wordembeddings alone, leaving more elaborate input representations to future work.
In these NER approaches, CNNs were used for low-level mapping feature extraction that feeds into alternative architectures. Overall, end-to-end CNNs have mainly been used in NLP for sentence classification, where the output representation is lower resolution is lower than that of the input Kim (2014); Kalchbrenner et al. (2014); Zhang et al. (2015); Toutanova et al. (2015).
Our work draws on the use of dilated convolutions for image segmentation in the computer vision community (Yu and Koltun, 2016; Chen et al., 2015). Similar to our block, Yu and Koltun (2016) employ a context-module of stacked dilated convolutions of exponentially increasing dilation width. Dilated convolutions were recently applied to the task of speech generation (van den Oord et al., 2016), and concurrent with this work, Kalchbrenner et al. (2016) posted a pre-print describing a network for machine translation that uses dilated convolutions in the encoder and decoder components. We are the first to use dilated convolutions for sequence labeling.
The broad receptive field of the ID-CNN helps aggregate document-level context. Ratinov and Roth (2009) incorporate document context in their greedy model by adding features based on tagged entities within a large, fixed window of tokens. Prior work has also posed a structured model that couples predictions across the whole document (Bunescu and Mooney, 2004; Sutton and McCallum, 2004; Finkel et al., 2005).
6 Experimental Results
We describe experiments on two benchmark English named entity recognition datasets. On CoNLL-2003 English NER, our ID-CNN outperforms a Bi-LSTM as a feature extractor for a CRF, and with greedy decoding performs on-par with the Bi-LSTM-CRF while running at more
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
than 14 times the speed. We also observe a performance boost in almost all models when broadening the context to incorporate entire documents, achieving an average F1 of 90.65 on CoNLL2003, out-performing the Bi-LSTM-CRF while decoding at nearly 8 times the speed.
6.1 Data and Evaluation
We evaluate using labeled data from the CoNLL2003 shared task (Tjong Kim Sang and De Meulder, 2003) and OntoNotes 5.0 (Hovy et al., 2006; Pradhan et al., 2006). Following previous work, we use the same OntoNotes data split used for co-reference resolution in the CoNLL-2012 shared task (Pradhan et al., 2012). For both datasets, we convert the IOB boundary encoding to BILOU as previous work found this encoding to result in improved performance (Ratinov and Roth, 2009). As in previous work we evaluate the performance of our models using segment-level micro-averaged F1 score. Hyperparameters that resulted in the best performance on the validation set were selected via grid search. A more detailed description of the data, evaluation, optimization and data pre-processing can be found in the Appendix.
6.2 Baselines
We compare our ID-CNN against strong LSTM and CNN baselines: a Bi-LSTM with local decoding, and one with CRF decoding (Bi-LSTMCRF). We also compare against a non-dilated CNN architecture with the same number of convolutional layers as our dilated network (4-layer CNN) and one with enough layers to incorporate a receptive field of the same size as that of the dilated network (5-layer CNN) to demonstrate that the dilated convolutions more effectively aggregate contextual information than simple convolutions (i.e. using fewer parameters). We also compare our document-level ID-CNNs to a baseline which does not share parameters between blocks (noshare) and one that computes loss only at the last block, rather than after every iterated block of dilated convolutions (1-loss).
We do not compare with deeper or more elaborate CNN architectures for a number of reasons: 1) Fast train and test performance are highly desirable for NLP practitioners, and deeper models require more computation time 2) more complicated models tend to over-fit on this relatively small dataset and 3) most accurate deep CNN architectures repeatedly up-sample and down-sample the
inputs. We do not compare to stacked LSTMs for similar reasons — a single LSTM is already slower than a 4-layer CNN. Since our task is sequence labeling, we desire a model that maintains the token-level resolution of the input, making dilated convolutions an elegant solution.
6.3 CoNLL-2003 English NER
6.3.1 Sentence-level prediction
Table 1 lists F1 scores of models predicting with sentence-level context on CoNLL-2003. The Viterbi-decoding Bi-LSTM-CRF and ID-CNNCRF obtain the highest average scores, with the ID-CNN-CRF outperforming the Bi-LSTM-CRF by 0.11 points of F1 on average, and the BiLSTM-CRF out-performing the greedy ID-CNN by 0.11 as well. Our greedy ID-CNN outperforms all other greedy models, including the 4- layer CNN which uses the same number of parameters as the ID-CNN, and the 5-layer CNN which uses more parameters but covers the same size receptive field. All CNN models out-perform the Bi-LSTM when paired with greedy decoding, suggesting that CNNs are better feature extractors than Bi-LSTMs for independent logistic regression. When paired with Viterbi decoding, our ID-CNN out-performs the Bi-LSTM as a feature extractor, showing that the D-CNN is also a better feature extractor for Viterbi inference.
Our ID-CNN is not only a better feature extractor than the Bi-LSTM but it is also faster. Table 2 lists relative decoding times on the CoNLL de-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
657
658
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Model b = 1 Fastest (b) Bi-LSTM-CRF 1⇥ 1⇥ (1024) ID-CNN-CRF 4.55⇥ 1.28⇥ (32) Bi-LSTM 1.01⇥ 9.92⇥ (2048) 5-layer CNN 6.56⇥ 12.38⇥ (2048) ID-CNN 6.00⇥ 14.10⇥ (128)
Table 2: Relative test-time speed of sentence models, using batch size b = 1, and the fastest batch size (b) for each model.4
velopment set, compared to the Bi-LSTM-CRF. We report decoding times for batch size 1, giving Viterbi-decoding algorithms the advantage since much of their computational overhead comes from single-thread decoding on the CPU, and with the fastest batch size for each model.2
The D-CNN model is about 6 times faster than the Bi-LSTM when decoding one sentence at a time, and with larger batch sizes is nearly 50% faster. With Viterbi decoding, the gap closes somewhat but the ID-CNN-CRF still comes out ahead, about 30% faster than the Bi-LSTM-CRF. The most vast speed improvements come when comparing the greedy ID-CNN to the Bi-LSTMCRF – our ID-CNN is more than 14 times faster than the Bi-LSTM-CRF at test time, but only 0.11 F1 points less accurate. The 5-layer CNN, which observes the same size receptive field as the IDCNN but with more parameters, performs at about the same speed as the ID-CNN while making less accurate predictions. With a better implementation of dilated convolutions than currently included in TensorFlow, we would expect the DCNN to be notably faster than the 5-layer CNN.
We emphasize the importance of the dropout regularizer of Ma et al. (2017) in Table 3, where we observe increased F1 for every model trained with expectation-linear dropout regularization. Dropout is important for training neural network models that generalize well, especially on relatively small NLP datasets such as CoNLL2003. We recommend this regularizer as a simple and helpful tool for practitioners training neural networks for NLP.
2At scale, speed should increase with batch size, as we could compose each batch of as many sentences of the same length as would fit in GPU memory, requiring no padding and giving CNNs and D-CNNs even more of a speed advantage.
4Our D-CNN could see up to 18⇥ speed-up with a less naive implementation than is included in TensorFlow as of this writing.
6.3.2 Document-level prediction
In Table 4 we show that adding documentlevel context improves every model on CoNLL2003. When incorporating document-level context, our greedy ID-CNN model out-performs the Bi-LSTM-CRF, attaining 90.65 average F1. We believe this model out-performs the Bi-LSTMCRF due to the ID-CNN learning a feature function better suited for representing broad context, in contrast with the Bi-LSTM which, though better than a simple RNN at encoding long memories of sequences, may reach its limit when provided with sequences more than 1,000 tokens long such as entire documents.
We also note that our combination of training objective (Eqn. 10) and tied parameters (Eqn. 7) more effectively learns to aggregate this broad context than a vanilla cross-entropy loss or deep CNN back-propagated from the final neural network layer. Table 5 compares models trained to incorporate entire document context using the document baselines described in Section 6.2.
In Table 6 we show that, in addition to being more accurate, our ID-CNN model is also much faster than the Bi-LSTM-CRF when incorporating context from entire documents, decoding at almost 8 times the speed. On these long sequences, it also
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Model F1-avg F1-max ID-CNN noshare 89.81 ± 0.19 89.75 ID-CNN 1-loss 90.06 ± 0.19 90.15 ID-CNN 90.65 ± 0.15 90.66
Table 5: Comparing ID-CNNs with 1) backpropagating loss only from the final layer (1-loss) and 2) untied parameters across blocks (noshare)
Model b = 1 Fastest (b) Bi-LSTM-CRF 1⇥ 1⇥ (1024) Bi-LSTM 1.05⇥ 4.60⇥ (1024) ID-CNN 40.83⇥ 7.96⇥ (32)
Table 6: Relative test-time speed of document models, using batch size b = 1, and the fastest batch size (b) for each model.
tags at more than 4.5 times the speed of the greedy Bi-LSTM, demonstrative of the benefit of our IDCNNs context-aggregating computation that does not depend on the length of the sequence.
6.4 OntoNotes 5.0 English NER
We observe similar patterns on OntoNotes as we do on CoNLL. Table 7 lists overall F1 scores of our models compared to those in the existing literature. The greedy Bi-LSTM out-performs the lexicalized greedy model of Ratinov and Roth (2009), and our ID-CNN out-performs the Bi-LSTM as well as the more complex model of Durrett and Klein (2014) which leverages the parallel coreference annotation available in the OntoNotes corpus to predict named entities jointly with entity linking and co-reference. Our greedy model is out-performed by the Bi-LSTM-CRF reported in Chiu and Nichols (2016) as well as our own re-implementation, which appears to be the new state-of-the-art on this dataset.
Model F1 Ratinov and Roth (2009)5 83.45 Durrett and Klein (2014) 84.04 Chiu and Nichols (2016) 86.19 ± 0.25 Bi-LSTM 83.76 ± 0.10 ID-CNN 84.23 ± 0.30 Bi-LSTM-CRF 86.99 ± 0.22 Bi-LSTM-CRF (doc) 86.81 ± 0.18 ID-CNN (doc) 85.76 ± 0.13
Table 7: F1 score of sentence and document models on OntoNotes.
The gap between our greedy model and those using Viterbi decoding is wider than on CoNLL. We believe this is due to the more diverse set of entities in OntoNotes, which also tend to be much longer – the average length of a multi-token named entity segment in CoNLL is about one token shorter than in OntoNotes. These long entities benefit more from explicit structured constraints enforced in Viterbi decoding. Still, our ID-CNN outperforms all other greedy methods, achieving our goal of learning a better feature extractor for structured prediction.
Incorporating greater context significantly boosts the score of our greedy model on OntoNotes, whereas the Bi-LSTM-CRF performs more poorly. In Table 7, we also list the F1 of our ID-CNN model and the Bi-LSTM-CRF model trained on entire document context. For the first time, we see the score decrease when more context is added to the Bi-LSTM-CRF model, though the ID-CNN, whose sentence model has a much lower score than that of the Bi-LSTM-CRF, sees an increase of more than 1.5 F1 points. We believe the decrease in the Bi-LSTM-CRF model occurs because of the nature of the OntoNotes dataset compared to CoNLL-2003: CoNLL-2003 contains a particularly high proportion of ambiguous entities,6 perhaps leading to more benefit from document context that helps with disambiguation. In this scenario, adding the wider context may just add noise to the high-scoring Bi-LSTM-CRF model, whereas the less accurate dilated model can still benefit from the refined predictions of the iterated dilated convolutions.
7 Conclusion
We present iterated dilated convolutional neural networks, fast feature extractors that efficiently aggregate broad context without losing resolution. These provide impressive speed improvements for sequence labeling, particularly when processing entire documents at a time. In the future we hope to extend this work to NLP tasks with richer structured output, such as parsing.
5Results as reported in Durrett and Klein (2014) as this data split did not exist at the time of publication.
6According to the ACL Wiki page on CoNLL-2003: “The corpus contains a very high ratio of metonymic references (city names standing for sport teams)”
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This work proposes to apply dilated convolutions for sequence tagging (specifically, named entity recognition). It also introduces some novel ideas (sharing the dilated convolution block, predicting the tags at each convolution level), which I think will prove useful to the community. The paper performs extensive ablation experiments to show the effectiveness of their approach. I found the writing to be very clear, and the experiments were exceptionally thorough.  Strengths:   - Extensive experiments against various architectures (LSTM, LSTM + CRF)        - Novel architectural training ideas (sharing blocks)    Weaknesses:   - Only applied to English NER--this is a big concern since the title of the paper seems to reference sequence-tagging directly.   - Section 4.1 could be clearer. For example, I presume there is padding to make sure the output resolution after each block is the same as the input resolution.  Might be good to mention this.   - I think an ablation study of number of layers vs perf might be interesting.  RESPONSE TO AUTHOR REBUTTAL:  Thank you very much for a thoughtful response. Given that the authors have agreed to make the content be more specific to NER as opposed to sequence-tagging, I have revised my score upward.