Chunk-based Decoder for Neural Machine Translation
Chunks (or phrases) had once played a pivotal role in machine translation. By using a chunk rather than a word as the basic translation unit, local (intrachunk) and global (inter-chunk) word orders/dependencies can be easily modeled. The chunk structure, despite its importance, has not been considered in the decoders of neural machine translation (NMT). In this paper, we propose chunkbased decoders for NMT, each of which consists of a chunk-level decoder and a word-level decoder. The chunk-level decoder models global dependencies, while the word-level decoder decides the word orders in a chunk. To generate a target sentence, the chunk-level decoder generates a chunk representation containing global information, based on which, the wordlevel decoder predicts the words inside the chunk. Experimental results show that our method can significantly improve translation performance in a WAT ’16 Englishto-Japanese translation task.
1 Introduction
Neural machine translation (NMT) performs an end-to-end translation based on a simple encoderdecoder model (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014b), and now has overwhelmed the classical, complex statistical machine translation (SMT) (Sennrich et al., 2016; Luong and Manning, 2016; Cromieres et al., 2016; Neubig, 2016). In NMT, an encoder first maps a source sequence into vector representations and a decoder then maps the vectors into a target sequence (§ 2). This simple framework allows researchers to incorporate the structure of the source
sentence as in SMT by leveraging various architectures as the encoder (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014b; Eriguchi et al., 2016b). Most of the NMT models, however, still rely on a sequential decoder based on recurrent neural network (RNN), due to the difficulty in capturing the structure of a target sentence that is unseen during translation.
With the sequential decoder, however, there are two problems to be solved. First, it is difficult to model long-distance dependencies (Bahdanau et al., 2015). A hidden state ht in an RNN is only conditioned by its previous output yt−1, previous hidden state ht−1 and current input xt. This makes it difficult to capture the dependencies between an older output yt−N if they are too far from the current output. This problem can become more serious when the target sequence become longer. For example in Figure 1, when one translates the English sentence into the Japanese one, after the decoder predicts the content word “噛ま (bite)”, it has to predict five function words “れ (passive)”, “た (past)”, “そう (hearsay)”, “だ (positive)”, and “けれど (but)” and a punctuation mark “、” before predicting the next content word “君 (you)”. In such a case, the decoder is required to capture the longer dependencies in a target sentence.
Another problem with the sequential decoder is that it is expected to cover possible word orders simply by memorizing the local word sequences in the limited training data. This problem can be more serious in free word-order languages such as Czech, German, Japanese, and Turkish. In the case of the example in Figure 1, the order of the subject phrase “だれかが (someone was)” and the modifier phrase “犬に (by a dog)” are flexible. This means that simply memorizing the word order in training data is not enough to train a model that can assign a high probability to a correct sentence regardless of its word order.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
!" # $ % & '( " ) *+ ! ,"- . / 0 12 3 4 5 65 7 !"#$"%$ &"' ()*$ +", )%-,.$
%,// %,//
0%1 233 34$5.&3 33 *45* !"#$"%$3 3365! ()**$%3 33(+ 5 &"'3 3373 336$.$%8* +",333 )%-,.$&3 339
:51
Figure 1: Translation from English to Japanese. The aligned words are content words and the underlined words are function words.
Looking back to the past, chunks (or phrases) are utilized to handle the above aforementioned problems in statistical machine translation (SMT) (Watanabe et al., 2003; Koehn et al., 2003) and in example-based machine translation (EBMT) (Kim et al., 2010). By using a chunk rather than a word as the basic translation unit, one can treat a sentence as a shorter sequence. This makes it easy to capture the longer dependencies in a target sentence. The order of words in a chunk is relatively fixed while that in a sentence is much more flexible. Thus, modeling intra-chunk (local) dependencies and inter-chunk (global) dependencies independently can help capture the difference of the flexibility between the word order and the chunk order in free word-order languages.
In this paper, we refine the original RNN decoder to consider chunk information in NMT. We propose three novel NMT models that capture and utilize the chunk structure in the target language (§ 3). Our focus is the hierarchical structure of a sentence: each sentence consists of chunks, and each chunk consists of words. To encourage an NMT model to capture the hierarchical structure, we start from a hierarchical RNN that consists of a chunk-level decoder and a word-level decoder (Model 1). Then, we improve the word-level decoder by introducing inter-chunk connections to capture the interaction between chunks (Model 2). Finally, we introduce a feedback mechanism to the chunk-level decoder to enhance the memory capacity of previous outputs (Model 3).
We evaluate the three models on the WAT ’16 English-to-Japanese translation task (§ 4). The experimental results show that our best model outperforms the best single NMT model reported in WAT ’16 (Eriguchi et al., 2016b).
Our contributions are twofold: (1) chunk information is firstly introduced into NMT to improve translation performance, and (2) a novel hierarchical decoder is devised to model the properties of chunk structure in the encoder-decoder framework.
2 Preliminaries: Attention-based Neural Machine Translation
In this section, we briefly introduce the architecture of the attention-based NMT model, which is the basis of our proposed models.
2.1 Neural Machine Translation
An NMT model usually consists of two connected neural networks: an encoder and a decoder. After the encoder maps a source sentence into a fixed-length vector, the decoder maps the vector into a target sentence. The implementation of the encoder can be a convolutional neural network (CNN) (Kalchbrenner and Blunsom, 2013), a long short-term memory (LSTM) (Sutskever et al., 2014; Luong and Manning, 2016), a gated recurrent unit (GRU) (Cho et al., 2014b; Bahdanau et al., 2015), or a Tree-LSTM (Eriguchi et al., 2016b). While various architectures are leveraged as an encoder to capture the structural information in the source language, most of the NMT models rely on a standard sequential network such as LSTM or GRU as its decoder.
Following (Bahdanau et al., 2015), we use GRU as the recurrent unit in this paper. A GRU unit computes its hidden state vector hi given an input vector xi and the previous hidden state hi−1:
hi = GRU(hi−1, xi). (1)
The function GRU(·) is calculated as:
ri = σ(Wrxi +Urhi−1 + br), (2)
zi = σ(Wzxi +Uzhi−1 + bz), (3)
h̃i = tanh(Wxi +U(ri hi−1 + b)), (4) hi = (1− zi) h̃i + zi hi−1, (5)
where the vectors ri and zi are reset gate and update gate, respectively. While the former gate allows the model to forget the previous states, the latter gate decides how much the model updates its content. All the W s and Us, or the bs above
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
!"
!"#$%&'( )*%%&"( +,-,&+
#"
#$"
だれ か が %"
&'
.
&( &).
/ )&-'% *)
.
.#(
#$(
#'
#$'
Figure 2: Standard word-based decoder.
are trainable matrices or vectors. σ(·) and denote the sigmoid function and element-wise multiplication operator, respectively.
We train a GRU that encodes a source sentence {x1, · · · , xI} into a single vector hI . At the same time, we jointly train another GRU that decodes hI to the target sentence {y1, · · · , yJ}. Here, the jth word in the target sentence yj can be predicted with this decoder GRU and a nonlinear function g(·) followed by a softmax layer, as
c = hI , (6)
sj = GRU(sj−1, [yj−1; c]), (7)
s̃j = g(yj−1, sj , c), (8) P (yj |y<j ,x) = softmax(s̃j), (9)
where c is a context vector of the encoded sentence and sj is a hidden state of the decoder GRU.
Following Bahdanau et al. (2015), we use a mini-batch stochastic gradient descent (SGD) algorithm with ADADELTA (Zeiler, 2012) to train the above two GRUs (i.e., the encoder and the decoder) jointly. The objective is to minimize the cross-entropy loss of the training data D, as
J = ∑
(x,y)∈D
− logP (y|x). (10)
2.2 Attention Mechanism for Neural Machine Translation
To use all the hidden states of the encoder and improve the translation performance of long sentences, Bahdanau et al. (2015) proposed using an attention mechanism. In their model, the context vector is not simply the last encoder state hI but rather the weighted sum of all encoder states, as follows:
cj = I∑
i=1
αjihi. (11)
Here, the weight αji decides how much a source word xi contributes to the target word yj . αji is
computed by a feedforward layer and a softmax layer as
eji = v · tanh(Wehi +Uesj + be), (12) αji = exp(eji)∑J
j′=1 exp(ej′i) , (13)
where We, Ue are trainable matrices and the be is a trainable vector. In a decoder using the attention mechanism, the obtained context vector cj in each timestep replaces cs in Eqs. (7) and (8). An illustration of the NMT model with the attention mechanism is shown in Figure 2.
The attention mechanism is expected to learn alignments between source and target words, and plays a similar role to the translation model in phrase-based SMT (Koehn et al., 2003).
3 Chunk-based Neural Machine Translation
Taking non-sequential information such as chunks (or phrases) structure into consideration is proved to be helpful for SMT (Watanabe et al., 2003; Koehn et al., 2003) and EBMT (Kim et al., 2010). We here focus on two important properties of chunks (Abney, 1991): (1) The word order in a chunk is almost always fixed; (2) A chunk consists of a few (typically one) content words surrounded by zero or more function words.
To fully utilize the above properties of a chunk, we propose to model the intra-chunk and the interchunk dependencies independently with a “chunkby-chunk” decoder (See Figure 3). In the standard word-by-word decoder described in § 2, a target word yj in the target sentence y is predicted by taking the previous outputs y<j and the source
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
!"
!"#$%&'('& )'*"$'#+ ,!-./.-0
1'23'4567&+84*"$'#+ ,!-././0
#$
%$ &'( %) &'(
9:34;+/ 9:34;+;%/ 9:34;+;
%" &'(
%*" &'(
だれ か が 犬 噛ま れ
%*$ &'( %*) &'(
+"
<
%,-./ &'(
+,-./
9:34;%&'('& )'*"$'#+ ,!-./.=0
#) #0<
> :'7#$ 10
<
<
<
!"#$%& '(&)*+$,-./0*1& ."**$2+3"*& 4§56'7
!"#$%& 5(&8",#-+"-./0*1& 9$$#:;21& 4§5657
%$ &2(
%3 &2( %34$ &2(
%" &2(
Figure 4: Proposed model: Chunk-based NMT. A chunk-level decoder generates a chunk representation for each chunk while a word-level decoder uses the representation to predict each word. The solid lines in the figure illustrate Model 1. The dashed blue arrows in the word-level decoder denote the connections added in Model 2. The dotted red arrows in the chunk-level decoder denote the feedback states added in Model 3; the connections in the thick black arrows are replaced with the dotted red arrows.
sentence x as input:
P (y|x) = J∏
j=1
P (yj |y<j ,x), (14)
where J is the length of the target sentence. Not assuming any structural information of the target language, the sequential decoder has to memorize long dependencies in a sequence. To release the model from the strong pressure of memorizing the long dependencies over a sentence, we redefine this problem as the combination of a word prediction problem and a chunk generation problem: P (y|x) = K∏ k=1 P (ck|c<k,x) Jk∏ j=1 P (yj |y<j , ck,x)  , (15) whereK is the number of chunks in the target sentence and Jk is the length of the k-th chunk (see Figure 3). The first term represents the generation probability of a chunk and the second term indicates the probability of a word in the chunk . We model the former term as a chunk-level decoder and the latter term as a word-level decoder. As we will later confirm in § 4, both K and Jk are much shorter than the sentence length J , which is why our decoders do not have to memorize the long dependencies like the standard decoder does.
In the above formulation, we model the information of the words and their orders in a chunk.
No matter which language we target, a chunk usually consists of some content words and function words, and the word order in the chunk is almost always fixed (Abney, 1991). Although our idea can be used in several languages, the optimal network architecture could be adapted depending on the word order of the target language. In this work, we design models for languages in which content words are followed by function words, such as Japanese and Korean. The details of our models are described in the following sections.
3.1 Model 1: Standard Chunk-based NMT
The model described in this section is the basis of our proposed models. It consists of three parts: a sequential encoder (§ 3.1.1), a chunk-level decoder (§ 3.1.2), and a word-level decoder (§ 3.1.3). The part drawn in black solid lines in Figure 4 illustrates the architecture of Model 1.
3.1.1 Sequential Encoder
We adopt a standard single-layer bidirectional GRU (Cho et al., 2014b; Bahdanau et al., 2015) as our encoder (see the right part in Figure 4). By using a standard sequential encoder, we need not perform any additional preprocessing on test data such as syntactic parsing. This design prevents our model from being affected by any errors that may occur as a result of additional preprocessing during test time.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
3.1.2 Chunk-level Decoder
Our chunk-level decoder (see Figure 3) outputs a chunk representation. The chunk representation contains the information about the words that should be predicted by the word-level decoder.
To generate the representation of the k-th chunk s̃ (c) k , the chunk-level decoder (see the bottom layer in Figure 4) takes the last states of the word-level decoder s(w)Jk−1 and updates its hidden state s (c) k as:
s (c) k = GRU(s (c) k−1, s (w) Jk−1 ), (16) s̃ (c) k = Wcs (c) k + bc. (17)
The obtained chunk representation s̃(c)k continues to be fed into the word-level decoder until it outputs all the words in current chunk.
3.1.3 Word-level Decoder
Our word-level decoder (see Figure 4) differs from the standard sequential decoder described in § 2 in that it takes the chunk representation s̃(c)k as input:
s (w) j = GRU(s (w) j−1, [s̃ (c) k ; yj−1; c (w) j−1]), (18) s̃ (w) j = g(yj−1, s (w) j , c (w) j ), (19)
P (yj |y<j ,x) = softmax(s̃(w)j ). (20)
In a standard sequential decoder, the hidden state iterates over the length of a target sentence. In other words, its hidden layers are required to memorize the long-term dependencies in the target language. In contrast, in our word-level decoder, the hidden state iterates only over the length of a chunk. Thus, our word-level decoder is released from the pressure of memorizing the long (interchunk) dependencies and can focus on learning the short (intra-chunk) dependencies.
3.2 Model 2: Inter-Chunk Connection
The second term in Eq. (15) only iterates over a chunk (j = 1 to Jk). This means that the last state and the last output of a chunk are not being fed into the word-level decoder at the next timestep (see the black part in Figure 4). In other words, s (w) 1 in Eq. (18) is always initialized before generating the first word in a chunk. This may affect the word-level decoder because it cannot access any previous information at the first word of each chunk.
To address this problem, we add new connections to Model 1 between the first state in a chunk
and the last state in the previous chunk, as
s (w) 1 = GRU(s (w) Jk−1 , [s̃ (c) k ; yJk−1 ; c (w) Jk−1 ]). (21)
The dashed blue arrows in Figure 4 illustrate the added inter-chunk connections.
3.3 Model 3: Word-to-Chunk Feedback
The chunk-level decoder in Eq. (16) is only conditioned by s(w)Jk−1 , the last word state in each chunk (see the black part in Figure 4). This may affect the chunk-level decoder because it cannot memorize what kind of information has already been generated by the word-level decoder. The information about the words in a chunk should not be included in the representation of the next chunk; otherwise, it may generate the same chunks for multiple times, or forget to translate some words in the source sentence.
To encourage the chunk-level decoder to remove the information about the previous outputs more carefully, we add feedback states to our chunk-level decoder in Model 2. The feedback state in the chunk-level decoder is updated at every timestep j as:
s (c) j = GRU(s (c) j−1, s (w) j ). (22)
The red lines in Figure 4 illustrate the added feedback states and their connection. The connections in the thick black arrows are replaced with the dotted red arrows in Model 3.
4 Experiments
4.1 Setup
Data To clarify the effectiveness of our decoders, we choose Japanese, a free word-order language, as the target language. Japanese sentences are easy to be broken into well-defined chunks (called bunsetsus (Hashimoto, 1934) in Japanese), and the accuracy of bunsetsu-chunking is over 99% (Murata et al., 2000; Yoshinaga and Kitsuregawa, 2014). The effect of chunking errors in training the decoder can be suppressed so we can evaluate the potential of our method. We use the English-Japanese training corpus in the Asian Scientific Paper Excerpt Corpus (ASPEC) (Nakazawa et al., 2016), which was provided in WAT ’16. To remove inaccurate translation pairs, we extracted the first 2 million data from the 3 million translation pairs following the setting that gave the best performances in WAT ’15 (Neubig et al., 2015).
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Corpus # words # chunks # sentences Train 44,286,317 13,707,397 1,505,871 Dev. 54,287 - 1,790 Test 54,088 - 1,812
Table 1: Statistics of the target language (Japanese) in extracted corpus after preprocessing.
Preprocessings For Japanese sentences, we performed tokenization using KyTea 0.4.7.1 (Neubig et al., 2011) Then we performed bunsetsuchunking with CaboCha 0.69.2 For English sentences, we performed the same preprocessings described on the WAT ’16 Website.3 To suppress having possible chunking errors affect the translation quality, we removed extremely long chunks from the training data. Specifically, among the 2 million preprocessed translation pairs, we excluded the sentence pairs that matched any of following conditions: (1) The length of the source sentence or target sentence is larger than 64; (2) The maximum length of a chunk in the target sentence is larger than 8 (around 1% of whole data); (3) The maximum number of chunks in the target sentence is larger than 20 (around 2% of whole data). The amount of the excluded sentences by the condition (2) or (3) is negligible (less than 3% of whole data). Table 1 shows the details of the extracted data.
Postprocessing To perform unknown word replacement (Luong et al., 2015), we built a bilingual English-Japanese dictionary from all of the 3 million translation pairs. The dictionary was extracted with the MGIZA++ 0.7.04 (Och and Ney, 2003; Gao and Vogel, 2008) word alignment tool by automatically extracting the alignments between English words and Japanese words.
Evaluation Following the WAT ’16 evaluation procedure, we used BLEU (Papineni et al., 2002) and RIBES (Isozaki et al., 2010) to evaluate our models. The BLEU scores were calculated with multi-bleu.pl in Moses 2.1.15 (Koehn et al., 2007); RIBES scores were calculated with RIBES.py 1.03.1.6 (Isozaki et al., 2010)
1http://www.phontron.com/kytea/ 2http://taku910.github.io/cabocha/ 3http://lotus.kuee.kyoto-u.ac.jp/WAT/
baseline/dataPreparationJE.html 4https://github.com/moses-smt/mgiza 5http://www.statmt.org/moses/ 6http://www.kecl.ntt.co.jp/icl/lirg/ ribes/index.html
Training Details We use a single layer bidirectional GRU for the encoder and standard single layer GRUs for the word-level decoder and the chunk-level decoder. The vocabulary sizes are set to 30k for both source and target languages. The conditional probability of each target word is computed with a deep-output (Pascanu et al., 2014) layer with maxout (Goodfellow et al., 2013) units. The maximum number of output chunks is set to 20 and the maximum length of a chunk is set to 9.
The models are optimized using ADADELTA following (Bahdanau et al., 2015). The hyperparameters of the training procedure are fixed to the values given in Table 2. Note that the learning rate is halved when the BLEU score on the development set does not increase for 30,000 batches. All the parameters are initialized randomly with Gaussian distribution. It takes about a week to train each model with an NVIDIA TITAN X (Pascal) GPU.
4.2 Results
Following (Cho et al., 2014a), we perform beam search8 with length-normalized log-probability to decode target sentences. We saved the trained models that performed best on the development set during training, and use them to test the systems with the test set. Table 3 shows the results on the test set. Note that all the models listed in Table 3, including our three models, are single models without ensemble techniques. We set the word-based sequence-to-sequence model (Li et al., 2016) as our baseline, which is a standard implementation of the attention-based NMT described in § 2. We also compare our methods with the tree-to-sequence model (Eriguchi et al., 2016b) to compare the effectiveness of capturing the structure in the source language and that in the target language. Our improved models (Model 2 and Model 3) outperform all the single models reported in WAT ’16. The best model (Model 3)
7http://lotus.kuee.kyoto-u.ac.jp/WAT/ evaluation
8Beam size is set to 20.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
System RNN |Vsrc| |Vtrg| BLEU RIBES Word-based Seq-to-Seq (Li et al., 2016) GRU 40k 30k 33.47 78.75 Word-based Tree-to-Seq (Eriguchi et al., 2016b) LSTM 88k 66k 34.87 81.58 Character-based Tree-to-Seq (Eriguchi et al., 2016a) LSTM 88k 3k 31.52 79.39 Proposed chunk-based model (Model 1) GRU 30k 30k 33.56 79.92 + Inter-chunk connection (Model 2) GRU 30k 30k 35.44 80.95 + Word-to-chunk feedback (Model 3) GRU 30k 30k 36.20 82.06
Table 3: The results and settings of the baseline systems and our systems. |Vsrc| and |Vtrg| denote the vocabulary size of the source language and the target language, respectively. Only single NMT models (w/o ensembling) reported in WAT ’16 are listed here. Full results are available on the WAT ’16 Website.7
!"#$%&' ()*+, -.-&$,,/&+%$*0&/, ()&,,1&2, ,.#()"$,,3+,,()&"$4,,"1,,56.++, ($.1+*(*"1,,-"*1(,,7$"8,,%$"++6*19&/ $&+*1,,(",,6*1&.$,,-"648&$, :,
;&7&$&1%&' 著者の 架橋 樹脂から 線型ポリマへの ガラス転移点 に関 する 新しい 理論に つい て 述べた 。
<"/&6,=' 架橋樹脂から 線状ポリマへ の ガラス転移点に 関する 著者の 理論につ いて 述べた。 .#()"$3+ ()&,()&"$4 /&+%$*0&/7$"8,%$"++6*19&/ $&+*1 (",6*1&.$,-"648&$ .0"#(,56.++,($.1+*(*"1,-"*1(
<"/&6,>' 架橋樹脂から 線状ポリマーへの ガラス転移点に 関する 著者の 新しい 理論につ いて 述べた。 1&2.#()"$3+ /&+%$*0&/()&,()&"$47$"8,%$"++6*19&/ $&+*1 (",6*1&.$,-"648&$ .0"#(,56.++,($.1+*(*"1,-"*1(
<"/&6,?' 架橋樹脂から 線状高 分子に ガラス転移点 に関 する 新しい 著者の 著者 の 理論に つい て 述べた 。 1&2 .#()"$3+ .#()"$3+ ()&,()&"$4 /&+%$*0&/7$"8,%$"++6*19&/ $&+*1 (",6*1&.$,-"648&$ .0"#(,56.++,($.1+*(*"1,-"*1(
Figure 5: Translation examples of a long sentence. Each sequence of underlined words correspond to a chunk recognized by our decoder. Model 1 outputs a chunk “著者の (author’s)” twice by mistake. Model 2 does not output a chunk “新しい (new)” by mistake. In contrast, Model 3 outputs a correct translation although there is a minor word order difference from the reference.
outperforms the tree-to-sequence model (Eriguchi et al., 2016b) by +1.33 BLEU score and +0.48 RIBES score. The results show that capturing the chunk structure in the target language is more effective than capturing the syntax structure in the source language. Compared with the characterbased NMT model, our Model 3 outperformed the model of (Eriguchi et al., 2016a) by +4.68 BLEU score and +2.67 RIBES score. The characterbased model has a great advantage in that it does not require a large vocabulary size. Although the character-based model is less time-consuming thanks to the small target vocabulary size (|Vtrg| = 3k), our chunk-based model significantly outperformed it in terms of translation quality. One possible reason for this is that using a character-based model rather than a word-based model makes it more difficult to capture long-distance dependencies because the length of a target sequence becomes much longer in the character-based model.
To understand the qualitative difference between our three models, we show translation examples in Figure 5. While Model 3 outputs a cor-
rect translation, there are some errors in the outputs of Model 1 and Model 2. Only Model 1 outputs a chunk “著者の (author’s)” twice continuously. This error indicates that the inter-chunk connections added in Model 2 play important roles in memorizing previous word states more efficiently. On the other hand, Model 2 does not output a chunk “新しい (new)” by mistake, which is probably because it does not have a good ability to memorize the previous chunks. This phenomenon supports the importance of the feedback states that are added in Model 3.
5 Related Work
There has been much work done on using chunk (or phrase) structure to improve machine translation quality. The most notable work was phrasebased SMT (Koehn et al., 2003), which has been the basis for a huge amount of work on SMT for more than ten years. Apart from this, Watanabe et al. (2003) proposed a chunk-based translation model that generates output sentences in a chunkby-chunk manner. The chunk structure is effective
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
not only for SMT but also for example-based machine translation (EBMT). Kim et al. (2010) proposed a chunk-based EBMT and showed that using chunk structures can help with finding better word alignments. Our work is different from their works in that our models are based on NMT, but not SMT or EBMT. The decoders in the above works can model the chunk structure by storing chunk pairs in a large table. On the other hand, we do that by separately training a chunk generation model and a word prediction model with two RNNs.
While most of the NMT models focus on the conversion between sequential data, some works have tried to incorporate non-sequential information into NMT (Eriguchi et al., 2016b; Su et al., 2017). Eriguchi et al. (2016b) use tree-based LSTM (Tai et al., 2015) to encode input sentence into context vectors. Given a syntactic tree of a source sentence, their tree-based encoder encodes words from the leaf nodes to the root nodes recursively. Su et al. (2017) proposed a lattice-based encoder that considers multiple tokenization results while encoding the input sentence. To prevent the tokenization errors from propagating to the whole NMT system, their lattice-based encoder can utilize multiple tokenization results. These works focus on the encoding process and propose better encoders that can exploit the structures of the source language. In contrast, our work focuses on the decoding process to capture the structure of the target language. The encoders described above and our proposed decoders are complementary so they can be combined into a single network.
Considering that our Model 1 described in § 3.1 can be seen as a hierarchical RNN, our work is also related to previous studies that utilize multilayer RNNs to capture hierarchical structures in data. Hierarchical RNNs are used not only in the field of machine translation (Luong and Manning, 2016) but also for various NLP tasks such as document modeling (Li et al., 2015; Lin et al., 2015), dialog generation (Serban et al., 2017), image captioning (Krause et al., 2016), and video captioning (Yu et al., 2016). In particular, Li et al. (2015) and Luong and Manning (2016) use hierarchical sequence-to-sequence models, but not for the purpose of learning syntactic structures of target sentences. Li et al. (2015) build hierarchical models at the sentence-word level to obtain better document representations. Luong and Manning (2016) build the word-character level to cope with the out-
of-vocabulary problem. In contrast, we build a chunk-word level model to explicitly capture the syntactic structure based on chunk segmentation.
In addition, the architecture of Model 3 is also related to stacked RNN, which has shown to be effective to improve the translation quality (Luong et al., 2015; Sutskever et al., 2014). Although these architectures look similar to each other, there is a fundamental difference between the directions of the connection between two layers. A stacked RNN consists of multiple RNN layers that are connected from the input side to the output side at every timestep. In contrast, our Model 3 has a different connection at each timestep. Before it generates a chunk, there is a feed-forward connection from the chunk-level decoder to the word-level decoder. However, after generating a chunk representation, the connection is to be reversed to feedback the information from the word-level decoder to the chunk-level decoder. By switching the connections between two layers, our model can capture the chunk structure explicitly. This is the first work that proposes the decoders for NMT that can capture plausible linguistic structures like chunk.
6 Conclusion
In this paper, we propose chunk-based decoders for NMT. As the attention mechanism in NMT plays a similar role to the translation model in phrase-based SMT, our chunk-based decoders are intended to capture the notion of chunk in chunkbased (or phrase-based) SMT. We utilize the chunk structure to efficiently capture long-distance dependencies and cope with the problem of free word-order languages like Japanese. We design three models that have hierarchical RNN-like architectures, each of which consists of a word-level decoder and a chunk-level decoder. We performed experiments on the WAT ’16 English-to-Japanese translation task and found that our best model outperforms all the single models that were reported in WAT ’16 by +4.68 to +1.33 BLEU scores and by +3.31 to +0.48 RIBES scores.
In future work, we will apply our method to other target languages and evaluate the effectiveness on different languages such as Czech, German or Turkish. In addition, we plan to combine our decoder with other encoders that capture language structure, such as a Tree-LSTM (Eriguchi et al., 2016b), or an order-free encoder, such as a CNN (Kalchbrenner and Blunsom, 2013).
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: The paper presents an interesting extension to attention-based neural MT approaches, which leverages source-sentence chunking as additional piece of information from the source sentence. The model is modified such that this chunking information is used differently by two recurrent layers: while one focuses in generating a chunk at a time, the other focuses on generating the words within the chunk. This is interesting. I believe readers will enjoy getting to know this approach and how it performs. The paper is very clearly written, and alternative approaches are clearly contrasted. The evaluation is well conducted, has a direct contrast with other papers (and evaluation tables), and even though it could be strengthened (see my comments below), it is convincing.  - Weaknesses: As always, more could be done in the experiments section to strengthen the case for chunk-based models. For example, Table 3 indicates good results for Model 2 and Model 3 compared to previous papers, but a careful reader will wonder whether these improvements come from switching from LSTMs to GRUs. In other words, it would be good to see the GRU tree-to-sequence result to verify that the chunk-based approach is still best.  Another important aspect is the lack of ensembling results. The authors put a lot of emphasis is claiming that this is the best single NMT model ever published. While this is probably true, in the end the best WAT system for Eng-Jap is at 38.20 (if I'm reading the table correctly) - it's an ensemble of 3. If the authors were able to report that their 3-way chunk-based ensemble comes top of the table, then this paper could have a much stronger impact.  Finally, Table 3 would be more interesting if it included decoding times. The authors mention briefly that the character-based model is less time-consuming (presumably based on Eriguchi et al.'16), but no cite is provided, and no numbers from chunk-based decoding are reported either. Is the chunk-based model faster or slower than word-based? Similar? Who know... Adding a column to Table 3 with decoding times would give more value to the paper.  - General Discussion: Overall I think the paper is interesting and worth publishing. I have minor comments and suggestions to the authors about how to improve their presentation (in my opinion, of course).   * I think they should clearly state early on that the chunks are supplied externally - in other words, that the model does not learn how to chunk. This only became apparent to me when reading about CaboCha on page 6 - I don't think it's mentioned earlier, and it is important.  * I don't see why the authors contrast against the char-based baseline so often in the text (at least a couple of times they boast a +4.68 BLEU gain). I don't think readers are bothered... Readers are interested in gains over the best baseline.  * It would be good to add a bit more detail about the way UNKs are being handled by the neural decoder, or at least add a citation to the dictionary-based replacement strategy being used here.  * The sentence in line 212 ( We train a GRU that encodes a source sentence into a single vector ) is not strictly correct. The correct way would be to say that you do a bidirectional encoder that encodes the source sentence into a set of vectors... at least, that's what I see in Figure 2.  * The motivating example of lines 69-87 is a bit weird. Does  you  depend on  bite ? Or does it depend on the source side? Because if it doesn't depend on  bite , then the argument that this is a long-dependency problem doesn't really apply.


We propose a new A* CCG parsing model in which the probability of a tree is decomposed into factors of CCG categories and its dependency structure both defined on bi-directional LSTMs. Our factored model allows the precomputation of all probabilities and runs very efficiently, while modeling sentence structures explicitly via dependencies. Our model achieves the stateof-the-art results on English and Japanese CCG parsing1.
1 Introduction
Supertagging in lexicalized grammar parsing is known as almost parsing (Bangalore and Joshi, 1999), in that each supertag is syntactically informative and most ambiguities are resolved once a correct supertag is assigned to every word. Recently this property is effectively exploited in A* Combinatory Categorial Grammar (CCG; Steedman (2000)) parsing (Lewis and Steedman, 2014; Lewis et al., 2016), in which the probability of a CCG tree y on a sentence x of length N is the product of the probabilities of supertags (categories) ci (locally factored model):
P (y|x) = ∏
i∈[1,N ]
Ptag(ci|x). (1)
By not modeling every combinatory rule in a derivation, this formulation enables us to employ efficient A* search (see Section 2), which finds the most probable supertag sequence that can build a well-formed CCG tree. Although much ambiguity is resolved with this supertagging, some ambiguity still remains. Figure 1 shows an example, where the two CCG
1 We provide our software as a supplementary material.
(a)
a house in Paris in France
NP (NP\NP)/NP NP (NP\NP)/NP NP > >
NP\NP NP\NP <
NP <
NP (b)
a house in Paris in France
NP (NP\NP)/NP NP (NP\NP)/NP NP >
NP\NP < NP >
NP\NP <
NP
parses are derived from the same supertags. Lewis et al.’s approach to this problem is resorting to some deterministic rule. For example, Lewis et al. (2016) employ the attach low heuristics, which is motivated by the right-branching tendency of English, and always prioritizes (b) for this type of ambiguity. Though for English it empirically works well, an obvious limitation is that it does not always derive the correct parse; consider a phrase “a house in Paris with a garden”, for which the correct parse has the structure corresponding to (a) instead. In this paper, we provide a way to resolve these remaining ambiguities under the locally factored model, by explicitly modeling bilexical dependencies as shown in Figure 1. Our joint model is still locally factored so that an efficient A* search can be applied. The key idea is to predict the head of every word independently as in Eq. 1 with a strong unigram model, for which we utilize the scoring model in the recent successful graph-based dependency parsing on LSTMs (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2016). Specif-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
ically, we extend the bi-directional LSTM (biLSTM) architecture of Lewis et al. (2016) predicting the supertag of a word to predict the head of it at the same time with a bilinear transformation. The importance of modeling structures beyond supertags is demonstrated in the performance gain in Lee et al. (2016), which adds a recursive component to the model of Eq. 1. Unfortunately, this formulation loses the efficiency of the original one since it needs to compute a recursive neural network every time it searches for a new node. Our model does not resort to the recursive networks while modeling tree structures via dependencies. We also extend the tri-training method of Lewis et al. (2016) to learn our model with dependencies from unlabeled data. On English CCGbank test data, our model with this technique achieves 88.8% and 94.0% in terms of labeled and unlabeled F1, which mark the best scores so far. Besides English, we provide experiments on Japanese CCG parsing. Japanese employs freer word order dominated by the case markers and a deterministic rule such as the attach low method may not work well. We show that this is actually the case; our method outperforms the simple application of Lewis et al. (2016) in a large margin, 10.0 points in terms of clause dependency accuracy.
2 Background
Our approach is build on A* CCG parsing (Section 2.1), which we extend in Section 3 with a head prediction model on bi-LSTMs (Section 2.2).
2.1 Supertag-factored A* CCG Parsing
CCG has a nice property that since every category is highly informative about attachment decisions, assigning it to every word (supertagging) determines most of its syntactic structure. Lewis and Steedman (2014) utilize this characteristics of the grammar. Let a CCG tree y be a list of categories ⟨c1, . . . , cN ⟩ and a derivation on it. Their model looks for the most probable y given a sentence x from the set Y (x) of possible CCG trees under the model of Eq. 1:
ŷ = arg max y∈Y (x) ∑ i∈[1,N ] logPtag(ci|x).
Since this score is factored into each supertag, they call it supertag-factored.
Exact inference of this problem is possible by A* parsing (Klein and D. Manning, 2003), which uses the following two scores on a chart:
b(Ci,j) = ∑
ck∈ci,j
logPtag(ck|x),
a(Ci,j) = ∑
k∈[1,N ]\[i,j]
max ck
logPtag(ck|x),
where Ci,j is a chart item called an edge, which abstracts parses spanning interval [i, j] rooted by category C. The chart maps each edge to the derivation with the highest score, i.e., the Viterbi parse for Ci,j . ci,j is the sequence of categories on such Viterbi parse, and thus b is called the Viterbi inside score, while a is the approximation (upper bound) of the Viterbi outside score. A* parsing is a kind of CKY chart parsing augmented with an agenda, a priority queue that keeps the edges to be explored. At every step it pops the edge e with the highest priority b(e) + a(e) and inserts that into the chart, and enqueue any edges that can be built by combining e with other edges in the chart. The algorithm terminates when an edge C1,N is popped from the agenda.
A* search for this model is quite efficient because both b and a can be obtained from the unigram category distribution on every word, which can be precomputed before search. The heuristics a gives an upperbound on the true Viterbi outside score (i.e., admissible). Along with this the condition that the inside score never decreases by expansion (monotonicity) guarantees that the first found derivation on C1,N is always optimal. a(Ci,j) matches the true outside score if the onebest category assignments on the outside words (argmaxck logPtag(ck|x)) can comprise a wellformed tree with Ci,j , which is generally not true.
Scoring model For modeling Ptag, Lewis and Steedman (2014) use a log-linear model with features from a fixed window context. Lewis et al. (2016) extend this with bi-LSTMs, which encode the complete sentence and capture the long range syntactic information. We base our model on this bi-LSTM architecture, and extend it to modeling a head word at the same time.
Attachment ambiguity In A* search, an edge with the highest priority b + a is searched first, but as shown in Figure 1 the same categories (with the same priority) may sometimes derive different
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
trees. In Lewis and Steedman (2014), they prioritize the parse with longer dependencies, which they judge with a conversion rule from a CCG tree to a dependency tree (Section 4). Lewis et al. (2016) employ another heuristics prioritizing low attachments of constituencies, but inevitably these heuristics cannot be flawless in any situations. We provide a simple solution to this problem by explicitly modeing bilexical dependencies.
2.2 Bi-LSTM Dependency Parsing
For modeling dependencies, we borrow the idea from the recent graph-based neural dependency parsing (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2016) in which each dependency arc is scored directly on the outputs of bi-LSTMs. Though the model is first-order, bi-LSTMs enable conditioning on the entire sentence and lead to the state-of-the-art performance. Note that this mechanism is similar to modeling of the supertag distribution discussed above, in that for each word the distribution of the head choice is unigram and can be precomputed. As we will see this keeps our joint model still locally factored and A* search tractable. For score calculation, we use an extended bilinear transformation by Dozat and Manning (2016) that models the bias term as well, which they call biaffine.
3 Proposed Method
3.1 A* parsing with Supertag and Dependency Factored Model
We define a CCG tree y for a sentence x = ⟨xi, . . . , xN ⟩ as a triplet of a list of CCG categories c = ⟨c1, . . . , cN ⟩, dependencies h = ⟨h1, . . . , hN ⟩, and the derivation, where hi is the head index of xi. Our model is defined as follows:
P (y|x) = ∏
i∈[1,N ]
Ptag(ci|x) ∏
i∈[1,N ]
Pdep(hi|x).
(2)
The added term Pdep is a unigram distribution of the head choice.
A* search is still tractable under this model. The search problem is changed as:
ŷ = arg max y∈Y (x) ( ∑ i∈[1,N ] logPtag(ci|x)
+ ∑
i∈[1,N ]
logPdep(hi|x) ) ,
and the inside score is given by:
b(Ci,j) = ∑
ck∈ci,j
logPtag(ck|x) (3)
+ ∑
k∈[i,j]\{root(hCi,j)}
logPdep(hk|x),
where hCi,j is a dependency subtree for the Viterbi parse on Ci,j and root(h) returns the root index. We exclude the head score for the subtree root token since it cannot be resolved inside [i, j]. This causes the mismatch between the goal inside score b(C1,N ) and the true model score (log of Eq. 2), which we adjust by adding a special unary rule that is always applied to the popped goal edge C1,N . We can calculate the dependency terms in Eq. 3 on the fly when expanding the chart. Let the current popped edge be Ai,k, which will be combined with Bk,j into Ci,j . The key observation is that only one dependency arc (between root(hAi,k) and root(hBk,j)) is resolved at every combination (see Figure 2). For every rule C → A B we can define the head direction (see Section 4) and Pdep is obtained accordingly. For example, when the right child B becomes the head, b(Ci,j) = b(Ai,k) + b(Bk,j) + logPdep(hl = m|x), where l = root(hAi,k) andm = root(h B k,j) (l < m).
The Viterbi outside score is changed as:
a(Ci,j) = ∑
k∈[1,N ]\[i,j]
max ck
logPtag(ck|x)
+ ∑ k∈L max hk logPdep(hk|x),
where L = [1, N ] \ [k′|k′ ∈ [i, j], root(hCi,j) ̸= k′]. We regard root(hCi,j) as an outside word since its head is undefined yet. For every outside word we independently assign the weight of its argmax
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
head, which may not comprise a well-formed dependency tree. We initialize the agenda by adding an item for every supertag C and word xi with the score a(Ci,i) = ∑ k∈I\{i}max logPtag(ck|x) +∑
k∈I max logPdep(hk|x). Note that the dependency component of it is the same for every word.
3.2 Network Architecture
Following Lewis et al. (2016) and Dozat and Manning (2016), we model Ptag and Pdep using biLSTMs for exploiting the entire sentence to capture the long range phenomena. See Figure 3 for the overall network architecture, where Ptag and Pdep share the common bi-LSTM hidden vectors. First we map every word xi to their hidden vector ri with bi-LSTMs. The input to the LSTMs is word embeddings, which we describe in Section 6. We add special start and end tokens to each sentence with the trainable parameters following Lewis et al. (2016). For Pdep, we use the biaffine transformation in Dozat and Manning (2016):
gdepi = MLP dep child(ri), gdephi = MLP dep head(rhi), Pdep(hi|x) (4)
∝ exp((gdepi ) TWdepg dep hi +wdepg dep hi ),
where MLP is a multilayered perceptron. Though Lewis et al. (2016) simply use a MLP for mapping ri to Ptag, we additionally utilize the hidden vector of the most probable head hi = argmaxh′i Pdep(h ′ i|x), and apply ri and rhi to a bilinear function:2
gtagi = MLP tag child(ri), gtaghi = MLP tag head(rhi), (5)
ℓ = (gtagi ) TUtagg tag hi +Wtag [ gtagi gtaghi ] + btag,
Ptag(ci|x) ∝ exp(ℓc),
where Utag is a third order tensor. As in Lewis et al. these values can be precomputed before search, which makes our A* parsing quite efficient.
4 CCG to Dependency Conversion
Nowwe describe our conversion rules from a CCG tree to a dependency one, which we use in two purposes: 1) creation of the training data for the de-
2 This is inspired by the formulation of label prediction in Dozat and Manning (2016).
pendency component of our model; and 2) extraction of a dependency arc at each combinatory rule during A* search (Section 3.1). Lewis and Steedman (2014) describe one way to extract dependencies from the CCG tree (LEWISRULE). Below in addition to this we describe two simpler alternatives (HEADFIRST and HEADFINAL), and see the effects on parsing performance in our experiments (Section 6). See Figure 4 for the overview.
LEWISRULE This is the same as the conversion rule in Lewis and Steedman (2014). As shown in Figure 4c the output looks a familiar English dependency tree. For forward application and (generalized) forward composition, we define the head to be the left argument of the combinatory rule, unless it matches either X/X or X/(X\Y ), in which case the right argument is the head. For example, on “Black Monday” in Figure 4a we choose Monday as the head of Black. For the backward rules, the conversions are defined as the reverse of the corresponding forward rules. For other rules, RemovePunctuation (rp) chooses the non punctuation argument as a head, while Conjunction (Φ) chooses the right argument.3
One issue when applying this method for ob-
3When applying LEWISRULE to Japanese, we ignore the identity of the feature values in determining the head argument; In “tabe ta” (eat PAST), the category of auxiliary verb “ta” is Sf1\Sf2 with f1 ̸= f2, thus Sf1 ̸= Sf2 . Though it is notX\X , we define “ta” is headed by “tabe”, as removing the feature values, it matches X\X .
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
No , it was n′t Black Monday .
S/S , NP (S\NP)/NP (S\NP )\(S\NP ) NP/NP NP . <B× >
S\NP/NP NP >
S\NP < S rp S >
S rp
S
(a) English sentence
I SUB English ACC speak want . Boku wa eigo wo hanasi tai .
NP NP\NP NP NP\NP (S\NP)\NP S\S S\S < < <B2
NP NP (S\NP)\NP <
S\NP <
S <
S
(b) Japanese sentence “I want to speak English.”
No , it was n’t Black Monday .
(c) LEWISRULE
No , it was n’t Black Monday .
(d) HEADFIRST
Boku wa eigo wo hanasi tai .
(e) HEADFINAL
Figure 4: Examples of applying conversion rules in Section 4 to English and Japanese sentences.
taining the training data is that due to the mismatch between the rule set of our CCG parser, for which we follow Lewis and Steedman (2014), and the grammar in English CCGbank (Hockenmaier and Steedman, 2007) we cannot extract dependencies from some of annotated CCG trees.4 For this reason, we instead obtain the training data for this method from the original dependency annotations on CCGbank. Forutnatelly the dependency annotations of CCGbank matches LEWISRULE above in most cases and thus they can be a good approximation to it.
HEADFINAL Among SOV languages, Japanese is known as a strictly head final language, meaning that the head of every word always follows it. Japanese dependency parsing (Uchimoto et al., 1999; Kudo and Matsumoto, 2002) has exploited this property explicitly by only allowing left-toright dependency arcs. Inspired by this tradition, we try a simple HEADFINAL rule in Japanese CCG parsing, in which we always select the right argument as a head. For example we obtain the head final dependency tree in Figure 4e from the Japanese CCG tree in Figure 4b.
HEADFIRST We apply the similar idea as HEADFINAL into English. Since English has the opposite, SVO word order, we define the simple “head first” rule, in which the left argument always becomes the head (Figure 4d). Though this conversion may look odd at first sight it also has some advantages over LEWIS4 For example, the combinatory rules in Lewis and Steedman (2014) do not contain Nconj → N N in CCGbank. Another difficulty is that in English CCGbank a combinatory rule name is not annotated explicitly.
RULE. First, since the model with LEWISRULE is trained on the CCGbank dependencies, at inference, occasionally the two components Pdep and Ptag cause some conflicts on their predictions. For example, the true Viterbi parse may have a lower score in terms of dependencies, in which case the parser slows down and may degrade the accuracy. HEADFIRST, in contract, does not suffer from such conflicts. Second, by forcing the direction of arcs, the prediction of heads becomes easier, meaning that the dependency predictions become more reliable. Later we show this is in fact the case for existing dependency parsers (see Section 5), and in practice, we find that this simple conversion rule leads to the higher parsing scores than LEWISRULE on English (Section 6).
5 Tri-training
We extend the existing tri-training method to our models and apply it to our English parsers. Tri-training is one of the semi-supervised methods, in which the outputs of two parsers on unlabeled data are intersected to create (silver) new training data. This method is successfully applied to dependency parsing (Weiss et al., 2015) and CCG supertagging (Lewis et al., 2016). We simply combine the two previous approaches. Lewis et al. (2016) obtain their silver data annotated with the high quality supertags. Since they make this data publicly available 5, we obtain our silver data by assigning dependency structures on top of them.6
5https://github.com/uwnlp/taggerflow 6We annotate POS tags on this data using Stanford POS
tagger (Toutanova et al., 2003).
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
We train two very different dependency parsers from the training data extracted from CCGbank Section 02-21. This training data differs depending on our dependency conversion strategy (Section 4). For LEWISRULE, we extract the original dependency annotations of CCGbank. For HEADFIRST, we extract the head first dependencies from the CCG trees. Note that we cannot annotate dependency labels so we assign a dummy “none” label to every arc. The first parser is graph-based RBGParser (Lei et al., 2014) with the default settings except that we train an unlabeled parser and use word embeddings of Turian et al. (2010). The second parser is transition-based lstm-parser (Dyer et al., 2015) with the default parameters. On the development set (Section 00), with LEWISRULE dependencies RBGParser shows 93.8% unlabeled attachment score while that of lstm-parser is 92.5% using gold POS tags. Interestingly, the parsers with HEADFIRST dependencies achive higher scores: 94.9% by RBGParser and 94.6% by lstm-parser, suggesting that HEADFIRST dependencies are more parsable. For both dependencies, we obtain more than 1.7 million sentences on which two parsers agree. Following Lewis et al. (2016), we include 15 copies of CCGbank training set when using these silver data. Also to make effects of the tritrain samples smaller we multiply their loss by 0.4.
6 Experiments
We perform experiments on English and Japanese CCGbanks.
6.1 English Experimental Settings
We follow the standard data splits and use Sections 02-21 for training, Section 00 for development, and Section 23 for final evaluation. We report labeled and unlabeled F1 of the extracted CCG semantic dependencies obtained using generate program supplied with C&C parser.
For our models, we adopt the pruning strategies in Lewis and Steedman (2014) and allow at most 50 categories per word, use a variable-width beam with β = 0.00001, and utilize a tag dictionary, which maps frequent words to the possible supertags7. Unless otherwise stated, we only al-
7We use the same tag dictionary provided with their biLSTM model.
low normal form parses (Eisner, 1996; Hockenmaier and Bisk, 2010), choosing the same subset of the constaints as Lewis and Steedman (2014). We use as word representation the concatenatination of word vectors initialized to GloVe8 (Pennington et al., 2014), and randomly initialized prefix and suffix vectors of the length 1 to 4, which is inspired by Lewis et al. (2016). All affixes appearing less than two times in the training data are mapped to “UNK”. Other model configurations are: 4-layer biLSTMs with left and right 300-dimensional LSTMs, 1-layer 100-dimensional MLPs with ELU non-linearity (Clevert et al., 2015) for all MLP depchild, MLP dep head, MLP tag child and MLP tag head, and the Adam optimizer with β1 = 0.9, β2 = 0.9, L2 norm (1e−6), and lerning rate decay with the ratio 0.75 for every 2,500 iteration starting from 2e−3, which is shown to be effective for training the biaffine parser (Dozat and Manning, 2016).
6.2 Japanese Experimental Settings
We follow the default train/dev/test splits of Japanese CCGbank (Uematsu et al., 2013). For the baselines, we use an existing shift-reduce CCG parser implemented in an NLP tool Jigg9 (Noji and Miyao, 2016), and our implementation of the supertag-factored model using bi-LSTMs. For Japanese, we use as word representation the concatenatination of word vectors initialized to Japanese Wikipedia Entity Vector10, and 100- dimensional vectors computed from randomly initialized 50-dimensional character embeddings through convolution (dos Santos and Zadrozny, 2014). We do not use affix vectors as affixes are less informative in Japanese. All characters appearing less than two times are mapped to “UNK”. We use the same parameter settings as English for bi-LSTMs, MLPs, and optimization. One issue in Japanese experiments is evaluation. The Japanese CCGbank is encoded in a different format than the English bank, and no standalone script for extracting semantic dependencies is available yet. For this reason, we evaluate the parser outputs by converting them to bunsetsu dependencies, the syntactic representation ordinary used in Japanese NLP (Kudo andMatsumoto, 2002). Given a CCG tree, we obtain this by first
8http://nlp.stanford.edu/projects/glove/ 9https://github.com/mynlp/jigg
10http://www.cl.ecei.tohoku.ac.jp/∼msuzuki/jawiki vector/
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Method Labeled Unlabeled CCGbank LEWISRULE w/o dep 85.8 91.7 LEWISRULE 86.0 92.5 HEADFIRST w/o dep 85.6 91.6 HEADFIRST 86.6 92.8
Tri-training LEWISRULE 86.9 93.0 HEADFIRST 87.6 93.3
Table 1: Parsing results (F1) on English development set. “w/o dep” means that the model discards dependency components at prediction.
Method Labeled Unlabeled # violations CCGbank LEWISRULE w/o dep 85.8 91.7 2732 LEWISRULE 85.4 92.2 283 HEADFIRST w/o dep 85.6 91.6 2773 HEADFIRST 86.8 93.0 89 Tri-training LEWISRULE 86.7 92.8 253 HEADFIRST 87.7 93.5 66
Table 2: Parsing results (F1) on English development set when excluding the normal form constraints. # violations is the number of combinations violating the constraints on the outputs.
segment a sentence into bunsetsu (chunks) using CaboCha11 and extract dependencies that cross a bunsetsu boundary after obtaining the word-level, head final dependencies as in Figure 4b. For example, the sentence in Figure 4e is segmented as “Boku wa | eigo wo | hanashi tai”, from which we extract two dependencies (Boku wa) ← (hanashi tai) and (eigo wo) ← (hanashi tai). We perform this conversion for both gold and output CCG trees and calculate the (unlabeled) attachment accuracy. Though this is imperfect, it can detect important parse errors such as attachment errors and thus can be a good proxy for the performance as a CCG parser.
6.3 English Parsing Results
Effect of Dependency We first see how the dependency components added in our model affect the performance. Table 1 shows the results on the development set with the several configurations, in which “w/o dep” means discarding the dependency terms of the model and applying the attach low heuristics (Section 1) instead (i.e., a supertagfactored model; Section 2.1). We can see that for
11http://taku910.github.io/cabocha/
both LEWISRULE and HEADFIRST, adding dependency terms improves the performance.
Choice of Dependency Conversion Rule To our surprise, our simple HEADFIRST strategy always leads to better results than the linguistically motivated LEWISRULE. The absolute improvements by tri-training are equally large (about 1.0 points), suggesting that our model with dependencies can also be benefited from the silver data.
Excluding Normal Form Constraints One advantage of HEADFIRST is that the direction of arcs is always right, making the structures simpler and more parsable (Section 5). From another viewpoint, this fixed direction means that the constituent structure behind a (head first) dependency tree is unique. Since the constituent structures of CCGbank trees basiclly follow the normal form (NF), we hypothesize that the model learned with HEADFIRST has an ability to force the outputs in NF automatically. We summarize the results without the NF constraints in Table 2, which shows that the above argument is correct; the number of violating NF rules on the outputs of HEADFIRST is much smaller than that of LEWISRULE (89 vs. 283). Interestingly the scores of HEADFIRST slighly increase from the models with NF (e.g., 86.8 vs. 86.6 for CCGbank), suggesting that the NF constraints hinder the search of HEADFIRST models occasionally.
Results on Test Set Parsing results on the test set (Section 23) are shown in Table 3, where we compare our best-performance HEADFIRST dependency model without NF constraints with the several existing parsers. In CCGbank experiment, our parser shows the better result than all the baseline parsers except C&C with LSTM supertagger (Vaswani et al., 2016). Our parser outper-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
neuralccg EasySRL reimpl Ours Tagging 21.7 14.0 8.6 A* Search 16.7 200.6 89.1 Total 9.33 13.1 7.9
Table 4: Results of the efficiency experiment, where each number is the number of sentences processed per second. We compare our proposed parser against neuralccg and our reimplementation of EasySRL.
forms EasySRL by 0.5% and our reimplementation of that parser (EasySRL reimpl) by 0.9% in terms of labeled F1. In tri-training experiment, our parser shows much increased performance of 88.8% labeled F1 and 94.0% unlabeled F1, outperforming the current state-of-the-art neuralccg (Lee et al., 2016) that uses recursive neural networks by 0.1 point and 0.3 point in terms of labeled and unlabeled F1. This is the best reported F1 in English CCG parsing.
Efficiency Comparison We compare the efficiency of our parser with neuralccg and EasySRL reimpl.12 The results are shown in Table 4. Our parser lags behind in the overall parsing speed to neuralccg and EasySRL reimpl. When we go into the details, our supertagger runs slower than those of neuralccg and EasySRL reimpl, while in A* search, our parser processes over 5 times more sentences than neuralccg. The delay in supertagging can be attributed to several factors, in particular, the implementation (Python vs. C++) and the number of parameters, especially the number of bi-LSTM layers (4 vs. 2). We note that there are many implementation differences in our parsers (C++ A* parser with neural network model implemented with Chainer (Tokui et al., 2015)) and neuralccg (Java parser with C++ TensorFlow (Abadi et al., 2015) supertagger and recursive neural model in C++ DyNet (Neubig et al., 2017))13.
6.4 Japanese Parsing Result
We show the results of the Japanese parsing experiment in Table 5. The simple application of Lewis
12This experiment is performed on a laptop with 4-thread 2.0 GHz CPU.
13 There seems to be a room for optimizing our parser’s efficiency. We found that our EasySRL reimpl is slower than the original implementation of the supertag-factored model by Lewis et al. (2016), which neuralccg uses internally.
Method Category Bunsetsu Dep. Noji and Miyao (2016) 93.0 87.5 Supertag model 93.7 81.5 LEWISRULE (Ours) 93.8 90.8 HEADFINAL (Ours) 94.1 91.5
Table 5: Results of Japanese CCGbank.
Yesterday buy−PAST curry−ACC eat−PAST Kinoo kat − ta karee − wo tabe − ta S/S S NP S\NP
> S un
NP/NP >
NP <
S
Yesterday buy−PAST curry−ACC eat−PAST Kinoo kat − ta karee − wo tabe − ta S/S S NP S\NP
un NP/NP
> NP
< S
> S
Figure 5: Examples of ambiguous Japanese sentence given fixed supertags. The English translation is “I ate the curry I bought yesterday”.
et al. (2016) (Supertag model) is not effective for Japanese, showing the lowest attachment score of 81.5%. We observe a performance boost with our method, especially with HEADFINAL dependencies, which outperforms the baseline shift-reduce parser by 1.1 points on category assignments and 4.0 points on attachments. The degraded results of the simple application of the supertag-factored model can be attributed to the fact that the structure of a Japanese sentence is still highly ambiguous given the supertags (Figure 5). This is particularly the case in constructions where phrasal adverbial/adnominal modifiers (with the supertag S/S) are involved. The result suggests the importance of modeling dependencies in some languages, at least Japanese.
7 Conclusion
In this work, we have proposed a new CCG A* parsing method, in which the probability of a CCG tree is decomposed into local factors of the CCG categories and its dependency structure. By explicitly modeling the dependency structure, we do not require any deterministic heuristics to resolve attachment ambiguities, and keep the model factored so that all the probabilities can be precomputed before running the search. Our parser efficiently finds the the optimal parses, achieving the state-of-the-art performance in both English and Japanese parsing.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper describes a state-of-the-art CCG parsing model that decomposes into tagging and dependency scores, and has an efficient A* decoding algorithm. Interestingly, the paper slightly outperforms Lee et al. (2016)'s more expressive global parsing model, presumably because this factorization makes learning easier. It's great that they also report results on another language, showing large improvements over existing work on Japanese CCG parsing. One surprising original result is that modeling the first word of a constituent as the head substantially outperforms linguistically motivated head rules.   Overall this is a good paper that makes a nice contribution. I only have a few suggestions: - I liked the way that the dependency and supertagging models interact, but it would be good to include baseline results for simpler variations (e.g. not conditioning the tag on the head dependency). - The paper achieves new state-of-the-art results on Japanese by a large margin. However, there has been a lot less work on this data - would it also be possible to train the Lee et al. parser on this data for comparison? - Lewis, He and Zettlemoyer (2015) explore combined dependency and supertagging models for CCG and SRL, and may be worth citing.