Neural Modeling of Multi-Predicate Interactions for Japanese Predicate Argument Structure Analysis
The accuracy of Japanese predicate argument structure (PAS) analysis has improved in recent years thanks to joint modeling of interactions between multiple predicates. However, this approach heavily relies on syntactic information predicted by parsers, and suffers from the error propagation. To remedy this problem, we introduce a model using grid-type recurrent neural networks (Grid-RNN), which automatically induces features sensitive to multi-predicate interactions from word sequence information of a sentence. The experiments on the NAIST Text Corpus show that our model exceeds the accuracy of the state-of-the-art Japanese PAS analyzer without syntactic information.
1 Introduction
Predicate argument structure (PAS) analysis is a basic semantic analysis task, in which systems are required to identify semantic units of a sentence, such as who did what to whom. In pro-drop languages such as Japanese, Chinese and Italian, arguments are often omitted in text, and such argument omission is regarded as one of the most problematic issues of PAS analysis (Iida and Poesio, 2011; Sasano and Kurohashi, 2011).
As an approach to the argument omission problem, in Japanese PAS analysis, joint modeling of interactions between multiple predicates has been gaining popularity and achieved the state-of-theart result (Ouchi et al., 2015; Shibata et al., 2016). This approach is based on the linguistic intuition that the predicates in a sentence are semantically related to each other and the interaction information can be a clue for PAS analysis. However, to model such multi-predicate interactions, this ap-
proach heavily relies on syntactic information predicted by parsers and suffers from the error propagation caused by the pipeline processing. To remedy this problem, we propose a neural model which automatically induces features sensitive to multi-predicate interactions from word sequence information of a sentence. This model takes as input all predicates and their argument candidates in a sentence at a time, and captures the interactions using grid-type recurrent neural networks (Grid-RNN) without syntactic information. In this paper, we firstly introduce a basic model using RNNs, which independently estimates arguments of each predicate without considering the multi-predicate interactions (Sec. 3). Then, extending this model, we propose a neural model using Grid-RNNs (Sec. 4). Performing experiments on the NAIST Text Corpus (Iida et al., 2007), we demonstrate that our neural models exceed the accuracy of the stateof-the-art Japanese PAS analyzer (Ouchi et al., 2015). In particular, the neural model using GridRNNs achieves the best result, which suggests that our grid-type neural architecture effectively captures multi-predicate interactions. 1
1Our source code is publicly available at http:xxx
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Figure 2: An overview of the neural models: (i) single-sequence and (ii) multi-sequence models.
2 Japanese Predicate Argument Structure Analysis
2.1 Task Description
In Japanese PAS analysis, we identify arguments taking part in the three major case roles, nominative (NOM), accusative (ACC) and dative (DAT) cases, for each predicate. Arguments can be divided into the following three categories according to the positions relative to their predicates (Hayashibe et al., 2011; Ouchi et al., 2015):
Dep: The arguments that have direct syntactic dependency with the predicate.
Zero: The arguments referred to by zero pronouns within the same sentence, which have no direct syntactic dependency with the predicate.
Inter-Zero: The arguments referred to by zero pronouns out of the same sentence.
For example, in Figure 1, the nominative argument “警察 (police)” for the predicate “逮捕した (arrested)” is regarded as a Dep argument since the argument has a direct syntactic dependency with the predicate. In contrast, the nominative argument “男 i (mani)” for the predicate “逃走した (escaped)” is regarded as a Zero argument since the argument has no direct syntactic dependency with the predicate.
In this paper, we focus on the analysis for these intra-sentential arguments, i.e., Dep and Zero. In order to identify inter-sentential arguments (Inter-Zero), it is required to search a much broader space, such as the whole document, resulting in a much harder analysis than intrasentential arguments.2 Thus, Ouchi et al. (2015)
2The F-measure remains 10-20% (Taira et al., 2008; Imamura et al., 2009; Sasano and Kurohashi, 2011).
and Shibata et al. (2016) focused on only intrasentential argument analysis. Following this trend, we focus on intra-sentential argument analysis.
2.2 Challenging Problem
Arguments are often omitted in Japanese sentences. In Figure 1, ϕi represents the omitted argument, called zero pronoun. This zero pronoun ϕi refers to “男 i (mani)”. In Japanese PAS analysis, when an argument of the target predicate is omitted, we have to identify the antecedent of the omitted argument (Zero argument). The analysis for such Zero arguments is much more difficult than that forDep arguments because of the lack of direct syntactic dependencies. For Dep arguments, the syntactic dependency between an argument and its predicate is a strong clue. In the sentence in Figure 1, for the predicate “逮捕し た (arrested)”, the nominative argument is “警察 (police)”. This argument can easily be identified by relying on the syntactic dependency. In contrast, since the nominative argument “男 i (mani)” has no syntactic dependency with its predicate “逃 走した (escaped)”, we have to use other information for such zero argument identification. As an solution to this problem, we exploit two kinds of information: (i) context in the entire sentence and (ii) multi-predicate interactions. For the former, we introduce single-sequence model, which induces context-sensitive representations from a sequence of argument candidates of a predicate. For the latter, we introduce multi-sequence model, which induces predicate-sensitive representations from multiple sequences of argument candidates of all predicates in a sentence (shown in Figure 2).
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 3: The overall architecture of the single sequence model. This model consists of three components: (i) Input Layer, (ii) RNN Layer and (iii) Output Layer.
3 Single-Sequence Model
The single-sequence model exploits stacked bidirectional RNNs (Bi-RNN) (Schuster and Paliwal, 1997; Graves et al., 2005, 2013; Zhou and Xu, 2015). Figure 3 shows the overall architecture, which consists of the following three components:
Input Layer: Map each word to a feature vector representation.
RNN Layer: Produce high-level feature vectors using Bi-RNNs.
Output Layer: Compute the probability of each case label for each word using the softmax function.
In the following subsections, we describe each of them in more detail.
3.1 Input Layer
Given an input sentence w1:T = (w1, · · · , wT) and a predicate p, each word wt is mapped to a feature representation xt, which is the concatenation (⊕) of three types of vectors:
xt = x arg t ⊕ x pred t ⊕ xmarkt (1)
where each vector is based on the following atomic features inspired by Zhou and Xu (2015):
ARG: Word index of each word. PRED: Word index of the target predicate and
words around the predicate.
MARK: Binary index that represents whether the word is the predicate or not.
Figure 4: An example of the feature extraction. The underlined word is the target predicate. From the sentence “彼女はパンを食べた。(She ate a bread.)”, the three types of features are extracted for the target predicate “食べた (ate)”.
Figure 5: An example of the process of creating a feature vector. The extracted features are mapped to each vector, and all the vectors are concatenated into one feature vector.
Figure 4 presents an example of the atomic features. As the ARG feature, we extract a word index xword ∈ V of each word. Similarly, as the PRED feature, we extract each word index xword of the C words taking the target predicate at the center, where C is the window size. The MARK feature xmark ∈ {0, 1} is a binary value that represents whether the word is the predicate or not. Then, using feature indices, we extract feature vector representations from each embedding matrix. Figure 5 shows the process of creating the feature vector x1 for the word w1 “彼女 (she)”. We set two embedding matrices: (i) word embedding matrix Eword ∈ Rdword×|V|, and (ii) mark embedding matrixEmark ∈ Rdmark×2. From each embedding matrix, we extract corresponding column vectors and concatenate them as a feature vector representation xt based on Eq. 1.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Each feature vector xt is multiplied with a parameter matrix Wx:
h (0) t = Wx xt (2)
The vector h(0)t is given to the first layer of the RNN layers as input.
3.2 RNN Layer
In the RNN layers, feature vectors are updated recurrently using Bi-RNNs. Bi-RNNs process an input sequence from the left-to-right manner in odd-numbered layers and the opposite in evennumbered layers. By stacking these layers, we can construct the deeper network structures.
Stacked Bi-RNNs consist of L layers, and the hidden state in the layer ℓ ∈ (1, · · · , L) is calculated as follows:
h (ℓ) t =
{ g(ℓ)(h
(ℓ−1) t , h (ℓ) t−1) (ℓ = odd)
g(ℓ)(h (ℓ−1) t , h (ℓ) t+1) (ℓ = even)
(3)
Both of the odd and even-numbered layers receive h(ℓ−1)t , the t-th hidden state of the ℓ − 1 layer, as the first input of the function g(ℓ), which is an arbitrary function 3. As the second input of g(ℓ), odd-numbered layers receive h(ℓ)t−1 while even-numbered layers receive h(ℓ)t+1. By calculating the hidden states until the L-th layer, we obtain a hidden state sequence h(L)1:T = (h (L) 1 , · · · ,h (L) T ). Using each vector h(L)t , we calculate the probability of case labels for each word in the output layer.
3.3 Output Layer
In the output layer, multi-class classification is performed using the softmax function:
yt = softmax(Wy h (L) t )
where h(L)t is a vector representation propagated from the last RNN layer (Fig 3). Each element of yt is a probability value corresponding to each label. The label with the maximum probability among them is output as a result. In this task, there are five labels: NOM, ACC, DAT, PRED, null. The labels NOM, ACC and DAT indicate the nominative, accusative and dative case, respectively. PRED is the label for the predicate. null represents a word that does not play any case role.
3In this work, we use the Gated Recurrent Unit (GRU) (Cho et al., 2014) as the function g(ℓ).
4 Multi-Sequence Model
While the single-sequence model assumes the independence between predicates, the multisequence model assumes the multi-predicate interactions. To capture such interactions between all predicates in a sentence, we extend the singlesequence model to the multi-sequence model using Grid-RNNs (Graves and Schmidhuber, 2009; Kalchbrenner et al., 2016). Figure 6 presents the overall architecture of the multi-sequence model, which consists of three components:
Input Layer: Map words to M sequences of feature vectors for M predicates.
Grid Layer: Update the hidden states over different sequences using Grid-RNNs.
Output Layer: Compute the probability of each case label for each word using the softmax function.
In the following subsections, we describe them in more detail.
4.1 Input Layer
The multi-sequence model takes as input a sentence w1:T = (w1, · · · , wT) and all predicates {pm}M1 in the sentence. For each predicate pm, the input layer creates a sequence of feature vectors Xm = (xm,1, · · · ,xm,T) by mapping each input word wt to a feature vector xm,t based on Eq 1. That is, for M predicates, M sequences of feature vectors {Xm}M1 are created. Then, using Eq. 2, each feature vector xm,t is mapped to h(0)m,t, and a feature sequence is created for a predicate pm, i.e.,H (0) m = (h (0) m,1, · · · ,h (0) m,T). Consequently, forM predicates, we obtainM feature sequences {H(0)m }M1 .
4.2 Grid Layer
Inter-Sequence Connections
In the grid layers, we use Grid-RNNs to propagate the feature information over the different sequences (inter-sequence connections). The right figure in Figure 6 shows an odd-numbered layer of the Grid layers. The hidden state is recurrently calculated from upper-left (m = 1, t = 1) to lowerright (m = M, t = T). Formally, in the ℓ-th layer, the hidden state h(ℓ)m,t is calculated as follows:
h (ℓ) m,t=
{ g(ℓ)(h
(ℓ−1) m,t ⊕ h (ℓ) m−1,t,h (ℓ) m,t−1) (ℓ = odd)
g(ℓ)(h (ℓ−1) m,t ⊕ h (ℓ) m+1,t,h (ℓ) m,t+1) (ℓ = even)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
This equation is similar to Eq. 3. The main difference is that the hidden state of a neighboring sequence, h(ℓ)m−1,t (or h (ℓ) m+1,t), is concatenated (⊕) with the hidden state of the previous (ℓ− 1) layer, h (ℓ−1) m,t , and is taken as input of the function g
(ℓ). In the right figure in Figure 6, the blue curve lines represent the inter-sequence connections. Taking as input the hidden states of neighboring sequences, the network propagates feature information over multiple sequences (predicates). By calculating the hidden states until the L-th layer, we obtain M sequences of the hidden states, i.e., {H(L)m }M1 , in which H (L) m = (h (L) m,1, · · · ,h (L) m,T).
Residual Connections
As more layers are stacked, it gets more difficult to learn the model parameters due to some problems such as gradient vanishment (Pascanu et al., 2013). In this work, we integrate residual connections (He et al., 2015;Wu et al., 2016) with our networks to connect between layers. Specifically, the input vector h(ℓ−1)m,t of the ℓ-th layer is added to the output vector h(ℓ)m,t. Residual connections can also be applied to the single-sequence model, and thus we perform the experiments on both models with/without residual connections.
4.3 Output Layer
Like the single-sequence model, using the softmax function, we calculate the probability of case labels of each word wt for each predicate pm:
ym,t = softmax(Wy h (L) m,t)
where h(L)m,t is a hidden state vector calculated in the last Grid Layer.
5 Related Work
5.1 Japanese PAS Analysis Approaches
Existing approaches for Japanese PAS analysis are divided into two categories: (i) pointwise approach and (ii) joint approach. In the pointwise approach, we estimate the score of each argument candidate for one predicate, and select the argument candidate with the maximum score as an argument (Taira et al., 2008; Imamura et al., 2009; Hayashibe et al., 2011; Iida et al., 2016). In the joint approach, we select the score of all the predicate-argument combinations in one sentence, and select the combination with the highest score (Yoshikawa et al., 2011; Sasano and Kurohashi, 2011; Ouchi et al., 2015; Shibata et al., 2016). Compared with the pointwise approach, the joint approach has achieved better results.
5.2 Multi-Predicate Interactions
Ouchi et al. (2015) reported that it is beneficial to Japanese PAS analysis to capture the interactions between all predicates in a sentence. This is based on the linguistic intuition that the predicates in a sentence are semantically related to each other, and the interaction information can be a clue for PAS analysis. Similarly, in semantic role labeling (SRL), Yang and Zong (2014) also reported that their reranking model capturing the multi-predicate interactions is effective for the English constituentbased SRL task (Carreras and Màrquez, 2005). Taking a step further in this direction, we propose the neural architecture that effectively models the multi-predicate interactions.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
5.3 Neural Approaches
Japanese PAS
In recent years, several attempts have been made to apply neural networks for Japanese PAS analysis (Shibata et al., 2016; Iida et al., 2016)4. Shibata et al. (2016) used a feed-forward neural network for the score calculation part of the joint model proposed by Ouchi et al. (2015). Iida et al. (2016) used multi-column convolutional neural networks for the zero anaphora resolution task.
Both models exploited syntactic and selectional preference information as atomic features of neural networks. Using neural networks, the good performance was realized with mitigating the cost of manually designing combination features. In this work, we demonstrate that even without such syntactic information, our neural models realize the state-of-the-art performance by using word sequence information of a sentence.
English SRL
Some neural models achieved high performance without syntactic information in English SRL. Collobert et al. (2011) worked on the English constituent-based SRL task (Carreras and Màrquez, 2005). Their model exploited a convolutional neural network and achieved 74.15% in F-measure without syntactic information. Zhou and Xu (2015) also worked on the same task using bidirectional RNNs with CRF and achieved the state-of-the-art result, 81.07% in F-measure. Our models can be regarded as an extension from their model.
The main differences between Zhou and Xu (2015) and our work are: (i) the constituent vs dependency-based argument identification and (ii) the multi-predicate consideration. In the constituent-based SRL, since systems are required to identify the spans of arguments for each predicate, Zhou and Xu (2015) used CRF to capture the IOB label dependencies. In contrast, in Japanese dependency-based PAS analysis, since arguments are infrequently adjacent to each other, we replaced the CRF with the softmax function. Also, while the model of Zhou and Xu (2015) predicts arguments for each predicate independently, our multi-sequence model jointly predicts arguments for all predicates in a sentence at a time by considering the multi-predicate interactions.
4These previous studies used unpublished datasets and evaluated the performance with different experimental settings, so we cannot compare their models with ours.
6 Experiments
6.1 Experimental Settings
Dataset
We use the NAIST Text Corpus 1.5, which consists of 40,000 sentences of Japanese newspaper text (Iida et al., 2007). In the experiments, we adopt the standard data splits (Taira et al., 2008; Imamura et al., 2009; Ouchi et al., 2015):
Train: Articles: Jan 1-11, Editorials: Jan-Aug Dev: Articles: Jan 12-13, Editorials: Sept Test: Articles: Jan 14-17, Editorials: Oct-Dec
We use the word boundaries annotated to the NAIST Text Corpus and the target predicates that have at least one argument in the same sentence. We do not use any external resources.
Learning
We train the model parameters by minimizing the cross-entropy loss function:
L(θ) = − ∑ n ∑ t logP (yt|xt) + λ 2 ||θ||2
where θ is a set of model parameters, and the hyper-parameter λ is the coefficient governing the L2 weight decay.
Implementation Details
We implement our neural models using Theano (Bastien et al., 2012). The number of epochs is set to 50, and we report the result of the test set in the epoch with the best F-measure of the development set. Parameter optimization is done by stochastic gradient descent method (SGD) using mini-batch, whose size is selected from {2, 4, 8}. The learning rate is automatically adjusted using Adam (Kingma and Ba, 2014). For the L2 weight decay, the hyper-parameter λ in Eq. 4 is selected from {0.001, 0.0005, 0.0001}. In the neural models, the number of the RNN and Grid layers are selected from {2, 4, 6, 8}. The window size C for the PRED feature (Sec. 3.1) is set to 5. Words with frequency 2 or more are mapped to each word index, and the remaining words are mapped to the unknown word index. The dimensions dword and dmark of the embeddings are set to 32. In the single-sequence model, the parameters of GRUs are set to 32× 32. In the multi-sequence model, the parameters of GRUs related to the input values are set to 64 × 32, and
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Dep Zero All Imamura+ 09 85.06 41.65 78.15
Ouchi+ 15 86.07 44.09 79.23 Single-Seq 88.10 46.10 81.15 Multi-Seq 88.17 † 47.12 † 81.42 †
Table 1: F-measures in the test set. Single-Seq is the single-sequence model, and Multi-Seq is the multi-sequence model. Imamura+ 09 is the model of Imamura et al. (2009) reimplemented by Ouchi et al. (2015), and Ouchi+ 15 is the ALLCases Joint Model of Ouchi et al. (2015). The mark † denotes the significantly better results with the significance level p < 0.05 comparing between Single-Seq and Multi-Seq.
others are 32× 32. The initial values of all the parameters are sampled according to the uniform distribution from [− √ 6√
row+col ,
√ 6√
row+col ], where row
and col are the number of rows and columns of each matrix, respectively.
Baseline Models
We compare our models with the models in the previous works (Sec. 5.1) that use the NAIST Text Corpus 1.5. As a baseline of the pointwise approach, we use the pointwise model5 of Imamura et al. (2009). In addition, as a baseline of the joint approach, we use the model of Ouchi et al. (2015), which achieved the best result on the NAIST Text Corpus 1.5.
6.2 Results
Neural Models vs Baseline Models
Table 1 presents F-measures of our neural sequence models with 8 RNN or Grid layers and the baseline models on the test set, in which as the significent test, we used the bootstrap resampling method. In all the metrics, both of the single-sequence (Single-Seq) and multisequence model (Multi-Seq) outperformed the baseline models. This confirms that our neural sequence models realize high-performance even without syntactic information by learning contextual information effective for PAS analysis from a word sequence of the sentence.
In particular, for zero arguments (Zero), our models achieved a considerable improvement compared with the state-of-the-art model of
5We compared the results of the model reimplemented by Ouchi et al. (2015).
Ouchi et al. (2015), i.e., the single model improved around 2.0 points and the multi-sequence model improved around 3.0 points in F-measure. These results suggest that it is beneficial to Japanese PAS analysis, particularly to the zero argument identification, to model the context in the entire sentence using RNNs.
Effects of Multiple Predicate Consideration
As Table 1 shows, the multi-sequence model significantly outperformed the single-sequence model in F-measure in total (81.42% vs 81.15%). This result demonstrates that the grid-type neural architecture can effectively capture the multipredicate interactions by connecting between the sequences of the argument candidates for all predicates in a sentence. Compared with the single-sequence model for different argument types, the multi-sequence model achieved slightly but significantly better result for the direct dependency arguments (Dep) (88.10% vs 88.17%). In addition, for zero arguments (Zero), which have no syntactic dependency with its predicate, the multi-sequence model significantly outperformed the single-sequence model by around 1.0 points in F-measure (46.10% vs 47.12%). This shows that capturing the multipredicate interactions is particularly effective for zero arguments, which is consistent with the results of Ouchi et al. (2015).
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Dep Zero NOM ACC DAT NOM ACC DAT
NAIST Text Corpus 1.5 Imamura+ 09 86.50 92.84 30.97 45.56 21.38 0.83
Ouchi+ 15 88.13 92.74 38.39 48.11 24.43 4.80 Single-Seq 88.32 93.89 65.91 49.51 35.07 9.83 Multi-Seq 88.75 93.68 64.38 50.65 32.35 7.52
NAIST Text Corpus 1.4β Taira+ 08* 75.53 88.20 89.51 30.15 11.41 3.66
Imamura+ 09* 87.0 93.9 80.8 50.0 30.8 0.0 Sasano+ 11* - - - 39.5 17.5 8.9
Table 3: Performance comparison for different case roles on the test set in F-measures. NOM, ACC or DAT is the nominal, accusative or dative case, respectively. The mark * indicates that the model uses external resources.
Effects of Network Depth
Table 2 presents F-measures of the neural sequence models with different network depths and with/without residual connections. The performance tends to get better as the RNN or Grid layers get deeper with residual connections. In particular, the two models with 8 layers and residual connections achieved considerable improvements of around 1.0 point in F-measure compared the models without residual connections, which means that the residual connections contribute to the effective parameter learning of deeper models.
Comparison per Case Role
Table 3 shows F-measures for each case role. For reference, we show the results of the previous studies using NAIST Text Corpus 1.4β with external resources as well.6
Comparing between the models using the NAIST Text Corpus 1.5, the single-sequence and multi-sequence models outperformed the baseline models in all the metrics. In particular, for the dative case, the two neural models achieved much higher results by around 30 points. This suggests that although dative arguments appear infrequently compared with the other two case arguments, the neural models can robustly learn it.
In addition, for zero arguments (Zero), the neural models achieved better results than the baseline models. Especially, for zero arguments of
6The major difference between NAIST Text Corpus 1.4β and 1.5 is the revision of the annotation criterion for the dative case (DAT) (corresponding to Japanese case marker “に”). Argument and adjunct usages of the case marker “に” are not distinguished in 1.4β, making the identification of the dative case seemingly easy (Ouchi et al., 2015).
the nominative case (NOM), the multi-sequence model achieved a considerable improvement of around 2.5 points in F-measure compared with the state-of-the-art model of Ouchi et al. (2015). To achieve high accuracies for the analysis of such zero arguments, it is necessary to capture long distance dependencies (Iida et al., 2005; Sasano and Kurohashi, 2011; Iida et al., 2015). Therefore, the improvements of the results suggest that the neural models effectively capture long distance dependencies using RNNs that can encode the context in the entire sentence.
7 Conclusion
In this work, we introduced neural sequence models that automatically induce effective feature representations from word sequence information of a sentence for Japanese PAS analysis. The experiments on NAIST Text Corpus 1.5 demonstrated that the models achieve the state-of-the-art result without syntactic information. In particular, our multi-sequence model improved the performance for zero argument identification, one of the problematic issues in Japanese PAS analysis, by considering the multi-predicate interactions using Grid-RNNs. Since our neural models are applicable to SRL, applying our models for multilingual SRL tasks is an interesting line of the future research. In addition, in this work, the model parameters were learned without any external resources. For future work, we plan to explore effective methods for exploiting large-scale unlabeled data to learn the neural models.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper proposes a joint neural modelling approach to PAS analysis in Japanese, based on Grid-RNNs, which it compares variously with a conventional single-sequence RNN approach.  This is a solidly-executed paper, targeting a well-established task from Japanese but achieving state-of-the-art results at the task, and presenting the task in a mostly accessible manner for those not versed in Japanese. Having said that, I felt you could have talked up the complexity of the task a bit, e.g. wrt your example in Figure 1, talking through the inherent ambiguity between the NOM and ACC arguments of the first predicate, as the NOM argument of the second predicate, and better describing how the task contrasts with SRL (largely through the ambiguity in zero pronouns). I would also have liked to have seen some stats re the proportion of zero pronouns which are actually intra-sententially resolvable, as this further complicates the task as defined (i.e. needing to implicitly distinguish between intra- and inter-sentential zero anaphors). One thing I wasn't sure of here: in the case of an inter-sentential zero pronoun for the argument of a given predicate, what representation do you use? Is there simply no marking of that argument at all, or is it marked as an empty argument? My reading of the paper is that it is the former, in which case there is no explicit representation of the fact that there is a zero pronoun, which seems like a slightly defective representation (which potentially impacts on the ability of the model to capture zero pronouns); some discussion of this would have been appreciated.  There are some constraints that don't seem to be captured in the model (which some of the ILP-based methods for SRL explicitly model, e.g.): (1) a given predicate will generally have only one argument of a given type (esp. NOM and ACC); and (2) a given argument generally only fills one argument slot for a given predicate. I would have liked to have seen some analysis of the output of the model to see how well the model was able to learn these sorts of constraints. More generally, given the mix of numbers in Table 3 between Single-Seq and Multi-Seq (where it is really only NOM where there is any improvement for Multi-Seq), I would have liked to have seen some discussion of the relative differences in the outputs of the two models: are they largely identical, or very different but about the same in aggregate, e.g.? In what contexts do you observe differences between the two models? Some analysis like this to shed light on the internals of the models would have made the difference between a solid and a strong paper, and is the main area where I believe the paper could be improved (other than including results for SRL, but that would take quite a bit more work).  The presentation of the paper was good, with the Figures aiding understanding of the model. There were some low-level language issues, but nothing major:  l19: the error propagation to error propagation l190: an solution to a solution l264 (and Figure 2): a bread to bread l351: the independence to independence l512: the good to good l531: from their model to of their model l637: significent to significance l638: both of to both  and watch casing in your references (e.g.  japanese ,  lstm ,  conll ,  ilp )


Neural Machine Translation via Binary Code Prediction
In this paper, we propose a new method for calculating the output layer in neural machine translation systems. The method is based on predicting a binary code for each word, and can reduce computation time/memory requirements of the output layer to be logarithmic in vocabulary size in the best case. In addition, we also introduce two advanced approaches to improve robustness of the proposed model: using error-correcting codes and combining softmax and binary codes. Experiments show the proposed model achieves translation accuracies that approach the softmax, while reducing memory usage on the order of 1/10 to 1/1000, and also improving decoding speed on CPUs by x5 to x20.
1 Introduction
When handling broad or open domains, machine translation systems usually have to handle a large vocabulary as their inputs and outputs. This is particularly a problem in neural machine translation (NMT) models (Sutskever et al., 2014), such as the attention-based model (Bahdanau et al., 2014; Luong et al., 2015) shown in Figure 1. In these models, the output layer is required to generate a specific word from an internal vector, and a large vocabulary size tends to require a large amount of computation to predict each of the candidate word probabilities.
Because this is a significant problem for neural language and translation models, there are a number of methods proposed to resolve this problem, which we detail in Section 2.2. However, none of these previous methods simultaneously satisfies the following desiderata, all of which, we argue, are desirable for practical use in NMT systems:
Memory efficiency: The method should not require large memory to store the parameters and calculated vectors to maintain scalability in resource-constrained environments.
Time efficiency: The method should be able to train the parameters efficiently, and possible to perform decoding efficiently with choosing the candidate words from the full probability distribution. In particular, the method should be performed fast on general CPUs to suppress physical costs of computational resources for actual production systems.
Compatibility with parallel computation: It should be easy for the method to be minibatched and optimized to run efficiently on GPUs, which are essential for training large NMT models.
In this paper, we propose a method that satisfies all of these conditions: requires significantly less memory, fast, and is easy to implement minibatched on GPUs. The method works by not pre-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
dicting a softmax over the entire output vocabulary, but instead by encoding each vocabulary word as a vector of binary variables, then independently predicting the bits of this binary representation. In order to represent a vocabulary size of 2n, the binary representation need only be at least n bits long, and thus the amount of computation and size of parameters required to select an output word is only O(log V ) in the size of the vocabulary V , a great reduction from the standard linear increase of O(V ) seen in the original softmax.
While this idea is simple and intuitive, we found that it alone was not enough to achieve competitive accuracy with real NMT models. Thus we make two improvements: First, we propose a hybrid model, where the high frequency words are predicted by a standard softmax, and low frequency words are predicted by the proposed binary codes separately. Second, we propose the use of convolutional error correcting codes with Viterbi decoding (Viterbi, 1967), which add redundancy to the binary representation, and even in the face of localized mistakes in the calculation of the representation, are able to recover the correct word.
In experiments on two translation tasks, we find that the proposed hybrid method with error correction is able to achieve results that are competitive with standard softmax-based models while reducing the output layer to a fraction of its original size.
2 Problem Description and Prior Work
2.1 Formulation and Standard Softmax
Most of current NMT models use one-hot representations to represent the words in the output vocabulary – each word w is represented by a unique sparse vector eid(w) ∈ RV , in which only one element at the position corresponding to the word ID id(w) ∈ {x ∈ N | 1 ≤ x ≤ V } is 1, while others are 0. V represents the vocabulary size of the target language. NMT models optimize network parameters by treating the one-hot representation eid(w) as the true probability distribution, and minimizing the cross entropy between it and the softmax probability v:
LH(v, id(w)) := H(eid(w),v), (1) = log sum expu− uid(w), (2)
v := expu/ sum expu, (3)
u := Whuh+ βu, (4)
where sumx represents the sum of all elements in x, xi represents the i-th element of x, Whu ∈
RV×H and βu ∈ RV are trainable parameters and H is the total size of hidden layers directly connected to the output layer.
According to Equation (4), this model clearly requires time/space computation in proportion to O(HV ), and the actual load of the computation of the output layer is directly affected by the size of vocabulary V , which is typically set around tens of thousands (Sutskever et al., 2014).
2.2 Prior Work on Suppressing Complexity of NMT Models
Several previous works have proposed methods to reduce computation in the output layer. The hierarchical softmax (Morin and Bengio, 2005) predicts each word based on binary decision and reduces computation time to O(H log V ). However, this method still requires O(HV ) space for the parameters, and requires calculation much more complicated than the standard softmax, particularly at test time. The differentiated softmax (Chen et al., 2016) divides words into clusters, and predicts words using separate part of the hidden layer for each word clusters. This method make the conversion matrix of the output layer sparser than a fully-connected softmax, and can reduce time/space computation amount by ignoring zero part of the matrix. However, this method restricts the usage of hidden layer, and the size of the matrix is still in proportion to V .
Sampling-based approximations (Mnih and Teh, 2012; Mikolov et al., 2013) to the denominator of the softmax have also been proposed to reduce calculation at training. However, these methods are basically not able to be applied at test time, still require heavy computation like the standard softmax.
Other methods using characters (Ling et al., 2015) or subwords (Sennrich et al., 2016; Chitnis and DeNero, 2015) can be applied to suppress the vocabulary size, but these methods also make for longer sequences, and thus are not a direct solution to problems of computational efficiency.
3 Binary Code Prediction Models
3.1 Representing Words using Bit Arrays
Figure 2(a) shows the conventional softmax prediction, and Figure 2(b) shows the binary code prediction model proposed in this study. Unlike the conventional softmax, the proposed method predicts each output word indirectly using dense
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 2: Designs of output layers.
bit arrays that correspond to each word. Let b(w) := [b1(w), b2(w), · · · , bB(w)] ∈ {0, 1}B be the target bit array obtained for word w, where each bi(w) ∈ {0, 1} is an independent binary function given w, and B is the number of bits in whole array. For convenience, we introduce some constraints on b. First, a wordw is mapped to only one bit array b(w). Second, all unique words can be discriminated by b, i.e., all bit arrays satisfy that:
id(w) 6= id(w′)⇒ b(w) 6= b(w′). (5)
Third, multiple bit arrays can be mapped to the same word as described in Section 3.5. By considering second constraint, we can also constrain B ≥ dlog2 V e, because b should have at least V unique representations to distinguish each word. The output layer of the network independently predicts B probability values q := [q1(h), q2(h), · · · , qB(h)] ∈ [0, 1]B using the current hidden values h by logistic regressions:
q(h) = σ(Whqh+ βq), (6) σ(x) := 1/(1 + exp(−x)), (7)
where Whq ∈ RB×H and βq ∈ RB are trainable parameters. When we assume that each qi is the probability that “the i-th bit becomes 1,” the joint probability of generating word w can be represented as:
Pr(w|h) := B∏ i=1 ( biqi + b̄iq̄i ) , (8)
Algorithm 1 Mapping words to bit arrays. Require: w ∈ V Ensure: b ∈ {0, 1}B = Bit array representing w
x :=  0, if w = UNK 1, if w = BOS 2, if w = EOS 2 + rank(w), otherwise bi := bx/2i−1c mod 2 b← [b1, b2, · · · , bB ]
where x̄ := 1 − x. We can easily obtain the maximum-probability bit array from q by simply assuming bi = 1 if qi ≥ 1/2, and bi = 0 otherwise. However, this calculation may generate invalid bit arrays which do not correspond to actual words according to the mapping between words and bit arrays. For now, we simply assume that w = UNK (unknown) when such bit arrays are obtained, and discuss alternatives later in Section 3.5.
The constraints described here are very general requirements for bit arrays, which still allows us to choose between a wide variety of mapping functions. However, designing the most appropriate mapping method for NMT models is not a trivial problem. In this study, we use a simple mapping method described in Algorithm 1, which was empirically effective in preliminary experiments.1 Here, V is the set of V target words including 3 extra markers: UNK, BOS (begin-of-sentence), and EOS (end-of-sentence), and rank(w) ∈ N>0 is the rank of the word according to their frequencies in the training corpus. Algorithm 1 is one of the minimal mapping methods (i.e., satisfying B = dlog2 V e), and generated bit arrays have the characteristics that their higher bits roughly represents the frequency of corresponding words (e.g., if w is frequently appeared in the training corpus, higher bits in b(w) tend to become 0).
3.2 Loss Functions
For learning correct binary representations, we can use any loss function that satisfies a constraint that:
LB(q, b) { = 0, if q = b, ≥ 0, otherwise. (9)
For example, the squared-distance:
LB(q, b) := B∑ i=1 (qi − bi)2, (10)
1Other methods examined included random codes, Huffman codes (Huffman, 1952) and Brown clustering (Brown et al., 1992) with zero-padding to adjust code lengths, and some original allocation methods based on the word2vec embeddings (Mikolov et al., 2013).
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
or the cross-entropy:
LB(q, b) := − B∑ i=1 ( bi log qi + b̄i log q̄i ) , (11)
are candidates for the loss function. We also examined both loss functions in the preliminary experiments, and in this paper, we only used the squared-distance function (Equation (10)), because this function achieved higher translation accuracies than Equation (11).
3.3 Efficiency of the Binary Code Prediction
The computational complexity for the parameters Whq and βq is O(HB). This is equal to O(H log V ) when using a minimal mapping method like that shown in Algorithm 1, and is significantly smaller than O(HV ) when using standard softmax prediction. For example, if we chose V = 65536 = 216 and use Algorithm 1’s mapping method, then B = 16 and total amount of computation in the output layer could be suppressed to 1/4096 of its original size.
On a different note, the binary code prediction model proposed in this study shares some ideas with the hierarchical softmax (Morin and Bengio, 2005) approach. Actually, when we used a binarytree based mapping function for b, our model can be interpreted as the hierarchical softmax with two strong constraints for guaranteeing independence between all bits: all nodes in the same level of the hierarchy share their parameters, and all levels of the hierarchy are predicted independently of each other. By these constraints, all bits in b can be calculated in parallel. This is particularly important because it makes the model conducive to being calculated on GPGPUs.
However, the binary code prediction model also introduces problems of robustness due to these strong constraints. As the experimental results show, the simplest prediction model which directly maps words into bit arrays seriously decreases translation quality. In Sections 3.4 and 3.5, we introduce two additional techniques to prevent reductions of translation quality and improve robustness of the binary code prediction model.
3.4 Hybrid Softmax/Binary Model
According to the Zipf’s law (Zipf, 1949), the distribution of word appearances in an actual corpus is biased to a small subset of the vocabulary. As a result, the proposed model mostly learns
characteristics for frequent words and cannot obtain enough opportunities to learn for rare words. To alleviate this problem, we introduce a hybrid model using both softmax prediction and binary code prediction as shown in Figure 2(c). In this model, the output layer calculates a standard softmax for the N − 1 most frequent words and an OTHER marker which indicates all rare words. When the softmax layer predicts OTHER, then the binary code layer is used to predict the representation of rare words. In this case, the actual probability of generating a particular word is separated into two equations according to the frequency of words:
Pr(w|h) := { v′id(w), if id(w) < N,
v′N · π(w,h), otherwise, (12)
v′ := expu′/ sum expu′, (13) u′ := Whu′h+ βu′ , (14)
π(w,h) := B∏ i=1 ( biqi + b̄iq̄i ) , (15)
where Whu′ ∈ RN×H and βu′ ∈ RN are trainable parameters, and id(w) assumes that the value corresponds to the rank of frequency of each word. We also define the loss function for the hybrid model using both softmax and binary code losses:
L := { lH(id(w)), if id(w) < N, lH(N) + lB, otherwise, (16) lH(i) := λHLH(v ′, i), (17)
lB := λBLB(q, b), (18)
where λH and λB are hyper-parameters to determine strength of both softmax/binary code losses. These also can be adjusted according to the training data, but in this study, we only used λH = λB = 1 for simplicity.
The computational complexity of the hybrid model is O(H(N + log V )), which is larger than the original binary code modelO(H log V ). However,N can be chosen asN V because the softmax prediction is only required for a few frequent words. As a result, we can control the actual computation for the hybrid model to be much smaller than the standard softmax complexity O(HV ),
The idea of separated prediction of frequent words and rare words comes from the differentiated softmax (Chen et al., 2016) approach. However, our output layer can be configured as a fullyconnected network, unlike the differentiated soft-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Figure 3: Example of the classification problem using redundant bit array mapping.
Figure 4: Training and generation processes with error-correcting code.
max, because the actual size of the output layer is still small after applying the hybrid model.
3.5 Applying Error-correcting Codes
The 2 methods proposed in previous sections impose constraints for all bits in q, and the value of each bit must be estimated correctly for the correct word to be chosen. As a result, these models may generate incorrect words due to even a single bit error. This problem is the result of dense mapping between words and bit arrays, and can be avoided by creating redundancy in the bit array. Figure 3 shows a simple example of how this idea works when discriminating 2 words using 3 bits. In this case, the actual words are obtained by estimating the nearest centroid bit array according to the Hamming distance between each centroid and the predicted bit array. This approach can predict correct words as long as the predicted bit arrays are in the set of neighbors for the correct centroid (gray regions in the Figure 3), i.e., up to a 1-bit error in the predicted bits can be corrected. This ability to be robust to errors is a central idea behind error-correcting codes (Shannon, 1948). In general, an error-correcting code has the ability to correct up to b(d−1)/2c bit errors when all centroids differ d bits from each other (Golay, 1949). d is known as the free distance determined
by the design of error-correcting codes. Errorcorrecting codes have been examined in some previous work on multi-class classification tasks, and have reported advantages from the raw classification (Dietterich and Bakiri, 1995; Klautau et al., 2003; Liu, 2006; Kouzani and Nasireding, 2009; Kouzani, 2010; Ferng and Lin, 2011, 2013). In this study, we applied an error-correcting algorithm to the bit array obtained from Algorithm 1 to improve robustness of the output layer in an NMT system. A challenge in this study is trying a large classification (#classes > 10,000) with error-correction, unlike previous studies focused on solving comparatively small tasks (#classes < 100). And this study also tries to solve a generation task unlike previous studies. As shown in the experiments, we found that this approach is highly effective in these tasks.
Figure 4 (a) and (b) illustrate the training and generation processes for the model with errorcorrecting codes. In the training, we first convert the original bit arrays b(w) to a center bit array b′ in the space of error-correcting code: b′(b) := [b′1(b), b ′ 2(b), · · · , b′B′(b)] ∈ {0, 1}B ′ , where B′(B) ≥ B is the number of bits in the error-correcting code. The NMT model learns its parameters based on the loss between predicted probabilities q and b′. Note that typical errorcorrecting codes satisfy O(B′/B) = O(1), and this characteristic efficiently suppresses the increase of actual computation cost in the output layer due to the application of the error-correcting code. In the generation of actual words, the decoding method of the error-correcting code converts the redundant predicted bits q into a dense representation q̃ := [q̃1(q), q̃2(q), · · · , q̃B(q)], and uses q̃ as the bits to restore the word, as is done in the method described in the previous sections.
It should be noted that the method for performing error correction directly affects the quality of the whole NMT model. For example, the mapping shown in Figure 3 has only 3 bits and it is clear that these bits represent exactly the same information as each other. In this case, all bits can be estimated using exactly the same parameters, and we can not expect that we will benefit significantly from applying this redundant representation. Therefore, we need to choose an error correction method in which the characteristics of original bits should be distributed in various positions of the resulting bit arrays so that errors in bits are
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Algorithm 2 Encoding into a convolutional code. Require: b ∈ {0, 1}B Ensure: b′ ∈ {0, 1}2(B+6) = Redundant bit array
x[t] := { bt, if 1 ≤ t ≤ B 0, otherwise y1t := x[t− 6 .. t] · [1001111] mod 2 y2t := x[t− 6 .. t] · [1101101] mod 2 b′ ← [y11 , y21 , y12 , y22 , · · · , y1B+6, y2B+6]
Algorithm 3 Decoding from a convolutional code. Require: q ∈ (0, 1)2(B+6) Ensure: q̃ ∈ {0, 1}B = Restored original bit array g(q, b) := b log q + (1− b) log(1− q)
φ0[s | s ∈ {0, 1}6]← {
0, if s = [000000] −∞, otherwise
for t = 1→ B + 6 do for scur ∈ {0, 1}6 do
sprev(x) := [x] ◦ scur[1 .. 5] o1(x) := ([x] ◦ scur) · [1001111] mod 2 o2(x) := ([x] ◦ scur) · [1101101] mod 2 g′(x) := g(q2t−1, o1(x)) + g(q2t, o2(x)) φ′(x) := φt−1[s
prev(x)] + g′(x) x̂← arg maxx∈{0,1} φ′(x) rt[s
cur]← sprev(x̂) φt[s
cur]← φ′(x̂) end for
end for s′ ← [000000] for t = B → 1 do
s′ ← rt+6[s′] q̃t ← s′1
end for q̃ ← [q̃1, q̃2, · · · , q̃B ]
not highly correlated with each-other. In addition, it is desirable that the decoding method of the applied error-correcting code can directly utilize the probabilities of each bit, because q generated by the network will be a continuous probabilities between zero and one.
In this study, we applied convolutional codes (Viterbi, 1967) to convert between original and redundant bits. Convolutional codes perform a set of bit-wise convolutions between original bits and weight bits (which are hyper-parameters). They are well-suited to our setting here because they distribute the information of original bits in different places in the resulting bits, work robustly for random bit errors, and can be decoded using bit probabilities directly.
Algorithm 2 describes the particular convolutional code that we applied in this study, with two convolution weights [1001111] and [1101101] as fixed hyper-parameters.2 Where x[i .. j] :=
2We also examined many configurations of convolutional codes which have different robustness and computation costs, and finally chose this one.
[xi, · · · , xj ] and x · y := ∑
i xiyi. On the other hand, there are various algorithms to decode convolutional codes with the same format which are based on different criteria. In this study, we use the decoding method described in Algorithm 3, where x ◦ y represents the concatenation of vectors x and y. This method is based on the Viterbi algorithm (Viterbi, 1967) and estimates original bits by directly using probability of redundant bits. Although Algorithm 3 looks complicated, this algorithm can be performed efficiently on CPUs at test time, and is not necessary at training time when we are simply performing calculation of Equation (6). Algorithm 2 increases the number of bits from B intoB′ = 2(B+6), but does not restrict the actual value of B.
4 Experiments
4.1 Experimental Settings
We examined the performance of the proposed methods on two English-Japanese bidirectional translation tasks which have different translation difficulties: ASPEC (Nakazawa et al., 2016) and BTEC (Takezawa, 1999). Table 1 describes details of two corpora. To prepare inputs for training, we used tokenizer.perl in Moses (Koehn et al., 2007) and KyTea (Neubig et al., 2011) for English/Japanese tokenizations respectively, applied lowercase.perl from Moses, and replaced out-of-vocabulary words such that rank(w) > V − 3 into the UNK marker.
We implemented each NMT model using C++ in the DyNet framework (Neubig et al., 2017) and trained/tested on 1 GPU (GeForce GTX TITAN X). Each test is also performed on CPUs to compare its processing time. We used the concat local attention model proposed in Luong et al. (2015) constructed using a 1-layer LSTM (input/forget/output gates and non-peepholes) (Gers et al., 2000) encoder/decoder with 30% dropout (Srivastava et al., 2014) for the input/output vectors of the LSTMs. Only output layers and loss functions are replaced, and other network architectures are identical for the conventional/proposed models. We used the Adam optimizer (Kingma and Ba, 2014) with fixed hyper-parameters α = 0.001, β1 = 0.9β2 = 0.999, ε = 10
−8, and minibatches with 64 sentences sorted according to their sequence lengths. For evaluating the quality of each model, we calculated BLEU (Papineni et al., 2002) every 1000 mini-batches. Table 2 lists all
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Table 1: Details of the corpus.
Name ASPEC BTEC
Languages En↔ Ja
#sentences Train 2.00 M 465. k Dev 1,790 510 Test 1,812 508 Vocabulary size V 65536 25000
Table 2: Evaluated models.
Name Summary Softmax Softmax prediction (Figure 2(a)) Binary Figure 2(b) w/ raw bit array Hybrid-N Figure 2(c) w/ softmax size N Binary-EC Binary w/ error-correction Hybrid-N-EC Hybrid-N w/ error-correction
models we examined in experiments.
4.2 Results and Discussion
Table 3 shows the BLEU on the test set, number of bits B (or B′) for the binary code, actual size of the output layer #out, number of parameters in the output layer #W,β, as well as the ratio of #W,β or amount of whole parameters compared with Softmax, and averaged processing time at training (per mini-batch on GPUs) and test (per sentence on GPUs/CPUs), respectively. Figure 5(a) and 5(b) shows training curves up to 180,000 epochs about some English→Japanese settings. To relax instabilities of translation qualities while training (as shown in Figure 5(a) and 5(b)), each BLEU in Table 3 is calculated by averaging actual test BLEU of 5 consecutive results around the epoch that has the highest dev BLEU.
First, we can see that each proposed method largely suppresses the actual size of the output layer from ten to one thousand times compared with the standard softmax. By looking at the total number of parameters, we can see that the proposed models require only 70% of the actual memory, and the proposed model reduces the total number of parameters for the output layers to a practically negligible level. Note that most of remaining parameters are used for the embedding lookup at the input layer in both encoder/decoder. These still occupy O(EV ) memory, where E represents the size of each embedding layer and usually O(E/H) = O(1). These are not targets to be reduced in this study because these values rarely are accessed at test time because we only need to access them for input words, and do not need them to always be in the physical memory. It might be
possible to apply a similar binary representation as that of output layers to the input layers as well, then express the word embedding by multiplying this binary vector by a word embedding matrix. This is one potential avenue of future work.
Taking a look at the BLEU for the simple Binary method, we can see that it is far lower than other models for all tasks. This is expected, as described in Section 3, because using raw bit arrays causes many one-off estimation errors at the output layer due to the lack of robustness of the output representation. In contrast, Hybrid-N and Binary-EC models clearly improve BLEU from Binary, and they approach that of Softmax. This demonstrates that these two methods effectively improve the robustness of binary code prediction models. Especially, Binary-EC generally achieves higher quality than Hybrid-512 despite the fact that it suppress the number of parameters by about 1/10. These results show that introducing redundancy to target bit arrays is more effective than incremental prediction. In addition, the Hybrid-NEC model achieves the highest BLEU in all proposed methods, and in particular, comparative or higher BLEU than Softmax in BTEC. This behav-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Table 3: Comparison of BLEU, size of output layers, number of parameters and processing time.
Corpus Method BLEU % B #out #W,β
Ratio of #params Time (En→Ja) [ms] EnJa JaEn #W,β All Train Test: GPU / CPU
ASPEC Softmax 31.13 21.14 — 65536 33.6 M 1/1 1 1026. 121.6 / 2539. Binary 13.78 6.953 16 16 8.21 k 1/4.10 k 0.698 711.2 73.08 / 122.3 Hybrid-512 22.81 13.95 16 528 271. k 1/124. 0.700 843.6 81.28 / 127.5 Hybrid-2048 27.73 16.92 16 2064 1.06 M 1/31.8 0.707 837.1 82.28 / 159.3 Binary-EC 25.95 18.02 44 44 22.6 k 1/1.49 k 0.698 712.0 78.75 / 164.0 Hybrid-512-EC 29.07 18.66 44 556 285. k 1/118. 0.700 850.3 80.30 / 180.2 Hybrid-2048-EC 30.05 19.66 44 2092 1.07 M 1/31.4 0.707 851.6 77.83 / 201.3
BTEC Softmax 47.72 45.22 — 25000 12.8 M 1/1 1 325.0 34.35 / 323.3 Binary 31.83 31.90 15 15 7.70 k 1/1.67 k 0.738 250.7 27.98 / 54.62 Hybrid-512 44.23 43.50 15 527 270. k 1/47.4 0.743 300.7 28.83 / 66.13 Hybrid-2048 46.13 45.76 15 2063 1.06 M 1/12.1 0.759 307.7 28.25 / 67.40 Binary-EC 44.48 41.21 42 42 21.5 k 1/595. 0.738 255.6 28.02 / 69.76 Hybrid-512-EC 47.20 46.52 42 554 284. k 1/45.1 0.744 307.8 28.44 / 56.98 Hybrid-2048-EC 48.17 46.58 42 2090 1.07 M 1/12.0 0.760 311.0 28.47 / 69.44
Figure 6: BLEU changes in the Hybrid-N methods according to the softmax size (En→Ja).
ior clearly demonstrates that these two methods are orthogonal, and combining them together can be effective. We hypothesize that the lower quality of Softmax in BTEC is caused by an over-fitting due to the large number of parameters required in the softmax prediction.
Proposed methods also improve actual computation time in both training and test. In particular, proposed methods can be performed significantly faster than Softmax at testing on CPUs by x5 to x20, which are directly affected by the complexity of output layers. In addition, we can also see that applying error-correcting code is also efficient at the point of the decoding speed.
Figure 6 shows the trade-off between the translation quality and the size of softmax layers in the hybrid prediction model (Figure 2(c)) without error-correction. According to the model definition in Section 3.4, the softmax prediction and
raw binary code prediction can be assumed to be the upper/lower-bound of the hybrid prediction model. The curves in Figure 6 move between Softmax and Binary models, and this behavior intuitively explains the characteristics of the hybrid prediction. In addition, we can see that the BLEU score in BTEC quickly improves, and saturates at N = 1024 in contrast to the ASPEC model, which is still improving at N = 2048. We presume that the shape of curves in Figure 6 is also affected by the difficulty of the corpus, i.e., when we train the hybrid model for easy datasets (e.g., BTEC is easier than ASPEC), it is enough to use a small softmax layer (e.g. N ≤ 1024).
5 Conclusion
In this study, we proposed neural machine translation models which indirectly predict output words via binary codes, and two model improvements: a hybrid prediction model using both softmax and binary codes, and introducing error-correcting codes to introduce robustness of binary code prediction. Experiments show that the proposed model can achieve comparative translation qualities to standard softmax prediction, while significantly suppressing the amount of parameters in the output layer, and improving calculation speeds while training and especially testing.
One interesting avenue of future work is to automatically learn encodings and error correcting codes that are well-suited for the type of binary code prediction we are performing here. In Algorithms 2 and 3 we use convolutions that were determined heuristically, and it is likely that learning these along with the model could result in improved accuracy or better compression capability.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: This paper has high originality, proposing a fundamentally different way of predicting words from a vocabulary that is more efficient than a softmax layer and has comparable performance on NMT. If successful, the approach could be impactful because it speeds up prediction.  This paper is nice to read with great diagrams. it's very clearly presented -- I like cross-referencing the models with the diagrams in Table 2. Including loss curves is appreciated.  - Weaknesses: Though it may not be possible in the time remaining, it would be good to see a comparison (i.e. BLEU scores) with previous related work like hierarchical softmax and differentiated softmax.  The paper is lacking a linguistic perspective on the proposed method. Compared to a softmax layer and hierarchical differentiated softmax, is binary code prediction a natural way to predict words? Is it more or less similar to how a human might retrieve words from memory? Is there a theoretical reason to believe that binary code based approaches should be more or less suited to the task than softmax layers?  Though the paper promises faster training speeds in the introduction, Table 3 shows only modest (less than x2) speedups for training. Presumably this is because much of the training iteration time is consumed by other parts of the network. It would be useful to see the time needed for the output layer computation only.  - General Discussion: It would be nice if the survey of prior work in 2.2 explicitly related those methods to the desiderata in the introduction (i.e. specify which they satisfy).  Some kind of analysis of the qualitative strengths and weaknesses of the binary code prediction would be welcome -- what kind of mistakes does the system make, and how does this compare to standard softmax and or hierarchical and differentiated softmax?  LOW LEVEL COMMENTS  Equation 5: what's the difference between id(w) = id(w') and w = w' ?  335: consider defining GPGPU  Table 3: Highlight the best BLEU scores in bold  Equation 15: remind the reader that q is defined in equation 6 and b is a function of w. I was confused by this at first because w and h appear on the LHS but don't appear on the right, and I didn't know what b and q were.