Neural Modeling of Multi-Predicate Interactions for Japanese Predicate Argument Structure Analysis
The accuracy of Japanese predicate argument structure (PAS) analysis has improved in recent years thanks to joint modeling of interactions between multiple predicates. However, this approach heavily relies on syntactic information predicted by parsers, and suffers from the error propagation. To remedy this problem, we introduce a model using grid-type recurrent neural networks (Grid-RNN), which automatically induces features sensitive to multi-predicate interactions from word sequence information of a sentence. The experiments on the NAIST Text Corpus show that our model exceeds the accuracy of the state-of-the-art Japanese PAS analyzer without syntactic information.
1 Introduction
Predicate argument structure (PAS) analysis is a basic semantic analysis task, in which systems are required to identify semantic units of a sentence, such as who did what to whom. In pro-drop languages such as Japanese, Chinese and Italian, arguments are often omitted in text, and such argument omission is regarded as one of the most problematic issues of PAS analysis (Iida and Poesio, 2011; Sasano and Kurohashi, 2011).
As an approach to the argument omission problem, in Japanese PAS analysis, joint modeling of interactions between multiple predicates has been gaining popularity and achieved the state-of-theart result (Ouchi et al., 2015; Shibata et al., 2016). This approach is based on the linguistic intuition that the predicates in a sentence are semantically related to each other and the interaction information can be a clue for PAS analysis. However, to model such multi-predicate interactions, this ap-
proach heavily relies on syntactic information predicted by parsers and suffers from the error propagation caused by the pipeline processing. To remedy this problem, we propose a neural model which automatically induces features sensitive to multi-predicate interactions from word sequence information of a sentence. This model takes as input all predicates and their argument candidates in a sentence at a time, and captures the interactions using grid-type recurrent neural networks (Grid-RNN) without syntactic information. In this paper, we firstly introduce a basic model using RNNs, which independently estimates arguments of each predicate without considering the multi-predicate interactions (Sec. 3). Then, extending this model, we propose a neural model using Grid-RNNs (Sec. 4). Performing experiments on the NAIST Text Corpus (Iida et al., 2007), we demonstrate that our neural models exceed the accuracy of the stateof-the-art Japanese PAS analyzer (Ouchi et al., 2015). In particular, the neural model using GridRNNs achieves the best result, which suggests that our grid-type neural architecture effectively captures multi-predicate interactions. 1
1Our source code is publicly available at http:xxx
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Figure 2: An overview of the neural models: (i) single-sequence and (ii) multi-sequence models.
2 Japanese Predicate Argument Structure Analysis
2.1 Task Description
In Japanese PAS analysis, we identify arguments taking part in the three major case roles, nominative (NOM), accusative (ACC) and dative (DAT) cases, for each predicate. Arguments can be divided into the following three categories according to the positions relative to their predicates (Hayashibe et al., 2011; Ouchi et al., 2015):
Dep: The arguments that have direct syntactic dependency with the predicate.
Zero: The arguments referred to by zero pronouns within the same sentence, which have no direct syntactic dependency with the predicate.
Inter-Zero: The arguments referred to by zero pronouns out of the same sentence.
For example, in Figure 1, the nominative argument “警察 (police)” for the predicate “逮捕した (arrested)” is regarded as a Dep argument since the argument has a direct syntactic dependency with the predicate. In contrast, the nominative argument “男 i (mani)” for the predicate “逃走した (escaped)” is regarded as a Zero argument since the argument has no direct syntactic dependency with the predicate.
In this paper, we focus on the analysis for these intra-sentential arguments, i.e., Dep and Zero. In order to identify inter-sentential arguments (Inter-Zero), it is required to search a much broader space, such as the whole document, resulting in a much harder analysis than intrasentential arguments.2 Thus, Ouchi et al. (2015)
2The F-measure remains 10-20% (Taira et al., 2008; Imamura et al., 2009; Sasano and Kurohashi, 2011).
and Shibata et al. (2016) focused on only intrasentential argument analysis. Following this trend, we focus on intra-sentential argument analysis.
2.2 Challenging Problem
Arguments are often omitted in Japanese sentences. In Figure 1, ϕi represents the omitted argument, called zero pronoun. This zero pronoun ϕi refers to “男 i (mani)”. In Japanese PAS analysis, when an argument of the target predicate is omitted, we have to identify the antecedent of the omitted argument (Zero argument). The analysis for such Zero arguments is much more difficult than that forDep arguments because of the lack of direct syntactic dependencies. For Dep arguments, the syntactic dependency between an argument and its predicate is a strong clue. In the sentence in Figure 1, for the predicate “逮捕し た (arrested)”, the nominative argument is “警察 (police)”. This argument can easily be identified by relying on the syntactic dependency. In contrast, since the nominative argument “男 i (mani)” has no syntactic dependency with its predicate “逃 走した (escaped)”, we have to use other information for such zero argument identification. As an solution to this problem, we exploit two kinds of information: (i) context in the entire sentence and (ii) multi-predicate interactions. For the former, we introduce single-sequence model, which induces context-sensitive representations from a sequence of argument candidates of a predicate. For the latter, we introduce multi-sequence model, which induces predicate-sensitive representations from multiple sequences of argument candidates of all predicates in a sentence (shown in Figure 2).
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 3: The overall architecture of the single sequence model. This model consists of three components: (i) Input Layer, (ii) RNN Layer and (iii) Output Layer.
3 Single-Sequence Model
The single-sequence model exploits stacked bidirectional RNNs (Bi-RNN) (Schuster and Paliwal, 1997; Graves et al., 2005, 2013; Zhou and Xu, 2015). Figure 3 shows the overall architecture, which consists of the following three components:
Input Layer: Map each word to a feature vector representation.
RNN Layer: Produce high-level feature vectors using Bi-RNNs.
Output Layer: Compute the probability of each case label for each word using the softmax function.
In the following subsections, we describe each of them in more detail.
3.1 Input Layer
Given an input sentence w1:T = (w1, · · · , wT) and a predicate p, each word wt is mapped to a feature representation xt, which is the concatenation (⊕) of three types of vectors:
xt = x arg t ⊕ x pred t ⊕ xmarkt (1)
where each vector is based on the following atomic features inspired by Zhou and Xu (2015):
ARG: Word index of each word. PRED: Word index of the target predicate and
words around the predicate.
MARK: Binary index that represents whether the word is the predicate or not.
Figure 4: An example of the feature extraction. The underlined word is the target predicate. From the sentence “彼女はパンを食べた。(She ate a bread.)”, the three types of features are extracted for the target predicate “食べた (ate)”.
Figure 5: An example of the process of creating a feature vector. The extracted features are mapped to each vector, and all the vectors are concatenated into one feature vector.
Figure 4 presents an example of the atomic features. As the ARG feature, we extract a word index xword ∈ V of each word. Similarly, as the PRED feature, we extract each word index xword of the C words taking the target predicate at the center, where C is the window size. The MARK feature xmark ∈ {0, 1} is a binary value that represents whether the word is the predicate or not. Then, using feature indices, we extract feature vector representations from each embedding matrix. Figure 5 shows the process of creating the feature vector x1 for the word w1 “彼女 (she)”. We set two embedding matrices: (i) word embedding matrix Eword ∈ Rdword×|V|, and (ii) mark embedding matrixEmark ∈ Rdmark×2. From each embedding matrix, we extract corresponding column vectors and concatenate them as a feature vector representation xt based on Eq. 1.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Each feature vector xt is multiplied with a parameter matrix Wx:
h (0) t = Wx xt (2)
The vector h(0)t is given to the first layer of the RNN layers as input.
3.2 RNN Layer
In the RNN layers, feature vectors are updated recurrently using Bi-RNNs. Bi-RNNs process an input sequence from the left-to-right manner in odd-numbered layers and the opposite in evennumbered layers. By stacking these layers, we can construct the deeper network structures.
Stacked Bi-RNNs consist of L layers, and the hidden state in the layer ℓ ∈ (1, · · · , L) is calculated as follows:
h (ℓ) t =
{ g(ℓ)(h
(ℓ−1) t , h (ℓ) t−1) (ℓ = odd)
g(ℓ)(h (ℓ−1) t , h (ℓ) t+1) (ℓ = even)
(3)
Both of the odd and even-numbered layers receive h(ℓ−1)t , the t-th hidden state of the ℓ − 1 layer, as the first input of the function g(ℓ), which is an arbitrary function 3. As the second input of g(ℓ), odd-numbered layers receive h(ℓ)t−1 while even-numbered layers receive h(ℓ)t+1. By calculating the hidden states until the L-th layer, we obtain a hidden state sequence h(L)1:T = (h (L) 1 , · · · ,h (L) T ). Using each vector h(L)t , we calculate the probability of case labels for each word in the output layer.
3.3 Output Layer
In the output layer, multi-class classification is performed using the softmax function:
yt = softmax(Wy h (L) t )
where h(L)t is a vector representation propagated from the last RNN layer (Fig 3). Each element of yt is a probability value corresponding to each label. The label with the maximum probability among them is output as a result. In this task, there are five labels: NOM, ACC, DAT, PRED, null. The labels NOM, ACC and DAT indicate the nominative, accusative and dative case, respectively. PRED is the label for the predicate. null represents a word that does not play any case role.
3In this work, we use the Gated Recurrent Unit (GRU) (Cho et al., 2014) as the function g(ℓ).
4 Multi-Sequence Model
While the single-sequence model assumes the independence between predicates, the multisequence model assumes the multi-predicate interactions. To capture such interactions between all predicates in a sentence, we extend the singlesequence model to the multi-sequence model using Grid-RNNs (Graves and Schmidhuber, 2009; Kalchbrenner et al., 2016). Figure 6 presents the overall architecture of the multi-sequence model, which consists of three components:
Input Layer: Map words to M sequences of feature vectors for M predicates.
Grid Layer: Update the hidden states over different sequences using Grid-RNNs.
Output Layer: Compute the probability of each case label for each word using the softmax function.
In the following subsections, we describe them in more detail.
4.1 Input Layer
The multi-sequence model takes as input a sentence w1:T = (w1, · · · , wT) and all predicates {pm}M1 in the sentence. For each predicate pm, the input layer creates a sequence of feature vectors Xm = (xm,1, · · · ,xm,T) by mapping each input word wt to a feature vector xm,t based on Eq 1. That is, for M predicates, M sequences of feature vectors {Xm}M1 are created. Then, using Eq. 2, each feature vector xm,t is mapped to h(0)m,t, and a feature sequence is created for a predicate pm, i.e.,H (0) m = (h (0) m,1, · · · ,h (0) m,T). Consequently, forM predicates, we obtainM feature sequences {H(0)m }M1 .
4.2 Grid Layer
Inter-Sequence Connections
In the grid layers, we use Grid-RNNs to propagate the feature information over the different sequences (inter-sequence connections). The right figure in Figure 6 shows an odd-numbered layer of the Grid layers. The hidden state is recurrently calculated from upper-left (m = 1, t = 1) to lowerright (m = M, t = T). Formally, in the ℓ-th layer, the hidden state h(ℓ)m,t is calculated as follows:
h (ℓ) m,t=
{ g(ℓ)(h
(ℓ−1) m,t ⊕ h (ℓ) m−1,t,h (ℓ) m,t−1) (ℓ = odd)
g(ℓ)(h (ℓ−1) m,t ⊕ h (ℓ) m+1,t,h (ℓ) m,t+1) (ℓ = even)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
This equation is similar to Eq. 3. The main difference is that the hidden state of a neighboring sequence, h(ℓ)m−1,t (or h (ℓ) m+1,t), is concatenated (⊕) with the hidden state of the previous (ℓ− 1) layer, h (ℓ−1) m,t , and is taken as input of the function g
(ℓ). In the right figure in Figure 6, the blue curve lines represent the inter-sequence connections. Taking as input the hidden states of neighboring sequences, the network propagates feature information over multiple sequences (predicates). By calculating the hidden states until the L-th layer, we obtain M sequences of the hidden states, i.e., {H(L)m }M1 , in which H (L) m = (h (L) m,1, · · · ,h (L) m,T).
Residual Connections
As more layers are stacked, it gets more difficult to learn the model parameters due to some problems such as gradient vanishment (Pascanu et al., 2013). In this work, we integrate residual connections (He et al., 2015;Wu et al., 2016) with our networks to connect between layers. Specifically, the input vector h(ℓ−1)m,t of the ℓ-th layer is added to the output vector h(ℓ)m,t. Residual connections can also be applied to the single-sequence model, and thus we perform the experiments on both models with/without residual connections.
4.3 Output Layer
Like the single-sequence model, using the softmax function, we calculate the probability of case labels of each word wt for each predicate pm:
ym,t = softmax(Wy h (L) m,t)
where h(L)m,t is a hidden state vector calculated in the last Grid Layer.
5 Related Work
5.1 Japanese PAS Analysis Approaches
Existing approaches for Japanese PAS analysis are divided into two categories: (i) pointwise approach and (ii) joint approach. In the pointwise approach, we estimate the score of each argument candidate for one predicate, and select the argument candidate with the maximum score as an argument (Taira et al., 2008; Imamura et al., 2009; Hayashibe et al., 2011; Iida et al., 2016). In the joint approach, we select the score of all the predicate-argument combinations in one sentence, and select the combination with the highest score (Yoshikawa et al., 2011; Sasano and Kurohashi, 2011; Ouchi et al., 2015; Shibata et al., 2016). Compared with the pointwise approach, the joint approach has achieved better results.
5.2 Multi-Predicate Interactions
Ouchi et al. (2015) reported that it is beneficial to Japanese PAS analysis to capture the interactions between all predicates in a sentence. This is based on the linguistic intuition that the predicates in a sentence are semantically related to each other, and the interaction information can be a clue for PAS analysis. Similarly, in semantic role labeling (SRL), Yang and Zong (2014) also reported that their reranking model capturing the multi-predicate interactions is effective for the English constituentbased SRL task (Carreras and Màrquez, 2005). Taking a step further in this direction, we propose the neural architecture that effectively models the multi-predicate interactions.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
5.3 Neural Approaches
Japanese PAS
In recent years, several attempts have been made to apply neural networks for Japanese PAS analysis (Shibata et al., 2016; Iida et al., 2016)4. Shibata et al. (2016) used a feed-forward neural network for the score calculation part of the joint model proposed by Ouchi et al. (2015). Iida et al. (2016) used multi-column convolutional neural networks for the zero anaphora resolution task.
Both models exploited syntactic and selectional preference information as atomic features of neural networks. Using neural networks, the good performance was realized with mitigating the cost of manually designing combination features. In this work, we demonstrate that even without such syntactic information, our neural models realize the state-of-the-art performance by using word sequence information of a sentence.
English SRL
Some neural models achieved high performance without syntactic information in English SRL. Collobert et al. (2011) worked on the English constituent-based SRL task (Carreras and Màrquez, 2005). Their model exploited a convolutional neural network and achieved 74.15% in F-measure without syntactic information. Zhou and Xu (2015) also worked on the same task using bidirectional RNNs with CRF and achieved the state-of-the-art result, 81.07% in F-measure. Our models can be regarded as an extension from their model.
The main differences between Zhou and Xu (2015) and our work are: (i) the constituent vs dependency-based argument identification and (ii) the multi-predicate consideration. In the constituent-based SRL, since systems are required to identify the spans of arguments for each predicate, Zhou and Xu (2015) used CRF to capture the IOB label dependencies. In contrast, in Japanese dependency-based PAS analysis, since arguments are infrequently adjacent to each other, we replaced the CRF with the softmax function. Also, while the model of Zhou and Xu (2015) predicts arguments for each predicate independently, our multi-sequence model jointly predicts arguments for all predicates in a sentence at a time by considering the multi-predicate interactions.
4These previous studies used unpublished datasets and evaluated the performance with different experimental settings, so we cannot compare their models with ours.
6 Experiments
6.1 Experimental Settings
Dataset
We use the NAIST Text Corpus 1.5, which consists of 40,000 sentences of Japanese newspaper text (Iida et al., 2007). In the experiments, we adopt the standard data splits (Taira et al., 2008; Imamura et al., 2009; Ouchi et al., 2015):
Train: Articles: Jan 1-11, Editorials: Jan-Aug Dev: Articles: Jan 12-13, Editorials: Sept Test: Articles: Jan 14-17, Editorials: Oct-Dec
We use the word boundaries annotated to the NAIST Text Corpus and the target predicates that have at least one argument in the same sentence. We do not use any external resources.
Learning
We train the model parameters by minimizing the cross-entropy loss function:
L(θ) = − ∑ n ∑ t logP (yt|xt) + λ 2 ||θ||2
where θ is a set of model parameters, and the hyper-parameter λ is the coefficient governing the L2 weight decay.
Implementation Details
We implement our neural models using Theano (Bastien et al., 2012). The number of epochs is set to 50, and we report the result of the test set in the epoch with the best F-measure of the development set. Parameter optimization is done by stochastic gradient descent method (SGD) using mini-batch, whose size is selected from {2, 4, 8}. The learning rate is automatically adjusted using Adam (Kingma and Ba, 2014). For the L2 weight decay, the hyper-parameter λ in Eq. 4 is selected from {0.001, 0.0005, 0.0001}. In the neural models, the number of the RNN and Grid layers are selected from {2, 4, 6, 8}. The window size C for the PRED feature (Sec. 3.1) is set to 5. Words with frequency 2 or more are mapped to each word index, and the remaining words are mapped to the unknown word index. The dimensions dword and dmark of the embeddings are set to 32. In the single-sequence model, the parameters of GRUs are set to 32× 32. In the multi-sequence model, the parameters of GRUs related to the input values are set to 64 × 32, and
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Dep Zero All Imamura+ 09 85.06 41.65 78.15
Ouchi+ 15 86.07 44.09 79.23 Single-Seq 88.10 46.10 81.15 Multi-Seq 88.17 † 47.12 † 81.42 †
Table 1: F-measures in the test set. Single-Seq is the single-sequence model, and Multi-Seq is the multi-sequence model. Imamura+ 09 is the model of Imamura et al. (2009) reimplemented by Ouchi et al. (2015), and Ouchi+ 15 is the ALLCases Joint Model of Ouchi et al. (2015). The mark † denotes the significantly better results with the significance level p < 0.05 comparing between Single-Seq and Multi-Seq.
others are 32× 32. The initial values of all the parameters are sampled according to the uniform distribution from [− √ 6√
row+col ,
√ 6√
row+col ], where row
and col are the number of rows and columns of each matrix, respectively.
Baseline Models
We compare our models with the models in the previous works (Sec. 5.1) that use the NAIST Text Corpus 1.5. As a baseline of the pointwise approach, we use the pointwise model5 of Imamura et al. (2009). In addition, as a baseline of the joint approach, we use the model of Ouchi et al. (2015), which achieved the best result on the NAIST Text Corpus 1.5.
6.2 Results
Neural Models vs Baseline Models
Table 1 presents F-measures of our neural sequence models with 8 RNN or Grid layers and the baseline models on the test set, in which as the significent test, we used the bootstrap resampling method. In all the metrics, both of the single-sequence (Single-Seq) and multisequence model (Multi-Seq) outperformed the baseline models. This confirms that our neural sequence models realize high-performance even without syntactic information by learning contextual information effective for PAS analysis from a word sequence of the sentence.
In particular, for zero arguments (Zero), our models achieved a considerable improvement compared with the state-of-the-art model of
5We compared the results of the model reimplemented by Ouchi et al. (2015).
Ouchi et al. (2015), i.e., the single model improved around 2.0 points and the multi-sequence model improved around 3.0 points in F-measure. These results suggest that it is beneficial to Japanese PAS analysis, particularly to the zero argument identification, to model the context in the entire sentence using RNNs.
Effects of Multiple Predicate Consideration
As Table 1 shows, the multi-sequence model significantly outperformed the single-sequence model in F-measure in total (81.42% vs 81.15%). This result demonstrates that the grid-type neural architecture can effectively capture the multipredicate interactions by connecting between the sequences of the argument candidates for all predicates in a sentence. Compared with the single-sequence model for different argument types, the multi-sequence model achieved slightly but significantly better result for the direct dependency arguments (Dep) (88.10% vs 88.17%). In addition, for zero arguments (Zero), which have no syntactic dependency with its predicate, the multi-sequence model significantly outperformed the single-sequence model by around 1.0 points in F-measure (46.10% vs 47.12%). This shows that capturing the multipredicate interactions is particularly effective for zero arguments, which is consistent with the results of Ouchi et al. (2015).
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Dep Zero NOM ACC DAT NOM ACC DAT
NAIST Text Corpus 1.5 Imamura+ 09 86.50 92.84 30.97 45.56 21.38 0.83
Ouchi+ 15 88.13 92.74 38.39 48.11 24.43 4.80 Single-Seq 88.32 93.89 65.91 49.51 35.07 9.83 Multi-Seq 88.75 93.68 64.38 50.65 32.35 7.52
NAIST Text Corpus 1.4β Taira+ 08* 75.53 88.20 89.51 30.15 11.41 3.66
Imamura+ 09* 87.0 93.9 80.8 50.0 30.8 0.0 Sasano+ 11* - - - 39.5 17.5 8.9
Table 3: Performance comparison for different case roles on the test set in F-measures. NOM, ACC or DAT is the nominal, accusative or dative case, respectively. The mark * indicates that the model uses external resources.
Effects of Network Depth
Table 2 presents F-measures of the neural sequence models with different network depths and with/without residual connections. The performance tends to get better as the RNN or Grid layers get deeper with residual connections. In particular, the two models with 8 layers and residual connections achieved considerable improvements of around 1.0 point in F-measure compared the models without residual connections, which means that the residual connections contribute to the effective parameter learning of deeper models.
Comparison per Case Role
Table 3 shows F-measures for each case role. For reference, we show the results of the previous studies using NAIST Text Corpus 1.4β with external resources as well.6
Comparing between the models using the NAIST Text Corpus 1.5, the single-sequence and multi-sequence models outperformed the baseline models in all the metrics. In particular, for the dative case, the two neural models achieved much higher results by around 30 points. This suggests that although dative arguments appear infrequently compared with the other two case arguments, the neural models can robustly learn it.
In addition, for zero arguments (Zero), the neural models achieved better results than the baseline models. Especially, for zero arguments of
6The major difference between NAIST Text Corpus 1.4β and 1.5 is the revision of the annotation criterion for the dative case (DAT) (corresponding to Japanese case marker “に”). Argument and adjunct usages of the case marker “に” are not distinguished in 1.4β, making the identification of the dative case seemingly easy (Ouchi et al., 2015).
the nominative case (NOM), the multi-sequence model achieved a considerable improvement of around 2.5 points in F-measure compared with the state-of-the-art model of Ouchi et al. (2015). To achieve high accuracies for the analysis of such zero arguments, it is necessary to capture long distance dependencies (Iida et al., 2005; Sasano and Kurohashi, 2011; Iida et al., 2015). Therefore, the improvements of the results suggest that the neural models effectively capture long distance dependencies using RNNs that can encode the context in the entire sentence.
7 Conclusion
In this work, we introduced neural sequence models that automatically induce effective feature representations from word sequence information of a sentence for Japanese PAS analysis. The experiments on NAIST Text Corpus 1.5 demonstrated that the models achieve the state-of-the-art result without syntactic information. In particular, our multi-sequence model improved the performance for zero argument identification, one of the problematic issues in Japanese PAS analysis, by considering the multi-predicate interactions using Grid-RNNs. Since our neural models are applicable to SRL, applying our models for multilingual SRL tasks is an interesting line of the future research. In addition, in this work, the model parameters were learned without any external resources. For future work, we plan to explore effective methods for exploiting large-scale unlabeled data to learn the neural models.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
This paper proposes a joint neural modelling approach to PAS analysis in Japanese, based on Grid-RNNs, which it compares variously with a conventional single-sequence RNN approach.  This is a solidly-executed paper, targeting a well-established task from Japanese but achieving state-of-the-art results at the task, and presenting the task in a mostly accessible manner for those not versed in Japanese. Having said that, I felt you could have talked up the complexity of the task a bit, e.g. wrt your example in Figure 1, talking through the inherent ambiguity between the NOM and ACC arguments of the first predicate, as the NOM argument of the second predicate, and better describing how the task contrasts with SRL (largely through the ambiguity in zero pronouns). I would also have liked to have seen some stats re the proportion of zero pronouns which are actually intra-sententially resolvable, as this further complicates the task as defined (i.e. needing to implicitly distinguish between intra- and inter-sentential zero anaphors). One thing I wasn't sure of here: in the case of an inter-sentential zero pronoun for the argument of a given predicate, what representation do you use? Is there simply no marking of that argument at all, or is it marked as an empty argument? My reading of the paper is that it is the former, in which case there is no explicit representation of the fact that there is a zero pronoun, which seems like a slightly defective representation (which potentially impacts on the ability of the model to capture zero pronouns); some discussion of this would have been appreciated.  There are some constraints that don't seem to be captured in the model (which some of the ILP-based methods for SRL explicitly model, e.g.): (1) a given predicate will generally have only one argument of a given type (esp. NOM and ACC); and (2) a given argument generally only fills one argument slot for a given predicate. I would have liked to have seen some analysis of the output of the model to see how well the model was able to learn these sorts of constraints. More generally, given the mix of numbers in Table 3 between Single-Seq and Multi-Seq (where it is really only NOM where there is any improvement for Multi-Seq), I would have liked to have seen some discussion of the relative differences in the outputs of the two models: are they largely identical, or very different but about the same in aggregate, e.g.? In what contexts do you observe differences between the two models? Some analysis like this to shed light on the internals of the models would have made the difference between a solid and a strong paper, and is the main area where I believe the paper could be improved (other than including results for SRL, but that would take quite a bit more work).  The presentation of the paper was good, with the Figures aiding understanding of the model. There were some low-level language issues, but nothing major:  l19: the error propagation to error propagation l190: an solution to a solution l264 (and Figure 2): a bread to bread l351: the independence to independence l512: the good to good l531: from their model to of their model l637: significent to significance l638: both of to both  and watch casing in your references (e.g.  japanese ,  lstm ,  conll ,  ilp )