Deep Semantic Role Labeling: What Works and What’s Next
We introduce a new deep learning model for semantic role labeling (SRL) that significantly improves the state of the art, along with detailed analyses to reveal its strengths and limitations. We use a deep highway BiLSTM architecture with constrained decoding, while observing a number of recent best practices for initialization and regularization. Our 8-layer ensemble model achieves 83.2 F1 on the CoNLL 2005 test set and 83.4 F1 on CoNLL 2012, roughly a 10% relative error reduction over the previous state of the art. Extensive empirical analysis of these gains show that (1) deep models excel at recovering long-distance dependencies but can still make surprisingly obvious errors, and (2) that there is still room for syntactic parsers to improve these results. All code and models will be publicly released.
1 Introduction
Semantic role labeling (SRL) systems aim to recover the predicate-argument structure of a sentence, to determine essentially “who did what to whom”, “when”, and “where.” Recently breakthroughs involving end-to-end deep models for SRL without syntactic input (Zhou and Xu, 2015; Marcheggiani et al., 2017) seem to overturn the long-held belief that syntactic parsing is a prerequisite for this task (Punyakanok et al., 2008). In this paper, we show that this result can be pushed further using deep highway bidirectional LSTMs with constrained decoding, again significantly moving the state of the art (another 2 points on CoNLL 2005). We also present careful empirical analysis to determine what works well and what might be done to progress even further.
Our model combines a number of best practices in the recent deep learning literature. Following Zhou and Xu (2015), we treat SRL as a BIO tagging problem and use deep bidirectional LSTMs. However, we differ by (1) simplifying the input and output layers, (2) introducing highway connections (Srivastava et al., 2015; Zhang et al., 2016), (3) using recurrent dropout (Gal, 2015), (4) decoding with BIO-constraints, and (5) ensembling with a product of experts. Our model gives a 10% relative error reduction over previous state of the art on the test sets of CoNLL 2005 and 2012. All code and models will be publicly released.
We present a set of detailed error analyses to better understand the performance gains, including (1) design choices on architecture, initialization, and regularization that have a surprisingly large impact on model performance, (2) analyses of different types of prediction errors, e.g. showing that deep models excel at predicting long-distance dependencies but still struggles with known challenges such as PP-attachment errors and adjunct-argument distinctions, (3) a careful analysis of the role of syntax, showing that there is significant room for improvement given oracle syntax but errors from existing parsers prevent effective use in SRL.
In summary, our main contributions are:
• A new state-of-the-art deep network for endto-end SRL, supported by code and models that will be publicly available. • An in-depth error analysis indicating where
the models work well and where they still struggle, including discussion of structural consistency and long-distance dependencies. • Experiments that point toward directions for
future improvements, including a detailed discussion of how and when syntactic parsers could be used to improve these results.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
2 Model
Two major factors contribute to the success of our deep SRL model: (1) applying recent advances in training deep recurrent neural networks such as highway connections (Srivastava et al., 2015; Zhang et al., 2016) and RNN-dropouts (Gal, 2015)1; and (2) using an A∗ decoding algorithm, similar to Lewis and Steedman (2014) to enforce structural consistency at prediction time, without adding more complexity to the training process.
Formally, the goal of our task is to predict a sequence y given a sentence-predicate pair (w, v) as input. Each yi belongs to a discrete set of BIO tags T . Words outside argument spans have the tag O, and words at the beginning and inside of argument with role r have the tags Br and Ir respectively. Let n = |w| = |y| be the length of the sequence.
Predicting an SRL structure under our model involves finding the highest-scoring tag sequence over the space of all possibilities Y:
ŷ = argmax y∈Y f(w,y) (1)
We use a deep bidirectional LSTM (BiLSTM) to learn a locally decomposed scoring function conditioned on the input: ∑n t=1 log p(yt | w).
To incorporate additional information (e.g. structural consistency, syntactic input), we augment the scoring function with penalization terms:
f(w,y) = n∑ t=1 log p(yt | w)− ∑ c∈C c(w, y1:t) (2)
Each constraint function c applies a non-negative penalty given the input w and a length-t prefix y1:t. These constraints can be hard or soft depending on whether the penalties are finite.
2.1 Deep BiLSTM Model
Our model computes the distribution over tags using stacked BiLSTMs, which we define as follows:
il,t = σ(W l i [hl,t+δl ,xl,t] + b l i) (3) ol,t = σ(W l o[hl,t+δl ,xl,t] + b l o) (4) fl,t = σ(W l f[hl,t+δl ,xl,t] + b l f + 1) (5) c̃l,t = tanh(W l c[hl,t+δl ,xl,t] + b l c) (6) cl,t = il,t ◦ c̃l,t + fl,t ◦ ct+δl (7) hl,t = ol,t ◦ tanh(cl,t) (8)
1We thank Mingxuan Wang for suggesting highway connections with simplified inputs and outputs. Part of our model is extended from his unpublished implementation.
where xl,t is the input to the LSTM at layer l and timestep t. δl is either 1 or −1, indicating the directionality of the LSTM at layer l.
To stack the LSTMs in an interleaving pattern, as proposed by Zhou and Xu (2015), the layerspecific inputs xl,t and directionality δl are arranged in the following manner:
xl,t =
{ [Wemb(wt),Wmask(t = v)] l = 1
hl−1,t l > 1 (9)
δl = { 1 if l is even −1 otherwise
(10)
The input vector x1,t is the concatenation of token wt’s word embedding and an embedding of the binary feature (t = v) indicating whether the current word is the given predicate.
Finally, the locally normalized distribution over output tags is computed via a softmax layer:
p(yt | x) ∝ exp(WytaghL,t + btag) (11)
Highway Connections To alleviate the vanishing gradient problem when training deep BiLSTMs, we use gated highway connections described in Zhang et al. (2016); Srivastava et al. (2015). We include transform gates rt to control the weight of linear and non-linear transformations between layers (See Figure 1). The output hl,t is changed to:
rl,t = σ(W l r[hl,t−1,xt] + b l r) (12) h′l,t = ol,t ◦ tanh(cl,t) (13) hl,t = rl,t ◦ h′l,t + (1− rl,t) ◦Wlhxl,t (14)
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Recurrent Dropout To reduce over-fitting, we use dropout as described in Gal (2015). A shared dropout mask zl is applied to the hidden state:
h̃l,t = rl,t ◦ h′l,t + (1− rl,t) ◦Wlhxl,t (15)
hl,t = zl ◦ h̃l,t (16)
zl is shared across timesteps at layer l to avoid amplifying the dropout noise along the sequence.
2.2 Constrained A∗ Decoding
The approach described so far does not model any dependencies between the output tags. To incorporate constraints on the output structure at decoding time, we use A∗ search over tag prefixes for decoding. Starting with an empty sequence, the tag sequence is built from left to right. The score for a partial sequence with length t is defined as:
f(w, y1:t) = t∑ i=1 log p(yi | w)− ∑ c∈C c(w, y1:i)
(17) An admissible A∗ heuristic can be computed efficiently by summing over the best possible tags for all timesteps after t:
g(w, y1:t) = n∑ i=t+1 max yi∈T log p(yi | w) (18)
Exploration of the prefixes is determined by an agenda A which is sorted by f(w, y1:t) + g(w, y1:t). In the worst case, A∗ explores exponentially many prefixes, but because the distribution p(yt | w) learned by the BiLSTM models is very peaked, the algorithm is efficient in practice. We list some example constraints as follows:
BIO Constraints These constraints reject any sequence that does not produce valid BIO transitions, such as BARG0 followed by IARG1.
SRL Constraints Punyakanok et al. (2008); Täckström et al. (2015) described a list of SRLspecific global constraints: • Unique core roles (U): Each core role
(ARG0-ARG5, ARGA) should appear at most once for each predicate. • Continuation roles (C): A continuation role
C-X can exist only when its base role X is realized before it. • Reference roles (R): A reference role R-X
can exist only when its base role X is realized (not necessarily before R-X).
We only enforce U and C constraints, since the R constraints are more commonly violated in gold data, and enforcing them results in worse performance (see discussions in Section 4.3).
Syntactic Constraints We can enforce consistency with a given parse tree by rejecting or penalizing arguments that are not constituents. In Section 4.4, we will discuss the motivation behind using syntactic constraints and experimental results using both predicted and gold syntax.
3 Experiments
3.1 Datasets
We measure the performance of our SRL system on two PropBank-style, span-based SRL datasets: CoNLL-2005 (Carreras and Màrquez, 2005) and CoNLL-2012 (Pradhan et al., 2013)2. Both datasets provide gold predicates (their index in the sentence) as part of the input. Therefore, each provided predicate corresponds to one training/test tag sequence. We follow the traindevelopment-test split for both datasets and use the official evaluation script from the CoNLL 2005 shared task for evaluation on both datasets.
3.2 Model Setup
Our network consists of 8 BiLSTM layers (4 forward LSTMs and 4 reversed LSTMs) with 300- dimensional hidden units, and a softmax layer for predicting the output distribution.
Initialization All the weight matrices in BiLSTMs are initialized with random orthonormal matrices as described in Saxe et al. (2013).
All the tokens are lower-cased and initialized with 100-dimensional GloVe embeddings pretrained on 6B tokens (Pennington et al., 2014) and updated during training. Tokens that are not covered by GloVe are replaced with a randomly initialized UNK embedding.
Training We use Adadelta (Zeiler, 2012) with = 1e−6 and ρ = 0.95 and mini-batches of size 80. We set RNN-dropout probability to 0.1 and clip gradients with norm larger than 1. All the models are trained for 500 epochs with early stopping based on development results 3.
2We used the version of OntoNotes downloaded at: http://cemantix.org/data/ontonotes.html.
3Training the full model on CoNLL 2005 takes about 5 days on a single Titan X Pascal GPU.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Development WSJ Test Brown Test Combined
Method P R F1 Comp. P R F1 Comp. P R F1 Comp. F1
Ours (PoE) 83.1 82.4 82.7 64.1 85.0 84.3 84.6 66.5 74.9 72.4 73.6 46.5 83.2 Ours 81.6 81.6 81.6 62.3 83.1 83.0 83.1 64.3 72.9 71.4 72.1 44.8 81.6
Zhou - - 79.6 - - - 82.8 - - - 69.4 - 81.1 FitzGerald (Struct.,PoE) 81.2 76.7 79.2 55.2 82.5 78.2 80.3 57.3 74.5 70.0 72.2 41.3 - Täckström (Struct.) 81.2 76.2 78.6 54.4 82.3 77.6 79.9 56.0 74.3 68.6 71.3 39.8 - Toutanova (Ensemble) - - 78.6 58.7 81.9 78.8 80.3 60.1 - - 68.8 40.8 - Punyakanok (Ensemble) 80.1 74.8 77.4 50.7 82.3 76.8 79.4 53.8 73.4 62.9 67.8 32.3 77.9
Table 1: Experimental results on CoNLL 2005, in terms of precision (P), recall (R), F1 and percentage of completely correct predicates (Comp.). We report results of our best single and ensemble (PoE) model. The comparison models are Zhou and Xu (2015), FitzGerald et al. (2015), Täckström et al. (2015), Toutanova et al. (2008) and Punyakanok et al. (2008).
Development Test
P R F1 Comp. P R F1 Comp.
Ours (PoE) 83.5 83.2 83.4 67.5 83.5 83.3 83.4 68.5 Ours 81.7 81.4 81.5 64.6 81.8 81.6 81.7 66.0
Zhou - - 81.1 - - - 81.3 - FitzGerald (Struct.,PoE) 81.0 78.5 79.7 60.9 81.2 79.0 80.2 62.6 Täckström (Struct.) 80.5 77.8 79.1 60.1 80.6 78.2 79.4 61.8 Pradhan (revised) 78.5 76.6 77.5 55.8 78.5 76.6 77.5 55.8
Table 2: Experimental results on CoNLL 2012 in the same metrics as above. We compare our best single and ensemble (PoE) models against Zhou and Xu (2015), FitzGerald et al. (2015), Täckström et al. (2015) and Pradhan et al. (2013).
Ensembling We use a product of experts (Hinton, 2002) to combine predictions of 5 models, each trained on 80% of the training corpus and validated on the remaining 20%. For the CoNLL 2012 corpus, we split the training data from each sub-genre into 5 folds, such that the training data will have similar genre distributions.
Constrained Decoding We experimented with different types of constraints on the CoNLL 2005 and CoNLL 2012 development sets. Only the BIO hard constraints significantly improve over the ensemble model. Therefore, in our final results, we only use BIO hard constraints during decoding 4.
3.3 Results
In Table 1 and 2, we compare our best single and ensemble model with previous work. Our ensemble (PoE) has an absolute improvement of 2.1 F1 on CoNLL 2005 and 2.3 on CoNLL 2012 over the previous state of the art. Our single model also achieves more than 0.4 improvement on both datasets. In comparison with the best reported results, our percentage of completely correct predicates improves by 5.9%. While the continuing trend of improving SRL without syntax seems to
4A∗ search in this setting finds the optimal sequence for all sentences and is therefore equivalent to Viterbi decoding.
0 100 200 300 400 500 Num. Epochs
66
68
70
72
74
76
78
80
82
D e v . F1
%
Our model No highway connections No RNN-dropout No orthonormal initialization
Figure 2: Learning curve of various ablations. The combination of highway layers, orthonormal parameter initialization and recurrent dropout is crucial to achieving strong performance. The numbers shown here are without constrained decoding.
suggest that neural end-to-end systems no longer needs parsers, our analysis in Section 4.4 will show that accurate syntactic information can improve these deep models.
3.4 Ablations
Figure 2 shows learning curves of our model ablations on the CoNLL 2005 development set. We ablate our full model by removing highway connections, RNN-dropout and orthonormal initialization independently. Without dropout, the model overfits at around 300 epochs at 78% F1. Or-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
thonormal parameter initialization is surprisingly important—without this, the model achieves only 65% F1 within the first 50 epochs. All 8 layer ablations suffer a loss of more than 1.7% in absolute F1 compared to the full model.
4 Analysis
To better understand our deep SRL model and its relation to previous work, we address the following questions with a suite of empirical analyses: • What is the model good at and what kind of
mistakes does the model make? • How well do LSTMs model global structural
consistency, despite conditionally independent tagging decisions? • Is our model implicitly learning syntax, and
could explicitly modeling syntax still help? All the analysis in this section is done on the CoNLL 2005 development set unless otherwise stated. We are also able to compare to previous systems with model predictions available online (Punyakanok et al., 2005; Pradhan et al., 2005)5.
4.1 Error Types Breakdown
Inspired Kummerfeld et al. (2012), we define a set of oracle transformations that fix various prediction errors and observe the relative improvement after each operation (See Table 3). Figure 3 shows how our work compare to the previous systems in terms of different types of mistakes. The analysis shows that while our model makes a similar number of labeling errors to traditional syntax-based systems, it has far fewer missing arguments (perhaps due to parser errors making some arguments difficult to recover for syntax-based systems).
Label Confusion As shown in Table 3, our system most commonly makes labeling errors, where the predicted span is an argument but the role was incorrectly labeled. Table 4 shows a confusion matrix for the most frequent labels. The model often confuses ARG2 with AM-DIR, AM-LOC and AM-MNR. These confusions can arise due to the use of ARG2 in many verb frames to represent semantic relations such as direction or location. For example, ARG2 in the frame move.01 is defined as Arg2-GOL: destination 6. This type of argument-
5Model predictions of CoNLL 2005 systems: http:// www.cs.upc.edu/˜srlconll/st05/st05.html
6Source: Unified verb index: http://verbs. colorado.edu.
Operation Description % Fix Labels Correct the span label its boundarymatches gold. 29.3 Move Arg. Move an unique core argument to itscorrect position. 4.5
Merge Spans
Combine two predicted spans into a gold span if they are separated by at most one word. 10.6
Split Spans
Split a predicted span into two gold spans that are separated by at most one word. 14.7
Fix Boundary Correct the boundary of a span if its label matches an overlapping gold span. 18.0 Drop Arg. Drop a predicted argument that does notoverlap with any gold span. 7.4 Add Arg. Add a gold argument does not overlapwith any predicted span. 11.0
Table 3: Oracle transformations paired with the relative error reduction after each operation. All the operations are permitted only if they do not cause any overlapping arguments.
adjunct distinction is known to be difficult (Kingsbury et al., 2002), and it is not surprising that our neural model has many such failure cases.
Attachment Mistakes A second common source of error is reflected by the Merge Spans transformation (10.6%) and the Split Spans transformation (14.7%). These errors are closely tied to prepositional phrase (PP) attachment errors, which are also known to be some of the biggest challenges for linguistic analysis (Kummerfeld et al., 2012). Figure 4 shows the distribution of syntactic span labels involved in an attachment mistake, where 62% of the syntactic spans are prepositional phrases. For example, in Sumitomo financed the acquisition from Sears our model mistakenly labels the prepositional phrase from
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
pred. \ gold A0 A1 A2 A3 ADV DIR LOC MNR PNC TMP A0 76 13 6 14 2 0 0 0 0 0 A1 16 74 25 0 0 18 9 11 19 2 A2 2 5 31 52 10 45 26 46 19 0 A3 1 0 1 57 2 0 0 0 19 2
ADV 0 0 0 5 33 0 11 33 19 5 DIR 0 0 3 5 0 27 9 2 0 0 LOC 1 2 7 0 2 0 34 11 0 2 MNR 1 0 7 29 21 0 0 43 0 3 PNC 0 1 3 5 0 9 3 2 44 0 TMP 0 2 3 0 26 9 20 7 0 71
Table 4: Confusion matrix of span labels, showing the percentage of predicted labels for each gold label. We only count predicted arguments that match gold span boundaries.
PP VP NP SBAR ADVP Other
0
20
40
60
80
100
62
10 10 5 4 9
% of
la be
ls
Figure 4: For cases where our model either splits a gold span into two (Z → XY ) or merges two gold constituents (XY → Z), we show the distribution of syntactic labels for the Y span. Results show the major cause of these errors is inaccurate prepositional phrase attachment.
Sears as the ARG2 of financed, whereas it should instead attach to acquisition.
4.2 Long-range Dependencies
To analyze the model’s ability to capture longrange dependencies, we compute the F1 of our model on arguments with various distances to the predicate. Figure 5 shows that for all models, performance tends to degrade for arguments further from the predicate. Interestingly, the gap between shallow and deep models becomes much larger for the long-distance predicate-argument structures. The absolute gap between our 2 layer and 8 layer models is 3-4 F1 for arguments that are within 2 words to the predicate, and 5-6 F1 for arguments that are farther away from the predicate. Surprisingly, the neural model performance deteriorates less severely on long-range dependencies than traditional syntax-based models.
4.3 Structural Consistency
We can quantify two types of structural consistencies: the BIO constraints and the SRL-specific constraints. Via our ablation study, we show that deeper BiLSTMs are better at enforcing these structural consistencies, although not perfectly.
BIO Violations The BIO format requires argument spans to begin with a B tag. Any I tag directly following an O tag or a tag with different label is considered a violation. Table 5 shows the number of BIO violations per token for BiLSTMs with different depths. The number of BIO violations decreases when we use a deeper model. The gap is biggest between 2-layer and 4-layer models, and diminishes after that.
It is surprising that although the deeper models generate impressively accurate token-level predictions, they still make enough BIO errors to significantly hurt performance—when these constraints are simple enough to be enforced by trivial rules. To understand the reason behind it, we compare the average entropy between tokens involved BIO violations with the averaged entropy of all tokens. For the 8-layer model, the average entropy on these tokens is 30 times higher than the averaged entropy on all tokens. This suggests that BIO inconsistencies occur when there is some ambiguity. For example, if the model is unsure whether two consecutive words should belong to an ARG0 or ARG1, it might generate inconsistent BIO sequences such as BARG0, IARG1 when decoding at the token level. Using BIO-constrained decoding can resolve this ambiguity and result in a structurally consistent solution.
SRL Structure Violations The model predictions can also violate the SRL-specific constraints commonly used in prior work (Punyakanok et al., 2008; Täckström et al., 2015). As shown in Table 6, the model occasionally violates these SRL constraints. With our constrained decoding algorithm, it is straightforward to enforce the unique core roles (U) and continuation roles (C) constraints during decoding. The constrained de-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
656
657
658
659
660
661
662
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Housing starts are expected to quicken a bit from August’s pace
ARG0
ARG1
ARG2ARG2
ARG1
V ARG2 ARG3
ARG2ARG0
V
V
Gold
Pred.
+SRL
Figure 6: Example where performance is hurt by enforcing the constraint that core roles may only occur once (+SRL).
coding results are shown with the model named L8+PoE+SRL in Table 6.
Although the violations are eliminated, the performance does not significantly improve. This is mainly due to two factors: (1) the model often already satisfies these constraints on its own, so the number of violations to be fixed are relatively small, and (2) the gold SRL structure sometimes violate the constraints and enforcing hard constraints can hurt performance. Figure 6 shows a sentence in the CoNLL 2005 development set. Our original model produces two ARG2s for the predicate quicken, and this violates the SRL constraints. When the A∗ decoder fixes this violation, it changes the first ARG1 into ARG2 because ARG0,ARG1,ARG2 is a more frequent pattern in the training data and has higher overall score.
4.4 Can Syntax Still Help SRL?
The Propbank-style SRL formalism is closely tied to syntax (Bonial et al., 2010; Weischedel et al., 2011). In Table 6, we show that 98.7% of the gold SRL arguments match an unlabeled constituent in the gold syntax tree. Similar to some recent work (Zhou and Xu, 2015), our model achieved strong performance without directly modeling syntax. A natural question follows: are neural SRL models implicitly learning syntax? Table 6 shows the trend of deeper models making predictions that are more consistent with the gold syntax in terms of span boundaries. With our best model, 94.3% of the predicted arguments spans are part of the gold parse tree. This consistency is on par with previous CoNLL 2005 systems that directly model constituency and use predicted parse trees as features (Punyakanok, 95.3% and Pradhan, 93.0%).
Constrained Decoding with Syntax The above analysis raises a further question: would improving consistency with syntax provide improvements for SRL? Our constrained decoding algorithm described in Section 2.2 enables us to inject syntax as a decoding constraint without having to re-train
the model. In Figure 7, we compare the SRL accuracy with syntactic constraints specified by gold parse or automatic parses. The penalty of disagreeing with the parse tree is a single parameter dictating how much the model should trust the provided syntax. When using gold syntax, the predictions improve up to 2 F1 as the penalty increases. The improvement from using Choe’s parser (Choe and Charniak, 2016) is much smaller, while using the Charniak parser (Charniak, 2000) hurts performance if the model places too much trust in it. These results suggest that high-quality syntax can still make a large impact on SRL.
A known challenge for syntactic parsers is robustness on out-of-domain data. Therefore we
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
0 1 10 100 1000 10000 ∞
82
83
84
85
Penalty
F1 %
Choe Charniak Gold
Figure 7: Performance of syntax-constrained decoding as the non-constituent penalty increases for syntax from two parsers (from Choe and Charniak (2016) and Charniak (2000)) and gold syntax. The best existing parser gives a small improvement, but the improvement from gold syntax shows that there is still potential for syntax to help SRL.
CoNLL-05 CoNLL-2012 Dev.
Dev. Test BC BN NW MZ PT TC WB
L8+PoE 82.7 84.6 81.4 82.8 82.8 80.4 93.6 84.8 81.0 +AutoSyntax 83.2 84.8 81.5 82.8 83.2 80.6 93.7 84.9 81.1
Table 7: F1 on CoNLL 2005, and the development set of CoNLL 2012, broken down by genres. Syntax-constrained decoding (+AutoSyntax) shows bigger improvement on in-domain data (CoNLL 05 and CoNLL 2012 NW).
provide experimental results in Table 7 for both CoNLL 2005 and CoNLL 2012, which consists of 8 different genres. The penalties are tuned on the two development sets separately (10000 on CoNLL 2005 and 20 on CoNLL 2012). On the CoNLL 2005 development set, the predicted syntax gives a 0.5 F1 improvement over our best model, while on in-domain test and outof-domain development sets, the improvement is much smaller. As expected, on CoNLL 2012, syntax improves most on the newswire (NW) domain. These improvements suggest that while decoding with hard constraints is beneficial, joint training or multi-task learning could be even more effective by leveraging full, labeled syntactic structures.
5 Related Work
Traditional approaches to semantic role labeling have used syntactic parsers to identify constituents and model long-range dependencies, and enforced global consistency using integer linear programming (Punyakanok et al., 2008) or dynamic programs (Täckström et al., 2015). More recently, neural methods have been employed on top of syntactic features (FitzGerald et al., 2015; Roth and Lapata, 2016). Our experiments show that off-
the-shelf neural methods have a remarkable ability to learn long-range dependencies, syntactic constituency structure, and global constraints without coding task-specific mechanisms for doing so.
An alternative line of work has attempted to reduce the dependency on syntactic input for semantic role labeling models. Collobert et al. (2011) first introduced an end-to-end neural-based approach with sequence-level training and uses a convolutional neural network to model the context window. However, their best system fell short of traditional feature-based systems. Neural methods have also been used as classifiers in transition-based SRL systems (Henderson et al., 2013; Swayamdipta et al., 2016). Most recently, several successful LSTM-based architectures have achieved state-of-the-art results in English span-based SRL (Zhou and Xu, 2015), Chinese SRL (Wang et al., 2015) and dependencybased SRL (Marcheggiani et al., 2017), with little to no syntactic input. Our techniques push results to more than 3 F1 over the best syntax-based models. However, we also show that there is potential for syntax to further improve performance.
6 Conclusion and Future Work
We presented a new deep learning model for spanbased semantic role labeling with a 10% relative error reduction over the previous state of the art. Our ensemble of 8 layer BiLSTMs incorporated some of the recent best practices such as orthonormal initialization, RNN-dropout, and highway connections, and we have shown that they are crucial for getting good results with deep models.
Extensive error analysis sheds light on the strengths and limitations of our deep SRL model, with detailed comparison against shallower models and two strong non-neural systems. While our deep model is better at recovering longdistance predicate-argument relations, we still observe structural inconsistencies, which can be alleviated by constrained A∗ decoding.
Finally, we posed the question of whether deep SRL still needs syntactic supervision. Despite recent success without syntactic input, we found that our best neural model can still benefit from accurate syntactic parser output via straightforward constrained decoding. Our oracle experiment, where the SRL system receives a 3 F1 gain from gold syntax showed potential for syntax to further improve deep SRL models.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
This paper presents a new state-of-the-art deep learning model for semantic role labeling (SRL) that is a natural extension of the previous state-of-the-art system (Zhou and Xu, 2015) with recent best practices for initialization and regularization in the deep learning literature. The model gives a 10% relative error reduction which is a big gain on this task. The paper also gives in-depth empirical analyses to reveal the strengths and the remaining issues, that give a quite valuable information to the researchers in this field.   Even though I understand that the improvement of 3 point in F1 measure is a quite meaningful result from the engineering point of view, I think the main contribution of the paper is on the extensive analysis in the experiment section and a further in-depth investigation on analysis section. The detailed analyses shown in Section 4 are performed in a quite reasonable way and give both comparable results in SRL literature and novel information such as relation between accuracies in syntactic parsing and SRL. This type of analysis had often been omitted in recent papers. However, it is definitely important for further improvement.  The paper is well-written and well-structured.  I really enjoyed the paper and would like to see it accepted.