Deep Semantic Role Labeling: What Works and What’s Next
We introduce a new deep learning model for semantic role labeling (SRL) that significantly improves the state of the art, along with detailed analyses to reveal its strengths and limitations. We use a deep highway BiLSTM architecture with constrained decoding, while observing a number of recent best practices for initialization and regularization. Our 8-layer ensemble model achieves 83.2 F1 on the CoNLL 2005 test set and 83.4 F1 on CoNLL 2012, roughly a 10% relative error reduction over the previous state of the art. Extensive empirical analysis of these gains show that (1) deep models excel at recovering long-distance dependencies but can still make surprisingly obvious errors, and (2) that there is still room for syntactic parsers to improve these results. All code and models will be publicly released.
1 Introduction
Semantic role labeling (SRL) systems aim to recover the predicate-argument structure of a sentence, to determine essentially “who did what to whom”, “when”, and “where.” Recently breakthroughs involving end-to-end deep models for SRL without syntactic input (Zhou and Xu, 2015; Marcheggiani et al., 2017) seem to overturn the long-held belief that syntactic parsing is a prerequisite for this task (Punyakanok et al., 2008). In this paper, we show that this result can be pushed further using deep highway bidirectional LSTMs with constrained decoding, again significantly moving the state of the art (another 2 points on CoNLL 2005). We also present careful empirical analysis to determine what works well and what might be done to progress even further.
Our model combines a number of best practices in the recent deep learning literature. Following Zhou and Xu (2015), we treat SRL as a BIO tagging problem and use deep bidirectional LSTMs. However, we differ by (1) simplifying the input and output layers, (2) introducing highway connections (Srivastava et al., 2015; Zhang et al., 2016), (3) using recurrent dropout (Gal, 2015), (4) decoding with BIO-constraints, and (5) ensembling with a product of experts. Our model gives a 10% relative error reduction over previous state of the art on the test sets of CoNLL 2005 and 2012. All code and models will be publicly released.
We present a set of detailed error analyses to better understand the performance gains, including (1) design choices on architecture, initialization, and regularization that have a surprisingly large impact on model performance, (2) analyses of different types of prediction errors, e.g. showing that deep models excel at predicting long-distance dependencies but still struggles with known challenges such as PP-attachment errors and adjunct-argument distinctions, (3) a careful analysis of the role of syntax, showing that there is significant room for improvement given oracle syntax but errors from existing parsers prevent effective use in SRL.
In summary, our main contributions are:
• A new state-of-the-art deep network for endto-end SRL, supported by code and models that will be publicly available. • An in-depth error analysis indicating where
the models work well and where they still struggle, including discussion of structural consistency and long-distance dependencies. • Experiments that point toward directions for
future improvements, including a detailed discussion of how and when syntactic parsers could be used to improve these results.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
2 Model
Two major factors contribute to the success of our deep SRL model: (1) applying recent advances in training deep recurrent neural networks such as highway connections (Srivastava et al., 2015; Zhang et al., 2016) and RNN-dropouts (Gal, 2015)1; and (2) using an A∗ decoding algorithm, similar to Lewis and Steedman (2014) to enforce structural consistency at prediction time, without adding more complexity to the training process.
Formally, the goal of our task is to predict a sequence y given a sentence-predicate pair (w, v) as input. Each yi belongs to a discrete set of BIO tags T . Words outside argument spans have the tag O, and words at the beginning and inside of argument with role r have the tags Br and Ir respectively. Let n = |w| = |y| be the length of the sequence.
Predicting an SRL structure under our model involves finding the highest-scoring tag sequence over the space of all possibilities Y:
ŷ = argmax y∈Y f(w,y) (1)
We use a deep bidirectional LSTM (BiLSTM) to learn a locally decomposed scoring function conditioned on the input: ∑n t=1 log p(yt | w).
To incorporate additional information (e.g. structural consistency, syntactic input), we augment the scoring function with penalization terms:
f(w,y) = n∑ t=1 log p(yt | w)− ∑ c∈C c(w, y1:t) (2)
Each constraint function c applies a non-negative penalty given the input w and a length-t prefix y1:t. These constraints can be hard or soft depending on whether the penalties are finite.
2.1 Deep BiLSTM Model
Our model computes the distribution over tags using stacked BiLSTMs, which we define as follows:
il,t = σ(W l i [hl,t+δl ,xl,t] + b l i) (3) ol,t = σ(W l o[hl,t+δl ,xl,t] + b l o) (4) fl,t = σ(W l f[hl,t+δl ,xl,t] + b l f + 1) (5) c̃l,t = tanh(W l c[hl,t+δl ,xl,t] + b l c) (6) cl,t = il,t ◦ c̃l,t + fl,t ◦ ct+δl (7) hl,t = ol,t ◦ tanh(cl,t) (8)
1We thank Mingxuan Wang for suggesting highway connections with simplified inputs and outputs. Part of our model is extended from his unpublished implementation.
where xl,t is the input to the LSTM at layer l and timestep t. δl is either 1 or −1, indicating the directionality of the LSTM at layer l.
To stack the LSTMs in an interleaving pattern, as proposed by Zhou and Xu (2015), the layerspecific inputs xl,t and directionality δl are arranged in the following manner:
xl,t =
{ [Wemb(wt),Wmask(t = v)] l = 1
hl−1,t l > 1 (9)
δl = { 1 if l is even −1 otherwise
(10)
The input vector x1,t is the concatenation of token wt’s word embedding and an embedding of the binary feature (t = v) indicating whether the current word is the given predicate.
Finally, the locally normalized distribution over output tags is computed via a softmax layer:
p(yt | x) ∝ exp(WytaghL,t + btag) (11)
Highway Connections To alleviate the vanishing gradient problem when training deep BiLSTMs, we use gated highway connections described in Zhang et al. (2016); Srivastava et al. (2015). We include transform gates rt to control the weight of linear and non-linear transformations between layers (See Figure 1). The output hl,t is changed to:
rl,t = σ(W l r[hl,t−1,xt] + b l r) (12) h′l,t = ol,t ◦ tanh(cl,t) (13) hl,t = rl,t ◦ h′l,t + (1− rl,t) ◦Wlhxl,t (14)
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Recurrent Dropout To reduce over-fitting, we use dropout as described in Gal (2015). A shared dropout mask zl is applied to the hidden state:
h̃l,t = rl,t ◦ h′l,t + (1− rl,t) ◦Wlhxl,t (15)
hl,t = zl ◦ h̃l,t (16)
zl is shared across timesteps at layer l to avoid amplifying the dropout noise along the sequence.
2.2 Constrained A∗ Decoding
The approach described so far does not model any dependencies between the output tags. To incorporate constraints on the output structure at decoding time, we use A∗ search over tag prefixes for decoding. Starting with an empty sequence, the tag sequence is built from left to right. The score for a partial sequence with length t is defined as:
f(w, y1:t) = t∑ i=1 log p(yi | w)− ∑ c∈C c(w, y1:i)
(17) An admissible A∗ heuristic can be computed efficiently by summing over the best possible tags for all timesteps after t:
g(w, y1:t) = n∑ i=t+1 max yi∈T log p(yi | w) (18)
Exploration of the prefixes is determined by an agenda A which is sorted by f(w, y1:t) + g(w, y1:t). In the worst case, A∗ explores exponentially many prefixes, but because the distribution p(yt | w) learned by the BiLSTM models is very peaked, the algorithm is efficient in practice. We list some example constraints as follows:
BIO Constraints These constraints reject any sequence that does not produce valid BIO transitions, such as BARG0 followed by IARG1.
SRL Constraints Punyakanok et al. (2008); Täckström et al. (2015) described a list of SRLspecific global constraints: • Unique core roles (U): Each core role
(ARG0-ARG5, ARGA) should appear at most once for each predicate. • Continuation roles (C): A continuation role
C-X can exist only when its base role X is realized before it. • Reference roles (R): A reference role R-X
can exist only when its base role X is realized (not necessarily before R-X).
We only enforce U and C constraints, since the R constraints are more commonly violated in gold data, and enforcing them results in worse performance (see discussions in Section 4.3).
Syntactic Constraints We can enforce consistency with a given parse tree by rejecting or penalizing arguments that are not constituents. In Section 4.4, we will discuss the motivation behind using syntactic constraints and experimental results using both predicted and gold syntax.
3 Experiments
3.1 Datasets
We measure the performance of our SRL system on two PropBank-style, span-based SRL datasets: CoNLL-2005 (Carreras and Màrquez, 2005) and CoNLL-2012 (Pradhan et al., 2013)2. Both datasets provide gold predicates (their index in the sentence) as part of the input. Therefore, each provided predicate corresponds to one training/test tag sequence. We follow the traindevelopment-test split for both datasets and use the official evaluation script from the CoNLL 2005 shared task for evaluation on both datasets.
3.2 Model Setup
Our network consists of 8 BiLSTM layers (4 forward LSTMs and 4 reversed LSTMs) with 300- dimensional hidden units, and a softmax layer for predicting the output distribution.
Initialization All the weight matrices in BiLSTMs are initialized with random orthonormal matrices as described in Saxe et al. (2013).
All the tokens are lower-cased and initialized with 100-dimensional GloVe embeddings pretrained on 6B tokens (Pennington et al., 2014) and updated during training. Tokens that are not covered by GloVe are replaced with a randomly initialized UNK embedding.
Training We use Adadelta (Zeiler, 2012) with = 1e−6 and ρ = 0.95 and mini-batches of size 80. We set RNN-dropout probability to 0.1 and clip gradients with norm larger than 1. All the models are trained for 500 epochs with early stopping based on development results 3.
2We used the version of OntoNotes downloaded at: http://cemantix.org/data/ontonotes.html.
3Training the full model on CoNLL 2005 takes about 5 days on a single Titan X Pascal GPU.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Development WSJ Test Brown Test Combined
Method P R F1 Comp. P R F1 Comp. P R F1 Comp. F1
Ours (PoE) 83.1 82.4 82.7 64.1 85.0 84.3 84.6 66.5 74.9 72.4 73.6 46.5 83.2 Ours 81.6 81.6 81.6 62.3 83.1 83.0 83.1 64.3 72.9 71.4 72.1 44.8 81.6
Zhou - - 79.6 - - - 82.8 - - - 69.4 - 81.1 FitzGerald (Struct.,PoE) 81.2 76.7 79.2 55.2 82.5 78.2 80.3 57.3 74.5 70.0 72.2 41.3 - Täckström (Struct.) 81.2 76.2 78.6 54.4 82.3 77.6 79.9 56.0 74.3 68.6 71.3 39.8 - Toutanova (Ensemble) - - 78.6 58.7 81.9 78.8 80.3 60.1 - - 68.8 40.8 - Punyakanok (Ensemble) 80.1 74.8 77.4 50.7 82.3 76.8 79.4 53.8 73.4 62.9 67.8 32.3 77.9
Table 1: Experimental results on CoNLL 2005, in terms of precision (P), recall (R), F1 and percentage of completely correct predicates (Comp.). We report results of our best single and ensemble (PoE) model. The comparison models are Zhou and Xu (2015), FitzGerald et al. (2015), Täckström et al. (2015), Toutanova et al. (2008) and Punyakanok et al. (2008).
Development Test
P R F1 Comp. P R F1 Comp.
Ours (PoE) 83.5 83.2 83.4 67.5 83.5 83.3 83.4 68.5 Ours 81.7 81.4 81.5 64.6 81.8 81.6 81.7 66.0
Zhou - - 81.1 - - - 81.3 - FitzGerald (Struct.,PoE) 81.0 78.5 79.7 60.9 81.2 79.0 80.2 62.6 Täckström (Struct.) 80.5 77.8 79.1 60.1 80.6 78.2 79.4 61.8 Pradhan (revised) 78.5 76.6 77.5 55.8 78.5 76.6 77.5 55.8
Table 2: Experimental results on CoNLL 2012 in the same metrics as above. We compare our best single and ensemble (PoE) models against Zhou and Xu (2015), FitzGerald et al. (2015), Täckström et al. (2015) and Pradhan et al. (2013).
Ensembling We use a product of experts (Hinton, 2002) to combine predictions of 5 models, each trained on 80% of the training corpus and validated on the remaining 20%. For the CoNLL 2012 corpus, we split the training data from each sub-genre into 5 folds, such that the training data will have similar genre distributions.
Constrained Decoding We experimented with different types of constraints on the CoNLL 2005 and CoNLL 2012 development sets. Only the BIO hard constraints significantly improve over the ensemble model. Therefore, in our final results, we only use BIO hard constraints during decoding 4.
3.3 Results
In Table 1 and 2, we compare our best single and ensemble model with previous work. Our ensemble (PoE) has an absolute improvement of 2.1 F1 on CoNLL 2005 and 2.3 on CoNLL 2012 over the previous state of the art. Our single model also achieves more than 0.4 improvement on both datasets. In comparison with the best reported results, our percentage of completely correct predicates improves by 5.9%. While the continuing trend of improving SRL without syntax seems to
4A∗ search in this setting finds the optimal sequence for all sentences and is therefore equivalent to Viterbi decoding.
0 100 200 300 400 500 Num. Epochs
66
68
70
72
74
76
78
80
82
D e v . F1
%
Our model No highway connections No RNN-dropout No orthonormal initialization
Figure 2: Learning curve of various ablations. The combination of highway layers, orthonormal parameter initialization and recurrent dropout is crucial to achieving strong performance. The numbers shown here are without constrained decoding.
suggest that neural end-to-end systems no longer needs parsers, our analysis in Section 4.4 will show that accurate syntactic information can improve these deep models.
3.4 Ablations
Figure 2 shows learning curves of our model ablations on the CoNLL 2005 development set. We ablate our full model by removing highway connections, RNN-dropout and orthonormal initialization independently. Without dropout, the model overfits at around 300 epochs at 78% F1. Or-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
thonormal parameter initialization is surprisingly important—without this, the model achieves only 65% F1 within the first 50 epochs. All 8 layer ablations suffer a loss of more than 1.7% in absolute F1 compared to the full model.
4 Analysis
To better understand our deep SRL model and its relation to previous work, we address the following questions with a suite of empirical analyses: • What is the model good at and what kind of
mistakes does the model make? • How well do LSTMs model global structural
consistency, despite conditionally independent tagging decisions? • Is our model implicitly learning syntax, and
could explicitly modeling syntax still help? All the analysis in this section is done on the CoNLL 2005 development set unless otherwise stated. We are also able to compare to previous systems with model predictions available online (Punyakanok et al., 2005; Pradhan et al., 2005)5.
4.1 Error Types Breakdown
Inspired Kummerfeld et al. (2012), we define a set of oracle transformations that fix various prediction errors and observe the relative improvement after each operation (See Table 3). Figure 3 shows how our work compare to the previous systems in terms of different types of mistakes. The analysis shows that while our model makes a similar number of labeling errors to traditional syntax-based systems, it has far fewer missing arguments (perhaps due to parser errors making some arguments difficult to recover for syntax-based systems).
Label Confusion As shown in Table 3, our system most commonly makes labeling errors, where the predicted span is an argument but the role was incorrectly labeled. Table 4 shows a confusion matrix for the most frequent labels. The model often confuses ARG2 with AM-DIR, AM-LOC and AM-MNR. These confusions can arise due to the use of ARG2 in many verb frames to represent semantic relations such as direction or location. For example, ARG2 in the frame move.01 is defined as Arg2-GOL: destination 6. This type of argument-
5Model predictions of CoNLL 2005 systems: http:// www.cs.upc.edu/˜srlconll/st05/st05.html
6Source: Unified verb index: http://verbs. colorado.edu.
Operation Description % Fix Labels Correct the span label its boundarymatches gold. 29.3 Move Arg. Move an unique core argument to itscorrect position. 4.5
Merge Spans
Combine two predicted spans into a gold span if they are separated by at most one word. 10.6
Split Spans
Split a predicted span into two gold spans that are separated by at most one word. 14.7
Fix Boundary Correct the boundary of a span if its label matches an overlapping gold span. 18.0 Drop Arg. Drop a predicted argument that does notoverlap with any gold span. 7.4 Add Arg. Add a gold argument does not overlapwith any predicted span. 11.0
Table 3: Oracle transformations paired with the relative error reduction after each operation. All the operations are permitted only if they do not cause any overlapping arguments.
adjunct distinction is known to be difficult (Kingsbury et al., 2002), and it is not surprising that our neural model has many such failure cases.
Attachment Mistakes A second common source of error is reflected by the Merge Spans transformation (10.6%) and the Split Spans transformation (14.7%). These errors are closely tied to prepositional phrase (PP) attachment errors, which are also known to be some of the biggest challenges for linguistic analysis (Kummerfeld et al., 2012). Figure 4 shows the distribution of syntactic span labels involved in an attachment mistake, where 62% of the syntactic spans are prepositional phrases. For example, in Sumitomo financed the acquisition from Sears our model mistakenly labels the prepositional phrase from
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
pred. \ gold A0 A1 A2 A3 ADV DIR LOC MNR PNC TMP A0 76 13 6 14 2 0 0 0 0 0 A1 16 74 25 0 0 18 9 11 19 2 A2 2 5 31 52 10 45 26 46 19 0 A3 1 0 1 57 2 0 0 0 19 2
ADV 0 0 0 5 33 0 11 33 19 5 DIR 0 0 3 5 0 27 9 2 0 0 LOC 1 2 7 0 2 0 34 11 0 2 MNR 1 0 7 29 21 0 0 43 0 3 PNC 0 1 3 5 0 9 3 2 44 0 TMP 0 2 3 0 26 9 20 7 0 71
Table 4: Confusion matrix of span labels, showing the percentage of predicted labels for each gold label. We only count predicted arguments that match gold span boundaries.
PP VP NP SBAR ADVP Other
0
20
40
60
80
100
62
10 10 5 4 9
% of
la be
ls
Figure 4: For cases where our model either splits a gold span into two (Z → XY ) or merges two gold constituents (XY → Z), we show the distribution of syntactic labels for the Y span. Results show the major cause of these errors is inaccurate prepositional phrase attachment.
Sears as the ARG2 of financed, whereas it should instead attach to acquisition.
4.2 Long-range Dependencies
To analyze the model’s ability to capture longrange dependencies, we compute the F1 of our model on arguments with various distances to the predicate. Figure 5 shows that for all models, performance tends to degrade for arguments further from the predicate. Interestingly, the gap between shallow and deep models becomes much larger for the long-distance predicate-argument structures. The absolute gap between our 2 layer and 8 layer models is 3-4 F1 for arguments that are within 2 words to the predicate, and 5-6 F1 for arguments that are farther away from the predicate. Surprisingly, the neural model performance deteriorates less severely on long-range dependencies than traditional syntax-based models.
4.3 Structural Consistency
We can quantify two types of structural consistencies: the BIO constraints and the SRL-specific constraints. Via our ablation study, we show that deeper BiLSTMs are better at enforcing these structural consistencies, although not perfectly.
BIO Violations The BIO format requires argument spans to begin with a B tag. Any I tag directly following an O tag or a tag with different label is considered a violation. Table 5 shows the number of BIO violations per token for BiLSTMs with different depths. The number of BIO violations decreases when we use a deeper model. The gap is biggest between 2-layer and 4-layer models, and diminishes after that.
It is surprising that although the deeper models generate impressively accurate token-level predictions, they still make enough BIO errors to significantly hurt performance—when these constraints are simple enough to be enforced by trivial rules. To understand the reason behind it, we compare the average entropy between tokens involved BIO violations with the averaged entropy of all tokens. For the 8-layer model, the average entropy on these tokens is 30 times higher than the averaged entropy on all tokens. This suggests that BIO inconsistencies occur when there is some ambiguity. For example, if the model is unsure whether two consecutive words should belong to an ARG0 or ARG1, it might generate inconsistent BIO sequences such as BARG0, IARG1 when decoding at the token level. Using BIO-constrained decoding can resolve this ambiguity and result in a structurally consistent solution.
SRL Structure Violations The model predictions can also violate the SRL-specific constraints commonly used in prior work (Punyakanok et al., 2008; Täckström et al., 2015). As shown in Table 6, the model occasionally violates these SRL constraints. With our constrained decoding algorithm, it is straightforward to enforce the unique core roles (U) and continuation roles (C) constraints during decoding. The constrained de-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
656
657
658
659
660
661
662
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Housing starts are expected to quicken a bit from August’s pace
ARG0
ARG1
ARG2ARG2
ARG1
V ARG2 ARG3
ARG2ARG0
V
V
Gold
Pred.
+SRL
Figure 6: Example where performance is hurt by enforcing the constraint that core roles may only occur once (+SRL).
coding results are shown with the model named L8+PoE+SRL in Table 6.
Although the violations are eliminated, the performance does not significantly improve. This is mainly due to two factors: (1) the model often already satisfies these constraints on its own, so the number of violations to be fixed are relatively small, and (2) the gold SRL structure sometimes violate the constraints and enforcing hard constraints can hurt performance. Figure 6 shows a sentence in the CoNLL 2005 development set. Our original model produces two ARG2s for the predicate quicken, and this violates the SRL constraints. When the A∗ decoder fixes this violation, it changes the first ARG1 into ARG2 because ARG0,ARG1,ARG2 is a more frequent pattern in the training data and has higher overall score.
4.4 Can Syntax Still Help SRL?
The Propbank-style SRL formalism is closely tied to syntax (Bonial et al., 2010; Weischedel et al., 2011). In Table 6, we show that 98.7% of the gold SRL arguments match an unlabeled constituent in the gold syntax tree. Similar to some recent work (Zhou and Xu, 2015), our model achieved strong performance without directly modeling syntax. A natural question follows: are neural SRL models implicitly learning syntax? Table 6 shows the trend of deeper models making predictions that are more consistent with the gold syntax in terms of span boundaries. With our best model, 94.3% of the predicted arguments spans are part of the gold parse tree. This consistency is on par with previous CoNLL 2005 systems that directly model constituency and use predicted parse trees as features (Punyakanok, 95.3% and Pradhan, 93.0%).
Constrained Decoding with Syntax The above analysis raises a further question: would improving consistency with syntax provide improvements for SRL? Our constrained decoding algorithm described in Section 2.2 enables us to inject syntax as a decoding constraint without having to re-train
the model. In Figure 7, we compare the SRL accuracy with syntactic constraints specified by gold parse or automatic parses. The penalty of disagreeing with the parse tree is a single parameter dictating how much the model should trust the provided syntax. When using gold syntax, the predictions improve up to 2 F1 as the penalty increases. The improvement from using Choe’s parser (Choe and Charniak, 2016) is much smaller, while using the Charniak parser (Charniak, 2000) hurts performance if the model places too much trust in it. These results suggest that high-quality syntax can still make a large impact on SRL.
A known challenge for syntactic parsers is robustness on out-of-domain data. Therefore we
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
0 1 10 100 1000 10000 ∞
82
83
84
85
Penalty
F1 %
Choe Charniak Gold
Figure 7: Performance of syntax-constrained decoding as the non-constituent penalty increases for syntax from two parsers (from Choe and Charniak (2016) and Charniak (2000)) and gold syntax. The best existing parser gives a small improvement, but the improvement from gold syntax shows that there is still potential for syntax to help SRL.
CoNLL-05 CoNLL-2012 Dev.
Dev. Test BC BN NW MZ PT TC WB
L8+PoE 82.7 84.6 81.4 82.8 82.8 80.4 93.6 84.8 81.0 +AutoSyntax 83.2 84.8 81.5 82.8 83.2 80.6 93.7 84.9 81.1
Table 7: F1 on CoNLL 2005, and the development set of CoNLL 2012, broken down by genres. Syntax-constrained decoding (+AutoSyntax) shows bigger improvement on in-domain data (CoNLL 05 and CoNLL 2012 NW).
provide experimental results in Table 7 for both CoNLL 2005 and CoNLL 2012, which consists of 8 different genres. The penalties are tuned on the two development sets separately (10000 on CoNLL 2005 and 20 on CoNLL 2012). On the CoNLL 2005 development set, the predicted syntax gives a 0.5 F1 improvement over our best model, while on in-domain test and outof-domain development sets, the improvement is much smaller. As expected, on CoNLL 2012, syntax improves most on the newswire (NW) domain. These improvements suggest that while decoding with hard constraints is beneficial, joint training or multi-task learning could be even more effective by leveraging full, labeled syntactic structures.
5 Related Work
Traditional approaches to semantic role labeling have used syntactic parsers to identify constituents and model long-range dependencies, and enforced global consistency using integer linear programming (Punyakanok et al., 2008) or dynamic programs (Täckström et al., 2015). More recently, neural methods have been employed on top of syntactic features (FitzGerald et al., 2015; Roth and Lapata, 2016). Our experiments show that off-
the-shelf neural methods have a remarkable ability to learn long-range dependencies, syntactic constituency structure, and global constraints without coding task-specific mechanisms for doing so.
An alternative line of work has attempted to reduce the dependency on syntactic input for semantic role labeling models. Collobert et al. (2011) first introduced an end-to-end neural-based approach with sequence-level training and uses a convolutional neural network to model the context window. However, their best system fell short of traditional feature-based systems. Neural methods have also been used as classifiers in transition-based SRL systems (Henderson et al., 2013; Swayamdipta et al., 2016). Most recently, several successful LSTM-based architectures have achieved state-of-the-art results in English span-based SRL (Zhou and Xu, 2015), Chinese SRL (Wang et al., 2015) and dependencybased SRL (Marcheggiani et al., 2017), with little to no syntactic input. Our techniques push results to more than 3 F1 over the best syntax-based models. However, we also show that there is potential for syntax to further improve performance.
6 Conclusion and Future Work
We presented a new deep learning model for spanbased semantic role labeling with a 10% relative error reduction over the previous state of the art. Our ensemble of 8 layer BiLSTMs incorporated some of the recent best practices such as orthonormal initialization, RNN-dropout, and highway connections, and we have shown that they are crucial for getting good results with deep models.
Extensive error analysis sheds light on the strengths and limitations of our deep SRL model, with detailed comparison against shallower models and two strong non-neural systems. While our deep model is better at recovering longdistance predicate-argument relations, we still observe structural inconsistencies, which can be alleviated by constrained A∗ decoding.
Finally, we posed the question of whether deep SRL still needs syntactic supervision. Despite recent success without syntactic input, we found that our best neural model can still benefit from accurate syntactic parser output via straightforward constrained decoding. Our oracle experiment, where the SRL system receives a 3 F1 gain from gold syntax showed potential for syntax to further improve deep SRL models.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper presents a new state-of-the-art deep learning model for semantic role labeling (SRL) that is a natural extension of the previous state-of-the-art system (Zhou and Xu, 2015) with recent best practices for initialization and regularization in the deep learning literature. The model gives a 10% relative error reduction which is a big gain on this task. The paper also gives in-depth empirical analyses to reveal the strengths and the remaining issues, that give a quite valuable information to the researchers in this field.   Even though I understand that the improvement of 3 point in F1 measure is a quite meaningful result from the engineering point of view, I think the main contribution of the paper is on the extensive analysis in the experiment section and a further in-depth investigation on analysis section. The detailed analyses shown in Section 4 are performed in a quite reasonable way and give both comparable results in SRL literature and novel information such as relation between accuracies in syntactic parsing and SRL. This type of analysis had often been omitted in recent papers. However, it is definitely important for further improvement.  The paper is well-written and well-structured.  I really enjoyed the paper and would like to see it accepted.


Reading Wikipedia to Answer Open-Domain Questions
This paper proposes to tackle opendomain question answering using Wikipedia as the unique knowledge source: the answer to any factoid question is a text span in a Wikipedia article. This problem combines the challenges of document retrieval, to find relevant articles, and of machine comprehension of text to identify the answer spans from those articles. Our approach combines a search component based on bigram hashing and TF-IDF matching with a multi-layer recurrent neural network model trained to detect answers in Wikipedia paragraphs. Our experiments on multiple existing QA datasets indicate that (1) both modules are highly competitive with respect to existing counterparts and (2) multitask learning using distant supervision on their combination is an effective complete system on this challenging task.
1 Introduction
This paper considers the problem of answering factoid questions in an open-domain setting using Wikipedia as the unique knowledge source, as one does when looking for answers in an encyclopedia. Wikipedia is a constantly evolving source of detailed information that could facilitate intelligent machines — if they are able to leverage its power. Unlike knowledge bases (KBs) such as Freebase (Bollacker et al., 2008) or DBPedia (Auer et al., 2007), which are easier for computers to process but too sparsely populated for open-domain question answering (Miller et al., 2016), Wikipedia contains up-to-date knowledge that humans are interested in, but is designed for humans, not machines, to read.
Using Wikipedia articles as the knowledge source causes the task of question answering (QA) to combine the challenges of both large-scale open-domain QA and of machine comprehension of text. In order to answer any question, one must first retrieve the few relevant articles among more than 5 millions items, and scan them carefully to identify the answer. Our work treats Wikipedia as a collection of articles and does not rely on its internal graph structure. As a result, our approach is generic and could be switched to another collection of documents.
Large-scale QA systems like IBM’s DeepQA (Ferrucci et al., 2010) rely on multiple sources to answer: Wikipedia can be one of them but it is also paired with KBs, dictionaries, and even news articles, books, etc. As a result, such systems heavily rely on information redundancy among the sources to answer correctly. Having a single knowledge source forces the model to be very precise while searching for an answer as the evidence might appear only once. This challenge thus encourages research in the ability of a machine to read, a key motivation for the machine comprehension subfield and the creation of datasets such as SQuAD (Rajpurkar et al., 2016), CNN/Daily Mail (Hermann et al., 2015) and CBT (Hill et al., 2016).
However, those machine comprehension resources typically assume that a short piece of relevant text is already identified and given to the model, which is not realistic for building an opendomain QA system. In sharp contrast, methods that use KBs or information retrieval over documents have to employ search as an integral part of the solution. Instead we aim at the setting of simultaneously maintaining the challenge of machine comprehension, which requires the deep understanding of text, while keeping the realistic constraint of searching over a large open resource.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Q: How many of Warsaw's inhabitants spoke Polish in 1933? QA datasets: TREC, WebQuestions WikiMovies, SQuAD
Document Reader 833,500 Document Retriever
Figure 1: An overview of our question answering system DrWiki.
In this paper, we show how multiple existing QA datasets can be used to evaluate this framework by requiring an open-domain system to perform well on all of them at once. We develop DrWiki, a strong system for question answering from Wikipedia composed of: (1) Document Retriever, a module using bigram hashing and TFIDF matching designed to, given a question, efficiently return a subset of relevant articles and (2) Document Reader, a multi-layer recurrent neural network machine comprehension model trained to detect answer spans in those few returned documents. Figure 1 gives an illustration of DrWiki.
Our experiments show that Document Retriever outperforms the built-in Wikipedia search engine and that Document Reader achieves state-of-theart results on the very competitive SQuAD benchmark (Rajpurkar et al., 2016). Finally, our full system is evaluated using multiple benchmarks. In particular, we show that performance is improved across almost all datasets through the use of multitask learning and distant supervision compared to single task training.
2 Related Work
Open-domain QA was originally defined as finding answers in collections of unstructured documents, following the setting of the annual TREC competitions1. With the development of KBs, many recent innovations have occurred in the context of QA from KBs with the creation of re-
1http://trec.nist.gov/data/qamain.html
sources like WebQuestions (Berant et al., 2013) and SimpleQuestions (Bordes et al., 2015) based on the Freebase KB (Bollacker et al., 2008), or on automatically extracted KBs, e.g., OpenIE triples and NELL (Fader et al., 2014). However, KBs have inherent limitations (incompleteness, fixed schemas) that motivated researchers to return to the original setting of answering from raw text.
A second motivation to cast a fresh look at this problem is that of machine comprehension of text, i.e., answering questions after reading a short text or story. That subfield has made considerable progress recently thanks to new deep learning architectures like attention-based and memory-augmented neural networks (Bahdanau et al., 2015; Weston et al., 2015; Graves et al., 2014) and the release of new training and evaluation datasets like QuizBowl (Iyyer et al., 2014), CNN/Daily Mail based on news articles (Hermann et al., 2015), CBT based on children books (Hill et al., 2016), or SQuAD (Rajpurkar et al., 2016) and WikiReading (Hewlett et al., 2016), both based on Wikipedia. An objective of this paper is to test how such new methods can perform in an open-domain QA framework.
QA using Wikipedia as a resource has been explored previously. The authors of (Ryu et al., 2014) perform open-domain QA using a Wikipedia-based knowledge model. They combine article content with multiple other answer matching modules based on different types of semi-structured knowledge such as infoboxes, article structure, category structure, and defini-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
tions. Similarly (Ahn et al., 2004) also combine Wikipedia as a text resource with other resources, in this case with information retrieval over other documents. Buscaldi and Rosso (2006) also mine knowledge from Wikipedia for QA. Instead of using it as a resource for seeking the answers to questions, they focused on validation of the answers as returned by their QA system, and the use of Wikipedia categories for determining a set of patterns that should fit with the expected answer. In our work, we consider the comprehension of text only, and use the Wikipedia text documents as a sole resource for the reasons described in the introduction, that is to help focus on the machine comprehension task.
There are a number of highly developed full pipeline QA approaches using Wikipedia as a resource, including Microsoft’s AskMSR (Brill et al., 2002), IBM’s DeepQA (Ferrucci et al., 2010) and YodaQA (Baudiš, 2015; Baudiš and Šedivỳ, 2015), the latter of which is open source and hence reproducible for comparison purposes. AskMSR is a search-engine based QA system that relies on “data redundancy rather than sophisticated linguistic analyses of either questions or candidate answers”, i.e., it does not focus on machine comprehension, as we do. DeepQA is a very sophisticated system that relies on both unstructured information including text documents as well as structured data such as KBs, databases and ontologies to generate candidate answers or vote over evidence. YodaQA is an open source system modeled after DeepQA, similarly combining websites, information extraction, databases and Wikipedia in particular. While our comprehension task is far more challenging as we do not allow the leveraging of extra resources, comparing to these methods as an “upper bound” benchmark of what we could achieve is a useful datapoint.
Multitask learning (Caruana, 1998) and task transfer (e.g., in the computer vision community using ImageNet (Huh et al., 2016)) have a rich history in machine learning, as well as in NLP in particular (Collobert and Weston, 2008). Several works have attempted to combine multiple QA training datasets via multitask learning to (i) achieve improvement across the datasets via task transfer; and (ii) to provide a single general system capable of asking different kinds of questions due to the inevitably different data distributions across the source datasets. Fader et al.
(2014) used WebQuestions, TREC and WikiAnswers with four KBs as knowledge sources and reported improvement on the latter two datasets through multitasking. Bordes et al. (2015) combined WebQuestions and SimpleQuestions using distant supervision with Freebase as the KB to give slight improvements on both datasets, although poor performance was reported training on only one dataset and testing on the other, showing that task transfer is indeed a challenging subject; see also (Kadlec et al., 2016) for a similar conclusion. Our work follows similar themes, but in the setting of having to retrieve and then read text documents, rather than using a KB.
3 Our System: DrWiki
In the following we describe our system DrWiki for open-domain Wikipedia question answering which consists of two components: (1) the Document Retriever module for finding relevant articles and (2) a machine comprehension model, Document Reader, for extracting answers from a single document or a small collection of documents.
3.1 Document Retriever
Following classical QA systems, we use an efficient (non-machine learning) document retrieval system to first narrow our search space and focus on reading only articles that are likely to be relevant. A simple inverted index lookup followed by term vector model scoring performs quite well on this task for many question types, compared to the built-in ElasticSearch based Wikipedia Search API (Gormley and Tong, 2015). Articles and questions are compared as TF-IDF weighted bag-ofword vectors. We further improve our system by taking local word order into account with n-gram features. Our best performing system uses bigram counts while preserving speed and memory efficiency by using the hashing of (Weinberger et al., 2009) to map the bigrams to 224 bins with an unsigned murmur3 hash.
We use Document Retriever as the first part of our full model, by setting it to return 5 Wikipedia articles given any question. Those articles are then processed by Document Reader.
3.2 Document Reader
Our Document Reader model is inspired by the recent success of neural network models on machine comprehension tasks, in a similar spirit to the At-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
tentiveReader described in (Hermann et al., 2015; Chen et al., 2016).
Given a question q consisting of l tokens {q1, . . . , ql} and a document or a small set of documents of n paragraphs where paragraph p(k) consists of lk tokens {p (k) 1 , . . . , p (k) lk } for 1 ≤ k ≤ n, we develop an RNN model that we apply to each paragraph in turn and then finally aggregate the predicted answers. Our method works as follows:
Paragraph encoding We first represent each token pi in a paragraph p as a feature vector p̃i ∈ Rd and pass it as the input to a multi-layer recurrent neural network and thus obtain:
{p1, . . . ,pm} = RNN({p̃1, . . . , p̃m}),
where pi is expected to encode useful context information around token pi. Specifically, we choose to use a three-layer bidirectional long short-term memory network (LSTM) with h hidden units at each layer/direction as our RNN architecture and concatenate all the hidden units in the end. Therefore the resulting pi contains 6h dimensions.
The feature vector p̃i is comprised of the following parts:
• Word embeddings: femb(pi) = E(pi). We use the 300-dimensional Glove word embeddings trained from 840B Web crawl data (Pennington et al., 2014). We keep most of the pre-trained word embeddings fixed and only fine-tune the 1, 000 most frequent words because we think that some question words such as what, how, which, many could be highly important for QA systems.
• Exact match: fexact match(pi) = I(pi ∈ q). We use three simple binary features, indicating whether pi can be exactly matched to one question word in q, either in its original, lowercased or lemma form. These simple features turn out to be extremely helpful, as we will show in Section 5.
• Token features: ftoken(pi) = (POS(pi),NER(pi),TF(pi)). We also add a few manual features which reflect some properties of token pi in its context, which include its part-of-speech (POS) and named entity recognition (NER) tags and its (normalized) term frequency (TF).
• Aligned question embedding: following (Lee et al., 2016) and other recent works, the last part we incorporate is an aligned question embedding falign(pi) = ∑ j α(E(pi),E(qj))E(qj),
where αj ∝ E(pi) ᵀE(qj)∑
j′ E(pi) ᵀE(qj′ ) , which captures the similarity between pi and all the question words in q. Compared to the exact match features, these features add soft alignments between similar but non-identical words (e.g., car and vehicle).
Question encoding The question encoding is simpler, as we only apply another RNN on top of the word embeddings of qi and combine the resulting hidden units into one single vector: {q1, . . . ,ql} → q. We compute q = ∑ β(qi)qi where β(·) is a normalized weighting function which learns the importance of each question word.
Prediction At the paragraph level, the goal is to predict the span of tokens that is most likely the correct answer. We take the the paragraph vectors {p1, . . . ,pm} and the question vector q as input, and simply train two classifers independently for predicting the two ends of the span. Concretely, we use a bilinear term to capture the similarity between pi and q:
Pstart(i) = softmax (piWsq)
Pend(i) = softmax (piWeq)
During prediction, we choose the best span from token i to token i′ such that i ≤ i′ ≤ i + 15 and Pstart(i)×Pend(i′) is maximized. To make scores compatible across paragraphs in one or several retrieved documents, we replace the softmax with an unnormalized exponential. Our final prediction is then the argmax over all considered paragraph spans.
4 Data
Our work relies on three types of data: (1) Wikipedia that serves as our knowledge source for finding answers, (2) the SQuAD dataset which is our main resource to train Document Reader and (3) three more QA datasets (CuratedTREC, WebQuestions and WikiMovies) that in addition to SQuAD, are used to test the open-domain QA abilities of our full system, and to evaluate the ability of our model to learn from multitasking and
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Dataset Example Article / Paragraph SQuAD Q: How many provinces did the Ottoman
empire contain in the 17th century? A: 32
Article: Ottoman Empire Paragraph: ... At the beginning of the 17th century the empire contained 32 provinces and numerous vassal states. Some of these were later absorbed into the Ottoman Empire, while others were granted various types of autonomy during the course of centuries.
CuratedTREC Q: What U.S. state’s motto is “Live free or Die”? A: New Hampshire Article: Live Free or Die Paragraph: ”Live Free or Die” is the official motto of the U.S. state of New Hampshire, adopted by the state in 1945. It is possibly the best-known of all state mottos, partly because it conveys an assertive independence historically found in American political philosophy and partly because of its contrast to the milder sentiments found in other state mottos. WebQuestions Q: What part of the atom did Chadwick discover?†
A: neutron
Article: Atom Paragraph: ... The atomic mass of these isotopes varied by integer amounts, called the whole number rule. The explanation for these different isotopes awaited the discovery of the neutron, an uncharged particle with a mass similar to the proton, by the physicist James Chadwick in 1932. ...
WikiMovies Q: Who wrote the film Gigli? A: Martin Brest
Article: Gigli Paragraph: Gigli is a 2003 American romantic comedy film written and directed by Martin Brest and starring Ben Affleck, Jennifer Lopez, Justin Bartha, Al Pacino, Christopher Walken, and Lainie Kazan.
Table 1: Example training data from each QA dataset. In each case we show an associated article where distant supervision (DS) correctly identified the answer within it, which is highlighted (See Sec. 4.4).
Dataset Train Test Plain DS SQuAD 87,385 31,775 10,570† CuratedTREC 1,486∗ 3,464 694 WebQuestions 3,778∗ 4,602 2,032 WikiMovies 96,185∗ 36,301 9,952
Table 2: Number of questions for each dataset used in this paper. DS: distantly supervised training data. ∗: These training sets are not used as is because no paragraph is associated with each question. †: Corresponds to SQuAD development set.
distant supervision. Statistics of the datasets are given in Table 2.
4.1 Wikipedia (Knowledge Source)
We use the 2016-12-21 dump2 of English Wikipedia for all of our full-scale experiments as the knowledge source used to answer questions. For each page, only the plain text is extracted and all structured data sections such as lists and figures are stripped3. After discarding disambiguation pages, we retain 5,236,178 articles consisting
2https://dumps.wikimedia.org/enwiki/ latest
3https://github.com/attardi/ wikiextractor
of 9,264,930 unique uncased token types.
4.2 SQuAD
The Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) is a dataset for machine comprehension based on Wikipedia. The dataset contains 90k/10k train/development examples with a large hidden test set, that can only be accessed by the SQuAD creators. Each example is composed of a paragraph extracted from a Wikipedia article and an associated human-generated question. The answer is always a span from this paragraph and a model is given credit if its predicted answer matches it. Two evaluation metrics are used: Exact string match (EM) and F1 score, which measures the weighted average of precision and recall at the token level.
In the following, we use SQuAD for training and evaluating our Document Reader for the standard machine comprehension task given the relevant paragraph as defined in (Rajpurkar et al., 2016). For the task of evaluating open-domain question answering over Wikipedia, we use the SQuAD development set QA pairs only, and we ask systems to uncover the correct answer spans without having access to the associated paragraphs. That is, a model is required to answer a given question given the whole of Wikipedia as a resource, it is not given the relevant paragraph as
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Dataset Wiki Doc. Retriever Search plain +bigrams SQuAD 63.5 76.1 77.8 CuratedTREC 82.9 85.3 86.2 WebQuestions 74.1 75.8 74.6 WikiMovies 61.0 53.3 68.7
Table 3: Document retrieval results. % of questions for which the answer segment appears in one of the top 5 pages returned by the method.
in the standard SQuAD setting.
4.3 Open-domain QA Evaluation Resources
SQuAD is one of the largest general purpose QA datasets currently available. SQuAD questions have been collected via a process involving showing a paragraph to each human annotator and asking them to write a question. As a result, their distribution is quite specific. We hence propose to train and evaluate our system on other datasets developed for open-domain QA that have been constructed in different ways (not necessarily in the context of answering from Wikipedia).
CuratedTREC This dataset is based on the benchmarks from the TREC QA tasks that have been curated by Baudiš and Šedivỳ (2015). We use the large version, which contains a total of 2,180 questions extracted from the datasets from TREC 1999, 2000, 2001 and 2002.4
WebQuestions Introduced in (Berant et al., 2013), this dataset is built to answer questions from the Freebase KB. It was created by crawling questions through the Google Suggest API, and then obtaining answers using Amazon Mechanical Turk. We converted each answer to text by using entity names so that the dataset does not reference Freebase IDs and is purely made of plain text question-answer pairs.
WikiMovies This dataset, introduced in (Miller et al., 2016), contains 96k question-answer pairs in the domain of movies. Originally created from Freebase, the examples are built such that they can also be answered by using a subset of Wikipedia as the knowledge source (the title and the first section of articles from the movie domain).
4This dataset is available at https://github.com/ brmson/dataset-factoid-curated.
4.4 Distantly Supervised Data
All the QA datasets presented above contain training portions, but CuratedTREC, WebQuestions and WikiMovies only contain question-answer pairs, and not an associated document or paragraph as in SQuAD, and hence cannot be used for training Document Reader directly. Following previous work on distant supervision (DS) for relation extraction (Mintz et al., 2009), we use a procedure to automatically associate paragraphs to such training examples, and then add these examples to our training set.
We use the following process for each questionanswer pair to build our training set. First, we run Document Retriever on the question to retrieve the top 5 Wikipedia articles. All paragraphs from those articles without an exact match of the known answer are directly discarded. All paragraphs shorter than 25 or longer than 1500 characters are also filtered out. If any named entities are detected in the question, we remove any paragraph that does not contain them at all. For every remaining paragraph in each retrieved page, we score all positions that match an answer using unigram and bigram overlap between the question and a 20 token window, and we keep the paragraph with the highest overlap. If there is no paragraph with nonzero overlap, the example is discarded; otherwise we add it to our DS training dataset.
Examples of generated examples are given in Table 1 and the size of each DS dataset is given in the third column (DS) of Table 2.
Note that we can also generate additional DS data for SQuAD by trying to find mentions of the answers in other paragraphs other than the original paragraph provided in the dataset (from other pages or the same page that the given paragraph was in). As shown in Table 2, we can generate such additional examples for around a third of the original SQuAD training examples.
5 Experiments
This section first presents evaluations of our Document Retriever and Document Reader modules separately, and then describes tests of their combination, DrWiki, for open-domain QA on the full Wikipedia.
5.1 Finding Relevant Articles
We first examine the performance of our Document Retriever module on all the QA datasets. Ta-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Method Dev Test EM F1 EM F1 Dynamic Coattention Networks (Xiong et al., 2016) 65.4 75.6 66.2 75.9 BiDAF (Seo et al., 2016) 67.7 77.3 68.0 77.3 Multi-Perspective Matching (Wang et al., 2016)† 66.1 75.8 68.9 77.8 R-net‡ n/a n/a 71.3 79.7 DrWiki (Our model, Document Reader Only) 69.5 78.8 70.0 79.0
Table 4: Evaluation results on the SQuAD dataset (single model only). All the test results reflect the SQuAD leaderboard6 as of Feb 6, 2017. †: Test set results have been updated on the leaderboard after the paper submission. ‡: Paper unavailable.
ble 3 compares the performance of the two approaches described in Section 3.1 with that of the Wikipedia Search Engine7 for the task of finding articles that contain the answer given a question. Specifically, we compute the ratio of questions for which the text span of any of their associated answers appear in at least one the top 5 relevant pages returned by each system. Results on all datasets indicate that our simple approach outperforms Wikipedia Search, especially with bigram hashing.
5.2 Reader Evaluation on SQuAD
Next we evaluate our Document Reader component on the standard SQuAD evaluation (Rajpurkar et al., 2016).
Implementation details We use a 3-layer bidirectional LSTMs with h = 128 hidden units for both paragraph and question encoding. We apply the Stanford CoreNLP toolkit (Manning et al., 2014) for tokenization and also generating lemma, part-of-speech, and named entity manual features.
Lastly, all the training examples are sorted by the length of paragraph and divided into minibatches of 32 examples each. We use Adamax for optimization as described in (Kingma and Ba, 2014). Dropout with p = 0.3 is applied to word embeddings and all the hidden units of LSTMs.
Result and analysis Table 4 presents our evaluation results on both development and test sets. SQuAD has been a very competitive machine comprehension benchmark since its creation and we only list the best-performing systems in the table. Our system (single model) can achieve 70.0% exact match and 79.0% F1 scores on the test set, which surpasses all the published results and can
7We used the Wikipedia Search API https://www. mediawiki.org/wiki/API:Search.
Features F1 Full 78.4 No ftoken 77.9 (-0.5) No fexact match 77.2 (-1.2) No faligned 76.1 (-2.3) No faligned and fexact match 59.4 (-19.0)
Table 5: Feature ablation analysis of the paragraph representations of our Document Reader. Results are reported on the SQuAD development set, using a slightly inferior model compared to Table 4.
match the top performance on the SQuAD leaderboard at the time of writing. Additionally, we think that our model is conceptually simpler than most of the existing systems. We conducted an ablation analysis on the feature vector of paragraph tokens. As shown in Table 5 all the features contribute to the performance of our final system. Without the aligned question embedding feature (only word embedding and a few manual features), our system is still able to achieve F1 over 76%. More interestingly, if we remove both faligned and fexact match, the performance drops dramatically, so we conclude that both features play a similar but complementary role in the feature representation related to the paraphrased nature of a question vs. the context around an answer.
5.3 Full Wikipedia Question Answering
Finally, we assess the performance of our full system DrWiki for answering open-domain questions using the four datasets introduced in Section 4. We compare three versions of DrWiki which evaluate the impact of using distant supervision and multitask learning across the training sources provided to Document Reader (Document Retriever remains the same for each case):
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Dataset YodaQA DrWiki SQuAD +Fine-tune (DS) +Multitask (DS) SQuAD (All Wikipedia) n/a 26.7 29.2 29.6 CuratedTREC 31.3 19.7 25.5 24.5 WebQuestions 39.8 19.6 19.3 18.8 WikiMovies n/a 23.6 33.1 34.4
Table 6: Full Wikipedia results. Top-1 accuracy (in %). +Fine-tune (DS): Document Reader models trained on SQuAD and fine-tuned on each DS training set independently. +Multitask (DS): Document Reader single model trained on SQuAD and all the distant supervision (DS) training sets jointly. YodaQA results are extracted from https://github.com/brmson/yodaqa/wiki/Benchmarks and use additional resources other than Wikipedia such as Freebase and DBpedia, see Section 2.
• SQuAD setting: A single Document Reader model is trained on the SQuAD training set only and used on all evaluation sets.
• Fine-tune (DS): A Document Reader model is pre-trained on SQuAD and then fine-tuned for each dataset independently using its dedicated distant supervision (DS) training set.
• Multitask (DS) setting: A single Document Reader model is jointly trained on the SQuAD training set and all the DS sources.
Results Table 6 presents the results. Despite the difficulty of the task compared to machine comprehension (where you are given the right paragraph) and unconstrained QA (using redundant resources), DrWiki still provides reasonable performance across all four datasets.
We are interested in a single full system that can answer any question using Wikipedia. The single model trained only on SQuAD is outperformed on three of the datasets by the multitask system that uses distant supervision. However performance when training on SQuAD alone is not far behind indicating that task transfer is occurring. The majority of the improvement from SQuAD to Multiclass(DS) however is likely not from task transfer as fine-tuning on each dataset alone using DS also gives improvements, showing that it is the introduction of extra data in the same domain that helps. Nevertheless, the best single model that we can find is our overall goal, and that is the Multitask(DS) system.
We compare to an unconstrained QA system using redundant resources (not just Wikipedia), YodaQA (Baudiš, 2015), giving results which were previously reported on CuratedTREC and WebQuestions. Despite the increased difficulty of our task it is reassuring that our performance is not
too far behind on CuratedTREC (31.3 vs. 25.5). The gap is slightly bigger on WebQuestions likely because that is based on Freebase which YodaQA uses directly.
DrWiki’s performance on SQuAD compared to its Document Reader component on machine comprehension in Table 4 shows a large drop (from 69.5 to 26.7) as we now are given Wikipedia to read, not a single paragraph. Given the correct document (but not the paragraph) we can achieve 49.6 indicating many false positives come from highly topical sentences. This is despite the fact that the Document Retriever works relatively well (77.8% of the time retrieving the answer, see Table 3). Overall, our results indicate that we have identified a key challenging task for researchers to focus on.
6 Conclusion
We studied the task of open-domain QA using Wikipedia as the unique knowledge source. This brings unique challenges for machine comprehension systems where integrating search, distant supervision and multitask learning provides an effective complete system, whereas machine comprehension systems alone cannot solve the overall task. Evaluating the individual components and the overall system across multiple benchmarks showed the efficacy of our approach.
Future work should aim to improve over our DrWiki system. Two obvious angles of attack are: (i) incorporate the fact that Document Reader aggregates over multiple paragraphs and documents directly in the training, as it currently trains on paragraphs independently; and (ii) perform endto-end training across the Document Retriever and Document Reader pipeline, rather than independent systems.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: *- Task *- Simple model, yet the best results on SQuAD (single model0 *- Evaluation and comparison  - Weaknesses: *- Analysis of errors results (See detailed comments below)  - General Discussion: In this paper the authors present a method for directly querying Wikipedia to answer open domain questions. The system consist of two components - a module to query fetch wikipedia articles and a module to answer the question given the fetched set of wikipedia articles.   The document retrieval system is a traditional IR system relying on term frequency models and ngram counts.  The answering system uses a feature representation for paragraphs that consists of word embeddings, indicator features to determine whether a paragraph word occurs in a question, token-level features including POS, NER etc and a soft feature for capturing similarity between question and paragraph tokens in embedding space. A combined feature representation is used as an input to a bi-direction LSTM RNN for encoding. For questions an RNN that works on the word embeddings is used.  These are then used to train an overall classifier independently for start and end spans of sentences within a paragraph to answer questions.  The system has been trained using different Open Domain QA datasets such as SQuAD and WebQuestions by modifying the training data to include articles fetched by the IR engine instead of just the actual correct document passage.  Overall, an easy to follow interesting paper but I had a few questions: 1) The IR system has a Accuracy@5 of over 75 %, and individually the document reader performs well and can beat the best single models on SquAD. What explains the significant drop in Table 6. The authors mention that instead of the fetched results, if they test using the best paragraph the accuracy reaches just 0.49 (from 0.26) but that is still significantly below the 0.78-79 in the SQuAD task.  So, presumably the error is this large because the neural network for matching isnt doing as good a job in learning the answers when using the modified training set (which includes fetched articles) instead of the case when training and testing is done for the document understanding task. Some analysis of whats going on here should be provided. What was the training accuracy in the both cases? What can be done to improve it? To be fair, the authors to allude to this in the conclusion but I think it still needs to be part of the paper to provide some meaningful insights.  2) I understand the authors were interested in treating this as a pure machine comprehension task and therefore did not want to rely on external sources such as Freebase which could have helped with entity typing        but that would have been interesting to use. Tying back to my first question -- if the error is due to highly relevant topical sentences as the authors mention, could entity typing have helped?  The authors should also refer to QuASE (Sun et. al 2015 at WWW2015) and similar systems in their related work. QuASE is also an Open domain QA system that answers using fetched passages - but it relies on the web instead of just Wikipedia.


A New Approach for Measuring Sentiment Orientation based on Multi-Dimensional Vector Space
This study implements a vector space model approach to measure the sentiment orientations of words. Two representative vectors for positive/negative polarity are constructed using high-dimensional vector space in both an unsupervised and a semisupervised manner. A sentiment orientation value per word is determined by taking the difference between the cosine distances against the two reference vectors. These two conditions (unsupervised and semi-supervised) are compared against an existing unsupervised method (Turney, 2002). As a result of our experiment, we demonstrate that this novel approach significantly outperforms the previous unsupervised approach and is more practical and data efficient as well.
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
This study implements a vector space model approach to measure the sentiment orientations of words. Two representative vectors for positive/negative polarity are constructed using high-dimensional vector space in both an unsupervised and a semisupervised manner. A sentiment orientation value per word is determined by taking the difference between the cosine distances against the two reference vectors. These two conditions (unsupervised and semi-supervised) are compared against an existing unsupervised method (Turney, 2002). As a result of our experiment, we demonstrate that this novel approach significantly outperforms the previous unsupervised approach and is more practical and data efficient as well.
1 Introduction
Previous research in sentiment analysis or opinion mining mostly focus on supervised methods, which requires labeled training data to identify properties of unseen input, and then classify input later. Probabilistic methods in particular often calculate which class a word or phrase most likely bears and then make predictions regarding the label of a given target text, using those estimations. While such methods have widely been adopted, there are few examples which measure the likelihood of lexical items in an unsupervised or semisupervised manner. However, there still exist situations where sentiment analysis should be performed using non-labeled datasets. In such cases, discovering information regarding the sentiment orientation of vocabulary in a non-supervised fashion becomes essential.
Our approach employs VSM (Vector Space Models) as its main component. VSM is deeply related to the distributional hypothesis (Turney and Pantel, 2010). The distributional hypothesis states that words in similar contexts tend to have similar meaning (Rubenstein and Goodenough, 1965; Schütze and Pederson, 1995; Deerwester et al., 1990). Traditionally, the relation of two words in a ‘similar context’ has been distinguished into two classes: syntagmatic or paradigmatic (Murphy, 2003; Sahlgren, 2006). Syntagmatic relations are concerned with whether or not two entities are in a co-occurrence relation, and paradigmatic relations are concerned with whether the two items in question are interchangeable (substitution relation). Many collocation models using N-grams and Point-wise Mutual Information (PMI) analyze the former-type of word relations. On the other hand, recent dense vector-based models (Skip-gram, Continuous Bag-of-Words) exploit the paradigmatic relation, and thus they both give a high weight to the similarity of words if they share similar neighboring entities (Mikolov et al., 2013).
Our work can best be understood as an exploration to find a sentiment dimension over a multidimensional vector space, which is constructed from the relations of words in a corpus. However, it is too complex to extract a specific type of relation between whole words on such a high-dimensional space. Thus, we start by selecting a small set of words that are believed to have an emotional value for a topic. We refer to these words as ‘point words’ and use them to construct a latent sentiment dimension. Our method for choosing these point words can be divided into two sub-types: unsupervised or semi-supervised.
If we use a supervised learning approach, one simple, intuitive way of calculating the sentiment
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
orientation of a word is to compute the log-likelihood ratio of probabilities in terms of occurrence per sentiment label. Should the labels not be given, one alternative method would be to use the similarity between words. Following the principle of the Distributional Hypothesis, we assume that positive or negative words will tend to share similar contexts relative to their opposing stance.
We argue that collocation-based methods are not a practical choice for obtaining the similarity due to data sparsity which is inherent to the model. Our implementation of PMI-IR method (Turney, 2002) for this problem demonstrates this point and hints as to why dense vector space models should be used instead. For this purpose, we also compared two well-known vector models (Word2Vec and GloVe), as the latter model mainly depends on the collocation relations of words, while Word2Vec’s Skip-gram model is dependent on the paradigmatic relations between words.
In the unsupervised condition, a dimensionality reduction algorithm is implemented to search for sentiment dimension using the selected point words. Under the semi-supervised condition, we depend on external estimations of the terms in order to skip the exploration stage. We believe that this study can be a direct comparison with the previous work of (Turney, 2002), because we use the same seed information and a similar procedure for calculating the semantic orientations of the words.
2 Related Work
Although the majority of previous studies on sentiment analysis have preferred to use supervised methods, some researchers have tried to develop unsupervised or semi-unsupervised approaches. For instance, Turney (2002) suggests the PMI-IR algorithm to estimate the semantic orientation of a phrase for the unsupervised classification of various reviews. He uses two pre-chosen words (‘poor’ and ‘excellent’) to calculate the semantic orientation of the target phrases, which is defined as the relative PMI difference of the phrase from the two seed words. His work borrows heavily on the theory of semantic orientation of adjectives by Hatzivassiloglou and McKeown (1997). In this study, the authors discuss the existence of linguistic constraints on the semantic orientations of words in conjunctions. In line with the approach of Turney 1 Yandex reported that it indexes more than 4 billion pages written in the Latin Alphabet with the majority of them being in English (https://yandex.com/company
(2002), Zagibalov and Carroll (2008) attempt to develop an automatic selection process for seed words in Chinese texts for unsupervised classifications.
One major constraint on Turney (2002) is the availability of a corpus to calculate the relevant PMI. If a given equipped corpus is not big enough for a PMI analysis, the problem of data sparseness will arise, and the PMI values become suspect. Note that Turney (2002) used a search engine (AltaVista) for his experiments, which contained 350 million web pages at the time. In our experiment, we instead use Yadex.com (http://www.yan-
dex.com/), as it provides a more reliable Near operator among the current major search engines.1 With the operator, words in a query have to be within 10 words of each other, regardless of order. We used the same formula (Eq. 1) from Turney (2002) for our experiment:
𝑆𝑂(𝑝𝑟ℎ𝑎𝑠𝑒) =
𝑙𝑜𝑔2 [ ℎ𝑖𝑡𝑠(𝑝ℎ𝑟𝑎𝑠𝑒,𝑁𝐸𝐴𝑅 "excellent") ℎ𝑖𝑡𝑠("poor")
ℎ𝑖𝑡𝑠(𝑝ℎ𝑟𝑎𝑠𝑒,𝑁𝐸𝐴𝑅 "poor") ℎ𝑖𝑡𝑠("excellent") ] (1)
Note that hits(query) is the number of the returns,
given the query. Additionally, we add 0.01 to hits when the number of the hits is zero, in order to prevent division-by-zero.
In contrast to collocation models, some researchers attempt to apply neural probabilistic language models to measure the semantic similarities of words based on context-window methods (Mikolov et al., 2013; Collobert and Weston, 2008), and word embedding methods (e.g., Word2Vec) have been found more effective for various tasks in NLP than other traditional techniques (Baroni et al., 2014). Continuous Bag-of-Words (CBOW) models and Skip-gram models used in Mikolov et al., (2013) both place a high weight on the similarity of words if they share similar neighboring entities.
Additionally, we consider another word-embedding model (GloVe). Unlike Skip-gram or the CBOW architecture of Word2Vec, GloVe uses the ratios of words’ probability of co-occurrence to learn word vectors (Pennington et al., 2014). We note that GloVe is a kind of word embedding model, but its vector space is constructed differently from Word2Vec, because it adopts the collocation modeling of words.
/press_center/press_releases/2010/201005-19)
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
We note that the implication of the Skipgram/CBOW architecture is very similar to the concept of paradigmatic relation of words. This is due to the fact that the distance between any two words in a paradigmatic relation is minimized when they share the most similar neighbors. For example, in a minuscule corpus which bears only two sentences (“This movie is very good” and “This movie is very bad”), the two words (‘good’ and ‘bad’) will likely have a high cosine similarity in the Word2Vec model.
This aspect can cause unexpected results when such a model is employed for clustering a set of items that share similar emotions, because two words in paradigmatic relations often instantiate a contrastive relation (e.g., antonym). However, as noted in Mikolov et al. (2013), words seem to have multiple syntactic/semantic relations to each other, and the Word2Vec model helps to observe the multiple degrees of similarity for words. From this perspective, we might find a specific relation to them in a subspace of the original vector space if the necessary vector calculation operations are known.
A feature-space approach (Osgood, 1952; Smith and Medin, 1981; Waltz and Pollack, 1985) discusses how words are represented by feature vectors whose attributional factors are annotated by human participants. In a similar vein, the Conceptual Space theory of Gärdenfors (2000) provides a theoretical framework for understanding the multiple degrees of similarity in the space. He also suggests using dimensionality reduction algorithms (e.g., Multidimensional scaling) to explore the quality dimensions of multi-dimensional vector space and claims that applying these algorithms to the similarity-based vector space will generate an ordering relation for data points on an interested domain.
Another relevant prior work is the Word-Space model, which is a vector-based computational model for the semantic similarity of words (Schütze, 1993; Sahlgren, 2006). The Word-Space model, as the name implies, models word meaning with a spatial representation. Thus, semantic similarity is represented as proximity in n-dimensional space.
Our goal is to find or construct a sentiment dimension from the similarity-based vector space of words. The vector representation for the sentiment dimension will be our ‘interested domain’ and the ‘ordering relation’ on the domain will map each
word object to a real valued point, indicating the level of its ‘positive’ or ‘negative’ significance.
3 Methods and Data
The purpose of our experiment is to find the sentiment orientations of words in a corpus and evaluate the effectiveness of the information by conducting an unsupervised/semi-supervised classification on our movie review datasets.
3.1 Data
Our data consists of two movie review corpora: one of which is the IMDB movie dataset (Maas et al., 2011) and the other is the Stanford Sentiment Treebank (Socher et al., 2013). The IMDB dataset provides 25,000 movie reviews for training and 25,000 for testing. The corpus also contains the expected polarity values for all individual tokens occurring in the reviews. This dataset is used for our vector space construction and the polarity values per word are employed under the semi-supervised condition.
The Stanford Sentiment Treebank is a corpus based on the 11,855 movie reviews presented by Pang and Lee (2005), and all 215,154 phrases are manually annotated by three judges per phrase. Because we want the classification task to be binary, reviews with neutral labels are excluded from the dataset, and the number of reviews in the resulting dataset is 9,613. We select this corpus as our test dataset since it allows us to compare our sentiment orientation values with the annotated polarity value for each word.
3.2 Experiment Methods
The first step of our methodology is to obtain a set of point words which are assumed to express positive/negative sentiment. Using the pattern extrac-
First Word Second Word Third Word
1 JJ NN or NNS anything
2 RB, RBR or
RBS
JJ not NN nor
NNS
3 JJ JJ not NN nor
NNS
4 NN or NNS VB, VBD not NN nor
NNS
5 RB, RBR, or
RBS
VBN, or VBG anything
Table 1. Pattern rules of tags for extraction twoword phrases (third word excluded)
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
tion rules (Table. 1) from Turney (2002), we obtained 17,716 phrases by applying these rules to the Stanford Sentiment Treebank.
We use the adjectives or adverbs of the phrases for our point words, since these syntactic categories have been found very useful for recognizing subjectivity in written texts (Hatzivassiloglou and Wiebe, 2000; Wiebe et al., 1999; Bruce and Wiebe, 2000).
To validate this assumption, we POS-tagged the word tokens of the Stanford corpus and observe the variance of sentiment ratings of words depending on their POS-tag (Fig. 1).
Figure 1. Variances of polarity values per POS-tags
Not surprisingly, the main tags of adjective and adverb occupy the top places in the ranking, indicating that terms bearing these tags are often used with stronger subjectivity. Numerically, the top four tags (JJS, RBR, JJ, and RB) make up 37% of total variances over the whole dataset.
Based on the theory and the observation, we select the top K modifiers (adjectives or adverbs) from the extracted patterns. The number of the modifiers are automatically determined by choosing a minimum frequency for the extracted phrases.
3.2.1 Unsupervised methodology
After the point words (K modifiers) are obtained, a high-dimensional vector space is implemented to build a point-wise distance matrix for the selected words, resulting in a K×K matrix. Since the distances between the tokens are measured by Cosinedistance, the distance data is Euclidean. As mentioned in Section 3.1, we use the IMDB dataset for vector space construction.
A local structure-oriented dimensionality reduction algorithm, the Principal Component Analysis (PCA) is then used to find the sentiment dimension between the modifiers. In the analysis, we only use
the value of two for the number of dimensions to project the entities into the reduced space.
Based on the principle of the Distribution Hypothesis, we assume that semantically close modifiers are closer to each other than the opposites. Thus, the principal component of the PCA for the emotional terms will represent the positive/negative aspects of their collective meaning.
When the dimensionality reduction phase is completed, it is possible to observe correlations between values on the found dimension and the goldstandard dataset (the annotated values of Stanford Sentiment Treebank). Since the signs of the coefficients are irrelevant for our purposes, only absolutes are considered.
Now, we can determine the two sets of words distinguished by the origin of zero on the principal axis. To construct two reference vectors, simple vector averaging is used, and the vectors are classified as positive or negative by the criterion of being closer to the vector of the seed word (“excellent”). Note that this method was inspired by Turney (2002) and makes our study comparable to the previous work. We define the sentiment orientation of a word using Equation (2):
SO(w) = 𝐶𝑜𝑠𝐷𝑠𝑡(𝑉𝑒𝑐𝑝𝑜𝑠, 𝑤) − 𝐶𝑜𝑠𝐷𝑠𝑡(𝑉𝑒𝑐𝑛𝑒𝑔, 𝑤) (2)
CosDst means Cosine distance in the equation. If the mean of the sentiment orientations in a review post is less than zero, the review is labeled as ‘negative’, and is ‘positive’ otherwise. We note that this approach allows us to calculate the orientations of all words in the vocabulary, unlike Turney’s phrase-oriented approach.
3.2.2 Semi-supervised methodology
This method does not use the dimensionality reduction algorithm to distinguish the point words into two sets, but instead employs the expected star ratings of the tokens from the IMDB dataset. The ratings represent how strongly a word belongs to positive or negative sentiment polarity.
Thus, in this case, we start with almost certain information on the polarity of the words. Because adjectives and adverbs are generally used with their own static stance, we believe that the information can be applied to unseen texts over different domains. Since this semi-supervised setting is designed to compare with the unsupervised condition, the remainder of the experiment methodology is identical to the unsupervised methodology.
0
0.005
0.01
0.015
0.02
0.025
0.03
JJS RBR JJ RB VBG VBN NN JJR VBD
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
3.2.3 PMI-IR methodology
We replicate the PMI-IR algorithm (Turney, 2002) against the Stanford corpus. The queries on the 17 thousand phrases are sent to the Search API of Yandex.com (https://yandex.com/search/) and the hits of the phrases with the two seeds (“excellent” and “poor”) are disregarded if both of the two hits are less than four counts. Since the method is only applicable to a review that contains at least the one of the defined pat terns (Table. 1), the test dataset consists of 7,646 posts out of the 9,613 reviews and the average number of phrases per review is 2.25.
4 Results
Our first observation in the unsupervised setting is the visualization of the PCA result on the selected
modifiers using Word2Vec (Skip-gram, no_components: 100) and GloVe models (epochs: 30, no_components: 100). Fig. 1 shows exemplary results of the 82 tokens, which are from the extracted patterns of the Stanford dataset. The point words are colored red (positive) or blue (negative) by the middle value on the annotation scale.
The principal dimension in the embedding by PCA gives us an ordering relation between the terms. As explained in Section 3, we interpret the dimension as a sentiment domain in the high-dimensional vector space of words. The correlation coefficients in Table 2 demonstrates how the dimension correlates with our gold-standard dataset (the signs of the correlations are ignored). Although there undoubtedly exists noise in the results, we could still find correlations between the unsupervised ratings and the annotated values. We also note that varying the size of context window for the embedding models produces slightly different patterns. As Table 2 shows, the Word2Vec model tends to lose its potency as the size of context window increases, while GloVe remains the same.
Our semi-supervised approach does not use the procedure incorporating the PCA, but the vector averaging and classification processes are identical to the unsupervised condition.
Fig. 2 represents the accuracy results of the two settings for the 9,613 reviews, while changing the minimum frequency of the extracted patterns in which the modifiers occurred. The cutoff frequency ranges from 2 to 9 and the corresponding number of the word tokens varies from 495 to 30. We set the context window size to 3 for the
Figure 2. 2-D embedding of the Principal Component Analysis of the point words. The left figure shows the result from Word2Vec-based space and the right figure from GloVe model. The color is coded by the ratings of Stanford Sentiment Treebank (red for positive and blue for negative).
Model Pearson Spear-
man
Word2Vec (size:3) 0.33 0.32 Word2Vec (size:5) 0.29 0.29 Word2Vec (size:7) 0.27 0.26 Word2Vec(size:10) 0.25 0.24
GloVe (size:3) 0.28 0.33 GloVe (size:5) 0.27 0.30 GloVe (size: 7) 0.28 0.34
GloVe (size: 10) 0.28 0.34
Table 2. Correlations between the unsupervised ratings and annotated ratings
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Word2Vec (Skip-gram based) and 10 for the GloVe model.
As can be seen in the graph, the Word2Vec conditions often outperform the GloVe conditions, and the unsupervised Word2Vec generally records higher accuracy than the GloVe. The highest accuracy is achieved by semi-supervised Word2Vec (66%) and the lowest score is falls to the unsupervised GloVe model (lower than chance). The best result for unsupervised Word2Vec is 61% (cutoff: 8), and the variance of the accuracies is relatively small compared to other models. Decreasing the number of the point words worsens the performance of all the semi-supervised models.
The general pattern of Fig. 2 is found in the results of our experiment combined with the PMI-IR algorithm against the reduced test dataset (Table 3). Semi-supervised/unsupervised Word2Vec models generally record higher accuracies than the GloVe models. On top of this, the semi-supervised methods show a better performance than the unsupervised settings.
The PMI-IR produces the lowest accuracy along with the unsupervised GloVe model (57%). As mentioned in Section 3, the PMI-based approach is based on the returned search results from Yandex engine, while our methods use vector space from the IMDB corpus of 50,000 movie reviews. The classification by PMI-IR is processed by calculating the mean of semantic orientations from the extracted phrases per review as suggested in Turney (2002).
5 Discussion
We present an unsupervised/semi-supervised approach that measures the sentiment orientation of words using vector space models. The core idea of our approach is to find a sentiment dimension from a high-dimensional vector space of words and use the extracted information to calculate the polarity of an individual word. We attempted to see whether or not the obtained sentiment orientations are useful by the sentiment classification task of movie reviews.
Our approach borrows from the general paradigm of Word Space models (Schütze, 1993) and is inspired by the theoretical tools of feature space models (Osgood, 1952; Smith and Medin, 1981; Waltz and Pollack, 1985; Gärdenfors, 2000). The general framework of our experiment is based on Turney (2002) and our computational methods rely on recent successes in word-embedding research (Mikolov et al., 2013; Pennington et al., 2014).
Our unsupervised methodology (Word2Vecbased) outperforms the PMI-IR approach, which is
Figure 2. Accuracy of Classifications. The x-axis indicates the cutoff frequency.
0.4
0.45
0.5
0.55
0.6
0.65
0.7
2 3 4 5 6 7 8 9
W2V-unsuper
W2V-semi
Glove-unsuper
Glove-semi
Model Cutoff Accuracy
W2V (semi-supervised) 8 0.66
W2V (unsupervised) 5 0.63
GloVe (semi-supervised) 3 0.63
GloVe (unsupervised) 9 0.57
PMI-IR N/A 0.57
Table 3. highest accuracy of classification results of the selected test set (7,646 reviews)
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
a well-known unsupervised tool in a sentiment classification task. Additionally, the unsupervised Word2Vec setting records much higher accuracy than the unsupervised GloVe model. We suspect that the cause of this low performance is from the common feature that the both models are based on word collocations.
Considered that the PMI-IR approach employs the results of a major search engine (Yandex.com), the low accuracy indicates that data-sparsity is a very difficult issue for a collocation-based method to overcome. Even the word-embedding model (GloVe) seems unable to break free of this problem, as the models show lower results than the those using Skip-gram. This conjecture is supported by the observation that the GloVe model does not lose its correlation coefficients as the context size increases. We note that Skip-grams essentially exploit paradigmatic relations between words and produce denser vector spaces for the relation of words. Thus, methods based on dense vector modeling should be more robust for the data-sparsity problem in our task, as demonstrated by our experiment.
It is worth noting that PMI-IR suffers from datadeficiency. In many practical situations, it is hard to collect information on the collocations of phrases using a sufficient search engine service, as major search engines often constrain the queries of users by their policies or do not provide the ‘Near’ operator for queries.
One issue in our study is how to find optimal reference vectors to represent the sentiment polarity of a vector space. We tried to approximate by using a traditional operation (vector averaging). Note that the performance of the semi-supervised approach did not dramatically increase with the added number of the word tokens. This likely indicates that the type of vector calculation is not efficient for the purpose in this research. Additionally, the results of our experiment do not meet the high standards of a supervised approach (usually above 80%). However, we believe that there is potential for huge improvement, as the reference vectors can be constructed in an optimal way to represent the sentiment domain between words. We predict that there could be two possible ways to achieve this goal. The first way is to select the point words that best capture the landscape of the sentiment entities in a corpus. And the second way is to exploit more relevant vector space models than the embedding
methods used in this study. We leave such explorations for future work.
6 Conclusion
In this study, we introduce a novel approach which implements distributional semantic models to measure the sentiment orientation of a word. We divide our experiment into two separate types (unsupervised or semi-supervised) and compare the results with a previous unsupervised approach (PMI-IR). Our new unsupervised methodology (Word2Vec-based) outperforms the existing approach, using much smaller datasets, while being robust enough to overcome the problem of datasparseness.

# Summary  This paper presents an empirical study to identify a latent dimension of sentiment in word embeddings.  # Strengths   S1) Tackles a challenging problem of unsupervised sentiment analysis.   S2) Figure 2, in particular, is a nice visualisation.  # Weaknesses   W1) The experiments, in particular, are very thin. I would recommend also measuring F1 performance and expanding the number of techniques compared.   W2) The methodology description needs more organisation and elaboration. The ideas tested are itemised, but insufficiently justified.    W3) The results are quite weak in terms of the reported accuracy and depth of analysis. Perhaps this work needs more development, particularly with validating the central assumption that the Distributional Hypothesis implies that opposite words, although semantically similar, are separated well in the vector space?