Investigating Different Context Types and Representations for Learning Word Embeddings
The number of word embedding models is growing every year. Most of them learn word embeddings based on the cooccurrence information of words and their contexts. However, it’s still an open question what is the best definition of context. We provide the first systematical investigation of different context types and context representations for learning word embeddings. Comprehensive experiments are conducted to evaluate their effectiveness under 6 tasks, which give us some insights about context selection. We hope that this paper, along with the published code, can serve as a guideline of choosing context for our community.
1 Introduction
Recently, there is a growing research interest on word embedding models, where words are embedded into low-dimensional real vectors. Words that share similar meanings tend to have short distances in the vector space. The trained word embeddings are not only useful by themselves (e.g. used for calculating word similarities) but also effective when used as the input of the downstream models, such as part-of-speech tagging, chunking, named entity recognition (Collobert and Weston, 2008; Collobert et al., 2011) and text classification (Socher et al., 2013; Kim, 2014).
For almost all word embedding models, the training objectives are based on the Distributed Hypothesis (Harris, 1954), which can be stated as: “words that occur in the same contexts tend to have similar meanings”. The “context” is usually defined as the words which precede and follow the target word within some fixed distance in most word embedding models with various architec-
tures (Bengio et al., 2003; Mnih and Hinton, 2007; Mikolov et al., 2013b; Pennington et al., 2014). Among them, Global Vectors (GloVe) proposed by Pennington et al. (2014), Continuous Skip-Gram (CSG) 1 and Continuous Bag-Of-Words (CBOW) proposed by Mikolov et al. (2013a) achieve stateof-the-art results on a wide range of linguistic tasks, and scales well to corpus with billion words.
Recently, Levy and Goldberg (2014b); Ling et al. (2015) 2 improve CSG and CBOW by introducing position-aware context representation, where each contextual word is associated with their relative position to the target word. Levy and Goldberg (2014a) propose DEPS, which takes the words that are connected to target word in dependency parse tree as context.
Despite all these efforts, there is still no clear answer to the following questions due to the lack of systematical comparison: 1) Is dependencybased context more reasonable than traditional linear one? 2) Do the relative position or the dependency relation between contextual word and target word contributes to the learning process? 3) Do different word embedding models have preferences for different contexts? 4) How different contexts affect models’ performances on different tasks?
To answer these questions, we first classify word embedding models based on different context types (linear or dependency-based) and different context representations (word or bound word) in Table 1. We implement the models that previously not proposed and give systematical comparisons on a wide range of word similarity, word analogy, part-of-speech tagging, chunking, named entity recognition, and text classification dataset-
1Many researches refer Continuous Skip-Gram as SG. However, in order to distinguish linear (continuous) context and dependency-based context, we refer it as CSG.
2In these two papers, the description of position-aware context are quite different. However, their ideas is actually identical.
2
100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Basic Model Context Representation
Context Type Linear Dependency-based
generalized (unbound) word CSG (Mikolov et al., 2013a) this work Skip-Gram bound word Structured SG (Ling et al., 2015)POSIT (Levy and Goldberg, 2014b) Deps
(Levy and Goldberg, 2014a) generalized (unbound) word CBOW (Mikolov et al., 2013a) this work
Bag-Of-Words bound word CWINDOW (Ling et al., 2015) this work generalized (unbound) word GloVe (Pennington et al., 2014) this work
GloVe bound word this work this work
Table 1: Research summarization of Generalized Skip-Gram, Bag-Of-Words and GloVe with different context types and context representations. For linear context, bound word indicates word associated with positional information. For dependency-based context, bound word indicates word associated with dependency relation.
s. Experimental results suggest that although it’s hard to find any universal insight (i.e. one context works consistently better than the other), the characteristics of different contexts on different models are concluded according to specific tasks. We expect this paper to be a useful complement in the word embedding literature.
2 Methodology
In this section, we first introduce different contexts in detail and discuss their strength and weakness. We then show how CSG, CBOW and GloVe can be generalized to use these contexts.
2.1 Context Types
It is necessary to discover more effective ways of defining “context”. In the current literature, there are mainly two types of contexts: linear (most word embedding models) and dependency-based (DEPS (Levy and Goldberg, 2014a)). Linear context is defined as the positional neighbours of the target word in texts. Dependency-based context is defined as the syntactic neighbours of the target word based on dependency parse tree, as shown in Figure 1 .
Compared to linear context, dependency-based context is more focused and can capture more long-range contexts. For example in Figure 1, linear context does not consider the word-context pair “discovers telescope”, while dependency-based context contains this information. Dependency-based context can also exclude some uninformative word-context pairs like “with star” and “telescope with”.
in the text. The context vocabulary C is thus identical to the word vocabulary W . However, this restriction is not r quired by the model; contexts need not corr spond to words, and the number of context-types can be substantially larger than the number of word-types. W generalize SKIPGRAM by replacing the bag-of-words contexts with arbitrary contexts.
In this paper we experiment with dependencybased syntactic contexts. Syntactic contexts capture different information than bag-of-word contexts, as we demon trate u ing the sentence “Au - tralian scientist discovers star with telesc p ”.
Linear Bag-of-Words Contexts This is the context used by word2vec and many other neural embeddings. Using a window of size k around the targ t word w, 2k contexts are produced: the k words before and the k words after w. For k = 2, the contexts f the target word w are −2, w−1, w+1, w+2. In our example, the contexts of discovers are Australian, scientist, star, with.2
Note that a context window of size 2 may miss some important contexts (telescope is n t a cont xt of discovers), wh le including s me accidental ones (Australian is a ontext discover ). Moreover, the contexts are unmarked, resulting in discovers being a context of both stars and scientist, which may result in stars and scientists ending up as neighbours in the embedded space. A window size of 5 is commonly used to capture broad topical content, whereas smaller windows contain more focused information about the target word.
Dependency-Based Contexts An alternative to the bag-of-words approach is to derive contexts based on the syntactic relations the word participates in. This is facilitated by recent advances in parsing technology (Goldberg and Nivre, 2012; Goldberg and Nivre, 2013) that allow parsing to syntactic dependencies with very high speed and near state-of-the-art accuracy.
After parsing each sentence, we derive word contexts as follows: for a target word w with modifiers m1, . . . ,mk and a head h, we consider the contexts (m1, lbl1), . . . , (mk, lblk), (h, lbl−1h ),
2word2vec’s implementation is slightly more complicated. The software defaults to prune rare words based on their frequency, and has an option for sub-sampling the frequent words. These pruning and sub-sampling happen before the context extraction, leading to a dynamic window size. In addition, the window size is not fixed to k but is sampled uniformly in the range [1, k] for each word.
Australian scientist discovers star with telescope
amod nsubj dobj
prep
pobj
Australian scientist discovers star telescope
amod nsubj dobj
prep with
WORD CONTEXTS
australian scientist/amod−1
scientist australian/amod, discovers/nsubj−1
discovers scientist/nsubj, star/dobj, telescope/prep with star discovers/dobj−1 telescope discovers/prep with−1
Figure 1: Dependency-based context extraction example. Top: preposition relations are collapsed into single arcs, making telescope a direct modifier of discovers. Bottom: the contexts extracted for each word in the sentence.
where lbl is the type of the dependency relation between the head and the modifier (e.g. nsubj, dobj, prep with, amod) and lbl−1 is used to mark the inverse-relation. Relations that include a preposition are “collapsed” prior to context extraction, by directly connecting the head and the object of the preposition, and subsuming the preposition itself into the dependency label. An example of the dependency context extraction is given in Figure 1.
Notice that syntactic dependencies are both more inclusive and more focused than bag-ofwords. They capture relations to words that are far apart and thus “out-of-reach” with small window bag-of-words (e.g. the instrument of discover is telescope/prep with), and also filter out “coincidental” contexts which are within the window but not directly related to the target word (e.g. Australian is not used as the context for discovers). In addition, the contexts are typed, indicating, for example, that stars are objects of discovery and scientists are subjects. We thus expect the syntactic contexts to yield more focused embeddings, capturing more functional and less topical similarity.
4 Experiments and Evaluation
We experiment with 3 training conditions: BOW5 (bag-of-words contexts with k = 5), BOW2 (same, with k = 2) and DEPS (dependency-based syntactic contexts). We modified word2vec to support arbitrary contexts, and to output the context embeddings in addition to the word embeddings. For bag-of-words contexts we used the original word2vec implementation, and for syntactic contexts, we used our modified version. The negative-sampling parameter (how many negative contexts to sample for every correct one) was 15.
Figure 1: Illustration of dependency parse tree for sentence “Australian scientist discovers star with telescope”. Note that preposition relation is collapsed in the bottom sub-figure, where telescope is con idered as a direct modifier of discovers. We use the collapsed version of dependency in all of our experiments the same as Levy and Goldberg (2014b)
2.2 Context Representations
In CSG and CBOW, contexts are represented by words without any additional information. Levy and Goldberg (2014b); Ling et al. (2015) improve them by introducing position-bound words, where each contextual word is associated with their relative position to the target word. This allows CSG and CBOW to distinguish different sequential positions and capture context’s structural information. We name the method that bind additional information with the contextual word as bound (context) representation, as opposited to unbound (context) representation where word is used alone.
For dependency-based context, the original DEPS uses bound representation by default: words are associated with their dependency relation to the target word. Similar to bound representation in linear context type, this allows word embedding models to capture more dependency information. An example is shown in Table 2. In this paper, we
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Context Representation
Context Type Linear Dependency-based
unbound australian, scientist, star, with
scientist, star, telescope
bound australian/-2, scientist/-1, star/+1, with/+2 scientist/nsubj, star/dobj, telescope/prep with
Table 2: Illustration of bound and unbound representations under linear and dependency-based context types. This example is based on Figure 1 and the target word is “discovers”.
also investigate the simpler context representation where no dependency relation is considered. This also makes a fair comparison with linear context models like CSG, CBOW and GloVe, since they do not use bound representation either.
Intuitively, bound representation should work better than unbound representation, since it is more sophisticated by considering position or dependency relation. However, this is not always the case in practice. The biggest drawback of word embeddings learned with bound context type is the ignorance of syntax. The bound representation already contains a certain degree of syntactic information, thus word embedding models can not learn it from the input word-context pairs. Another drawback is that bound context representation is sparse, especially for dependency-based context. There are 47 dependency relations in dependency parse tree. Although not every combination of dependency relations and words appear in the wordcontext pair collection, it still enlarges the context vocabulary about 5 times in practice.
Compared to context types (linear and dependency-based), the choice of context representations (bound and unbound) have more effects to the quality of the learned word embeddings. Bound representation transfers each contextual word into a new one, and the word-context pairs are changed completely. As for context types, a lot of word-context pairs in linear context type also appear in dependency-based context type. For example, in Table 2, “scientist” and “star” are considered as the contexts of “discovers” in both context types.
2.3 Generalization
For convenience, we first define the collection of word-context pairs as P . P can be merged based on the words to form a collection M with size of
Linear (window size 1) Dependency-based
P (australian, scientist) (scientist, australian) (scientist, discovers) (discovers, scientist) (discovers, star) . . .
(australian, scientist) (scientist, australian) (scientist, discovers) (discovers, scientist) (discovers, star) (discovers, telescope) . . .
M (australian, scientist) (scientist, australian, discovers) (discovers, scientist, star) . . .
(australian, scientist) (scientist, australian, discovers) (discovers, scientist, star, telescope) . . .
M (australian, scientist, 1) (scientist, australian, 1) (scientist, discovers, 1) (discovers, scientist, 1) (discovers, star, 1) . . . (australian, scientist, 1) (scientist, australian, 1) (scientist, discovers, 1) (discovers, scientist, 1) (discovers, star, 1) (discovers, telescope, 1) . . .
Table 3: Illustration of collection P , M andM for sentence “australian scientist discovers star with telescope”. Unbound representation is used in this example. Words in the collections are Bold. Contexts and numbers in the collections are Normal.
|C|. Each element (w, c1, c2, .., cnw) ∈ M is the word w and its contexts, where nw is the number of word w’s contexts. P can also be merged based on both words and contexts to form a collection M . Each element (w, c,#(w, c)) ∈ M is the word w, context c, and the times they appear in collection P . An example of these collections is shown in Table 3.
2.3.1 Generalized Bag-Of-Words
The objective function of Generalized Bag-OfWords (GBOW) is defined as:
∑ (w,c1,..,cnw )∈M log p
( w ∣∣∣∣∣ nw∑ i=1 ~ci ) (1)
With negative sampling technique, the log probability is calculated by:
log σ ( ~w ·
nw∑ i=1 ~ci
) −
K∑ k=1 log σ
( ~wN ·
nw∑ 1=i ~ci ) (2)
where σ is the sigmoid function, K is the negative sampling size, ~w and ~c is the vector for word w and c respectively. The negatively sampled word wN is randomly selected based on its unigram distribution ( #(w)∑
w #(w) )ds, where #(w) is the number
of times that word w appears in the corpus, ds is the distribution smoothing hyper-parameter which is usually defined as 0.75.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Note that in the original CBOW (Mikolov et al., 2013a) with negative sampling technique, the probability is actually p (c| ∑ ~wi) instead of
p (w| ∑ ~ci). In another word, the original CBOW uses the sum of word vectors to predict context. This works well for linear context. But for dependency-based context with bound representation, there is only one word available for predicting its context. For example in Figure 1, the context “scientist/nsubj” can only be predicted by word “discovers”. However, a word can be predicted by the sum of several contexts. Due to this reason, we exchange the role of word and context in GBOW. The negative sampling objective is also changed from context cN to word wN .
2.3.2 Generalized Skip-Gram
For generalized Skip-Gram (GSG), the definition is straightforward and the objective function actually needs no modification (Levy and Goldberg, 2014b). However, in order to make it consistent with our GBOW, we also exchange the role of word and context. The objective function of GSG is defined as:∑
(w,c)∈P log p (w|~c)
= ∑
(w,c)∈P
[ log σ (~w · ~c)−
K∑ k=1 log σ ( ~wN · ~c) ] (3)
2.3.3 GloVe
Unlike GSG and GBOW, GloVe explicitly optimizes a log-bilinear regression model based on word co-occurrence matrix. Since GloVe is already a very generalized model, with the previous defined collection M , the final objective function is written as:∑ (w,c)∈M f(#(w, c))(~w ·~c+ ~bw+ ~bc− log#(w, c)) (4) where ~bw and ~bc are biases for word and context. f is a non-decreasing weighting function and ensures that large #(w, c) is not over-weighted.
Note that the inputs of GSG, GBOW and Glove are the collections P , M and M respectively. Once the corpus and hyper-parameters are fixed, these collections (and thus the learned word embeddings) are determined only by the choice of context types and representations.
3 Experiments
We evaluate the effectiveness of different context types and representations on word similarity, word analogy, part-of-speech tagging, chunking, named entity recognition, and text classification tasks. In this section, we first describe the training details of word embedding models. We then report and discuss the experimental results on each task. Detailed numerical results can be found in Supplemental Material.
3.1 Training Details
Previously, the word2vecf toolkit 3 (Levy et al., 2015) extends the word2vec toolkit 4 (Mikolov et al., 2013b) to accept the input of collection P rather than raw corpus. This makes CSG model accept arbitrary contexts (e.g. dependencybased context). However, CBOW and GloVe are not considered in that toolkit. We implement word2vecPM toolkit, a further extension of word2vecf, which supports generalized SG, CBOW and GloVe with the input of collection P , M and M respectively.
We use English Wikipedia (August 2013 dump) as the training corpus in all of our experiments. The Stanford CoreNLP (Manning et al., 2014) is used for dependency parsing. All words and contexts are converted to lower case after parsing. Words and contexts that appear less than 100 times in the collection P are directly ignored. Note that this is slightly different from ignoring rare words that appear less than 100 times in the vocabulary based on the corpus, since each word may appear more times in the collection than that in the vocabulary.
Most hyper-parameters are the same as Levy et al. (2015)’s best configuration. For example, negative sampling size K is set to 5 for GSG and 2 for GBOW. Distribution smoothing cds is set to 0.75. No dynamic context or “dirty” sub-sampling is used. The window size wn is fixed to 2 for constructing linear context, which ensures the number of the (merged) word-context pair collection for both linear context and dependency-based context is comparable. The number of iteration is set to 2, 5 and 30 for GSG, GBOW and GloVe respectively. Unless otherwise noted, the number of word embedding dimension is set to 500. Since the aim
3https://bitbucket.org/yoavgo/ word2vecf
4http://code.google.com/p/word2vec/
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
0 100 200 300 400 500 dimension
0.65
0.70
0.75 0.80 co rr e la ti o n
GSG on WordSim353 similarity
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.80
co rr
e la
ti o n
GBOW on WordSim353 similarity
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
0.55
0.60
0.65
0.70
0.75
co rr
e la
ti o n
GloVe on WordSim353 similarity
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
0.30
0.35
0.40
0.45
0.50
0.55
0.60
co rr
e la
ti o n
GSG on WordSim353 relatedness
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.55
co rr
e la
ti o n
GBOW on WordSim353 relatedness
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
0.30
0.35
0.40
0.45
0.50
0.55
co rr
e la
ti o n
GloVe on WordSim353 relatedness
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
Figure 2: Results on WordSim353 (similarity and relatedness) dataset.
of this paper is not comparing the performance of different word embedding models, the results of GSG, GBOW and GloVe are reported respectively.
3.2 Word Similarity Task
Word similarity task aims at producing semantic similarity scores of word pairs, which are compared with the human scores using Spearman’s correlation. The cosine distance is used for generating similarity scores between two word vectors. WordSim353 (Finkelstein et al., 2001) dataset with similarity and relatedness partition (Zesch et al., 2008; Agirre et al., 2009) is used for this task.
Previous researches (Levy and Goldberg, 2014a; Melamud et al., 2016) conclude that compared to linear context, dependency-based context can capture more functional similarity (e.g. tiger/cat) rather than topical similarity (relatedness) (e.g., tiger/jungle). However, their experiments do not distinguish the effect of differen-
t context representations: unbound representation is used for linear context (Mikolov et al., 2013b) while bound representation is used for dependency-based context (Levy and Goldberg, 2014a). Moreover, only CSG model is compared.
We revisit those claims based on more systematical experiments. As shown in Figure 2’s top-left sub-figure, compared to linear context (solid and dotted blue line), the better results of dependencybased context for GSG and GloVe (solid and dotted red line) on ws353’s similarity partition confirms its ability of capturing functional similarity. However, the good performances of dependencybased context do not fully transfer to GBOW. Although dependency-based context with bound representation (dotted red line) for GBOW is still the best performer, dependency-based context with unbound representation (solid red line) for GBOW performs worst on ws353’s similarity partition. Distinguishing bound representation from unbound representation is important.
Note that the results are also reversed on ws353’s relatedness partition (Figure 2’s right subfigures), which shows the use of linear context is more suitable for capturing topical similarity.
Overall, dependency-based context type does not get all the credit for capturing functional similarity. Context representations play an important role for word similarity task. It’s only safe to say that dependency-based context captures functional similarity with the “help” of bound representation. In contrast, linear context type captures topical similarity with the “help” of unbound representation.
3.3 Word Analogy Task
Word analogy task aims at answering the questions like “a is to b as c is to ?”. For example, “London is to Britain as Tokyo is to Japan”. We follow the evaluation protocol in Levy and Goldberg (2014b), answering the questions using both 3CosAdd (additive) and 3CosMul (multiplicative) functions. Our experiments show that 3CosMul works consistently better than 3CosAdd, thus only the results of 3CosMul are reported. We follow previous researches and use Google’s analogy dataset (Mikolov et al., 2013a) (with semantic and syntactic partition) in our experiments.
As shown in Figure 3, we observe that context representation plays an important role in word analogy task. The choice of context representa-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
0 100 200 300 400 500 dimension
0.1
0.2
0.3
0.4
0.5
0.6
0.7 0.8 a cc u ra cy ( % )
GSG on Google Sem (3CosMul)
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
a cc
u ra
cy (
% )
GBOW on Google Sem (3CosMul)
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
0.0
0.1
0.2
0.3
0.4
0.5
0.6
a cc
u ra
cy (
% )
GloVe on Google Sem (3CosMul)
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
0.1
0.2
0.3
0.4
0.5
0.6
0.7
a cc
u ra
cy (
% )
GSG on Google Syn (3CosMul)
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
0.0
0.1
0.2
0.3
0.4
0.5
0.6
a cc
u ra
cy (
% )
GBOW on Google Syn (3CosMul)
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
a cc
u ra
cy (
% )
GloVe on Google Syn (3CosMul)
Figure 3: Results on Google (Sem and Syn) dataset.
tion (word or bound word) actually has much larger impact than the choice of context type (linear or dependency). The results on Google Syn dataset (Figure 3’s sub-figures in the second column) is perhaps the most evident. The performance of linear context and dependency-based context with unbound representation is similar. However, when bound representation is used, the performance of GSG and GBOW drops more than 30 percent for dependency-based context and around 20 percent for linear context. The main reason for this phenomenon is that the bound representation already contains syntactic information, thus word embedding models can not learn it from the input wordcontext pairs. It can also be observed that GloVe is more sensitive to different context representations than Skip-Gram and CBOW, which is probably due to its explicitly defined/optimized objective function.
3.4 POS, Chunking and NER Tasks
Although intrinsic evaluations like word similarity and word analogy tasks could provide direct
insights of different context types and representations, the experimental results above cannot be directly translated to the typical uses of word embeddings. For example, these tasks aren’t necessarily correlated with downstream tasks’ performances, as shown in (Schnabel et al., 2015; Linzen, 2016; Chiu et al., 2016). More extrinsic tasks should be considered.
In this subsection, we evaluate the effectiveness of different word embedding models with different contexts on Part-of-Speech Tagging (POS), Chunking and Named Entity Recognition (NER) tasks. These tasks can be categorized as sequence labeling. It aims at automatically assigning words in texts with labels. CoNLL 2000 shared task 5 is used as benchmark for POS and Chunking. CoNLL 2003 shared task 6 is used as benchmark for NER.
Inspired by the evaluation protocol used in Kiros et al. (2015), we restrict the predicting model to simple linear classifier. The classifier’s input for predicting the label of word wi is simply the concatenation of vectors ~wi−2, ~wi−1, ~wi, ~wi+1, ~wi+2. This ensures the quality of embedding models is directly evaluated, and their strengths and weaknesses are easily observed.
As shown in Figure 4, the overall trends of GSG, GBOW and GloVe are similar. When the same context type is used, bound representation (dotted line) outperforms unbound representation (solid line) on all datasets. Sequence labeling tasks tend to classify words with the same syntax to the same category. The ignorance of syntax for word embeddings which are learned by bound representation becomes beneficial. Moreover, dependencybased context type works better than linear context type in most cases. These results suggest that linear context type with unbound representations (as in traditional CSG and CBOW) may not be the best choice of input word vectors for sequence labeling. Bound representations should always be used and dependency-based context type is also worth considering. Again, similar to that on word analogy task, GloVe is more sensitive to different context representations than Skip-Gram and CBOW on sequence labeling tasks.
5http://www.cnts.ua.ac.be/conll2000/ chunking
6http://www.cnts.ua.ac.be/conll2003/ ner
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
0 100 200 300 400 500 dimension
84
86
88
90
92
94
96
98
a cc
u ra
cy (
% )
GSG on Part-of-Speech Tagging
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
86
88
90
92
94
96
98
a cc
u ra
cy (
% )
GBOW on Part-of-Speech Tagging
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
80
82
84
86
88
90
92
94
96
a cc
u ra
cy (
% )
GloVe on Part-of-Speech Tagging
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
65
70
75
80
85
90
F1 -s
co re
( %
)
GSG on Chunking
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
65
70
75
80
85
90
F1 -s
co re
( %
)
GBOW on Chunking
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
60
65
70
75
80
85
90
F1 -s
co re
( %
)
GloVe on Chunking
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
66
68
70
72
74
76
78
F1 -s
co re
( %
)
GSG on Named Entity Recognition
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
66
68
70
72
74
76
F1 -s
co re
( %
)
GBOW on Named Entity Recognition
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
0 100 200 300 400 500 dimension
60
62
64
66
68
70
72
74
76
F1 -s
co re
( %
)
GloVe on Named Entity Recognition
linear ctx w/ unbound rep linear ctx w/ bound rep dep ctx w/ unbound rep dep ctx w/ bound rep
Figure 4: Results on POS, Chunking and NER tasks.
3.5 Text Classification Task
Finally, we evaluate the effectiveness of different word embedding models with different contexts on text classification task. Text classification is one of the most popular and well-studied tasks in natural language processing. Recently, deep neural networks are dominant on this task (Socher et al., 2013; Kim, 2014; Dai and Le, 2015). They often need pre-trained word embeddings as inputs to improve their performances. Similar to the previous evaluation of sequence labeling tasks, instead of building complex deep neural networks, we use a simpler classification method called Neural Bag-of-Words to directly evaluate the word embeddings: texts are first represented by the sum of their belonging words’ vectors, then a Logistic Regression Classifier is built upon them for classification.
Different word embedding models are evaluated on 5 text classification datasets. The first 3 datasets are sentence-level: short movie review sentiment (MR) (Pang and Lee, 2005), customer product re-
Model Context Context Sentence-level Document-levelType Rep. MR CR Subj RT-2k IMDB
GSG linear word 76.1 78.3 90.9 83.5 85.2bound 75.3 79.0 90.4 82.2 85.2
dep word 76.0 77.7 90.7 84.8 85.1bound 75.0 77.5 90.0 84.7 84.5
GBOW linear word 74.9 77.9 90.4 82.0 85.0bound 74.1 77.8 90.3 80.7 84.1
dep word 75.0 77.6 90.1 82.4 84.9bound 73.5 78.2 89.9 80.7 83.4
GloVe linear word 73.4 76.7 89.6 79.2 83.5bound 73.2 77.5 90.0 79.8 83.4
dep word 74.0 77.7 89.5 81.3 83.5bound 72.5 76.7 88.8 79.2 83.5 Random word embeddings 63.9 72.8 79.9 72.2 77.2
Table 4: Results on 5 text classification datasets.
views (CR) (Nakagawa et al., 2010), and subjectivity/objectivity classification (SUBJ) (Pang and Lee, 2004). The other 2 datasets are documentlevel with multiple sentences: full-length movie review (RT-2k) (Pang and Lee, 2004), and IMDB movie review (IMDB) (Maas et al., 2011).
As shown in Table 4, pre-trained word embeddings outperform random word embeddings by a large margin. This further strengthens previous
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
researches that pre-trained word embeddings are crucial for text classification. Unlike that on previous tasks, different models’ results are actually very similar on text classification task. Text classification has less focus on syntax and function similarity. This leads to the phenomenon that models which use bound representation perform worse than those which use unbound representation on all datasets except CR. Models that use dependency-based context type and linear context type are comparable. These observations suggest that simple linear context type with unbound representations (as in traditional CSG and CBOW) is still the best choice of pre-training word embeddings for text classification, which is already used in most researches.
4 Related Work
Previously, there are researches which directly compare different word embedding models. Lai et al. (2016) compare 6 word embedding models using different corpora and hyper-parameters. Levy and Goldberg (2014c) show the theoretical equivalence of CSG and PPMI matrix factorization. Levy et al. (2015) further discuss the connections between 4 word embedding models (PPMI, PPMI+SVD, CSG, GloVe) and re-evaluate them with the same hyper-parameters. Suzuki and Nagata (2015) investigate different configurations of CSG and Glove, then merge them into a unified form. Yin and Schutze (2016) propose 4 ensemble methods and show their effectiveness over individual word embeddings.
There are also researches which focus on evaluating different context types in learning word embeddings. Vulic and Korhonen (2016) compare CSG and dependency-based models on various languages. The results suggest that dependencybased models are able to detect functional similarity in English. However, the advantages of dependency-based context over linear context on other languages are not as promising as that on English. Bansal et al. (2014) investigate different embedding models for parsing task and show that dependency-based context is more suitable than linear context on this task. Melamud et al. (2016) investigate the performance of CSG, Deps and a substitute-based word embedding models (Yatbaz et al., 2012) 7, which shows that different types of
7We do not consider this type of context, since it performs consistently worse than the other two context types. The
intrinsic tasks have clear preference to particular types of contexts. On the other hand, for extrinsic tasks, the optimal context types need to be carefully tuned on specific dataset. However, context representations (bound and unbound) are not evaluated in these models. Moreover, they focus only on the more popular and intuitive CSG model, but not on CBOW and GloVe.
5 Conclusion
To the best of our knowledge, this paper provides the first systematical investigation of different context types and representations for learning word embeddings. We evaluate different models on intrinsic property analysis (word similarity and word analogy), sequence labeling tasks (POS, Chunking and NER) and text classification task.
Overall, the tendency of different models on different tasks is similar. However, most tasks have clear preference for different context types and representations. Context representations play a more important role than context types for learning word embeddings. More precisely: 1) Unbound representation is more suitable for syntactic word analogy than bound representation. Bound representation already contains syntactic information, which makes it difficult to learn syntactic aware word embeddings based on the input wordcontext pairs. 2) No matter which type of context to be used, bound representation is essential for sequence labeling tasks, which benefits from its ability of capturing functional similarity. In contrast, unbound representation, which is suitable for capturing topical similarity, doesn’t contribute to sequence labeling tasks. 3) Linear context with unbound representation (Skip-Gram) is still the best choice for text classification task. Linear context is enough for capturing topical similarity compared to dependency-based context. Words’ position information is generally useless for text classification, which makes bound representation contribute less to this task.
In the spirit of transparent and reproducible experiments, the word2vecPM toolkit in Supplemental Material will be published online. We hope researchers will take advantage of the code for further improvements and applications to other tasks.
same observation is made by Melamud et al. (2016); Vulic and Korhonen (2016)
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Supplemental Material
A.1 Numerical Results
For simplicity and clarity, most experimental results are shown in the form of line chart. However, numerical results are more accurate and can be directly used by other researches. We list them in Table 5, 6 and 7. Upon acceptance, they will be added to the final version.
Model Context Context WS353
Type Rep. Sim Related
GSG linear
word .757 .563 bound .762 .543
dep word .776 .531 bound .792 .483
GBOW linear
word .747 .503 bound .689 .427
dep word .669 .395 bound .799 .502
GloVe linear
word .645 .545 bound .670 .481
dep word .696 .539 bound .734 .468
Table 5: Numerical results on word similarity task. Best results in group are marked Bold.
Model Context Context Google
Type Rep. Sem Syn
GSG linear
word .708 .639 bound .702 .454
dep word .716 .661 bound .600 .307
GBOW linear
word .628 .566 bound .602 .376
dep word .573 .553 bound .495 .248
GloVe linear
word .471 .719 bound .502 .218
dep word .513 .700 bound .402 .121
Table 6: Numerical results on word analogy task. Best results in group are marked Bold.
Model Context Context
POS Chunking NER Type Rep.
GSG linear
word 95.3 87.2 76.6 bound 96.0 88.5 77.4
dep word 95.6 87.5 75.5 bound 96.3 88.5 76.2
GBOW linear
word 95.2 87.7 74.7 bound 95.7 88.3 75.2
dep word 95.4 87.3 74.3 bound 96.0 88.6 75.5
GloVe linear
word 91.6 79.6 70.8 bound 95.5 88.2 74.8
dep word 92.8 82.0 70.7 bound 95.5 87.5 72.0

- Strengths:  This paper presents a 2 x 2 x 3 x 10 array of accuracy results based on systematically changing the parameters of embeddings models:  (context type, position sensitive, embedding model, task), accuracy  - context type  u2208 {Linear, Syntactic} - position sensitive  u2208 {True, False} - embedding model  u2208 {Skip Gram, BOW, GLOVE} - task  u2208 {Word Similarity, Analogies, POS, NER, Chunking, 5 text classific. tasks}  The aim of these experiments was to investigate the variation in performance as these parameters are changed. The goal of the study itself is interesting for the ACL community and similar papers have appeared before as workshop papers and have been well cited, such as Nayak et al.'s paper mentioned below.  - Weaknesses: Since this paper essentially presents the effect of systematically changing the  context types and position sensitivity, I will focus on the execution of the investigation and the analysis of the results, which I am afraid is not  satisfactory.  A) The lack of hyper-parameter tuning is worrisome. E.g.    - 395 Unless otherwise notes, the number of word embedding dimension is set to 500.    - 232 It still enlarges the context vocabulary about 5 times in practice.    - 385 Most hyper-parameters are the same as Levy et al' best configuration.    This is worrisome because lack of hyperparameter tuning makes it difficult to make statements like method A is better than method B. E.g. bound methods may perform better with a lower dimensionality than unbound models, since their effective context vocabulary size is larger.  B) The paper sometimes presents strange explanations for its results. E.g.    - 115  Experimental results suggest that although it's hard to find any  universal insight, the characteristics of different contexts on different models are concluded according to specific tasks.      What does this sentence even mean?      - 580 Sequence labeling tasks tend to classify words with the same syntax  to the same category. The ignorance of syntax for word embeddings which  are learned by bound representation becomes beneficial.      These two sentences are contradictory, if a sequence labeling task    classified words with  same syntax  to same category then syntx becomes    a ver valuable feature. Bound representation's ignorance of syntax    should cause a drop in performance just like other tasks which does not    happen.  C) It is not enough to merely mention Lai et. al. 2016 who have also done a    systematic study of the word embeddings, and similarly the paper      Evaluating Word Embeddings Using a Representative Suite of Practical    Tasks , Nayak, Angeli, Manning. appeared at the repeval workshop at     ACL 2016. should have been cited. I understand that the focus of Nayak    et al's paper is not exactly the same as this paper, however they    provide recommendations about hyperparameter tuning and experiment    design and even provide a web interface for automatically running    tagging experiments using neural networks instead of the  simple linear    classifiers  used in the current paper.  D) The paper uses a neural BOW words classifier for the text classification tasks    but a simple linear classifier for the sequence labeling tasks. What is    the justification for this choice of classifiers? Why not use a simple    neural classifier for the tagging tasks as well? I raise this point,    since the tagging task seems to be the only task where bound    representations are consistently beating the unbound representations,    which makes this task the odd one out.   - General Discussion: Finally, I will make one speculative suggestion to the authors regarding the analysis of the data. As I said earlier, this paper's main contribution is an analysis of the following table. (context type, position sensitive, embedding model, task, accuracy) So essentially there are 120 accuracy values that we want to explain in terms of the aspects of the model. It may be beneficial to perform factor analysis or some other pattern mining technique on this 120 sample data.


Linguistically Regularized LSTM for Sentiment Classification
This paper deals with sentence-level sentiment classification. Though a variety of neural network models have been proposed recently, however, previous models either depend on expensive phrase-level annotation, most of which has remarkably degraded performance when trained with only sentence-level annotation; or do not fully employ linguistic resources (e.g., sentiment lexicons, negation words, intensity words). In this paper, we propose simple models trained with sentence-level annotation, but also attempt to model the linguistic role of sentiment lexicons, negation words, and intensity words. Results show that our models are able to capture the linguistic role of sentiment words, negation words, and intensity words in sentiment expression.
1 Introduction
Sentiment classification aims to classify text to sentiment classes such as positive or negative, or more fine-grained classes such as very positive, positive, neutral, etc. There has been a variety of approaches for this purpose such as lexicon-based classification (Turney, 2002; Taboada et al., 2011), and early machine learning based methods (Pang et al., 2002; Pang and Lee, 2005), and recently neural network models such as convolutional neural network (CNN) (Kim, 2014; Kalchbrenner et al., 2014; Lei et al., 2015), recursive autoencoders (Socher et al., 2011, 2013), Long ShortTerm Memory (LSTM) (Mikolov, 2012; Chung et al., 2014; Tai et al., 2015; Zhu et al., 2015), and many more.
In spite of the great success of these neural models, there are some defects in previous studies.
First, tree-structured models such as recursive autoencoders and Tree-LSTM (Tai et al., 2015; Zhu et al., 2015), depend on parsing tree structures and expensive phrase-level annotation, whose performance drops substantially when only trained with sentence-level annotation. Second, linguistic knowledge such as sentiment lexicon, negation words or negators (e.g., not, never), and intensity words or intensifiers (e.g., very, absolutely), has not been fully employed in neural models. The goal of this research is to developing simple sequence models but also attempts to fully employing linguistic resources to benefit sentiment classification. Firstly, we attempts to develop simple models that do not depend on parsing trees and do not require phrase-level annotation which is too expensive in real-world applications. Secondly, in order to obtain competitive performance, simple models can benefit from linguistic resources. Three types of resources will be addressed in this paper: sentiment lexicon, negation words, and intensity words. Sentiment lexicon offers the prior polarity of a word which can be useful in determining the sentiment polarity of longer texts such as phrases and sentences. Negators are typical sentiment shifters (Zhu et al., 2014), which constantly change the polarity of sentiment expression. Intensifiers change the valence degree of the modified text, which is important for fine-grained sentiment classification. In order to model the linguistic role of sentiment, negation, and intensity words, our central idea is to regularize the difference between the predicted sentiment distribution of the current position 1, and that of the previous or next positions, in a sequence model. For instance, if the current position is a negator not, the negator should change the sentiment distribution of the next posi-
1Note that in sequence models, the hidden state of the current position also encodes forward or backward contexts.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
tion accordingly. To summarize, our contributions lie in two folds:
• We discover that modeling the linguistic role of sentiment, negation, and intensity words can enhance sentence-level sentiment classification. We address the issue by imposing linguistic-inspired regularizers on sequence LSTM models.
• Unlike previous models that depend on parsing structures and expensive phrase-level annotation, our models are simple and efficient, but the performance is on a par with the stateof-the-art.
The rest of the paper is organized as follows: In the following section, we survey related work. In Section 3, we briefly introduce the background of LSTM and bidirectional LSTM, and then describe in detail the lingistic regularizers for sentiment/negation/intensity words in Section 4. Experiments are presented in Section 5, and Conclusion follows in Section 6.
2 Related Work
2.1 Neural Networks for Sentiment Classification
There are many neural networks proposed for sentiment classification. The most noticeable models may be the recursive autoencoder neural network which builds the representation of a sentence from subphrases recursively (Socher et al., 2011, 2013; Dong et al., 2014; Qian et al., 2015). Such recursive models usually depend on a tree structure of input text, and in order to obtain competitive results, usually require annotation of all subphrases. Sequence models, for instance, convolutional neural network (CNN), do not require tree-structured data, which are widely adopted for sentiment classification (Kim, 2014; Kalchbrenner et al., 2014; Lei et al., 2015). Long short-term memory models are also common for learning sentence-level representation due to its capability of modeling the prefix or suffix context (Hochreiter and Schmidhuber, 1997). LSTM can be commonly applied to sequential data but also tree-structured data (Zhu et al., 2015; Tai et al., 2015).
2.2 Applying Linguistic Knowledge for Sentiment Classification
Linguistic knowledge and sentiment resources, such as sentiment lexicons, negation words (not,
never, neither, etc.) or negators, and intensity words (very, extremely, etc.) or intensifiers, are useful for sentiment analysis in general.
Sentiment lexicon (Hu and Liu, 2004; Wilson et al., 2005) usually defines prior polarity of a lexical entry, and is valuable for lexicon-based models (Turney, 2002; Taboada et al., 2011), and machine learning approaches (Pang and Lee, 2008). There are recent works for automatic construction of sentiment lexicons from social data (Vo and Zhang, 2016) and for multiple languages (Chen and Skiena, 2014). A noticeable work that ultilizes sentiment lexicons can be seen in (Teng et al., 2016) which treats the sentiment score of a sentence as a weighted sum of prior sentiment scores of negation words and sentiment words, where the weights are learned by a neural network.
Negation words play a critical role in modifying sentiment of textual expressions. Some early negation models adopt the reversing assumption that a negator reverses the sign of the sentiment value of the modified text (Polanyi and Zaenen, 2006; Kennedy and Inkpen, 2006). The shifting hyothesis assumes that negators change the sentiment values by a constant amount (Taboada et al., 2011; Liu and Seneff, 2009). Since each negator can affect the modified text in different ways, the constant amount can be extended to be negatorspecific (Zhu et al., 2014), and further, the effect of negators could also depend on the syntax and semantics of the modified text (Zhu et al., 2014). Other approaches to negation modeling can be seen in (Jia et al., 2009; Wiegand et al., 2010; Benamara et al., 2012; Lapponi et al., 2012).
Sentiment intensity of a phrase indicates the strength of associated sentiment, which is quite important for fine-grained sentiment classification or rating. Intensity words can change the valence degree (i.e., sentiment intensity) of the modified text. In (Wei et al., 2011) the authors propose a linear regression model to predict the valence value for content words. In (Malandrakis et al., 2013), a kernel-based model is proposed to combine semantic information for predicting sentiment score. In the SemEval-2016 task 7 subtask A, a learningto-rank model with a pair-wise strategy is proposed to predict sentiment intensity scores (Wang et al., 2016). Linguistic intensity is not limited to sentiment or intensity words, and there are works that assign low/medium/high intensity scales to adjectives such as okay, good, great (Sharma et al.,
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
2015) or to gradable terms (e.g. large, huge, gigantic) (Shivade et al., 2015).
In (Dong et al., 2015), a sentiment parser is proposed, and the authors studied how sentiment changes when a phrase is modified by negators or intensifiers.
3 Long Short-term Memory Network
3.1 Long Short-Term Memory (LSTM)
Long Short-Term Memory has been widely adopted for text processing. Briefly speaking, in LSTM, the hidden states ht and memory cell ct is a function of their previous ct−1 and ht−1 and input vector xt, or formally as follows:
ct, ht = g (LSTM)(ct−1, ht−1, xt) (1)
The hidden state ht ∈ Rd denotes the representation of position t while also encoding the preceding contexts of the position. For more details about LSTM, we refer readers to (Hochreiter and Schmidhuber, 1997).
3.2 Bidirectional LSTM
In LSTM, the hidden state of each position (ht) only encodes the prefix context in a forward direction while the backward context is not considered. Bidirectional LSTM (Graves et al., 2013) exploited two parallel passes (forward and backward) and concatenated hidden states of the two LSTMs as the representation of each position. The forward and backward LSTMs are respectively formulated as follows:
−→c t, −→ h t = g (LSTM)(−→c t−1, −→ h t−1, xt) (2)
←−c t, ←− h t = g (LSTM)(←−c t+1, ←− h t+1, xt) (3)
where g(LSTM) is the same as that in Eq (1). Particularly, parameters in the two LSTMs are shared. The representation of the entire sentence is [ −→ h n, ←− h 1], where n is the length of the sentence. At each position t, the new representation is ht = [ −→ h t, ←− h t], which is the concatenation of hidden states of the forward LSTM and backward LSTM. In this way, the forward and backward contexts can be considered simultaneously.
4 Linguistically Regularized LSTM
The central idea of the paper is to model the linguistic role of sentiment, negation, and intensity words in sentence-level sentiment classification by
regularizing the outputs at adjacent positions of a sentence. For example, in sentence “this movie is interesting”, the predicted sentiment distributions at “this*2”, “this movie*”, and “this movie is*” should be close to each other, while the predicted sentiment distribution at “this movie is very interesting*” should be quite different from the preceeding positions (“this movie is very*”) since a sentiment word (“interesting”) is seen. We propose a generic regularizer and three special regularizers based on the following linguistic observations:
• Non-Sentiment Regularizer: if the two adjacent positions are all non-opinion words, the sentiment distributions of the two positions should be close to each other. Though this is not always true (e.g., soap movie), this assumption holds at most cases.
• Sentiment Regularizer: if the word is a sentiment word found in a lexicon, the sentiment distribution of the current position should be significantly different from that of the next or previous positions. We approach this phenomenon with a sentiment class specific shifting distribution.
• Negation Regularizer: Negation words such as “not” and “never” are critical sentiment shifter or converter: in general they shift sentiment polarity from the positive end to the negative end, but sometimes depend on the negation word and the words they modify. The negation regularizer models this linguistic phenomena with a negator-specific transformation matrix.
• Intensity Regularizer: Intensity words such as “very” and “extremely” change the valence degree of a sentiment expression: for instance, from positive to very positive. Modeling this effect is quite important for finegrained sentiment classification, and the intensity regularizer is designed to formulate this effect by a word-specific transformation matrix.
More formally, the predicted sentiment distribution (pt, based on ht, see Eq. 5) at position t should be linguistically regularized with respect to that of the preceding (t − 1) or following (t + 1)
2The asterisk denotes the current position.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
positions. In order to enforce the model to produce coherent predictions, we plug a new loss term into the original cross entropy loss:
L(θ) = − ∑ i yi log pi + α ∑ i ∑ t Lt,i + β||θ||2 (4)
where yi is the gold distribution for sentence i, pi is the predicted distribution, Lt,i is one of the above regularizers or combination of these regularizers on sentence i, α is the weight for the regularization term, and t is the word position in a sentence.
Note that we do not consider the modification span of negation and intensity words to preserve the simplicity of the proposed models. Negation scope resolution is another complex problem which has been extensively studied (Zou et al., 2013; Packard et al., 2014; Fancellu et al., 2016), which is beyond the scope of this work. Instead, we resort to sequence LSTMs for encoding surrounding contexts at a given position.
4.1 Non-Sentiment Regularizer (NSR)
This regularizer constrains that the sentiment distributions of adjacent positions should not vary much if the additional input word xt is not a sentiment word, formally as follows:
L (NSR) t = max(0, DKL(pt||pt−1)−M) (5)
whereM is a hyperparameter for margin, pt is the predicted distribution at state of position t, (i.e., ht), andDKL(p||q) is a symmetric KL divergence defined as follows:
DKL(p||q) = 1
2 C∑ l=1 p(l) log q(l) + q(l) log p(l) (6)
where p, q are distributions over sentiment labels l and C is the number of labels.
4.2 Sentiment Regularizer (SR)
The sentiment regularizer constrains that the sentiment distributions of adjacent positions should drift accordingly if the input word is a sentiment word. Let’s revisit the example “this movie is interesting” again. At position t = 4 we see a positive word “interesting” so the predicted distribution would be more positive than that at position t = 3. This is the issue of sentiment drift.
In order to address the sentiment drift issue, we propose a polarity shifting distribution sc ∈ RC for each sentiment class defined in a lexicon. For
instance, a sentiment lexicon may have class labels like strong positive, weakly positive, weakly negative, and strong negative, and for each class, there is a shifting distribution which will be learned by the model. The sentiment regularizer states that if the current word is a sentiment word, the sentiment distribution drift should be observed in comparison to the previous position, in more details:
p (SR) t−1 = pt−1 + sc(xt) (7)
L (SR) t = max(0, DKL(pt||p (SR) t−1 )−M) (8)
where p(SR)t−1 is the drifted sentiment distribution after considering the shifting sentiment distribution corresponding to the state at position t, c(xt) is the prior sentiment class of word xt, and sc ∈ θ is a parameter to be optimized but could also be set fixed with prior knowledge. Note that in this way all words of the same sentiment class share the same drifting distribution, but in a refined setting, we can learn a shifting distribution for each sentiment word if large-scale datasets are available.
4.3 Negation Regularizer (NR)
The negation regularizer approaches how negation words shift the sentiment distribution of the modified text. When the input xt is a negation word, the sentiment distribution should be shifted/reversed accordingly. However, the negation role is more complex than that by sentiment words, for example, the word “not” in “not good” and “not bad” have different roles in polarity change. The former changes the polarity to negative, while the latter changes to neutral instead of positive. To respect such complex negation effects, we propose a transformation matrix Tm ∈ RC×C for each negation word m, and the matrix will be learned by the model. The regularizer assumes that if the current position is a negation word, the sentiment distribution of the current position should be close to that of the next or previous position with the transformation.
p (NR) t−1 = softmax(Txj × pt−1) (9)
p (NR) t+1 = softmax(Txj × pt+1) (10)
L (NR) t = min
{ max(0, DKL(pt||p(NR)t−1 )−M)
max(0, DKL(pt||p(NR)t+1 )−M) (11)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
where p(NR)t−1 and p (NR) t+1 is the sentiment distuibution after transformation, Txj ∈ θ is the transformation matrix for a negation word xj , a parameter to be learned during training. In total, we train m transformation matrixs for m negation words. Such negator-specific transformation is in accordance with the finding that each negator has its individual negation effect (Zhu et al., 2014).
4.4 Intensity Regularizer (IR)
Sentiment intensity of a phrase indicates the strength of associated sentiment, which is quite important for fine-grained sentiment classification or rating. Intensifier can change the valence degree of the content word. The intensity regularizer models how intensity words influence the sentiment valence of a phrase or a sentence.
The formulation of the intensity effect is quite the same as that in the negation regularizer, but with different parameters of course. For each intensity word, there is a transform matrix to favor the different roles of various intensifiers on sentiment drift. For brevity, we will not repeat the formulas here.
4.5 Applying Linguistic Regularizers to Bidirectional LSTM
To preserve the simplicity of our proposals, we do not consider the modification span of negation and intensity words, which is a quite challenging problem in the NLP community (Zou et al., 2013; Packard et al., 2014; Fancellu et al., 2016). However, we can alleviate the problem by leveraging bidirectional LSTM.
For a single LSTM, we employ a backward LSTM from the end to the beginning of a sentence. This is because, at most times, the modified words of negation and intensity words are usually at the right side of the modified text. But sometimes, the modified words are at the left side of negation and intensity words. To better address this issue, we employ bidirectional LSTM and let the model determine which side should be chosen.
More formally, in Bi-LSTM, we compute a transformed sentiment distribution on−→p t−1 of the forward LSTM and also that on←−p t+1 of the backward LSTM, and compute the minimum distance of the distribution of the current position to the two distributions. This could be formulated as follows:
−→p (R)t−1 = softmax(Txj × −→p t−1) (12)
←−p (R)t+1 = softmax(Txj × ←−p t+1) (13)
L (R) t = min
{ max(0, DKL( −→p t||−→p (R)t−1)−M)
max(0, DKL( ←−p t||←−p (R)t+1)−M)
(14)
where −→p (R)t−1 and ←−p (R)t+1 are the sentiment distributions transformed from the previous distribution −→p t−1 and next distribution ←−p t+1 respectively. Note that R ∈ {NR, IR} indicating the formulation works for both negation and intensity regularizers. Due to the same consideration, we redefine L (NSR) t and L (SR) t with bidirectional LSTM similarly. The formulation is the same and omitted for brevity.
4.6 Discussion
Our models address these linguistic factors with mathematical operations, parameterized with shifting distribution vectors or transformation matrices. In the sentiment regularizer, the sentiment shifting effect is parameterized with a classspecific distribution (but could also be wordspecific if with more data). In the negation and intensity regularizers, the effect is parameterized with word-specific transformation matrices. This is to respect the fact that the mechanism of how negation and intensity words shift sentiment expression is quite complex and highly dependent on individual words. Negation/Intensity effect also depends on the syntax and semantics of the modified text, however, for simplicity we resort to sequence LSTM for encoding surrounding contexts in this paper. We partially address the modification scope issue by applying the minimization operator in Eq. 11 and Eq. 14, and the bidirectional LSTM.
5 Experiment
5.1 Dataset and Sentiment Lexicon
Two datasets are used for evaluating the proposed models: Movie Review (MR) (Pang and Lee, 2005) where each sentence is annotated with two classes as negative, positive and Stanford Sentiment Treebank (SST) (Socher et al., 2013) with five classes { very negative, negative, neutral, positive, very positive}. Note that SST has provided phrase-level annotation on all inner nodes, but we only use the sentence-level annotation since one of our goals is to avoid expensive phrase-level annotation.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
The sentiment lexicon contains two parts. The first part comes fromMPQA (Wilson et al., 2005), which contains 5, 153 sentiment words, each with polarity rating. The second part consists of the leaf nodes of the SST dataset (i.e., all sentiment words) and there are 6, 886 polar words except neural ones. We combine the two parts and ignore those words that have conflicting sentiment labels, and produce a lexicon of 9, 750words with 4 sentiment labels. For negation and intensity words, we collect them manually since the number is small, some of which can be seen in Table 2.
Due to the length limit, we present the implementation details and a full list of resources in the supplementary file.
Dataset MR SST # sentences in total 10,662 11,885
#sen containing sentiment word 10,446 11,211 #sen containing negation word 1,644 1,832 #sen containing intensity word 2,687 2,472
Table 1: The data statistics.
Negation word no, nothing, never, neither, not, seldom, scarcely, etc. Intensity word terribly, greatly, absolutely, too, very, completely, etc.
Table 2: Examples of negation and intensity words.
5.2 Overall Comparison
We include several baselines, as listed below:
RNN/RNTN: Recursive Neural Network over parsing trees, proposed by (Socher et al., 2011) and Recursive Tensor Neural Network (Socher et al., 2013) employs tensors to model correlations between different dimensions of child nodes’ vectors.
LSTM/Bi-LSTM: Long Short-Term Memory (Cho et al., 2014) and the bidirectional variant as introduced previously.
Tree-LSTM: Tree-Structured Long Short-Term Memory (Tai et al., 2015) introduces memory cells and gates into tree-structured neural network.
CNN: Convolutional Neural Network (Kalchbrenner et al., 2014) generates sentence representation by convolution and pooling operations.
CNN-Tensor: In (Lei et al., 2015), the convolution operation is replaced by tensor product and
a dynamic programming is applied to enumerate all skippable trigrams in a sentence. Very strong results are reported. DAN: Deep Average Network (DAN) (Iyyer et al., 2015) averages all word vectors in a sentence and connects an MLP layer to the output layer. Neural Context-Sensitive Lexicon: NCSL (Teng et al., 2016) treats the sentiment score of a sentence as a weighted sum of prior scores of words in the sentence where the weights are learned by a neural network. Firstly, we evaluate our model on the MR dataset and the results are shown in Table 3. We have the following observations: First, both LR-LSTM and LR-Bi-LSTM outperforms their counterparts (81.5% vs. 77.4% and 82.1% vs. 79.3%, resp.), demonstrating the effectiveness of the linguistic regularizers. Second, LR-LSTM and LR-Bi-LSTM perform slightly better than Tree-LSTM but Tree-LSTM leverages a constituency tree structure while our model is a simple sequence model. As future work, we will apply such regularizers to tree-structured models. Last, on the MR dataset, our model is comparable to or slightly better than CNN. For fine-grained sentiment classification, we evaluate our model on the SST dataset which has five sentiment classes { very negative, negative, neutral, positive, very positive} so that we can evaluate the sentiment shifting effect of intensity words. The results are shown in Table 3. We have the following observations: First, linguistically regularized LSTM and BiLSTM are better than their counterparts. It’s worth noting that LR-Bi-LSTM (trained with just sentence-level annotation) is even comparable to Bi-LSTM trained with phrase-level annotation. That means, LR-Bi-LSTM can avoid the heavy phrase-level annotation but still obtain comparable results. Second, our models are comparable to Tree-LSTM but our models are not dependent on a parsing tree and more simple, and hence more efficient. Further, for Tree-LSTM, the model is heavily dependent on phrase-level annotation, otherwise the performance drops substantially (from 51% to 48.1%). Last, on the SST dataset, our model is better than CNN, DAN, and NCSL. We conjecture that the strong performance of CNNTensor may be due to the tensor product operation, the enumeration of all skippable trigrams, and the
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
concatenated representations of all pooling layers for final classification.
Method MR SST phrase-level SST
sentence-level RNN 77.7* 44.8# 43.2* RNTN 75.9# 45.7* 43.4# LSTM 77.4# 46.4* 45.6# Bi-LSTM 79.3# 49.1* 46.5# Tree-LSTM 80.7# 51.0* 48.1# CNN 81.5* 48.0* 46.9# CNN-Tensor - 51.2* 50.6* DAN - - 47.7* NCSL - 51.1* 47.1# LR-Bi-LSTM 82.1 - 48.6 LR-LSTM 81.5 - 48.2
Table 3: The accuracy on MR and SST. Phraselevel means the models use phrase-level annotation for training. And sentence-level means the models only use sentence-level annotation. Results marked with * are re-printed from the references, while those with # are obtained either by our own implementation or with the same codes shared by the original authors.
5.3 The Effect of Different Regularizers
In order to reveal the effect of each individual regularizer, we conduct ablation experiments. Each time, we remove a regularizer and observe how the performance varies. First of all, we conduct this experiment on the entire datasets, and then we experiment on sub-datasets that only contain negation words or intensity words.
The experiment results are shown in Table 4 where we can see that the non-sentiment regularizer (NSR) and sentiment regularizer (SR) play a key role3, and the negation regularizer and intensity regularizer are effective but less important than NSR and SR. This may be due to the fact that only 14% of sentences contains negation words in the test datasets, and 23% contains intensity words, and thus we further evaluate the models on two subsets, as shown in Table 5.
The experiments on the subsets show that: 1) With linguistic regularizers, LR-Bi-LSTM outperforms Bi-LSTM remarkably on these subsets; 2) When the negation regularizer is removed from the model, the performance drops significantly on
3Kindly note that almost all sentences contain sentiment words, see Tab. 1.
Method MR SST LR-Bi-LSTM 82.1 48.6 LR-Bi-LSTM (-NSR) 80.8 46.9 LR-Bi-LSTM (-SR) 80.6 46.9 LR-Bi-LSTM (-NR) 81.2 47.6 LR-Bi-LSTM (-IR) 81.7 47.9 LR-LSTM 81.5 48.2 LR-LSTM (-NSR) 80.2 46.4 LR-LSTM (-SR) 80.2 46.6 LR-LSTM (-NR) 80.8 47.4 LR-LSTM (-IR) 81.2 47.4
Table 4: The accuracy for LR-Bi-LSTM and LRLSTM with regularizer ablation. NSR, SR, NR and IR denotes Non-sentiment Regularizer, Sentiment Regularizer, Negation Regularizer, and Intensity Regularizer respectively.
both MR and SST subsets; 3) Similar observations can be found regarding the intensity regularizer.
Method Neg. Sub. Int. Sub. MR SST MR SST BiLSTM 72.0 39.8 83.2 48.8 LR-Bi-LSTM (-NR) 74.2 41.6 - - LR-Bi-LSTM (-IR) - - 85.2 50.0 LR-Bi-LSTM 78.5 44.4 87.1 53.2
Table 5: The accuracy on the negation sub-dataset (Neg. Sub.) that only contains negators, and intensity sub-dataset (Int. Sub.) that only contains intensifiers.
5.4 The Effect of the Negation Regularizer
To further reveal the linguistic role of negation words, we compare the predicted sentiment distributions of a phrase pair with and without a negation word. The experimental results performed on MR are shown in Fig. 1. Each dot denotes a phrase pair (for example, <interesting, not interesting>), where the x-axis denotes the positive score4 of a phrase without negators (e.g., interesting), and the y-axis indicates the positive score for the phrase with negators (e.g., not interesting). The curves in the figures show this function: [1 − y, y] = softmax(Tnw ∗ [1 − x, x]) where [1 − x, x] is a sentiment distribution on [negative, positive], x is the positive score of the phrase without negators (x-axis) and y that of the phrase with negators (y-
4 The score is obtained from the predicted distribution, where 1 means positive and 0 means negative.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
axis), and Tnw is the transformation matrix for the negation word nw (see Eq. 9). By looking into the
Figure 1: The sentiment shifts with negators. Each dot< x, y > indicates that x is the sentiment score of a phrase without negator and y is that of the phrase with a negator.
detailed results of our model, we have the following statements:
First, there is no dot at the up-right and bottomleft blocks, indicating that negators generally shift/convert very positive or very negative phrases to other polarities. Typical phrases include not very good, not too bad.
Second, the dots at the up-left and bottom-right respectively indicates the negation effects: changing negative to positive and positive to negative. Typical phrases include never seems hopelessly (up-left), no good scenes (bottom-right), not interesting (bottom-right), etc. There are also some positive/negative phrases shifting to neutral sentiment such as not so good, and not too bad.
Last, the dots located at the center indicate that neutral phrases maintain neutral sentiment with negators. Typical phrases include not at home, not here, where negators typically modify nonsentiment words.
5.5 The Effect of the Intensity Regularizer
To further reveal the linguistic role of intensity words, we perform experiments on the SST dataset, as illustrated in Figure 2. We show the matrix that indicates how the sentiment shifts after being modified by intensifiers. Each number in a cell (mij) indicates how many phrases are predicted with a sentiment label i but the prediction of the phrases with intensifiers changes to label j. For instance, the number 20 (m21) in the second matrix , means that there are 20 phrases predicted with a class of negative (-) but the prediction changes to very negative (- -) after being modified by intensifier “very”. Results in the first matrix show that, for intensifier “most”, there are 21/21/13/12 phrases whose sentiment is shifted after being modified by intensifiers, from negative
Figure 2: The sentiment shifting with intensifiers. The number in cell(mij) indicates howmany phrases are predicted with sentiment label i but the prediction of phrases with intensifiers changes to label j.
to very negative (eg. most irresponsible picture), positive to very positive (eg. most famous author), neutral to negative (eg. most plain), and neutral to positive (eg. most closely), respectively. There are also many phrases retaining the sentiment after being modified with intensifiers. Not surprisingly, for very positive/negative phrases, phrases modified by intensifiers still maintain the strong sentiment. For the left phrases, they fall into three categories: first, words modified by intensifiers are non-sentiment words, such as most of us, most part; second, intensifiers are not strong enough to shift sentiment, such as most complex (from neg. to neg.), most traditional (from pos. to pos.); third, our models fail to shift sentiment with intensifiers such as most vital, most resonant film.
6 Conclusion and Future Work
We present linguistically regularized LSTMs for sentence-level sentiment classification. The proposed models address the sentient shifting effect of sentiment, negation, and intensity words. Furthermore, our models are sequence LSTMs which do not depend on a parsing tree-structure and do not require expensive phrase-level annotation. Results show that our models are able to address the linguistic role of sentiment, negation, and intensity words. To preserve the simplicity of the proposed models, we do not consider the modification scope of negation and intensity words, though we partially address this issue by applying a minimization operartor (see Eq. 11, Eq. 14) and bi-directional LSTM. As future work, we plan to apply the linguistic regularizers to tree-LSTM to address the scope issue since the parsing tree is easier to indicate the modification scope explicitly.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Strengths:  - Innovative idea: sentiment through regularization - Experiments appear to be done well from a technical point of view - Useful in-depth analysis of the model  Weaknesses:  - Very close to distant supervision - Mostly poorly informed baselines  General Discussion:  This paper presents an extension of the vanilla LSTM model that incorporates sentiment information through regularization.  The introduction presents the key claims of the paper: Previous CNN approaches are bad when no phrase-level supervision is present. Phrase-level annotation is expensive. The contribution of this paper is instead a  simple model  using other linguistic resources.  The related work section provides a good review of sentiment literature. However, there is no mention of previous attempts at linguistic regularization (e.g., [YOG14]).  The explanation of the regularizers in section 4 is rather lengthy and repetitive. The listing on p. 3 could very well be merged with the respective subsection 4.1-4.4. Notation in this section is inconsistent and generally hard to follow. Most notably, p is sometimes used with a subscript and sometimes with a superscript.  The parameter   beta is never explicitly mentioned in the text. It is not entirely clear to me what constitutes a  position  t in the terminology of the paper. t is a parameter to the LSTM output, so it seems to be the index of a sentence. Thus, t-1 is the preceding sentence, and p_t is the prediction for this sentence. However, the description of the regularizers talks about preceding words, not sentences, but still uses. My assumption here is that p_t is actually overloaded and may either mean the sentiment of a sentence or a word. However, this should be made clearer in the text.  One dangerous issue in this paper is that the authors tread a fine line between regularization and distant supervision in their work. The problem here is that there are many other ways to integrate lexical information from about polarity, negation information, etc. into a model (e.g., by putting the information into the features). The authors compare against a re-run or re-implementation of Teng et al.'s NSCL model. Here, it would be important to know whether the authors used the same lexicons as in their own work. If this is not the case, the comparison is not fair. Also, I do not understand why the authors cannot run NSCL on the MR dataset when they have access to an implementation of the model. Would this not just be a matter of swapping the datasets? The remaining baselines do not appear to be using lexical information, which makes them rather poor. I would very much like to see a vanilla LSTM run where lexical information is simply appended to the word vectors.  The authors end the paper with some helpful analysis of the models. These experiments show that the model indeed learns intensification and negation to some extent. In these experiments, it would be interesting to know how the model behaves with out-of-vocabulary words (with respect to the lexicons). Does the model learn beyond memorization, and does generalization happen for words that the model has not seen in training? Minor remark here: the figures and tables are too small to be read in print.  The paper is mostly well-written apart from the points noted above.  It could benefit from some proofreading as there are some grammatical errors and typos left. In particular, the beginning of the abstract is hard to read.  Overall, the paper pursues a reasonable line of research. The largest potential issue I see is a somewhat shaky comparison to related work. This could be fixed by including some stronger baselines in the final model. For me, it would be crucial to establish whether comparability is given in the experiments, and I hope that the authors can shed some light on this in their response.  [YOG14] http:  www.aclweb.org anthology P14-1074  --------------  Update after author response  Thank you for clarifying the concerns about the experimental setup.   NSCL: I do now believe that the comparison is with Teng et al. is fair.  LSTM: Good to know that you did this. However, this is a crucial part of the paper. As it stands, the baselines are weak. Marginal improvement is still too vague, better would be an open comparison including a significance test.  OOV: I understand how the model is defined, but what is the effect on OOV words? This would make for a much more interesting additional experiment than the current regularization experiments.