Improving sentiment classification with task-specific data
Current state-of-the-art sentiment analysis techniques rely heavily on pre-trained word embeddings. However, the data used to train these embeddings normally comes from large, generic datasets, such as Wikipedia or GoogleNews, which may not include enough task-specific information to create reliable representations. This paper proposes a method to determine the subjectivity of a corpus using available tools and shows that word embeddings trained on task-specific corpora tend to outperform those trained on generic data. We then examine ways to combine information from generic and task-specific datasets and finally demonstrate that our method can work well for under-resourced languages.
1 Introduction
Sentiment analysis techniques requiring only word embeddings as features have proven to be successful (Socher et al., 2013; Tang et al., 2014; Iyyer et al., 2015). Research has also shown the importance of properly initializing the word embeddings, either through careful manipulation of the initialization of random vectors, or more commonly by pretraining the word embeddings (Kim, 2014). The content of the corpora used to train these word embeddings, however, has not been examined in depth.
Most research tends toward using larger and larger datasets in order to build high-quality word embeddings, with some studies using corpora on the magnitude of several billion tokens (Mikolov et al., 2013; Pennington et al., 2014). This amount of data, however, is not available in all languages, nor is it necessarily an optimal use of available re-
sources. In fact, there is evidence suggesting that tailoring a dataset for a task can achieve better results with less data (Suster et al., 2016; Barnes et al., 2016).
The subjectivity or polarity content of a corpus could provide a good clue to the quality of word embeddings created for the task of sentiment analysis. Pang and Lee (2004) showed that using a pipeline technique where they first classified subjectivity before classifying polarity, they were able to achieve better results than performing classification without filtering subjective sentences. Therefore, we attempt to use subjectivity as a metric for the appropriateness of a corpus for our task.
In this work we aim to:
• discover if small task-specific datasets can provide better representations of words than large generic datasets for the task of sentiment analysis
• quantify the amount of subjectivity in the data in order to predict the appropriateness of a dataset for the task of sentiment analysis.
• discover the best way to combine this information for our task, given a large generic dataset and a smaller domain-specific dataset.
• extract the subjective information as a way of approximating a smaller task-specific dataset, given only large generic datasets,
As far as we know, this is the first work on the effects of the subjectivity and task-specificity of a corpus on the creation of word embeddings for the task of sentiment analysis. Given the importance of initialization and the wide-spread use of sentiment analysis techniques using word embeddings, the results are of importance to the community.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
This paper is organized as follows. Section 2 discusses previous work on subjectivity, generic word embeddings and task-specific word embeddings. Section 3 outlines how to quantify the subjectivity of a corpus. Section 4 determines how subjectivity and task-specificity effects word embeddings, and how to combine information from different datasets in order to improve classification. In Section 5 we propose a method to extract task-specific information from generic corpora and, by using the techniques from Section 4.2, improve over our baseline. Finally, in Section 6 we show that this technique of concatenation representations works even better for underresourced languages.
2 Related Work
2.1 Subjectivity as a filter for polarity
Subjectivity can be a clear indicator of whether a sentence or phrase contains an opinion. In the case of movie or product reviews, this is especially relevant as there are often sentences that seemingly contain sentiment information, yet do not give an opinion towards any relevant aspect: for example, ”The main character tries to protect her good name.” Here, ’good’ gives no relevant sentiment information and could easily be found within a negative review.
Pang and Lee (2004) use a subjectivity classifier to extract relevant sentences before passing them on to a sentiment classifier. Reducing the amount of irrelevant or potentially misleading data presumably leads to an improvement in polarity classification.
This approach could potentially improve vector representations of words by providing only relevant examples to the word embedding algorithm.
2.2 Representing Words as Vectors
Representing words as feature vectors is an idea that has created interest for many years (Deerwester et al., 1990; Lund and Burgess, 1996). The simplest example is to represent a word as a onehot vector the length of the vocabulary, where all entries are zeros except for the index of the word, which is a 1. However, this simple technique does not allow for generalization, as the words ’dog’ and ’cat’ are as different as ’dog’ and ’dogs’.
Lower-dimensional dense vectors, often known as word embeddings, are able to avoid this problem to a degree. They do this by attempting to
force similar words to occupy a similar area in vector space, given an objective function. The most successful of these applications to date is the Skip Gram with Negative Sampling (SGNS) algorithm (Mikolov et al., 2013). This algorithm applies PMI factorization over an implicit word/feature matrix (Levy and Goldberg, 2014).
Although generic word embeddings are often used as features for sentiment classifiers (Socher et al., 2013; Kim, 2014), there are good reasons to believe that the generality of these vectors may lead to inferior results. Most state-of-the-art sentiment classifiers are based on deep neural networks. With these architectures it is rather difficult, if not impossible, to find the global optimum and therefore, we must usually be content to find a local optimum. This fact also means that they are sensitive to the initialization of their word embeddings. For this reason, sentiment classifiers initialized with random vectors consistently underperform those initialized with pretrained vectors (Kim, 2014; Iyyer et al., 2015).
2.3 Task-specific Word Embeddings
Task-specific word embeddings are a further development of word embeddings. The purpose is to provide a representation that helps a classifier, often by removing or adding task-specific information.
These vectors can be trained as normal and then retrofitted to the task (Yu and Dredze, 2014; Xu et al., 2014; Faruqui et al., 2015). The advantage of this technique is that any kind of word embedding can be used, as the vectors are modified post training. Faruqui et al. (2015) propose a method to incorporate information from semantic lexicons (WordNet, FrameNet, and ParaphraseNet) into word embeddings.
Task-specific word embeddings can also be trained in a supervised manner, using the labels from annotated corpora (Maas et al., 2011; Tang et al., 2014, 2016) or task-specific dictionaries (Gouws and Søgaard, 2015). For the inclusion of sentiment information in word vectors, this is the most common approach.
Maas et al. (2011) build a probabilistic model, similar to LDA (Blei et al., 2003), and attempt to model both word relationships and sentiment at the same time. However, their sentiment vectors performed worse than generic word vectors without the sentiment information.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Tang et al. (2014) attempt to create sentimentspecific word vectors by modifying the feedforward network architecture used by Collobert and Weston (2011) with a hinge-objective that is designed to maximize both the word representation and the model’s ability to classify polarity. They collected 10 million distantly supervised tweets by searching for positive and negative emoticons. They trained a sentiment classifier and achieved state-of-the-art results on the SemEval2013 shared task (Nakov et al., 2013).
Both of these techniques attempt to use large amounts of annotated data to extract sentiment information, but do not consider different sources of data or ways to combine them. However, it would be beneficial to examine more in depth the type of data that we use to create word vectors.
The work that most closely resembles our is that of Joshi et al. (2015). They explore the effects of using in-domain and generic datasets to create embeddings for e-commerce NER. Their work shows that training on smaller amounts of task-specific data is better than training on large amounts of generic data. Unlike our work, they do not look into ways of combining information from different sources.
3 Quantifying the amount of subjectivity in corpora
3.1 Datasets
Our first goal was to quantify the amount of subjective information contained in a corpus. We collected the following corpora, which we believe represent both generic datasets often used in the literature (Wikipedia, Europarl, Multi UN) as well as task-specific corpora (Amazon Movie Dataset). All corpora were sentence tokenized, word tokenized and lower-cased using Stanford Core NLP (Manning et al., 2014). We give the details of each corpus in the following subsections as well as in Table 1.
Wikipedia Corpus
After downloading a 2016 Wikipedia dump, we removed duplicates and markup using a freely available script1.
1http://attardi.github.io/wikiextractor/
English Europarl The English part of the Europarl v7 corpus2 (Koehn, 2005) is composed of around 2 million sentences from the European Parliament.
Multi UN MultiUN3 is a corpus extracted from the official documents of the United Nations (Eisele and Chen, 2010). We use the English side of the English-Spanish corpus. It contains around 11 million sentences.
Amazon Movie Corpus This dataset is composed of more than 7.5 million sentences taken from nearly 1.7 million reviews about movies (McAuley et al., 2015).
3.2 Method
In order to determine the amount of subjectivity contained in each corpus, we ran OpinionFinder on each of them. This is a system that was designed for subjectivity and polarity detection (Wilson et al., 2005). It has a high-precision and highrecall subjectivity classifier. The high-precision classifier uses a rule-based method, looking for well-established lexical clues. The high-recall classifier is a naive bayes classifier trained on the MPQA corpus. After running OpinionFinder on each corpus, we retrieved the results for highprecision subjectivity and high-recall subjectivity. In order to quantify the amount of subjectivity, we took the number of sentences reported as subjective and divide them by the total number of sentences in the corpus, (see Formula 1).
Subjectivity Score = |subjective sentences| |total sentences|
(1)
This gives us an idea of the subjectivity content of one corpus compared to the others, although it is not a precise measurement. We did the same with high-precision and high-recall to get two scores for each corpus (see Table 1). As we expected, the Wikipedia corpus has the lowest scores for subjectivity. This is likely because of its encyclopedic format. The corpora with the highest subjectivity scores are the Amazon (high precision) and Europarl (high recall) corpora.
2http://www.statmt.org/europarl 3http://opus.lingfil.uu.se/
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Wiki Multi UN Europarl Amazon High recall subj. 14.8% 33% 49% 41.5% High precision subj. 2.8% 5.1% 6% 9.3% # Sentences 118,859,889 11,350,967 1,965,734 7,540,838 # Tokens 2,099,860,107 322,056,452 54,474,544 162,390,631 Avg. Sent Length 17.7 28.4 27.7 21.5
Table 1: Statistics of corpora used to train word embeddings.
4 Experiments
4.1 Baseline experiment
We created 50- 100- and 300-dimensional word vectors for each corpus using the Skip-gram with Negative Sampling algorithm (Mikolov et al., 2013) with a window of 10 words, and 5 negative samples.
We tested the word vectors on the Rotten Tomatoes dataset (Pang and Lee, 2005), which contains 10662 annotated sentences. We train on 7996, leave 1066 as a development set and test on 1600.
In order to see how well these representations work on a different domain, we also tested them on the English OpeNER sentiment corpus (Agerri et al., 2013). Specifically, we take the subcorpus from the hotel domain. It contains 3709 opinion phrases annotated for opinion holder, opinion target and four levels of sentiment (Strong Positive, Positive, Negative and Strong Negative). Each training example is a tuple of the opinion holder, opinion target and sentiment phrase, such as (”we”, ”hotel”, ”didn’t like at all”). We train on 2780, leave 186 as a development set and test on 743. This dataset has a class imbalance, which means that macro f1 is a more appropriate metric.
Following the work of Iyyer et al. (2015), we use a Deep Averaging Network to perform sentiment classification. This neural bag-of-words model does not take the order of the words into consideration, but has proven to give results similar to more sophisticated approaches. Our datasets do not allow us to train on labeled phrases. Also the small size of the training data, especially in the OpeNER dataset, means that updating the word vectors while training the classifier actually hurts classification when using our task-specific vectors. Therefore, we do not update while training.
For each sentence or phrase in the datasets, we create a training example. A training example is the average of the word embeddings for each word in the sentence or phrase. If a word is not found in
our word embedding model, it is replaced with a generic ’unknown’ vector, which is randomly initialized. The results are shown in Figures 1 and 2.
Figure 1: Accuracy of sentiment classifiers on Rotten Tomatoes dataset
As we can see, the embeddings trained on the much smaller Amazon corpus give consistently better results than the Wikipedia corpus on the RT dataset. However, the vectors trained on Wikipedia perform quite well and even outperform their Amazon trained counterparts on the OpeNER dataset. This is likely due to the fact that the domain for the OpeNER dataset is different (Hotel reviews) compared to the RT dataset (Movie Reviews).
4.2 Combining information from different sources
Given that vectors trained on task-specific data regularly outperform those trained on much larger generic datasets, we should ask ourselves if it is possible to combine both of these datasets to get even better results. We compare three methods:
1. Appending the task-specific dataset to the generic dataset and training one representation
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Figure 2: Accuracy of sentiment classifiers on OpeNER dataset
2. Splicing the task-specific dataset into the generic dataset and training one representation
3. Training two separate representations and combining them through vector concatenation.
Appending involves simply adding the taskspecific Amazon Movie corpus to the Wikpedia corpus and training a single set of vectors on this data. This should improve the representations of those words that are important for the task, as they will receive more positive examples while training the vectors.
Splicing is similar to appending, but instead of adding the Amazon sentences, we splice them into the Wikipedia corpus. We alternate sentences from the two corpora until we have exhausted all of the Amazon sentences, after which we add the remaining Wikipedia sentences. As above, we train a single set of vectors.
For concatenation, we train one set of vectors on Wikipedia and a second on the Amazon Movie Corpus separately. Given two m dimensional vectors, we concatenate them, resulting in an m +m dimensional vector, in order to provide both views to the sentiment classifier. For a fair comparison, we concatenate vectors that are half as large as those of the other techniques. This technique may provide better representations for words that often appear in one corpus but not in the other.
Note that the embeddings in these three techniques were trained on exactly the same data, the only difference being the method of combining
this data. Our baseline vectors are trained only on the Wikipedia corpus. We also test the 50 dimensional sentiment embeddings (SSWEu) provided by Tang et. al (2014) (see Section 2.3). The results are shown in Table 2.
We can see that including task-specific data with any of the techniques tends to improve over the baseline. This in itself is not surprising. However, it is important to notice that the concatenation of two vectors trained on the two datasets separately always improve the macro f1 score.
4.3 How much task-specific data is needed in order to see benefits
Given that we can combine generic and taskspecific data to improve our feature space, we would like to know how much task-specific data is necessary to see these improvements. Therefore, we perform experiments on the RT dataset with increasing amounts of task-specific data. The results are shown in Figure 3.
As we can see from the results, we see improvements over the baseline after 10-12 million tokens. This could potentially benefit low-resource languages which often have access only to Wikipedia pages and little task-specific data.
5 Extracting task-specific information from generic datasets
Finally, we ask if there is a way to approximate this task-specific data given only large, generic corpora. This again is a situation that is very common among under-resourced languages. There-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
RT OpeNER 50 100 200 300 50 100 200 300
Wikipedia Baseline acc 66.2 71.9 71.4 71.3 77.6 78.2 82.1 83.6 f1 59.8 72.4 .67.4 .67.5 37.2 36.7 39.8 40.6 SSWEu acc 66.3 – – – 81 – – – f1 56.5 – – – 39.4 – – – Amazon acc 73.8 76.3 76.7 76.9 73.5 73.8 77.8 76.9 f1 69.9 74.7 74.6 75.3 34.9 33.6 37.4 38 Appended acc 69.4 73.8 74.5 74.3 77.6 76.5 82.7 78.4 f1 65.1 75 72.5 72.8 36.9 35.8 40.5 36.4 Spliced acc 70.8 71.2 73.1 73.6 77.4 73.9 84.6 82.7 f1 62.5 73.4 71 70 37.2 32.9 41.4 39.9 Amazon + Wiki acc 74.4 75.2 75.9 75.9 64.2 65.7 69.4 72.3 f1 72.5 75 78 77.7 57.8 62.4 63.8 68
Table 2: Accuracy and F1 of sentiment classifier with embeddings trained on Wikipedia and Amazon corpora with different techniques. For the OpeNER dataset, accuracy is the macro average. Bold indicates the highest performance and underline indicates those that beat the generic baseline.
fore, we extract the sentences labeled as subjective from the Wikipedia, Europarl and Multi UN corpora, referred to from here on as subj-Wikipedia, subj-Europarl, subj-MultiUN. We train word embeddings on each of these and test them as in Section 4.2. We also report macro-averaged f1 scores. Results are shown in Table 3.
We can see that by filtering the corpora for subjectivity, we lose a degree of accuracy, but gain in f1. The poor f1 scores of the Wikipedia baseline on the OpeNER dataset are due to the fact that the classifier nearly always chooses the two majority classes and completely ignores the others. Including more subjective information seems to ameliorate this situation. The improvements in f1 also coincide with the subjectivity level predicted by our method. This suggests that it is possible to predict the amount of improvement that you can gain by extracting subjective data from a generic dataset.
5.1 Lexical Overlap
Given that the classification error is still relatively large, our hypothesis was that many words which are important for classification are not found in the corpora used to train the word embeddings. In order to test this possibility, we found the lexical overlap in vocabulary between the corpora used to train the word embeddings and the RT and OpeNER corpora. Let Vtr be the set of words in a training corpus and Vtest be the set of words in each test corpus test.
LexicalOverlap = |Vtest| ∩ |Vtr| |Vtest|
(2)
If the lexical overlap of the corpora used to train our representations (Wikipedia, Europarl, Multiun) and the datasets we use to test our classifier (RT, OpeNER) is small, this will negatively affect the results of the classifier, as it will lack necessary information during both training and testing. We also test for lexical diversity of the filtered subjectivity training corpora as a measure of vocabulary. We use the Average Type Token Ratio (ATTR) as a measure. Let k be a corpus, nk be a subset of n tokens of k, Vn = the vocabulary of n, and Tn = the tokens of n.
Average Type Token Ratio = 1
n ∗ N∑ n=1 |Vn| |Tn| (3)
We set n to 1000. The results are shown in Table 4.
As we can see in Table 4, the number of words that do not have a vector representation in the full Wikipedia model is much smaller than all other models. Therefore, the Wikipedia word embeddings have a larger vocabulary yet they do not always have a representation that is helpful for the task. The more subjective datasets may lack some vocabulary, but it seems that the representations that they have are able to overcome the sparsity.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
RT OpeNER 50 100 200 300 50 100 200 300
Wikipedia Baseline acc 66.2 71.9 71.4 71.3 77.6 78.2 82.1 83.6 f1 59.8 72.4 67.4 67.5 37.2 36.7 39.8 40.6 subj-Wiki + Wiki acc 66.9 69.1 70.3 72.4 62.1 67.6 67.8 70.3 f1 67.5 71.8 74.6 75.5 51.2 58.3 57.9 60.8 subj-Multiun + Wiki acc 68.4 69.6 69.3 73.4 65.1 68.3 68.2 71 f1 69 73.2 74 75.6 60.2 60.7 57.7 62.8 subj-Europarl + Wiki acc 68.2 68.4 71.3 74.1 63.7 68.3 69.6 71.7 f1 69.7 73.7 75.5 76.2 56.9 59.5 66.3 63.4
Table 3: Accuracy and F1 scores of sentiment classifier trained on concatenation of filtered subjectivity vectors and wikipedia vectors. Bold indicates the highest performance and underline indicates those that beat the Wikipedia baseline.
Wiki Amazon subj-Wiki subj-MulUN subj-Euro # tokens 1.8B˜ 162M˜ 24M˜ 43M˜ 33M˜ ATTR 41.5 39.9 44.3 36.7 39.4 Overlap RT-Train 83.7 61.6 73.3 70.9 68.6 Overlap RT-Test 87.5 69.9 76.7 75.3 74 Overlap OpeNER-Train 89.1 70.3 81.3 78.1 75 Overlap OpeNER-Test 96 89.1 92 90 90
Table 4: Average Type/Token ratio (ATTR) and overlap with test vocabulary in each training corpus.
5.2 Types of missing words
After seeing the number of words which were missing, we decided to study these qualitatively. We collected a list of the words that appear in the RT and OpeNER datasets which were not found in the embeddings.
A revision of each part of speech category revealed that the nouns not found in the embeddings models were often misspellings of more common nouns (fantasi, alientation) or creative use of language (vidgame, dateflick, splatterfests, artsploitation, witlessness, drippiness, naturedness, diciness, gooeyness, baaaaaaaad), both of which are important for categorization.
Even more important are the adjectives (unrecommendable, unsuspenseful, uncinematic, unslick, unfakeable, snazzy) and adverbs (heartwarmingly, dullingly, repellently, forgettably, uncharismatically, appallingly) which are not found in these models. These are words that often carry the most sentiment information in a sentence. The number of words lost due to filtering for subjectivity also seems to correlate with the performance of each model, as the subj-Europarl model lost the least number of wordforms and performed the best among the filtered corpora.
6 Experiments on under-resourced languages
In order to confirm our theory that this technique can improve representations for under-resourced languages, we perform the same concatenation experiment as in Section 4.2 but with Catalan data. The Catalan Wikipedia used in this experiment was from a wikipedia dump in January, 2016 and contains 182 million tokens. In order to gather task-specific data, we scraped 21,778 reviews in Catalan from www.booking.com which contain a total 4,161,719 tokens. We preprocessed both datasets in a similar fashion (lowercased, sentence tokenized, tokenized) using Freeling (Padró and Stanilovsky, 2012).
We test this on the Catalan Aspect-level Sentiment Dataset. This dataset was compiled from 879 Booking.com reviews collected in December of 2016. Three annotators annotated opinion holders, opinion targets, and four levels of sentiment (Strong Positive, Positive, Negative and Strong Negative). Statistics are shown in Table 5.
The results on the Catalan dataset (see Table 6) suggest that under-resourced languages benefit more from concatenating vectors trained separately than languages such as English. This is
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Training Dev Test Num. phrases 922 62 248 Strong Pos 27.3% 25.8% 27.4% Pos 34.2% 33.9% 34.3% Neg 33.3% 33.9% 33% Strong Neg 5.2% 6.5% 5.2%
Table 5: Statistics of the Catalan Aspect-level Sentiment Dataset.
50 100 200 300 Wikipedia 44.9 61 72.2 75.4 Booking 55.61 62 72.7 69.5 Appended 61.4 60.4 72.2 70.6 Spliced 49.2 62 70.58 75.4 Booking + Wiki 59.4 62.6 73.3 75.4
Table 6: Macro average of classification scores on the Catalan Aspect-level Sentiment Dataset (4 class). Bold indicates the highest performance and underline indicates those that beat the generic baseline.
most likely due to the size and breadth of the English Wikipedia corpus. Therefore, for languages that do not have large, broad Wikipedia, we recommend training separate vectors and concatenating them afterwards.
7 Conclusion and Future Work
In this paper, we have explored several methods for improving sentiment analysis using word embeddings. First, we demonstrated that the subjectivity of a corpus is a viable method to determine the usefulness of a corpus for the task of sentiment analysis. We then show that concatenation of vectors trained on generic and taskspecific datasets outperforms a single representation trained on both datasets. We also showed that it is possible to approximate task-specific data by extracting subjective portions from generic corpora. Finally, we showed that it is possible to use this technique to improve sentiment classification for under-resourced languages, even if there is little task-specific data available.
Instead of relying on subjectivity as metric, one could naturally use polarity to determine the usefulness of a passage for training embeddings for a sentiment related task. By removing neutral passages and keeping only those with positive or negative sentiment, it may be possible to achieve similar results to ours. We plan to explore this possi-
bility in the future. We also believe that these techniques could improve results on tasks other than sentiment analysis, but we leave this for future work.

This paper compares different ways of inducing embeddings for the task of polarity classification. The authors focus on different types of corpora and find that not necessarily the largest corpus provides the most appropriate embeddings for their particular task but it is more effective to consider a corpus (or subcorpus) in which a higher concentration of subjective content can be found. The latter type of data are also referred to as  task-specific data . Moreover, the authors compare different embeddings that combine information from  task-specific  corpora and generic corpora. A combination outperforms embeddings just drawn from a single corpus. This combination is not only evaluated on English but also on a less resourced language (i.e. Catalan).  - Strengths: The paper addresses an important aspect of sentiment analysis, namely how to appropriately induce embeddings for training supervised classifers for polarity classification. The paper is well-structured and well-written. The major claims made by the authors are sufficiently supported by their experiments.  - Weaknesses: The outcome of the experiments is very predictable. The methods that are employed are very simple and ad-hoc. I found hardly any new idea in that paper. Neither are there any significant lessons that the reader learns about embeddings or sentiment analysis. The main idea (i.e. focusing on more task-specific data for training more accurate embeddings) was already published in the context of named-entity recognition by Joshi et al. (2015). The additions made in this paper are very incremental in nature.  I find some of the experiments inconclusive as (apparently) no statistical signficance testing between different classifiers has been carried out. In Tables 2, 3 and 6, various classifier configurations produce very similar scores. In such cases, only statistical signficance testing can really give a proper indication whether these difference are meaningful. For instance, in Table 3 on the left half reporting results on RT, one may wonder whether there is a significant difference between  Wikipedia Baseline  and any of the combinations. Furthermore, one doubts whether there is any signficant difference between the different combinations (i.e. either using  subj-Wiki ,  subj-Multiun  or  subj-Europarl ) in that table. The improvement by focusing on subjective subsets is plausible in general. However, I wonder whether in real life, in particular, a situation in which resources are sparse this is very helpful. Doing a pre-selection with OpinionFinder is some pre-processing step which will not be possible in most languages other than English. There are no equivalent tools or fine-grained datasets on which such functionality could be learnt. The fact that in the experiments for Catalan, this information is not considered proves that.   Minor details:  - lines 329-334: The discussion of this dataset is confusing. I thought the task is plain polarity classification but the authors here also refer to  opinion holder  and  opinion targets . If these information are not relevant to the experiments carried out in this paper, then they should not be mentioned here.  - lines 431-437: The variation of  splicing  that the authors explain is not very well motivated. First, why do we need this? In how far should this be more effective than simple  appending ?  - lines 521-522: How is the subjective information isolated for these configurations? I assume the authors here again employ OpinionFinder? However, there is no explicit mention of this here.  - lines 580-588: The definitions of variables do not properly match the formula (i.e. Equation 3). I do not find n_k in Equation 3.  - lines 689-695: Similar to lines 329-334 it is unclear what precise task is carried out. Do the authors take opinion holders and targets in consideration?  ***AFTER AUTHORS' RESPONSE*** Thank you very much for these clarifying remarks. I do not follow your explanations regarding the incorporation of opinion holders and targets, though.  Overall, I will not change my scores since I think that this work lacks sufficient novelty (the things the authors raised in their response are just insufficient to me). This submission is too incremental in nature.


A Multigraph-based Model for Overlapping Entity Recognition
In this paper, we propose a new model that is capable of recognizing overlapping entities based on multigraphs, as opposed to simple graphs commonly used in graphical models for structured prediction. Through extensive experiments on standard datasets containing overlapping and non-overlapping entities, we demonstrate that our model outperforms previous models. We also present some analysis on the differences between our model and the previous models and discuss the possible implications of the differences. To the best of our knowledge, this is the first structured prediction model utilizing multigraphs to predict overlapping structures.
1 Introduction
Named entity recognition (NER), or in general the task of recognizing entities in a text, has been a research topic for many years (McCallum and Li, 2003; Nadeau and Sekine, 2007; Ratinov and Roth, 2009). However, as previously noted by Finkel and Manning (2009), many previous works ignored overlapping entities, although they are quite common. For example, the location entity China appears within the organization entity Bank of China. In practice, overlapping entities have been found in many existing datasets, including ACE (Doddington et al., 2004), GENIA (Kim et al., 2003), and in a clinical text dataset (Suominen et al., 2013). Figure 1 illustrates some examples of overlapping entities adapted from existing datasets.
Solving this task is non-trivial, as the typical way of modeling entity recognition as a sequence prediction problem (e.g., using linear-chain CRF
which utilizes simple graphs) has difficulties handling overlapping entities (Alex et al., 2007). Finkel and Manning (2009) proposed to use a treebased constituency parsing model to handle nested entities with a time complexity that is cubic in n for its inference procedure with n being the number of words in the sentence. This complexity was later improved by Lu and Roth (2015) with a hypergraph-based model, which shows a time complexity that is linear in n.
In this work, as opposed to using simple graphs or hypergraphs, we introduce a novel multigraphbased model to tackle the problem of overlapping entity extraction. We show it is possible to assign explicit semantics to different edges connecting the same pair of nodes when representing structures, leading to the novel multigraph representations that can be used to represent overlapping structures. We present the training and inference procedures over such a novel representation. To the best of our knowledge, this is the first structured prediction model utilizing multigraphs to predict overlapping structures.
In this paper we make the following major contributions: • We propose a novel multigraph-based model
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
for predicting overlapping entities. • Empirically, we show that our model is able
to achieve higher F1-scores compared to previous models in multiple datasets. • Theoretically, we show that unlike a previ-
ous state-of-the-art model that we compare against, our model does not present the spurious structures issue in its inference procedure and is therefore nondeficient. On the other hand, it still maintains the same inference time complexity as the previous model.
We also believe our proposed multigraph-based structured prediction framework can be used to solve other problems involving overlapping structures, and we hope this work can inspire further research along such a direction. We will make our system and code available for research purposes.
2 Related Work
Named entity recognition (NER) has been a research focus for quite some time. It is normally regarded as chunking task similar to base NP chunking (Kudo and Matsumoto, 2001; Shen and Sarkar, 2005), and hence the entities are usually represented in a similar way, using BILOU (Beginning of an entity, In the middle of an entity, Last token of an entity, Outside of any entity, Unit-length entity) or the simpler BIO annotation scheme (Ratinov and Roth, 2009). Some previous works have focused on getting the best representation of the chunks (Sang and Veenstra, 1999; Loper, 2007) and Ratinov and Roth (2009) showed that between BIO and BILOU scheme, the latter is the better approach for recognizing non-overlapping entities. As a chunking task, it is commonly modeled using sequence labeling models, such as the linear-chain CRF (Lafferty et al., 2001), which has time complexityO(nT 2) with n being the number of words in the sentence and T the number of entity types.
On the task of recognizing entities that may overlap with one another, one of the earliest works that attempted to regard this task as a structured prediction task was McDonald et al. (2005). They represented entities as top-k predictions with positive score from a structured multilabel classification model. This model resembles structured SVM (Tsochantaridis et al., 2005) in terms of the objective function used. Since they set k = n, their model has a time complexity of O(n3T ).
Alex et al. (2007) proposed a cascading approach using multiple sequence labeling models,
each handling a subset of all the possible entity types, where the models which come later in the pipeline have access to the predictions of the models earlier in the pipeline. For each entity type, they used a standard linear-chain CRF model to predict entities, resulting in the time complexity of roughly O(nT ) depending on how the pipeline is designed. However, they noted that “it involves extensive amounts of experimentation to determine the best model combination”, due to the many possible ways to arrange the order of the pipeline. Their model only handles overlapping entities of different types.
Finkel and Manning (2009) later proposed a constituency parser to handle nested entities by converting each sentence as a tree, and each entity is represented as one of the subtrees. Their model has the standard time complexity for a binary parser: O(n3 |G|), where |G| is the size of the grammar that is proportional to T in the best case, and T 3 in the worst case. They showed that their model is able to outperform a semi-CRF baseline. By design, their model cannot handle crossing entities (overlapping entities which are not nested).
More recently, Lu and Roth (2015) proposed a hypergraph-based model called mention hypergraphs that is able to handle overlapping entities, yet it retains the linear time complexity O(nT ). The model was shown to achieve competitive results compared to previous models on standard datasets. As we will be making extensive comparisons against this pervious state-of-the-art model, we will briefly discuss this approach in the next section.
3 Mention Hypergraph
In the mention hypergraph model of Lu and Roth (2015), nodes and directed hyperedges1 are used together to encode entities and their combinations. The following 5 types of states are used at the position k of a sentence: • Ak denotes all entities starting at k or later • Ek denotes all entities starting at k • Tkt denotes all entities of type t starting at k • Ikt denotes all entities of type t covering k • X denotes the end of an entity (the leaf node) Different hyperedges connecting these nodes are used to represent how the semantics of a node is composed from those of its child nodes.
1For brevity, in this paper we may also use edge to refer to hyperedge in some discussions.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
T
O O O O
I I I I
X
the human TCF-1 proteinX [ -[ ]- ]
X
S
CS EC
E
T T T T
I I I I
X X X X
X X X X
the [human [TCF-1]2 protein]1
Figure 2: (Top) Our multigraph model representing overlapping entities, with the entity separators shown in red. (Bottom) The corresponding mention hypergraph (A- and E-nodes not shown).
Specifically, each Ak is connected to Ak+1 and Ek through the hyperedge Ak → (Ak+1,Ek), denoting the fact that the set of entities that start at k or later is the union of the set of entities that start at k+1 or later and the set of entities that start at k. Each Ek is connected to Tk1,T k 2, . . . ,T k T through a hyperedge, denoting the fact that the entities that start at k must be one of the T types. Each Tkt is connected to Ikt through an edge (denoting there is an entity of type t that starts at the k-th token) and to X through another edge (denoting there are no entities of type t that start at the k-th token). Each Ikt is connected to I k+1 t (denoting there is an entity continuing to the next token), to X (denoting there is an entity ending here), or to both (with a single hyperedge, denoting the two cases above occur at the same time, a case of overlapping entities).
In this mention hypergraph, each possible entity is represented as a path from a T-node to the X-node through a sequence of I-nodes (each denoting the words which are part of the entity), and the set of all entities present in a sentence forms a sub-hypergraph of the overall hypergraph.
The bottom part of Figure 2 shows how the mention hypergraph represents the two entities in the phrase “the human TCF-1 protein”, which are “TCF-1” and “human TCF-1 protein” 2.
4 Multigraph-based Model
We now describe our proposed multigraph-based model for recognizing overlapping entities. In our
2Note that the word “protein” does not form an entity, due to the missing edge between the corresponding T node and I node.
C CS EC ECS
model, we assign for each token two states for each entity type, representing whether the token is part of an entity of a certain type. Formally, we define the following three types of nodes for each token: • Tt denotes the set of entities of type t • Ikt denotes that the k-th token is part of an
entity of type t • Okt denotes that the k-th token is not part of
any entities of type t To represent the distinct roles a token can take, we consider the incoming edge and outgoing edge of that state. Unlike previous approaches, we assign explicit semantics to the edges in our multigraph-based model, which we call entity separators. We consider the 8 possible types of entity separators based on the combination of the following three cases: 1. An entity is starting at the next token (S) 2. An entity is ending at the current token (E) 3. An entity is continuing to the next token (C)
For each token, the possible combinations of cases are as follows: ECS, EC, CS, C, ES, E, S, and X, where Xmeans none of the three cases applies. For example, the separator EC means there is one entity ending at the current token and another entity (overlapping) continuing to the next token. See Figure 3 for an illustration of these separators.
Next we define the edges between the states according to the 8 possible entity separators between adjacent tokens. Each entity separator is mapped to an edge connecting one state in the current position to another state in the next position depending on whether the separator defines current and next words as part of an entity, so in total we have 8 edges between two positions in the model. Some entity separators may connect the same two states, for example, the ES and C separator both connect Ikt to I k+1 t since in both cases the current token and the next token are part of an entity. In those cases, we simply define multiple edges between the pair of states, hence the multigraph aspect of
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
R
T0
T1
O0 O0 O0 O0
O1 O1 O1 O1
I0 I0 I0 I0
I1 I1 I1 I1
X
X
S
X
S
E ES C CS EC ECS
X
S
E ES C CS EC ECS
X
S
E ES C CS EC ECS
X
E
Figure 4: Example multigraph used in our model.
our model. The first I- and O-nodes in the sentence are connected to the T-node of the corresponding entity type, and the last I- and O-nodes are connected to the unique leaf node X. Finally, a designated root node R is added and is connected to all T-nodes by a single hyperedge to complete the multigraph structure. Figure 4 shows an example of our multigraph model considering 2 entity types and 4 input tokens. Note that the edges in our multigraph representations are directed, with nodes on the left serving as parents to the nodes on the right. Such directed edges will be helpful when performing inference, to be discussed in the next section.
The top part of Figure 2 shows how we can model the two entities “human TCF-1 protein” and “TCF-1” in the phrase “the human TCF-1 protein”. Note how each edge maps to a distinct entity separator visualized in the text. Note how the roles of T-I edge and I-I edge in mention hypergraph are combined in our model using the single edge connecting the two I nodes for the token “human” and “TCF-1” to represent the entity separator CS.
4.1 Training, Inference and Decoding
We follow the log-linear approach to define our model, using the following regularized loglikelihood in training dataD as our objective function:
LD(w)= ∑
(x,y)∈D [ ∑ e∈y w · f(e)− logZw(x) ] −λ||w||2
(1)
where (x,y) is a training instance consisting of the sentence x and the correct output y, w is the weight vector, f(e) is the feature vector defined over the edge e, Zw(x) is the normalization term, and λ is the l2-regularization parameter. The objective function is then optimized until convergence using L-BFGS (Liu and Nocedal, 1989).
We note the mention hypergraph model also defines the objective in a similar manner. For both models, the inference is done based on a generalized inside-outside algorithm. Both models involve directed structures, on top of which the inference algorithm first calculates the inside score for each node from the leaf node to root, and then the outside score from the root to the leaf node, in very much the same way as how inference is done in a classic graphical model. Specifically, for our multigraph-based model, the inside scores are calculated using a bottom-up (right-to-left) dynamic programming procedure, where we calculate the inside score at each node by summing up the scores associated with each path connecting the current node to one of its child nodes. Each such path score is in turn defined as the product of the inside score stored in that child node and the score defined over the edge connecting them. The computation of the outside scores can be done in an analogous manner from left to right. It can be verified that the time complexity of such an inference procedure for our model is O(nT ), which is the same as the mention hypergraph model.
During decoding, we perform MPE inference using a max-product procedure that is analogous to how the Viterbi decoding algorithm is used in conventional tree-structured graphical models to find out from the overall multigraph the highestscoring subgraph, from which we extract entities3
5 Model Analysis
In this section we make two analytical comparisons between our model and the previous models.
5.1 State-based vs Edge-based Paradigm
Based on how previous works model the role of each token in defining the predicted entity spans, we can classify the previous works into two paradigms, namely state-based paradigm and edge-based paradigm, which we detail below.
State-based Paradigm In conventional graph-based models for structured prediction such as linear-chain conditional random fields (Lafferty et al., 2001) and tree-based models (Finkel and Manning, 2009), typically nodes
3We note that both models involve interpretation of the output structures when recovering the entities. We implemented the same interpretation process as that was done in the mention hypergraph model, and we resolved ambiguous structures by considering them as nested entities instead of crossing entities.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
B B B B
I I I I
O O O O
R X
the [human TCF-1 protein]1
Figure 5: A linear-chain CRF model representing an entity in BIO scheme.
and edges are used together to encode structured output information. A linear-chain CRF model is shown in Figure 5 to model the protein entity “human TCF-1 protein” using the BIO scheme. In general, given an input sentence, a possible output can be represented using a linear-chain structure that involves states (nodes) and their connections (edges). One could observe that, in this case, given the state at each position in the structure, we already have enough information to interpret the output, without relying on the edge information. Similarly, this is also true for tree-based models. This leads to the following definition of what we call the state-based paradigm – the approach where the set of states alone can be used to interpret the predicted entities.
We observe that the majority of previous works fall into this paradigm (McDonald et al., 2005; Alex et al., 2007; Finkel and Manning, 2009; Tang et al., 2013).
Edge-based Paradigm Modeling the roles each token can take as states is, however, not the only possible paradigm. In the mention hypergraph model by Lu and Roth (2015), they used only a small number of possible states for each token. Their approach does not model the roles that each token can take directly using the states only. In fact, in addition to states, they used edges (or hyperedges, and their combinations) to capture the complex combination of roles each token can take. This leads to an alternative paradigm called edge-based paradigm: the approach where the set of states together with the edges are used to define the predicted entities.
To see the importance of edges in the edgebased paradigm, we show a simple illustrative example based on the hypergraph model of Lu and Roth (2015) with two words in Figure 6. The left graph and the right graph have exactly the same states. However, due to different edges (or hyperedges), these two graphs correspond to different entity combinations. Specifically, the left graph captures the set of non-overlapping entities {w1,
w2}, whereas the right graph captures the the following three entities: {w1, w2, w1 w2} with the third entity being overlapping with the first two.
It is easy to see that our multigraph-based model falls into this paradigm. Since there are multiple edges between the same pair of states, the combination of states alone do not define the roles of the tokens, and semantics are clearly assigned to the edges.
One might notice that there exists some equivalence between the two paradigms. For example, in our multigraph-based model, we can represent each edge (separator) as a separate state (such as a state named ES or CS) and assign such states to the gaps between two adjacent words using a linear-chain sequence labeling model, arriving at a “state-based” model. While this is feasible, we note that such an approach, however, will typically result in modeling unnecessary dependencies between the adjacent “states” (such as the dependencies between ES and CS, under the simple firstorder assumption), leading to additional model complexity and computation overhead. Alternative approaches such as using nodes to replace the edges such as ES in our representation between adjacent I nodes are also possible. However, such approaches essentially exploit ad-hoc representations derived from the multigraph representation, making the model significantly less intuitive and less concise.
Since the edge-based paradigm is relatively unexplored compared to the state-based paradigm, we hope that this discussion can ignite further research in this direction.
5.2 Spurious Structures
In previous section we see that our model falls into the same edge-based paradigm as the mention hypergraph model. One might notice that our model is similar to the mention hypergraph model, in the sense that the edges in our model represent combinations of multiple edges in the mention hypergraph model. Since the objective function is de-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
ACE2004 ACE2005 GENIA Train (%) Dev (%) Test (%) Train (%) Dev (%) Test (%) Train (%) Dev (%) Test (%)
# sentence 6,799 829 879 7,336 958 1,047 14,836 1,855 1,855 w/ o.l. 2,685 (39) 293 (35) 373 (42) 2,686 (37) 341 (36) 330 (32) 3,199 (22) 366 (20) 448 (24) # entities 22,207 2,511 3,031 24,687 3,217 3,027 46,473 5,014 5,600 o.l. 10,170 (46) 1,091 (43) 1,418 (47) 9,937 (40) 1,192 (37) 1,184 (39) 8,337 (18) 915 (18) 1,217 (22)
Table 1: Statistics of the datasets used in this experiments. w/ o.l.: sentences containing overlapping entities; o.l.: overlapping/nested entities.
fined using features over the edges, this raises the question: are the two models actually optimizing the same objective function?
To answer this question, let us look into the normalization term Zw(x) calculated by each model using the inside-outside algorithm. Let β(m) denote the inside score of the node m. Recall that in the inside-outside algorithm, the inside score of a node m represents the sum of the scores of all paths from m to the leaf node. Since in mention hypergraph a path from a T-node to the leaf node represents an entity, β(Tkt ) represents the scores of all entities starting at position k of type t. Similarly, β(Ek) represents the scores of all entities starting at position k of any type. Now, since β(Ak) is calculated as the product between β(Ak+1) and β(Ek), it represents the score of all combinations of entities starting at k or later. Therefore, the normalization term Zw(x) = β(A0) represents the score of all combinations of entities in the sentence.
Consider a sentence with three words w0, w1, w2. The normalization term includes the scores of both these distinct entity combinations: 1. Entities w0w1 and w1w2 (crossing entities) 2. Entities w0w1w2 and w1 (nested entities)
However, as also noted by Lu and Roth (2015)4, these two combinations share the same subgraph in the model.5 On the other hand, in the mention hypergraph model, only one of the many combinations would be considered in the first term of the objective of equation 1 – ∑ e∈y w · f(e), whose score can be calculated efficiently using the inside algorithm. The other structures that share the same subgraph will become what we call spurious structures – the structures that would never be predicted by the model. Also note that the score of each of these distinct interpretation of the same sub-hypergraph might differ from each other. Refer to the supplementary material for more details. We call a model with normalization term Zw(x)
4See Figure 3 in Section 3.2 in their paper. 5In fact, there are 7 distinct entity combinations which
have the same subgraph.
that contains spurious structures a deficient model. In contrast, we can observe that, unlike the mention hypergraph model, our model is nondeficient – the normalization term Zw(x) calculated by our multigraph-based model does not contain spurious structures. This can be verified by the fact that the inside score of each node in our multigraph is calculated as the sum of the scores associated with each path following that node.
6 Experiments
6.1 Datasets
To assess our model’s capability in recognizing overlapping entities and make comparisons with previous models, we looked at datasets where overlapping entities are explicitly annotated. Specifically, we looked at the following standard datasets which were used in several previous works (Finkel and Manning, 2009; Lu and Roth, 2015): ACE2004, ACE2005, and GENIA. For ACE datasets, we used the same splits as used by Lu and Roth (2015) published in their website6. For GENIA, we used GENIAcorpus3.02p7 that comes with POS tags for each word (Tateisi and Tsujii, 2004). Following previous works (Finkel and Manning, 2009; Lu and Roth, 2015), we first split the last 10% of the data as the test set. Next we used the first 80% and the subsequent 10% for training and development, respectively. We made the same modifications as described by Finkel and Manning (2009) by collapsing all DNA, RNA, and protein subtypes into DNA, RNA, and protein, keeping cell line and cell type, and removing other entity types, resulting in 5 entity types. The statistics of each dataset is shown in Table 1. We can see overlapping entities are common in such datasets.
6.2 Features
For both models (mention hypergraph and our model) that fall under the edge-based paradigm, we define features over the edges in the models.
6http://statnlp.org/research/ie#mention-hypergraph 7http://geniaproject.org/genia-corpus/pos-annotation
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
ACE2004 ACE2005 P R F1 P R F1
Lin-CRF 70.3 42.0 52.6 67.0 45.6 54.3 MH 80.7 46.1 58.7 77.6 47.3 58.8 MH (*) 71.0 55.4 62.2 67.0 57.3 61.7 Ours 76.3 52.0 61.9 73.5 51.6 60.6 Ours (*) 71.6 58.3 64.3 67.5 59.1 63.0
Table 2: Results on ACE2004 and ACE2005.
Features are defined as string concatenations of input features – information defined over the inputs (such as current word and POS tags of surrounding words) and output features – structured information defined over the output structure. In our model, when an edge represents a combination of roles, a feature is defined for each output feature. This allows us to use to make use of the identical set of features for both our multigraph model and the baseline mention hypergraph model, in order to make a proper comparison. We also followed Lu and Roth (2015) to add the additional mention penalty feature defined over the edges that represents the start of an entity to tune F1-scores on the development set.
When defining the input features for both our model and the mention hypergraph model, we tried to follow as close as possible to the features used by previous works in each dataset: we followed Lu and Roth (2015) for the features used in ACE datasets, and Finkel and Manning (2009) for features used in GENIA dataset. In general, they include surrounding words, surrounding POS tags, bag-of-words, Brown clusters (for GENIA only), and orthographic features. See the supplementary material for more details on the features.
6.3 Experimental Setup
We trained each model in the training set, then tuned the l2-regularization parameter based on the development set from the values in {0.0, 0.001, 0.01, 0.1, 1.0}. For GENIA experiments, we also tuned the number of Brown clusters from the values in {100, 1000}. Following (Lu and Roth, 2015), we also used each development set to tune the mention penalty to optimize the F1-score and report the scores on the corresponding test sets separately. Similar to Finkel and Manning (2009), as another baseline model we also trained a standard linear-chain CRF using BILOU scheme. Although this model does not support overlapping entities, it gives us a baseline to see the extent to which our model’s ability to recognize overlapping entities can help the overall performance.
7 Results and Discussion
Table 2 and 3 show the results on the ACE and GENIA datasets, respectively. Following previous works (Finkel and Manning, 2009; Lu and Roth, 2015), we report standard precision (P ), recall (R) and F1 percentage scores. For the ACE datasets, we make comparisons with the linear-chain CRF baseline (Lin-CRF), which does not support overlapping entities, as well as our implementation of the mention hypergraph baseline (MH). For the GENIA experiments, besides our implementation of the mention hypergraph baseline, we also make comparisons with the results of the two systems reported in (Finkel and Manning, 2009) – a model based on semi-Markov CRF (Semi-CRF) that cannot handle overlapping entities and their proposed constituency tree based model (F&M (’09)) that can support overlapping entities. The (*) marks the results after optimizing the F1-score in development set. Best results are highlighted in bold.
From such empirical results we can see that our proposed multigraph-based model yields significantly better results than the mention hypergraph model, with the highest improvement in F1 of 2.1 points observed in ACE2004. In GENIA dataset, our implementation of mention hypergraph matches the performance of the constituency parser-based model of Finkel and Manning (2009), while our multigraph-based model outperforms both by about 0.4 point. As expected, the linear-chain CRF baseline yields relatively lower results compared to the other models, since it cannot predict overlapping entities. However, such results give us some idea on how much performance increase we can gain by properly recognizing overlapping entities.
As these datasets consist of both overlapping and non-overlapping entities, to further understand the effectiveness of each model in recognizing overlapping entities (and non-overlapping entities), we performed some additional experiments. Specifically, we split the test data into two portions, one that consists of only sentences that
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
% MH (*) Ours (*) P R F1 P R F1
ACE2004 w/ o.l. 42 72.8 51.0 60.0 71.3 56.2 62.8 w/o o.l. 58 73.1 62.8 67.5 72.5 64.2 68.1 ACE2005 w/ o.l. 32 68.6 52.8 59.7 68.5 55.7 61.4 w/o o.l. 68 64.8 64.8 64.8 66.0 64.7 65.4 GENIA w/ o.l. 24 77.1 60.3 67.7 75.6 60.7 67.3 w/o o.l. 76 73.9 70.1 71.9 74.1 71.6 72.9
Table 4: Scores on different types of sentences.
contain at least one entity that overlaps with another (w/ o.l.), and the other consisting of sentences that do not contain any overlapping entity (w/o o.l.). The results are shown in Table 4.
We can see that in ACE datasets, our model achieves higher F1-score compared to the mention hypergraph for both portions, but it achieves slightly lower results in GENIA dataset for the portion that contains overlapping entities. Noticing that GENIA dataset has relatively lower percentage of overlapping entities compared to ACE, we believe that this low recall is due to insufficient amount of training data on overlapping entities for this dataset, which affected the models’ confidence in predicting overlapping entities. When given sufficient training data on overlapping entities, which is the case for ACE, our model is able to better recognize overlapping entities.
Such results also lead to the interesting empirical finding that our model appears to be able to do well on the task of recognizing non-overlapping entities. As a further investigation, we performed yet one additional set of experiments, on the standard CONLL2003 dataset, which consists of only entities that do not overlap with one another.
The results are shown in Table 5. Again we see that our multigraph model outperforms all baseline models, including the Illinois NER system where external resources are not used (Ratinov and Roth, 2009), and a linear-chain CRF model, although the linear-chain CRF baseline models some interactions between distinct entity types and our model does not. Such results also suggest that modeling the interactions between distinct entity types may not be crucial for achieving a good performance in entity recognition. By examining the outputs, we found that although the training set is not annotated with overlapping entities, our model (and mention hypergraph model) will be able to predict overlapping entities when it is confident, leading to improved performance.
When comparing the basic version of our model (without optimizing F1) against that of the men-
0 100 200 300 400 500 600 700 800 900 1,000 0
20
40
60
80
100
120 ·103
# iterations
O b j.
V al u e
Mention Hypergraph
Multigraph (This work)
Figure 7: Objective vs. training iterations.
tion hypergraph model, we note that our model consistently yields a higher recall. We speculate this is due to the fact that as a nondeficient model that resolves the issue of spurious structures we discussed in Section 5.2, our model is more confident in making its predictions.
We also empirically analyzed the training speed and convergence properties of the two models. Empirically, as illustrated in Figure 7 which shows how the objective improves when the training progresses on ACE2004, we found our multigraphbased model requires significantly less iterations to converge than mention hypergraph, though each training iteration requires 12% more time.
8 Conclusion and Future Work
We presented a novel multigraph-based model for entity recognition from text where entities may overlap with one another. We showed that empirically our model is able to yield better recognition results compared to previous models. We also performed theoretical analysis on the model and showed that our model resolves the spurious structures issue associated with a previous stateof-the-art model, while still maintaining the same inference time complexity.
Future work includes further investigations on how to apply the proposed multigraph-based formalism to other structured prediction problems involving complex structures, as well as finding applications of the proposed model in other related NLP tasks that involve the prediction of overlapping structures, such as equation parsing (Roy et al., 2016).
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Details on Spurious Structures
About mention hypergraph, we remarked that the normalization term includes spurious structures. This section shows in more details how this is the case using some examples.
Consider the simplified mention hypergraph as shown in Figure 8 (left) consisting of three words and where the possible edges have been restricted to what are shown in the figure. Namely, there are only three possible subgraphs here, one for each of the three (hyper-)edges coming out from the node I1 associated with the word “Apache”. Let A, B, C, D, E, F respectively denote the edges T1 → (I1), I0 → (I1), I1 → (I2), I1 → (X), I1 → (I2,X), and I2 → (X) as shown in Figure 8 (left). Further assume that features are only defined on these labeled edges.
Now we consider the normalization term Zw(x) = β(A0) calculated by inside-outside algorithm on this graph. The inside score β(I1) is the sum of the scores of the three possible entity combinations that include the word “Apache” as the first token in the entities, namely: {Apache} (using the edge I1 → (X)), {Apache helicopter} (using the edge I1 → (I2)), and {Apache helicopter, Apache} (using the hyperedge I1 → (I2,X)). Let us call this set of three entities as S1. The inside scores of I0, A1, the T-nodes, the E-nodes will be the same as this as there are no features along the unique edges between these
nodes and the node I1. Note that the inside score I(T0) represents the set of entities S2 which is the same as S1 except that each entity also includes the word “an” as the first word.
Now, the normalization term A0 includes the 9 possible entity combinations which are the results of taking all possible combinations between S1 and S2. The list of paths used to represent each of these combinations is as follows (only part of the paths is shown, the rest is implied as there is only one way to connect to the root node A0): 1. A-C-F and B-C-F (Apache helicopter, an
Apache helicopter) 2. A-C-F and B-E-F (Apache helicopter, an
Apache helicopter, an Apache) 3. A-C-F and B-D (Apache helicopter, an
Apache) 4. A-E-F and B-C-F (Apache helicopter,
Apache, an Apache helicopter) 5. A-E-F and B-E-F (Apache helicopter,
Apache, an Apache helicopter, an Apache) 6. A-E-F and B-D (Apache helicopter, Apache,
an Apache) 7. A-D and B-C-F (Apache, an Apache heli-
copter) 8. A-D and B-E-F (Apache, an Apache heli-
copter, an Apache) 9. A-D and B-D (Apache, an Apache)
Note that except the first one (the X-node associated with I1 is missing) and the last one (I2 and the last X-node are missing), the other 7 entity combinations are actually represented using the same subgraph consisting of the edges {A, B, E, F}, which is what is calculated in the fifth item (A-E-F and B-E-F). The other combinations represent entity combinations which, while valid, are not represented in the model as such, and so never appear as numerator in the likelihood. This is because whenever that entity combination appears, the model will represent it using the subgraph shown in Figure 8 (right).
B Features
For ACE datasets we used these features: 1. Words and POS tags (with window of 3
words to the left and right of current word) 2. Words and POS tags n-gram (up to length 4
containing current word) 3. Bag-of-word (with window of 5 words to the
left and to the right of current word) 4. Orthographic (following Lu and Roth (2015))
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
A0 A1
E0 E1
T0 T1
I0 I1 I2
X X
an Apache helicopter
A
D F
B C
E
A0 A1
E0 E1
T0 T1
I0 I1 I2
X X
an Apache helicopter
A
F
B
E
Figure 8: (left) A simplified example of a mention hypergraph with restricted edges. (right) The subgraph associated with 7 out of 9 possible entity combinations.
5. Parent node type For GENIA we used these features: 1. Words and POS tags (with window of 2 words to the left and right of current word) 2. Words and POS tags n-gram (up to length 4 containing current word) 3. Bag-of-word (with window of 5 words to the left and right of current word) 4. Brown clusters with window of 1 word to the left and right of current word (using 100 or 1000 clusters built from training data only) 5. Word shape (Finkel and Manning (2009)) 6. Prefixes and suffixes of current word (up to
length 6) 7. Edge type
For CoNLL 2003 we used these features: 1. Words and POS tags (with window of 2
words to the left and right of current word) 2. Words and POS tags n-gram (up to length 4
containing current word) 3. Bag-of-word (with window of 5 words to the
left and right of current word) 4. Word shape (with window of 2 words to the
left and right of current word) 5. Prefixes and suffixes of current word (up to
length 5) 6. Orthographic (following Lu and Roth (2015)) 7. Edge type
C Hyperparameter
Table 6 lists the best l2-regularization coefficient λ for each dataset and model. For GENIA, the
ACE2004 ACE2005 GENIA CoNLL MH 0.001 0.0 1.0 0.01 MH (*) 0.001 0.1 1.0 0.001 Ours 0.001 0.001 1.0 0.001 Ours (*) 0.1 0.1 1.0 0.001
optimal Brown cluster size was found to be 1000, except for ‘Ours (*)’, where the best cluster size is 100.
D GENIA Preprocessing
For GENIA, we used GENIAcorpus3.02p that comes with POS tags for each word (Tateisi and Tsujii, 2004). Similar to the problem faced by Finkel and Manning (2009) on JNLPBA dataset, we also find tokenization issues in this corpus. As described by Tateisi and Tsujii (2004), when a hyphenated word such as IL-2-induced is partially annotated as an entity (in this case IL-2), the POS annotation corpus splits it into two tokens, which when done in test set will leak some information about the presence of entity. Unlike Finkel and Manning (2009) which tried to match the tokenization during testing, we simply further split all tokens at some punctuations (those matching the regular expression [-/,.+]), while keeping the information that they all originally come from the same word. This has the advantage of simplifying the tokenization procedure, although it makes the task slightly more difficult due to the higher number of tokens.
Also, to handle the discontiguous entities present in GENIA dataset (mainly due to coordinated entities involving ellipsis), following the approach used by the JNLPBA shared task organizer (Kim et al., 2004), we consider a group of coordinated entities as one structure. For example, in “. . . the [T- and B-lymphocytes] count in . . . ”, the entities “T-lymphocytes” and “Blymphocytes” are annotated as one structure “Tand B-lymphocytes”.

The paper suggests an approach based on multigraphs (several edges may link two nodes) for detecting potentially overlapping entities.  Strengths: The problem itself could be rather interesting especially for crossing entities to decide which one might actually be mentioned in some text. The technique seems to work although the empirically results do not show some  dramatic  effect. I like that some words are spent on efficiency compared to a previous system. The paper in general is well-written but also needs some further polishing in some details (see minor remarks below).  Weaknesses: The problem itself is not really well motivated. Why is it important to detect China as an entity within the entity Bank of China, to stay with the example in the introduction? I do see a point for crossing entities but what is the use case for nested entities? This could be much more motivated to make the reader interested. As for the approach itself, some important details are missing in my opinion: What is the decision criterion to include an edge or not? In lines 229--233 several different options for the I^k_t nodes are mentioned but it is never clarified which edges should be present!  As for the empirical evaluation, the achieved results are better than some previous approaches but not really by a large margin. I would not really call the slight improvements as  outperformed  as is done in the paper. What is the effect size? Does it really matter to some user that there is some improvement of two percentage points in F_1? What is the actual effect one can observe? How many  important  entities are discovered, that have not been discovered by previous methods? Furthermore, what performance would some simplistic dictionary-based method achieve that could also be used to find overlapping things? And in a similar direction: what would some commercial system like Google's NLP cloud that should also be able to detect and link entities would have achieved on the datasets. Just to put the results also into contrast of existing  commercial  systems.  As for the result discussion, I would have liked to see some more emphasis on actual crossing entities. How is the performance there? This in my opinion is the more interesting subset of overlapping entities than the nested ones. How many more crossing entities are detected than were possible before? Which ones were missed and maybe why? Is the performance improvement due to better nested detection only or also detecting crossing entities? Some general error discussion comparing errors made by the suggested system and previous ones would also strengthen that part.  General Discussion: I like the problems related to named entity recognition and see a point for recognizing crossing entities. However, why is one interested in nested entities? The paper at hand does not really motivate the scenario and also sheds no light on that point in the evaluation. Discussing errors and maybe advantages with some example cases and an emphasis on the results on crossing entities compared to other approaches would possibly have convinced me more. So, I am only lukewarm about the paper with maybe a slight tendency to rejection. It just seems yet another try without really emphasizing the in my opinion important question of crossing entities.  Minor remarks:  - first mention of multigraph: some readers may benefit if the notion of a multigraph would get a short description  - previously noted by ... many previous: sounds a little odd  - Solving this task: which one?  - e.g.: why in italics?  - time linear in n: when n is sentence length, does it really matter whether it is linear or cubic?  - spurious structures: in the introduction it is not clear, what is meant  - regarded as _a_ chunk  - NP chunking: noun phrase chunking?  - Since they set: who?  - pervious to previous  - of Lu and Roth~(2015)  - the following five types: in sentences with no large numbers, spell out the small ones, please  - types of states: what is a state in a (hyper-)graph? later state seems to be used analogous to node?!  - I would place commas after the enumeration items at the end of page 2 and a period after the last one  - what are child nodes in a hypergraph?  - in Figure 2 it was not obvious at first glance why this is a hypergraph. colors are not visible in b w printing. why are some nodes edges in gray. it is also not obvious how the highlighted edges were selected and why the others are in gray ...  - why should both entities be detected in the example of Figure 2? what is the difference to  just  knowing the long one?  - denoting ...: sometimes in brackets, sometimes not ... why?  - please place footnotes not directly in front of a punctuation mark but afterwards  - footnote 2: due to the missing edge: how determined that this one should be missing?  - on whether the separator defines ...: how determined?  - in _the_ mention hypergraph  - last paragraph before 4.1: to represent the entity separator CS: how is the CS-edge chosen algorithmically here?  - comma after Equation 1?  - to find out: sounds a little odd here  - we extract entities_._  footnote  - we make two: sounds odd; we conduct or something like that?  - nested vs. crossing remark in footnote 3: why is this good? why not favor crossing? examples to clarify?  - the combination of states alone do_es_ not?  - the simple first order assumption: that is what?  - In _the_ previous section  - we see that our model: demonstrated? have shown?  - used in this experiments: these  - each of these distinct interpretation_s_  - published _on_ their website  - The statistics of each dataset _are_ shown  - allows us to use to make use: omit  to use   - tried to follow as close ... : tried to use the features suggested in previous works as close as possible?  - Following (Lu and Roth, 2015): please do not use references as nouns: Following Lu and Roth (2015)  - using _the_ BILOU scheme  - highlighted in bold: what about the effect size?  - significantly better: in what sense? effect size?  - In GENIA dataset: On the GENIA dataset  - outperforms by about 0.4 point_s_: I would not call that  outperform   - that _the_ GENIA dataset  - this low recall: which one?  - due to _an_ insufficient  - Table 5: all F_1 scores seems rather similar to me ... again,  outperform  seems a bit of a stretch here ...  - is more confident: why does this increase recall?  - converge _than_ the mention hypergraph  - References: some paper titles are lowercased, others not, why?