Learning Cognitive Features from Gaze Data for Sentiment and Sarcasm Classification using Convolutional Neural Network
Cognitive NLP systemsi.e., NLP systems that make use of behavioral data augment traditional text based features with cognitive features extracted from eye-movement patterns, EEG signals, brain-imaging etc.. Such extraction of features is typically manual. We contend that manual extraction of features is not good enough to tackle text subtleties that characteristically prevail in complex classification tasks like sentiment analysis and sarcasm detection, and that even the extraction and choice of features should be delegated to the learning system. We introduce a framework to automatically extract cognitive features from the eye-movement / gaze data of human readers reading the text and use them as features along with textual features for the tasks of sentiment polarity and sarcasm detection. Our proposed framework is based on Convolutional Neural Network (CNN). The CNN learns features from both gaze and text and uses them to classify the input text. We test our technique on published sentiment and sarcasm labeled datasets, enriched with gaze information, to show that using a combination of automatically learned text and gaze features yields better classification performance over (i) CNN based systems that rely on text input alone and (ii) existing systems that rely on handcrafted gaze and textual features.
1 000
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
1 Introduction
Detection of sentiment and sarcasm in usergenerated short reviews is of primary importance for social media analysis, recommendation and dialog systems. Traditional sentiment analyzers and
sarcasm detectors face challenges that arise at lexical, syntactic, semantic and pragmatic levels (Liu and Zhang, 2012; Mishra et al., 2016c). Featurebased systems (Akkaya et al., 2009; Sharma and Bhattacharyya, 2013; Poria et al., 2014) can aptly handle lexical and syntactic challenges (e.g. learning that the word deadly conveys a strong positive sentiment in opinions such as Shane Warne is a deadly bowler, as opposed to The high altitude Himalayan roads have deadly turns). It is, however, extremely difficult to tackle subtleties at semantic and pragmatic levels. For example, the sentence “I really love my job. I work 40 hours a week to be this poor.” requires an NLP system to be able to understand that the opinion holder has not expressed a positive sentiment towards her / his job. In the absence of explicit clues in the text, it is difficult for automatic systems to arrive at a correct classification decision, as they often lack external knowledge about various aspects of the text being classified.
Mishra et al. (2016b) and Mishra et al. (2016c) show that NLP systems based on cognitive data (or simply, Cognitive NLP systems) , that of leverage eye-movement information obtained from human readers, can tackle the semantic and pragmatic challenges better. The hypothesis here is that human gaze activities are related to the cognitive processes in the brain, that combines the “external knowledge” that the a reader possesses with textual clues that she / he perceives. While incorporating behavioral information obtained from gaze-data in NLP systems is intriguing and quite plausible, especially due to the availability of low cost eye-tracking machinery (Wood and Bulling, 2014; Yamamoto et al., 2013), few methods exist for text classification and they rely on handcrafted features extracted from gaze data (Mishra et al., 2016b,c). These systems have limited capabilities due to two reasons: (a) Manually designed gaze based features may not adequately capture all
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
162
163
164
165
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
forms of textual subtleties (b) Eye-movement data is not as intuitive to analyze as text which makes the task of designing manual features more difficult. So, in this work, instead of handcrafting the gaze based and textual features, we try to learn feature representations from both gaze and textual data using Convolutional Neural Networks (CNNs). We test our technique on two publicly available datasets enriched with eye-movement information, used for binary classification tasks of sentiment polarity and sarcasm detection. Our experiments show that the automatically extracted features help to achieve significant classification performance improvement over (a) existing systems that rely on handcrafted gaze and textual features and (b) CNN based systems that rely on text input alone.
The rest of the paper is organized as follows. Section 2 discusses the motivation behind using readers’ eye-movement data in a text classification setting. Section 3 discusses on why CNNs is preferred over other available alternatives for feature extraction. The CNN architecture is proposed and discussed in Section 4. Section 5 describes our experimental setup and results are discussed in Section 6. We provide a detailed analysis of the results along with some insightful observations in Section 7. Section 8 points to relevant literature followed by Section 9 that concludes the paper. Terminology: A fixation is a relatively long stay of gaze on a visual object (such as words in text) where as a sacccade corresponds to quick shifting of gaze between two positions of rest. Forward and backward saccades are called progressions and regressions respectively. A scanpath is a line graph that contains fixations as nodes and saccades as edges.
2 Eye-movement and Linguistic Subtleties
Presence of linguistic subtleties often induces (a) surprisal (Kutas and Hillyard, 1980; Malsburg et al., 2015), due to the underlying disparity /context incongruity or (b) higher cognitive load (Rayner and Duffy, 1986), due to the presence of lexically and syntactically complex structures. While surprisal accounts for irregular saccades (Malsburg et al., 2015), higher cognitive load results in longer fixation duration (Kliegl et al., 2004).
Mishra et al. (2016b) find that presence of sarcasm in text triggers either irregular saccadic patterns or unusually high duration fixations than non-sarcastic texts (illustrated through example scanpath representations in Figure 1). For sentiment bearing texts, highly subtle eye-movement patterns are observed for semantically / pragmatically complex negative opinions (expressing irony, sarcasm, thwarted expectations etc.) than the simple ones (Mishra et al., 2016b). The association between linguistic subtleties and eye-movement patterns could be captured through sophisticated feature engineering that considers both gaze and text inputs. In our work, CNNs take the onus of feature engineering.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
T ex
t C
om p
on en
t
Non-static
Static
Saccade
Fixation
G az
e C
om p
on en t P1 P2 P3 P4 P5 P6 P7 P8
N×K representation of sentences with static and non static channels
P×G representation of sentences
with fixation and saccade channels
1-D convolution operation with multiple filter width
and feature maps
2-D convolution operation with multiple filter row and
Column widths
Max-pooling for each filter width
Max-pooling over
multiple dimensions for
multiple filter widths
Fully connected with dropouts and softmax
output
Merged pooled values
Figure 3: Deep convolutional model for feature extraction from both text and gaze inputs
3 Why Convolutional Neural Network?
CNNs have been quite effective in learning filters for image processing tasks, filters being used to transform the input image into more informative feature space (Krizhevsky et al., 2012). Filters learned at various CNN layers are quite similar to handcrafted filters used for detection of edges, contours and removal of redundant backgrounds. We believe, a similar technique can also be applied to eye-movement data, where the learned filters will, hopefully, extract informative cognitive features. For instance, for sarcasm, we expect the network to learn filters that detect long distance saccades (refer to Figure 2 for an analogical illustration). With more number of convolution filters of different dimensions, the network may extract multiple features related to different gaze attributes (such as fixations, progressions, regressions and skips) and will be free from any form of human bias that manually extracted features are susceptible to.
4 Learning Feature Representations: The CNN architecture
Figure 3 shows the CNN architecture with two components for processing and extracting features from text and gaze inputs. The components are explained below.
4.1 Text Component
The text component is quite similar to the one proposed by Kim (2014) for sentence classification.
Words (in the form of one-hot representation) in the input text are first replaced by their embeddings of dimension K (ith word in the sentence represented by an embedding vector xi ∈ RK). As per Kim (2014), a multi-channel variant of CNN (referred to as MULTICHANNELTEXT) can be implemented by using two channels of embeddingsone that remains static through out training (referred to as STATICTEXT), and the other one that gets updated during training (referred to as NONSTATICTEXT). We separately experiment with static, non-static and multi-channel variants.
For each possible input channel of the text component, a given text is transformed into a tensor of fixed length N (padded with zero-tensors wherever necessary to tackle length variations) by concatenating the word embeddings.
x1:N = x1 ⊕ x2 ⊕ x3 ⊕ ...⊕ xN (1)
where ⊕ is the concatenation operator. To extract local features1, convolution operation is applied. Convolution operation involves a filter, W ∈ RHK , which is convolved with a window of H embeddings to produce a local feature for the H words. A local feature, ci is generated from a window of embeddings xi:i+H−1 by applying a non linear function (such as a hyperbolic tangent) over the convoluted output. Mathematically,
ci = f(W.xi:i+H−1 + b) (2)
1features specific to a region in case of images or window of words in case of text
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
where b ∈ R is the bias and f is the non-linear function. This operation is applied to each possible window of H words to produce a feature map (c) for the window size H .
c = [c1, c2, c3, ..., cN−H+1] (3)
A global feature is then obtained by applying max pooling operation2 (Collobert et al., 2011) over the feature map. The idea behind max-pooling is to capture the most important feature - one with the highest value - for each feature map.
We have described the process by which one feature is extracted from one filter (for illustration, red bordered portions in Figure 3 depict the case of H = 2). The model uses multiple filters (with varying window sizes) to obtain multiple features representing the text. In the MULTICHANNELTEXT variant, for a window of H words, convolution operation is separately applied on both the embedding channels. Local features learned from both the channels are concatenated before applying max-pooling.
4.2 Gaze Component
The gaze component deals with scanpaths of multiple participants annotating the same text. Scanpaths can be pre-processed to extract two sequences of gaze data to form separate channels of input: (1) A sequence of normalized3 durations of fixations (in milliseconds) in the order in which they appear in the scanpath and (2) A sequence of position of fixations (in terms of word id) in the order in which they appear in the scanpath. These channels are related to two fundamental gaze attributes such as fixation and saccade respectively. With two channels, we thus have three possible configurations of the gaze component such as (i) FIXATION, where the input is normalized fixation duration sequence, (ii) SACCADE, where the input is fixation position sequence, and (iii) MULTICHANNELGAZE, where both the inputs channels are considered.
For each possible input channel, the input is in the form of a P × G matrix (with P → number of participants and G → length of the input sequence). Each element of the matrix gij ∈ R, with i ∈ P and j ∈ G, corresponds to the jth gaze attribute (either fixation duration or word id, depending on the channel) of the input sequence of
2mean pooling does not perform well. 3scaled across participants using min-max normalization
to reduce subjectivity
the ith participant. Now, unlike the text component, here we apply convolution operation across two dimensions i.e. choosing a two dimensional convolution filter W ∈ RJK (for simplicity, we have kept J = K, thus , making the dimension of W , J2). For the dimension size of J2, a local feature cij is computed from the window of gaze elements gij:(i+J−1)(j+J−1) by,
cij = f(W.gij:(i+J−1)(j+J−1) + b) (4)
where b ∈ R is the bias and f is a non-linear function. This operation is applied to each possible window of size J2 to produce a feature map (c),
c =[c11, c12, c13, ..., c1(G−J+1),
c21, c22, c23, ..., c2(G−J+1), ...,
c(P−J+1)1, c(P−J+1)2, ..., c(P−J+1)(G−J+1)]
(5)
A global feature is then obtained by applying max pooling operation. Unlike the text component, max-pooling operator is applied to a 2D window of local features size M × N (for simplicity, we set M = N , denoted henceforth as M2). For the window of size M2, the pooling operation on c will result in as set of global features ĉJ = max{cij:(i+M−1)(j+M−1)} for each possible i, j.
We have described the process by which one feature is extracted from one filter (of 2D window size J2 and max-pooling window size of M2). In Figure 3, red and blue bordered portions illustrate the cases of J2 = [3, 3] and M2 = [2, 2] respectively. Like the text component, the gaze component uses multiple filters (also with varying window size) to obtain multiple features representing the gaze input. In the MULTICHANNELGAZE variant, for a 2D window of J2, convolution operation is separately applied on both fixation duration and saccade channels and local features learned from both the channels are concatenated before max-pooling is applied.
Once the global features are learned from both the text and gaze components, they are merged and passed to a fully connected feed forward layer (with number of units set to 150) followed by a SoftMax layer that outputs the the probabilistic distribution over the class labels.
The gaze component of our network is not invariant of the order in which the scanpath data is given as input- i.e., the P rows in the P × G can not be shuffled, even if each row is independent from others. The only way we can think of for
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
addressing this issue is by applying convolution operations to all P × G matrices formed with all the permutations of P , capturing every possible ordering. Unfortunately, this makes the training process significantly less scalable, as the number of model parameters to be learned becomes huge. As of now, training and testing are carried out by keeping the order of the input constant.
5 Experiment Setup
We now share several details regarding our experiments below. 1. Dataset: We experiment on sentiment and sarcasm tasks using two publicly available datasets enriched with eye-movement information. Dataset 1 has been released by Mishra et al. (2016a). It contains 994 text snippets with 383 positive and 611 negative examples. Out of the 994 snippets, 350 are sarcastic. Dataset 2 has been used by Joshi et al. (2014) and it consists of 843 snippets comprising movie reviews and normalized tweets out of which 443 are positive and 400 are negative. Eye-movement data of 7 and 5 readers is available for each snippet for dataset 1 and 2 respectively. 2. CNN Variants: With text component alone we have three variants such as STATICTEXT, NONSTATICTEXT and MULTICHANNELTEXT (refer to Section 4.1). Similarly, with gaze component we have variants such as FIXATION, SACCADE and MULTICHANNELGAZE (refer to Section 4.2). With both text and gaze components, 9 more variants could be experimented with. 3. Hyper-parameters: For text component, we experiment with filter widths (H) of [3, 4]. For the gaze component, 2D filters (J2) set to [3 × 3], [4× 4] respectively. The max pooling 2D window, M2, is set to [2 × 2]. In both gaze and text components, number of filters is set to 150, resulting in 150 feature maps for each window. These model hyper-parameters are fixed by trial and error and are possibly good enough to provide a first level insight into our system. Tuning of hyperparameters might help in improving the performance of our framework, which is on our future research agenda. 4. Regularization: For regularization dropout is employed on the penultimate layer with a constraint on l2-norms of the weight vectors (Hinton et al., 2012). Dropout prevents co-adaptation of hidden units by randomly dropping out - i.e., setting to zero - a proportion p of the hidden units
during forward propagation. We set p to 0.25. 5. Training: We use ADADELTA optimizer (Zeiler, 2012), with a learning rate of 0.1. The input batch size is set to 32 and number of training iterations (epochs) is set to 200. 10% of the training data is used for validation. 6. Use of pre-trained embeddings: Initializing the embedding layer with of pre-trained embeddings can be more effective than random initialization (Kim, 2014). In our experiments, we have used embeddings using word2vec facilitated by Mikolov et al. (2013) (best results obtained with embedding dimension of 50). We have also tried randomly initializing the embeddings but better results are obtained with pre-trained embeddings. 7. Comparison with existing work: For sentiment analysis, we compare our systems’s accuracy (for both datasets 1 and 2) with Mishra et al. (2016c)’s systems that rely on handcrafted text and gaze features. For sarcasm detection, we compare Mishra et al. (2016b)’s sarcasm classifier with ours using dataset 1 (with available gold standard labels for sarcasm). We follow the same 10-fold traintest configuration as these existing works for consistency.
6 Results
In this section, we discuss the results for different model variants for sentiment polarity and sarcasm detection tasks.
6.1 Results for Sentiment Analysis Task
Table 1 presents results for sentiment analysis task. For dataset 1, different variants of our CNN architecture outperform the best systems reported by Mishra et al. (2016c), with a maximum F-score improvement of 3.8%. This improvement is statistically significant of p < 0.05 as confirmed by McNemar test. Moreover, we observe an F-score improvement of around 5% for CNNs with both gaze and text components as compared to CNNs with only text components (similar to the system by Kim (2014)), which is also statistically significant (with p < 0.05).
For dataset 2, CNN based approaches do not perform better than manual feature based approaches. However, variants with both text and gaze components outperform the ones with only text component (Kim, 2014), with a maximum Fscore improvement of 2.9%. We observe that for dataset 2, training accuracy reaches 100 within
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Dataset1 Dataset2
Configuration P R F P R F
Traditional systems based on
Näive Bayes 63.0 59.4 61.14 50.7 50.1 50.39 Multi-layered Perceptron 69.0 69.2 69.2 66.8 66.8 66.8
textual features SVM (Linear Kernel) 72.8 73.2 72.6 70.3 70.3 70.3 Systems by Mishra et al. (2016c) Gaze based (Best) 61.8 58.4 60.05 53.6 54.0 53.3 Text + Gaze (Best) 73.3 73.6 73.5 71.9 71.8 71.8
CNN with only text input (Kim, 2014) STATICTEXT 63.85 61.26 62.22 55.46 55.02 55.24 NONSTATICTEXT 72.78 71.93 72.35 60.51 59.79 60.14 MULTICHANNELTEXT 72.17 70.91 71.53 60.51 59.66 60.08
CNN with only gaze Input
FIXATION 60.79 58.34 59.54 53.95 50.29 52.06 SACCADE 64.19 60.56 62.32 51.6 50.65 51.12 MULTICHANNELGAZE 65.2 60.35 62.68 52.52 51.49 52
CNN with both text and gaze Input
STATICTEXT + FIXATION 61.52 60.86 61.19 54.61 54.32 54.46 STATICTEXT + SACCADE 65.99 63.49 64.71 58.39 56.09 57.21 STATICTEXT + MULTICHANNELGAZE 65.79 62.89 64.31 58.19 55.39 56.75 NONSTATICTEXT + FIXATION 73.01 70.81 71.9 61.45 59.78 60.60 NONSTATICTEXT + SACCADE 77.56 73.34 75.4 65.13 61.08 63.04 NONSTATICTEXT + MULTICHANNELGAZE 79.89 74.86 77.3 63.93 60.13 62 MULTICHANNELTEXT + FIXATION 74.44 72.31 73.36 60.72 58.47 59.57 MULTICHANNELTEXT + SACCADE 78.75 73.94 76.26 63.7 60.47 62.04 MULTICHANNELTEXT + MULTICHANNELGAZE 78.38 74.23 76.24 64.29 61.08 62.64
Table 1: Results for different traditional feature based systems and CNN model variants for the task of sentiment analysis. Abbreviations (P,R,F)→ Precision, Recall, F-score. SVM→Support Vector Machine
25 epochs with validation accuracy stable around 50%, indicating the possibility of overfitting. Tuning the regularization parameters specific to dataset 2 may help here. Even though CNN might not be proving to be a choice as good as handcrafted features for dataset 2, the bottom line remains that incorporation of gaze data into CNN consistently improves the performance over onlytext-based CNN variants.
6.2 Results for Sarcasm Detection Task
For sarcasm detection, our CNN model variants outperform traditional systems by a maximum margin of 11.27% (Table 2). However, the improvement by adding the gaze component to the CNN network is just 1.36%, which is statistically insignificant over CNN with text component. While inspecting the sarcasm dataset, we observe a clear difference between the vocabulary of sarcasm and non-sarcasm classes in our dataset. This, perhaps, was captured well by the text component, especially the variant with only non-static embeddings.
7 Discussion
In this section, some important observations from our experiments are discussed. • Effect of embedding dimension variation: Embedding dimension has proven to have a deep impact on the performance of neural systems (dos
Santos and Gatti, 2014; Collobert et al., 2011). We repeated our experiments by varying the embedding dimensions in the range of [50-300]4 and observed that reducing embedding dimension improves the F-scores by a little margin. Best results are obtained when the embedding dimension is as low as 50. Small embedding dimensions are probably reducing the chances of over-fitting when the data size is small. We also observe that for different embedding dimensions, performance of CNN with both gaze and text components is consistently better than that with only text component. • Effect of static / non static text channels: Nonstatic embedding channel has a major role in tuning embeddings for sentiment analysis by bringing adjectives expressing similar sentiment close to each other (e.g, good and nice), where as static channel seems to prevent over-tuning of embeddings (over-tuning often brings verbs like love closer to the pronoun I in embedding space, purely due to higher co-occurrence of these two words in sarcastic examples). • Effect of fixation / saccade channels: For sentiment detection, saccade channel seems to be handing text having semantic incongruity (due to the presence of irony / sarcasm) better. Fixation channel does not help much, may be because of higher variance in fixation duration. For sarcasm
4a standard range (Liu et al., 2015; Melamud et al., 2016)
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Configuration P R F
Traditional systems based on
Näive Bayes 69.1 60.1 60.5 Multi-layered Perceptron 69.7 70.4 69.9
textual features SVM (Linear Kernel) 72.1 71.9 72 Systems by Riloff et al. (2013) Text based (Ordered) 49 46 47 Text + Gaze (Unordered) 46 41 42
System by Joshi et al. (2015)
Text based (best) 70.7 69.8 64.2
Systems by Mishra et al. (2016b)
Gaze based (Best) 73 73.8 73.1 Text based (Best) 72.1 71.9 72 Text + Gaze (Best) 76.5 75.3 75.7
CNN with only text input (Kim, 2014) STATICTEXT 67.17 66.38 66.77 NONSTATICTEXT 84.19 87.03 85.59 MULTICHANNELTEXT 84.28 87.03 85.63
CNN with only gaze input
FIXATION 74.39 69.62 71.93 SACCADE 68.58 68.23 68.40 MULTICHANNELGAZE 67.93 67.72 67.82
CNN with both text and gaze Input
STATICTEXT + FIXATION 72.38 71.93 72.15 STATICTEXT + SACCADE 73.12 72.14 72.63 STATICTEXT + MULTICHANNELGAZE 71.41 71.03 71.22 NONSTATICTEXT + FIXATION 87.42 85.2 86.30 NONSTATICTEXT + SACCADE 84.84 82.68 83.75 NONSTATICTEXT + MULTICHANNELGAZE 84.98 82.79 83.87 MULTICHANNELTEXT + FIXATION 87.03 86.92 86.97 MULTICHANNELTEXT + SACCADE 81.98 81.08 81.53 MULTICHANNELTEXT + MULTICHANNELGAZE 83.11 81.69 82.39
Table 2: Results for different traditional feature based systems and CNN model variants for the task of sarcasm detection on dataset 1. Abbreviations (P,R,F)→ Precision, Recall, F-score
detection, fixation and saccade channels perform with similar accuracy when employed separately. Accuracy reduces with gaze multichannel, may be because of higher variation of both fixations and saccades across sarcastic and non-sarcastic classes, as opposed to sentiment classes.
• Effectiveness of the CNN learned features To examine how good the features learned by the CNN are, we analyzed the features for a few example cases. Figure 4 presents some of the testexamples for the task of sarcasm detection. Example 1 contains sarcasm while examples 2, 3 and 4 are non-sarcastic. To see if there is any difference in the automatically learned features between text-only and combined text and gaze variants, we examine the feature vector (of dimension 150) for the examples obtained from different model variants. Output of the hidden layer after merge layer is considered as features learned by the network. We plot the features, in the form of color-bars, following Li et al. (2016) - denser colors representing higher feature values. In Figure 4, we show only two (representative) model variants viz., MULTICHANNELTEXT and MULTICHANNELTEXT+ MULTICHANNELGAZE. As one can see, addition of gaze information helps
to generate features with more subtle differences (marked by blue rectangular boxes) for sarcastic and non-sarcastic texts. It is also interesting to note that in the marked region, features for the sarcastic texts exhibit more intensity than the nonsarcastic ones - perhaps capturing the notion that sarcasm typically conveys an intensified negative opinion. This difference is not clear in feature vectors learned by text only systems for instances like example 2, which has been incorrectly classified by MULTICHANNELTEXT. Example 4 is incorrectly classified by both the systems, perhaps due to lack of context. In cases like this, addition of gaze information does not help much in learning more distinctive features, as it becomes difficult for even humans to classify such texts.
8 Related Work
Sentiment and sarcasm classification are two important problems in NLP and have been the focus of research for many communities for quite some time. Popular sentiment and sarcasm detection systems are feature based and are based on unigrams, bigrams etc. (Dave et al., 2003; Ng et al., 2006), syntactic properties (Martineau and Finin, 2009; Nakagawa et al., 2010), semantic properties
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
1. I would like to live in Manchester, England. The transition between Manchester and death would be unnoticeable. (Sarcastic, Negative Sentiment)
2. We really did not like this camp. After a disappointing summer, we switched to another camp, and all of us much happier on all fronts! (Non Sarcastic, Negative Sentiment)
3. Helped me a lot with my panics attack I take 6 mg a day for almost 20 years can't stop of course but make me feel very comfortable (Non Sarcastic, Positive Sentiment)
4. Howard is the King and always will be, all others are weak clones. (Non Sarcastic, Positive Sentiment)
(a) MultichannelText + MultichannelGaze (b) MultichannelText
Figure 4: Visualization of representations learned by two variants of the network for sarcasm detection task. The output of the Merge layer (of dimension 150) are plotted in the form of colour-bars. Plots with thick red borders correspond to wrongly predicted examples.
(Balamurali et al., 2011). For sarcasm detection, supervised approaches rely on (a) Unigrams and Pragmatic features (González-Ibánez et al., 2011; Barbieri et al., 2014; Joshi et al., 2015) (b) Stylistic patterns (Davidov et al., 2010) and patterns related to situational disparity (Riloff et al., 2013) and (c) Hastag interpretations (Liebrecht et al., 2013; Maynard and Greenwood, 2014). Recent systems are based on variants of deep neural network built on the top of embeddings. A few representative works in this direction for sentiment analysis are based on CNNs (dos Santos and Gatti, 2014; Kim, 2014; Tang et al., 2014), RNNs (Dong et al., 2014; Liu et al., 2015) and combined architecture (Wang et al., 2016). Few works exist on using deep neural networks for sarcasm detection, one of which is by (Ghosh and Veale, 2016) that uses a combination of RNNs and CNNs.
Eye-tracking technology is a relatively new NLP, with a very few systems directly making use of gaze data in prediction frameworks. Klerke et al. (2016) present a novel multi-task learning approach for sentence compression using labeled data, while, Barrett and Søgaard (2015) discriminate between grammatical functions using gaze features. The closest works to ours is by Mishra et al. (2016b) and Mishra et al. (2016c) that introduce feature engineering based on both gaze
and text data for sentiment and sarcasm detection tasks. These recent advancements motivate us to explore the cognitive NLP paradigm .
9 Conclusion and Future Directions
In this work, we proposed a multimodal ensemble of features, automatically learned using variants of CNNs from text and readers’ eye-movement data, for the tasks of sentiment and sarcasm classification. On multiple published datasets for which gaze information is available, our systems could achieve significant performance improvements over (a) systems that rely on handcrafted gaze and textual features and (b) CNN based systems that rely on text input alone. An analysis of the learned features confirms that the combination of automatically learned features is indeed capable of representing deep linguistic subtleties in text that pose challenges to sentiment and sarcasm classifiers. Our future agenda includes: (a) optimizing the CNN framework hyper-parameters (e.g., filter width, dropout, embedding dimensions etc.) to obtain better results, (b) exploring the applicability of our technique for document-level sentiment analysis and (c) applying our framework on related problems, such as emotion analysis, text summarization and question-answering.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  This paper tackles an interesting problem and provides a (to my knowledge) novel and reasonable way of learning and combining cognitive features with textual features for sentiment analysis and irony detection. The paper is  clearly written and organized, and the authors provided a lot of useful detail and informative example and plots. Most of the results are convincing, and the authors did a good job comparing their approach and results with previous work.  - Weaknesses:  1. Just from the reading abstract, I expected that the authors' approach would significantly outperform previous methods, and that using both the eye-gaze and textual features consistently yields the best results. Upon reading the actual results section, however, it seems like the findings were more mixed. I think it would be helpful to update the abstract and introduction to reflect this.  2. When evaluating the model on dataset 1 for sentiment analysis, were the sarcastic utterances included? Did the model do better on classifying the non-sarcastic utterances than the sarcastic ones? 3. I understand why the eye-movement data would be useful for sarcasm detection, but it wasn't as obvious to me why it would be helpful for (non-sarcastic) sentiment classification beyond the textual features.   - General Discussion:  This paper contains a lot of interesting content, and the approach seems solid and novel to me. The results were a little weaker than I had anticipated from the abstract, but I believe would still be interesting to the larger community and merits publication.


Determining Gains Acquired from Word Embedding Quantitatively using Discrete Distribution Clustering
Word embeddings have become widelyused in document analysis. A large number of models have been proposed, but the net gain these models can achieve expectably beyond the traditional bag-ofwords based approaches remains undetermined. Our empirical studies, conducted from a nonparametric unsupervised perspective, reveal where and how word embeddings can contribute to document analysis. Our approach is based on a recent algorithmic advance in nonparametric clustering for empirical measures, which neither invents nor relies on any document vector representations. The new document clustering approach proposed in this work is easy to use and stably outperforms other existing methodologies on a variety of document-clustering tasks.
1 000
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
1 Introduction
Word embeddings, or word vectors, have been broadly adopted for document analysis (Mikolov et al., 2013b,a). A key appeal of word embedding methods is that they can be obtained from external large-scale corpus and then be easily utilized for different data. Before choosing word embeddings for data analysis, researchers must first consider how much extra gain can be brought from the “embedded” knowledge of words in comparison with that achieved by existing bag-of-words based approaches. Moreover, they must also consider how to quantify that gain. Such a preliminary evaluation is often necessary before any further decisions can be made about the data.
Answering such questions is important: almost every model used in practice exploits some basic
representations — bag-of-words and word embeddings — for the sake of its computational tractability. Based on word embeddings, high-level models are designed for various tasks. Examples include entity representations, similarity measures, data manifolds, hierarchical structures, language models, and neural architectures. Therefore, it is important to investigate whether the gain or loss found in practice should be credited to the extra assumptions associated with those high-level models or to the use of basic word embeddings. As our experiments demonstrate, introducing these extra assumptions will make individual methods effective only if certain constraints are met. We will address this issue from an unsupervised perspective.
Our proposed clustering framework has several advantages. Instead of suppressing a document into a fixed-length vector feeding post-analysis, our framework uses the Wasserstein distance (or the Earth Mover’s Distance, EMD) as a metadistance to quantify the dissimilarity between two empirical nonparametric measures (or discrete distributions) over word embedding space (Wan, 2007; Kusner et al., 2015). Hence, it excludes any vector representation of the documents and sidesteps extra high-level assumptions, which is crucial and beneficial to evaluating the gain from basic word embeddings.
Our approach is intuitive and robust. the Wasserstein distance considers the cross-term relationship between different words in a principled fashion. As defined, the distance between two documents, say A and B, are the minimum cumulative cost that words from document A need to “travel” to match exactly the point cloud of document B. Here, the travel cost of a path between two words is their (squared) Euclidean distance in the word embedding space. Therefore, how much
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
benefit Wasserstein distance brings also depends on how the utilized word embedding space flattens document data as distributions, disentangling independent factors into different regions, subject to the interest of task.
While Wasserstein distance is well suited for document analysis, a major obstacle of EMDbased approaches is the high-magnitude computation involved, especially for the original D2clustering method (Li and Wang, 2008) — an EMD-based clustering framework. The main technical hurdle is to compute the Wasserstein barycenter, which is a discrete distribution, efficiently for a given set of discrete distributions. Thanks to the recent advance of algorithms for efficient solving of Wasserstein barycenters, one can now perform document clustering in a nonparametric way by directly treating them as empirical measures over word embedding space (Cuturi and Doucet, 2014; Ye and Li, 2014; Benamou et al., 2015; Ye et al., 2017). In short, the intuition behind such a nonparametric framework is: By keeping D2-clustering to a minimal assumption set, we can achieve results of the highest possible clustering quality regardless the size and patterns of data. Obtaining highest quality clustering of unstructured text data merits consuming extra computational resources. Because it is a crucial step in many techniques and applications of natural language processing, such as cross-document co-reference resolution (Singh et al., 2011), document summarization (Radev et al., 2004; Wang and Li, 2010), retrospective events detection (Yang et al., 1998), and opinion mining (Zhai et al., 2011).
Our contributions. Our work has two main contributions. First, we create a basic tool of document clustering with mere hyper-parameters at scale. Our tool leverages the state-of-the-art numerical toolbox developed for optimal transport to achieve computational feasibility. Meanwhile, it gives state-of-the-art clustering performances across heterogeneous text data — an advantage over other methods in the literature. Second, with our tool, one can quantitatively inspect how well a word-embedding model can fit the data and how much gain or loss will be obtained compared to traditional bag-of-words models. Acquiring insights on these questions is valuable for document analysis beyond clustering. The exploration of D2-clustering for documents will provide a win-
dow for investigating these questions.
2 Related Work
In the original D2-clustering framework (Li and Wang, 2008), calculating Wasserstein barycenter involves solving a large-scale LP problem at each inner iteration, severely limiting the scalability and robustness of the framework. Such high magnitude of computations had prohibited it from many real-world applications until recently. To accelerate the computation of Wasserstein barycenter, and ultimately to improve D2clustering, multiple numerical algorithmic efforts have been made in the recent few years (Cuturi and Doucet, 2014; Ye and Li, 2014; Benamou et al., 2015; Ye et al., 2017).
Although the effectiveness of Wasserstein distance has been well recognized in the computer vision and multimedia literature, the property of Wasserstein barycenter has not been well understood. To our knowledge, there still lacks systematic study of applying Wasserstein barycenter and D2-clustering in document analysis with word embeddings.
A closely related work to ours was proposed by authors of (Kusner et al., 2015) who recently connected the Wasserstein distance to the word embeddings for comparing documents. Our work differs from theirs in the methodology. We directly pursue a scalable clustering setting rather than construct a nearest neighbor graph based on calculated distances, because the calculation of the Wasserstein distances of all pairs is too expensive to be practical. The authors of (Kusner et al., 2015) used a lower bound that was cheaper to compute in order to prune unnecessary full distance calculation, but the scalability of this modified approach is still considered very limited, an issue to be discussed in Section 4.3. On the other hand, our approach adopts the framework similar to K-means which is of complexity O(n) per iteration, and usually converges within tens of iterations. The computation of D2-clustering, though in its original form was magnitudes heavier than other document clustering methods, can now be done with efficient parallelization and proper implementations (Ye et al., 2017).
3 Our Approach
This section introduces the distance, the D2clustering technique, the fast computation frame-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
work, and how they are used in the proposed document clustering method.
3.1 Wasserstein Distance
Suppose we represent each document d k consisting m
k unique words by a discrete measure or a discrete distribution, where k = 1, . . . , N with N being the sample size:
d k =
X mk
i=1 w(k) i x (k) i . (1)
Here x denotes the Dirac measure with support x, and w(k)
i 0 is the “importance weight” for the i-th word in the k-th document, with P mk i=1w (k) i = 1. And x(k) i
2 Rd, called a support point, is the semantic embedding vector of the i-th word. The 2nd-order Wasserstein distance between two documents d1 and d2 (and likewise for any document pairs) is defined by the following LP problem: W 2(d1, d2) :=
min
⇧
P i,j ⇡ i,j kx(1) i x(2) j
k22 s.t. P m2 j=1 ⇡i,j = wi, 8i, P m1 i=1 ⇡i,j = wj , 8j
⇡ i,j 0, 8i, j , (2)
where ⇧ = {⇡ i,j } is a m1 ⇥ m2 coupling matrix, and let {C
i,j := kx(1) i x(2) j k22} be transportation costs between words. Wasserstein distance is a true metric (Villani, 2003) for measures, and its best exact algorithm has a complexity of O(m3 logm) (Pele and Werman, 2009; Cuturi, 2013), if m1 = m2 = m.
3.2 Discrete Distribution (D2-) Clustering
D2-clustering (Li and Wang, 2008) iterates between the assignment step and centroids updating step in a similar way as the Lloyd’s K-means. Suppose we are to find K clusters. The assignment step finds each member distribution its nearest mean from K candidates. The mean of each cluster is again a discrete distribution with m support points, denoted by c
i , i = 1, . . . ,K. Each mean is iteratively updated to minimize its total within cluster variation. We can write the D2clustering problem as follows: given sample data {d
k }N k=1, support size of means m, and desired
number of clusters K, D2-clustering solves
min
c1,...,cK
X N
k=1 min 1iK W 2(d k , c i ) , (3)
where c1, . . . , cK are Wasserstein barycenters. At the core of solving the above formulation is an optimization method that searches the Wasserstein barycenters of varying partitions. Therefore, we concentrate on the following problem. For each cluster, we reorganize the index of member distributions from 1, . . . , n. The Wasserstein barycenter (Agueh and Carlier, 2011; Cuturi and Doucet, 2014) is by definition the solution of
min
c
X n
k=1 W 2(d k , c) , (4)
where c = P m
i=1wi xi . The above Wasserstein barycenter formulation involves two levels of optimization: the outer level finding the minimizer of total variations, and the inner level solving Wasserstein distances. We remark that in D2clustering, we need to solve multiple Wasserstein barycenters rather than a single one. This constitutes the third level of optimization.
3.3 Modified Bregman ADMM for Computing Wasserstein Barycenter
The recent modified Bregman alternating direction method of multiplier (B-ADMM) algorithm (Ye et al., 2017), motivated by (Wang and Banerjee, 2014), is a practical choice for computing Wasserstein barycenters. We briefly sketch their algorithmic procedure of this optimization method here for the sake of completeness. To solve for Wasserstein barycenter defined in Eq. (4), the key procedure of the modified Bregman ADMM involves iterative updates of four block of primal variables: the support points of c — {x
i }m i=1 (with trans-
portation costs {C i,j }(k) for k = 1, . . . , n), the importance weights of c — {w
i }m i=1, and two sets of
split matching variables — {⇡(k,1) i,j } and {⇡(k,2) i,j }, for k = 1, . . . , n, as well as Lagrangian variables { (k)
i,j } for k = 1, . . . , n. In the end, both {⇡(k,1) i,j } and {⇡(k,2)
i,j } converge to the matching weight in Eq. (2) with respect to d(c, d
k ). The iterative algorithm proceeds as follows until c converges or a maximum number of iterations are reached: given constant ⌧ 10, ⇢ / P i,j,k C(k) i,jP
n k=1mkm and round-off
tolerance ✏ = 10 10, those variables are updated in the following order. Update {x
i }m i=1 and {C (k) i,j } in every ⌧ iterations:
x i :=
1
nw i
X n
k=1
X mk
j=1 ⇡(k,1) i,j x(k) j , 8i, (5)
C(k) i,j := kx i x(k) j k22, 8i, j and k, (6)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Update {⇡(k,1) i,j } and {⇡(k,2) i,j }. For each i, j and k,
⇡(k,2) i,j := ⇡(k,2) i,j exp
C(k)
i,j
(k) i,j
⇢
! + ✏ (7)
⇡(k,1) i,j := w(k) j ⇡(k,2) i,j
.⇣X m
l=1 ⇡(k,2) l,j
⌘ (8)
⇡(k,1) i,j := ⇡(k,1) i,j exp ⇣ (k) i,j /⇢ ⌘ + ✏ (9)
Update {w i }m i=1. For i = 1, . . . ,m ,
w i :=
nX
k=1
P mk j=1 ⇡ (k,1) i,j
P i,j ⇡(k,1) i,j
(10)
w i := w i
.⇣X m
i=1 w i
⌘ (11)
Update {⇡(k,2) i,j } and { (k) i,j }. For each i, j and k,
⇡(k,2) i,j := w i ⇡(k,1) i,j
.⇣X mk
l=1 ⇡(k,1) i,l
⌘ (12)
(k) i,j := (k) i,j + ⇢ ⇣ ⇡(k,1) i,l ⇡(k,2) i,l ⌘ . (13)
Eq. (5)-(13) can all be vectorized as very efficient numerical routines. In a data parallel implementation, only Eq. (5) and Eq. (10) (involving P n
k=1) needs to be synchronized. The software package detailed in (Ye et al., 2017) was used to generate relevant experiments. We make available our codes and pre-processed datasets for reproducing all experiments of our approach.
4 Experiments
4.1 Datasets and Evaluation Metrics
We prepare six datasets to conduct a set of experiments. Two short-text datasets are created as follows. (1) BBCNews abstract: we concatenate the title and the first sentence of news posts from BBCNews dataset1 to create an abstract version. (2) Wiki events: each cluster/class contains a set of news abstracts on the same story such as “2014 Crimean Crisis” crawled from Wikipedia current events following (Wu et al., 2015); this dataset offers more challenges because it has more finegrained classes and fewer documents (with shorter length) per class than the others have. It also shows more realistic nature of real-world applications such as news event clustering.
We also experiment with two long-text datasets and two domain-specific text datasets.
1BBCNews and BBCSport are downloaded from http://mlg.ucd.ie/datasets/bbc.html
(3) Reuters-21578: we obtain the original Reuters21578 text dataset and process as follows: remove documents with multiple categories, remove documents with empty body, remove duplicates, and select documents from the largest 10 categories. Reuters dataset is a highly unbalanced dataset (the top category has more than 3000 documents while the 10-th category has fewer than 100), this imbalance induces some extra randomness in comparing the results. (4) 20Newsgroups “bydate” version: we obtain the raw “bydate” versionand process them as follows: remove headers and footers, remove URLs and Email addresses, delete documents with less than 10 words. 20Newsgroups have roughly comparable sizes of categories. (5) BBCSports. (6) Ohsumed and Ohsumed-full: documents are medical abstracts from the MeSH categories of the year 1991. Specifically, there are 23 cardiovascular diseases categories.
Evaluating clustering results is known to be nontrivial. We use the following three sets of quantitative metrics to assess the quality of clusters by knowing the groundtruth categorical labels of documents: (1) Homogeneity, Completeness, and V-measure (Rosenberg and Hirschberg, 2007); (2) Adjusted Mutual Information (AMI) (Vinh et al., 2010); (3) Adjusted Rand Index (ARI) (Rand, 1971). For sensitivity analysis, we use the homogeneity score (Rosenberg and Hirschberg, 2007) as a projection dimension of other metrics, creating a 2D plot to visualize the metrics of a method along different homogeneity levels. Generally speaking, more clusters leads to higher homogeneity by chance.
4.2 Methods in Comparison
We examine four categories of methods that place a vector-space model over documents, and compare them to our D2-clustering framework. When needed, we use K-means++ to obtain clusters from dimension reduced vectors. To diminish the randomness brought by K-mean initialization, we ensemble the clustering results of 50 repeated runs (Strehl and Ghosh, 2003), and report the metrics for the ensembled one. The largest possible vocabulary used, excluding word embedding based approaches, is composed of words appearing in at least two documents. On each dataset, we select the same set of Ks, the number of clusters, for all methods. Typically, Ks are chosen around
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
the number of groundtruth categories in logarithmic scale.
We prepare two versions of the TF-IDF vectors as the unigram model. The ensembled K-means methods are used to obtain clusters. (1) TF-IDF vector (Sparck Jones, 1972). (2) TF-IDF-N vector is found by choosing the most frequent N words in a corpus, where N 2 {500, 1000, 1500, 2000}. The difference between the two methods highlights the sensitivity issue brought by the size of chosen vocabulary.
We also compare our approach with the following seven additional baselines. They are (3) Spectral Clustering (Laplacian) (4) Latent Semantic Indexing (LSI) (Deerwester et al., 1990). (5) Locality Preserving Projection (LPP) (He and Niyogi, 2004; Cai et al., 2005). (6) Nonnegative Matrix Factorization (NMF) (Lee and Seung, 1999; Xu et al., 2003). (7) Latent Dirichlet Allocation (LDA) (Blei et al., 2003; Hoffman et al., 2010; Pritchard et al., 2000). (8) Average of word vectors (AvgDoc) (9) Paragraph Vectors (PV) (Le and Mikolov, 2014). Details on their experimental setups and hyper-parameter search strategies can be found in the Appendix.
4.3 Runtime
We report the runtime for our approach upon two largest datasets. The experiments regarding other smaller datasets all finish within minutes in a single machine, which we omit due to page constraints. Like K-means, the running time spent by our approach depends on the number of actual iterations before a termination criterion is met. In the Newsgroups dataset, with m = 100 and K = 45, the time per iteration is 121 seconds on 48 processors. In Reuters dataset, with m = 100 and K = 20, the time per iteration is 190 seconds on 24 processors. Each run finishes in around tens of iterations typically, upon which the percentage of label changes is less than 0.1%.
Our approach adopts the Elkan’s algorithm pruning unnecessary computations of Wasserstein distance in assignment steps of K-means (Elkan, 2003). For the Newsgroups data (with m = 100 and K = 45), our approach terminates in 36 iterations, and totally computes 12, 162, 717 (⇡ 3.5% ⇥ 186122) distance pairs in assignment steps, saving 60% (⇡ 1 12,162,71736⇥45⇥18612 ) distance pairs to calculate in the standard D2-clustering. In comparison, the clustering approaches based on
K nearest neighbor graph with the prefetch-andprune method of (Kusner et al., 2015) needs substantially more pairs to compute Wasserstein distance, meanwhile the speed-ups also suffer from the curse of dimensionality. Their detailed statistics are reported in Table 1. Based on the results, our approach is much more practical as a basic document clustering tool.
4.4 Results
We summarize our numerical results in this section.
Regular text datasets. The first four datasets in Table 2 cover quite general and broad topics. We consider them to be regular and representative datasets encountered more frequently in applications. We report the clustering performances of the ten methods in Fig. 1, where three different metrics are plotted against the clustering homogeneity. The higher result at the same level of homogeneity is better, and the ability to achieve higher homogeneity is also welcomed. Clearly, D2-clustering is the only method that shows ro-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
bustly superior performances among all ten methods. Specifically, it ranks first in three datasets, and second in the other one. In comparison, LDA performs competitively on the “Reuters” dataset, but is substantially unsuccessful on others. Meanwhile, LPP performs competitively on the “Wiki events” and “Newsgroups” datasets, but it underperforms on other two. Laplacian, LSI, and TfidfN can achieve comparably performance if their reduced dimensions are fine tuned, which unfortunately is not realistic in practice. NMF is a simple and effective method which always gives stable, though subpar, performance.
Short texts vs. long texts. D2-clustering performs much more impressive on short texts (“BBC abstract” and “Wiki events”) than it does on long texts (“Reuters” and “Newsgroups”). This outcome is somewhat expected, because the bag-ofwords method suffers from high sparsity for short texts, and word-embedding based methods in theory should have an edge here. As shown in Fig. 1, D2-clustering has indeed outperformed other nonembedding approaches by a large margin on short texts (improved by about 40% and 20% respectively). Nevertheless, we find lifting from word embedding to document clustering is not a free
lunch. Neither AvgDoc nor PV can perform as competitively as D2-clustering performs on both. Domain-specific text datasets. We are also interested in how word embedding can help group domain-specific texts into clusters. In particular, does the semantic knowledge “embedded” in words provides enough clues to discriminate fine-grained concepts? We report the best AMI achieved by each method in Table 3. Our preliminary result indicates state-of-the-art word embeddings do not provide enough gain here to exceed the performance of existing methodologies. On the easy one, aka “BBCSport” dataset, basic bagof-words approach (Tfidf and Tfidf-N) already suffices to discriminate different sport categories; and on the hard one, aka “Ohsumed” dataset, D2clustering only slightly improves over Tf-idf and others, ranking behind LPP. Meanwhile, we feel the overall quality of clustering “Ohsumed” texts is quite far from useful in practice, no matter which method to use. (See next section for more discussions.)
4.5 Sensitivity to Word Embeddings.
We validate the robustness of D2 clustering with different word embedding models, and we also
7
612
613
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
662
663
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
show all their results in Fig. 2. As we mentioned, the effectiveness of Wasserstein document clustering depends on how relevant the utilized word embeddings are with the tasks. In those general document clustering tasks, however, word embedding models trained on general corpus perform robustly well with acceptably small variations. This outcome reveals our framework as generally effective and not dependent on a specific word embedding model. In addition, we also conduct experiments with word embeddings with a smaller dimension, aka 50 and 100. Their results are not as good as those we reported (therefore detailed numbers are not included due to space limit).
Inadequate embeddings may not be disastrous. In addition to our standard running set, we also try D2-clustering with purely random word embeddings, meaning each word vector is independently sampled from spherical Gaussian at 300 dimen-
ARI AMI V-measure BBCNews .146 .187 .190
abstract .792 +442% .759+306% .762+301%
Wiki events .194 .369 .463 .277 +43% .545+48% .611+32%
Reuters .498 .524 .588 .515 +3% .534+2% .594+1%
Newsgroups .194 .358 .390 .305 +57% .493+38% .499+28%
BBCSport .755 .740 .760 .801 +6% .812+10% .817+8%
Ohsumed .080 .204 .292 .116 +45% .260+27% .349+20%
Table 4: Comparison between random word embeddings (upper row) and meaningful pre-trained word embeddings (lower row), based on their best ARI, AMI, and V-measures. The improvements by percentiles are also shown in the subscripts.
8
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
sion, to see how deficient it can be. From our experimental results, random word embeddings degrade the performance of D2-clustering, but it still performs much better than purely random clustering does, and is even consistently better than LDA. Its performances across different datasets is highly correlated with the bag-of-words (Tfidf and TfidfN). By comparing a pre-trained word embedding model to a randomly generated one, we find that the extra gain is significant (> 10%) in clustering four of the six datasets. Their detailed statistics are in Table 4 and Fig. 3.
5 Discussions
Performance advantage. There has been one immediate observation from these studies, D2clustering always outperforms two of its degenerated cases, namely Tf-idf and AvgDoc, and three other popular methods: LDA, NMF, and PV, on all tasks. Therefore, for document clustering, users can expect to gain performance improvements by using our approach. Clustering sensitivity. From the four 2D plots in Fig. 1, we notice that the results of Laplacian, LSI and Tfidf-N are rather sensitive to their extra hyper-parameters. Once the vocabulary set, weight scheme and embeddings of words are fixed, our framework involves only two additional hyper-parameters: the number of intended clusters, K, and the selected support size of centroid distributions, m. We have chosen more than one m in all related experiments (m = {64, 100} for long documents, and m = 10, 20 for short documents). Our empirical experiments show that the effect of m on different metrics is less sensitive than the change of K. As shown in Fig. 1, results at different K are plotted for each method. The gray dots denote results of multiple runs of D2clustering. They are always contracted around the top-right region of the whole population, revealing
the predictive and robustly supreme performance of our approach. When bag-of-words suffices. Among the results of “BBCSport” dataset, Tfidf-N shows that by restricting the vocabulary set into a smaller one (which may be more relevant to the interest of tasks), it already can achieve highest clustering AMI without any other techniques. Other unsupervised regularization over data is likely unnecessary, or even degrades the performance slightly. Toward better word embeddings. Our experiments on the Ohsumed dataset have been limited. The result shows that it could be highly desirable to incorporate certain domain knowledge to derive more effective vector embeddings of words and phrases to encode their domainspecific knowledge, such as jargons that have knowledge dependencies and hierarchies in educational data mining, and signal words that capture multi-dimensional aspects of emotions in sentiment analysis.
Finally, we report the best AMIs of all methods on all datasets in Table 3. By looking at each method and the average of best AMIs over six datasets, we find our proposed clustering framework often performs competitively and robustly, which is the only method reaching more than 90% of the best AMI on each dataset. Furthermore, this observation holds for varying lengths of documents and varying difficulty levels of clustering tasks. Our nonparametric framework benefits from both bag-of-words and word embeddings.
6 Conclusions
This paper introduces a nonparametric clustering framework for document analysis. Its computational tractability, robustness and supreme performance, as a fundamental tool, are empirically validated. Its ease of use enables data scientists to use it for the pre-screening purpose of examining word embeddings in a specific task. Finally, the gains acquired from word embeddings are quantitatively measured from a nonparametric unsupervised perspective.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: Introduces  a new document clustering approach and compares it to several established methods, showing that it improves results in most cases. The analysis is very detailed and thorough--quite dense in many places and requires careful reading.  The presentation is organized and clear, and I am impressed by the range of comparisons and influential factors that were considered. Argument is convincing and the work should influence future approaches.  - Weaknesses:   The paper does not provide any information on the availability of the software described.  - General Discussion:  Needs some (minor) editing for English and typos--here are just a few:  Line 124: regardless the size > regardless of the size Line 126: resources. Because > resources, because Line 205: consist- ing mk > consisting of mk Line 360: versionand > version and