Determining Gains Acquired from Word Embedding Quantitatively using Discrete Distribution Clustering
Word embeddings have become widelyused in document analysis. A large number of models have been proposed, but the net gain these models can achieve expectably beyond the traditional bag-ofwords based approaches remains undetermined. Our empirical studies, conducted from a nonparametric unsupervised perspective, reveal where and how word embeddings can contribute to document analysis. Our approach is based on a recent algorithmic advance in nonparametric clustering for empirical measures, which neither invents nor relies on any document vector representations. The new document clustering approach proposed in this work is easy to use and stably outperforms other existing methodologies on a variety of document-clustering tasks.
1 000
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
1 Introduction
Word embeddings, or word vectors, have been broadly adopted for document analysis (Mikolov et al., 2013b,a). A key appeal of word embedding methods is that they can be obtained from external large-scale corpus and then be easily utilized for different data. Before choosing word embeddings for data analysis, researchers must first consider how much extra gain can be brought from the “embedded” knowledge of words in comparison with that achieved by existing bag-of-words based approaches. Moreover, they must also consider how to quantify that gain. Such a preliminary evaluation is often necessary before any further decisions can be made about the data.
Answering such questions is important: almost every model used in practice exploits some basic
representations — bag-of-words and word embeddings — for the sake of its computational tractability. Based on word embeddings, high-level models are designed for various tasks. Examples include entity representations, similarity measures, data manifolds, hierarchical structures, language models, and neural architectures. Therefore, it is important to investigate whether the gain or loss found in practice should be credited to the extra assumptions associated with those high-level models or to the use of basic word embeddings. As our experiments demonstrate, introducing these extra assumptions will make individual methods effective only if certain constraints are met. We will address this issue from an unsupervised perspective.
Our proposed clustering framework has several advantages. Instead of suppressing a document into a fixed-length vector feeding post-analysis, our framework uses the Wasserstein distance (or the Earth Mover’s Distance, EMD) as a metadistance to quantify the dissimilarity between two empirical nonparametric measures (or discrete distributions) over word embedding space (Wan, 2007; Kusner et al., 2015). Hence, it excludes any vector representation of the documents and sidesteps extra high-level assumptions, which is crucial and beneficial to evaluating the gain from basic word embeddings.
Our approach is intuitive and robust. the Wasserstein distance considers the cross-term relationship between different words in a principled fashion. As defined, the distance between two documents, say A and B, are the minimum cumulative cost that words from document A need to “travel” to match exactly the point cloud of document B. Here, the travel cost of a path between two words is their (squared) Euclidean distance in the word embedding space. Therefore, how much
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
benefit Wasserstein distance brings also depends on how the utilized word embedding space flattens document data as distributions, disentangling independent factors into different regions, subject to the interest of task.
While Wasserstein distance is well suited for document analysis, a major obstacle of EMDbased approaches is the high-magnitude computation involved, especially for the original D2clustering method (Li and Wang, 2008) — an EMD-based clustering framework. The main technical hurdle is to compute the Wasserstein barycenter, which is a discrete distribution, efficiently for a given set of discrete distributions. Thanks to the recent advance of algorithms for efficient solving of Wasserstein barycenters, one can now perform document clustering in a nonparametric way by directly treating them as empirical measures over word embedding space (Cuturi and Doucet, 2014; Ye and Li, 2014; Benamou et al., 2015; Ye et al., 2017). In short, the intuition behind such a nonparametric framework is: By keeping D2-clustering to a minimal assumption set, we can achieve results of the highest possible clustering quality regardless the size and patterns of data. Obtaining highest quality clustering of unstructured text data merits consuming extra computational resources. Because it is a crucial step in many techniques and applications of natural language processing, such as cross-document co-reference resolution (Singh et al., 2011), document summarization (Radev et al., 2004; Wang and Li, 2010), retrospective events detection (Yang et al., 1998), and opinion mining (Zhai et al., 2011).
Our contributions. Our work has two main contributions. First, we create a basic tool of document clustering with mere hyper-parameters at scale. Our tool leverages the state-of-the-art numerical toolbox developed for optimal transport to achieve computational feasibility. Meanwhile, it gives state-of-the-art clustering performances across heterogeneous text data — an advantage over other methods in the literature. Second, with our tool, one can quantitatively inspect how well a word-embedding model can fit the data and how much gain or loss will be obtained compared to traditional bag-of-words models. Acquiring insights on these questions is valuable for document analysis beyond clustering. The exploration of D2-clustering for documents will provide a win-
dow for investigating these questions.
2 Related Work
In the original D2-clustering framework (Li and Wang, 2008), calculating Wasserstein barycenter involves solving a large-scale LP problem at each inner iteration, severely limiting the scalability and robustness of the framework. Such high magnitude of computations had prohibited it from many real-world applications until recently. To accelerate the computation of Wasserstein barycenter, and ultimately to improve D2clustering, multiple numerical algorithmic efforts have been made in the recent few years (Cuturi and Doucet, 2014; Ye and Li, 2014; Benamou et al., 2015; Ye et al., 2017).
Although the effectiveness of Wasserstein distance has been well recognized in the computer vision and multimedia literature, the property of Wasserstein barycenter has not been well understood. To our knowledge, there still lacks systematic study of applying Wasserstein barycenter and D2-clustering in document analysis with word embeddings.
A closely related work to ours was proposed by authors of (Kusner et al., 2015) who recently connected the Wasserstein distance to the word embeddings for comparing documents. Our work differs from theirs in the methodology. We directly pursue a scalable clustering setting rather than construct a nearest neighbor graph based on calculated distances, because the calculation of the Wasserstein distances of all pairs is too expensive to be practical. The authors of (Kusner et al., 2015) used a lower bound that was cheaper to compute in order to prune unnecessary full distance calculation, but the scalability of this modified approach is still considered very limited, an issue to be discussed in Section 4.3. On the other hand, our approach adopts the framework similar to K-means which is of complexity O(n) per iteration, and usually converges within tens of iterations. The computation of D2-clustering, though in its original form was magnitudes heavier than other document clustering methods, can now be done with efficient parallelization and proper implementations (Ye et al., 2017).
3 Our Approach
This section introduces the distance, the D2clustering technique, the fast computation frame-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
work, and how they are used in the proposed document clustering method.
3.1 Wasserstein Distance
Suppose we represent each document d k consisting m
k unique words by a discrete measure or a discrete distribution, where k = 1, . . . , N with N being the sample size:
d k =
X mk
i=1 w(k) i x (k) i . (1)
Here x denotes the Dirac measure with support x, and w(k)
i 0 is the “importance weight” for the i-th word in the k-th document, with P mk i=1w (k) i = 1. And x(k) i
2 Rd, called a support point, is the semantic embedding vector of the i-th word. The 2nd-order Wasserstein distance between two documents d1 and d2 (and likewise for any document pairs) is defined by the following LP problem: W 2(d1, d2) :=
min
⇧
P i,j ⇡ i,j kx(1) i x(2) j
k22 s.t. P m2 j=1 ⇡i,j = wi, 8i, P m1 i=1 ⇡i,j = wj , 8j
⇡ i,j 0, 8i, j , (2)
where ⇧ = {⇡ i,j } is a m1 ⇥ m2 coupling matrix, and let {C
i,j := kx(1) i x(2) j k22} be transportation costs between words. Wasserstein distance is a true metric (Villani, 2003) for measures, and its best exact algorithm has a complexity of O(m3 logm) (Pele and Werman, 2009; Cuturi, 2013), if m1 = m2 = m.
3.2 Discrete Distribution (D2-) Clustering
D2-clustering (Li and Wang, 2008) iterates between the assignment step and centroids updating step in a similar way as the Lloyd’s K-means. Suppose we are to find K clusters. The assignment step finds each member distribution its nearest mean from K candidates. The mean of each cluster is again a discrete distribution with m support points, denoted by c
i , i = 1, . . . ,K. Each mean is iteratively updated to minimize its total within cluster variation. We can write the D2clustering problem as follows: given sample data {d
k }N k=1, support size of means m, and desired
number of clusters K, D2-clustering solves
min
c1,...,cK
X N
k=1 min 1iK W 2(d k , c i ) , (3)
where c1, . . . , cK are Wasserstein barycenters. At the core of solving the above formulation is an optimization method that searches the Wasserstein barycenters of varying partitions. Therefore, we concentrate on the following problem. For each cluster, we reorganize the index of member distributions from 1, . . . , n. The Wasserstein barycenter (Agueh and Carlier, 2011; Cuturi and Doucet, 2014) is by definition the solution of
min
c
X n
k=1 W 2(d k , c) , (4)
where c = P m
i=1wi xi . The above Wasserstein barycenter formulation involves two levels of optimization: the outer level finding the minimizer of total variations, and the inner level solving Wasserstein distances. We remark that in D2clustering, we need to solve multiple Wasserstein barycenters rather than a single one. This constitutes the third level of optimization.
3.3 Modified Bregman ADMM for Computing Wasserstein Barycenter
The recent modified Bregman alternating direction method of multiplier (B-ADMM) algorithm (Ye et al., 2017), motivated by (Wang and Banerjee, 2014), is a practical choice for computing Wasserstein barycenters. We briefly sketch their algorithmic procedure of this optimization method here for the sake of completeness. To solve for Wasserstein barycenter defined in Eq. (4), the key procedure of the modified Bregman ADMM involves iterative updates of four block of primal variables: the support points of c — {x
i }m i=1 (with trans-
portation costs {C i,j }(k) for k = 1, . . . , n), the importance weights of c — {w
i }m i=1, and two sets of
split matching variables — {⇡(k,1) i,j } and {⇡(k,2) i,j }, for k = 1, . . . , n, as well as Lagrangian variables { (k)
i,j } for k = 1, . . . , n. In the end, both {⇡(k,1) i,j } and {⇡(k,2)
i,j } converge to the matching weight in Eq. (2) with respect to d(c, d
k ). The iterative algorithm proceeds as follows until c converges or a maximum number of iterations are reached: given constant ⌧ 10, ⇢ / P i,j,k C(k) i,jP
n k=1mkm and round-off
tolerance ✏ = 10 10, those variables are updated in the following order. Update {x
i }m i=1 and {C (k) i,j } in every ⌧ iterations:
x i :=
1
nw i
X n
k=1
X mk
j=1 ⇡(k,1) i,j x(k) j , 8i, (5)
C(k) i,j := kx i x(k) j k22, 8i, j and k, (6)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Update {⇡(k,1) i,j } and {⇡(k,2) i,j }. For each i, j and k,
⇡(k,2) i,j := ⇡(k,2) i,j exp
C(k)
i,j
(k) i,j
⇢
! + ✏ (7)
⇡(k,1) i,j := w(k) j ⇡(k,2) i,j
.⇣X m
l=1 ⇡(k,2) l,j
⌘ (8)
⇡(k,1) i,j := ⇡(k,1) i,j exp ⇣ (k) i,j /⇢ ⌘ + ✏ (9)
Update {w i }m i=1. For i = 1, . . . ,m ,
w i :=
nX
k=1
P mk j=1 ⇡ (k,1) i,j
P i,j ⇡(k,1) i,j
(10)
w i := w i
.⇣X m
i=1 w i
⌘ (11)
Update {⇡(k,2) i,j } and { (k) i,j }. For each i, j and k,
⇡(k,2) i,j := w i ⇡(k,1) i,j
.⇣X mk
l=1 ⇡(k,1) i,l
⌘ (12)
(k) i,j := (k) i,j + ⇢ ⇣ ⇡(k,1) i,l ⇡(k,2) i,l ⌘ . (13)
Eq. (5)-(13) can all be vectorized as very efficient numerical routines. In a data parallel implementation, only Eq. (5) and Eq. (10) (involving P n
k=1) needs to be synchronized. The software package detailed in (Ye et al., 2017) was used to generate relevant experiments. We make available our codes and pre-processed datasets for reproducing all experiments of our approach.
4 Experiments
4.1 Datasets and Evaluation Metrics
We prepare six datasets to conduct a set of experiments. Two short-text datasets are created as follows. (1) BBCNews abstract: we concatenate the title and the first sentence of news posts from BBCNews dataset1 to create an abstract version. (2) Wiki events: each cluster/class contains a set of news abstracts on the same story such as “2014 Crimean Crisis” crawled from Wikipedia current events following (Wu et al., 2015); this dataset offers more challenges because it has more finegrained classes and fewer documents (with shorter length) per class than the others have. It also shows more realistic nature of real-world applications such as news event clustering.
We also experiment with two long-text datasets and two domain-specific text datasets.
1BBCNews and BBCSport are downloaded from http://mlg.ucd.ie/datasets/bbc.html
(3) Reuters-21578: we obtain the original Reuters21578 text dataset and process as follows: remove documents with multiple categories, remove documents with empty body, remove duplicates, and select documents from the largest 10 categories. Reuters dataset is a highly unbalanced dataset (the top category has more than 3000 documents while the 10-th category has fewer than 100), this imbalance induces some extra randomness in comparing the results. (4) 20Newsgroups “bydate” version: we obtain the raw “bydate” versionand process them as follows: remove headers and footers, remove URLs and Email addresses, delete documents with less than 10 words. 20Newsgroups have roughly comparable sizes of categories. (5) BBCSports. (6) Ohsumed and Ohsumed-full: documents are medical abstracts from the MeSH categories of the year 1991. Specifically, there are 23 cardiovascular diseases categories.
Evaluating clustering results is known to be nontrivial. We use the following three sets of quantitative metrics to assess the quality of clusters by knowing the groundtruth categorical labels of documents: (1) Homogeneity, Completeness, and V-measure (Rosenberg and Hirschberg, 2007); (2) Adjusted Mutual Information (AMI) (Vinh et al., 2010); (3) Adjusted Rand Index (ARI) (Rand, 1971). For sensitivity analysis, we use the homogeneity score (Rosenberg and Hirschberg, 2007) as a projection dimension of other metrics, creating a 2D plot to visualize the metrics of a method along different homogeneity levels. Generally speaking, more clusters leads to higher homogeneity by chance.
4.2 Methods in Comparison
We examine four categories of methods that place a vector-space model over documents, and compare them to our D2-clustering framework. When needed, we use K-means++ to obtain clusters from dimension reduced vectors. To diminish the randomness brought by K-mean initialization, we ensemble the clustering results of 50 repeated runs (Strehl and Ghosh, 2003), and report the metrics for the ensembled one. The largest possible vocabulary used, excluding word embedding based approaches, is composed of words appearing in at least two documents. On each dataset, we select the same set of Ks, the number of clusters, for all methods. Typically, Ks are chosen around
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
the number of groundtruth categories in logarithmic scale.
We prepare two versions of the TF-IDF vectors as the unigram model. The ensembled K-means methods are used to obtain clusters. (1) TF-IDF vector (Sparck Jones, 1972). (2) TF-IDF-N vector is found by choosing the most frequent N words in a corpus, where N 2 {500, 1000, 1500, 2000}. The difference between the two methods highlights the sensitivity issue brought by the size of chosen vocabulary.
We also compare our approach with the following seven additional baselines. They are (3) Spectral Clustering (Laplacian) (4) Latent Semantic Indexing (LSI) (Deerwester et al., 1990). (5) Locality Preserving Projection (LPP) (He and Niyogi, 2004; Cai et al., 2005). (6) Nonnegative Matrix Factorization (NMF) (Lee and Seung, 1999; Xu et al., 2003). (7) Latent Dirichlet Allocation (LDA) (Blei et al., 2003; Hoffman et al., 2010; Pritchard et al., 2000). (8) Average of word vectors (AvgDoc) (9) Paragraph Vectors (PV) (Le and Mikolov, 2014). Details on their experimental setups and hyper-parameter search strategies can be found in the Appendix.
4.3 Runtime
We report the runtime for our approach upon two largest datasets. The experiments regarding other smaller datasets all finish within minutes in a single machine, which we omit due to page constraints. Like K-means, the running time spent by our approach depends on the number of actual iterations before a termination criterion is met. In the Newsgroups dataset, with m = 100 and K = 45, the time per iteration is 121 seconds on 48 processors. In Reuters dataset, with m = 100 and K = 20, the time per iteration is 190 seconds on 24 processors. Each run finishes in around tens of iterations typically, upon which the percentage of label changes is less than 0.1%.
Our approach adopts the Elkan’s algorithm pruning unnecessary computations of Wasserstein distance in assignment steps of K-means (Elkan, 2003). For the Newsgroups data (with m = 100 and K = 45), our approach terminates in 36 iterations, and totally computes 12, 162, 717 (⇡ 3.5% ⇥ 186122) distance pairs in assignment steps, saving 60% (⇡ 1 12,162,71736⇥45⇥18612 ) distance pairs to calculate in the standard D2-clustering. In comparison, the clustering approaches based on
K nearest neighbor graph with the prefetch-andprune method of (Kusner et al., 2015) needs substantially more pairs to compute Wasserstein distance, meanwhile the speed-ups also suffer from the curse of dimensionality. Their detailed statistics are reported in Table 1. Based on the results, our approach is much more practical as a basic document clustering tool.
4.4 Results
We summarize our numerical results in this section.
Regular text datasets. The first four datasets in Table 2 cover quite general and broad topics. We consider them to be regular and representative datasets encountered more frequently in applications. We report the clustering performances of the ten methods in Fig. 1, where three different metrics are plotted against the clustering homogeneity. The higher result at the same level of homogeneity is better, and the ability to achieve higher homogeneity is also welcomed. Clearly, D2-clustering is the only method that shows ro-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
bustly superior performances among all ten methods. Specifically, it ranks first in three datasets, and second in the other one. In comparison, LDA performs competitively on the “Reuters” dataset, but is substantially unsuccessful on others. Meanwhile, LPP performs competitively on the “Wiki events” and “Newsgroups” datasets, but it underperforms on other two. Laplacian, LSI, and TfidfN can achieve comparably performance if their reduced dimensions are fine tuned, which unfortunately is not realistic in practice. NMF is a simple and effective method which always gives stable, though subpar, performance.
Short texts vs. long texts. D2-clustering performs much more impressive on short texts (“BBC abstract” and “Wiki events”) than it does on long texts (“Reuters” and “Newsgroups”). This outcome is somewhat expected, because the bag-ofwords method suffers from high sparsity for short texts, and word-embedding based methods in theory should have an edge here. As shown in Fig. 1, D2-clustering has indeed outperformed other nonembedding approaches by a large margin on short texts (improved by about 40% and 20% respectively). Nevertheless, we find lifting from word embedding to document clustering is not a free
lunch. Neither AvgDoc nor PV can perform as competitively as D2-clustering performs on both. Domain-specific text datasets. We are also interested in how word embedding can help group domain-specific texts into clusters. In particular, does the semantic knowledge “embedded” in words provides enough clues to discriminate fine-grained concepts? We report the best AMI achieved by each method in Table 3. Our preliminary result indicates state-of-the-art word embeddings do not provide enough gain here to exceed the performance of existing methodologies. On the easy one, aka “BBCSport” dataset, basic bagof-words approach (Tfidf and Tfidf-N) already suffices to discriminate different sport categories; and on the hard one, aka “Ohsumed” dataset, D2clustering only slightly improves over Tf-idf and others, ranking behind LPP. Meanwhile, we feel the overall quality of clustering “Ohsumed” texts is quite far from useful in practice, no matter which method to use. (See next section for more discussions.)
4.5 Sensitivity to Word Embeddings.
We validate the robustness of D2 clustering with different word embedding models, and we also
7
612
613
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
662
663
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
show all their results in Fig. 2. As we mentioned, the effectiveness of Wasserstein document clustering depends on how relevant the utilized word embeddings are with the tasks. In those general document clustering tasks, however, word embedding models trained on general corpus perform robustly well with acceptably small variations. This outcome reveals our framework as generally effective and not dependent on a specific word embedding model. In addition, we also conduct experiments with word embeddings with a smaller dimension, aka 50 and 100. Their results are not as good as those we reported (therefore detailed numbers are not included due to space limit).
Inadequate embeddings may not be disastrous. In addition to our standard running set, we also try D2-clustering with purely random word embeddings, meaning each word vector is independently sampled from spherical Gaussian at 300 dimen-
ARI AMI V-measure BBCNews .146 .187 .190
abstract .792 +442% .759+306% .762+301%
Wiki events .194 .369 .463 .277 +43% .545+48% .611+32%
Reuters .498 .524 .588 .515 +3% .534+2% .594+1%
Newsgroups .194 .358 .390 .305 +57% .493+38% .499+28%
BBCSport .755 .740 .760 .801 +6% .812+10% .817+8%
Ohsumed .080 .204 .292 .116 +45% .260+27% .349+20%
Table 4: Comparison between random word embeddings (upper row) and meaningful pre-trained word embeddings (lower row), based on their best ARI, AMI, and V-measures. The improvements by percentiles are also shown in the subscripts.
8
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
sion, to see how deficient it can be. From our experimental results, random word embeddings degrade the performance of D2-clustering, but it still performs much better than purely random clustering does, and is even consistently better than LDA. Its performances across different datasets is highly correlated with the bag-of-words (Tfidf and TfidfN). By comparing a pre-trained word embedding model to a randomly generated one, we find that the extra gain is significant (> 10%) in clustering four of the six datasets. Their detailed statistics are in Table 4 and Fig. 3.
5 Discussions
Performance advantage. There has been one immediate observation from these studies, D2clustering always outperforms two of its degenerated cases, namely Tf-idf and AvgDoc, and three other popular methods: LDA, NMF, and PV, on all tasks. Therefore, for document clustering, users can expect to gain performance improvements by using our approach. Clustering sensitivity. From the four 2D plots in Fig. 1, we notice that the results of Laplacian, LSI and Tfidf-N are rather sensitive to their extra hyper-parameters. Once the vocabulary set, weight scheme and embeddings of words are fixed, our framework involves only two additional hyper-parameters: the number of intended clusters, K, and the selected support size of centroid distributions, m. We have chosen more than one m in all related experiments (m = {64, 100} for long documents, and m = 10, 20 for short documents). Our empirical experiments show that the effect of m on different metrics is less sensitive than the change of K. As shown in Fig. 1, results at different K are plotted for each method. The gray dots denote results of multiple runs of D2clustering. They are always contracted around the top-right region of the whole population, revealing
the predictive and robustly supreme performance of our approach. When bag-of-words suffices. Among the results of “BBCSport” dataset, Tfidf-N shows that by restricting the vocabulary set into a smaller one (which may be more relevant to the interest of tasks), it already can achieve highest clustering AMI without any other techniques. Other unsupervised regularization over data is likely unnecessary, or even degrades the performance slightly. Toward better word embeddings. Our experiments on the Ohsumed dataset have been limited. The result shows that it could be highly desirable to incorporate certain domain knowledge to derive more effective vector embeddings of words and phrases to encode their domainspecific knowledge, such as jargons that have knowledge dependencies and hierarchies in educational data mining, and signal words that capture multi-dimensional aspects of emotions in sentiment analysis.
Finally, we report the best AMIs of all methods on all datasets in Table 3. By looking at each method and the average of best AMIs over six datasets, we find our proposed clustering framework often performs competitively and robustly, which is the only method reaching more than 90% of the best AMI on each dataset. Furthermore, this observation holds for varying lengths of documents and varying difficulty levels of clustering tasks. Our nonparametric framework benefits from both bag-of-words and word embeddings.
6 Conclusions
This paper introduces a nonparametric clustering framework for document analysis. Its computational tractability, robustness and supreme performance, as a fundamental tool, are empirically validated. Its ease of use enables data scientists to use it for the pre-screening purpose of examining word embeddings in a specific task. Finally, the gains acquired from word embeddings are quantitatively measured from a nonparametric unsupervised perspective.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
- Strengths: Introduces  a new document clustering approach and compares it to several established methods, showing that it improves results in most cases. The analysis is very detailed and thorough--quite dense in many places and requires careful reading.  The presentation is organized and clear, and I am impressed by the range of comparisons and influential factors that were considered. Argument is convincing and the work should influence future approaches.  - Weaknesses:   The paper does not provide any information on the availability of the software described.  - General Discussion:  Needs some (minor) editing for English and typos--here are just a few:  Line 124: regardless the size > regardless of the size Line 126: resources. Because > resources, because Line 205: consist- ing mk > consisting of mk Line 360: versionand > version and