Latent Support Measure Machines for Bag-of-Words Data Classification
In many classification problems, the input is represented as a set of features, e.g., the bag-of-words (BoW) representation of documents. Support vector machines (SVMs) are widely used tools for such classification problems. The performance of the SVMs is generally determined by whether kernel values between data points can be defined properly. However, SVMs for BoW representations have a major weakness in that the co-occurrence of different but semantically similar words cannot be reflected in the kernel calculation. To overcome the weakness, we propose a kernel-based discriminative classifier for BoW data, which we call the latent support measure machine (latent SMM). With the latent SMM, a latent vector is associated with each vocabulary term, and each document is represented as a distribution of the latent vectors for words appearing in the document. To represent the distributions efficiently, we use the kernel embeddings of distributions that hold high order moment information about distributions. Then the latent SMM finds a separating hyperplane that maximizes the margins between distributions of different classes while estimating latent vectors for words to improve the classification performance. In the experiments, we show that the latent SMM achieves state-of-the-art accuracy for BoW text classification, is robust with respect to its own hyper-parameters, and is useful to visualize words.
Latent Support Measure Machines for Bag-of-Words Data ClassiﬁcationThis paper presents a kernel-based latent support measure machine for bag-of-word data to jointly learn word embedding and a document classifier. It assumes each word can be represented as a latent vector and each document is represented as a distribution of the latent vectors associated with the words in the document. Then latent support measure machine learns a model to classify documents based on the latent distribution. The proposed method is a nice extension of support measure machines. Experiments show that the proposed method outperforms other methods using word embedding or word cluster.   The paper is nice written and easy to follow. Comments:	- My main concern is the scalability of the proposed method. All the experiments are conducted using less than 1,000 examples. However, it is easy to encounter a bag-of-word data in the scale of hundred thousand in text classification problems. It is unclear if the proposed method can scale up to deal with such datasets.	- The paper demonstrates promising results using a small dataset. However, it is hard to judge if Latent SMM can improve the accuracy of text classification when the training samples are sufficient. For example, new20 achieves 0.82 accuracy using SVM with linear kernel when training on the whole dataset (http://web.ist.utl.pt/acardoso/datasets/). The accuracy of the best method trained on 1,000 samples reported in Figure 1 (c) is around 0.6. It would be more promising to report the results on the whole dataset as well.	- I wonder if the proposed method can be extended for other applications.	- In figures 4, it seems that the method performs better when using a large C to fit training data. Is there an explanation for that?Minor comments/typos:	- Lines 361, 364: should be Figure 3, Figure 4, respectively.I like the experimental analysis about the word occurrence threshold (line 356~363). However, it is hard to see the performance difference with different thresholds in Figure 3.