In this paper, we propose a novel representation for text documents based on aggregating word embedding vectors into document embeddings. Our approach is inspired by the Vector of Locally-Aggregated Descriptors used for image representation, and it works as follows. First, the word embeddings gathered from a collection of documents are clustered by k-means in order to learn a codebook of semnatically-related word embeddings. Each word embedding is then associated to its nearest cluster centroid (codeword). The Vector of Locally-Aggregated Word Embeddings (VLAWE) representation of a document is then computed by accumulating the differences between each codeword vector and each word vector (from the document) associated to the respective codeword. We plug the VLAWE representation, which is learned in an unsupervised manner, into a classifier and show that it is useful for a diverse set of text classification tasks. We compare our approach with a broad range of recent state-of-the-art methods, demonstrating the effectiveness of our approach. Furthermore, we obtain a considerable improvement on the Movie Review data set, reporting an accuracy of 93 . 3% , which represents an absolute gain of 10% over the state-of-the-art approach. Our code is available at https://github.com/raduionescu/vlawe-boswe/. In recent years, word embeddings (Bengio et al., 2003; Collobert and Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) have had a huge impact in natural language processing (NLP) and related fields, being used in many tasks including sentiment analysis (Dos Santos and Gatti, 2014; Fu et al., 2018), information retrieval (Clin-chant and Perronnin, 2013; Ye et al., 2016) and word sense disambiguation (Bhingardive et al., 2015; Butnaru et al., 2017; Chen et al., 2014; Ia-cobacci et al., 2016), among many others. Starting from word embeddings, researchers proposed various ways of aggregating word embedding vectors to obtain efficient sentence-level or document-level representations (Butnaru and Ionescu, 2017; Cheng et al., 2018; Clinchant and Perronnin, 2013; Conneau et al., 2017; Cozma et al., 2018; Fu et al., 2018; Hill et al., 2016; Kiros et al., 2015; Kusner et al., 2015; Le and Mikolov, 2014; Shen et al., 2018; Torki, 2018; Zhao et al., 2015; Zhou et al., 2016, 2018). Although the mean (or sum) of word vectors is commonly adopted because of its simplicity (Mitchell and Lapata, 2010), it seems that more complex approaches usually yield better performance (Cheng et al., 2018; Conneau et al., 2017; Cozma et al., 2018; Fu et al., 2018; Hill et al., 2016; Kiros et al., 2015; Torki, 2018; Zhao et al., 2015; Zhou et al., 2016, 2018). To this end, we propose a simple yet effective approach for aggregating word embeddings into document embeddings. Our approach is inspired by the Vector of Locally-Aggregated Descriptors (VLAD) (Jegou et al., 2010, 2012) used in computer vision to efficiently represent images for various image classification and retrieval tasks. To our knowledge, we are the first to adapt and use VLAD in the text domain. Our document-level representation is constructed as follows. First, we apply a pre-trained word embedding model, such as GloVe (Penning-ton et al., 2014), on all the words from a set of training documents in order to obtain a set of training word vectors. The word vectors are clustered by k-means in order to learn a codebook of semnatically-related word embeddings. Each word embedding is then associated to its nearest cluster centroid (codeword). The Vector of Locally-Aggregated Word Embeddings (VLAWE) representation of a text document is then computed by accumulating the differences between each codeword vector and each word vector that is both present in the document and associated to the respective codeword. Since our approach considers cluster centroids as reference for building the representation, it can easily accommodate new words, not seen during k-means training, simply by associating them to the nearest cluster centroids. Thus, VLAWE is robust to vocabulary distribution gaps between training and test, which can appear when the training set is particularly smaller or from a different domain. Certainly, the robustness holds as long as the word embeddings are pre-trained on a very large set of documents, e.g. the entire Wikipedia. We plug the VLAWE representation, which is learned in an unsupervised manner, into a classifier, namely Support Vector Machines (SVM), and show that it is useful for a diverse set of text classification tasks. We consider five benchmark data sets: Reuters-21578 (Lewis, 1997), RT-2k (Pang and Lee, 2004), MR (Pang and Lee, 2005), TREC (Li and Roth, 2002) and Subj (Pang and Lee, 2004). We compare VLAWE with recent state-of-the-art methods (Butnaru and Ionescu, 2017; Cheng et al., 2018; Fu et al., 2018; Hill et al., 2016; Iyyer et al., 2015; Kim, 2014; Kiros et al., 2015; Le and Mikolov, 2014; Liu et al., 2017; Shen et al., 2018; Torki, 2018; Xue and Zhou, 2009; Zhao et al., 2015; Zhou et al., 2016, 2018), demonstrating the effectiveness of our approach. Furthermore, we obtain a considerable improvement on the Movie Review (MR) data set, surpassing the state-of-the-art approach of Cheng et al. (2018) by almost 10% . The rest of the paper is organized as follows. We present related works on learning document-level representations in Section 2. We describe the Vector of Locally-Aggregated Word Embeddings in Section 3. We present experiments and results on various text classification tasks in Section 4. Finally, we draw our conclusion in Section 5. 2 Related Work There are various works (Butnaru and Ionescu, 2017; Cheng et al., 2018; Conneau et al., 2017; Fu et al., 2018; Hill et al., 2016; Iyyer et al., 2015; Kim, 2014; Kiros et al., 2015; Kusner et al., 2015; Le and Mikolov, 2014; Clinchant and Perronnin, 2013; Shen et al., 2018; Torki, 2018; Zhao et al., 2015; Zhou et al., 2018) that propose to build effective sentence-level or document-level representations based on word embeddings. While most of these approaches are based on deep learning (Cheng et al., 2018; Conneau et al., 2017; Hill et al., 2016; Iyyer et al., 2015; Kim, 2014; Kiros et al., 2015; Le and Mikolov, 2014; Zhao et al., 2015; Zhou et al., 2018), there have been some approaches that are inspired by computer vision research, namely by the bag-of-visual-words (But-naru and Ionescu, 2017) and by Fisher Vectors (Clinchant and Perronnin, 2013). The relationship between the bag-of-visual-words, Fisher Vectors and VLAD is discussed in (Jegou et al., 2012). The discussion can be transferred to describe the relantionship of our work and the closely-related works of Butnaru and Ionescu (2017) and Clinchant and Perronnin (2013). The Vector of Locally-Aggregated Descriptors (VLAD) (J  egou et al., 2010, 2012) was introduced in computer vision to efficiently represent images for various image classification and retrieval tasks. We propose to adapt the VLAD representation in order to represent text documents instead of images. Our adaptation consists of replacing the Scale-Invariant Feature Transform (SIFT) image descriptors (Lowe, 2004) useful for recognizing object patterns in images with word embeddings (Mikolov et al., 2013; Pennington et al., 2014) useful for recognizing semantic patterns in text documents. We coin the term Vector of Locally-Aggregated Word Embeddings (VLAWE) for the resulting document representation. The VLAWE representation is derived as follows. First, each word in the collection of training documents is represented as a word vector using a pre-trained word embeddings model. The result is a set X = { x 1 , x 2 , ..., x n } of n word vectors. As for the VLAD model, the next step is to learn a codebook {  1 ,  2 , ...,  k } of representative meta-word vectors (codewords) using k-means. Each codeword  i is the centroid of the cluster C i  X :  i = 1 | C i | (cid:88) x t  C i x t ,  i  { 1 , 2 , ..., k } , (1) where | C i | is the number of word vectors assigned to cluster C i and k is the number of clusters. Since word embeddings carry semantic information by projecting semantically-related words in the same region of the embedding space, it means that the resulting clusters contain semantically-related words. The formed centroids are stored in a randomized forest of k-d trees to reduce search cost, as described in (Philbin et al., 2007; Ionescu et al., 2013; Ionescu and Popescu, 2014, 2015a). Each word embedding x t is associated to a single cluster C i , such that the Euclidean distance between x t and the corresponding codeword  i is minimum, for all i  { 1 , 2 , ..., k } . For each document D and each codeword  i , the differences x t   i of the vectors x t  C i  D and the codeword  i are accumulated into column vectors: v i,D = (cid:88) x t  C i  D x t   i , (2) where D  X is the set of word embeddings in a given text document. The final VLAWE embedding for a given document D is obtained by stacking together the d -dimensional residual vectors v i,D , where d is equal to the dimension of the word embeddings:  D =   v 1 ,D v 2 ,D ... v k,D   . Therefore, the VLAWE document embedding is has k  d components. The VLAWE vector  D undergoes two normalization steps. First, a power normalization is performed by applying the following operator independently on each component (element): f ( z ) = sign ( z )  | z |  , (4) where 0    1 and | z | is the absolute value of z . Since words in natural language follow the Zipf's law (Powers, 1998), it seems natural to apply the power normalization in order to reduce the influence of highly frequent words, e.g. common words or stopwords, which can corrupt the representation. As Jegou et al. (2012), we empirically observed that this step consistently improves the quality of the representation. The power normalized document embeddings are then L 2 -normalized. After obtaining the normalized VLAWE representations, we employ a classification method to learn a discriminative model for each specific text classification task. We exhibit the performance of VLAWE on five public data sets: Reuters-21578 (Lewis, 1997), RT-2k (Pang and Lee, 2004), MR (Pang and Lee, 2005), TREC (Li and Roth, 2002) and Subj (Pang and Lee, 2004). The Reuters-21578 data set (Lewis, 1997) contains articles collected from Reuters newswire. Following Joachims (1998) and Yang and Liu (1999), we select the categories (topics) that have at least one document in the training set and one in the test set, leading to a total of 90 categories. We use the ModeApte evaluation (Xue and Zhou, 2009), in which unlabeled documents are eliminated, leaving a total of 10787 documents. The collection is already divided into 7768 documents for training and 3019 documents for testing. The RT-2k data set (Pang and Lee, 2004) consists of 2000 movie reviews taken from the IMDB movie review archives. There are 1000 positive reviews rated with four or five stars, and 1000 negative reviews rated with one or two stars. The task is to discriminate between positive and negative reviews. The Movie Review (MR) data set (Pang and Lee, 2005) consists of 5331 positive and 5331 negative sentences. Each sentence is selected from one movie review. The task is to discriminate between positive and negative sentiment. TREC (Li and Roth, 2002) is a question type classification data set, where questions are divided into 6 classes. The collection is already divided into 5452 questions for training and 500 questions for testing. The Subjectivity (Subj) (Pang and Lee, 2004) data set contains 5000 objective and 5000 subjective sentences. The task is to classify a sentence as being either subjective or objective. In the experiments, we used the pre-trained word embeddings computed with the GloVe toolkit provided by Pennington et al. (2014). The pre-trained GloVe model contains 300 -dimensional vectors for 2 . 2 million words and phrases. Most of the steps required for building the VLAWE representation, such as the k-means clustering and the randomized forest of k-d trees, are implemented using the VLFeat library (Vedaldi and Fulkerson, 2008). We set the number of clusters (size of the codebook) to k = 10 , leading to a VLAWE representation of k  d = 10  300 = 3000 components. Similar to Jegou et al. (2012), we set  = 0 . 5 for the power normalization step in Equation (4), which consistently leads to near-optimal results on all data sets. In the learning stage, we employ the Method Reuters-21578 RT-2k MR TREC Subj Average of word embeddings (baseline) 85 . Support Vector Machines (SVM) implementation provided by LibSVM (Chang and Lin, 2011). We set the SVM regularization parameter to C = 1 in all our experiments. In the SVM, we use the linear kernel. For optimal results, the VLAWE representation is combined with the BOSWE representation (Butnaru and Ionescu, 2017), which is based on the PQ kernel (Ionescu and Popescu, 2013, 2015b). We follow the same evaluation procedure as Kiros et al. (2015) and Hill et al. (2016), using 10 fold cross-validation when a train and test split is not pre-defined for a given data set. As evaluation metrics, we employ the micro-averaged F 1 measure for the Reuters-21578 data set and the standard classification accuracy for the RT-2k, the MR, the TREC and the Subj data sets, in order to fairly compare with the related art. 4.3 Results We compare VLAWE with several state-of-the-art methods (Butnaru and Ionescu, 2017; Cheng et al., 2018; Fu et al., 2018; Hill et al., 2016; Iyyer et al., 2015; Kim, 2014; Kiros et al., 2015; Le and Mikolov, 2014; Liu et al., 2017; Shen et al., 2018; Torki, 2018; Xue and Zhou, 2009; Zhao et al., 2015; Zhou et al., 2016, 2018) as well as two baseline methods, namely the average of word embeddings and the standard bag-of-words (BOW). The corresponding results are presented in Table 1. First, we notice that our approach outperforms both baselines on all data sets, unlike other related methods (Le and Mikolov, 2014; Hill et al., 2016). In most cases, our improvements over the baselines are higher than 5% . On the Reuters-21578 data set, we surpass the closely-related approach of Butnaru and Ionescu (2017) by around 2% . On Method MR VLAWE ( k = 2 ) 93 . the RT-2k data set, we surpass the related works of Fu et al. (2018) and Butnaru and Ionescu (2017) by around 4% . To our knowledge, our accuracy of 94 . 1% on RT-2k (Pang and Lee, 2004) surpasses all previous results reported in literature. On the MR data set, we surpass most related works by more than 10% . To our knowledge, the best accuracy on MR reported in previous literature is 83 . 6% , and it is obtained by Cheng et al. (2018). We surpass the accuracy of Cheng et al. (2018) by almost 10% , reaching an accuracy of 93 . 3% using VLAWE. On the TREC data set, we reach the third best performance, after methods such as (Cheng et al., 2018; Zhou et al., 2016, 2018). Our performance on TREC is about 2% lower than the state-of-the-art accuracy of 96 . 1% . On the Subj data set, we obtain an accuracy of 95 . 0% . There are two state-of-the-art methods (Cheng et al., 2018; Zhao et al., 2015) reporting better performance on Subj. Compared to the best one of them (Cheng et al., 2018), our accuracy is 1% lower. Overall, we consider that our results are noteworthy. The k-means clustering algorithm and, on some data sets, the cross-validation procedure can induce accuracy variations due to the random choices involved. We have conducted experiments to determine how large are the accuracy variations. We observed that the accuracy can decrease by up to 1% , which does not bring any significant differences to the results reported in Table 1. Even for a small number of clusters, e.g. k = 10 , the VLAWE document representation can grow up to thousands of features, as the number of features is k  d , where d = 300 is the dimensionality of commonly used word embeddings. However, there are several document-level representations that usually have a dimensionality much smaller than k  d . Therefore, it is desirable to obtain a more compact VLAWE representation. We hereby propose two approaches that lead to more compact representations. The first Figure 1: Accuracy on MR for different numbers of k-means clusters. one is simply based on reducing the number of clusters. By setting k = 2 for instance, we obtain a 600 -dimensional representation. The second one is based on applying Principal Component Analysis (PCA), to reduce the dimension of the feature vectors. Using PCA, we propose to reduce the size of the VLAWE representation to 300 components. In Table 2, the resulting compact representations are compared against the full VLAWE representation on the MR data set. Although the compact VLAWE representations provide slightly lower results compared to the VLAWE representation based on 3000 components, we note that the differences are insignificant. Furthermore, both compact VLAWE representations are far above the state-of-the-art method (Cheng et al., 2018). In Figure 1, we illustrate the performance variation on MR, when using different values for k . We notice that the accuracy tends to increase slightly, as we increase the number of clusters from 2 to 30 . Overall, the VLAWE representation seems to be robust to the choice of k , always surpassing the state-of-the-art approach (Cheng et al., 2018). We proposed a novel representation for text documents which is based on aggregating word embeddings using k-means and on computing the residuals between each word embedding allocated to a given cluster and the corresponding cluster centroid. Our experiments on five benchmark data sets prove that our approach yields competitive results with respect to the state-of-the-art methods. We thank the reviewers for their useful comments. This research is supported by University of Bucharest, Faculty of Mathematics and Computer Science, through the 2019 Mobility Fund.
objective method abstain abstain abstain result objective result abstain other other abstain other other objective method objective method abstain abstain abstain abstain objective abstain abstain result method other result abstain method abstain result abstain other other abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain objective result other other