We present V ector of L ocally A ggregated E mbeddings ( VLAE ) for effective and, ultimately, lossless representation of textual content. Our model encodes each input text by effectively identifying and integrating the representations of its semantically-relevant parts. The proposed model generates high quality representation of textual content and improves the classification performance of current state-of-the-art deep averaging networks across several text classification tasks. Representation learning algorithms can reveal intrinsic low-dimensional structure in data (Rumel-hart et al., 1986; Bengio et al., 2013; LeCun et al., 2015). In particular, deep averaging networks (DANs) are effective for text classification (Shen et al., 2018; Arora et al., 2017; Wieting et al., 2016; Iyyer et al., 2015). They achieve their improvement through use of word embeddings, weighted averaging, and deepening networks. The above works show that DANs can outperform RNNs and CNNs in text classification while taking only a fraction of their training time. In this work, with a special focus on DANs, we study the effect of information loss associated with average word embeddings and develop algorithms that are robust against information loss for text representation. We show that divergence of word embeddings from their average can be considered as a good proxy to quantify information loss; in particular, longer documents suffer from significant information loss when represented by average word embeddings. These results inspire our work to develop a novel representation learning approach based on Vector of Locally Aggregated Descriptors (VLAD) (Jegou et al., 2010; Arandjelovic and Zisserman, 2013)an effective approach to integrate image descriptors for large scale image datasets. Our model identifies semantically-relevant parts of documents and locally integrates their representations through clustering and autoencoding. In contrast to averaging, our model prevents larger semantically-relevant parts of inputs to dominate final representations. It improves DANs by 5 . 30 macro-F1 points in classifying longer texts and show comparable performance to them on shorter text. How can information loss be quantified when word embeddings are averaged? How important it is to address information loss when representing textual content? Are representation learning algorithms robust against information loss? We conduct experiments to answer these questions with respect to deep averaging network (DANs). Our study can inspire works in more complex averaging approaches such as those reported in (Torabi Asr et al., 2018; Kiela et al., 2015) as well as recent works on unsupervised semantic similarity (Pagliardini et al., 2018). We use the DAN developed in (Joulin et al., 2017) and several datasets containing short and long documents to answer these questions. Let's assume a d -dimensional word embedding space. We quantify the amount of information loss in the average word embedding vector of a given document S  R n  d by computing the average divergence (or distance) between its word embeddings, w i  R d  i  { 1 . . . n } , and their average vector, s = 1 /n (cid:80) i w i , s  R d , as follows: divergence = 1 n (cid:88) i (1  cosine ( s , w i )) . Figures 1(a)1(c) show strong positive correlation between divergence and document length IMDb Reddit Twitter 0.54 0.56 0.58 0.60 0.62 0.64 0.66 0.68 0.70 divergence from average embedding 0 50 100 150 200 250 a v e r a g e d o c u m e n t l e n g t h document_length trend_line across long and short text datasets. Given these results and if we assume longer documents should suffer from greater information loss if represented by average word embeddings, divergence from mean can be a good proxy to quantify information loss associated with average embeddings. As Figures 1(d) and 1(e) show, DAN's macro-F1 classification performance considerably decreases as divergence (or text length) increases for IMDb and Reddit datasets; note that we sort and bin instances based on their divergence values and report average macro-F1 for each bin. In particular, as the trend lines in Figures 1(d) and 1(e) show, the average macro-F1 performance drops from 0 . 86 and 0 . 82 on shorter IMDb and Reddit posts to 0 . 82 and 0 . 71 on their longer posts respectively. In addition, the result on Twitter dataset, Figure 1(f), shows that DANs are robust against small information loss, i.e. small divergence values below 0 . 55 do not inversely affect macro-F1 performance. This result is also observed on the other two datasets, see macro-F1 performance for small divergence values (  0 . 55 ) in Figures 1(d) and 1(e). The above experiments show that (a): significant information loss can occur when word embeddings are averaged, in particular, when representing longer documents, and (b) such information loss can inversely affect the performance of downstream classifiers like DANs on longer texts. In this paper, we develop an effective representation learning model to tackle this problem. We propose to utilize semantically-relevant parts of inputs to tackle information loss associated with average word embeddings. Assuming that semantically-relevant words are closer to each other in semantic space (constructed over a global vocabulary), we expect divergence between words in semantically-relevant parts of inputs (i.e. information loss associated to their average word embedding) to be very small. Thus, as Figure 2 illustrates, we propose to cluster the semantic space to first identify semantically-relevant parts of inputs over a global vocabulary; we then effectively integrate these parts to represent documents. Let's assume a global vocabulary V in which words are represented in a d -dimensional space, w  R d . As Figure 2 shows, we first cluster this semantic space into k clusters through the following objective function over V : min (cid:88) w V || f ( C , w )  w || 2 , (2) where C is the set of k cluster centers, |C| = k , and f ( C , w ) returns the nearest cluster center c  C to the embedding vector w based on cosine similarity among embeddings or Euclidean distance in case of K-Means. 1 Given a document S  R n  d with an arbitrary number of n  1 words, and the above k cluster centers, we compute the representation of the document in each cluster c i , i = 1 . . . k as follows: a i = 1 z i (cid:88) j : f ( C , w j )= c i w j (3) z i = | j : f ( C , w j ) = c i | , where a i  R d indicates the representation of the document at cluster c i and is obtained by taking the average embedding of words of the document that have been assigned to cluster c i according to Equation (2), and z i is the number of such words in cluster c i . To this end, each document can be represented by A  R d  k which is obtained by concatenating its cluster-level representations. Note that we didn't observe any performance difference between the above averaging process versus computing residuals (differ-ences between word embeddings and corresponding cluster centroids) which is commonly used to represent cluster-level image descriptors (Jegou et al., 2010; Arandjelovic and Zisserman, 2013) in image processing. Since A s are of fixed length, they can be readily used as features in traditional classification and clustering algorithms. However, they can cause ef-ficiency issues because of their large size ( d  k ); note that the typical value for embedding dimension d is 300 (Pennington et al., 2014; Mikolov et al., 2013). To tackle this issue, we further integrate cluster-level representations, at the cost of some further information loss, to create representations of lower dimension for inputs. In particular, given all input documents with k cluster-level representations A  R d  k for each document, we develop an autoencoder with one 1 This problem can be solved through gradient descent seeded with an initial set of k examples drawn uniformly at random from V (Bottou and Bengio, 1995; Sculley, 2010). hidden layer that integrates these cluster-level representations to create a final representation for each document, vector a  R d  m where m is the dimensionality reduction parameter and m  d is length of the representation (final layer of the encoder) and is smaller than d  k for m < k . Training a single-layer autoencoder corresponds to optimizing the learning parameters to minimize the overall loss between inputs and their reconstructions. For real-valued A , squared loss is often used (Vincent et al., 2010), i.e. l ( A ) = || A   A || 2 where  A  R d  k is reconstruction of A and generated by the decoder from a . Our intuition is that if a leads to a good reconstruction of A , it has retained all information available in the input. We refer to a  R d  m as the Vector of Locally Aggregated Embeddings (VLAE). We expect this final representation to be robust against information loss due to its cluster-level local aggregation which prevents larger portions of semantically-similar words to dominate the representation. Data: We investigate VLAEs in three binary classification tasks: sentiment classification on IMDb (Maas et al., 2011), disease-text classification on Reddit, where the task is to classify reddit posts as relevant or irrelevant to specific diseases, and churn prediction on Twitter (Amiri and Daume III, 2015), where the task is to clas-sify/predict if given tweets indicate user intention about leaving brands, e.g. the tweet my days with BRAND are numbered is a churny tweet. See details in Table 1. For pre-processing, we change all texts to lowercase, and remove stop words, user names, and URLs from texts. Settings: We use validation data for hyperpa-rameter tuning and model selection. We use 300 dimensional word embeddings ( d = 300 ) provided by Google (Mikolov et al., 2013), and for greater number of d s, we train word2vec on unlabeled data, see Table 1. In addition, we set the dimensionality reduction parameter m from { 1 . . . 4 } using validation data. The best value of m is the same across tasks/datasets, m = 2 . Furthermore, we determine the number of clusters k for VLAEs by choosing the optimal k from { 2 i , i = { 1 . . . 7 }} using validation data of each dataset. We learn optimal k with respect to task, but not embedding space, due to significant density of the semantic space of word embeddings, see Note on Clustering Word Embeddings. Baselines: We consider two versions of DANs as baselines: Avg small and Avg large which represent documents by average word embedding of size d = 300 and d = m  300 respectively. Note that, for fair comparison, Avg large has the exact same size as our model ( VLAE ); however, depending on m , their network size is 1 . 3 1 . 6 times greater than that of Avg small due to difference in input dimensions. We use 3 hidden layers of size 300 for above networks. Also, to directly evaluate the effect of averaging, we do not adjust initial word embeddings during training. Experimental Results: Table 2 shows the performance of different models across datasets. The results show that VLAE significantly outperforms Avg small and Avg large by 2 . 6 and 7 . 2 points in Macro-F1 on IMDb. The corresponding values on Reddit dataset are 6 . 7 and 3 . 4 points respectively. We believe these improvements are due to more effective and lossless representation of inputs. We note that Avg large performs worse than Avg small on IMDb. This could be attributed to the size of training data which may not be enough to train Avg large , or to lower quality of input representations in Avg large compared to Avg small in case of IMDb. Note that although VLAE has the same number of parameters as Avg large , it uses autoencoding to effectively filter redundant information. Verify-Avg small Avg large VLAE IMDb 83.11 78.52 85.72* Reddit 59.42 62.72 66.10* Twitter 61.42 73.08* 72.62 AVG 67.98 71.44 74.81 Table 2: Macro-F1 performance of different models across datasets. ing these hypotheses will be the subject of future work. In addition, VLAE show lower performance than Avg large on Twitter dataset, F1 of 72 . 62 versus 73 . 08 . We attribute this result to the shorter length of tweets for which, as we experimentally showed before, averaging does not cause major divergence in representations. On average, VLAE improves Avg small and Avg large by 4 . 7 and 5 . 3 F1 points on IMDb and Reddit (longer texts) respectively. It also shows comparable performance to best performing model on Twitter (shorter texts). We also compare models in terms of the quality of their representations. For this comparison, we ignore input preparation time and assume a model that generates better representations should converge faster than other models; note that the overall turnaround time of VLAE is greater than that of Avg small or Avg large because of its input preparation time which we ignore for the purpose of this experiment. The result show that VLAE leads to 7 . 5 , 1 . 3 , and 1 . 3 times faster convergence than Avg small and 14 . 9 , 2 . 6 , and 1 . 8 times faster convergence than Avg large on IMDb, Reddit, and Twitter datasets respectively. Considering the size of these networks, these results indicate that representations obtained from VLAE are much better than those of its counterparts. Note on Clustering Word Embeddings: In experiments, we observe clusters obtained from word embeddings are often very dense. This is a challenge for our model because with small number of clusters ( k s) potentially dissimilar words can appear in the same cluster, while with large k s semantically-similar words may appear in different clusters. Neither of these are desired. To illustrate the above challenge, we report Silhouette Coefficient (SC) (Rousseeuw, 1987) of k-means with different number of clusters obtained from words embeddings across datasets. SC indicates how well cluster boundaries are detected 2 4 8 16 32 64 128 #cluster 0.000 0.025 0.050 0.075 0.100 0.125 0.150 S il h o u e tt e C o e ff i c i e n t IMDB Reddit Figure 3: Mean Silhouette Coefficient computed for different number of clusters; a higher Silhouette Coefficient score indicates better defined clusters. by a clustering model. It is calculated using the mean intra-cluster distance and the mean nearest-cluster distance for each sample. Specifically, the mean distance between each embedding and all other embeddings in the same cluster ( mc ), and the mean distance between the embedding and all other embeddings in the next nearest cluster (the nearest cluster that the embedding is not part of) ( mn ) are used to measure SC for the embedding: mn  mc max( mn, mc ) . The best and worst SC scores are 1 and  1 which indicate ideal and worst clustering respectively. Also, values near 0 indicate overlapping clusters. Figure 3 shows the mean SCs computed over all word embeddings for IMDb and Reddit datasets. 2 The results show that (a): the best number of clusters is k = 2 on both datasets, and (b): Silhouette Coefficient scores generally home in on values close to zero as the number of clusters increases. These results show significant density of embeddings in semantic space. Therefore, we optimize the number of clusters for creating VLAE s by resorting to validation data and measuring task-specific performance. From these results, we conclude that a hierarchical clustering approach that recursively combines pairs of semantically-similar clusters could help better defining these clusters and perhaps improve the performance of our model. Deep averaging networks (DANs) (Joulin et al., 2017; Iyyer et al., 2015; Arora et al., 2017; Shen et al., 2018) were developed based on the successes of vector operations in embedding space. In contrast to their simplicity, DANs showed high performance in text classification tasks. Arora et al. (2017) showed that sentences can be effectively represented by the weighted average of their word embeddings modified by PCA/SVD. In addition, the DANs developed in (Iyyer et al., 2015), (Joulin et al., 2017), and (Shen et al., 2018) were feed-forward networks that used average word embeddings to represent inputs; they were effective for several NLP tasks such as document categorization, text pair similarity, and short sentence classification. Furthermore, feed-forward architectures like DANs have been used for language modeling (Bengio et al., 2003) and greedy transition-based dependency parsing (Chen and Manning, 2014) with fast turnaround time. In addition, previous research investigated a variety of vector operations that could replace the averaging operation used in the DANs. Many of these operations have been studied in (Mitchell and Lapata, 2008) for modeling the composition-ality of short phrases, or showing the utility of simple vector computations(Banea et al., 2014). The operations in (Mitchell and Lapata, 2008) were also extended to use syntactic relation between words and grammar (Erk and Pado, 2008; Collobert and Weston, 2008). Also, clustering semantic space was studied in (Mekala et al., 2017) to learn context information for words and for tasks like topic coherence and information retrieval. In this work, we built on previous work on DANs and investigated and tackled information loss associated with average word embeddings. We investigate information loss associated with average word embeddings. We show that averaging lead to significant information loss and propose to tackle the issue by identify semantically-similar parts of documents through clustering of semantic space at word-level and integrating cluster-level representations through autoencoding. A promising future direction is to use hierarchical clustering to create better cluster-level representations. We thank anonymous reviewers for their insightful comments and constructive feedback.
method method abstain abstain abstain abstain abstain objective result abstain method objective abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method method abstain method abstain abstain method method abstain abstain abstain method method method other abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other other other other other other other other other other objective objective objective abstain other