Unsupervised document representation learning is an important task providing pre-trained features for NLP applications. Unlike most previous work which learn the embedding based on self-prediction of the surface of text, we explicitly exploit the inter-document information and directly model the relations of documents in embedding space with a discriminative network and a novel objective. Extensive experiments on both small and large public datasets show the competitiveness of the proposed method. In evaluations on standard document classification, our model has errors that are relatively 5 to 13% lower than state-of-the-art unsupervised embedding models. The reduction in error is even more pronounced in scarce label setting. Rapid advance in deep methods for natural language processing has contributed to a growing need for vector representation of documents as input features. Applications for such vector representations include machine translation (Sutskever et al., 2014), text classification (Dai and Le, 2015), image captioning (Mao et al., 2015), multi-lingual document matching (Pham et al., 2015), question answering (Rajpurkar et al., 2016), and more. This work studies unsupervised training for encoders that can efficiently encode long paragraph of text into compact vectors to be used as pre-trained features. Existing solutions are mostly based on the assumption that a good document embedding can be learned through modeling the intra -document information by predicting the occurrence of terms inside the document itself. We argue that such an assumption might not be sufficient to obtain meanEqually contribution. Traditional document representation models such as Bag-of-words (BoW) and TF-IDF show competitive performance in some tasks (Wang and Manning, 2012). However, these models treat words as flat tokens which may neglect other useful information such as word order and semantic distance. This in turn can limit the models effectiveness on more complex tasks that require deeper level of understanding. Further, BoW models suffer from high dimensionality and sparsity. This is likely to prevent them from being used as input features for downstream NLP tasks. Continuous vector representations for documents are being developed. A successful thread of work is based on the distributional hypothesis, and use contextual information for context-word predictions. Similar to Word2Vec (Mikolov et al., 2013), PV (Le and Mikolov, 2014) is optimized by predicting the next words given their contexts in a document, but it is conditioned on a unique document vector. Word2Vec-based methods for computing document embeddings achieve state-of-the-art performance on document embedding. Such methods rely on one strong underlying assumption: it is necessary to train the document embedding to optimize the prediction of the target words in the document. In other words, the objective requires the model to learn to predict the target words in surface text. We argue that there are several concerns with such a self-prediction assumption. The strategy of predicting target words therefore only exploits in-document information, and do not explicitly model the inter-document distances. We believe an ideal embedding space should also infer the relations among training documents. For example, if all documents in the corpus are about machine learning , then the concept of machine learning becomes less critical in the embedding. However, if the corpus contains documents from different areas of computer science, then the concept of machine learning should be encoded in any document relevant to it. We therefore claim that the embedding of a document should not only depend on the document itself but also the other documents in the corpus , even though previous work seldom makes this consideration. In addition, accurate predictions at the lexicon or word level do not necessarily reflect that the true semantics have been learned. For example, in IMDB dataset review No.10007:  ... the father did such a good job. Obviously, good can be replaced with synonyms like nice without significantly altering the meaning of the sentence. However, since the synonyms are treated as independent tokens in PV and Doc2VecC, the lexicon good must be predicted exactly. Moreover, to accurately predict the final word job , the embedding probably only needs to know that did a good job is a very common phrase, without having to understand the true meaning of job . This example shows that in order to accurately predict a local lexicon, the embedding might opt to encode the syntactic relationship instead of true semantics. Enforcing document embeddings to make predictions at the word level could be too strong of an objective. More specifically, we argue that the true semantics should not only depend on a small context, but also the relations with other training documents at document level. To address the above concerns we propose a novel model for learning document embedding unsupervisedly. In contrast with previous work (PV and Doc2Vec), we model documents according to two aspects. First, we abandon the concept of context word prediction when training an embedding model. Instead we propose a self-supervision learning framework to model inter-document information. Conceptually, we use the embedding to determine whether a sentence belongs to a document. Our encoder is equipped with a discriminator to classify whether a sentence embedding is derived from a document given that document's embedding. This explicitly enforces documents to be spread reasonably in the embedding space without any labels so that they can be discriminated. To the best of our knowledge, this is the first deep embedding work to explicitly model the inter-document relationship. Second, in our approach the predictions are inferred at the sentence level. This avoids the effect of only predicting the surface meaning in word level (e.g. good vs. nice). Unlike previous work, our model is explicitly optimized to represent documents as combinations of sequence embedding beyond words seen in training. Below we summarize the key contributions:  We present a deep and general framework and a novel objective for learning document representation unsupervisedly. Our models are end-to-end, easy to implement, and flexible to extend. We perform experiments through sentiment analysis and topic classification to show that our model, referred to as self-discriminative document embedding (SDDE) , is competitive to the state-of-the-art solutions based on traditional context-prediction objectives. Our extensive experiments quantitatively and qualitatively show that SDDE learns more effective features that capture more document-level information. To the best of our knowledge, SDDE is the first deep network to model inter-instance information at document level. We further propose to evaluate unsupervised document embedding models in weakly-supervised classification. That is, lots of unlabeled documents with only few labels attached to some of them, which is a realistic scenario that unsupervised embedding could be particularly useful. Here we give an overview of other related methods on learning unsupervised text representations. Besides BoW and TF-IDF, Latent Dirichlet Allocation models (Deerwester et al., 1990; Blei et al., 2003) leverage the orthogonality of high-dimensional BoW features by clustering a probabilistic BoW for latent topics. Several models extend from Word2Vec (Mikolov et al., 2013), using context-word predictions for training document embedding end-to-end. PV (Le and Mikolov, 2014) keeps a document embedding matrix in memory and is jointly trained. The required training parameters are linear to the number of documents and thus prohibit PV from being trained on a large corpus. Moreover, expensive inference for new documents is required during testing. To address the above concerns, Doc2VecC (Chen, 2017) combines PV and a denoising autoencoder (DEA) (Chen et al., 2012) with BoW vectors as global document information instead. The final document embedding are then produced by simply averaging the jointly trained word embedding. Another thread of work uses two-stage pipelines to construct sentence/document embedding from pre-trained word embedding. Arora et al. (2017) propose post-processing weighting strategies on top of word embedding to build sentence representations. WME (Wu et al., 2018) propose a random feature kernel method based on distance between pairs of words, which also shows inter-document information helps. However, the cost scales with the size of training samples such that it is hard to be applied on large-scale dataset. There have been more embedding work on sentences compared to documents. These approaches mostly learn the sentence embedding by modeling the sentence-level (Kiros et al., 2015; Tang et al., 2017b,a; Logeswaran and Lee, 2018) or word-level (Pagliardini et al., 2018; Kenter et al., 2016; Arora et al., 2018) distribution hypothesis (Harris, 1954; Polajnar et al., 2015) in a large ordered corpus. We note that the main difference between learning embedding for sentences and documents is that documents are not ordered in a corpus. Some other work model sentences with RNN autoencoders (Hill et al., 2016a; Gan et al., 2017). Documents often refer to long-length text containing multiple sentences, which might be hard to model with RNNs (Pascanu et al., 2013; Jing et al., 2017) and time-consuming on large corpus. To facilitate downstream tasks, a document embedding is required to compress useful features into a compact representation. It is not an easy task to learn discriminable features unsupervisedly since validation information is not accessible for training. We first introduce some notations:  V : the training corpus vocabulary of size |V| ;  X = { X 1 ,    , X n } : a training corpus of document size n = |X | , in which each document X i is a set of sentences S i ;  S i = { s 1 i ,    , s | x i | i } : a document divided into a set of sentences, of set size |S i | , in which each sentence s ji  R |V| T j contains a sequence of variable length T j of word one-hot vectors w 1 j ,    , w T j j , each in R |V| 1 . S is the set of total sentences n (cid:83) i =1 S i in the training corpus, of size m = |S| ;  h w : the size of the word embedding and U  R h w |V| : the word embedding projection matrix. We use u w to denote the column in U for word w  h s : the size of the sentence embedding and e s  R h s : the embedding of sentence s . d i  R h s : document X i 's embedding. that maps document X i to d i unsupervisedly. Next, we formulate how SDDE represents a document, then introduce our self-discriminative learning procedure we use to train it. We consider a document as mean of sentences , i.e., breaking a document into several subsequences. We demonstrate several benefits of the design in SDDE. First, decomposing long documents into shorter but reasonable semantic unit (e.g., sentences) makes encoding easier and faster since they can be processed in parallel. Similar concepts of modeling documents hierarchically have shown benefits in some supervised tasks such as text classification (Yang et al., 2016). It also makes the model insensitive to document length, which is important because length varies greatly in real documents (see Table 1). In training, we further propose to represent a document during training using the average of only a subset of its sentences. This special sentence-level dropout is beneficial for training by creating up to (cid:0) | X i | q (cid:1) combinations for each document, where q is the number of sentences to keep. This enforces the local sentence representations to capture global information of a document by representing it with a subset of sentences in it. The word embedding is used as globally shared building blocks for sentence embedding. For a document X i = { s ji } , or S i , the embedding is derived from averaging the respective representations of subsequences. Noted as: d i = 1 q q (cid:88) j =1 , s  PS i ( s ) e j s , (1) where e s = E ( s ) , (2) where a sentence encoder E is introduced to produce sentence embedding for s ji . In practice, sentences can be obtained by simply segmenting documents with punctuation. In testing, the document embedding is obtained by averaging all the sentences in which: d = 1 |S i | (cid:88)  s S i e s . We note that averaging subsequences differs from averaging of words in two aspects. First, each sentence is encoded individually before being averaged, allowing incorporation of word order into design rationale at least in a reasonable range. Second, subsequences may have different lengths that reveal syntactic information. To illustrate, BoW/mean-of-word models suffer from ambiguously modeling two different documents which are similar in word distributions but differ in some aspects of interest. Mean-of-sentence model avoids such concern by modeling documents at the sentence level. It could be expected that it is much less likely to find two documents with similar sentence distribution than similar word distribution. Mean-of-sentences formulation can be smoothly reduced to mean-of-word models (by treating each word as a sentence) or pure sequence models (by treating each document as a very long sentence). Unlike PV or Doc2VecC which emphasize modeling distributional information within individual documents, we model relations across documents. The basic idea is that we hope to learn an embedding for each sentence in the document as well as a discriminator that determines whether a sentence belongs to a document. Self-discriminative learning uses a discriminator network D to determine whether a sentence belongs to a document. The aim is to learn a suitable embedding and a good discriminator to determine if a sentence belongs to Algorithm 1 Self-Discriminative Learning for Unsupervised Document Embedding Input: Documents X = { X i } n 1 , p , k , h w , h s . We propose an objective that explicitly optimizes SDDE towards representing a document with mean of (encoded) sentences. To optimize the discriminator D , we formulate it as a binary classifier that takes pairs of document embedding d of a document X i and a sentence embedding e s , ( d , e s ), as inputs. The discriminator is asked to discriminate using d whether the sentence s belongs to the document X i or the other documents X (cid:48)  X \ { X i } . The loss then becomes: log (1  D ( d , e p s )) + k (cid:88) (cid:96) =1 E s (cid:48)  PS , s (cid:48) / S i (cid:104) log ( D ( d , e (cid:96) s (cid:48) )) (cid:105) , (4) for each document with one positive sample e p s and k negative samples of sentences e s (cid:48) , where s (cid:48) are not in the sentence set S i of X i , as s (cid:48) /  S i . Note that e p s is not used for d otherwise it would be trivial to be solved by the discriminator. The spirits of self-discriminative learning can be understood as unsupervisedly mimicking supervised inference without knowledge of any label information by treating sentences from other documents as fake/negative samples. One main concern that it is possible to find similar sentences in two different documents. Our discriminator particularly addresses this issue by optimizing for the most discriminative sentences rather than similar ones that might not be critical to shape the embedding. To minimize the loss, the encoder would tend to preserve the most essential feature to facilitate the discriminator to push away any two documents, which should encourage the embedding points spread even more widely across the space. This in turn should result in more ease in downstream tasks: for example in learning a decision hyperplane in a classification task. Next, we narrow down to sentence encoder E . Given a sequence of word one-hot vectors as a sentence s = [ w 1 , . . . , w T ] , we project them into an embedding layer U to retrieve their corresponding word embedding. Note that the word embedding are trained jointly. where G is a single-layer RNN encoder using GRU cells to process the word embedding sequences and  is a linear transform for dimension h s of sentence embedding. Our second method, we use a schema of mean-of-word for advantage of fast generation, we average the word embedding w within a sentence s along time axis as AVG encoder: E ( s ) =  ( ReLU ( 1 | s | | s | (cid:88) i =1 Uw ti )) , (6) Let us stress that the role of encoder E is to extract local feature from every sentence, and the overall objective encourages SDDE to represent documents as mean of sentence embedding. An undesired pitfall comes from a learned weak encoder with a powerful discriminator causing the embedding produced by the encoder useless for downstream tasks. To avoid such a pitfall, we Dataset #Class #Train / #Test Doc Length Sent Length IMDB 2 75k / 25k 124.6  8,856.7 11.6  105.5 Yelp P. 2 560k / 38k 70.0  4,117.8 8.2  48.7 AG's News 4 120k / 7.6k 27.2  66.1 11.8  66.7 DBPedia 14 560k / 70k 32.8  231.3 9.7  68.6 Table 1: Statistics of the datasets. adopt lightweight network structures for discriminators. For the IMDB datasets, we find inner product ( dV ) t E ( s ) with a learnable matrix V sufficient. For the other datasets in Table 1, two fully-connected layers with ReLU activations in latent are used. We relate our method to the Negative Sampling (Mikolov et al., 2013) technique which is a simpli-fied objective of softmax approximation (Mikolov et al., 2013; Mnih and Teh, 2012; Zoph et al., 2016). Negative sampling has been used as an efficient and effective technique in learning word embedding. We reformulate it to train document embedding by sampling in sentence level, which is easy to implement and efficient to train just like Word2Vec (Mikolov et al., 2013). In practice, when training with mini-batches the documents for negative samples are from the same mini-batch, which requires small extra computation efforts. SDDE requires a similar number of parameters as Doc2VecC does, but much less than PV. In addition, the sentence encoder is flexible and can incorporate other techniques of text processing such as attention methods. Public datasets on sentiment analysis and topic classification across diverse domains including online reviews, news, and Wiki pages are used including IMDB dataset (Maas et al., 2011) and the others from Zhang et al. (2015). Table 1 provides a summary. Only the training splits are used in training embedding with subsampled training split for cross-validations. We preprocess the datasets by normalizing the text to lower class and replacing words appearing less than 10 times with a UNK token. Out-of-vocabulary words in testing set are also replaced. All our baseline models use the same input data. To define the sentences for experiment, we utilize the sentence tokenizer from NLTK. For the documents containing only one sentence we simply divide it into multiple subsequences for sampling. We use RMSProp method for optimization. Dropout 50% of input to the discriminator. Weights are random-uniformly initialized between [-1, 1]. The other hyperparameters are summarized in Table 2. All the models use the same embedding size for fair comparison. The trained document embedding are used for all the evaluations without specific tuning. Generally, it is not easy to evaluate an unsupervised embedding model. In Section 4.3 and 4.4, we evaluate the performance on standard document classification following the common practice used by previous work (Chen, 2017; Wu et al., 2018; Arora et al., 2017): a classification task with a linear SVM (Fan et al., 2008) trained on the labels in each dataset. Next, we study unsupervised document embedding on two novel aspects. In Section 4.5, we study a weakly-supervised classification setting that fits the realistic scenario of using unsupervised embedding with only a few labels. In Section 4.6, we provide a metric to evaluate the effectiveness of modeling inter-document information. We first compare our models with the others state-of-the-art competitors. RNN-LM (Mikolov et al., 2010) and Skip-thought (Kiros et al., 2015) are RNN-based. SIF (Arora et al., 2017), W2V-AVG (Mikolov et al., 2013), and WME (Wu et al., 2018) are two-stage approach that post-processing on word embedding. We collect the results reported on the widely-used benchmark sentiment classification dataset IMDB. For PV, we use Gensim implementation ( Rehurek and Sojka, 2010); versions https://www.nltk.org/ Model Error% Skip-thought* (Kiros et al., 2015) 17.4 SIF (GloVe) (Arora et al., 2017) 15.0 RNN-LM* (Mikolov et al., 2010) 13.6 W2V-AVG* (Mikolov et al., 2013) 12.7 DEA* (Chen et al., 2012) 12.5 PV-DM 20.0 PV-DBoW 12.0 WME (Wu et al., 2018) 11.5 Doc2VecC* (Chen, 2017) 11.7 SDDE-AVG 10.6 SDDE-RNN 10.2 Table 3: Sentiment Classification on IMDB Benchmark. of both Distributed Memory (DM) and Distributed Bag of Words (DBoW) are reported. For different encoders in SDDE, AVG is for averaging word embedding and RNN is for the RNN encoder. Self-Discriminative Learning is Effective From the experiment result in Table 3, we can see that our self-discriminative learning is effective and superior on the document embedding models for both AVG and RNN versions. SDDE-RNN achieves best accuracy on IMDB dataset 1.5% margin against Doc2VecC. Study the Property of SDDE Unlike previous work modeling documents on the word or short context level, SDDE operates on the sentence level. We study the false and true predictions output by SVM upon SDDE in comparison with Doc2VecC. Table 5 show some examples that have the largest difference. We observed SDDE can better capture contradicting or contrasting opinions. We observe some wrong predictions (Row 3) are due to the ambivalent reviews. SDDE is insensitive to the number of sentences; we found the effect of the number of sentences per document was trivial as shown in Table 4. Next, we borrow some public large-scale dataset in Table 1 to further validate the effectiveness of SDDE compared to the other models. For Doc2vecC and SIF, we use the code from the au-Label: 1 i don t even like watching those late night talk shows , but i found this one really interesting . thors. We use SIF to generate document embedding with Word2Vec trained on each dataset as its inputs. Results are shown in Table 6. SDDE-AVG performs slightly better across different dataset. We hypothesis SDDE gets larger improvement on IMDB dataset since SDDE can handle longer documents better by exploiting sentence embeddings. On the other hand, the RNN version of SDDE performs significantly worse than the word-averaging version. We may remind the reader that state-of-the-art unsupervised document embedding models are not RNN-based. The effects of word order are still unclear. Wieting and Gimpel (2017) provides a study of sentence embedding. We hypothesize that it may be difficult for an RNN encoder to learn to incorporate multi-domain information in datasets with many classes (e.g., DBpedia) unsupervisedly. This would be our future work. Next, we consider a more real-world weakly-supervised learning scenario: classification on the datasets we have used in previous experiments, but this time only when very few labels are available. We hypothesize that SDDE is particularly useful for classification with few labels since the self-discriminative learning has exploited the possible features to map the text onto the embedding space properly during the representation learning phase. The embedding is expected to be more discriminable to facilitate finding the classification decision hyperplanes with fewer labeled data. We randomly sample equal number of instances from each class to train a SVM and verify with the whole testing set. PV-DBoW, Doc2VecC, and SDDE-AVG are examined in this experiment. We use the same pre-trained document embedding as in the previous experiments. We repeat the whole procedure 30 times and report the means and tune the penalty parameter C to find the best value for each model. Results in Figure 2 and Table 7 show SDDEs outperform PV and Doc2VecC. We visualize the training points with t-SNE. As shown in Figure 1, SDDE seems to be able to spread the embedded data more widely, which eventually leads to better usage of scarce data for classification. Definition We examine our assumption of the ability of SDDE to model inter-document feature. Similar to (Hill et al., 2016b), we consider pairwise cosine similarity between documents with topic labels, this allows us to quantitatively evaluate unsupervised document embedding at inter-document level. Our assumption is that: if pairwise similarity between documents is calculated based on different kinds of embedding, the better embedding results should comply with the properties of both high similarities between those documents within the same underlying class, denoted as IntraCos ( d, d (cid:48) ) and low similarities between document pairs from different classes, or InterCos ( d,  d ) . The mean distance: mean ( IntraCos )  mean ( InterCos ) , (7) is considered as our metric to avoid simply maximizing IntraCos or minimizing InterCos. SDDE Provides High Separation Table 8 shows the evaluation. The distances (Eq. 7) for the baseline models are small, which support our assumption that these methods are not able to model inter-document features properly. On the other hand, distances for SDDE are significantly larger. With the classification experiments, we believe SDDE better preserves meaningful inter-document features. Figure 3 shows some meaningful clusters in SDDE in the World class as cohesive sub-classes. Compared to mainstream unsupervised document embedding models (trained to perform predictions on the lexicon level) SDDE embeddings capture information at the inter-document level, as they are trained to maximize the distance between a sentence and a corresponding document. We hope the underlying idea of SDDE offers the document-embedding community a new investigation direction. Self-discriminative learning shows potential for real-world scarcely-labeled scenarios, and our future work will focus on joint training of representations for semi-supervised learning. In NAACL HLT , pages 13671377. Tom Kenter, Alexey Borisov, and Maarten de Rijke. 2016. Siamese CBOW: optimizing word embeddings for sentence representations. In ACL . Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urta-sun, and Sanja Fidler. 2015. Skip-thought vectors. In NIPS , pages 32943302. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In ICML , pages 11881196. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In ICLR . Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their composition-ality. In NIPS , pages 31113119. This material is based upon work supported by Mi-crosoft Research Asia (MSRA) grant, and by Taiwan Ministry of Science and Technology (MOST) under grant number 108-2634-F-002 -019.
abstain objective abstain method abstain abstain abstain method abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective method method abstain objective abstain abstain result objective objective objective result objective objective objective abstain other other other other other other other other other other other other other method other other abstain abstain other abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain method abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain result abstain abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other