Skip-Thought Vectors
We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoderdecoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice.
TL; DR This paper presents a natural generalization of neural word embeddings: sentence embeddings! The training objective of the proposed "skip-thoughts" model is to predict the previous and next sentence given an encoding of the current sentence. Thus, it may be trained without supervision from raw text, and the resulting embeddings used as "features" for downstream tasks. The authors present experiments on 8 tasks, and while the results of the proposed method are not state-of-the-art, it outperforms many previous baselines with less feature engineering.The proposed "skip-thoughts" model is a fairly ingenious application of the recently popularized sequence-to-sequence RNN framework. It's been known for a long time now (e.g., Ando & Zhang, 2005) that learning representations of observed data X via auxiliary tasks can be helpful in downstream tasks, in which one is interested in making predictions Y. However, doing so at the sentence level is a nice twist, though not an entirely novel one (e.g. the cited paragraph vector work, and earlier work cited therein). I think the technical approach in the paper, being largely derived from existing RNN frameworks, is sensible. I also liked the vocabulary expansion strategy.I do have some concerns regarding the experimental evaluation. The main question in my mind is if the proposed approach is a better way of learning from a big pile of unlabeled data X than alternatives, in terms of downstream performance on predicting Y. Unfortunately, I don't believe this question is addressed in the experiments; there is no baseline which uses to the same X (the Book11K corpus). Thus, it is not clear if the reported results are simply as a result of (a) using a lot more data, or (b) the model is capturing something interesting beyond superficial word co-occurence statistics.I would have liked to see a comparison to a strong baseline which uses the same training data. Short of this, it would have at least been good to see a *weak* baseline, e.g. a bag-of-words model. Yet another option would have been to train word embeddings on Book11K, and use these in some of the more engineered baselines so that at least the data conditions are matched.One common issue with RNNs in the encoder-decoder framework is the encoder must summarize the entire input in a single vector. One heuristic that has been used to alleviate this problem is to reverse the order of the tokens. More recently, attention models have been used. I would have liked to see a little discussion of this issue. In a similar vein, I would have liked to see more details about:-how the proposed approach was tuned, and how sensitive it is to hyperparameters-what alternate architectures were used, if any. For instance, an obvious alternative is to only predict the next sentence given the current sentence (it's not 100% obvious why it makes sense to predict sentences in reverse, even though I'm willing to believe it's helpful as it doubles the amount of data).-how performance improves with increasing amounts of unlabeled data X (presumably could be derived from model checkpoints from a prior training run).UPDATE AFTER AUTHOR RESPONSE:Including a baseline that uses word-level embeddings trained on Book11K would improve the paper. However, it seems to me that the natural baseline to compare to is the paragraph vector model [1], with matched train and test conditions. I don't understand why this comparison wasn't made; perhaps this should be clarified in the paper.[1] http://arxiv.org/pdf/1405.4053v2.pdf