Semi-supervised Sequence Learning
We present two approaches to use unlabeled data to improve Sequence Learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a language model in NLP. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a "pretraining" algorithm for a later supervised sequence learning algorithm. In other words, the parameters obtained from the pretraining step can then be used as a starting point for other supervised training models. In our experiments, we find that long short term memory recurrent networks after pretrained with the two approaches become more stable to train and generalize better. With pretraining, we were able to achieve strong performance in many classification tasks, such as text classification with IMDB, DBpedia or image recognition in CIFAR-10.
This is a mostly empirical paper where the authors apply a new style of pretraining to LSTMs and show major improvements over randomly initialized LSTM on sentiment analysis, text classification, and character-level document classification. The main pretraining approach is to simply train the LSTM as auto-encoder to be able to echo a sequence in order.Generally this paper shows very striking results with gains over temporal convolutional networks and document vector methods like paragraph vectors.General comments:- "Unsupervised Sequence Learning" is a confusing title for this paper. While the pretraining method does use an unsupervised sub-task, there are no results reported on any unsupervised tasks or even analysis of this initial subtask.- The LSTM autoencoder itself (not for pretraining) does not seem novel. The seq2seq work for instance uses "unsupervised" reconstruction as an example (particularly in their talks).This should be made clear in the paper.- I was not able to follow the explanation in paragraph l.97- l. 182 "Random word dropout" seems important. Could you describe?- It was not clear exactly how pretraining was done. Unless I am mistaked, it seems like the autoencoder is trained on in-domain documents possibly from the same dataset? Is this important? What happens if you pretrain on wikipedia or external data. For section 4.3 how is SA pretrained?- l.243. It seems unfair that CNN-non-static is not given the Amazon reviews as well. Should be possible to run word2vec on amazon instead of google news for this comparison.  - One aspect that would improve this paper is some direct evidence of the claim that long-range dependencies are really being captured. It seems possible that the LSTM is just somehow better than CNN at picking up local phrases. Even anecdotal evidence of this process would be interesting.     ## Small - l103: typo "the weights" verb agreement - l114: typo verb agreement. - l188: same