Highly regularized LSTMs achieve impressive results on several benchmark datasets in language modeling. We propose a new regularization method based on decoding the last token in the context using the predicted distribution of the next token. This biases the model towards retaining more contextual information, in turn improving its ability to predict the next token. With negligible overhead in the number of parameters and training time, our Past Decode Regularization (PDR) method achieves a word level perplexity of 55.6 on the Penn Treebank and 63.5 on the WikiText-2 datasets using a single softmax. We also show gains by using PDR in combination with a mixture-of-softmaxes, achieving a word level perplexity of 53.8 and 60.5 on these datasets. In addition, our method achieves 1.169 bits-per-character on the Penn Treebank Character dataset for character level language modeling. These results constitute a new state-of-the-art in their respective settings.
This paper proposes an additional loss term to use when training an LSTM LM.  The authors argue that, intuitively, we want the output distribution to retain some information about the context, or "past".  Given this, they use the output distribution as input to a one layer network that must predict the current token.  The loss for this network is incorporated as an additional term used when training the LM.  The authors show that by adding this loss term they can achieve SOTA (for single softmax model) perplexity on a number of LM benchmarks.
The technical contribution is proposing a new loss term to use when training a language model.  The idea is clear, simple, and well explained, and it seems to be effective in practice.  One drawback is that it is highly specific to language models.  Other recent works which have demonstrated effective regularization of LSTM LMs have proposed methods that can be used in any LSTM model, but that is not the case here.  In addition, there is not much theoretical justification for it, it seems like a one-off trick.  The loss term is motivated by the idea that we want the output distribution to retain some information about the context, but why should that be the case?
Although it is specific to language models, there are a few reasons it might be of broader significance:
- It falls in the recent line of work in incorporating auxiliary losses for various tasks.  This idea has touched many problems and seen success in practice.
- Perhaps it can be applied to other sequence models.  For example in encoder-decoder models, the decoder can be thought of as a conditional LM.
Experiments are comprehensive and rigorous.  They might be more convincing if there were results on a very large corpus such as 1 billion word corpus.
Pros:
- New SOTA for single softmax model on LM benchmarks.
- Simple, clearly explained idea.
- Demonstrates effectiveness of auxiliary losses.
- Rigorous experiments.
Cons
- Trick is specific to LM.
- No large corpus results.