Published as a conference paper at ICLR 2018 MEMORY ARCHITECTURES IN RECURRENT NEURAL NETWORK LANGUAGE MODELS
We compare and analyze sequential, random access, and stack memory architectures for recurrent neural network language models. Our experiments on the Penn Treebank and Wikitext-2 datasets show that stack-based memory architectures consistently achieve the best performance in terms of held out perplexity. We also propose a generalization to existing continuous stack models (Joulin & Mikolov, 2015;Grefenstette et al., 2015) to allow a variable number of pop operations more naturally that further improves performance. We further evaluate these language models in terms of their ability to capture non-local syntactic dependencies on a subject-verb agreement dataset (Linzen et al., 2016) and establish new state of the art results using memory augmented language models. Our results demonstrate the value of stack-structured memory for explaining the distribution of words in natural language, in line with linguistic theories claiming a context-free backbone for natural language.
The authors propose to compare three different memory architecture for recurrent neural network language models:
vanilla LSTM, random access based on attention and continuous stack. The second main contribution of the paper is to propose an extension of continuous stacks, which allows to perform multiple pop operations at a single time step.
The way to do that is to use a similar mechanism as the adaptive computation time from Graves (2016): all the pop operations are performed, and the final state of the continuous stack is weighted average of all the intermediate states. The different memory models are evaluated on two standard language modeling tasks: PTB and WikiText-2, as well as on the verb number prediction dataset from Linzen et al (2016). On the language modeling tasks, the stack model performs slightly better than the attention models (0-2 ppl points) which performs slightly better than the plain LSTM (2-3 ppl). On the verb number prediction tasks, the stack model tends to outperforms the two other models (which get similar results) for hard examples (2 or more attractors).
Overall, I enjoy reading this paper: it is clearly written, and contains interesting analysis of different memory architecture for recurrent neural networks. As far as I know, it is the first thorough comparison of the different memory architecture for recurrent neural network applied to language modeling. The experiments on the Linzen et al. (2016) dataset is also interesting, as it shows that for hard examples, the different models do have different behavior (even when the difference are not noticeable on the whole test set).
One small negative aspect of the paper is that the substance might be a bit limited. The only technical contribution is to merge the ideas from the continuous stack with the adaptive computation time to obtain the "multi-pop" model. In the experimental section, which I believe is the main contribution of the paper, I would have liked to see more "in-depth" analysis of the different models. I found the experiments performed on the Linzen et al. (2016) dataset (Table 2) to be quite interesting, and would have liked more analysis like that. On the other hand, I found Figures 2 or 3 not very informative, as it is (would like to see more). For example, from Fig. 2, it would be interesting to get a better understanding of what errors are made by the different models (instead of just the distribution).
Finally, I have a few questions for the authors:
- In Figure 1. shouldn't there be an arrow from h_{t-1} to m_t instead of x_{t-1} to m_t?
- What are the equations to update the stack? I assume something similar to Joulin & Mikolov (2015)?
- Do you have any ideas why there is a sharp jump between 4 and 5 attractors (Table 2)?
- Why no "pop" operations in Figure 3 and 4?
pros/cons:
+ clear and easy to read
+ interesting analysis
- not very original
Overall, while not groundbreaking, this is a serious paper with interesting analysis. Hence, I am weakly recommending to accept this paper.