Can Active Memory Replace Attention?
Several mechanisms to focus attention of a neural network on selected parts of its input or memory have been used successfully in deep learning models in recent years. Attention has improved image classification, image captioning, speech recognition, generative models, and learning algorithmic tasks, but it had probably the largest impact on neural machine translation. Recently, similar improvements have been obtained using alternative mechanisms that do not focus on a single part of a memory but operate on all of it in parallel, in a uniform way. Such mechanism, which we call active memory, improved over attention in algorithmic tasks, image processing, and in generative modelling. So far, however, active memory has not improved over attention for most natural language processing tasks, in particular for machine translation. We analyze this shortcoming in this paper and propose an extended model of active memory that matches existing attention models on neural machine translation and generalizes better to longer sentences. We investigate this model and explain why previous active memory models did not succeed. Finally, we discuss when active memory brings most benefits and where attention can be a better choice.
This paper is built on the top of Neural GPU model by making changes to the decoder to handle sequence generation tasks like machine translation better. While the original Neural GPU model was applied for algorithmic tasks, author try to use this model for machine translation task. Results show that proposed active memory model is comparable to Attention model for NMT.
After Rebuttal:  I have read the rebuttal and I see that authors have addressed most of my concerns. I am changing my rating for impact from 2 to 3.  Before Rebuttal:  This paper proposes 2 extensions to Neural GPU model. Markovian Neural GPU is an obvious extension of Neural GPU for sequence generation. Extended Neural GPU is a non-trivial extension.  Firstly the claim (in abstract and everywhere) that active memory has not improved over attention for NLP tasks is wrong. Dynamic Memory Networks and End-to-end memory networks have been used for comprehension based question answering task. For machine translation, Meng et al., 2016 (http://arxiv.org/pdf/1506.06442v4.pdf) proposes a deep memory based architecture which is an active memory model and it does perform better than the attention based NMT model. In fact their model is very similar to Neural GPU, except for the fact that they use a different transformation instead of convolution.  Input structure as described in line 106-110 is not very clear. If I understand correctly, authors use w*n*m matrix as input where only w[0] is filled with n m-dimensional word vectors and w[k] for k > 0 are all set to 0? Also if number of words are less than n, then remaining part of the w[0] matrix are also set to 0? Please clarify.  From lines 39-41 I understand that authors consider NTM as a complex attention model than an active memory model? If so, I disagree with that point of view.   Experiments:  1. I understand that the authors want to compare their model with attention based model without beam search. But I do not understand the reason for choosing two attention models – one trained with sentences upto 64 tokens and another one with sentences upto 128 tokens. Why is Extended Neural GPU trained with sentences upto 64 tokens only? Specifically, the comparison is incomplete without the following rows:  a. Attention model trained with sentences upto 64 + beam search b. Attention model trained with sentence upto 128 tokens + greedy decoder. c. Neural GPU models trained with sentences upto 128 tokens. d. Meng et al., 2016  2. Authors should also highlight the limitations of the proposed model clearly. a. Is it not possible to train Neural GPU with 128 tokens? Or the results are worse? In either case, this is a limitation. b. Neural GPU models expect the output sentence size. In lines 203-208, authors say that they consider all sizes between input size and double of it and pick the result with smallest perplexity. This is very inefficient when compared to simple decoding done with LSTM decoder. c. I see that it is very difficult to do beam search with Neural GPU, because of the proposed complex decoder. This is another limitation.  3. Just based on Figure 6 (which is approximate), authors cannot make the claim that less sensitive to sentence length than the attention baseline. Because other 2 models where trained with UNK token and few other variations.  Overall, this paper proposes an interesting extension to Neural GPU, but the application is not very convincing. It is not very clear, what is the gain in using a complex architecture like this when compared to the simple attention model if the results are only comparable and not better. I would like to see a comparison with Meng et al., 2016 to see if this complex architecture really helps to improve the performance.