Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feedforward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers. Building upon this intuition, we propose a new model that solely consists of attention layers. More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer. Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer. Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.
This paper proposes a simple modification to the ubiquitous Transformer model. Noticing that the feed-forward layer of a Transformer layer looks a bit like an attention over "persistent" memory vectors, the authors propose to explicitly incorporate this notion directly into the self-attention layer. This involves concatenating the contextual representations with global, learned memory vectors, which are attended over. 
The model is tested on widedly-utilized character/word-level language modeling benchmarks, where it is found to outperform, or be on par with, existing models while using fewer number of parameters. 
Insofar as architectural advancements can translate to general improvements across multiple NLP tasks, this paper could be seen as important. However, I am not sure that in 2019, demonstrating arguably-marginal perplexity improvements on standard datasets is enough. I would love to see if this type of layer can result in better conditional generation models (e.g. translation, summarization), or can train GPT2/BERT/XLNet-style models whose representations better transfer to other tasks.
I had some further questions/comments:
- I found the motivation of the persistent memory vector as replacing the FF-layer somewhat tenuous. Eq(5) is definitely different from Eq(9)! In my opinion this work can be better motivated/presented as just a standalone modification to the Transformer layer.
- While it is impressive that the proposed approach performs better (or on par with, in the case of character-level language modeling) than the previous state-of-the-art models which are larger, to me it is not immediately clear if the benefit is coming from the proposed modifications, or something else (e.g. some of the things mentioned in 4.3). I understand most of this is taking from prior work, but as we all know, in deep models various architectural/hyperparameter modifications can interact in unexpected ways. Therefore, at a minimum, I would like to see the performance of a comparable model with the same exact setting, except for the persistent attention layer (i.e. N = 0 but using the feedforward sublayers). (If I understand the work correctly, this baseline should have comparable number of parameters?)
- The ablation studies are good but it would be good to see them on the word-level task as well.
- What is the performance of the FF-attn baseline with the same depth? (I understand the number of parameters would be  larger, but does this model perform as well as the Dai et al. 2019 work?)
- What if you combine this with the FF sublayer as well?
- Have you tried qualitatively analyzing the attention distributions? What are some examples in which the persistent memory vectors are attended to most? What are some examples in which the attention distribution for this model differs considerly versus regular self-attention layers?
[EDIT after author rebuttal]
Thank you very much for the rebuttal. I have updated my score to reflect the latest iteration of the paper.