Recurrent Neural Networks have long been the dominating choice for sequence modeling. However, it severely suffers from two issues: impotent in capturing very long-term dependencies and unable to parallelize the sequential computation procedure. Therefore, many non-recurrent sequence models that are built on convolution and attention operations have been proposed recently. Notably, models with multi-head attention such as Transformer have demonstrated extreme effectiveness in capturing long-term dependencies in a variety of sequence modeling tasks. Despite their success, however, these models lack necessary components to model local structures in sequences and heavily rely on position embeddings that have limited effects and require a considerable amount of design efforts. In this paper, we propose the R-Transformer which enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks. The proposed model can effectively capture both local structures and global longterm dependencies in sequences without any use of position embeddings. We evaluate R-Transformer through extensive experiments with data from a wide range of domains and the empirical results show that R-Transformer outperforms the state-of-the-art methods by a large margin in most of the tasks. We have made the code and data publicly available 1 .
Summary: This paper proposes a new architecture, R-Transformer, that blends the Transformer networks and the recurrent networks, so as to better capture both the long- and short-term features. By injecting a local RNN layer at every level of the network, the authors hoped to enhance the Transformer's ability to model locality structure. To demonstrate the modeling power of R-Trasnformer, the paper evaluates the effectiveness of R-Transformer on 4 different sequence tasks (seqMNIST, polyphonic music, character- and word-level PTB).
Contribution: The authors propose an architecture that combines the practices of recurrent and feed-forward sequence models. However, I have major concerns regarding the novelty this paper, the various claims it makes, as well as its experiment setting.
----------------------------------------------------------------------------
Major issues/questions:
1. The techniques proposed by this paper lack novelty. For instance, the entire section 3.2 is simply the original design of the multi-head self-attention by Vaswani et al. The major difference between R-Transformer and the original Transformer is the replacement of positional embedding with an RNN layer, but (in my opinion) the authors did not demonstrate sufficiently its effectiveness via ablative studies (see below). Moreover, some prior works have already exploited the locality structure in Transformers. For instance, [1] showed that a sparse, local Transformer can work extremely well and be very efficient (they achieved SOTA on large-scale char-level language modeling tasks).
2. The experiments do not entirely convince me. 
    i) The authors use the "same hidden size for R-Transformer and Transformer." But in fact, as the R-Transformer has one extra RNN/LSTM/GRU layer at every level of the network, the tests were carried out (in effect) using a larger model than the baselines. I think the authors should instead control the # of model parameters, especially since you are running only on small tasks with small-sized models.
    ii) It is nice that the authors tested R-Transformer on a variety of tasks--- this is important. However, in no way do these number achieve the levels of the "state-of-the-art", which the authors claim at the end of Section 1 (e.g., [2] has better number on seqMNIST and character-level PTB, and the Transformer-XL actually achieves <55 perplexity on word-level PTB). Therefore, the numbers don't look particularly appealing to me. 
    iii) Lack of more challenging, or large-scale experiments. Sequential MNIST is known to be a relatively simple task, and alternatives such as sequential CIFAR-10 or permuted sequential MNIST are more valuable "small sequence tasks." (e.g., prior works studying long-term sequence model dependencies [3]) I also think that benchmarking R-Transformer on large-scale tasks like WikiText-103 or 1Bword (which prior works like QRNN, TrellisNet, RMC and Transformer-XL all explored) would be much more indicative of its usefulness (and should not be left out in such a paper). For instance, it would be useful to compare on small models with controlled model size, even if larger models that require TPUs are not available.
    iv) Lack of ablative study. Table 4 seems to suggest that Transformer-XL is better than R-Transformer. Is it because of their usage of the relative positional embedding? How does the "finite window size" affect the performance of R-Transformer? Why don't you use Transformer-XL for the other 3 tasks? Lots of interesting questions are unanswered here. More details in (3) below.
3. Regarding the motivation to insert a local RNN at the start of each layer. The authors claim (by citing Al-Rfou et al.) that the positional embedding (PE) have only limited effect. But in fact, Al-Rfou only says that the PE features (which is added only at the first layer, by the design of Vaswani et al.) can get lost once the transformer gets very deep--- which is totally expected. Therefore, Al-Rfou et al. propose to use a learnable embedding for each layer. This in no way suggests that positional embedding has "limited effect" (Note that in their ablative study, when they turned off the learnable PE, they also added back the original PE). Moreover, the authors didn't use convolution to capture local data, because it "completely ignores the sequential information of positions within the local window". That is true. However, **stacking** convolutional layers does capture sequential information. Prior works like [2] showed exactly how temporal convolutions are related to finite-window RNNs. I would suggest the authors to at least compare these different options with ablative studies. I would at least expect a comparison of i) temporal convolution; 2) fixed-window RNN (LocalRNN); 3) unlimited-window RNN; 4) positional embedding; 5) relative PE (which Transformer-XL uses; is that why it's better in Table 4?).
4. The authors claim that the finite-window RNN captures local features. But doesn't that claim only applies to the first layer? Once the first layer multi-head attention mixes all input elements across the sequence, the "local" features fed into the second layer RNN will be, actually, **global** features? Doesn't that "defeat" the purpose of using a local RNN though?
============================
Minor issues that have mild or zero impact on the score:
5. Inconsistent notations. Equation (2) and (3) both describe the LocalRNN(...) function, but clearly have a different input-output signature. Instead of using italics, it's better to have well-defined notations. Another case is Equation (7): you have 'FeedForward(mt)', but 'mt' is not on the right-hand side of the equation at all. Moreover, the symbols used in Eq. (7) are currently inconsistent with Eq. (8). 
6. Show the # of parameters in the model in Table 1-4.
7. According to the code released in the dropbox URL, you precompute the index in the finite window and later called 'torch.index_select' to produce a tensor that is 'ksize' larger than the original sequence (cf. Line 130-143 in models/RTransformers.py in the dropbox folder). How does R-Transformer compare to Transformers in terms of speed and memory? Since you convert the batch dimension to (batch_size * seq_len) at every level of the network, I imagine that could slow down the process, especially in high-dimensional/large-scale experiments?
8. There are some typos (e.g., Sec. 4.2, LTSM) and grammatical mistakes in the paper (Sec. 3.3). 
9. In Section 4.2, the authors claim that both LSTM and TCN performed better than Transformers on the polyphonic music dataset because "these music tunes exhibit strong local structures". However, the difference between Transformer and LSTM is actually very small, and it's even better than GRU/vanilla RNN. Is that too big a claim to make? In polyphonic music datasets like JSB or Nottingham, there are still longer sequences with longer dependencies...
============================
Overall, I feel that this paper has a good motivation to combine different sequence model families (Transformers, TCNs, RNNs) to improve their modeling power. But at the same time, I feel that the experiments can be a lot stronger, and the paper has limited novelty when compared to prior works. I'm happy to consider adjusting my score if my concerns above are addressed. 
[1] "Generating Long Sequences with Sparse Transformers", https://arxiv.org/abs/1904.10509
[2] "Trellis Networks for Sequence Modeling", https://arxiv.org/abs/1810.06682
[3] "Learning Longer-term Dependencies in RNNs with Auxiliary Losses", https://arxiv.org/abs/1803.00144