We propose a novel neural architecture, Transformer-XL, for modeling longerterm dependency. To address the limitation of fixed-length contexts, we introduce a notion of recurrence by reusing the representations from the history. Empirically, we show state-of-the-art (SoTA) results on both word-level and character-level language modeling datasets, including WikiText-103, One Billion Word, Penn Treebank, and enwiki8. Notably, we improve the SoTA results from 1.06 to 0.99 in bpc on enwiki8, from 33.0 to 18.9 in perplexity on WikiText-103, and from 28.0 to 23.5 in perplexity on One Billion Word. Performance improves when the attention length increases during evaluation, and our best model attends to up to 1,600 words and 3,800 characters. To quantify the effective length of dependency, we devise a new metric and show that on WikiText-103 Transformer-XL manages to model dependency that is about 80% longer than recurrent networks and 450% longer than Transformer. Moreover, Transformer-XL is up to 1,800+ times faster than vanilla Transformer during evaluation.
This paper proposes a Transformer based RNN structure "Transformer-XL" to capture long-range contextual relations and targets on language model task. The idea is straightforward: it splits the input sequence into equal and fixed length segments, and recurrently apply the Transformer over the sequence of segments, in which the hidden states for the previous segment are treated as a memory to attend for the next segment. 
This paper is well-organized and well-written, and easy to follow. The empirical results also demonstrate the proposed model can achieve SoTA performance on several word- and character-based language model benchmarks. 
Pros:
1. The model is designed based on a careful engineering: 1) taking into account the history hidden states for long-term dependency modeling and 2) alignment scores calculated from multiple perspectives for relative position modeling and global significance capturing. In addition, in contrast to the previous Transformer-based language model, benefiting from the recurrent architecture, both training and decoding can be accelerated.
2. The experimental results show that the proposed Transformer-XL can surpass the baseline model and achieve new state-of-the-art perplexity or bpc on word- or char-based language model task. And, based on the proposed new metric, RECL, the analysis for context length modeling verifies the proposed model can make the best of long-range dependencies.
Cons:
1. The proposed model is ad-hoc and is only compatible with language model task. Is it possible to extend the proposed model to more general and practical tasks (e.g., seq2seq tasks)?
2. The absence of a popular language model benchmark, WikiText-2, which has been evaluated in most previous papers.
3. It is notable that there are no ubiquitous decoding techniques for the language used in both the proposed model and baselines, such as dynamical evaluation and continuous cache pointer. However, these techniques are essential for the RNN-LM baselines to achieve state-of-the-art performance, and has been standardly used in most previous works. Therefore, the comparison seems unfair. 
Minor comments: In Figure 1 and 2, it is better to include a legend explaining the meaning of different colors for different nodes.