We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new openvocabulary language modelling benchmark derived from books, PG-19.
## Updated review
I have read the rebuttal. First I'd like to thank the authors for the detailled rebuttal. 
The latest version of the paper adressed all my concerns, hence I change my rating to Accept.
## Original review
This paper presents a new variation of the Transformer model, named Compressive Transformer. The key novelty of this model is to preserve long range memory in a compressed form, instead of discarding them as previous models have done. This improves the long-range dependencies modelling capabilities of the approach. The model is evaluated on two common language modelling benchmarks and yields state of the art results in both of them. The paper also introduces a new benchmark for long-range dependencies modelling composed of thousands of books. The paper finally presents an analysis of the compressed memory and provide some insights, including the fact that the attention model uses the compressed memory. The model is also evaluated on two other tasks: speech generation and reinforcement learning on videos.
I think this paper should be accepted, mainly because:
- The proposed model is novel as far as I can tell. 
- The presented approach is significant, as modelling long-range dependencies is an important milestone in sequence modelling.
- The new benchmark is a good addition.
- The comparison with the relevant literature is thorough and well done.
- The experiments are convincing and demonstrate the viability of the approach, although some aspects can be improved (see below).
Detailed comments:
- About the character-level language modelling on Enwik8, the improvement is very small, it seems that the task doesn't benefit from have long-range memory, could it be because character-level modelling is less dependent on the long-range past? can the authors comment on that? It would also been interesting to evaluate the gain of the memory, for instance by varying the size of the compressed memory from 0 to 1152. 
- The WikiText-103 evaluation is interesting, specially Table 6, which shows the advantages of the model. However when comparing with the literature, it's not clear if the performance gain is due to the compressed memory or to the network capacity. A study with different lengths of the compressed memory (starting at 0) would bring some insights about that.
- In Section 5.6.2: can the authors justify why the attention weights were split in only 6 bins? creating a trended curve on only 6 points could be problematic, and I don't see why more bins couldn't be used.
- The speech analysis section (5.7) is not very insightful. It shows that the proposed model is on par with WaveNet on unconstrained speech generation, which is not very useful and feels a bit half-finished. I think that the authors should either commit to this study by constraining the model with linguistic features like in (Oord et al. 2018) and evaluate it in a TTS framework with subjective evaluation or discard this section entirely.