Self-attention-based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called Explicit Sparse Transformer. Explicit Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on a series of natural language processing and computer vision tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Explicit Sparse Transformer in model performance. We also show that our proposed sparse attention method achieves comparable or better results than the previous sparse attention method, but significantly reduces training and testing time. For example, the inference speed is twice that of sparsemax in Transformer model.
CONTRIBUTIONS:
C1. Sparse Transformer: A modification of the Transformer, limiting attention to the top-k locations. (That is a complete statement of the proposed model.)
C2. Experiments showing that, quantitatively, the Sparse Transformer out-performs the standard Transformer on translation, language modeling, and image captioning.
C3. Experiments showing that, qualitatively, in translation, when generating a target word, the Sparse Transformer better focusses attention on the aligned source word 
RATING: Reject
REASONS FOR RATING (SUMMARY). The innovativeness seems low given the several previous proposals for sparse attention, the results are not dramatic enough to compensate for the lack of originality, and the comparison to other models is wanting.
REVIEW
Strengths: The paper is clearly written. The question of whether the Transformer’s attention is too diffuse is of interest. The proposal is admirably simple. The quantitative metrics include comparison against many alternative models.
Weaknesses: A primary area of deficiency concerns the relation of the proposed model to other proposals for sparse attention: the authors cite 5 of them (and 2 more are cited in the comment by Cui). The paper should clearly identify the differences between the proposed model and earlier models: it does not discuss this at all. The deficiencies in these previous models should be clearly stated and demonstrated:  they are only described as “either restricted range of attention or training difficulty” (Sec 6). A rationale for why the proposal can be expected to remedy these deficiencies should be stated clearly: it is not stated at all. Experimental demonstration that the proposed innovation actually remedies the identified deficiencies should be provided, but is not.
A proposal to use a top-k filter immediately raises the question of the value of k. This is not discussed at all. In particular, no empirical results are given concerning the sensitivity of the reported successes to choosing the correct value for k. We are only told that “k is usually a small number such as 5 or 10” (Sec 3). The experimental details in the appendix do not even state the value of k used in the models reported.
It is an interesting discovery that in the translation task, attention at the top layer of the standard Transformer is strongly focused on the end of the input. This is described as an “obvious problem” (Sec 7). But it can’t obviously be a problem because the performance of the standard Transformer is only very slightly lower than that of the Sparse Transformer: if anything is obvious, it is that processing in the standard Transformer packs a lot of information into its final encoding of the end of the input string, which functions rather like an encoding of the entire sentence.
Presumably, the experimental results reported are those from a single model, since we are not told otherwise. There should be multiple tests of the models with different random initializations, with the means and variances of measures reported. It is possible, however, that limitations of computational resources made that infeasible, although the Appendix seems to indicate that no hyperparameter tuning was done, which greatly reduces computational cost.
COMMENTS FOR IMPROVEMENT, NOT RELEVANT TO RATING DECISION
Although the tiny sample of visualized attention weights provided is useful, a large-scale quantitative assessment of a main claim concerning translation might well be possible: that attention is in fact concentrated on the aligned word might be testable using an aligned bilingual corpus or perhaps an existing forced aligner could be used.
Much space could be saved: it is not necessary to review the standard Transformer, and the modification proposed is so simple that it can be precisely stated in one sentence (see C1 above): the entire page taken up by Sec. 3 is unnecessary, as it adds only implementation details.
Errors that took more than a moment to mentally correct, all on p. 12:
The definition of the BPC should be E[log P(x(t+1) | h(t))]: all parentheses are missing
“regrad” should be “regard”
“derivative” should be “differentiable” in the final sentence