The attention layer in a neural network model provides insights into the model's reasoning behind its prediction, which are usually criticized for being opaque. Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019;Vig & Belinkov, 2019). Amid such confusion arises the need to understand attention mechanism more systematically. In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not). Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.
CONTRIBUTIONS:
I use (unqualified) “self-attention” to refer to attention of tokens in a sequence to other tokens in the same sequence, as described by [some corrected version of] Eq (1) and the paragraph following it (citing Bahdanau et al. 2015). This contrasts with “Transformer self-attention” and “cross-sequence attention”.
C1. Self-attention is gating. (Prop. 4.1, Sec. 4)
C2. Gating cannot provide explanations in the way that attention is alleged to do.
C3. Single input sequence models deploy only self-attention, so by C1-C2, attention cannot be used for explanation of these models; attention in two input sequence models is not limited to self-attention and is not equivalent to gating, so attention can provide explanations in these models.
C4. Outputs are sensitive to alteration of attention weights for two-sequence but not one-sequence models. (Tables 2-3 vs. 1; Figs. 2-3) 
C5. Human judgments (“manual evaluation”) of the intuitive importance, for the output, of items with highest attention weights show that such intuitive importance is found for both one- and two-sequence models. (Fig. 5)
RATING: Reject
REASONS FOR RATING (SUMMARY). Because the modeling experiments closely follow previous work, the primary contribution rests on the account provided of why explanation-by-attention works only sometimes --- on the basis of the Proposition identifying attention with gating. But the reasoning and the math as presented are problematic. There is a worthwhile contribution from the human judgment experiment, but this is not sufficient to overcome the weakness with the main argument of the paper. (I also have reasons to question whether one- vs two-sequence inputs is the right distinction that needs to be accounted for.) 
REVIEW
C1. If this point were made in plain English, ‘attention weights and gating units both multiply activation values’, I would say, “yes, obviously”. But the point is stated as a Proposition, with a Proof, so the math given should be correct. I don’t see a way to make it correct, and that shouldn’t be the job of the reader anyway. There are several errors in the dimensions of the matrices which make the equations incoherent. Eq. (1) contains W*h, a matrix product, where the dimensions stated are W in R^{d x d’} and h in R{T x m}; these cannot be multiplied. This unclarity about matrix dimensions propagates into Prop. 4.1. In the definition of g(X), we have WX + b, where b is presumably a vector. Addition then requires that WX also be a vector, but X is stated to be in R^{n x d}, so WX cannot be a vector. Whether WX + b is actually a vector or a matrix, g(X) = c^T tanh(WX + b) is not a matrix: it is either a scalar or a vector. But this can’t be. The definition of h uses the elementwise product, which requires that both arguments have the same dimensions, so g(X) must have the same dimensions as f(X). We’re told f is the identity function, so f(X), like X, must be a matrix. Furthermore, the statement of Prop. 4.1 says that self-attention can be interpreted as *a* gating unit. By the standard definition of ‘unit’, this should mean that self-attention is a scalar.
Throughout the paper, we are never told what kind of “RNN” is being assumed. If the RNN unit contains gates, as in an LSTM or a GRU, I can imagine that the intention is for Prop. 4.1 to say that (*) “the effect of self-attention can be reproduced without attention by adjusting the weights in the gating mechanism already present in the RNN”, so that attention doesn’t increase the capacity of the model. But what I see in the paper does not convince me that (*) is true. (Because of the kind of global normalization required by the softmax, I actually suspect it is not.)
C2. I don’t see why (formally or intuitively) gating is not a legitimate candidate for explaining the behavior of networks containing gates; I would assume just the opposite, actually. How can it *not* be part of a satisfactory explanation? And why should changing a name from “attention” to “gating” have any bearing on whether it (whatever it is called) can potentially serve for explanation of network behavior?
C3. Leaving the formalism aside, I don’t see intuitively why, whatever an analysis of self-attention might entail about explanation, the same implication shouldn’t apply to straightforward (not Rocktaeschel) attention when two sequences are present. Why can’t we just treat the concatenation of the input sequences as a single input sequence, as standardly done for example for the Transformer? If the formal content of Prop. 4.1 were clear, perhaps this could be justified, but it is simply asserted without justification in the proof that “the same reduction does not hold in the case of pair sequence”.
C4. Claims C1-C3 attempt to give an account for why various tests of the explanatory role of attention turn out positive for two-sequence but not one-sequence tasks, a pattern previously reported and verified with new results in the present paper. I fear however that one- vs two- is not the correct empirical generalization about attention that one should try to account for. Messing about with attention weights would not be expected to alter outputs if the output is determined by the input considered as a bag of words. And there is a troubling confound in the tasks compared: the one-sequence tasks are sentiment and topic classification, where a BOW model should do very well – and I suspect that is the real reason why these tasks don’t show strong sensitivity to attention weight distribution. But the two-sequence tasks are NLI and QA, where (ideally) BOW models should not do nearly so well: paying attention to the right tokens should be important. The same is true of translation. So the confound in the tasks examined leaves it undetermined whether the crucial factor to account for is one- vs two-sequence or BOW-friendly vs BOW-unfriendly tasks.
C5. Put together, the human judgments and the sensitivity-to-altering-attention-weights seem to indicate that attention tends always to be allocated to intuitively important tokens, and that matters for the output of the two-sequence models but not for the one-sequence models. This is what we’d expect if attention is always being allocated appropriately, but for BOW-friendly tasks that doesn’t make much difference.