In this work we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and the aggregation of context into hidden tokens. We show that attention weights are not unique and propose effective attention as a complementary tool for improving explanatory interpretations based on attention. Furthermore, we are the first to show that input tokens retain to a large degree their identity across the model. We also study how identity information propagates and find that it is mainly encoded in the angle of the embeddings and gradually decreases with depth. Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantification method based on gradient attribution. Overall, we show that self-attention distributions are not directly interpretable and present tools to further investigate Transformer models.
CONTRIBUTIONS:
Topic: Analysis of self-attention in the standard Transformer, applied to models of language. 
Notation for review (see also definitions in C1-C4): cell(t,L) is the cell in layer L at input-string-position t (“corresponding to input token t”); Z(t,L) is the output vector for cell(t,L). Also, in the review, points are sometimes labeled by “(label)”, which is intended to label the point that immediately *follows* it in the text.
C1. “Attention identifiability.” Uniquely identifying the attention-weight vector from the output of attention is not possible when D < T: For an individual head, the vector R returned by attention uniquely determines the attention-weight vector A only if the dimension D of the value vector equals or exceeds the length T of the input. 
C2. “Effective attention vector” A1: When D < T, there is a method for uniquely decomposing A as A = A1 + A2 where A2 has no effect on R. 
C3. “Token identifiability” method M. Identifying t from Z(t,L): A method M is proposed for this, and experimental results presented showing that, roughly, M succeeds at lower levels L but can fail at higher levels.
C4. “Hidden token attribution” method: A gradient-attribution-based method is proposed to quantify C(t,t’), the proportional contribution of input token t’ to Z(t,L). C(t,t) is the “[same-]token contribution”; the “contextual contributions” are the C(t,t’) for t’ =! t. Experimental results are presented showing that after L = 1, 6, 12 the median token values are about 30%, 15%, 10%.
MAIN ARGUMENT: Interpretation of the weight of cell(t,L)’s attention to cell(t’,L) as a measure of the influence of token t’ on the level-L representation of token t is (P0) frequent, but problematic, because (P1) the attention weight itself includes an irrelevant quantity (the A2 contribution, which should be omitted to get the effective attention [C2]) and because (P2) Z(t,L) is not properly viewed as representing token t, since (P2) it can be insufficient to even determine which token t it corresponds to [C3] as (P3) it can contain strong contributions from other tokens [C4]. 
RATING: Weak accept
REASONS FOR RATING (SUMMARY). The paper raises important challenges to a standard practice for interpreting the knowledge in Transformer models (especially BERT) by examining attention weights. It proposes methods for reaching more theoretically justified interpretations. It introduces important new general interpretive concepts in the process. The main argument is sufficiently well supported by theoretical and empirical results to justify making the ICLR community aware of it, although there is a missing step in the argument: justification that the proposed means of resolving the unidentifiability of attention yields an optimal result.
REVIEW
Weaknesses.
Minor complaint concerning exposition: The central result, C1 above, is actually rather obvious, unless I’m missing a subtlety (in which case it would be useful for that subtlety to be pointed out explicitly in the paper). (*) The dimension-D vector R is a linear combination of T vectors; if T > D, these T vectors cannot be linearly independent, so the weighting coefficients that yield R cannot be unique. The argument presented is rather overblown and may hide the obvious correctness of the result. However, I think over-deriving is clearly preferable to under-deriving, so this complaint truly is extremely minor. It might be worthwhile to add a sentence like (*) however to make sure that simple observation doesn’t get lost in the formalism. A further advantage of the more detailed derivation is that it sets up nicely the unique decomposition of A into the component A2 in the null space and the component A1 orthogonal to the null space (of the matrix containing the T vectors being linearly combined). 
A more substantial question here: What argument or evidence is there that this particular decomposition A = A1 + A2  is the one that yields a definition of “effective attention” that is best for interpretative purposes? Is there a metric of success that would validate this? Given that A is underdetermined, there is a whole subspace of attention-weight vectors that yield the same result R of attention. The proposed choice of A1 within that space is a natural one, but what makes it the *right* one? The incontrovertible point is that the A vector a model happens to use has a degree of arbitrariness in it that makes using it for interpretation problematic. But what is the right criterion for picking an alternative “effective” version of A, A1? Being orthogonal to the null space is one well-defined choice, but is it clearly the right choice? (I believe this choice is also the one with minimal L2 norm. Is that important?) For example, Sec. 3.3 shows that raw attention, but not effective attention, gives evidence that attention shifts from [CLS] to [SEP] to periods and commas as L increases. This shows they are different, but what is the argument that the raw-attention conclusion is less ‘correct’ than the effective-attention conclusion here? 
More generally, it would strengthen the paper even further to document more concrete cases where effective attention yields different interpretive conclusions than raw attention, especially to the extent that it can be argued that the former interpretations are somehow better than that latter.
Strengths.
The paper gives convincing evidence of the MAIN ARGUMENT’s propositions (P0) – (P3).
(P0) [Interpretation of the attention weight as a measure of the influence of one token on another] is frequent] Sec 4, 2nd paragraph, cites 16 papers to illustrate the point. While I have not examined all of them, even allowing for some possible disagreements of interpretation, I am confident there is ample relevant evidence among these papers and probably others as well.
(P1) [the attention weight itself includes an irrelevant quantity [C2]] The argument provided for this in Sec 3 seems correct to me, but as stated above as a ‘minor complaint concerning exposition’, it may not make as clear as possible the obviousness of the conclusion. 
(P2) [Z(t,L) can be insufficient to determine which token t it corresponds to [C3]] This point is operationalized in Sec 4.2 by picking a function-approximation architecture G, and attempting to use it to perform the map Z(t,L) -> t, for a given L and for all input strings and tokens t. A linear and a non-linear MLP G are used. An experiment applying the pre-trained BERT-base Transformer to the Microsoft Research Paraphrase Corpus gives success rates that fall to 73% and 82% by final layer 12. That 18% of tokens cannot be recovered from their corresponding layer-12 encoding is the basis of the claim that equating attention from cell(t,L) to cell(t’,L) with attention from token t to token t’ is not justified in general. 
That the non-linear approximator does 9% better than the linear one is the basis of the claim that ‘the non-linearities in BERT play an important role in maintaining token identifiability’ (Sec. 4.2). This claim should be spelled out explicitly. It reads to me to say that if the non-linearities in BERT were removed, token identifiability would be reduced, but the experiment does not show that. Rather, it shows that, in the presence of the non-linearities in BERT, a linear approximation to the token-identifying map G is significantly less accurate than the non-linear approximation: hardly a surprising conclusion. This bit of the argument needs to be clarified.
(P3) [Z(t,L) can contain strong contributions from tokens t’ =! t [C4]] As defined in Sec. 5, the contribution of token t’ to Z(t,L), for a given L, is proportional to the norm of the gradient of Z(t,L) with respect to the input from token t’; these are normalized to sum to 1. This is a transparent definition that resembles the previous gradient attribution proposal, but applied to internal rather than output representations. Using the same experimental set-up, it is shown that the average contribution of token t to Z(t,L) declines from 30% to 10% as L goes from 1 to 12. The proportion of values t such that the largest contribution to Z(t,L) is from a token t’ =! T rises from 0 for L = 1 through 3 to over 30% for L = 10 through 12. This is further evidence for the claim that equating attention between Z(t,L) and Z(t’,L) with attention between tokens t and t’ is problematic.