Clarification regarding past/layer_past in GPT-2
## ❓ Questions & Help

I'm hoping to get some clarification regarding how past/layer_past are meant to work in GPT-2. My prior impression was that the query/key/value at every layer (other than the first) should be influenced by all tokens the model is able to see. As such, it shouldn't make much sense to use pre-computed key/value activations from prior steps. Could you clarify this issue?

(Hope my question makes sense.)
the `past` variable in GTP-2 stores all previously computed key and value vectors. Because GPT-2 uses masked self-attention only the query vectors of previous tokens are updated at every step, but not the key and value vectors. Therefore the `past` variable can be used to speed up decoding. 

To better understand how GPT-2 works, I highly recommend reading [the Illustrated GPT-2](http://jalammar.github.io/illustrated-gpt2) especially the part: 

**The Illustrated Masked Self-Attention**<|||||>Thanks for the response! 

I'm still not sure this is intuitive for me. The linked article mentions: "Now in the next iteration, when the model processes the word robot, it does not need to generate query, key, and value queries for the a token. It just reuses the ones it saved from the first iteration:", which seems to imply Q, K and V are reused, whereas it seems we're only (optionally) reusing K and V. 

On the other hand, I'm not seeing where in the code that masked self-attention only affected the query vectors. It seems to be that attention masking is applied to the scoring vectors at each layer, and that should affect the generated Q/K/V for all downstream layers.

It feels like there may be some key part of the intuition I'm missing, so thanks for the help.<|||||>I read the article @patrickvonplaten pointed. I am still very confused about the usage of attention when there are "past" vectors. The dimension just doesn't match.

19 is the batch size here; I am using gpt-2 small. 40 is the encoding seq len; 23 is the decoding seq len.
Input document past[0] size: torch.Size([2, 19, 12, 40, 64])
to-decode-sequence input embedding: torch.Size([19, 23, 768])
mask of the input document: torch.Size([19, 40])
mask of the to-decode-sequence: torch.Size([19, 23])
concat of two masks: torch.Size([19, 63])    // 63=23+40

concat doesn't work          // _decoder_output = self.transformer(past=presents,attention_mask=concat_attention_mask,   inputs_embeds=gt_input_embedding) fails

to-decode-mask doesn't work      // _decoder_output = self.transformer(past=presents,attention_mask=partial_oracle_trajectory_mask,inputs_embeds=gt_input_embedding ) fails


Error message for concat:
`    attention_mask = attention_mask.view(-1, input_shape[-1])
RuntimeError: shape '[-1, 23]' is invalid for input of size 1197`

Error message for to-decode-mask only
`RuntimeError: The size of tensor a (63) must match the size of tensor b (23) at non-singleton dimension 3
`

@zphang any idea? not sure if this correlates with what you said tho. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hi @jiacheng-xu - sorry for answering that late. I think the problem with the attention mask was recently fixed in PR #3033. The issue was also mentioned in #3031 I think. Let me know if you still have problems when using the current version of master.