Unexpected behaviour with past_key_values in GPT2
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.11.2
- Platform: Ubuntu 20.04
- Python version: 3.8.10
- PyTorch version (GPU?): 1.9 GPU
- Tensorflow version (GPU?):
- Using GPU in script?: rtx 3070
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten, @LysandreJik
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): GPT2LMHeadModel,  gpt2-medium

When using GPT2LMHeadModel with past_key_values as an input, the output logits are different than expected.
I expected that when I run inference with the context ''Image of a" and when I run inference with the context "a" and past_key_values of previous run with the context "Image of", the output logits will be the same, but they are slightly different. I added code below.
Is this a bug, or am I missing something?

## To reproduce

Run the code:

```
from transformers.models.gpt2 import GPT2LMHeadModel, GPT2Tokenizer
import numpy as np
import torch

seed = 0
device = "cuda" if torch.cuda.is_available() else "cpu"
# set Random seed
torch.manual_seed(seed)
np.random.seed(seed)

lm_tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
lm_model = GPT2LMHeadModel.from_pretrained('gpt2-medium', output_hidden_states=True)
lm_model.to(device)
lm_model.eval()

raw_text = 'Image of a'
tokenized_cond_text = lm_tokenizer.encode(lm_tokenizer.bos_token + raw_text)

context_t = torch.tensor(tokenized_cond_text, device=device, dtype=torch.long)
while len(context_t.shape) < 2:
    context_t = context_t.unsqueeze(0)
output_so_far = context_t

past = lm_model(output_so_far[:, :-1])["past_key_values"]
last = output_so_far[:, -1:]

logits_past = lm_model(last, past_key_values=past)['logits']
logits_normal = lm_model(output_so_far)['logits'][:, -1:, :]

print(logits_normal.mean().cpu().item(), logits_past.mean().cpu().item())
```
My output:
`-59.50511169433594 -59.50926971435547`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I expected for the logits to be the same (and therfore have the same mean). Am I missing something?

<!-- A clear and concise description of what you would expect to happen. -->

Hey @YoadTew - good observation! 

We consider a tolerance of 1e-3 to be the same, so they are the same in this case :-) 
To give you a bit more background why they are slighly different:

If we make use of `past_key_values` then only the last query vector is forwarded through the model. If we don't make use of `past_key_values` then all query vectors are forwarded whereas all but the last query vector are masked with -10,000 a softmax of -10,000 is in comparison to other values almost always 0, but is not exactly 0 which leads to some small differences between padded forward and non-padded forward<|||||>@patrickvonplaten Oh really, I didn't know it. So the masked self attention is not perfect and it's the reason for the difference.

The reason I noticed it is that the slight difference resulted in a slightly diffrent text generation. Thanks for the response.<|||||>I just ran into a similar issue which seems to be related to this. I ran the above setup myself and discovered that the error compounds over multiple iterations and the difference becomes quite significant:

```python
from transformers.models.gpt2 import GPT2LMHeadModel, GPT2Tokenizer
import numpy as np
import torch

# set Random seed
seed = 0
torch.manual_seed(seed)
np.random.seed(seed)

tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
model = GPT2LMHeadModel.from_pretrained('gpt2-medium', output_hidden_states=True)
model.eval()

tokens = tokenizer.encode(tokenizer.bos_token + 'My name is')
x = torch.tensor(tokens).unsqueeze(0)

with torch.no_grad():
  out = model(x)
  x_ = out.logits[:, -1].argmax(-1).long().unsqueeze(1)
  print(tokenizer.decode(x_.item()))

  for i in range(16):
    x = torch.cat([x[:, :-1], x_], dim=1)

    out = model(x_, past_key_values=out.past_key_values)
    logits1 = out.logits[:, -1]
    logits2 = model(x).logits[:, -1]
    print(logits1.mean().item(), logits2.mean().item())

    x1, x2 = logits1.argmax(-1), logits2.argmax(-1)
    t1, t2 = tokenizer.decode(x1.item()), tokenizer.decode(x2.item())
    print(t1, t2)
    assert x1 == x2, f"predictions weren't equal: '{t1}' and '{t2}'"
    
    x_ = x1.long().unsqueeze(1)
```
Running this snippet gives the following output before failing the assertion:
```
 John
-43.53556823730469 -48.440670013427734
, ,
-38.2783088684082 -59.84267807006836
 and  and
-39.929866790771484 -71.5892562866211
 I  the
```
So not only does the difference in logits become quite large, eventually the output will be different as well, which is causing issues in my personal project.

Is there a reason for masking the query vectors with -10'000 instead of -infinity? Or am I doing something wrong that's causing this non-deterministic behavior?<|||||>the reason we picked -10,000 is because it works fine with `fp16`. Does changing `-10,000` to `-inf` solve your problem?<|||||>Thank you for the followup! Is there a quick way to modify the attention masking or do I need to edit the Huggingface source code?<|||||>I'm afraid you probably have to edit the Hugging Face source code at the moment...out of curiosity - what is the use case that requires `use_cache=True` to give exactly the same outputs as `use_cache=False`?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.