Masking issues with GPT2LMHeadModel.generate()
Is this intended behavior, that padding a sentence and attention_mask it will not give the exact same generation result comparing to the same sentence unpadded?

Edit: [This notebook](https://colab.research.google.com/drive/1oyFRFigtSNUYwKO1EQPRHfEqke0-F6_N?usp=sharing) demonstrates this, with the newest version available on colab. I realized that I didn't turn sampling off with the example below but the colab one has sampling off.

```
>>> gpt2_model = transformers.GPT2LMHeadModel.from_pretrained("gpt2")
>>> gpt2_model.generate(torch.tensor([[100,200,300]]))
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
tensor([[100, 200, 300,  84,  12,  75,  84,  12,  75,  84,  12,  75,  84,  12,
          75,  84,  12,  75,  84,  12]])
>>> gpt2_model.generate(torch.tensor([[100,200,300,50256]]),attention_mask=torch.tensor([[1,1,1,0]]))
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
tensor([[  100,   200,   300, 50256,   198,   198,     7,    16,     8,   383,
          3381,   366,    75,     1,  1724,   262,  4129,   286,   262,  4731]])
```
Hey @xxbidiao, 

For batched generation GPT2 has to be used in quite a special way... -> could you check out [this](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/2) forum post to see whether this makes sense for you?<|||||>```
import torch,transformers
gpt2_model = transformers.GPT2LMHeadModel.from_pretrained("gpt2")
print(gpt2_model.generate(torch.tensor([[100,200,300]]),do_sample=False))
print(gpt2_model.generate(torch.tensor([[100,200,300,50256]]),attention_mask=torch.tensor([[1,1,1,0]]),do_sample=False))
print(gpt2_model.generate(torch.tensor([[50256,100,200,300]]),attention_mask=torch.tensor([[0,1,1,1]]),do_sample=False))
```

```
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
tensor([[100, 200, 300,  84,  12,  75,  84,  12,  75,  84,  12,  75,  84,  12,
          75,  84,  12,  75,  84,  12]])
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
tensor([[  100,   200,   300, 50256,   198,   198,     7,    16,     8,   383,
          3381,   366,    75,     1,  1724,   262,  4129,   286,   262,  4731]])
tensor([[50256,   100,   200,   300,    84,    12,    75,    84,    12,    75,
            84,    12,    75,    84,    12,    75,    84,    12,    75,    84]])
```

Looks like it works! Will double check. Thank you!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.