RuntimeError: The size of tensor a (1025) must match the size of tensor b (1024) at non-singleton dimension 3
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): gpt2

Language I am using the model on (English, Chinese ...): english

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)





The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
I want to generate samples with length = 2000
## To reproduce

Steps to reproduce the behavior:
1. run the command
python run_generation.py     --model_type=gpt2     --model_name_or_path=<output_dir_of_finetuned_model>     --length=2000     --num_return_sequences=10     --stop_token='<|endoftext|>
'


## Expected behavior
The error that I am getting is
w = torch.where(mask.bool(), w, self.masked_bias.to(w.dtype))
RuntimeError: The size of tensor a (1024) must match the size of tensor b (1025) at non-singleton dimension 3

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Python 3.8
- Python version:
- PyTorch version (GPU?): 10.1
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

Length of 2000 is too long for GPT2, this won't be possible. 
You can do two things here:

1) You chunk your generation which means that you first produce a length of up to 1000 and then use a bit of that (100 or so tokens) as context to generate the next 900 tokens and the same again until you hit 2000.

2) You can use this Reformer model: https://huggingface.co/google/reformer-enwik8 which can handle sequences up to 65000. Currently generation with Reformer is painfully slow though :-/ This should be improved in the coming weeks :-) 