position_ids parameter cannot work with past parameter for GPT2Model during batch inference
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
Hi, @patrickvonplaten.  I just finished training a GPT-2 model by using the `GPT2Model` class, and I try to speed up the inference by using the batch inference (very similar to #3021 ). However, I found that the `position_ids` parameter cannot work with the `past` parameter, and it raises the Error: `RuntimeError: The size of tensor a (32) must match the size of tensor b (2592) at non-singleton dimension 0`.
![微信图片_20201007002211](https://user-images.githubusercontent.com/27548710/95229447-2b6f4980-0833-11eb-8cf4-07465f8935ec.png)
I found the exception happens in the `modeling_gpt2.py` line 471, so I check the original codes of the `GPT2Model` class. In `modeling_gpt2.py` line 426 to line 427 (line 558 to line 559 in the lastest original code):
```python
if position_ids is not None:
    position_ids = position_ids.view(-1, input_shape[-1])
```

Actually, during using the `past` parameter for speeding up the inference, the input_shape is `[batch_size, 1]`, but the `position_ids` is `[batch_size, seq_length]`. So, when we use the `past` and `position_ids` at the same time, the position_ids will be converted into a wrong shape `[batch_size*seq_length, 1]` (the shape we want should be `[batch_size, seq_length]`). For example, as shown in the figure, the `batch_size` is 32 and the `seq_length` is 81, and the generated position_ids shape is `[2592, 1]` (32*81=2592), but the correct position_ids shape should be `[32, 81]`.
So I think it may be a bug, but I am not so sure about it. Can you guys help me to figure it out?

Here are the environment variables in my system:
* transformers==2.11.0
* pytorch==1.5.1
* python==3.6.11

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:
Hey @gmftbyGMFTBY - I think the better approach to tackle your problem would actually be this one here: https://github.com/huggingface/transformers/issues/3021#issuecomment-681792104 .

This way you should not run into any errors regarding the position_ids<|||||>Hey, @patrickvonplaten, it works for me. Thank you so mcuh.