How to use GPT-2 for predicting the next word in batch
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:transformers 4.16.0.ver0
- Python version:3.7
- PyTorch version (GPU?):1.10.1
- Tensorflow version (GPU?):
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?:no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.-->
@patrickvonplaten @LysandreJik 

### Information


I'm using GPT2 for predicting the next word and I expect to get the loss. For running the model in batch, I should make paddings so that all the samples are of the same length. But when padding I got the significantly different loss values.

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
import numpy as np

device = 'cuda'
model_id = 'gpt2-medium'
model = GPT2LMHeadModel.from_pretrained(model_id).to(device)
model.eval()
tokenizer = GPT2Tokenizer.from_pretrained(model_id)
def get_loss(input,target,encoded=True):
    input=input.to(device)
    target=torch.tensor(target).to(device)
    #print(final_input,target)
    output=model(**input,return_dict=True)
    output = output.logits[0][-1:]
    print(output)
    loss_fct = torch.nn.CrossEntropyLoss()
    #print(output.view(-1, output.size(-1)).shape, target.shape)
    loss = loss_fct(output.view(-1, output.size(-1)), target)
    return loss.cpu().detach()

tokenizer.pad_token = '!'
tokenizer.padding_side = 'left'
text='I like playing the'
target=' piano'
target=tokenizer.encode(target)
text1=tokenizer(text, return_tensors="pt")
print(text1)
text2=tokenizer(text,padding='max_length',max_length=300, return_tensors="pt")
print(text2)
print(get_loss(text1,target))
print(get_loss(text2,target))
```

And I got the following results:

#### not padding

```
tensor([[-67.2690, -65.8415, -70.7127,  ..., -70.0382, -72.1321, -67.0170]],
       device='cuda:0', grad_fn=<SliceBackward>)
tensor(4.4344)
```

#### padding

```
tensor([[-205.1836, -207.1315, -211.1725,  ..., -218.3117, -216.3883,
         -205.4879]], device='cuda:0', grad_fn=<SliceBackward>)
tensor(16.0859)
```

The loss is much higher after padding. And the logits value is also strange. 
I have to pad on the left to make sure logits[-1:]  is always the encode state of the last input. 

## Expected behavior

I think that GPT2 can do this task but I don't know how to use it in batch. 
 
<!-- A clear and concise description of what you would expect to happen. -->
