T5 shows weird behaviour in a sanity check of its pre-training task.
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): T5

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* no script, just a few simple commands. watch below.

## To reproduce

Steps to reproduce the behavior:

1. 
```
from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5ForConditionalGeneration.from_pretrained('t5-small')
input_ids = tokenizer.encode('The <extra_id_1> walks in <extra_id_2> park', return_tensors='pt')
lm_labels = tokenizer.encode('<extra_id_1> cute dog <extra_id_2> the <extra_id_3> </s>', return_tensors='pt')

outputs = model(input_ids=input_ids, lm_labels=lm_labels)
probs, preds = torch.topk(outputs[1], 10)
preds = preds[0]
for ind_lst in preds:
      tokenizer.convert_ids_to_tokens(ind_lst)
```
This yeilds the following:
```
['▁The', '▁Walking', '▁Park', '▁Walk', '▁', '▁walking', '▁the', '▁We', '▁In', '▁La']
['▁park', '▁Park', '▁parks', '▁Walking', '▁National', '▁walk', '▁', '▁Forest', '▁walking', 'park']
['▁park', '<extra_id_2>', '▁Park', '▁walks', '▁walking', '▁parks', '▁nature', '▁walk', '▁Walking', '▁and']
['▁park', '▁walks', '<extra_id_2>', '▁parks', '▁Park', '▁walk', '▁walking', 'walk', 'park', '▁also']
['▁the', '▁The', '▁', '▁park', '▁to', '▁Park', '▁and', '▁at', '▁walking', '▁in']
['▁park', '<extra_id_3>', '▁Park', '▁The', '▁parks', '▁parking', '▁', '▁car', '▁beautiful', '▁the']
['▁park', '▁Park', '▁Walking', '▁parks', '▁walk', '▁walking', '▁Walk', 'park', '▁National', '▁walks']
```

## Expected behavior
I would expect that '<extra_id_1>' will show up as a possible top10 prediction.
during pre-training T5 gets supervision to return the first token to be <extra_id_1>. So if it doesn't return it, it is weird behavior. I hope that I didn't get it wrong. Thank you. @patrickvonplaten 

## Environment info
     
- `transformers` version: 2.8.0
- Platform: Darwin-19.3.0-x86_64-i386-64bit
- Python version: 3.7.0
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
Hi @yuvalkirstain,


Thanks for your message! 

When you put in `input_ids` and `lm_labels` at the some time the following happens:
the `decoder_input_ids` are created automatically depending on the `lm_labels` (shift `lm_labels` to the right and pre-prend `<PAD>`) and look as follows: 
`<PAD> <extra_id_1> cute dog <extra_id_2> the <extra_id_3>`. 

see https://github.com/huggingface/transformers/blob/4e817ff41885063e08bb3bcd63e5adfd835b9911/src/transformers/modeling_t5.py#L1060 

Now in order for `<extra_id_1>` to show up, it should be guessed from `<PAD>`, but I'm not sure whether this makes much sense. After `<extra_id_1>` has been processed though from the input, it's much more likely that another `<extra_id_2>` is in the labels, so it does make more sense to me that `<extra_id_2>` is guessed.

Let me know if it does not makes sense to you :-) 
<|||||>ahh ok. Thank you so much and sorry that I bothered you with it. Really enjoying working with it so far :)