How to instantiate custom T5ForConditionalGeneration
## Information

I have problems using `T5ForConditionalGeneration`.

I want to evaluate performance of the model without any information from the encoder, i.e., figure out what is the lower bound of decoder performance. To do so, I use a model as follows during both training and validation:
```python
bs = labels.shape[0]
outputs = model(   
   input_ids=torch.zeros(bs, 1).long(),
   attention_mask=torch.ones(bs, 1).float(),
   labels=labels
)
```
where labels look like this:
``` python
[[ 103,   25,  214,  125,   34, 1416,  114,    3,   58,    1],
 [  11,  125,  103,   62,  217,    3,    5,    1, -100, -100],
 [ 131,  600,  631,   21,   80,  568,    3,    5,    1, -100]]
```
so the sequences apparently end with `</s>` and have paddings replaced with `-100`.

I take the loss from outputs as `outputs.loss` and get the most probable tokens as `tokens = outputs.logits.argmax(-1)` to calculate token accuracy and get a better notion about what is happening.

**I do not want to use pre-trained weights**, so I instantiate the model as follows:
``` python
config = AutoConfig.from_pretrained('t5-small')
model = T5ForConditionalGeneration(config)
```
At the beginning of the training, everything seems to be ok. The token accuracy is **~41%** and output distributions make sense (the token distribution on the first token in labels should roughly reflect frequency of the first tokens in the dataset, the decoder does not have any information from the encoder). However, after a while of training (1k steps), the model improves rapidly, train and val losses drop, and the model achieves accuracy hitting **80%** which is not expected at all. It is absolutely certain about the first token and predicts it correctly (which is not possible!), so I suspect that the input tokens somehow leak into the predictions or that the masking in the decoder is not causal.

**The issue does not occur when using the pre-trained weights**, i.e., instantiating the model as:
``` python
model = T5ForConditionalGeneration.from_pretrained('t5-small')
```
And the token accuracy is constantly around **49%** (verified after 15k steps).

Any ideas why or what am I doing wrong?   @patrickvonplaten

## Environment info

- `transformers` version: 4.9.2
- Python version: 3.7.0
- PyTorch version (GPU?): 1.10.1+cu113 (True)
