Different logits for single/batch inputs on T5ForConditionalGeneration
### System Info

```shell
transformers                 4.18.0 
python 3.8.10
ubuntu
pytorch
T5ForConditionalGeneration
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("t5-small")
model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
model.resize_token_embeddings(len(tokenizer))
model.to("cuda")
model.eval()



# sequences
seq1 = "summarize: Calling the model (which means the forward method) uses the labels for teacher forcing. This means inputs to the decoder are the labels shifted by one"
output1 = "calling the model uses the labels for teacher forcing. inputs to the decoder"

seq2 = "summarize: When you call the generate method, the model is used in the autoregressive fashion"
output2 = "the model is used in the autoaggressive fashion."

seq3 = "summarize: However, selecting the token is a hard decision, and the gradient cannot be propagated through this decision"
output3 = "the token is a hard decision, and the gradient cannot be propagated through this decision"

input_sequences = [seq1, seq2, seq3]
output_seq = [output1, output2, output3]

# encoding input and attention mask
encoding = tokenizer.batch_encode_plus(
    input_sequences,
    padding="longest",
    truncation=True,
    return_tensors="pt",
)

input_ids, attention_mask = encoding.input_ids.to("cuda"), encoding.attention_mask.to("cuda")

# labels
target_encoding = tokenizer.batch_encode_plus(
    output_seq, padding="longest",  truncation=True, return_tensors="pt"
)
labels = target_encoding.input_ids.to("cuda")
labels[labels == tokenizer.pad_token_id] = -100

# Call the models
logits = model(input_ids=input_ids, labels=labels).logits

# Apply softamx() and batch_decode()

X = logits
X = F.softmax(X, dim=-1)
ids = X.argmax(dim=-1)
y = tokenizer.batch_decode(sequences=ids, skip_special_tokens=True)

print(y)
# results: batch_size=3

# [
#   'call the model uses the labels for teacher forcing  inputs to the decoder are',
#   'model is used in the constructegressgressive fashion ',
#   'token can a token decision, and the gradient cannot be propagated through this decision '

# ]

# results: batch_size =1 i.e. consider 1 seq each time

# ['call the model uses the labels for teacher forcing  inputs to the decoder are']

# ['the model is used in the auto-gressgressive fashion ']

# ['the token is a hard decision, and the gradient cannot be propagated through this decision ']
```

### Expected behavior

```shell
Having the same output sequences.
```

@patrickvonplaten can you please have a look ?