Fine-tuning t5-large: greedy predictions don't match teacher-forcing results
## Environment info
- `transformers` version: 4.9.0.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.0
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

Tagged contributors: @patrickvonplaten, @patil-suraj

## Information

Model I am using t5-large.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x] my own task or dataset: (give details below)

## Context
I am fine-tuning the T5-large pretrained model with my own training script on a custom task (learning to generate a simplified version of the parse tree for each sentence in the SST dataset).  

All works well when using normal model(inputs) to get logits (teacher-forcing inference).  When I try to use "generate()" to do a greedy inference (to predict a sequence without using the labels), the results that I get are **very different** than the TF inference (for the same training examples).  See below for test case and output.

## To reproduce

Steps to reproduce the behavior:

1. after model has been fine-tuned for about 100 steps, take a sample of your training data and compare the results from 
model(input_ids=src, decoder_input_ids=trg) to model.generate(input_ids=src)

Here is a **test case** I added to my eval loop, that reproduces the problem:

```
        x = "You 'll probably love it ."
        y = "( * ( ( ( * * ) ( * * ) ) * ) )"

        t5 = self.model.model

        src = self.tokenizer.tokenizer(x)["input_ids"]
        trg = self.tokenizer.tokenizer(y)["input_ids"]

        src = torch.tensor(src).to(torch.long).to(t5.device).unsqueeze(0)
        trg = torch.tensor(trg).to(torch.long).to(t5.device).unsqueeze(0)

        # get TF results
        logits = t5(input_ids=src, decoder_input_ids=trg)["logits"]
        tf_preds = torch.argmax(logits, dim=-1)[0]

        # get GREEDY results
        gi_preds = t5.generate(input_ids=src)[0]

        tf_y = self.arr_to_str(tf_preds, " ", True)
        gi_y = self.arr_to_str(gi_preds, " ", True)

        print("x: ", x)
        print("y: ", y)
        print("tf_y: ", tf_y)
        print("gi_y: ", gi_y)

```

## Test Case output
```
x:  You 'll probably love it .
y:  ( * ( ( ( * * ) ( * * ) ) * ) )
tf_y:  ▁( ▁( ▁* ▁* ▁* ▁( ▁ ) ▁ ▁* ▁( ▁ ) ▁ ) ▁ ▁ ) ▁ ) ▁ </s>
gi_y:  <extra_id_0> . <extra_id_1> . <extra_id_2> . <extra_id_3> . ▁You ▁ ' ll ▁probably ▁love ▁it ▁ . <extra_id_4> . <extra_id_5> ll ▁probably ▁love ▁it ▁ . ▁You ▁ ' ll ▁probably ▁love ▁it ▁ . ▁You ▁ ' ll ▁probably ▁love ▁it ▁ . <extra_id_6> . <extra_id_7> ll ▁probably ▁love ▁it ▁ . <extra_id_8> ▁ . <extra_id_9> ▁ . ▁ . <extra_id_10> ▁You ▁ ' ll ▁probably ▁love ▁it ▁ . <extra_id_11> ▁You ▁ ' ll ▁probably ▁love ▁it ▁ . 
<extra_id_14> ▁ . <extra_id_15> ▁ . ▁ . ▁ . <extra_id_16> ▁ . ▁ . ▁ . ▁ . ▁ . <extra_id_26> . ▁ . ▁ . ▁ . <extra_id_27> . ▁ </s>
```

## Expected behavior
The results should match exactly, but instead, the greedy/generate output seems to come from a completely different context.  Is generate() the wrong way to get a greedy prediction?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hey @rfernand2 - could you please upload the fine-tuned checkpoint to the hub and provide a fully reproducible code snippet? :-) 

It would be great if we could just copy paste the code to reproduce the error <|||||>Hi @patrickvonplaten - I'll work on that (may take a day or 2 given other ongoing work).  Thanks.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>I happened to encounter something similar; in my case, the decoder had very few tokens (3), and the decoder learned to predict the token given by the teacher because that was the trivial solution and worked most of the time. When the model needed to make a prediction based on its own predictions it just went haywire and regressed to a single label because that's what it had learned.