T5 should not use teacher-forcing when under evaluation
### System Info

- `transformers` version: 4.23.1
- Platform: Linux-6.0.2-76060002-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@patrickvonplaten 



### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce behaviour:

1. Find a task where the same token in the target sequence is repeated multiple times, e.g. IOB Tagging.
1. Train a t5 model on a task
3. Evaluate by predicting
4. Evaluate using generate and giving the same input_ids

### Expected behavior

The prediction task during evaluation should produce identical results to generate using num beams=1 ie greedy decoding.

Due to the nature of the task above, the target sequence has the form ((a|b|c){n})+, meaning sequences of target id repetitions, e.g. `000011110000`, due to teacher forcing, the model can learn to repeat the input token and quickly minimize the loss except for the boundary cases of `01` and `10`.

This is not visible when evaluating the model (e.g. via `.eval()`), and due to teacher forcing, the model gives excellent results. However, when using `.generate` with the same inputs, the model behaves horribly.

This is similar to 
#12488
Please use the [forums](https://discuss.huggingface.co/) for discussions like this one as we keep issues for bugs and feature requests in the library.<|||||>Then perhaps the ability to turn off teacher-forcing could be listed as a feature for the model?

I don’t see how using teacher forcing in an auto regressive model during evaluation is not a bug.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.