Trainer do model generation during evaluation loop
### Feature request

Current trainer only supports teacher-forcing generation for computing evaluation loss but not auto-regressive generation for other metrics. Seq2SeqTrainer supports this but seems that it only accepts encoder-decoder models like T5 instead of GPT-style (decoder-only) models. Would this feature be added in the future?

### Motivation

I am training a decoder-only model and want to use model.generate to evaluate it during training.

### Your contribution

I haven't investigated deeply into Trainer code.
You can write your own subclass of the Trainer, it's not supported and we don't plan on adding it.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.