Having to specify too many `ignore_keys` in `Trainer.prediction_step`
Since all model output dicts that have logits give it with the key `logits` I think this code could be simplified to just use the `logits` key. (Rather than having to specify a bunch of `ignore_keys`.)

from:
https://github.com/huggingface/transformers/blob/e20ac6611df97f66148ce8b7886f01ffe9d17484/src/transformers/trainer.py#L1471-L1473
to:
```python
 if isinstance(outputs, dict): 
     loss = outputs["loss"].mean().detach() 
     logits = (outputs.get('logits', None),)
```

This prevents other keys for being sent to `nested_concat` causing an error:
https://github.com/huggingface/transformers/blob/e20ac6611df97f66148ce8b7886f01ffe9d17484/src/transformers/trainer.py#L1367-L1368

I'd be happy to make this change, let me know if I'm missing something here.
No, this is too simplistic. First of all, all question-answering  models return two logits called `start_logits` and `end_logits`. Then a user might want to get the predictions for all their `all_hidden_states` or `all_attentions` when the model has the proper config keys, which is why the `Trainer` gather all the tensors different from the loss.<|||||>Ahh I see, guess there's no simple answer here. Thanks for the info!