OOM Issue when evaluating with Trainer
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: Ubuntu 20.04
- Python version: Python 3.8.2
- PyTorch version (GPU?): 1.5.1
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
 
 Trainer: @sgugger 

## Information

Model I am using BartForConditionalGeneration:

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

Using the normal Trainer with trainer.evaluate() on the BillSum Dataset

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

BillSum Dataset

## To reproduce

Steps to reproduce the behavior:

1. Load Billsum
2. FineTune Bart on Billsum 
3. try to evaluate with trainer.evaluate()

`
Traceback (most recent call last):
  File "main.py", line 211, in <module>
    main()

  File "main.py", line 182, in main
    trainer.evaluate()

  File "/home/mypc/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1157, in evaluate
    output = self.prediction_loop(eval_dataloader, description="Evaluation")
  File "/home/mypc/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1244, in prediction_loop
    preds = logits if preds is None else torch.cat((preds, logits), dim=0)
RuntimeError: CUDA out of memory. Tried to allocate 2.58 GiB (GPU 0; 7.79 GiB total capacity; 2.72 GiB already allocated; 1.94 GiB free; 4.88 GiB reserved in total by PyTorch)
`

## Expected behavior

The Tensors for the logits are getting really big (x, 1024, 52000) and will be concatenated every prediction step in the trainer.py in line 1244:
`             
if logits is not None:
    preds = logits if preds is None else torch.cat((preds, logits), dim=0)
` 
One possible solution is to argmax them while iterating through the prediction loop
We know there is a limitation on the size of dataset for evaluation/prediction because every tensors are concatenated together (this is necessary for distributed training). We plan to work on it using the work done in the `datasets` library to have the predictions cached in temporary files as the prediction loop go. However this is not going to be ready before a few weeks, so in the meantime, I encourage you to feed your dataset to predictions by small slices.<|||||>Okay, thank you. 

Currently i'm only debugging on my machine so the:
` if logits is not None: preds = logits.argmax(-1) if preds is None else torch.cat((preds, logits.argmax(-1)), dim=0)`
-hack does it for me. But yeah as soon as I train on multiple machines I will feed it in small slices.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> We know there is a limitation on the size of dataset for evaluation/prediction because every tensors are concatenated together (this is necessary for distributed training). We plan to work on it using the work done in the `datasets` library to have the predictions cached in temporary files as the prediction loop go. However this is not going to be ready before a few weeks, so in the meantime, I encourage you to feed your dataset to predictions by small slices.

Hi, has this problem been solved :)