a memory leak in qqp prediction using bart
### System Info

```shell
- `transformers` version: 4.19.0.dev0
- Platform: Linux-5.11.0-43-generic-x86_64-with-glibc2.17
- Python version: 3.8.10
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I met the same issue #11011. If not using `--eval_accumulation_steps`, it caused CUDA out of memory. If using it, it caused out of RAM and killed by system.

I only did prediction on GLUE QQP dataset using bart without fine-tuning. Considering QQP having a large test set (300k), the prediction got slower and slower, and finally got out of memory.

This is the script to reproduce:
```
CUDA_VISIBLE_DEVICES=0 python run_glue.py   --model_name_or_path facebook/bart-large   --task_name qqp   --output_dir bart-large_qqp --eval_accumulation_steps 100 --do_predict --per_device_eval_batch_size 24
```

### Expected behavior

```shell
Prediction without out memory.
```

There is nothing we can do to help with that, as you don't seem to have the RAM necessary to hold all predictions. The only advise I have is that you should predict on parts of the dataset and not the whole. 

This is not a bug in Transformers, removing the label.<|||||>Sorry, I don't think that. I have 512GB RAM. And I can conduct training and evaluation well, but this issue only occurs during prediction.<|||||>The training does not accumulate predictions and the evaluation uses the evaluation set which is smaller.<|||||>But I don't understand why a tensor of shape (300k,) will exceed RAM? Does trainer save intermediate hidden states during prediction?<|||||>Mmmmmm, it may be that the model is outputting more tensors than just the logits. I see it has `use_cache=True` in its config, can you try again by setting it to `False`?<|||||>OK! I try it now! Thanks!<|||||>Sorry, it doesn't work.<|||||>It takes more time and more memory with every 100 steps.<|||||>I have found how to solve it!<|||||>The `logits` in the [https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2635](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2635) returns two tensors of shape (batch_size, num_classes) and (batch_size, max_sequence_length, hidden_size).
And the second shape will be accumulated, and finally will become a tensor of (300k, 256, 1024), and it will be out of memory.

And if i don't accumulate it, the code can work well.<|||||>Yes, that is what I was saying earlier: the model does not return the predictions only but some hidden states. Not sure which option will deactivate the second one.<|||||>Yes, thanks for your help! And it would be nice if this could be improved.