Keep getting an OOM when doing a evaluation
## Environment info

  - `transformers` version: 4.8.2
  - Platform: Linux-5.8.0-55-generic-x86_64-with-debian-bullseye-sid
  - Python version: 3.7.9
  - PyTorch version (GPU?): 1.9.0+cu111 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Flax version (CPU?/GPU?/TPU?): not installed (NA)
  - Jax version: not installed
  - JaxLib version: not installed
  - Using GPU in script?: Yes cuda:0 only one GPU
  - Using distributed or parallel set-up in script?: No

### Who can help
No idea.

Models:
- albert, bert

Library:
- trainer: @sgugger

## Information
The problem arises when using evaluating a model, I train using a batch size of 64 and 80% of my data but when I do an evaluation (600 000 data points) it ALWAYS give a OOM at 94% even with a batch size of one and the memory allocation IS ALWAYS 3.75 GB no matter the size of the batch. I've deleted the .cache many times and still got the problem.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: which is a sequence classification of postal address (max 30 tokens).

## To reproduce
My script is base on the [token classification with trainer](https://github.com/huggingface/transformers/blob/master/examples/pytorch/token-classification/run_ner.py).

Found [this](https://discuss.huggingface.co/t/cuda-out-of-memory-during-evaluation-but-training-is-fine/1783/2) explication that fixed my problem.

i.e. Huggingface keeps all the prediction in the GPU.