eval OOM when loading a pretrained model with output_hidden_states set to True for BertForSequenceClassification
### System Info

- `transformers` version: 4.23.1
- Platform: Linux-4.15.0-166-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.10.0+cu102 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run the following snippet

```
model = BertForSequenceClassification.from_pretrained('snunlp/KR-BERT-char16424', output_hidden_states=True)

# Now create any training_args, tokenized_train, tokenized_valid, and compute_metrics function (such as those in the offical tutorial).

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_valid,
    compute_metrics=compute_metrics,
)
trainer.train()
```

Eval will experience CUDA OOM on a gpu with 24Gb, after about 100 examples.



### Expected behavior

I tried warmstarting a BERT classification model from a pretrained embedding model, which sets output_hidden_states to True in config.json. But eval runs into OOM issue. 

You will need to use the [eval_accumulation_steps](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.eval_accumulation_steps) argument in your `TrainingArguments` as it's not possible to accumulate all those tensors coming from the hidden states on the GPU.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.