GPU memory not completely free after one BERT/RoBERTa fwd pass
### System Info

Hello. I am batch processing a list of sentences to solely extract a contextualized word embedding for each one (for the same word, this is just an example). I have to iterate this for different words and sentences, but after one forward pass and all the "del" and "torch.cuda.empty_cache()", the GPU memory has still "residuals" I can't delete in any way. I even tried to delete the model. I am not sure whether the one below is the correct deleting order. Any suggestion is **highly** appreciated.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

model = AutoModelForMaskedLM.from_pretrained("roberta-base", output_hidden_states = True,).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model.eval()

context_sentences = ["Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec porttitor erat urna, sit amet vulputate purus rutrum in. Donec et laoreet velit."] * 100
toks = tokenizer(context_sentences, return_tensors="pt").input_ids.to("cuda")

with torch.no_grad():
    out = model(toks)
    
    del context_sentences
    del model
    del out
    del toks
    del tokenizer
    torch.cuda.empty_cache()
```

Result: `NVIDIA GeForce RTX 3070 Laptop GPU memory: 878 MiB/8192 MiB`

### Expected behavior

Expected: `NVIDIA GeForce RTX 3070 Laptop GPU memory: 0 MiB/8192 MiB`
Hi @halixness 
In the past I have experienced that, and adding `gc.collect()` before and after the `torch.cuda.empty_cache()` seemed to help. Can you quickly try that out? 🙏 <|||||>> `gc.collect()`
@younesbelkada 
Hi! Unfortunately that did not help :(
```
with torch.no_grad():
    out = model(toks)
    
    del context_sentences
    del model
    del out
    del toks
    del tokenizer
    gc.collect()
    torch.cuda.empty_cache()
    gc.collect()
```
Also tried before `torch.no_grad()`.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.