BertTokenizer not release gpu memory after del 
### System Info

transformers                       4.20.1
tensorflow                         2.9.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import BertTokenizer, TFBertModel
import torch
max_seq_len = 2028
tokenizer = BertTokenizer.from_pretrained('klue/bert-base', truncation=True, max_seq_len=max_seq_len)
del tokenizer
torch.cuda.empty_cache()
```



### Expected behavior

In ubuntu20.04 with RTX3090, after del tokenizer, gpu memory is not released. 
how can i release gpu memory?
This code doesn't use any GPU memory as tokenizers don't even import `torch`.<|||||>@sgugger In my case, after run this code, gpu memory is fully occupied.  Is your pytorch the gpu version correct?<|||||>This is because you import `TFBertModel` (I didn't catch it in your first code sample). This imports `tensorflow` which then takes all the GPU memory.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.