Memory leak after repeated inference
### System Info

- `transformers` version: 4.30.2
- Platform: Linux-3.10.0-1160.90.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I am using the following script, the memory rises slowly and memory leak happend. I change the transformers version and the torch version, but it doesn't work. When using TFBertModel instead of BertModel, or moving model to CPU, the memory leak still exists.

##########
from transformers import BertTokenizer, BertModel
import torch
import psutil

device = "cuda:0"

model_path = "bert-base-uncased"
model = BertModel.from_pretrained(model_path)
tokenizer = BertTokenizer.from_pretrained(model_path)
model.to(device)
model.eval()

query = "Replace me by any text you'd like."

for i in range(10000):
    with torch.no_grad():
        encoded_input = tokenizer(query, return_tensors='pt').to(device)
        a = psutil.Process(os.getpid()).memory_info().rss / 1024  # memory before
        output = model(**encoded_input)
        b = psutil.Process(os.getpid()).memory_info().rss / 1024  # memory after
        if b != a and i > 0:
            print(i)
            print("b-a=%s kb" % (b - a))

##########
result:
62
b-a=80.0 kb
6195
b-a=8.0 kb

### Expected behavior

Memory is almost stable.
Hi @huu3301 

Please format the code snippet properly (with proper indent too). See for example

<img width="332" alt="Screenshot 2023-06-27 111112" src="https://github.com/huggingface/transformers/assets/2521628/ec2852fb-695a-456b-b09f-8f99ef0bdd30">

Regarding your question about leak:

Do you only get iteration `62` and `6195` with `b != a`? In this case, there is no memory leak: it's just python process gets to use a bit more memory to perform something under the hood.


<|||||>@ydshieh 
The indent disappeared when I submitted the issue. 
I increased the number of iteration to 1000000 , and the memory increment was less than 50kb. There was no memory leak. Thank you for answering.