Memory leak when using GIT for image captioning in reference
### System Info

transformers version: 4.28.1
Platform: Linux-5.8.0+-x86_64-with-Ubuntu-20.04
Python version: 3.7.11
PyTorch version (GPU?): 1.10.0+cu113 (True)
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I am using the following script some times, the memory rises and memory leak happend.

processor = AutoProcessor.from_pretrained("microsoft/git-large-textcaps"))
model = AutoModelForCausalLM.from_pretrained("microsoft/git-large-textcaps")).to(device)
def test(image):
  with torch.no_grad():
      pixel_values_org = processor(images=image_cv2, return_tensors="pt").pixel_values
      pixel_values1 = pixel_values_org.to(device).detach()
      generated_ids = model.generate(pixel_values=pixel_values1, max_length=20)
      generated_ids1 = generated_ids.to(device).detach()
      generated_caption = processor.batch_decode(generated_ids1, skip_special_tokens=True)[0]
      return generated_caption

### Expected behavior

The memory is nearly stable
Hi @XingyuZhu-Pamela, thanks for raising this issue!

A few questions from my side so that I can best try and help: 
* In the above snippet, what values does `device` have (I'm assuming `"cuda"`)? 
* Are there any patterns to when the memory leak occurs e.g. certain images, running inside certain scripts, running after a long time? How often do you observe this - is it every time or more sporadic? 
* Would you be able to share an example image being used so that I can run the script? 
* Is there any chance that you're using `decord` as part of the pipeline? There was a known issue where [decord would cause crashes when moving the models to GPU](https://github.com/huggingface/transformers/issues/21085).

As a side note - soon python 3.7 will no longer be officially support by the transformer library. I'd suggest upgrading the python version to make sure your code remains compatible with the library.  <|||||>> 

Thank you! Here are my answers for your questions:

- The value of device is cuda, but when I try to use cpu, the memory leak problem still exists.
- This problem happened no matter what image I use, and the memory increase is small for each call and continues to increase after multiple calls.
- Here is a example  url of image : http://p6.music.126.net/obj/w5zDmcODw6PDjj7DiMOi/4497423202/f55d/1ece/b350/7d32fea2646ce85fba6756f84761a702.jpg
I'm sorry that there is a slight problem with the code provided above, which has been fixed as follows:
**import Image
from transformers import AutoProcessor, AutoModelForCausalLM
device = "cuda"
processor = AutoProcessor.from_pretrained("microsoft/git-large-textcaps"))
model = AutoModelForCausalLM.from_pretrained("microsoft/git-large-textcaps")).to(device)
def test(image):
  image_cv2 = Image.open(requests.get(pic_url, stream=True).raw).convert('RGB')
  with torch.no_grad():
    pixel_values_org = processor(images=image_cv2, return_tensors="pt").pixel_values
    pixel_values1 = pixel_values_org.to(device).detach()
    generated_ids = model.generate(pixel_values=pixel_values1, max_length=20)
    generated_ids1 = generated_ids.to(device).detach()
    generated_caption = processor.batch_decode(generated_ids1, skip_special_tokens=True)[0]
 return generated_caption**

- I don't use decord in my code
<|||||>Along with @amyeroberts 's comments, I would suggest:
- First, please format the code snippet in a better way
  - you can enclose the code inside like done in the following screenshop
    
    <img width="500" alt="Screenshot 2023-06-27 111112" src="https://github.com/huggingface/transformers/assets/2521628/baadbd25-f2b0-4d80-9d7a-68fa413de429">


  - with proper indent (currently it's difficult to read the code snippet)
- provide a for loop that run the model and print some cpu/gpu memory usage 

That would help us to provide help 🙏 <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.