CLIP not releasing GPU memory after each inference batch
### System Info

- `transformers` version: 4.25.1
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.31
- Python version: 3.10.8
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@patil-suraj

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import CLIPTokenizerFast, CLIPProcessor, CLIPModel
import torch

model_id = 'openai/clip-vit-base-patch32'
device = 'cuda'

tokenizer = CLIPTokenizerFast.from_pretrained(model_id)
processor = CLIPProcessor.from_pretrained(model_id)
model = CLIPModel.from_pretrained(model_id).to(device)

images = glob.glob('/data/index/abo/images/small/*/*.jpg')
dataset = Dataset.from_dict({'image': images}).cast_column('image', Image())

for i in range(0, len(dataset), 500):
  print(i)
  batch = processor(
    text=None,
    images=dataset[i:i+500]['image'],
    return_tensors='pt'
  )['pixel_values'].to(device)

  model.get_image_features(batch)
```

### Expected behavior

Each time I call `model.get_image_features(batch)` about 20GB of GPU memory is consumed. However, the GPU memory is never cleared, so I quickly run into a `CUDA out of memory` error. This memory also isn't cleared if I manually call `torch.cuda.empty_cache()`.

It's possible I'm missing a step, but to me it looks like there may be a bug in the model code causing it not to free GPU memory after it finishes inference on a batch?
Maybe it's missing a `torch.no_grad`? Not sure though, cc @amyeroberts and @ArthurZucker if you have time to dive into this a bit more :-)<|||||>@sgugger thanks for the tip; I think that's the source of the issue! When I wrap my code in a `with torch.no_grad():` context it starts releasing GPU memory correctly. Not going to close the issue just yet since I'm not sure whether this *should* be the caller's responsibility or not when calling `get_image_features`, but at any rate the problem is solved for me. 🙂<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.