Unexpected GPU requests during training
### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.19.0-46-generic-x86_64-with-glibc2.31
- Python version: 3.11.4
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


**When CUDA_VISIBLE_DEVICES=1 is specified, the program suddenly applies for calling GPU:0 memory after executing thousands of steps.**

I executed the command below on a two-card 3090 server, and when I ran the program, GPU:0 card was already occupied.
`CUDA_VISIBLE_DEVICES=1 nohup python train.py &`
In the beginning, everything seemed to be normal, because I have done more than 10,000 steps
![image](https://github.com/huggingface/transformers/assets/88258534/90fa40f3-d34d-4f14-8a33-94614cbce155)
However, very suddenly, the program starts to apply for the memory of GPU0 during eval stage（although it has go through a lot of eval stage before）
![image](https://github.com/huggingface/transformers/assets/88258534/79d16b30-92d7-4aac-bc22-54e9c450f65b)
The error entry of the program is displayed as model.generate from the error report
![image](https://github.com/huggingface/transformers/assets/88258534/901567df-5672-493b-9486-0b3077049e60)


this is a related document retirval code run on t5-large,not sure the rest of code will grant help or not.
![image](https://github.com/huggingface/transformers/assets/88258534/2133aae7-d92a-474d-83ec-265845f503ca)
![image](https://github.com/huggingface/transformers/assets/88258534/4dbec245-d923-449e-b5e4-7ade02f54f11)
![image](https://github.com/huggingface/transformers/assets/88258534/7cf39de4-dd3d-4699-9e7b-0b57cea7c17a)


### Who can help?

@gan @ArthurZucker @younesbelkada  Since it's a nlp project and happened in generation stage, so maybe I need help from you sincerely.

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

It's a public project on github [DSI-transformers](https://github.com/ArvinZhuang/DSI-transformers), just run``CUDA_VISIBLE_DEVICES=1 nohup python train.py &`` to reproduce this problem.

### Expected behavior

The program can correctly realize that it can only use GPU specified in command.
solved