AutoModelFromCausalLLM of Bloom not releasing GPU memory after each inference batch
### System Info

Hi there, I have set torch.no_grad() and torch.cuda.empty_cache(), but the GPU still encounters out-of-memory (OOM) errors after a few inferences. My torch version is 1.13.1, deepspeed version is 0.9, and transformer version is 4.28.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I have set torch.no_grad() and torch.cuda.empty_cache() for AutoModelFromCausalLLM

### Expected behavior

auto release the memory usage 
Hi @chuckhope, thanks for raising this issue. 

So that we can best help, could you provide the following information: 
* a minimal code snippet to reproduce the error
* Information about the hardware - are you running on a single GPU? 
* Information about the model - are you using a single checkpoint? If so, which one? If not, is this observed for all checkpoints? <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.