chatglm2 load_in_8bit=true can't reduce gpu memory when using transformer==4.31.0
### System Info

- `transformers` version: 4.31.0
- Platform: Linux-3.10.0-1160.92.1.el7.x86_64-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from transformers import AutoTokenizer, AutoModelForCausalLM, LlamaForCausalLM, TextIteratorStreamer
import transformers
from peft import PeftModel
import bitsandbytes as bnb
import torch
from threading import Thread, currentThread
import time
model = "/workspace/model-files/chatglm2"

model = AutoModelForCausalLM.from_pretrained(model, device_map='auto', trust_remote_code=True, load_in_8bit=True)
cls = bnb.nn.Linear8bitLt
print(model.get_memory_footprint())
for name, module in model.named_modules():
    # print(name)
    if isinstance(module, cls):
        names = name.split('.')
        print(names)
```
Regardless of whether load_in8bit is set or not, the gpu memory usage is always 12487168064
but when use transformer==4.29.2 load_in_8bit=True  the gpu memory usage is 6776623168

### Expected behavior

transformers latest version work well
ref:https://github.com/THUDM/ChatGLM2-6B/issues/163<|||||>cc @younesbelkada <|||||>+1<|||||>Thanks, my feeling is that it is related with the issue described in https://github.com/huggingface/transformers/pull/25105 
Can you try that version of transformers meanwhile and let me know if this fixes your issue?

```bash
pip install -U git+https://github.com/ranchlai/transformers.git@fix_get_keys_to_not_convert
```<|||||>> 

it's work,but i get other problem when i use git+https://github.com/ranchlai/transformers.git@fix_get_keys_to_not_convert , please ref: https://github.com/huggingface/transformers/issues/25197<|||||>Now #250105 is on main, you can install it with:

```bash
pip install -U git+https://github.com/huggingface/transformers.git
```

I will close this issue as this issue is solved with the above PR. Feel free to re-open if you think that's not the case