Same size on memory usage when loading gpt2 from torch.float16 and 8bit Quantization
### System Info

- `transformers` version: 4.30.0.dev0
- Platform: Linux-5.15.0-1026-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:No

### Who can help?

@pacman100 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM
from transformers import BitsAndBytesConfig
import torch

model_8bit = AutoModelForCausalLM.from_pretrained(
    "gpt2", 
    load_in_8bit=True,
    device_map='auto',
)
print('8 bit: ', model_8bit.get_memory_footprint())


model_float16 = AutoModelForCausalLM.from_pretrained(
    "gpt2", 
    load_in_8bit=False, 
    device_map='auto',
    torch_dtype=torch.float16
)
print('float16: ', model_float16.get_memory_footprint())
```

### Expected behavior

Result is
 8 bit:  274045464
float16:  261462552

Should see 8bit smaller than float16
cc @younesbelkada <|||||>Hi @czhang-trinity 
Thanks for the issue, running your script on the current main branch gives me:

```bash 
8 bit:  261462552
16 bit:  261462552
```

This is because GPT2 uses Conv1D in replacement to all linear layers. Therefore the 8bit conversion ends up converting no Linear layers !
```bash
GPT2LMHeadModel(
  (transformer): GPT2Model(
    (wte): Embedding(50257, 768)
    (wpe): Embedding(1024, 768)
    (drop): Dropout(p=0.1, inplace=False)
    (h): ModuleList(
      (0-11): 12 x GPT2Block(
        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (attn): GPT2Attention(
          (c_attn): Conv1D()
          (c_proj): Conv1D()
          (attn_dropout): Dropout(p=0.1, inplace=False)
          (resid_dropout): Dropout(p=0.1, inplace=False)
        )
        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (mlp): GPT2MLP(
          (c_fc): Conv1D()
          (c_proj): Conv1D()
          (act): NewGELUActivation()
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
    )
    (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  )
  (lm_head): Linear(in_features=768, out_features=50257, bias=False)
)
```