Abnormally slow inference speed of quantized model?
### System Info



To reproduce, I am running on CUDA 12.1/Driver 530 on an A100 with Ubuntu 20.04. I am running with the following packages
accelerate                    0.21.0.dev0
triton                        2.0.0
transformers                  4.31.0.dev0
torch                         2.0.1
bitsandbytes                  0.40.0.post3

Output from the transformers-cli env is 
- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.15.0-1015-aws-x86_64-with-glibc2.17
- Python version: 3.8.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi guys,
I am trying out the load_in_4bit/load_in_8bit options to see if it speed up model inference. My understanding is that by quantizing the model, the inference speed will improve. However that's not the case. I use the following simple script to test out speed on a T5 XXL model for 4bits/8bits/fp32, and actually fp32 model runs the fastest (0.07sec), and the 4bits/8bits run almost in the same speed (0.1sec). So I want to double check. 

The script I am using to test the speed is here

```
from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch
import pdb
import gc
import time
import os
device_id = 3
tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
input_text = "translate English to German: How old are you?" * 20
input_ids = tokenizer(input_text, return_tensors="pt").input_ids[:, :128].to(f"cuda:{device_id}")

# Actual model loading
max_memory = f'{int(torch.cuda.mem_get_info()[0]/1024**3)-2}GB'
model_4bits = T5ForConditionalGeneration.from_pretrained(
    "google/flan-t5-xxl", 
    load_in_4bit=True,
    max_memory=max_memory)

# model_8bits = T5ForConditionalGeneration.from_pretrained(
#     "google/flan-t5-xxl", 
#     load_in_8bit=True,
#     max_memory=max_memory)

# model_fp32 = T5ForConditionalGeneration.from_pretrained(
#     "google/flan-t5-xxl").to(f"cuda:{device_id}")

def benchmark(model_name, model, input_ids):
    # warmup
    for _ in range(200):
        model.encoder(input_ids)

    torch.cuda.synchronize()
    with torch.no_grad():
        start = time.time()
        for i in range(200):
            model.encoder(input_ids)
            
        torch.cuda.synchronize()
        end = time.time()

        print(f"{model_name} inference time is {(end-start)/200} sec")

benchmark("model_4bits", model_4bits, input_ids) # model_4bits inference time is 0.1036052393913269 sec
# benchmark("model_8bits", model_8bits, input_ids) # model_8bits inference time is 0.1006016504764557 sec
# benchmark("model_fp32", model_fp32, input_ids) # model_fp32 inference time is 0.0731453263759613 sec
```

### Expected behavior

I expect the 4 bits model runs faster than the 8bits model, which in turn runs faster than the fp32 model. That's not the case I observe. 
Having the same issue here. Also want to ask if it is true that 8bit model is slower than fp16 during inference<|||||>cc @younesbelkada for `4-bit` 🙏 <|||||>Hi everyone,
Sadly 8bit models are expected to be slower than fp16 models, see this: https://huggingface.co/blog/hf-bitsandbytes-integration#faster-inference-speed-for-smaller-models for reference
bitsandbytes juste released a new version for faster inference (batch_size=1) https://github.com/TimDettmers/bitsandbytes/releases/tag/0.40.0
Can you try to upgrade bitsandbytes and run the benchmark again?<|||||>> Hi everyone, Sadly 8bit models are expected to be slower than fp16 models, see this: https://huggingface.co/blog/hf-bitsandbytes-integration#faster-inference-speed-for-smaller-models for reference bitsandbytes juste released a new version for faster inference (batch_size=1) https://github.com/TimDettmers/bitsandbytes/releases/tag/0.40.0 Can you try to upgrade bitsandbytes and run the benchmark again?

Hi younesbelkada, I get my bitsandbytes from directly pulling the github repository and compile from source yesterday. So it is already the latest 0.40 version that includes the 4 bits bs1 inference. <|||||>This is strange .. Can you report that to bitsandbytes library? 🙏 <|||||>> This is strange .. Can you report that to bitsandbytes library? 🙏

Sure. I will post on the issues of bitsandbytes library and link this issue. 