Bloom-6b3 not utilizing much from GPU
### System Info

GPU: Nvidia V100

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Using the following code, when I do some inferences, GPU utilization is below 10%.
```
tokenizer = AutoTokenizer.from_pretrained("models/bloom_1b3")
model = AutoModelForCausalLM.from_pretrained(
        "models/bloom_1b3",
        device_map="auto",
        torch_dtype=torch.float16
)
pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, device=torch.device(0))
```

### Expected behavior

GPU utilization should be larger than 50%.
Hi @farzanehnakhaee70 !
Thanks a lot for your message!
Could you give us the output of `nvidia-smi` when running your script? Also could you share with us the version of `accelerate` you are using?<|||||>Hi @younesbelkada 
Thanks for your support.
nvidia-smi:
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.29.05    Driver Version: 495.29.05    CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:3E:00.0 Off |                    0 |
| N/A   38C    P0    54W / 300W |  10358MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
```
Also accelerate version is `0.10.0`.
It should also be mentioned that the same behavior existed if I use deep-speed or even if I didn't use any of accelerate and deep-speed.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.