load_in_8bit=True returns gibberish when inferencing on multi GPU
### System Info

```Shell
- `Accelerate` version: 0.18.0
- Platform: Linux-5.15.0-72-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Numpy version: 1.24.3
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- `Accelerate` default config:
	Not found
- using transformers from here, as recommended by openassistant: https://huggingface.co/OpenAssistant/oasst-rlhf-2-llama-30b-7k-steps-xor
    git clone https://github.com/huggingface/transformers.git
    cd transformers
    git checkout d04ec99bec8a0b432fc03ed60cea9a1a20ebaf3c

other info:
- ubuntu 22.04
- bitsandbytes = 0.38.1
- CUDA 118 detected by bitsandbytes
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100S-PCI...  On   | 00000000:00:06.0 Off |                    0 |
| N/A   31C    P0    41W / 250W |  32198MiB / 32768MiB |     37%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100S-PCI...  On   | 00000000:00:07.0 Off |                    0 |
| N/A   31C    P0    36W / 250W |  31784MiB / 32768MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100S-PCI...  On   | 00000000:00:08.0 Off |                    0 |
| N/A   33C    P0    36W / 250W |  31784MiB / 32768MiB |     23%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100S-PCI...  On   | 00000000:00:09.0 Off |                    0 |
| N/A   33C    P0    36W / 250W |  31784MiB / 32768MiB |     16%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
```




### Who can help?

Big Model Inference: @sgugger @muellerzr

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below): https://huggingface.co/OpenAssistant/oasst-rlhf-2-llama-30b-7k-steps-xor

### Reproduction

create a fresh venv and run this:
```
python3.10 -m venv dev_1
source dev_1/bin/activate
pip install --upgrade pip
git clone https://github.com/huggingface/transformers.git
cd transformers
git checkout d04ec99bec8a0b432fc03ed60cea9a1a20ebaf3c
pip install .
pip install torch==1.13.1 accelerate==0.18.0 sentencepiece==0.1.98 protobuf==3.20.1
pip install scipy
pip install bitsandbytes==0.38.1
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
python -m bitsandbytes

```
Then open python, load weights and infer with `load_in8bit=True`. The `model.generate` arguments differ, due to the inf/nan bug with `CUDA 11.8` and bitsandbytes `0.38.1` see https://github.com/tloen/alpaca-lora/issues/408
**Update: see section expected behaviour where I run the exact same `model.generate` call with the same parameters.**
```
python

from transformers import LlamaTokenizer, LlamaForCausalLM, TextStreamer

tokenizer = LlamaTokenizer.from_pretrained("/mnt/models/oasst-rlhf-2-llama-30b")
model = LlamaForCausalLM.from_pretrained("/mnt/models/oasst-rlhf-2-llama-30b", device_map="auto", load_in_8bit=True) streamer = TextStreamer(tokenizer, skip_prompt=True)
message = "<|prompter|>This is a demo of a text streamer. What's a cool fact about ducks?<|assistant|>"
inputs = tokenizer(message, return_tensors="pt").to(model.device)

tokens = model.generate(**inputs,  max_new_tokens=25, do_sample=True, num_beams=1, temperature=0.9, streamer=streamer, remove_invalid_values=True) #if I don't use remove invalid, I get the inf/nan bug, see https://github.com/tloen/alpaca-lora/issues/408

 ⁇  <|prompter|> This is a demo of a text streamer. What's a cool fact about ducks? <|assistant|> 
enaracht blood searches anomдів kun Nap wherever learned Laufcalendar ^C #manu
```

Here's what happens when I load the model without the `load_in_8bit=True` flag (good!):
```
python

from transformers import LlamaTokenizer, LlamaForCausalLM, TextStreamer

tokenizer = LlamaTokenizer.from_pretrained("/mnt/models/oasst-rlhf-2-llama-30b")
model = LlamaForCausalLM.from_pretrained("/mnt/models/oasst-rlhf-2-llama-30b", device_map="auto")
streamer = TextStreamer(tokenizer, skip_prompt=True)
message = "<|prompter|>This is a demo of a text streamer. What's a cool fact about ducks?<|assistant|>"
inputs = tokenizer(message, return_tensors="pt").to(model.device)

tokens = model.generate(**inputs,  max_new_tokens=25, do_sample=True, temperature=0.9, streamer=streamer)

Response: The Duck: Small yet Mighty

Did you know that, while ducks are relatively

```
I also tried running it without `do_sample=True`:
```
>>> tokens = model.generate(**inputs,  max_new_tokens=25, temperature=0.9, streamer=streamer)
 ⁇  <|prompter|> This is a demo of a text streamer. What's a cool fact about ducks? <|assistant|>  
⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇  ⁇ 


```


### Expected behavior

I would expect the following:
```
python

from transformers import LlamaTokenizer, LlamaForCausalLM, TextStreamer

tokenizer = LlamaTokenizer.from_pretrained("/mnt/models/oasst-rlhf-2-llama-30b")
model = LlamaForCausalLM.from_pretrained("/mnt/models/oasst-rlhf-2-llama-30b", device_map="auto", load_in_8bit=True) 
streamer = TextStreamer(tokenizer, skip_prompt=True)
message = "<|prompter|>This is a demo of a text streamer. What's a cool fact about ducks?<|assistant|>"
inputs = tokenizer(message, return_tensors="pt").to(model.device)

tokens = model.generate(**inputs,  max_new_tokens=25, do_sample=True,  num_beams=1, temperature=0.9, streamer=streamer, remove_invalid_values=True)


Response: The Duck: Small yet Mighty

Did you know that, while ducks are relatively
```

edit (additional testing):
- I also tried setting `use_cache=False` in `model.generate()`, as hinted in https://huggingface.co/OpenAssistant/falcon-40b-sft-top1-560/discussions/1, but still gibberish output.
- I also tried running this with `torch==2.0.1`, but same error behavior.
- I tried downgrading from `CUDA 11.8` to `CUDA 11.6` and `bitsandbytes` from `0.38.1` to `0.31.8`, which solves the inf/nan problem (see https://github.com/tloen/alpaca-lora/issues/408). So now I can run the exact same `model.generate()` code with the only difference between the two being `load_in_8bit=True` in the model loading step:
```
python
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import LlamaTokenizer, LlamaForCausalLM, TextStreamer
>>> 
>>> tokenizer = LlamaTokenizer.from_pretrained("/mnt/models/oasst-rlhf-2-llama-30b")
>>> model = LlamaForCausalLM.from_pretrained("/mnt/models/oasst-rlhf-2-llama-30b", device_map="auto", load_in_8bit=True) #returns gibberish
Overriding torch_dtype=None with `torch_dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in mixed int8. Either pass torch_dtype=torch.float16 or don't pass this argument at all to remove this warning.

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please use this form: ......
================================================================================
dev_1/lib/python3.10/site-packages/bitsandbytes/cuda_setup/paths.py:110: UserWarning: /usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu: did not contain libcudart.so as expected! Searching further paths...
  warn(
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA_SETUP: Detected CUDA version 116
CUDA_SETUP: Loading binary dev_1/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda116_nocublaslt.so...
Loading checkpoint shards:   0%|                               | 0/7 [00:00<?, ?it/s]
dev_1/lib/python3.10/site-packages/bitsandbytes/functional.py:227: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return ct.c_void_p(A.data.storage().data_ptr())
Loading checkpoint shards: 100%|███████████████████████| 7/7 [00:54<00:00,  7.82s/it]
>>> streamer = TextStreamer(tokenizer, skip_prompt=True)
>>> message = "<|prompter|>This is a demo of a text streamer. What's a cool fact about ducks?<|assistant|>"
>>> inputs = tokenizer(message, return_tensors="pt").to(model.device)
>>> tokens = model.generate(**inputs,  max_new_tokens=25, do_sample=True, temperature=0.9, streamer=streamer)

xeabaselogiccmtnzak accomplish MeyifullylandaMP Marshallcitaemann beskre Gil zoomaki Bon companion Vert Mindsetti

```

The issue persists, so it's independent from the inf/nan bug and 100% confirmed caused by a combination of using both `load_in_8bit=True` and multi gpu. This code returns comprehensible language when:
- it fits on a single GPU's VRAM and use `load_in_8bit=True`,
- or when you load on multi GPU, but without the argument `load_in_8bit=True`.
Not OP's issue but for others finding this issue like I did: be aware that if you're using 2x RTX 4090, there's a [driver bug](https://forums.developer.nvidia.com/t/standard-nvidia-cuda-tests-fail-with-dual-rtx-4090-linux-box/233202/51) in Linux causing corrupt results. For me switching from the 530 to the 525 drivers fixed a multi-gpu gibberish issue when using `load_in_4bit` from the latest bitsnbytes.<|||||>cc @younesbelkada <|||||>thanks for the issue, I see you are using a V100, can you try to upgrade `bitsandbytes` to the version `0.39.0`?
Also I think that the kernels for 4bit inference are much more robust in my experience (tried 4bit inference in a V100 and it seemed to work fine) - can you try them and let us know how it goes?<|||||>Thanks, I will check it out! If it runs in 4 bit, then the multi-gpu issue is immediately solved, since the model would fit on a single card. Slightly worried about the inference speed from what I read. It did 1 token/s in 16-bit, spread across 4 V100Ss. But that'd be a new issue :)
<|||||>**Sort of solved**
Thanks @younesbelkada 
by updating to 4bit requirements as mentioned in https://huggingface.co/blog/4bit-transformers-bitsandbytes
```
accelerate               0.20.0.dev0
bitsandbytes             0.39.0
peft                     0.4.0.dev0
transformers             4.30.0.dev0
```

`load_in_4bit=True` produces comprehensible text on multi-gpu!!! (Even though it now only takes 28GB of VRAM, and thus would also fit on a single V100S GPU.)
```
python

from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer
tokenizer = AutoTokenizer.from_pretrained("falcon-40b-sft-mix-1226", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("falcon-40b-sft-mix-1226", device_map="auto", offload_folder="offload", trust_remote_code=True, load_in_4bit=True) 
streamer = TextStreamer(tokenizer, skip_prompt=True)
message = "<|prompter|>This is a demo of a text streamer. What's a cool fact about ducks?<|endoftext|><|assistant|>"
inputs = tokenizer(message, return_tensors="pt").to(model.device)

tokens = model.generate(**inputs,  max_new_tokens=25, do_sample=True, temperature=0.9, streamer=streamer) 

```

yields a very cool:
```
/generation/utils.py:1140: UserWarning: The following `model_kwargs` are not used by the model: ['token_type_ids'] (note: typos in the generate arguments will also show up in this list)
  warnings.warn(
Setting `pad_token_id` to `eos_token_id`:11 for open-end generation.
Ducks have waterproof feathers which are excellent at repelling water.<|endoftext|>
```

remarks/questions:
- Yes it no longer blocks me, but the original issue remains for 8 bit. So shall I keep this bug open, or is the solution for everyone to move to `load_in_4bit`, instead of `load_in_8bit`?
- 4 bit inference is not noticeably slower (or faster) than 16 bit, great!
- using `load_in 4bit` also solves the inf/nan bug that `load_in_8bit` has.<|||||>Awesome!
Great also to hear that load_in_4bit is as fast (maybe faster) than 16bit in V100s, this is very interesting! 
We can keep this issue open for community members to chime in and comment on your observations. I think that the potential 8bit incompatibility issue on V100s might need to be reported on `bitsanbytes` library. Again thanks @Daryl149 !<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.