Multi-threaded parallel inferrence problem
### System Info

accelerate == 0.19.0
bitsandbytes == 0.37.1
- `transformers` version: 4.29.2
- Platform: Linux-3.10.0-1160.92.1.el7.x86_64-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I use chatglm2/bloomz-7b for multithreaded parallel inference:

```
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from transformers import AutoTokenizer, AutoModelForCausalLM, LlamaForCausalLM, TextIteratorStreamer
import transformers
import torch
from threading import Thread, currentThread
import time
model = "/workspace/model-files/chatglm2"

tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model, device_map='auto', trust_remote_code=True, load_in_8bit=True)

def infer(prompt):
    inputs = tokenizer(prompt, return_tensors='pt')
    inputs = inputs.to(model.device)
    print('before generate')

    # out = model.generate(**inputs)
    # print('after generate')
    # out_text = tokenizer.decode(out[0])
    # print('out_text is:', out_text)
    t = currentThread()
    streamer = TextIteratorStreamer(tokenizer)
    generation_kwargs = dict(inputs, streamer=streamer, max_length=2048)
    thread = Thread(target=model.generate, kwargs=generation_kwargs)
    thread.start()
    print("------******-------")
    for new_text in streamer:
        print("thread id:", t.ident ,"new text:",new_text)
    print("------******-------")

if __name__ == '__main__':
    prompt1 = '写一篇关于黄鹤楼的800字作文'
    prompt2 = 'Describe each state in the United States in detail'
    t1 = Thread(target=infer, args=(prompt1,))
    t2 = Thread(target=infer, args=(prompt2,))
    t1.start()
    time.sleep(5)
    t2.start()
    t1.join()
    t2.join()
```

I get below error info:
```
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/generation/utils.py", line 1515, in generate
    return self.greedy_search(
  File "/usr/local/lib/python3.8/dist-packages/transformers/generation/utils.py", line 2332, in greedy_search
    outputs = self(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 932, in forward
    transformer_outputs = self.transformer(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 828, in forward
    hidden_states, presents, all_hidden_states, all_self_attentions = self.encoder(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 638, in forward
    layer_ret = layer(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 563, in forward
    mlp_output = self.mlp(layernorm_output)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 499, in forward
    output = self.dense_4h_to_h(intermediate_parallel)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/nn/modules.py", line 242, in forward
    out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/autograd/_functions.py", line 488, in matmul
    return MatMul8bitLt.apply(A, B, out, bias, state)
  File "/usr/local/lib/python3.8/dist-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/autograd/_functions.py", line 397, in forward
    output += torch.matmul(subA, state.subB)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x4 and 2x4096)
```

### Expected behavior

able to inference normally
Hi @zhaotyer, thanks for opening this issue! 

Could you edit the issue details to make sure that the code and traceback are properly formatted? This will make it easier for us to read and understand what's going on. 

The code examples should be wrapped around three backticks. You can also specify the language to get color coded formatting :) i.e. ` ```python CODE GOES HERE ``` `

Same for the traceback - it should go between a pair of three tickbacks ` ``` error message ``` `

cc @gante as this seems to be related to generate :) <|||||>@zhaotyer that seems to be a bitsandbytes issue -- would you be able to update this library to its latest version and confirm whether the issue persists? :)<|||||>> Collaborator

Thanks for the reminder, I have edited it
<|||||>> @zhaotyer that seems to be a bitsandbytes issue -- would you be able to update this library to its latest version and confirm whether the issue persists? :)
Should have nothing to do with bitsandbytes
The following are the test results of different transformers/accelerate versions
```
1.transformers==4.31.0 accelerate==0.21.0 bitsandbytes==0.37.1
 	1.1 chatglm2 load_in_8bit=true  singlethread normal, multithread normal
 	1.2 chatglm2 load_in_8bit=false singlethread normal, multithread normal
2.transformers==4.29.2 accelerate==0.19.0 bitsandbytes==0.37.1
	1.1 chatglm2 load_in_8bit=true singlethread normal, multithread have error(RuntimeError: mat1 and mat2 shapes cannot be multiplied)
 	1.2 chatglm2 load_in_8bit=false singlethread normal, multithread normal
```
@gante <|||||>@zhaotyer OK, thanks for running with updated versions and reporting. If I've understood correctly, it looks like everything is running as expected on the most recent versions of accelerate and transformers and so the issue has been resolved in the most recent releases. <|||||>> @zhaotyer OK, thanks for running with updated versions and reporting. If I've understood correctly, it looks like everything is running as expected on the most recent versions of accelerate and transformers and so the issue has been resolved in the most recent releases.

Can you explain what changes have been made between the two releases to solve this problem? thks<|||||>@zhaotyer The best way to find this is running `git bisect` to identify the commit and PR which resolved this for your example script. 