Unable to quantize Meta's new AudioCraft MusicGen model
### System Info

- Windows 11 64bit
- Python 3.10.12
- Torch v2.0.1+cu117
- Transformers v4.31.0
- audiocraft v0.0.2
- bitsandbytes v0.41.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi, I'm attempting to quantize Meta's new MusicGen model with bitsandbytes (through the Transformers library) and I've run into a bug with the `deepcopy` function. I'm not familiar with PyTorch's deepcopy function or why this error may be occurring, but I am able to side-step it with a hack and get a bit further until I reach another error, this time with the Transformers library.

The first error:
```python
>>> from transformers import AutoProcessor, MusicgenForConditionalGeneration
bin C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
>>> processor = AutoProcessor.from_pretrained("facebook/musicgen-small", load_in_8bit=True)
>>> model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small", load_in_8bit=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\transformers\models\musicgen\modeling_musicgen.py", line 1599, in from_pretrained
    return super().from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\transformers\modeling_utils.py", line 2719, in from_pretrained
    modules_to_not_convert = get_keys_to_not_convert(model)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\transformers\utils\bitsandbytes.py", line 257, in get_keys_to_not_convert
    tied_model = deepcopy(model)  # this has 0 cost since it is done inside `init_empty_weights` context manager`
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 297, in _reconstruct
    value = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 297, in _reconstruct
    value = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 297, in _reconstruct
    value = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 297, in _reconstruct
    value = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 297, in _reconstruct
    value = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\copy.py", line 153, in deepcopy
    y = copier(memo)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\torch\_tensor.py", line 86, in __deepcopy__
    raise RuntimeError(
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
```

The hack:
```python
torch.save(model, "temp.pt")
tied_model = torch.load("temp.pt")
```

The second error after using the hack:
```python
>>> from transformers import AutoProcessor, MusicgenForConditionalGeneration
bin C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
>>> processor = AutoProcessor.from_pretrained("facebook/musicgen-small", load_in_8bit=True)
>>> model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small", load_in_8bit=True)
>>> inputs = processor(text=["80s pop track with bassy drums and synth"], padding=True, return_tensors="pt")
>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\transformers\models\musicgen\modeling_musicgen.py", line 2430, in generate
    outputs = self.sample(
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\transformers\generation\utils.py", line 2642, in sample
    outputs = self(
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\transformers\models\musicgen\modeling_musicgen.py", line 1916, in forward
    decoder_outputs = self.decoder(
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\transformers\models\musicgen\modeling_musicgen.py", line 1029, in forward
    outputs = self.model(
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\transformers\models\musicgen\modeling_musicgen.py", line 938, in forward
    decoder_outputs = self.decoder(
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\transformers\models\musicgen\modeling_musicgen.py", line 848, in forward
    layer_outputs = decoder_layer(
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\transformers\models\musicgen\modeling_musicgen.py", line 394, in forward
    hidden_states = self.self_attn_layer_norm(hidden_states)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\torch\nn\modules\normalization.py", line 190, in forward
    return F.layer_norm(
  File "C:\Users\fkdlam\anaconda3\envs\audiocraft\lib\site-packages\torch\nn\functional.py", line 2515, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: expected scalar type Float but found Half
```

This is the same code provided in [an example](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen#textconditional-generation) for generating music in the Transformers documentation, except I've added the `load_in_8bit` flag. I'm not sure how to fix this one though. I've created [an issue](https://github.com/TimDettmers/bitsandbytes/issues/669) in the bitsandbytes repository too.

### Expected behavior

Being able to run the MusicGen quantized model with bitsandbytes and obtain audio data output.
I figured out a fix by adding the line
```python
inputs_embeds = inputs_embeds.to(torch.float16)
```
right after line 776, but I noticed commit https://github.com/huggingface/transformers/commit/03f98f96836477f6f5b86957d3ce98778cad5d94 which also fixes this bug. So the second bug is fixed if you're using a version of transformers since that commit a week ago.

Now we are down to two problems: the original `deepcopy` bug and the fact that for some reason the quantized MusicGen model runs over 2x as slow as the non-quantized one. Not sure why that is because quantized models should be faster. I can't do anything about it so I'm at a dead end here.<|||||>Also, non-quantized, normal musicgen-large is about 2x slower on Transformers than Meta's own code. Interestingly musicgen-small is a bit faster than Meta's own code. About 10% faster.<|||||>cc @younesbelkada @sanchit-gandhi <|||||>For benchmarking `transformers` vs `audiocraft` - could you ensure that the `transformers` model is put in half (fp16) precision? By default, we always load in fp32 precision on CPU, whereas `audiocraft` always loads the model in fp16 precision on the GPU. Running the `transformers` model in fp16 half precision should give a considerable speed-up vs fp32 full precision:

```python
model = MusicGenForConditionalGeneration.from_pretrained("facebook/musicgen-large", torch_dtype=torch.float16)
```

We can make this faster still by adding Flash Attention with a Better Transformers integration! This should give a further 10-15% speed-up<|||||>Regarding the quantisation, I was **not** able to load the model using bitsandbytes==0.40.0 using the following code snippet:
```python
from transformers import MusicgenForConditionalGeneration

model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small", load_in_8bit=True)
```

<details>

<summary> Traceback </summary>

```python
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[6], line 1
----> 1 model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small", load_in_8bit=True)

File ~/transformers/src/transformers/models/musicgen/modeling_musicgen.py:1595, in MusicgenForConditionalGeneration.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1589     logger.warning(
   1590         "Fast initialization is currently not supported for MusicgenForConditionalGeneration. "
   1591         "Falling back to slow initialization..."
   1592     )
   1593 kwargs["_fast_init"] = False
-> 1595 return super().from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)

File ~/transformers/src/transformers/modeling_utils.py:2744, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)
   2742 # We keep some modules such as the lm_head in their original dtype for numerical stability reasons
   2743 if llm_int8_skip_modules is None:
-> 2744     modules_to_not_convert = get_keys_to_not_convert(model)
   2745 else:
   2746     modules_to_not_convert = llm_int8_skip_modules

File ~/transformers/src/transformers/utils/bitsandbytes.py:257, in get_keys_to_not_convert(model)
    245 r"""
    246 An utility function to get the key of the module to keep in full precision if any For example for CausalLM modules
    247 we may want to keep the lm_head in full precision for numerical stability reasons. For other architectures, we want
   (...)
    253     Input model
    254 """
    255 # Create a copy of the model and tie the weights, then
    256 # check if it contains tied weights
--> 257 tied_model = deepcopy(model)  # this has 0 cost since it is done inside `init_empty_weights` context manager`
    258 tied_model.tie_weights()
    260 tied_params = find_tied_parameters(tied_model)

File /usr/lib/python3.10/copy.py:172, in deepcopy(x, memo, _nil)
    170                 y = x
    171             else:
--> 172                 y = _reconstruct(x, memo, *rv)
    174 # If is its own copy, don't memoize.
    175 if y is not x:

File /usr/lib/python3.10/copy.py:271, in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    269 if state is not None:
    270     if deep:
--> 271         state = deepcopy(state, memo)
    272     if hasattr(y, '__setstate__'):
    273         y.__setstate__(state)

File /usr/lib/python3.10/copy.py:146, in deepcopy(x, memo, _nil)
    144 copier = _deepcopy_dispatch.get(cls)
    145 if copier is not None:
--> 146     y = copier(x, memo)
    147 else:
    148     if issubclass(cls, type):

File /usr/lib/python3.10/copy.py:231, in _deepcopy_dict(x, memo, deepcopy)
    229 memo[id(x)] = y
    230 for key, value in x.items():
--> 231     y[deepcopy(key, memo)] = deepcopy(value, memo)
    232 return y

File /usr/lib/python3.10/copy.py:172, in deepcopy(x, memo, _nil)
    170                 y = x
    171             else:
--> 172                 y = _reconstruct(x, memo, *rv)
    174 # If is its own copy, don't memoize.
    175 if y is not x:

File /usr/lib/python3.10/copy.py:297, in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    295     for key, value in dictiter:
    296         key = deepcopy(key, memo)
--> 297         value = deepcopy(value, memo)
    298         y[key] = value
    299 else:

    [... skipping similar frames: deepcopy at line 172 (1 times)]

File /usr/lib/python3.10/copy.py:271, in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    269 if state is not None:
    270     if deep:
--> 271         state = deepcopy(state, memo)
    272     if hasattr(y, '__setstate__'):
    273         y.__setstate__(state)

File /usr/lib/python3.10/copy.py:146, in deepcopy(x, memo, _nil)
    144 copier = _deepcopy_dispatch.get(cls)
    145 if copier is not None:
--> 146     y = copier(x, memo)
    147 else:
    148     if issubclass(cls, type):

File /usr/lib/python3.10/copy.py:231, in _deepcopy_dict(x, memo, deepcopy)
    229 memo[id(x)] = y
    230 for key, value in x.items():
--> 231     y[deepcopy(key, memo)] = deepcopy(value, memo)
    232 return y

    [... skipping similar frames: deepcopy at line 172 (1 times)]

File /usr/lib/python3.10/copy.py:297, in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    295     for key, value in dictiter:
    296         key = deepcopy(key, memo)
--> 297         value = deepcopy(value, memo)
    298         y[key] = value
    299 else:

    [... skipping similar frames: deepcopy at line 172 (6 times), _deepcopy_dict at line 231 (3 times), _reconstruct at line 271 (3 times), deepcopy at line 146 (3 times), _reconstruct at line 297 (2 times)]

File /usr/lib/python3.10/copy.py:297, in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    295     for key, value in dictiter:
    296         key = deepcopy(key, memo)
--> 297         value = deepcopy(value, memo)
    298         y[key] = value
    299 else:

File /usr/lib/python3.10/copy.py:172, in deepcopy(x, memo, _nil)
    170                 y = x
    171             else:
--> 172                 y = _reconstruct(x, memo, *rv)
    174 # If is its own copy, don't memoize.
    175 if y is not x:

File /usr/lib/python3.10/copy.py:271, in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    269 if state is not None:
    270     if deep:
--> 271         state = deepcopy(state, memo)
    272     if hasattr(y, '__setstate__'):
    273         y.__setstate__(state)

File /usr/lib/python3.10/copy.py:146, in deepcopy(x, memo, _nil)
    144 copier = _deepcopy_dispatch.get(cls)
    145 if copier is not None:
--> 146     y = copier(x, memo)
    147 else:
    148     if issubclass(cls, type):

File /usr/lib/python3.10/copy.py:231, in _deepcopy_dict(x, memo, deepcopy)
    229 memo[id(x)] = y
    230 for key, value in x.items():
--> 231     y[deepcopy(key, memo)] = deepcopy(value, memo)
    232 return y

File /usr/lib/python3.10/copy.py:153, in deepcopy(x, memo, _nil)
    151 copier = getattr(x, "__deepcopy__", None)
    152 if copier is not None:
--> 153     y = copier(memo)
    154 else:
    155     reductor = dispatch_table.get(cls)

File ~/hf/lib/python3.10/site-packages/torch/_tensor.py:86, in Tensor.__deepcopy__(self, memo)
     84     return handle_torch_function(Tensor.__deepcopy__, (self,), self, memo)
     85 if not self.is_leaf:
---> 86     raise RuntimeError(
     87         "Only Tensors created explicitly by the user "
     88         "(graph leaves) support the deepcopy protocol at the moment"
     89     )
     90 if id(self) in memo:
     91     return memo[id(self)]

RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
```

</details>

However, I was with:
```python
from transformers import MusicgenForConditionalGeneration
import torch

with torch.no_grad():
    model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small", load_in_8bit=True)
```

I can take a deeper look into why the bnb conversion is failing unless @younesbelkada has an idea from this behaviour!

Note that if you care about inference speed, your best bet is to stick with fp16 inference here:
```python
from transformers import MusicgenForConditionalGeneration

model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small", torch_dtype=torch.float16)
```
