Mask2Former Model Doesn't Move to GPU
### System Info

transformers version: 4.31.0 (same bug occurs until version 4.27.0)
pytorch 2.0.1+cu118 (same bug occurs with cu117)
python: python 3.10
systems: NVIDIA RTX 3090 CUDA 12.0


### Who can help?

@amyeroberts @sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

First of all, my full error traceback is this:
Traceback (most recent call last):
  File "C:\Users\labuser\hubmap\mask2former\train.py", line 428, in <module>
    model, history = run_training(model, optimizer, scheduler,train_dataloader = train_dataloader, val_dataloader= val_dataloader,
  File "C:\Users\labuser\hubmap\mask2former\train.py", line 255, in run_training
    train_loss = epoch_train(model, optimizer, scheduler,
  File "C:\Users\labuser\hubmap\mask2former\train.py", line 166, in epoch_train
    outputs = model(
  File "C:\ProgramData\Anaconda3\envs\hubmap\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\hubmap\lib\site-packages\transformers\models\mask2former\modeling_mask2former.py", line 2496, in forward
    outputs = self.model(
  File "C:\ProgramData\Anaconda3\envs\hubmap\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\hubmap\lib\site-packages\transformers\models\mask2former\modeling_mask2former.py", line 2271, in forward
    transformer_module_output = self.transformer_module(
  File "C:\ProgramData\Anaconda3\envs\hubmap\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\hubmap\lib\site-packages\transformers\models\mask2former\modeling_mask2former.py", line 2066, in forward
    self.input_projections[i](multi_scale_features[i]).flatten(2)
  File "C:\ProgramData\Anaconda3\envs\hubmap\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\hubmap\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\ProgramData\Anaconda3\envs\hubmap\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

Steps to reproduce error:
1. Use any dataset of choice, doesn't matter since the inputs to the model is in cuda already, the model is the issue. (And I've made sure there is the line "model = model.to(device)", and there is only one model loaded. 
2. Now write any training code (dummy code) and make sure to run below two lines when loading empty model for Mask2FormerForUniversalSegmentation:

config = config = Mask2FormerConfig(feature_size=512, mask_feature_size=512)
model = Mask2FormerForUniversalSegmentation(config)

3. Then run dummy training code and you get this error. 

I'm really not sure how to resolve this issue- I've moved my model to my device, and by doing nvidia-smi I can confirm that the inputs are being transferred over to my GPU memory, I just cannot understand why the model weights are not being transferred when literally writing the code "model = model.to(device)". This also only happens with transformers, on other models on torch it works perfectly fine on the same exact environment so I doubt it's a bug with torch. Thank you!

Also, the issue is also discussed without any solutions in this [forum link](https://discuss.huggingface.co/t/mask2former-cuda-training/47072) 

### Expected behavior

As discussed above, expected behavior would be so that the model is also moved to cuda (GPU). 
Hi @chokevin8 

Thank you for reporting. Could you provide a self-contained code snippet so the reproduction of the error is direct.

Also you can enclose the error log in a block like the following

\`\`\`bash
  error log ...
\`\`\`

to make it easier to read. 

Thank you in advance 🤗 