Fails to create Trainer object. IndexError: list index out of range at --> torch.device(devices[0]);
### System Info

The system is google colab, transformers related packages are installed from git.
```

- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0.dev0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (gpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: using one GPU


```

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
!pip install -q datasets
!pip install git+https://github.com/microsoft/LoRA
!pip install git+https://github.com/huggingface/accelerate.git
!pip install -q git+https://github.com/huggingface/peft.git
!pip install -q git+https://github.com/huggingface/transformers.git
!pip install -i https://test.pypi.org/simple/ bitsandbytes
!pip install -q sentencepiece

import torch
import os
os.environ["CUDA_VISIBLE_DEVICES"]="0"
import torch
import torch.nn as nn
import bitsandbytes as bnb
from transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM
from peft import AutoPeftModelForCausalLM

MODEL_NAME = <some lora llama2 checkpoint>

model = AutoPeftModelForCausalLM.from_pretrained(
    MODEL_NAME,
    device_map='auto',
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16,
    is_trainable=True
)

class CastOutputToFloat(nn.Sequential):
  def forward(self, x): return super().forward(x).to(torch.float32)
model.lm_head = CastOutputToFloat(model.lm_head)

for param in model.parameters():
    if param.ndim == 1:
        # cast the small parameters (e.g. layernorm) to fp32 for stability
        param.data = param.data.to(torch.float32)

model.gradient_checkpointing_enable()
model.enable_input_require_grads()


from datasets import load_dataset

qa_dataset = load_dataset("squad_v2")

def create_prompt(context, question, answer):
  if len(answer["text"]) < 1:
    answer = "Cannot Find Answer"
  else:
    answer = answer["text"][0]
  prompt_template = f"### CONTEXT\n{context}\n\n### QUESTION\n{question}\n\n### ANSWER\n{answer}</s>"
  return prompt_template

mapped_qa_dataset = qa_dataset.map(lambda samples: tokenizer(create_prompt(samples['context'], samples['question'], samples['answers'])))

import transformers

train_args = transformers.TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=1,
    warmup_steps=100,
    max_steps=100,
    learning_rate=1e-3,
    fp16=True,
    logging_steps=1,
    output_dir='outputs',
)

trainer = transformers.Trainer(
    model=model,
    train_dataset=mapped_qa_dataset["train"],
    args=train_args,
    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False),
)

```

Trainer init crashes here:

```
IndexError                                Traceback (most recent call last)
[<ipython-input-114-29de745c4455>](https://localhost:8080/#) in <cell line: 14>()
     12 )
     13 
---> 14 trainer = transformers.Trainer(
     15     model=model,
     16     train_dataset=mapped_qa_dataset["train"],

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in __init__(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers, preprocess_logits_for_metrics)
    380                 self.is_model_parallel = True
    381             else:
--> 382                 self.is_model_parallel = self.args.device != torch.device(devices[0])
    383 
    384             # warn users

IndexError: list index out of range
```

### Expected behavior

Trainer object should be constructed correctly.
Same issue as: https://discuss.huggingface.co/t/indexerror-on-devices-0-when-initializing-a-trainer/46410<|||||>I can fix that particular issue but you won't be able to actually train a model with CPU/disk offload, only do evaluation.<|||||>I figured out in my case removing
`os.environ["CUDA_VISIBLE_DEVICES"]="0"`
seem to fix the issue. 
But it is still stange as an original tutorial I followed had it set and worked on colab https://colab.research.google.com/drive/1Jt9Rpd9J1mEnf5NXREYqM5hSj-UqL24M#scrollTo=o0BZjNgEBvXH
<|||||>[Edit: it was caused by device_map="auto"  and is probably what you have meant in your reply. I managed to train by not using device_map="auto". Thank you for your fast reply.]

Also then I instantly run into 
```

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
[<ipython-input-25-c52c20b5cf4b>](https://localhost:8080/#) in <cell line: 14>()
     12 )
     13 
---> 14 trainer = transformers.Trainer(
     15     model=model,
     16     train_dataset=mapped_qa_dataset["train"],

13 frames
[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in __init__(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers, preprocess_logits_for_metrics)
    496         # Quantized models doesn't support `.to` operation.
    497         if self.place_model_on_device and not getattr(model, "is_quantized", False):
--> 498             self._move_model_to_device(model, args.device)
    499 
    500         # Force n_gpu to 1 to avoid DataParallel as MP will manage the GPUs

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _move_model_to_device(self, model, device)
    725 
    726     def _move_model_to_device(self, model, device):
--> 727         model = model.to(device)
    728         # Moving a model to an XLA device disconnects the tied weights, so we have to retie them.
    729         if self.args.parallel_mode == ParallelMode.TPU and hasattr(model, "tie_weights"):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in to(self, *args, **kwargs)
   1143             return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
   1144 
-> 1145         return self._apply(convert)
   1146 
   1147     def register_full_backward_pre_hook(

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    795     def _apply(self, fn):
    796         for module in self.children():
--> 797             module._apply(fn)
    798 
    799         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    795     def _apply(self, fn):
    796         for module in self.children():
--> 797             module._apply(fn)
    798 
    799         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    795     def _apply(self, fn):
    796         for module in self.children():
--> 797             module._apply(fn)
    798 
    799         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    795     def _apply(self, fn):
    796         for module in self.children():
--> 797             module._apply(fn)
    798 
    799         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    795     def _apply(self, fn):
    796         for module in self.children():
--> 797             module._apply(fn)
    798 
    799         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    795     def _apply(self, fn):
    796         for module in self.children():
--> 797             module._apply(fn)
    798 
    799         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    795     def _apply(self, fn):
    796         for module in self.children():
--> 797             module._apply(fn)
    798 
    799         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    795     def _apply(self, fn):
    796         for module in self.children():
--> 797             module._apply(fn)
    798 
    799         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    795     def _apply(self, fn):
    796         for module in self.children():
--> 797             module._apply(fn)
    798 
    799         def compute_should_use_set_data(tensor, tensor_applied):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _apply(self, fn)
    818             # `with torch.no_grad():`
    819             with torch.no_grad():
--> 820                 param_applied = fn(param)
    821             should_use_set_data = compute_should_use_set_data(param, param_applied)
    822             if should_use_set_data:

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in convert(t)
   1141                 return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
   1142                             non_blocking, memory_format=convert_to_format)
-> 1143             return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
   1144 
   1145         return self._apply(convert)

NotImplementedError: Cannot copy out of meta tensor; no data!

```