Quantized models + PEFT + multi-gpu setup failing during training
### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.10.178-162.673.amzn2.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.8
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0

### Who can help?

@younesbelkada

### Information

- [] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

To repoduce:

(Note, this is related to https://github.com/huggingface/accelerate/pull/1523) 

```
    accelerator = Accelerator()
    model_id = "t5-base"

    # Load tokenizer of FLAN-t5-XL
    tokenizer = AutoTokenizer.from_pretrained(model_id, cache_dir = 'model_cache')
    
    dataset = get_data()
    
    tokenized_dataset = dataset.map(lambda sample: preprocess_function(sample, tokenizer), batched=True, remove_columns=["source", "target"])
    
    # print(dist.get_rank())
    model = AutoModelForSeq2SeqLM.from_pretrained(
        model_id,
        load_in_8bit=True,
        device_map='auto',
        cache_dir='model_cache')
    
    # Define LoRA Config
    lora_config = LoraConfig(
     r=16,
     lora_alpha=32,
     lora_dropout=0.05,
     bias="none",
     task_type=TaskType.SEQ_2_SEQ_LM
    )
    # prepare int-8 model for training
    model = prepare_model_for_int8_training(model)

    # add LoRA adaptor
    model = get_peft_model(model, lora_config)
    model = accelerator.prepare(model)
    
    label_pad_token_id = -100

    data_collator = DataCollatorForSeq2Seq(
        tokenizer,
        label_pad_token_id=label_pad_token_id,
        pad_to_multiple_of=None, 
        padding=False
    )

    # Define training args
    training_args = TrainingArguments(
        per_device_train_batch_size=1, 
        learning_rate=1e-3, 
        num_train_epochs=10,
        logging_strategy='steps',
        logging_steps=5,
        weight_decay=0,
        output_dir = 'weights',
        seed=22
    )
    
    # Create Trainer instance
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_dataset['train'].select(range(10)),
        data_collator=data_collator,
    )
        
    train_result = trainer.train() 
```

`tokenized_dataset` can be an arbitrary dataset.

The problem arises when running `python -m torch.distributed.launch --nproc_per_node=4 multi-gpu.py`. 
Note that it works fine if just using `python multi-gpu.py` (since only 1 GPU is used here).

I am running with four T4s.



### Expected behavior

Error message:

```
Traceback (most recent call last):
  File "/home/ec2-user/SageMaker/training/scripts/multi-gpu.py", line 131, in <module>
    main()
  File "/home/ec2-user/SageMaker/training/scripts/multi-gpu.py", line 125, in main
    train_result = trainer.train() 
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/transformers/trainer.py", line 1656, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/accelerate/accelerator.py", line 1202, in prepare
    result = tuple(
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/accelerate/accelerator.py", line 1203, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/accelerate/accelerator.py", line 1030, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/accelerate/accelerator.py", line 1270, in prepare_model
    raise ValueError(
ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices in any distributed mode. In order to use 8-bit models that have been loaded across multiple GPUs the solution is to use Naive Pipeline Parallelism. Therefore you should not specify that you are under any distributed regime in your accelerate config.
```

Some notes:

- this works if I remove 8 bit training
- I have tried this with and without `accelerator.prepare(model)` and this makes no difference (although when I remove 8bit training but keep this line, I get another error. When I remove the line, it trains fine). 

Any help appreciated!
@younesbelkada maybe you can have a look at it? 