LoRA is incompatible with DeepSpeed ZeRO3
### System Info

`pytorch==2.0.0, transformers==4.28.0, peft==0.2.0`

When use LoRA to wrap model in `__init__` and enable deepspeed ZeRO3, i will get the following errors:

```
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/peft/peft_model.py:287 in __getattr__                            │
│                                                                              │
│   284 │   def __getattr__(self, name: str):                                  │
│   285 │   │   """Forward missing attributes to the wrapped module."""        │
│   286 │   │   try:                                                           │
│ ❱ 287 │   │   │   return super().__getattr__(name)  # defer to nn.Module's l │
│   288 │   │   except AttributeError:                                         │
│   289 │   │   │   return getattr(self.base_model, name)                      │
│   290                                                                        │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/torch/nn/modules/module.py:1614 in __getattr__                   │
│                                                                              │
│   1611 │   │   │   modules = self.__dict__['_modules']                       │
│   1612 │   │   │   if name in modules:                                       │
│   1613 │   │   │   │   return modules[name]                                  │
│ ❱ 1614 │   │   raise AttributeError("'{}' object has no attribute '{}'".form │
│   1615 │   │   │   type(self).__name__, name))                               │
│   1616 │                                                                     │
│   1617 │   def __setattr__(self, name: str, value: Union[Tensor, 'Module'])  │
╰──────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'PeftModelForCausalLM' object has no attribute 
'_ds_child_entered'

During handling of the above exception, another exception occurred:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/peft/peft_model.py:287 in __getattr__                            │
│                                                                              │
│   284 │   def __getattr__(self, name: str):                                  │
│   285 │   │   """Forward missing attributes to the wrapped module."""        │
│   286 │   │   try:                                                           │
│ ❱ 287 │   │   │   return super().__getattr__(name)  # defer to nn.Module's l │
│   288 │   │   except AttributeError:                                         │
│   289 │   │   │   return getattr(self.base_model, name)                      │
│   290                                                                        │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/torch/nn/modules/module.py:1614 in __getattr__                   │
│                                                                              │
│   1611 │   │   │   modules = self.__dict__['_modules']                       │
│   1612 │   │   │   if name in modules:                                       │
│   1613 │   │   │   │   return modules[name]                                  │
│ ❱ 1614 │   │   raise AttributeError("'{}' object has no attribute '{}'".form │
│   1615 │   │   │   type(self).__name__, name))                               │
│   1616 │                                                                     │
│   1617 │   def __setattr__(self, name: str, value: Union[Tensor, 'Module'])  │
╰──────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'PeftModelForCausalLM' object has no attribute 'base_model'
```

It seems like that deepspeed begins to partition parameters before `PeftModelForCausalLM` finish its `__init__`, since it can not get the attribute `base_model`.

It's also notable that this error leads to a infinite recursion, since `PeftModel` catch the AttributeError when trying to get the attribute `base_model` while this attribute does not exist so the AttributeError will be raised again and again.

```
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /mnt/petrelfs/wangweiyun/projects/region_wise_model/main_clip_v6.py:120 in   │
│ <module>                                                                     │
│                                                                              │
│   117                                                                        │
│   118                                                                        │
│   119 if __name__ == '__main__':                                             │
│ ❱ 120 │   main()                                                             │
│   121                                                                        │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/projects/region_wise_model/main_clip_v6.py:42 in    │
│ main                                                                         │
│                                                                              │
│    39 │                                                                      │
│    40 │   if config.use_window_attn:                                         │
│    41 │   │   state_dict = preprocess_state_dict(model_args.model_name_or_pa │
│ ❱  42 │   │   model = HuskyForCLIP.from_pretrained(model_args.model_name_or_ │
│    43 │   else:                                                              │
│    44 │   │   model = HuskyForCLIP.from_pretrained(model_args.model_name_or_ │
│    45                                                                        │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/transformers/modeling_utils.py:2629 in from_pretrained           │
│                                                                              │
│   2626 │   │   │   init_contexts.append(init_empty_weights())                │
│   2627 │   │                                                                 │
│   2628 │   │   with ContextManagers(init_contexts):                          │
│ ❱ 2629 │   │   │   model = cls(config, *model_args, **model_kwargs)          │
│   2630 │   │                                                                 │
│   2631 │   │   # Check first if we are `from_pt`                             │
│   2632 │   │   if use_keep_in_fp32_modules:                                  │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/deepspeed/runtime/zero/partition_parameters.py:382 in wrapper    │
│                                                                              │
│    379 │   │   │   │   │   is_child_module = True                            │
│    380 │   │   │   │   │   setattr(module, "_ds_child_entered", True)        │
│    381 │   │   │   │                                                         │
│ ❱  382 │   │   │   │   f(module, *args, **kwargs)                            │
│    383 │   │   │   │                                                         │
│    384 │   │   │   │   if is_child_module:                                   │
│    385 │   │   │   │   │   # child's __init__ is done, now we can run a sing │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/projects/region_wise_model/custom_models/husky_clip │
│ _ablate.py:1472 in __init__                                                  │
│                                                                              │
│   1469 # shared align token + Both flatten + soft prompt (best)              │
│   1470 class HuskyForCLIPV6(WindowRegionHusky):                              │
│   1471 │   def __init__(self, config: WindowRegionHuskyConfig):              │
│ ❱ 1472 │   │   super().__init__(config)                                      │
│   1473 │   │                                                                 │
│   1474 │   │   self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1 / 0 │
│   1475 │   │   self.text_projection = nn.Parameter(torch.empty(self.language │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/deepspeed/runtime/zero/partition_parameters.py:382 in wrapper    │
│                                                                              │
│    379 │   │   │   │   │   is_child_module = True                            │
│    380 │   │   │   │   │   setattr(module, "_ds_child_entered", True)        │
│    381 │   │   │   │                                                         │
│ ❱  382 │   │   │   │   f(module, *args, **kwargs)                            │
│    383 │   │   │   │                                                         │
│    384 │   │   │   │   if is_child_module:                                   │
│    385 │   │   │   │   │   # child's __init__ is done, now we can run a sing │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/projects/region_wise_model/custom_models/husky_wind │
│ ow.py:47 in __init__                                                         │
│                                                                              │
│    44 │   │   │   │   │   self.vision_model.encoder.layers[idx] = WindowBLIP │
│    45 │   │   │                                                              │
│    46 │   │   │   if self.config.lora:                                       │
│ ❱  47 │   │   │   │   self.wrap_lora()                                       │
│    48 │   │   │   if self.config.lora_vision:                                │
│    49 │   │   │   │   self.wrap_lora_vision()                                │
│    50 │   │   self.post_init()                                               │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/projects/region_wise_model/custom_models/husky_src/ │
│ husky_chat.py:436 in wrap_lora                                               │
│                                                                              │
│   433 │   │   │   lora_dropout=lora_dropout,                                 │
│   434 │   │   │   target_modules=target_modules                              │
│   435 │   │   )                                                              │
│ ❱ 436 │   │   self.language_model = get_peft_model(self.language_model, peft │
│   437 │   │   self.config.lora = True                                        │
│   438 │   │   self.language_model.print_trainable_parameters()               │
│   439                                                                        │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/peft/mapping.py:145 in get_peft_model                            │
│                                                                              │
│   142 │   │   peft_config = _prepare_lora_config(peft_config, model_config)  │
│   143 │   else:                                                              │
│   144 │   │   peft_config = _prepare_prompt_learning_config(peft_config, mod │
│ ❱ 145 │   return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](mod │
│   146                                                                        │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/deepspeed/runtime/zero/partition_parameters.py:377 in wrapper    │
│                                                                              │
│    374 │   │   │   │   print_rank_0(f'Before initializing {module.__class__. │
│    375 │   │   │   │                                                         │
│    376 │   │   │   │   is_child_module = False                               │
│ ❱  377 │   │   │   │   if not hasattr(module, "_ds_child_entered"):          │
│    378 │   │   │   │   │   # child's __init__ was called, since parents all  │
│    379 │   │   │   │   │   is_child_module = True                            │
│    380 │   │   │   │   │   setattr(module, "_ds_child_entered", True)        │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/peft/peft_model.py:289 in __getattr__                            │
│                                                                              │
│   286 │   │   try:                                                           │
│   287 │   │   │   return super().__getattr__(name)  # defer to nn.Module's l │
│   288 │   │   except AttributeError:                                         │
│ ❱ 289 │   │   │   return getattr(self.base_model, name)                      │
│   290 │                                                                      │
│   291 │   def forward(self, *args, **kwargs):                                │
│   292 │   │   """                                                            │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/peft/peft_model.py:289 in __getattr__                            │
│                                                                              │
│   286 │   │   try:                                                           │
│   287 │   │   │   return super().__getattr__(name)  # defer to nn.Module's l │
│   288 │   │   except AttributeError:                                         │
│ ❱ 289 │   │   │   return getattr(self.base_model, name)                      │
│   290 │                                                                      │
│   291 │   def forward(self, *args, **kwargs):                                │
│   292 │   │   """                                                            │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/peft/peft_model.py:289 in __getattr__                            │
│                                                                              │
│   286 │   │   try:                                                           │
│   287 │   │   │   return super().__getattr__(name)  # defer to nn.Module's l │
│   288 │   │   except AttributeError:                                         │
│ ❱ 289 │   │   │   return getattr(self.base_model, name)                      │
│   290 │                                                                      │
│   291 │   def forward(self, *args, **kwargs):                                │
│   292 │   │   """                                                            │
│                                                                              │
│ /mnt/petrelfs/wangweiyun/miniconda3/envs/recognize_anything/lib/python3.9/si │
│ te-packages/peft/peft_model.py:289 in __getattr__                            │
│                                                                              │
│   286 │   │   try:                                                           │
│   287 │   │   │   return super().__getattr__(name)  # defer to nn.Module's l │
│   288 │   │   except AttributeError:                                         │
│ ❱ 289 │   │   │   return getattr(self.base_model, name)                      │
│   290 │                                                                      │
│   291 │   def forward(self, *args, **kwargs):                                │
│   292 │   │   """                                                            │
```

### Who can help?

@pacman100

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

environments: `pytorch==2.0.0, transformers==4.28.0, peft==0.2.0`

slurm launch command: `srun --gres=gpu:8 --ntasks=8 --ntasks-per-node=8 --cpus-per-task=8 python -u bug_unit_test.py --output_dir ./outputs/debug --deepspeed ./configs/default_offload_opt_param_zero3.json`

deepspeed config to reproduce:

```json
{
    "bf16": {
        "enabled": "auto"
    },
    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_16bit_weights_on_model_save": true
    },
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false
}
```

code to reproduce:

```python
import os
import subprocess
import torch
from transformers import (
    HfArgumentParser, TrainingArguments,
    PreTrainedModel, LlamaModel, LlamaConfig
)
from peft import LoraConfig, TaskType, get_peft_model


class BugModel(PreTrainedModel):
    config_class = LlamaConfig

    def __init__(self, config):
        super().__init__(config)
        self.model = LlamaModel(config)
        self.wrap_lora()
        # init code for other modules, which is not important to reproduce this bug
        pass

    def wrap_lora(
        self,
        r=16,
        lora_alpha=32,
        lora_dropout=0.05,
        target_modules=("q_proj", "k_proj", "v_proj", "o_proj"),
    ):
        peft_config = LoraConfig(
            task_type=TaskType.CAUSAL_LM,
            inference_mode=False,
            r=r,
            lora_alpha=lora_alpha,
            lora_dropout=lora_dropout,
            target_modules=target_modules
        )
        self.model = get_peft_model(self.model, peft_config)
        self.model.print_trainable_parameters()


def init_distributed_mode():
    if 'SLURM_PROCID' in os.environ:
        rank = int(os.environ['SLURM_PROCID'])
        local_rank = rank % torch.cuda.device_count()

        world_size = int(os.environ["SLURM_NTASKS"])
        local_size = int(os.environ["SLURM_NTASKS_PER_NODE"])

        if "MASTER_PORT" not in os.environ:
            port = 22110
            print(f'MASTER_PORT = {port}')
            os.environ["MASTER_PORT"] = str(port)

        node_list = os.environ["SLURM_NODELIST"]
        addr = subprocess.getoutput(f"scontrol show hostname {node_list} | head -n1")
        if "MASTER_ADDR" not in os.environ:
            os.environ["MASTER_ADDR"] = addr

        os.environ['RANK'] = str(rank)
        os.environ['LOCAL_RANK'] = str(local_rank)
        os.environ['LOCAL_WORLD_SIZE'] = str(local_size)
        os.environ['WORLD_SIZE'] = str(world_size)


parser = HfArgumentParser(TrainingArguments)
init_distributed_mode()
training_args = parser.parse_args_into_dataclasses()


model_name_or_path = '/mnt/petrelfs/share_data/wangweiyun/share_hf/vicuna-7b'
model = BugModel.from_pretrained(model_name_or_path)  # Error!


print('finish')
```

### Expected behavior

I expect to wrap the model with LoRA during `__init__` successfully when i enable ZeRO3.
Hello, please refer this doc for the correct way of using PEFT + DeepSpeed: https://huggingface.co/docs/peft/accelerate/deepspeed-zero3-offload<|||||>> Hello, please refer this doc for the correct way of using PEFT + DeepSpeed: https://huggingface.co/docs/peft/accelerate/deepspeed-zero3-offload

 Thank you for your response!

I note that this doc is based on `accelerate`. However, my code is based on `transformers.Trainer`. Can you provide me any example to use PEFT + DeepSpeed with `transformers.Trainer` correctly?<|||||>The following steps work for me:
1. Create `TrainingArguments(..., deepspeed="ds_config_zero3.json")`
2. Load model with `from_pretrained()`
3. Wrap it with `get_peft_model()`
4. Run `Trainer.train()`

Few important notes:
1. You have to create `TrainingArguments` before initialising the model with Zero3 partitioning.
2. If you use `TaskType.SEQ_CLS` task, `get_peft_model` will break the forward path. A quick workaround is recreate unpartitioned classification head after the model initialised with `deepspeed.zero.Init()`, i.e. after `from_pretrained()`.