T5-Flan Resuming Int-8 / LoRA / Deepspeed Checkpoint
### System Info

transformers 4.29
accelerate 0.19
peft 0.3



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

My training ended unexpectedly and I want to resume my T5-Flan training from a checkpoint. Inside my checkpoint I have:

```
global_step25000
latest
pytorch_model.bin 
rng_state.pth 
trainer_state.json 
training_args.bin
zero_to_fp32.py
```

I am unable to load the checkpoint in the following ways:

```

# try 1
model = AutoModelForSeq2SeqLM.from_pretrained("/checkpoint-25000",
                                                           load_in_8bit=True,
                                                           device_map='auto',
                                                           )
# ValueError: weight is on the meta device, we need a `value` to put in on 0.

# try 2
model = AutoModelForSeq2SeqLM.from_pretrained("/checkpoint-25000",
                                                           )
# ValueError: weight is on the meta device, we need a `value` to put in on 0.

# try 3
config = AutoConfig.from_pretrained("google/flan-t5-large")
with init_empty_weights():
    model = AutoModelForSeq2SeqLM.from_config(config)
model.tie_weights()
device_map = infer_auto_device_map(model)
model = load_checkpoint_and_dispatch(model,  "checkpoint-25000/pytorch_model.bin", device_map=device_map)

# AttributeError: 'T5ForConditionalGeneration' object has no attribute 'model'
```




### Expected behavior

The checkpoint to attach so I could resume training.
Totally didnt try the trainer, but it works!
`trainer.train('checkpoint-xxxx')`