NotImplementedError: offload_to_cpu=True and NO_SHARD is not supported yet
### System Info

I was using fsdp with settings "full_shard auto_wrap" on a A100 GPU. The training went well but was interupted when saving the checkpoints. The error stated `NotImplementedError: offload_to_cpu=True and NO_SHARD is not supported yet`. I understand that I am using a single GPU so fsdp defaluts to NO_SHAPR. However, I dont understand why offload_to_cpu was set to True. Or anywhere I can reset it to false?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

following https://github.com/lm-sys/FastChat to fine-tune an LLM

### Expected behavior

the error as stated.
Can you please give us a code reproducer of the issue? cc @pacman100 <|||||>> Can you please give us a code reproducer of the issue? cc @pacman100

Thanks. Sure, here is the code:

`torchrun train.py \
    --model_name_or_path openlm-research/open_llama_3b  \
    --data_path /path/to/data \
    --bf16 True \
    --output_dir /path/to/output \
    --num_train_epochs 3 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps 4 \
    --evaluation_strategy "no" \
    --eval_steps 1500 \
    --save_strategy "steps" \
    --save_steps 2 \
    --save_total_limit 3 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.04 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "shard_grad_op auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True
`

`train.py` can be found from [here](https://github.com/lm-sys/FastChat/blob/main/fastchat/train/train.py)
You can use dummy data from [here](https://github.com/lm-sys/FastChat/blob/main/data/dummy_conversation.json)<|||||>Hello, thank you @linkailuo1986, the above PR should fix it. But it doesn't make sense to use FSDP on a single GPU<|||||>Thanks @pacman100 for the quick fix. I used FSDP because it seemd to reduce VRAM for a larger batch size, which otherwise got an OOM error without using it.