'UserWarning: Module is put on CPU' when use FSDP by Accelerate
script: https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm_no_trainer.py

accelerate launch run_mlm_no_trainer.py \
    --model_name_or_path bert-base-chinese \
    --train_file $DATA_DIR/train.txt \
    --validation_file $DATA_DIR/val.txt \
    --line_by_line True \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 16 \
    --output_dir $MODEL_DIR/test-mlm

![image](https://user-images.githubusercontent.com/12816241/186734597-0c70137b-bfdb-44a4-add0-3a44ef6bc71e.png)

![image](https://user-images.githubusercontent.com/12816241/186734845-c99752c7-050c-49a5-abcf-2a266c32d238.png)

UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.

**How can I solve this problem?** 


Maybe cc @pacman100 ? :)<|||||>Hello @scuyjzh ,  as mentioned in https://github.com/huggingface/accelerate/issues/659, you can safely ignore that warning as it is only during model initialization under FSDP. I will look into this later as and when time permits because this doesn't impact model correctness or performance gains through fsdp.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.