fixes issue when saving fsdp via accelerate's FSDP plugin
# What does this PR do?
1. Fixes https://github.com/huggingface/transformers/issues/24057#issuecomment-1595152783
2. When using Accelerate's integration for FSDP, fsdp_plugin saves the optimizer state under various configs such as Full_Dict, Sharded_Dict ... properly. For other cases such as with FSDP-XLA, the trainer's behaviour is unchanged.

_The documentation is not available anymore as the PR was closed or merged._<|||||>Thanks again for this issue! I just finished training a largeish (7b param) model using this fix and have some questions. 
I noticed the directory where the model was meant to save looks a bit odd. While the training code seems to have finished without error, the directory contains some checkpoint directories and a directory called pytorch_model_0 (which has a bunch of .distcp files), but none of the files I previously would see in my trained model directories, like the config.json and .bin files. Is this expected save behavior? <|||||>Hello @amartino1, this is because it is now using the FSDP's recommended way of saving ckpts, see this: https://github.com/pytorch/pytorch/blob/e71ab214226af1f9dbded944e939c6447e0e8f09/torch/distributed/checkpoint/examples/fsdp_checkpoint_example.py#L59

You will only notice that if you are using `SHARDED_STATE_DICT` as the `fsdp_state_dict_type`. 

with PR https://github.com/huggingface/transformers/pull/24591, it should save the whole model in transformers format as well as FSDP ckpt following what you have chosen as `fsdp_state_dict_type`. 