Saving with Trainer missing config.json and tokenizer files. 
### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.4.119-19.0009.28-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
- - `Accelerate` version: 0.20.3
- Platform: Linux-5.4.119-19.0009.28-x86_64-with-glibc2.35
- Python version: 3.10.6
- Numpy version: 1.22.2
- PyTorch version (GPU?): 2.0.0 (True)
- PyTorch XPU available: False
- System RAM: 1877.62 GB
- GPU type: NVIDIA H800
- `Accelerate` default config:
	Not found
- [2023-07-10 14:40:30,136] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/usr/local/lib/python3.10/dist-packages/torch']
torch version .................... 2.0.0
deepspeed install path ........... ['/usr/local/lib/python3.10/dist-packages/deepspeed']
deepspeed info ................... 0.9.5, unknown, unknown
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 12.1
deepspeed wheel compiled w. ...... torch 2.0, cuda 12.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

run run_clm.py i have add one Callback as following to trainfer. with deepspeed zero3 enable.

```
class CheckPointFinishCallBack(TrainerCallback):

    def on_save(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):
        # Save model checkpoint
        checkpoint_folder = f"{PREFIX_CHECKPOINT_DIR}-{state.global_step}"
        log_file = os.path.join(args.output_dir,"checkpoint_saved")
        with open(log_file,"w") as writer:
            writer.write(checkpoint_folder)
```



### Expected behavior

save the mode config.json file and tokenizer files. 
@pacman100 <|||||>Hi @dumpmemory !

Could you verify if any of the 3 places `self._save` in the code snippet below is triggered?

(line 2742, 2753, 2762)

https://github.com/huggingface/transformers/blob/25411085647a4dbcbd4e7ba6f381881a3e49c33e/src/transformers/trainer.py#L2734-L2762

<|||||>> Hi @dumpmemory !
> 
> Could you verify if any of the 3 places `self._save` in the code snippet below is triggered?
> 
> (line 2742, 2753, 2762)
> 
> https://github.com/huggingface/transformers/blob/25411085647a4dbcbd4e7ba6f381881a3e49c33e/src/transformers/trainer.py#L2734-L2762

I can check it later. currently my training is using zero3 and multi gpus setting <|||||>It would be nice if you can check the execution flow at this place, to see if any `self._save` is triggered 🙏 or not (and why). Thank you! No worry, we can wait :-)<|||||>> It would be nice if you can check the execution flow at this place, to see if any `self._save` is triggered 🙏 or not (and why). Thank you! No worry, we can wait :-)

 I’m checking now. My transformer code base may not be the newest one.<|||||>I am using save_16bit_model=true<|||||>I almost found the reason and I will update my code base to the current main base and test again. <|||||>after update to the main commit code base for accelerate and transformers. it was fixed 