pattern), and are saved under the normal checkpoint.
FP16 Weights:
When a model is saved under ZeRO-2, you end up having the normal pytorch_model.bin file with the model weights, but
they are only the fp16 version of the weights.
Under ZeRO-3, things are much more complicated, since the model weights are partitioned out over multiple GPUs,
therefore "stage3_gather_fp16_weights_on_model_save": true is required to get the Trainer to save the fp16
version of the weights. If this setting is False pytorch_model.bin` won't be created. This is because by default DeepSpeed's `state_dict` contains a placeholder and not the real weights. If we were to save this `state_dict it
won't be possible to load it back.
json
{
    "zero_optimization": {
        "stage3_gather_fp16_weights_on_model_save": true
