You can also set the value explicitly:
json
{
    "gradient_accumulation_steps": 3
}
But then you're on your own synchronizing the [Trainer] command line arguments and the DeepSpeed
configuration.
Gradient Clipping
To configure gradient gradient clipping set:
json
{
    "gradient_clipping": "auto"
}
and the [Trainer] will automatically set it to the value of args.max_grad_norm.
You can also set the value explicitly:
json
{
    "gradient_clipping": 1.0
}
But then you're on your own synchronizing the [Trainer] command line arguments and the DeepSpeed
configuration.
Getting The Model Weights Out
As long as you continue training and resuming using DeepSpeed you don't need to worry about anything. DeepSpeed stores
fp32 master weights in its custom checkpoint optimizer files, which are global_step*/*optim_states.pt (this is glob
