going to be used again in near future (less than stage3_max_reuse_distance) then we keep it to reduce communication
overhead. This is super helpful when you have activation checkpointing enabled, where we do a forward recompute and
backward passes a a single layer granularity and want to keep the parameter in the forward recompute till the backward
The following configuration values depend on the model's hidden size:
reduce_bucket_size: hidden_size*hidden_size
stage3_prefetch_bucket_size: 0.9 * hidden_size * hidden_size
stage3_param_persistence_threshold: 10 * hidden_size
therefore set these values to auto and the [Trainer] will automatically assign the recommended
values. But, of course, feel free to set these explicitly as well.
stage3_gather_fp16_weights_on_model_save enables model fp16 weights consolidation when model gets saved. With large
