models and multiple GPUs this is an expensive operation both in terms of memory and speed. It's currently required if
you plan to resume the training. Watch out for future updates that will remove this limitation and make things more
flexible.
If you're migrating from ZeRO-2 configuration note that allgather_partitions, allgather_bucket_size and
reduce_scatter configuration parameters are not used in ZeRO-3. If you keep these in the config file they will just
be ignored.
sub_group_size: 1e9
sub_group_size controls the granularity in which parameters are updated during optimizer steps. Parameters are
grouped into buckets of sub_group_size and each buckets is updated one at a time. When used with NVMe offload in
ZeRO-Infinity, sub_group_size therefore controls the granularity in which model states are moved in and out of CPU
