memory from NVMe during the optimizer step. This prevents running out of CPU memory for extremely large models.
You can leave sub_group_size to its default value of 1e9 when not using NVMe offload. You may want to change its
default value in the following cases:
Running into OOM during optimizer step: Reduce sub_group_size to reduce memory utilization of temporary buffers
Optimizer Step is taking a long time: Increase sub_group_size to improve bandwidth utilization as a result of
   the increased data buffers.
NVMe Support
ZeRO-Infinity allows for training incredibly large models by extending GPU and CPU memory with NVMe memory. Thanks to
smart partitioning and tiling algorithms each GPU needs to send and receive very small amounts of data during
offloading so modern NVMe proved to be fit to allow for an even larger total memory pool available to your training
