example will contain global_step1.
Note: currently the script requires 2x general RAM of the final fp32 model weights.
ZeRO-3 and Infinity Nuances
ZeRO-3 is quite different from ZeRO-2 because of its param sharding feature.
ZeRO-Infinity further extends ZeRO-3 to support NVMe memory and multiple other speed and scalability improvements.
While all the efforts were made for things to just work without needing any special changes to your models, in certain
circumstances you may find the following information to be needed.
Constructing Massive Models
DeepSpeed/ZeRO-3 can handle models with Trillions of parameters which may not fit onto the existing RAM. In such cases,
but also if you want the initialization to happen much faster, initialize the model using deepspeed.zero.Init()
context manager (which is also a function decorator), like so:
