Hello again.
After taking your advice, I tried run_mlm.py on roberta-base under deepspeed using ZeRO-3 + cpu_offload.  This is on an AWS p4d.24xlarge instance, so 8 x A100 GPUs.
Do you have any tips on tuning the deepspeed parameters so as to maximize GPU utilization?  Right now, no matter how I tune the parameters, I cannot get the volatile GPU utilization (as reported by nvidia-smi) to above ~50% on average.  At first I thought it was due to cpu_offload causing communication stalls, but then I turned off cpu_offload and the GPUs were still very much underutilized.  I’ve tuned the per-device batch size to just below OOM, so I thought maybe the batch size is still too small for the A100s to break a sweat!  But…
… using fairscale and --sharded_ddp=‘zero_dp_3’, I am able to max out the GPU utilization (and train almost 2x faster), even though I have a slightly smaller per-device batch size.
I should note that I’m using deepspeed not so much for training a big model (roberta-base is not that big) but rather to try to jam large batch sizes onto the GPUs to accelerate training.
Any tips would be greatly appreciated!