Another question,(sorry I’m new to pytorch)
image1080×353 130 KB
By this image in some blog about nn.DataParallel.
The first step in Backward(Compute loss gradient on GPU-1) results in imbalanced GPU usage.
Is that means in DataParallel loss.backward() only happens in GPU-1 not other GPUs, but optimizer.step(),optimizer.zero_grad() are parallel(step 2,3,4 in backward)?
Thank you very much.