I was able to run the program with reduced batch-size via nn.Distributed, I noticed the time per training step is reduced from 1s (nn.DataParallel) to 0.8s (distributed) however, the loss per epoch is twice the loss I was getting from nn.DataParallel and I am getting two losses at end of epoch (i can imagine they are coming from two processes), but doesn’t it have to be single loss? Any thoughts why the loss is twice or will it do averaging at back end? Also, any chance if the training speed can be further improved using two nodes, each with a single GPU? Currently I am using a single node with two GPUs.
Output for nn.DataParallel:
[Training] Epoch: 0, LossG: 4.1266
Output with nn.Distributed:
[Training] Epoch: 0, LossG: 9.2402                                                                           
[Training] Epoch: 0, LossG: 9.2885