samra-irshad:

I am getting two losses at end of epoch (i can imagine they are coming from two processes), but doesn’t it have to be single loss?


The loss will be local to each process and DDP only synchronizes gradients. So there will be two losses. For more details, please see this note 5 and this paper 3.

Any thoughts why the loss is twice or will it do averaging at back end?

hmm… this is weird. If you are not using DataParallel or DistributedDataParallel (just local model), how does the loss vary with batch size?

Also, any chance if the training speed can be further improved using two nodes, each with a single GPU?

With two servers, do you mean 2 GPUs per node or 2 GPUs in total? If it is the latter, probably not, as the across-machine comm can be slower than same-machine comm, especially if NVLink is available. If it is the former, it can be, but we need to first understand two things:

What is the dominating delay in each training iteration? If it is the forward/backward/opt.step(), then scale out can be helpful. Otherwise, if it is the the dataloader, then using more machines won’t help.
If you use smaller batch size in local training, how much does it speed up each iteration? If the speedup is significant, then using more machines can help.