When I train my network with a single GPU, the training process terminates successfully after 120 epochs. However, if I use two GPUs, I get nan loss after a dozen epochs. The only thing I change is the batch size. For single GPU I use a batch size of 2 and for 2 GPUs I use a batch size of 1 for each GPU. The other parameters are exactly the same. I also replace every batchnorm2d layer with a syncbatchnorm layer. Strangely, syncbatchnorm gives higher loss. What could be the possible reasons?