Hi, I’m confronted with a similiar problem, maybe even worth.
For a model training on ModelNet 40 dataset, with the same setting, when using a single GPU, the  error decreases to 5e-4 and accuracy achieves 80% in 88 epochs.
However, using 2 GPUs wrapped in DDP,  at epoch 88 the error was around 15e-4 and accuracy 48%; using 4 GPUs, at epoch 88 was around 14e-4 and accuracy around 53%. And in either case, the error and accuracy fail to be improved effectively given more epochs.
For the possibility of batch size of one, I didn’t use batchnorm, instead I applied layernorm. Therefore no SyncNorm was added. Is this choice of normalization the reason for such huge discrepancy? Also I used torch.cuda.amp throughout.
Hope to have some discussions.
The platform is Pytorch 1.7.1 with cuda 10.1.