I’m converting a DataParallel (DP) model into DistributedDataParallel (DDP).
However, the DDP degradates the testing performance and leads to zigzag loss curves during training.
Below is the loss curve during training (blue is DDP and orange is DP):

企业微信截图_163056778822342131×414 24.9 KB

In general these two behave similar. But when zooming in, the DDP presents periodic zigzags:

企业微信截图_163056796453662150×402 33.5 KB

My training configurations are:


DP:

batch size 48
3 GPUs



DDP

batchsize per GPU: 16
3 GPUs



Random seeds are fixed and learning rate and all other configurations are the same.