Well, what I facing is that, DDP has better result than DP.
However many question in here actually said that DP give better result than DDP.
Second, our task is class imbalance and binary semantic segmentation. The task is real world image with very complex background. In this cases, DP gives us the 82 % mIoU and DDP achieves the 88% in the same loss function and same learning rate.
The Loss function is the IoU Loss.
What grad syncronization and accumulation is another new question. I will read your paper first and ask question again. Thank you. It is really difficult but I hope I can make it.