Hey, I am facing somme issue with data parallel. I am training on 4 v100 with a batchsize of 1. The time of forward pass seems to scale but the time of backward pass is taking 4* times in comparison to 1 V100. So there is no significant boost in speed when using 4 gpus. I guess the backward pass is taking place on single gpu. I am using nn.parallel.DataParallel, is there any solution to this problem? I can share more details if you want.