wasiahmad:

I am facing exactly the same issue when I run my program with 3 GPUs. In this case, do you suggest to increase the batch size? For example in my case, I set the batch size as 30 and ran on 3 Titan X GPUs. The difference between running time is huge.
While running in 1 GPU, one complete epoch takes 30 minutes but using DataParallel with 3 GPUs takes around 90 minutes. Do you have any suggestion to overcome this issue?


If you use batch_size=30 using a single GPU, then when you use DataParallel with 3 GPUs, you should use batch_size=90 to make a fair comparison. The point of using DataParallel is that you can use a larger batch_size which then requires less number of iterations to complete one full epoch.