I am facing exactly the same issue when I run my program with 3 GPUs. In this case, do you suggest to increase the batch size? For example in my case, I set the batch size as 30 and ran on 3 Titan X GPUs. The difference between running time is huge.
While running in 1 GPU, one complete epoch takes 30 minutes but using DataParallel with 3 GPUs takes around 90 minutes. Do you have any suggestion to overcome this issue?