Depending on the workload you are dealing with, you might see the communication overhead using nn.DataParallel.
Could you post some information on the models and data (batch size) you are currently using?