Have a look at this blog post 57 to see how nn.DataParallel works internally and how to counter some effects of an imbalanced memory usage.
PS: you could also try out nn.DistributedDataParallel, which shouldn’t introduce the imbalance.