It seems that wrapping the complete model into nn.DataParallel uses less memory on GPU1 and 2 than wrapping just a single part of the model?
The imbalance is expected, as the default device will be used to gather the input data, loss etc. as explained here 32.
What batch size are you using in both approaches?