I am by no mean a specialist on DataParallel and I have a cpu-only install so I couldn’t test the code.
The doc states explicitly that if the module returns a 0dim tensor, then you will get a 1D tensor with size the number of gpus. That does not work? Maybe that’s only supported in the latest version.
Sharing the weights among gpus is not possible as they need to be on the device to perform computations.
I am not sure at the moment what creates this inbalance…
Maybe @smth knows?