I see – perhaps the better approach then is to create your own version of nn.DataParallel that scatters straight from CPU to the right destination device. Then you don’t pay the cost of first going to GPU 0 and then scatter from there to the other GPUs.
edit: It looks like nn.DataParallel already supports this if you just keep your input on CPU.