do you want to retain the outputs on their respective GPUs without being gathered back onto a particular GPU?
If so, you want to do the scatter, parallel_apply and avoid gather yourself. These are primitives under nn.parallel. DataParallel is effectively a composition scatter + parallel_apply + gather