What DataParallel does is more version 3 of this image: split the input on each GPUs and run on each of them independently. Then accumulate.
Note that the backward will run on the same device as the forward. whatever the device of the Tensor on which you call .backward().