Not sure where .to(device) is getting called twice. I believe we are supposed to call it only once, for just one device.
I have used DataParallel and not DistributedDataParallel, but I believe the semantics are the same. In my usage, I call model.to(device) only once (where device is cuda:0). After that, I do dpmodel = DataParallel(model) only once. This replicates the model onto both devices.
The input is also on the original device (cuda:0). Now, if I perform output = dpmodel(input), then the half of the input batch gets replicated to cuda:1. The computation is performed in parallel on both GPUs, and then output is computed by concatenating the two half-batches. output is on the same device as input.