Regarding my experience with DataParallel, I stopped using it because while it runs faster, i usually need more epochs for good convergence (assuming the same minibatch size per GPU). So, I usually utilize multiple GPUs for hyperparameter tuning rather than splitting the dataset across multiple GPUs.
Regarding what to do with the loss. The loss is usually super cheap to compute, and since the gradients are gathered on the main GPU device anyways, I don’t think it’s worth the effort computing it on parallel devices. However, it is possible and not much hassle. For an example, see Uneven GPU utilization during training backpropagation 11.
Regarding

So does  .to() modify the the model or tensor in place, or we have to reassign it back like so:  model = model.to(device) ?

I think it works in-place, but doing
model = model.to(device) 

does not hurt. I usually always do that because I remember there were some places where this was inconsistent in PyTorch (like with the tensor example you mentioned). In my opinion, it should be named
model.to_(device) 

to be clear with regard to the rest of the API (and the tensor example you mentioned)
Hope that helps!