Hi,
The .to() operator is differentiable (just move values to the right device). And so the backward will work exactly the same as if  everything was on a single device. The backward ops will run on the same device as the forward ones.