Zhaofeng_Wu:

but the gradient device is wrong during the optimizer step.


What kind of error are you getting?
The to() operation is differentiable, thus the parameters of different models can be on different devices.