Thanks for your kindly help!
So, you are meaning I shouldn’t force any tensor to a specific GPU device during the forward function? How about I just create some new tensors in the sub-modules and force them to be in the same device with the normal existing tensors (which should be in correct device by auto-scattering if I didn’t assign any device to them, right? )?