If there is some tensor on GPU that requires_grad but is later copied to CPU the grad_fn for the CopyBackwards would set the src_device to be GPU so during the backwards pass would this result in the tensor being copied back to the GPU?
If so, is there a way to disable the second copy back to GPU?
Thanks!