Hi,
You can actually move stuff between cpu and cuda and the gradients will still be computed properly.
Note that the backward ops are performed on the same device where the forward op was done.
So moving the result of your gpu computation to cpu won’t reduce the cpu usage.