Is it possible to forward a model on gpu but calculate the loss of the last layer on cpu?
If so, how does pytorch know during backprop where the tensor is? Or is it expecting all tensors to lie consistently on one device?
If it is possible, is there a documentation article or other resource which explains this process?
Background: I calculate a loss with torch.pca_lowrank which is significantly faster on cpu in my usecase (more than 8x).
Cheers!