Hi,
The backward operation is always performed on the same device where the forward was performed. So moving the loss to the cpu does not force the backward to be computed on the cpu.
There is no way to do this at the moment.
The usual way to deal with this is to reduce the batch size. Potentially doing 2 forward/backward before doing an optimizer.step to double the effective batch size.