The intermediate activations in the model will still be on the GPU, as you’ve just moved the outputs to the CPU. All operations afterwards will be executed on the CPU. However, once your backward call reaches the model, it will transfer back to the GPU.