Solved by albanD in post #2 
              
              
                This will not send data to the cpu indeed. But you want to add a .detach() to make sure that the computational graph associated with loss is not kept around otherwise your memory usage is going to quickly explode.