Solved by ptrblck in post #3 
              
              
                Yes, you are detaching the loss tensor from the computation graph by recreating a new tensor, so you would have to create the loss tensor by using torch.cat, torch.stack, torch.where etc. which are all differentiable.