what if my second loss function requires some computed value from first loss (or even the grad of first loss?) in that case I can’t add two loss together; they must be gradients respectively; and retain_graph=True gives wrong results as well as the intermediate grads not correct
see this one

  
    
    
    Backward() w.r.t. to different parameters with different loss function 
  
  
    Consider 
I have Variable x 

y = f(x) 
z = Q(y) # Q here is a neural net 

Step(1): gradient w.r.t. x. 

z.backward(retain_graph = True) 
x.grad 
g = x.grad.clone() 
x.grad.data.zero_() 

Step(2): have another function that take the gradients we just compute 

L(g) 

I want to take gradient of it w.r.t. to the weights of neural net Q 
as following 

var_opt = torch.optim.Adam(Q.parameters(), lr=lr)
while not converge:
    var_opt.zero_grad()
    variance_loss = torch.mean(L(g))
    variance_los…