Ho, I think I misunderstood that the two are actually share the parameters but you want each loss to only participate to the gradient of a subset of weights.
In that case, you want to do two backwards indeed.
If you’re using the nightly build, you can either just get the grad with autograd.grad. Or if you’re using nightly pytorch, you can specify to .backward() which inputs you want the gradients to be computed for.
In particular, if you want to reproduce
    nu_grads = tape.gradient(nu_loss, self.nu_net.variables) 
    pi_grads = tape.gradient(pi_loss, self.actor.variables)

I think you just want
    # Set retain_grad in the first one because your call to backwards on the same graph
    nu_grads = autograd.grad(nu_loss, self.nu_net.variables, retain_graph=True) 
    pi_grads = autograd.grad(pi_loss, self.actor.variables)