Thank you for answer. But we seem to be talking different things.
# Part of valueDice code
      loss = (non_linear_loss - linear_loss)

      # maybe loss.backward() ? I think this is not a problem.

      nu_loss = loss + nu_grad_penalty * nu_reg
      pi_loss = -loss + keras_utils.orthogonal_regularization(self.actor.trunk)

    nu_grads = tape.gradient(nu_loss, self.nu_net.variables) 
    pi_grads = tape.gradient(pi_loss, self.actor.variables)
    # or (nu_loss + pi_loss).backward() ?

    self.nu_optimizer.apply_gradients(zip(nu_grads, self.nu_net.variables))
    self.actor_optimizer.apply_gradients(zip(pi_grads, self.actor.variables))

In this example, it doesn’t seem like it’s possible to simply add and backward in pytorch.
The main purpose is not for a(nu) and b(pi), but we have to find the slopes for different parameters in different directions for the loss(non_linear_loss - linear_loss).
Am I misunderstanding?