I am trying to implement TRPO, and I need the gradient of the network parameters w.r.t. the KL divergence between the current action distribution and the action distribution after the parameters are changed.
Put simply, we have:
action_dstribution0 <-- Network(state)
We need to calculate something to the effect of:
autograd.backward(KL(Network(state), action_distribution0), [torch.ones(1)])
What is the best way to implement this? How do I make sure action_distribution0 doesn’t get backpropagated through?