Using previous output values as part of a loss function
I am trying to implement TRPO, and I need the gradient of the network parameters w.r.t. the KL divergence between the current action distribution and the action distribution after the parameters are changed. What is the best way to implement this? How do I make sure action_distribution0 doesn’t get backpropagated through?
yoonholee
<SCODE>action_distribution0 = model(state).detach()
# detach() blocks the gradient
# also action_distribution0 doesn't require grad now, so it can be a loss target

# =========================================
# change the parameters of the network here
# =========================================

KL(Network(state), action_distribution0).backward()
<ECODE>
apaszke
This worked great, thanks!
yoonholee
<SCODE>fake = netG(noise)
fake = fake.detach() # this is not an inplace op, so need to re-assign?
output = netD(fake)<ECODE>
tomsercu
smth