Train Neural network problem in reinforcement learning
<SCODE>self.value_optimizer.zero_grad()
preValue = self.value_net(state).data
nextValue = self.value_net(next_state).data
expectedValue = (self.gamma * nextValue) + reward

preValue = Variable(preValue)
expectedValue = Variable(expectedValue)
loss = F.smooth_l1_loss(preValue, expectedValue)
loss.backward()
self.value_optimizer.step()

**self._execution_engine.run_backward((self,), (gradient,), retain_variables)**
**RuntimeError: there are no graph nodes that require computing gradients**
<ECODE>
cumttang
smth
<SCODE>self.value_optimizer.zero_grad()
# Here, when you unpack the data, you detach the data from the graph
# No backpropagation through the model is possible, because you got rid
# of the reference to the grpah.
preValue = self.value_net(state).data
nextValue = self.value_net(next_state).data
expectedValue = (self.gamma * nextValue) + reward

# Here, you repack the tensors in Variables, but the history of
# operations is not retained - they are leaf Variables.
# Also you didn't specify that they require gradients (they don't
# by default).
preValue = Variable(preValue)
expectedValue = Variable(expectedValue)
loss = F.smooth_l1_loss(preValue, expectedValue)
# At this point your while graph looks like this - no model there:
#  preValue        expectedValue
#         \          /
#        smooth_f1_loss
#              |
#            loss
loss.backward()
self.value_optimizer.step()
<ECODE>
apaszke
I have tried it, but it seems not work.
cumttang
cumttang
Try this: <SCODE>self.value_optimizer.zero_grad()
preValue = self.value_net(state)
nextValue = self.value_net(next_state).detach() # don't backprop this way
expectedValue = (self.gamma * nextValue) + reward
loss = F.smooth_l1_loss(preValue, expectedValue)
loss.backward()
self.value_optimizer.step()
<ECODE>
apaszke
I have tried it, but this problem occurs. <SCODE>**expectedValue = (self.gamma * nextValue) + reward**
<ECODE>
cumttang
apaszke