Body: <p>newbie here. I am studying the REINFORCE method in "Deep Reinforcement Learning Hands-On". I can't understand how, after computing the loss of the episode, that loss is backpropagated in a NN with multiple output nodes. To be more precise, in Supervised Learning, when we have multiple output nodes we know the loss of each of them, but in RL, how do we compute the loss of each output node (or maybe the partial derivative of the total loss with respect to each output layer)?</p>
  
  <p>I hope to have been clear, thanks in advance.  </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-05T20:17:33.310Z
Comments.Text: You only backpropagate the gradient through the node for the action that you took at specific timestep in trajectory. You cannot backpropagate through the rest of the nodes because you cannot estimate the return for those other nodes and you cannot estimate return for them because you never took those actions.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-05T21:41:50.543Z
Comments.Text: @Brale_ Thanks for the help. This would make much sense, but if you check [the example code here](https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/8ac75e0a7ae28b6dc49182f9ca3b2fe36ec1b161/Chapter09/02_cartpole_reinforce.py#L98) the loss function is computed for a whole minibatch and averaged, regardless of the node for the action we took, and then backpropagated to all the network. How does this work? And how can PyTorch compute such gradients with just the loss value?
Comments.UserDisplayName: Giovanni Del Gallo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-05T22:40:37.730Z
Comments.Text: I wouldn't exactly know since I didn't work with PyTorch, I would say that `batch_actions_t` holds all taken actions during trajectory, `log_prob_v` holds all action log probabilities calculated from the states encountered during trajectory. Then "correct" nodes for taken actions are extracted and that is then multiplied with returns which are calculated for all timesteps. Then they take the mean of that (to average the gradient) and `backward` method does it's automatic differentiation magic that I can't really explain. That would be my rough guess.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-06T10:50:57.060Z
Comments.Text: That's the point. I want to know the math behind the magic. How is the gradient of the loss with respect to the output of each output neuron computed?
Comments.UserDisplayName: Giovanni Del Gallo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-06T12:38:10.240Z
Comments.Text: like I said you only compute for the node of the action that you took. It's too long to discuss further in the comments read chapter 13 of this [book](http://incompleteideas.net/book/bookdraft2017nov5.pdf). This is only a draft, if you can find full book even better.
Comments.UserDisplayName: Brale
Tags: <reinforcement-learning><reinforce>
Title: How is computed the gradient with respect to each output node from a loss value?
UserDisplayName: Giovanni Del Gallo
ThreadText: # How is computed the gradient with respect to each output node from a loss value?
  
  (Asked by: username_0 on 2019-08-05)
  
  newbie here. I am studying the REINFORCE method in "Deep Reinforcement Learning Hands-On". I can't understand how, after computing the loss of the episode, that loss is backpropagated in a NN with multiple output nodes. To be more precise, in Supervised Learning, when we have multiple output nodes we know the loss of each of them, but in RL, how do we compute the loss of each output node (or maybe the partial derivative of the total loss with respect to each output layer)?
  
  I hope to have been clear, thanks in advance.
  
  **Tags:** reinforcement-learning, reinforce
  > **Comments:**
  > **username_1**: You only backpropagate the gradient through the node for the action that you took at specific timestep in trajectory. You cannot backpropagate through the rest of the nodes because you cannot estimate the return for those other nodes and you cannot estimate return for them because you never took those actions.  
  > **username_0**: @Brale_ Thanks for the help. This would make much sense, but if you check [the example code here](https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/8ac75e0a7ae28b6dc49182f9ca3b2fe36ec1b161/Chapter09/02_cartpole_reinforce.py#L98) the loss function is computed for a whole minibatch and averaged, regardless of the node for the action we took, and then backpropagated to all the network. How does this work? And how can PyTorch compute such gradients with just the loss value?  
  > **username_1**: I wouldn't exactly know since I didn't work with PyTorch, I would say that `batch_actions_t` holds all taken actions during trajectory, `log_prob_v` holds all action log probabilities calculated from the states encountered during trajectory. Then "correct" nodes for taken actions are extracted and that is then multiplied with returns which are calculated for all timesteps. Then they take the mean of that (to average the gradient) and `backward` method does it's automatic differentiation magic that I can't really explain. That would be my rough guess.  
  > **username_0**: That's the point. I want to know the math behind the magic. How is the gradient of the loss with respect to the output of each output neuron computed?  
  > **username_1**: like I said you only compute for the node of the action that you took. It's too long to discuss further in the comments read chapter 13 of this [book](http://incompleteideas.net/book/bookdraft2017nov5.pdf). This is only a draft, if you can find full book even better.  
  
  ---
  
  ## 0 answers
  
  