Answers.Body: <blockquote>
  <p>If so, then would it be correct to take the gradient of this expression of <span class="math-container">$\hat{q}(S,A,w)$</span> with respect to the weights of the network</p>
  </blockquote>
  <p>Yes, your expression for <span class="math-container">$\hat{q}(S,A,w)$</span> looks correct for your network, and can be differentiated.</p>
  <blockquote>
  <p>and plug into into the following update equation for back propagation?</p>
  </blockquote>
  <p>Not clear what you intend. If you already have the gradient <span class="math-container">$\nabla\hat{q}(S,A,w)$</span> then you have <em>already</em> performed backpropagation from the current output value to all weights. The additional factor of <span class="math-container">$[R + \gamma \hat{q}(S',A',w) - \hat{q}(S,A,w)]$</span> is effectively the gradient of the MSE cost function w.r.t. the current output that needs to be multiplied in so that the backprop starts from the MSE loss.</p>
  <p>The equations given in Sutton&amp;Barto are not that easy to apply when you already have a neural network library that has everything defined around supervised learning.</p>
  <p>You may find it easier to calculate a target value for the whole action value vector and use your library's definitions of loss functions and update steps. That would go something like this:</p>
  <ul>
  <li>Set the loss function of the neural network to MSE.</li>
  <li>Set the optimisation method to SGD (although you now have the option of using others e.g. Adam, momentum, mini-batches etc)</li>
  <li>Calculate <span class="math-container">$\begin{bmatrix}
      \hat{q}(S,a_1,w)  &amp;  \hat{q}(S,a_2,w) &amp;  \hat{q}(S,a_3,w)         
  \end{bmatrix}$</span> forward using the neural network.</li>
  <li>Calculate the TD target <span class="math-container">$G = R + \gamma \hat{q}(S',A',w)$</span></li>
  <li>Substitute the target value into the forward matrix e.g. <span class="math-container">$\begin{bmatrix}
      \hat{q}(S,a_1,w)  &amp;  \hat{q}(S,a_2,w) &amp;  G         
  \end{bmatrix}$</span> assuming it was action 3 that the agent took.</li>
  <li>Use the adjusted forward matrix as the SGD training value.</li>
  </ul>
  <p>This will automatically provide a gradient 0 associated with actions that were not taken, and is effectively the same update as per Sutton&amp;Barto, but using the NN library as-is without trying to tweak the gradient calculations directly.</p>
  <p>If you are using a library like PyTorch, then it is definitely possible to calculate the pure gradient of the selected action as you are attempting. However, you may find yourself in more familiar and supported territory if you re-frame the updates as supervised learning as suggested above.</p>
  <p>I suggest one of the first things you try if this does not converge is to batch up the updates and only perform them every e.g. 100 steps. Again, this is <em>possible</em> using the S&amp;B update rule, and something like PyTorch, but things will be easier if you go with the grain and re-frame the updates as supervised learning &quot;inside&quot; the RL mechanism.</p>
  <p>I am not sure how well SARSA will cope with Lunar Lander. I have solved it using DQN, which is similar, but has the advantage of re-using older off-policy experience, sampling from it randomly, which SARSA cannot do. To correct for SARSA's weakness there, you may need quite large batches to avoid too much correlation between experience. A common pattern with &quot;Deep SARSA&quot; is to sample multiple environments at once under the current policy, to build a large batch (you see this in A2C and other on-policy policy gradient methods too).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T17:45:45.480Z
Answers.LastActivityDate: 2022-05-30T17:45:45.480Z
Answers.UserDisplayName: Neil Slater
Body: <p><a href="https://i.sstatic.net/KDGyW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KDGyW.png" alt="enter image description here" /></a></p>
  <p>For the back weight update step, I need to calculate <span class="math-container">$\nabla\hat{q}(S,A,w)$</span>. My neural network takes in the state vector <span class="math-container">$S$</span> and gives out the action values for state <span class="math-container">$S$</span> and each action in the action space.</p>
  <p>Since <span class="math-container">$\epsilon$</span>-greedy randomly chooses between the greedy action and a random action, I was not sure what the loss for the network should be. Let's say I have 3 possible actions - <span class="math-container">$a_1, a_2, a_3$</span> and my neural network outputs the vector of action-values for each of these actions in state <span class="math-container">$S$</span>. If <span class="math-container">$a_3$</span> is the action chosen by <span class="math-container">$\epsilon$</span>-greedy, then is this a correct expression for <span class="math-container">$\hat{q}(S,A,w)$</span>?</p>
  <p><span class="math-container">\begin{equation}
  \hat{q}(S,A,w) = 
  \begin{bmatrix}
      \hat{q}(S,a_1,w)  &amp;  \hat{q}(S,a_2,w) &amp;  \hat{q}(S,a_3,w)         
  \end{bmatrix}
  \times
  \begin{bmatrix}
      0       \\
      0       \\
      1
  \end{bmatrix} 
  \end{equation}</span></p>
  <p>If so, then would it be correct to take the gradient of this expression of <span class="math-container">$\hat{q}(S,A,w)$</span> with respect to the weights of the network and plug into into the following update equation for back propagation?</p>
  <p><span class="math-container">\begin{equation}
  w = w + \alpha~[R + \gamma \hat{q}(S',A',w) - \hat{q}(S,A,w)]\nabla\hat{q}(S,A,w)
  \end{equation}</span></p>
  <p>I was trying to use this in the Lunar-Lander Open-AI gym environment but it never converged. So I wanted to make sure this approach is correct before trying other things.</p>
  
Tags: <reinforcement-learning><deep-rl><sutton-barto><sarsa>
Title: How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?
UserDisplayName: ketan dhanuka
ThreadText: # How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?
  
  (Asked by: username_0 on 2022-05-30)
  
  [![enter image description here](https://i.sstatic.net/KDGyW.png)](https://i.sstatic.net/KDGyW.png)
  
  For the back weight update step, I need to calculate $\nabla\hat{q}(S,A,w)$. My neural network takes in the state vector $S$ and gives out the action values for state $S$ and each action in the action space.
  
  Since $\epsilon$-greedy randomly chooses between the greedy action and a random action, I was not sure what the loss for the network should be. Let's say I have 3 possible actions - $a\_1, a\_2, a\_3$ and my neural network outputs the vector of action-values for each of these actions in state $S$. If $a\_3$ is the action chosen by $\epsilon$-greedy, then is this a correct expression for $\hat{q}(S,A,w)$?
  
  \begin{equation}
  \hat{q}(S,A,w) =
  \begin{bmatrix}
  \hat{q}(S,a\_1,w) & \hat{q}(S,a\_2,w) & \hat{q}(S,a\_3,w)
  \end{bmatrix}
  \times
  \begin{bmatrix}
  0 \\
  0 \\
  1
  \end{bmatrix}
  \end{equation}
  
  If so, then would it be correct to take the gradient of this expression of $\hat{q}(S,A,w)$ with respect to the weights of the network and plug into into the following update equation for back propagation?
  
  \begin{equation}
  w = w + \alpha~[R + \gamma \hat{q}(S',A',w) - \hat{q}(S,A,w)]\nabla\hat{q}(S,A,w)
  \end{equation}
  
  I was trying to use this in the Lunar-Lander Open-AI gym environment but it never converged. So I wanted to make sure this approach is correct before trying other things.
  
  **Tags:** reinforcement-learning, deep-rl, sutton-barto, sarsa
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-30. Score: 2 | Accepted answer
  
  > If so, then would it be correct to take the gradient of this expression of $\hat{q}(S,A,w)$ with respect to the weights of the network
  
  Yes, your expression for $\hat{q}(S,A,w)$ looks correct for your network, and can be differentiated.
  
  > and plug into into the following update equation for back propagation?
  
  Not clear what you intend. If you already have the gradient $\nabla\hat{q}(S,A,w)$ then you have *already* performed backpropagation from the current output value to all weights. The additional factor of $[R + \gamma \hat{q}(S',A',w) - \hat{q}(S,A,w)]$ is effectively the gradient of the MSE cost function w.r.t. the current output that needs to be multiplied in so that the backprop starts from the MSE loss.
  
  The equations given in Sutton&Barto are not that easy to apply when you already have a neural network library that has everything defined around supervised learning.
  
  You may find it easier to calculate a target value for the whole action value vector and use your library's definitions of loss functions and update steps. That would go something like this:
  
  * Set the loss function of the neural network to MSE.
  * Set the optimisation method to SGD (although you now have the option of using others e.g. Adam, momentum, mini-batches etc)
  * Calculate $\begin{bmatrix}
    \hat{q}(S,a\_1,w) & \hat{q}(S,a\_2,w) & \hat{q}(S,a\_3,w)
    \end{bmatrix}$ forward using the neural network.
  * Calculate the TD target $G = R + \gamma \hat{q}(S',A',w)$
  * Substitute the target value into the forward matrix e.g. $\begin{bmatrix}
    \hat{q}(S,a\_1,w) & \hat{q}(S,a\_2,w) & G
    \end{bmatrix}$ assuming it was action 3 that the agent took.
  * Use the adjusted forward matrix as the SGD training value.
  
  This will automatically provide a gradient 0 associated with actions that were not taken, and is effectively the same update as per Sutton&Barto, but using the NN library as-is without trying to tweak the gradient calculations directly.
  
  If you are using a library like PyTorch, then it is definitely possible to calculate the pure gradient of the selected action as you are attempting. However, you may find yourself in more familiar and supported territory if you re-frame the updates as supervised learning as suggested above.
  
  I suggest one of the first things you try if this does not converge is to batch up the updates and only perform them every e.g. 100 steps. Again, this is *possible* using the S&B update rule, and something like PyTorch, but things will be easier if you go with the grain and re-frame the updates as supervised learning "inside" the RL mechanism.
  
  I am not sure how well SARSA will cope with Lunar Lander. I have solved it using DQN, which is similar, but has the advantage of re-using older off-policy experience, sampling from it randomly, which SARSA cannot do. To correct for SARSA's weakness there, you may need quite large batches to avoid too much correlation between experience. A common pattern with "Deep SARSA" is to sample multiple environments at once under the current policy, to build a large batch (you see this in A2C and other on-policy policy gradient methods too).
  
  
  ---
  
  