Answers.Body: <p>The value <span class="math-container">$\delta$</span> is <em>already</em> representing a derivative equivalent to derivative of MSE loss for the difference between observed and predicted return. Multiplying it by the gradient of <span class="math-container">$\hat{v}$</span> to get a gradient for the weights is applying the chain rule.</p>
  <p>Your equations would work and be compatible with the book, if instead of trying to redefine <span class="math-container">$\delta$</span>, you created the implied loss function <span class="math-container">$L = \frac{1}{2}(\hat{v}(S_t, w) - G)^2$</span> and set <span class="math-container">$\delta = \frac{d L}{d \hat{v}}$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-11T10:50:35.473Z
Answers.LastActivityDate: 2024-02-11T12:40:07.800Z
Answers.LastEditDate: 2024-02-11T12:40:07.800Z
Answers.UserDisplayName: Neil Slater
Body: <p>I was looking at the algorithm for REINFORCE with baseline from the Book 'Introduction to Reinforcement Learning' from Sutton:</p>
  <p><a href="https://i.sstatic.net/R577y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/R577y.png" alt="enter image description here" /></a></p>
  <p>I do not quite understand the update rule for <span class="math-container">$w$</span>:</p>
  <p><span class="math-container">$w = w  + \alpha \delta \nabla \hat{v}(S_t, w) $</span></p>
  <p>If I understand correctly, this would be the update rule, if <span class="math-container">$ \delta$</span> would be the MSE:</p>
  <p><span class="math-container">$\delta = (G-\hat{v}(S_t, w))^2$</span></p>
  <p><span class="math-container">$\nabla\delta = -2(G-\hat{v}(S_t, w))\nabla \hat{v}(S_t, w)$</span></p>
  <p>which would give the same update as shown in the algorithm, when we include the <span class="math-container">$2$</span> into the learning rate <span class="math-container">$\alpha$</span></p>
  <p><span class="math-container">$w = w  + \alpha \delta \nabla \hat{v}(S_t, w) $</span></p>
  <p>However, in the algorithm they do not use the MSE for <span class="math-container">$\delta$</span>, but simply</p>
  <p><span class="math-container">$\delta = G-\hat{v}(S_t, w)$</span></p>
  <p><span class="math-container">$\nabla\delta = - \nabla \hat{v}(S_t, w)$</span></p>
  <p>which would yield a different update rule (compared to the one in the algorithm)</p>
  <p><span class="math-container">$w = w  + \alpha \nabla \hat{v}(S_t, w) $</span></p>
  <p>What am I missing?</p>
  
Tags: <reinforcement-learning><gradient-descent><reinforce>
Title: REINFORCE with Baseline update rule
UserDisplayName: kklaw
ThreadText: # REINFORCE with Baseline update rule
  
  (Asked by: username_0 on 2024-02-11)
  
  I was looking at the algorithm for REINFORCE with baseline from the Book 'Introduction to Reinforcement Learning' from Sutton:
  
  [![enter image description here](https://i.sstatic.net/R577y.png)](https://i.sstatic.net/R577y.png)
  
  I do not quite understand the update rule for $w$:
  
  $w = w + \alpha \delta \nabla \hat{v}(S\_t, w) $
  
  If I understand correctly, this would be the update rule, if $ \delta$ would be the MSE:
  
  $\delta = (G-\hat{v}(S\_t, w))^2$
  
  $\nabla\delta = -2(G-\hat{v}(S\_t, w))\nabla \hat{v}(S\_t, w)$
  
  which would give the same update as shown in the algorithm, when we include the $2$ into the learning rate $\alpha$
  
  $w = w + \alpha \delta \nabla \hat{v}(S\_t, w) $
  
  However, in the algorithm they do not use the MSE for $\delta$, but simply
  
  $\delta = G-\hat{v}(S\_t, w)$
  
  $\nabla\delta = - \nabla \hat{v}(S\_t, w)$
  
  which would yield a different update rule (compared to the one in the algorithm)
  
  $w = w + \alpha \nabla \hat{v}(S\_t, w) $
  
  What am I missing?
  
  **Tags:** reinforcement-learning, gradient-descent, reinforce
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-11. Score: 3 | Accepted answer
  
  The value $\delta$ is *already* representing a derivative equivalent to derivative of MSE loss for the difference between observed and predicted return. Multiplying it by the gradient of $\hat{v}$ to get a gradient for the weights is applying the chain rule.
  
  Your equations would work and be compatible with the book, if instead of trying to redefine $\delta$, you created the implied loss function $L = \frac{1}{2}(\hat{v}(S\_t, w) - G)^2$ and set $\delta = \frac{d L}{d \hat{v}}$
  
  
  ---
  
  