<p>The value <span class="math-container">$\delta$</span> is <em>already</em> representing a derivative equivalent to derivative of MSE loss for the difference between observed and predicted return. Multiplying it by the gradient of <span class="math-container">$\hat{v}$</span> to get a gradient for the weights is applying the chain rule.</p>
<p>Your equations would work and be compatible with the book, if instead of trying to redefine <span class="math-container">$\delta$</span>, you created the implied loss function <span class="math-container">$L = \frac{1}{2}(\hat{v}(S_t, w) - G)^2$</span> and set <span class="math-container">$\delta = \frac{d L}{d \hat{v}}$</span></p>

CC BY-SA 4.0
2024-02-11T10:50:35.473Z
2024-02-11T12:40:07.800Z
2024-02-11T12:40:07.800Z
Neil Slater
<p>I was looking at the algorithm for REINFORCE with baseline from the Book 'Introduction to Reinforcement Learning' from Sutton:</p>
<p><a href="https://i.sstatic.net/R577y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/R577y.png" alt="enter image description here" /></a></p>
<p>I do not quite understand the update rule for <span class="math-container">$w$</span>:</p>
<p><span class="math-container">$w = w  + \alpha \delta \nabla \hat{v}(S_t, w) $</span></p>
<p>If I understand correctly, this would be the update rule, if <span class="math-container">$ \delta$</span> would be the MSE:</p>
<p><span class="math-container">$\delta = (G-\hat{v}(S_t, w))^2$</span></p>
<p><span class="math-container">$\nabla\delta = -2(G-\hat{v}(S_t, w))\nabla \hat{v}(S_t, w)$</span></p>
<p>which would give the same update as shown in the algorithm, when we include the <span class="math-container">$2$</span> into the learning rate <span class="math-container">$\alpha$</span></p>
<p><span class="math-container">$w = w  + \alpha \delta \nabla \hat{v}(S_t, w) $</span></p>
<p>However, in the algorithm they do not use the MSE for <span class="math-container">$\delta$</span>, but simply</p>
<p><span class="math-container">$\delta = G-\hat{v}(S_t, w)$</span></p>
<p><span class="math-container">$\nabla\delta = - \nabla \hat{v}(S_t, w)$</span></p>
<p>which would yield a different update rule (compared to the one in the algorithm)</p>
<p><span class="math-container">$w = w  + \alpha \nabla \hat{v}(S_t, w) $</span></p>
<p>What am I missing?</p>

<reinforcement-learning><gradient-descent><reinforce>
REINFORCE with Baseline update rule
kklaw
# REINFORCE with Baseline update rule

(Asked by: username_0 on 2024-02-11)

I was looking at the algorithm for REINFORCE with baseline from the Book 'Introduction to Reinforcement Learning' from Sutton:

[![enter image description here](https://i.sstatic.net/R577y.png)](https://i.sstatic.net/R577y.png)

I do not quite understand the update rule for $w$:

$w = w + \alpha \delta \nabla \hat{v}(S\_t, w) $

If I understand correctly, this would be the update rule, if $ \delta$ would be the MSE:

$\delta = (G-\hat{v}(S\_t, w))^2$

$\nabla\delta = -2(G-\hat{v}(S\_t, w))\nabla \hat{v}(S\_t, w)$

which would give the same update as shown in the algorithm, when we include the $2$ into the learning rate $\alpha$

$w = w + \alpha \delta \nabla \hat{v}(S\_t, w) $

However, in the algorithm they do not use the MSE for $\delta$, but simply

$\delta = G-\hat{v}(S\_t, w)$

$\nabla\delta = - \nabla \hat{v}(S\_t, w)$

which would yield a different update rule (compared to the one in the algorithm)

$w = w + \alpha \nabla \hat{v}(S\_t, w) $

What am I missing?

**Tags:** reinforcement-learning, gradient-descent, reinforce


---

## 1 answers

### Answer by username_1 on 2024-02-11. Score: 3 | Accepted answer

The value $\delta$ is *already* representing a derivative equivalent to derivative of MSE loss for the difference between observed and predicted return. Multiplying it by the gradient of $\hat{v}$ to get a gradient for the weights is applying the chain rule.

Your equations would work and be compatible with the book, if instead of trying to redefine $\delta$, you created the implied loss function $L = \frac{1}{2}(\hat{v}(S\_t, w) - G)^2$ and set $\delta = \frac{d L}{d \hat{v}}$


---

