<p>Can you please elucidate the math behind the update rule for the critic? I've seen in other places that just a squared distance of  <span class="math-container">$R + \hat{v}(S', w) - \hat{v}(S,w)$</span> is used, but Sutton suggests an update rule (and the math behind) that is beyond my understanding?</p>

<p><a href="https://i.sstatic.net/h5t1V.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/h5t1V.png" alt="enter image description here"></a></p>

<p>Also, why do we need <span class="math-container">$I$</span>?</p>

CC BY-SA 4.0
2020-01-23T15:32:48.747Z
In the pseudocode you provide, what you call the "squared distance" is also present. So, what's exactly confusing you, apart from the need for $I$ in the pseudocode?
nbro
CC BY-SA 4.0
2020-01-23T20:39:11.070Z
Please, fix your last comment to include the MathJax equations between the dollar symbols `$`.
nbro
CC BY-SA 4.0
2020-01-23T20:42:42.903Z
deleted the previous comment because could not edit it. instead of $w + \alpha \nabla (R + \hat{v}(S',w) - \hat{v}(S,w))^2$ why does the update of w look like $w + \alpha (R + \hat{v}(S',w) - \hat{v}(S,w)) \nabla \hat{v}(S,w)$? I think i don't get the math behind derivation – d56 12 mins ago   Delete
d56
CC BY-SA 4.0
2020-01-28T18:45:47.047Z
@nbro I think i get it. It is just an application of the derivative chain rule. First, we bring down the exponent, and then take derivative of the second term (we treat the first term as the constant). The questions remaing are: why do we treat the first term ($R + \hat{v}(S',w))$ as a constant although it is being estimated by the same neural network. Where does the exponent disappear to (is it being reduced by the learning rate $\alpha^\theta$)? What is the term $I$?
d56
CC BY-SA 4.0
2020-01-28T19:14:57.110Z
Your ideas seem reasonable, but where did you see that the loss must be the square of that thing? If the gradient is taken with respect to the parameters $w$, then $R$ is a constant, but $\hat{v}(S', w)$ should not be a constant (as you say). The learning rate could indeed absorb the $2$. The $I$ is used in the update step of the policy parameters and it changes by the factor $\gamma$. I would need to review that part of the book to help you more. Maybe later.
nbro
CC BY-SA 4.0
2020-01-28T21:12:18.150Z
Regarding the squaring, it is mentioned in the Sergey Levine's lectures on RL: http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-6.pdf
d56
CC BY-SA 4.0
2020-01-28T22:43:28.200Z
Which slide exactly are you referring to?
nbro
CC BY-SA 4.0
2020-01-28T23:05:57.430Z
Take a look at slides 13-14
d56
CC BY-SA 4.0
2020-01-29T09:08:19.250Z
This is a semi-gradient algorithm, meaning it only takes derivative with respect to $v(S, \mathbf w)$ and disregards derivative with respect to $v(S', \mathbf w)$.
Brale
CC BY-SA 4.0
2020-01-31T17:09:53.930Z
@Brale_ Cool thanks! Do you know of a resource that expains the semi-gradient?
d56
CC BY-SA 4.0
2020-01-31T18:23:26.743Z
Book by Sutton and Barto: Reinforcement Learning: An Introduction
Brale
<reinforcement-learning><math><actor-critic-methods>
How does the update rule for the one-step actor-critic method work?
d56
# How does the update rule for the one-step actor-critic method work?

(Asked by: username_0 on 2020-01-22)

Can you please elucidate the math behind the update rule for the critic? I've seen in other places that just a squared distance of $R + \hat{v}(S', w) - \hat{v}(S,w)$ is used, but Sutton suggests an update rule (and the math behind) that is beyond my understanding?

[![enter image description here](https://i.sstatic.net/h5t1V.png)](https://i.sstatic.net/h5t1V.png)

Also, why do we need $I$?

**Tags:** reinforcement-learning, math, actor-critic-methods

> **Comments:**
> **username_1**: In the pseudocode you provide, what you call the "squared distance" is also present. So, what's exactly confusing you, apart from the need for $I$ in the pseudocode?  
> **username_1**: Please, fix your last comment to include the MathJax equations between the dollar symbols `$`.  
> **username_0**: deleted the previous comment because could not edit it. instead of $w + \alpha \nabla (R + \hat{v}(S',w) - \hat{v}(S,w))^2$ why does the update of w look like $w + \alpha (R + \hat{v}(S',w) - \hat{v}(S,w)) \nabla \hat{v}(S,w)$? I think i don't get the math behind derivation – username_0 12 mins ago   Delete  
> **username_0**: @username_1 I think i get it. It is just an application of the derivative chain rule. First, we bring down the exponent, and then take derivative of the second term (we treat the first term as the constant). The questions remaing are: why do we treat the first term ($R + \hat{v}(S',w))$ as a constant although it is being estimated by the same neural network. Where does the exponent disappear to (is it being reduced by the learning rate $\alpha^\theta$)? What is the term $I$?  
> **username_1**: Your ideas seem reasonable, but where did you see that the loss must be the square of that thing? If the gradient is taken with respect to the parameters $w$, then $R$ is a constant, but $\hat{v}(S', w)$ should not be a constant (as you say). The learning rate could indeed absorb the $2$. The $I$ is used in the update step of the policy parameters and it changes by the factor $\gamma$. I would need to review that part of the book to help you more. Maybe later.  
> **username_0**: Regarding the squaring, it is mentioned in the Sergey Levine's lectures on RL: http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-6.pdf  
> **username_1**: Which slide exactly are you referring to?  
> **username_0**: Take a look at slides 13-14  
> **username_2**: This is a semi-gradient algorithm, meaning it only takes derivative with respect to $v(S, \mathbf w)$ and disregards derivative with respect to $v(S', \mathbf w)$.  
> **username_0**: @username_2_ Cool thanks! Do you know of a resource that expains the semi-gradient?  
> **username_2**: Book by Sutton and Barto: Reinforcement Learning: An Introduction  

---

## 0 answers

