Body: <p>Can you please elucidate the math behind the update rule for the critic? I've seen in other places that just a squared distance of  <span class="math-container">$R + \hat{v}(S', w) - \hat{v}(S,w)$</span> is used, but Sutton suggests an update rule (and the math behind) that is beyond my understanding?</p>
  
  <p><a href="https://i.sstatic.net/h5t1V.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/h5t1V.png" alt="enter image description here"></a></p>
  
  <p>Also, why do we need <span class="math-container">$I$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-23T15:32:48.747Z
Comments.Text: In the pseudocode you provide, what you call the "squared distance" is also present. So, what's exactly confusing you, apart from the need for $I$ in the pseudocode?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-23T20:39:11.070Z
Comments.Text: Please, fix your last comment to include the MathJax equations between the dollar symbols `$`.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-23T20:42:42.903Z
Comments.Text: deleted the previous comment because could not edit it. instead of $w + \alpha \nabla (R + \hat{v}(S',w) - \hat{v}(S,w))^2$ why does the update of w look like $w + \alpha (R + \hat{v}(S',w) - \hat{v}(S,w)) \nabla \hat{v}(S,w)$? I think i don't get the math behind derivation – d56 12 mins ago   Delete
Comments.UserDisplayName: d56
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-28T18:45:47.047Z
Comments.Text: @nbro I think i get it. It is just an application of the derivative chain rule. First, we bring down the exponent, and then take derivative of the second term (we treat the first term as the constant). The questions remaing are: why do we treat the first term ($R + \hat{v}(S',w))$ as a constant although it is being estimated by the same neural network. Where does the exponent disappear to (is it being reduced by the learning rate $\alpha^\theta$)? What is the term $I$?
Comments.UserDisplayName: d56
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-28T19:14:57.110Z
Comments.Text: Your ideas seem reasonable, but where did you see that the loss must be the square of that thing? If the gradient is taken with respect to the parameters $w$, then $R$ is a constant, but $\hat{v}(S', w)$ should not be a constant (as you say). The learning rate could indeed absorb the $2$. The $I$ is used in the update step of the policy parameters and it changes by the factor $\gamma$. I would need to review that part of the book to help you more. Maybe later.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-28T21:12:18.150Z
Comments.Text: Regarding the squaring, it is mentioned in the Sergey Levine's lectures on RL: http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-6.pdf
Comments.UserDisplayName: d56
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-28T22:43:28.200Z
Comments.Text: Which slide exactly are you referring to?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-28T23:05:57.430Z
Comments.Text: Take a look at slides 13-14
Comments.UserDisplayName: d56
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-29T09:08:19.250Z
Comments.Text: This is a semi-gradient algorithm, meaning it only takes derivative with respect to $v(S, \mathbf w)$ and disregards derivative with respect to $v(S', \mathbf w)$.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-31T17:09:53.930Z
Comments.Text: @Brale_ Cool thanks! Do you know of a resource that expains the semi-gradient?
Comments.UserDisplayName: d56
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-31T18:23:26.743Z
Comments.Text: Book by Sutton and Barto: Reinforcement Learning: An Introduction
Comments.UserDisplayName: Brale
Tags: <reinforcement-learning><math><actor-critic-methods>
Title: How does the update rule for the one-step actor-critic method work?
UserDisplayName: d56
ThreadText: # How does the update rule for the one-step actor-critic method work?
  
  (Asked by: username_0 on 2020-01-22)
  
  Can you please elucidate the math behind the update rule for the critic? I've seen in other places that just a squared distance of $R + \hat{v}(S', w) - \hat{v}(S,w)$ is used, but Sutton suggests an update rule (and the math behind) that is beyond my understanding?
  
  [![enter image description here](https://i.sstatic.net/h5t1V.png)](https://i.sstatic.net/h5t1V.png)
  
  Also, why do we need $I$?
  
  **Tags:** reinforcement-learning, math, actor-critic-methods
  
  > **Comments:**
  > **username_1**: In the pseudocode you provide, what you call the "squared distance" is also present. So, what's exactly confusing you, apart from the need for $I$ in the pseudocode?  
  > **username_1**: Please, fix your last comment to include the MathJax equations between the dollar symbols `$`.  
  > **username_0**: deleted the previous comment because could not edit it. instead of $w + \alpha \nabla (R + \hat{v}(S',w) - \hat{v}(S,w))^2$ why does the update of w look like $w + \alpha (R + \hat{v}(S',w) - \hat{v}(S,w)) \nabla \hat{v}(S,w)$? I think i don't get the math behind derivation – d56 12 mins ago   Delete  
  > **username_0**: @username_1 I think i get it. It is just an application of the derivative chain rule. First, we bring down the exponent, and then take derivative of the second term (we treat the first term as the constant). The questions remaing are: why do we treat the first term ($R + \hat{v}(S',w))$ as a constant although it is being estimated by the same neural network. Where does the exponent disappear to (is it being reduced by the learning rate $\alpha^\theta$)? What is the term $I$?  
  > **username_1**: Your ideas seem reasonable, but where did you see that the loss must be the square of that thing? If the gradient is taken with respect to the parameters $w$, then $R$ is a constant, but $\hat{v}(S', w)$ should not be a constant (as you say). The learning rate could indeed absorb the $2$. The $I$ is used in the update step of the policy parameters and it changes by the factor $\gamma$. I would need to review that part of the book to help you more. Maybe later.  
  > **username_0**: Regarding the squaring, it is mentioned in the Sergey Levine's lectures on RL: http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-6.pdf  
  > **username_1**: Which slide exactly are you referring to?  
  > **username_0**: Take a look at slides 13-14  
  > **username_2**: This is a semi-gradient algorithm, meaning it only takes derivative with respect to $v(S, \mathbf w)$ and disregards derivative with respect to $v(S', \mathbf w)$.  
  > **username_0**: @Brale_ Cool thanks! Do you know of a resource that expains the semi-gradient?  
  > **username_2**: Book by Sutton and Barto: Reinforcement Learning: An Introduction  
  
  ---
  
  ## 0 answers
  
  