Answers.Body: <ol>
  <li><p><span class="math-container">$\mu(s)$</span> is not in equation (9.4) because we are <em>assuming</em> that the examples by which we update our parameter <span class="math-container">$w$</span>, i.e. the frequency of which we will observe the states during online training, is the same. That is, it is a constant with respect to <span class="math-container">$w$</span> and since we are differentiating it can be somewhat disregarded as a constant of proportionality -- it essentially can be 'absorbed' by <span class="math-container">$\alpha$</span>.</p>
  </li>
  <li><p>The minus is there because we are performing gradient <em><strong><em>descent</em></strong></em>. For more information on this, see e.g. <a href="https://en.wikipedia.org/wiki/Gradient_descent" rel="nofollow noreferrer">the wikipedia page</a></p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-07T23:34:59.943Z
Answers.LastActivityDate: 2020-10-07T23:34:59.943Z
Answers.UserDisplayName: David
Body: <p>I am reading &quot;Reinforcement Learning: An Introduction (2nd edition)&quot; authored by Sutton and Barto. In Section 9, On-policy prediction with approximation, it first gives the mean squared value error objective function in (9.1):</p>
  <p><span class="math-container">$\bar{VE}(\boldsymbol{w}) = \sum_{s \in S} \mu(s)[v_{\pi}(s) - \hat{v}(s,\boldsymbol{w})]^2$</span>. (9.1)</p>
  <p><span class="math-container">$\boldsymbol{w}$</span> is a vector of the parameterized function <span class="math-container">$\hat{v}(s,\boldsymbol{w})$</span> that approximates the value function <span class="math-container">$v_{\pi}(s)$</span>. <span class="math-container">$\mu(s)$</span> is the fraction of time spent in <span class="math-container">$s$</span>, which measures the &quot;importance&quot; of state <span class="math-container">$s$</span> in <span class="math-container">$\bar{VE}(\boldsymbol{w})$</span>.</p>
  <p>In (9.4), it states an update rule of <span class="math-container">$\boldsymbol{w}$</span> by gradient descent:
  <span class="math-container">$\boldsymbol{w}_{t+1} = \boldsymbol{w} -\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$</span>. (9.4)</p>
  <p>I have two questions regarding (9.4).</p>
  <ol>
  <li>Why <span class="math-container">$\mu(s)$</span> is not in (9.4)?</li>
  <li>Why is it the &quot;minus&quot; instead of &quot;+&quot; in (9.4)? In other words, why is it <span class="math-container">$\boldsymbol{w} -\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$</span> instead of <span class="math-container">$\boldsymbol{w} +\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$</span>?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-07T20:42:27.267Z
Comments.Text: I think your equation number 9.4 is not correct? There is indeed + sign in Sutton's book and also $\nabla{V(s,w)}$ at the end.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-07T20:45:25.097Z
Comments.Text: They are giving same importance to all the states, so there is no $\mu(s)$
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-07T20:50:09.247Z
Comments.Text: @SwaksharDeb the + sign is in (9.5) because it expands the gradient equation. But why it is a - sign in the first place in (9.4)?
Comments.UserDisplayName: JYY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-07T20:51:24.177Z
Comments.Text: @SwaksharDeb So in (9.4) it "assumes" all the states are equally important, otherwise $\mu(s)$ will appear in the equation?
Comments.UserDisplayName: JYY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-07T20:52:10.060Z
Comments.Text: This is a stochastic gradient descent update. I think your problem is related to gradient descent.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-07T20:53:54.420Z
Comments.Text: Yes. for simplicity they are treating all-state with the same importance
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-07T21:17:22.543Z
Comments.Text: I now understand why it is a minus sign in (9.4). The objective is to minimize $VE(w)$, so it needs to move to the negative of the gradient direction.
Comments.UserDisplayName: JYY
Tags: <reinforcement-learning><gradient-descent><function-approximation><sutton-barto>
Title: Why is the fraction of time spent in state $s$, $\mu(s)$, not in the update rule of the parameters?
UserDisplayName: JYY
ThreadText: # Why is the fraction of time spent in state $s$, $\mu(s)$, not in the update rule of the parameters?
  
  (Asked by: username_0 on 2020-10-07)
  
  I am reading "Reinforcement Learning: An Introduction (2nd edition)" authored by Sutton and Barto. In Section 9, On-policy prediction with approximation, it first gives the mean squared value error objective function in (9.1):
  
  $\bar{VE}(\boldsymbol{w}) = \sum\_{s \in S} \mu(s)[v\_{\pi}(s) - \hat{v}(s,\boldsymbol{w})]^2$. (9.1)
  
  $\boldsymbol{w}$ is a vector of the parameterized function $\hat{v}(s,\boldsymbol{w})$ that approximates the value function $v\_{\pi}(s)$. $\mu(s)$ is the fraction of time spent in $s$, which measures the "importance" of state $s$ in $\bar{VE}(\boldsymbol{w})$.
  
  In (9.4), it states an update rule of $\boldsymbol{w}$ by gradient descent:
  $\boldsymbol{w}\_{t+1} = \boldsymbol{w} -\frac{1}{2}\alpha \nabla[v\_{\pi}(S\_t) - \hat{v}(S\_t,\boldsymbol{w})]^2$. (9.4)
  
  I have two questions regarding (9.4).
  
  1. Why $\mu(s)$ is not in (9.4)?
  2. Why is it the "minus" instead of "+" in (9.4)? In other words, why is it $\boldsymbol{w} -\frac{1}{2}\alpha \nabla[v\_{\pi}(S\_t) - \hat{v}(S\_t,\boldsymbol{w})]^2$ instead of $\boldsymbol{w} +\frac{1}{2}\alpha \nabla[v\_{\pi}(S\_t) - \hat{v}(S\_t,\boldsymbol{w})]^2$?
  
  **Tags:** reinforcement-learning, gradient-descent, function-approximation, sutton-barto
  
  > **Comments:**
  > **username_1**: I think your equation number 9.4 is not correct? There is indeed + sign in Sutton's book and also $\nabla{V(s,w)}$ at the end.  
  > **username_1**: They are giving same importance to all the states, so there is no $\mu(s)$  
  > **username_0**: @username_1 the + sign is in (9.5) because it expands the gradient equation. But why it is a - sign in the first place in (9.4)?  
  > **username_0**: @username_1 So in (9.4) it "assumes" all the states are equally important, otherwise $\mu(s)$ will appear in the equation?  
  > **username_1**: This is a stochastic gradient descent update. I think your problem is related to gradient descent.  
  > **username_1**: Yes. for simplicity they are treating all-state with the same importance  
  > **username_0**: I now understand why it is a minus sign in (9.4). The objective is to minimize $VE(w)$, so it needs to move to the negative of the gradient direction.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-10-07. Score: 2 | Accepted answer
  
  1. $\mu(s)$ is not in equation (9.4) because we are *assuming* that the examples by which we update our parameter $w$, i.e. the frequency of which we will observe the states during online training, is the same. That is, it is a constant with respect to $w$ and since we are differentiating it can be somewhat disregarded as a constant of proportionality -- it essentially can be 'absorbed' by $\alpha$.
  2. The minus is there because we are performing gradient ****descent****. For more information on this, see e.g. [the wikipedia page](https://en.wikipedia.org/wiki/Gradient_descent)
  
  
  ---
  
  