Answers.Body: <p>This is formally known as a <em>semi-gradient method</em>.</p>
  <p>What we would like to do is to minimize <span class="math-container">$\big(v(S) - \hat v(S, w)\big)^2$</span>, where   <span class="math-container">$v(S)$</span> is the true value function. This would give the gradient descent update
  <span class="math-container">\begin{align*} 
  w \leftarrow w + \alpha[v(S) - \hat v(S, w)]\nabla \hat v(S, w) . \tag{1}
  \end{align*}</span>
  Of course we don't have access to <span class="math-container">$v(S)$</span>. So instead, we could use the monte carlo return (the observed, discounted, episodic return). The other option is to use a bootstraped estimate of <span class="math-container">$v(S)$</span>, e.g.  the estimate <span class="math-container">$r + \gamma \hat v(S', w)$</span>, which would give the update</p>
  <p><span class="math-container">\begin{align*} 
  w \leftarrow w + \alpha[r + \gamma \hat v(S', w) - \hat v(S, w)]\nabla \hat v(S, w) \tag{2}
  \end{align*}</span></p>
  <p>As you correctly point out, Eq. 2 is no longer a true gradient descent method. To directly quote Sutton and Barto, section 9.3 page 165,</p>
  <blockquote>
  <p>This step [Eq. 1] would not be valid if a bootstrapping estimate were
  used in place of v(S). Bootstrapping methods are not in fact instances
  of true gradient descent (Barnard, 1993). They take into account the
  effect of changing the weight vector w on the estimate, but ignore its
  effect on the target. They include only a part of the gradient and,
  accordingly, we call them semi-gradient methods.</p>
  </blockquote>
  <p>The 'estimate' here is refering to <span class="math-container">$\hat v(S, w)$</span>, whereas the 'target' is <span class="math-container">$v(S)$</span> (or its approximation). Indeed the parameter update you give would be the true gradient descent update for minimizing <span class="math-container">$\big(r + \gamma \hat v(S', w) - \hat v(S, w)\big)^2$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-22T17:06:00.797Z
Answers.LastActivityDate: 2021-12-22T17:06:00.797Z
Answers.UserDisplayName: Taw
Body: <p>I am reading <a href="http://web.archive.org/web/20210506214610/https://web.stanford.edu/class/cs234/slides/lecture5.pdf" rel="nofollow noreferrer">these slides</a>. On page 38, the update for the parameters for the linear function approximation of TD(0) is given. I have a doubt regarding this.
  The cost function (RMSE) is given on page 37.</p>
  <p>My doubt is: why is the gradient of <span class="math-container">$\hat v(S^{\prime}, \mathbf w)$</span> with respect to parameters <span class="math-container">$w$</span> not considered?</p>
  <p>I think the parameter update should be:
  <span class="math-container">$$\mathbf w \leftarrow \mathbf w +\alpha [R + \gamma \hat v(S', \mathbf w) - \hat v(S, \mathbf w)] (\nabla \hat v(S, \mathbf w)- \gamma \nabla \hat v(S', \mathbf w))$$</span>
  Instead in the material it is given as:-
  <span class="math-container">$$\mathbf w \leftarrow \mathbf w +\alpha [R + \gamma \hat v(S', \mathbf w) - \hat v(S, \mathbf w)] \nabla \hat v(S, \mathbf w)$$</span></p>
  <p>Could someone please explain?</p>
  
Tags: <reinforcement-learning><value-functions><function-approximation><weights><temporal-difference-methods>
Title: In TD(0) with linear function approximation, why is the gradient of $\hat v(S^{\prime}, \mathbf w)$ wrt parameters $\mathbf w$ not considered?
UserDisplayName: A Yoghes
ThreadText: # In TD(0) with linear function approximation, why is the gradient of $\hat v(S^{\prime}, \mathbf w)$ wrt parameters $\mathbf w$ not considered?
  
  (Asked by: username_0 on 2021-11-14)
  
  I am reading [these slides](http://web.archive.org/web/20210506214610/https://web.stanford.edu/class/cs234/slides/lecture5.pdf). On page 38, the update for the parameters for the linear function approximation of TD(0) is given. I have a doubt regarding this.
  The cost function (RMSE) is given on page 37.
  
  My doubt is: why is the gradient of $\hat v(S^{\prime}, \mathbf w)$ with respect to parameters $w$ not considered?
  
  I think the parameter update should be:
  $$\mathbf w \leftarrow \mathbf w +\alpha [R + \gamma \hat v(S', \mathbf w) - \hat v(S, \mathbf w)] (\nabla \hat v(S, \mathbf w)- \gamma \nabla \hat v(S', \mathbf w))$$
  Instead in the material it is given as:-
  $$\mathbf w \leftarrow \mathbf w +\alpha [R + \gamma \hat v(S', \mathbf w) - \hat v(S, \mathbf w)] \nabla \hat v(S, \mathbf w)$$
  
  Could someone please explain?
  
  **Tags:** reinforcement-learning, value-functions, function-approximation, weights, temporal-difference-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-22. Score: 2 | Accepted answer
  
  This is formally known as a *semi-gradient method*.
  
  What we would like to do is to minimize $\big(v(S) - \hat v(S, w)\big)^2$, where $v(S)$ is the true value function. This would give the gradient descent update
  \begin{align\*}
  w \leftarrow w + \alpha[v(S) - \hat v(S, w)]\nabla \hat v(S, w) . \tag{1}
  \end{align\*}
  Of course we don't have access to $v(S)$. So instead, we could use the monte carlo return (the observed, discounted, episodic return). The other option is to use a bootstraped estimate of $v(S)$, e.g. the estimate $r + \gamma \hat v(S', w)$, which would give the update
  
  \begin{align\*}
  w \leftarrow w + \alpha[r + \gamma \hat v(S', w) - \hat v(S, w)]\nabla \hat v(S, w) \tag{2}
  \end{align\*}
  
  As you correctly point out, Eq. 2 is no longer a true gradient descent method. To directly quote Sutton and Barto, section 9.3 page 165,
  
  > This step [Eq. 1] would not be valid if a bootstrapping estimate were
  > used in place of v(S). Bootstrapping methods are not in fact instances
  > of true gradient descent (Barnard, 1993). They take into account the
  > effect of changing the weight vector w on the estimate, but ignore its
  > effect on the target. They include only a part of the gradient and,
  > accordingly, we call them semi-gradient methods.
  
  The 'estimate' here is refering to $\hat v(S, w)$, whereas the 'target' is $v(S)$ (or its approximation). Indeed the parameter update you give would be the true gradient descent update for minimizing $\big(r + \gamma \hat v(S', w) - \hat v(S, w)\big)^2$
  
  
  ---
  
  