Answers.Body: <p>It is our &quot;current&quot; target. We assume that the value we get now is at least a closer approximation to the &quot;true&quot; target.</p>
  <p>We're not so much moving towards a wrong value as we are moving away from a more wrong value.</p>
  <p>Of course, it is all base on random trials, so saying anything definite (such as: &quot;we are guaranteed to improve at each step&quot;) is hard to show without working probabilistically. The <em>expectation</em> of the error of the value function (as compared to the true value function) will decrease, that is all we can say.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T19:16:01.940Z
Answers.Text: Maybe to clarify your answer, you could add the formula of the "target" and why it represents an estimate of the "true" target. Anyway, welcome to Artificial Intelligence SE :)
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:02:14.273Z
Answers.Text: This isn’t clear to me, why is it closer to the true target?
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:16:03.457Z
Answers.Text: I understand that we use a sample return as the target, but what I don’t get is how it moves closer to the true target, the sample return doesn’t get closer to the target, the mean of those samples is what gets Closer which is our value function, so why do we then call the it target
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T23:13:19.490Z
Answers.Text: The new value is equal to $(1 - \alpha)(\mathrm{old estimate})  + \alpha(\mathrm{new estimate})$. We're gradually "forgetting" the old estimate with a decay rate of $(1 - \alpha)$.
Answers.UserDisplayName: Robby Goetschalckx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T17:46:53.627Z
Answers.LastActivityDate: 2020-08-28T17:46:53.627Z
Answers.UserDisplayName: Robby Goetschalckx
Answers.Body: <p>It would be heplful for me if you specify the section and page number of the Sutton's book.
  But as far as I understand your question I will try explain this.
  Think of TD update. The sample contains <span class="math-container">$(s_t,a_t,r_{t+1},s_{t+1})$</span>. Using incremental update we can write:
  <span class="math-container">$$ v_{t}(s) = \frac{1}{t} \sum_{j=1}^{t}(r_{t+1} + \gamma v_{s_{t+1}})$$</span>
  <span class="math-container">$$ v_{t}(s) = v_{t-1}(s) + \alpha (r_{t+1} + \gamma v_{t-1}(s_{t+1}) - v_{t-1}(s_t))$$</span>
  We are calling this <span class="math-container">$r_{t+1} + \gamma v_{t-1}(s_{t+1})$</span> as the TD target. From the above equation you can already see that <span class="math-container">$r_{t+1} + \gamma v_{t-1}(s_{t+1})$</span> is actually the unbaised estimate for <span class="math-container">$v(s)$</span>. We are calling <span class="math-container">$r_{t+1} + \gamma v_{t-1}(s_{t+1})$</span> an unbiased estimate since <span class="math-container">$E[r_{t+1} + \gamma v_{t-1}(s_{t+1})] = v_t(s_t)$</span>. That means expectation over <span class="math-container">$r_{t+1} + \gamma v_{t-1}(s_{t+1})$</span> lead us to true state value function, <span class="math-container">$v_t(s)$</span>.</p>
  <p>For the monte carlo update same explain will be applied. I hope that this answer your question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T19:59:37.953Z
Answers.Text: I understand the unbiased estimate part, which means by law of large numbers it’ll eventually converge to the optimal value, what I don’t get is why it’s called the target , normally in supervised learning it’s the actual target value, not a sample return , we’re using a sample return as our target TD, which I do not understand
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:08:19.703Z
Answers.Text: In RL targets are not fixed like supervised learning. Here we have moving targets. It is called target in the sense that it is an unbiased estimate of the true value.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:20:23.170Z
Answers.Text: Ok, so why do people say the target will eventually converge, isn’t it the mean of the target that actually converges to the optimal q value, I mean that’s what unbiased estimate means
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:34:32.743Z
Answers.Text: People say this in the sense that initially what you get $r_{t+1} + \gamma v(s_{t+1})$ is not actually an unbiased estimate. We randomly initally all $v(s) = 0$. Then gradually they started to update. Initially, those values do not represent the unbiased estimate. But as time passes those estimate get close to the true unbiased estimate and converges to the unbiased estimate. For this reason they say target will eventually converge to the ubiased estimate.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:42:59.710Z
Answers.Text: Chill I think I get what you mean, we use the  estimate r + yV(s’), the unbiased estimate is the formula with the optimal V(s) ie R +yV*(s’), but since we use an estimate instead of the actual value ,so as our V(s) estimate converges to the V*(s) we can then say the target now becomes the unbiased estimate of our true value function ?
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:48:50.530Z
Answers.Text: For TD learning or monte carlo learning we are not finding optimal $v(s)$. We are just evaluating the policy, $\pi$,
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:51:42.657Z
Answers.Text: What I mean is the V(s) we initially start with is wrong, so it’s not the unbiased estimate, but as our V(s) converges to the true value, the target now converges to the unbiased estimate
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:51:52.417Z
Answers.Text: When the target  converges after that we can say the target is an unbiased estimate of the true value
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:53:50.390Z
Answers.Text: When the target converges to the true value, then you can get the true $v(s)$ for each state.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:54:10.077Z
Answers.Text: That makes sense, because our target is dependent on a value function that is initially wrong, it isn’t really the unbiased estimate, but as time goes on and we interact more, our Value function becomes more accurate , and so out TD target converges to the unbiased estimate
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T20:55:44.257Z
Answers.Text: Yes,  you are right.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T21:10:04.840Z
Answers.Text: Finally, thank you so much, I appreciate your time and effort
Answers.UserDisplayName: Chukwudi Ogbonna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T18:10:20.437Z
Answers.LastActivityDate: 2020-08-28T19:40:28.757Z
Answers.LastEditDate: 2020-08-28T19:40:28.757Z
Answers.UserDisplayName: Swakshar Deb
Body: <p>I was going through Sutton's book and, using sample-based learning for estimating the expectations, we have this formula</p>
  <p><span class="math-container">$$
  \text{new estimate} = \text{old estimate} + \alpha(\text{target} - \text{old estimate})
  $$</span></p>
  <p>What I don't quite understand is why it's called the target, because since it's the sample, it’s not the actual target value, so why are we moving towards a wrong value?</p>
  
Tags: <machine-learning><reinforcement-learning><terminology><monte-carlo-methods><temporal-difference-methods>
Title: Why is the target called "target" in Monte Carlo and TD learning if it is not the true target?
UserDisplayName: Chukwudi Ogbonna
ThreadText: # Why is the target called "target" in Monte Carlo and TD learning if it is not the true target?
  
  (Asked by: username_0 on 2020-08-28)
  
  I was going through Sutton's book and, using sample-based learning for estimating the expectations, we have this formula
  
  $$
  \text{new estimate} = \text{old estimate} + \alpha(\text{target} - \text{old estimate})
  $$
  
  What I don't quite understand is why it's called the target, because since it's the sample, it’s not the actual target value, so why are we moving towards a wrong value?
  
  **Tags:** machine-learning, reinforcement-learning, terminology, monte-carlo-methods, temporal-difference-methods
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-08-28. Score: 6
  
  It is our "current" target. We assume that the value we get now is at least a closer approximation to the "true" target.
  
  We're not so much moving towards a wrong value as we are moving away from a more wrong value.
  
  Of course, it is all base on random trials, so saying anything definite (such as: "we are guaranteed to improve at each step") is hard to show without working probabilistically. The *expectation* of the error of the value function (as compared to the true value function) will decrease, that is all we can say.
  
  > **Comments:**
  > **username_2**: Maybe to clarify your answer, you could add the formula of the "target" and why it represents an estimate of the "true" target. Anyway, welcome to Artificial Intelligence SE :)  
  > **username_0**: This isn’t clear to me, why is it closer to the true target?  
  > **username_0**: I understand that we use a sample return as the target, but what I don’t get is how it moves closer to the true target, the sample return doesn’t get closer to the target, the mean of those samples is what gets Closer which is our value function, so why do we then call the it target  
  > **username_1**: The new value is equal to $(1 - \alpha)(\mathrm{old estimate})  + \alpha(\mathrm{new estimate})$. We're gradually "forgetting" the old estimate with a decay rate of $(1 - \alpha)$.  
  
  ---
  
  ### Answer by username_3 on 2020-08-28. Score: 0
  
  It would be heplful for me if you specify the section and page number of the Sutton's book.
  But as far as I understand your question I will try explain this.
  Think of TD update. The sample contains $(s\_t,a\_t,r\_{t+1},s\_{t+1})$. Using incremental update we can write:
  $$ v\_{t}(s) = \frac{1}{t} \sum\_{j=1}^{t}(r\_{t+1} + \gamma v\_{s\_{t+1}})$$
  $$ v\_{t}(s) = v\_{t-1}(s) + \alpha (r\_{t+1} + \gamma v\_{t-1}(s\_{t+1}) - v\_{t-1}(s\_t))$$
  We are calling this $r\_{t+1} + \gamma v\_{t-1}(s\_{t+1})$ as the TD target. From the above equation you can already see that $r\_{t+1} + \gamma v\_{t-1}(s\_{t+1})$ is actually the unbaised estimate for $v(s)$. We are calling $r\_{t+1} + \gamma v\_{t-1}(s\_{t+1})$ an unbiased estimate since $E[r\_{t+1} + \gamma v\_{t-1}(s\_{t+1})] = v\_t(s\_t)$. That means expectation over $r\_{t+1} + \gamma v\_{t-1}(s\_{t+1})$ lead us to true state value function, $v\_t(s)$.
  
  For the monte carlo update same explain will be applied. I hope that this answer your question.
  
  > **Comments:**
  > **username_0**: I understand the unbiased estimate part, which means by law of large numbers it’ll eventually converge to the optimal value, what I don’t get is why it’s called the target , normally in supervised learning it’s the actual target value, not a sample return , we’re using a sample return as our target TD, which I do not understand  
  > **username_3**: In RL targets are not fixed like supervised learning. Here we have moving targets. It is called target in the sense that it is an unbiased estimate of the true value.  
  > **username_0**: Ok, so why do people say the target will eventually converge, isn’t it the mean of the target that actually converges to the optimal q value, I mean that’s what unbiased estimate means  
  > **username_3**: People say this in the sense that initially what you get $r_{t+1} + \gamma v(s_{t+1})$ is not actually an unbiased estimate. We randomly initally all $v(s) = 0$. Then gradually they started to update. Initially, those values do not represent the unbiased estimate. But as time passes those estimate get close to the true unbiased estimate and converges to the unbiased estimate. For this reason they say target will eventually converge to the ubiased estimate.  
  > **username_0**: Chill I think I get what you mean, we use the  estimate r + yV(s’), the unbiased estimate is the formula with the optimal V(s) ie R +yV*(s’), but since we use an estimate instead of the actual value ,so as our V(s) estimate converges to the V*(s) we can then say the target now becomes the unbiased estimate of our true value function ?  
  > **username_3**: For TD learning or monte carlo learning we are not finding optimal $v(s)$. We are just evaluating the policy, $\pi$,  
  > **username_0**: What I mean is the V(s) we initially start with is wrong, so it’s not the unbiased estimate, but as our V(s) converges to the true value, the target now converges to the unbiased estimate  
  > **username_3**: When the target  converges after that we can say the target is an unbiased estimate of the true value  
  > **username_3**: When the target converges to the true value, then you can get the true $v(s)$ for each state.  
  > **username_0**: That makes sense, because our target is dependent on a value function that is initially wrong, it isn’t really the unbiased estimate, but as time goes on and we interact more, our Value function becomes more accurate , and so out TD target converges to the unbiased estimate  
  > **username_3**: Yes,  you are right.  
  > **username_0**: Finally, thank you so much, I appreciate your time and effort  
  
  ---
  
  