Answers.Body: <p>Say you have an estimate of <span class="math-container">$V(s_t)$</span> and you visited <span class="math-container">$s_t$</span> <span class="math-container">$\#s_t$</span> times, using the mean the update would be:
  <span class="math-container">$$
  V'(s_t) = \frac{\#s_t \cdot V(s_t) + G_t}{\#s_t+1}
  $$</span>
  which can also be seen as:
  <span class="math-container">$$
  V'(s_t) = \frac{\#s_t}{\#s_t+1} \cdot V(s_t) + \frac{1}{\#s_t+1}G_t
  $$</span>
  Where the sum of the two coefficients sum to 1, and they are strictly positive... in other words, is a linear combination of your current estimate, and your new estimate</p>
  <p>Now, check the constant step size formula:
  <span class="math-container">$$
  V'(s_t) = V(s_t) + \alpha[G_t - V(s_t)]\\
  V'(s_t) = V(s_t) + \alpha G_t - \alpha V(s_t)\\
  V'(s_t) = (1 - \alpha)V(s_t) + \alpha G_t 
  $$</span>
  And again, you can see that you are taking a linear combination of your old estimate and the new estimate</p>
  <p>Hence the similarity/being in-line</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-25T20:00:30.440Z
Answers.LastActivityDate: 2024-06-25T20:00:30.440Z
Answers.UserDisplayName: Alberto
Body: <p>This is my understanding thus far about Monte Carlo method for approximating value function:</p>
  <p>Instead of using a recursive Bellman equations and knowledge of environment dynamics, Monte Carlo methods use statistics to evaluate a value function. For a given policy π and starting state S_t, multiple episodes are generated. The return for each of these episodes is calculated and averaged out. If the number of samples is large enough, the calculated value converges to the expected return.</p>
  <p><a href="https://i.sstatic.net/Cw5bkrkA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Cw5bkrkA.png" alt="enter image description here" /></a></p>
  <p>But according to constant-α Monte Carlo, the value function is evaluated using following update rule:</p>
  <p><a href="https://i.sstatic.net/AJpP3D48.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AJpP3D48.png" alt="enter image description here" /></a></p>
  <p>where V(S_t) is current estimate of value function for state S_t and G_t is the return after time step t. I don't understand how this is in-line with the averaging method.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-27T11:12:14.730Z
Comments.Text: Can you please write the equations using mathjax and ask your specific question in the title? "Understanding x" is not really a question and very specific. Thanks.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><value-functions><monte-carlo-methods>
Title: Please help me understand constant-α Monte Carlo method
UserDisplayName: DeadAsDuck
ThreadText: # Please help me understand constant-α Monte Carlo method
  
  (Asked by: username_0 on 2024-06-25)
  
  This is my understanding thus far about Monte Carlo method for approximating value function:
  
  Instead of using a recursive Bellman equations and knowledge of environment dynamics, Monte Carlo methods use statistics to evaluate a value function. For a given policy π and starting state S\_t, multiple episodes are generated. The return for each of these episodes is calculated and averaged out. If the number of samples is large enough, the calculated value converges to the expected return.
  
  [![enter image description here](https://i.sstatic.net/Cw5bkrkA.png)](https://i.sstatic.net/Cw5bkrkA.png)
  
  But according to constant-α Monte Carlo, the value function is evaluated using following update rule:
  
  [![enter image description here](https://i.sstatic.net/AJpP3D48.png)](https://i.sstatic.net/AJpP3D48.png)
  
  where V(S\_t) is current estimate of value function for state S\_t and G\_t is the return after time step t. I don't understand how this is in-line with the averaging method.
  
  **Tags:** reinforcement-learning, value-functions, monte-carlo-methods
  
  > **Comments:**
  > **username_1**: Can you please write the equations using mathjax and ask your specific question in the title? "Understanding x" is not really a question and very specific. Thanks.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-06-25. Score: 2 | Accepted answer
  
  Say you have an estimate of $V(s\_t)$ and you visited $s\_t$ $\#s\_t$ times, using the mean the update would be:
  $$
  V'(s\_t) = \frac{\#s\_t \cdot V(s\_t) + G\_t}{\#s\_t+1}
  $$
  which can also be seen as:
  $$
  V'(s\_t) = \frac{\#s\_t}{\#s\_t+1} \cdot V(s\_t) + \frac{1}{\#s\_t+1}G\_t
  $$
  Where the sum of the two coefficients sum to 1, and they are strictly positive... in other words, is a linear combination of your current estimate, and your new estimate
  
  Now, check the constant step size formula:
  $$
  V'(s\_t) = V(s\_t) + \alpha[G\_t - V(s\_t)]\\
  V'(s\_t) = V(s\_t) + \alpha G\_t - \alpha V(s\_t)\\
  V'(s\_t) = (1 - \alpha)V(s\_t) + \alpha G\_t
  $$
  And again, you can see that you are taking a linear combination of your old estimate and the new estimate
  
  Hence the similarity/being in-line
  
  
  ---
  
  