Answers.Body: <p>I did not read the link but I am giving a standard derivation as it can be found in Sutton-Barto for instance.</p>
  <p>The average return formula can be reformulated:
  <span class="math-container">\begin{align}
  V_{n+1}(s) &amp;= \frac{1}{n}\sum_{i=1}^{n}R_{i} \\
             &amp;= \frac{1}{n}\left(R_{n} +\sum_{i=1}^{n-1}R_{i}   \right) \\
             &amp;= \frac{1}{n}\left(R_{n} +(n-1) \frac{1}{n-1}\sum_{i=1}^{n-1}R_{i}   \right) \\
             &amp;= \frac{1}{n}\left(R_{n} + (n-1) V_{n}(s) \right) \\
             &amp;= \frac{1}{n}\left(R_{n} + nV_{n}(s) - V_{n}(s) \right) \\
             &amp;= V_{n}(s) + \frac{1}{n} (R_{n} - V_{n}(s))
  \end{align}</span>
  where <span class="math-container">$V_{n}$</span> is the estimate of the value function after <span class="math-container">$n-1$</span> averages over the return of the state <span class="math-container">$s$</span>. We can interpret each <span class="math-container">$i$</span> as a single visit to <span class="math-container">$s$</span> or as a batch of visits of <span class="math-container">$s$</span> in which case <span class="math-container">$R_{i}$</span> would be the averaged return of this batch. Setting <span class="math-container">$\frac{1}{n} = \alpha$</span> directly results in the formula for incremental updates. Note that if we set <span class="math-container">$n = \alpha = 1$</span> we just set the value function of the state to the return of the first episode or the average of the first batch <span class="math-container">$V(s) = R_{1}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-25T20:58:09.780Z
Answers.LastActivityDate: 2023-02-25T20:58:09.780Z
Answers.UserDisplayName: pythonic833
Body: <p>Here is a part of slide 4 from the link:</p>
  <p><a href="https://tao.lri.fr/tiki-download_wiki_attachment.php?attId=1683" rel="nofollow noreferrer">https://tao.lri.fr/tiki-download_wiki_attachment.php?attId=1683</a></p>
  <p><a href="https://i.sstatic.net/4cIIB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4cIIB.png" alt="enter image description here" /></a></p>
  <p>Why does <span class="math-container">$\alpha=1$</span> mean batch MC Learning? I do not see this clearly when I compare with averaging returns formula.</p>
  
Tags: <reinforcement-learning>
Title: Why does $\alpha=1$ mean batch MC Learning?
UserDisplayName: DSPinfinity
ThreadText: # Why does $\alpha=1$ mean batch MC Learning?
  
  (Asked by: username_0 on 2023-02-19)
  
  Here is a part of slide 4 from the link:
  
  <https://tao.lri.fr/tiki-download_wiki_attachment.php?attId=1683>
  
  [![enter image description here](https://i.sstatic.net/4cIIB.png)](https://i.sstatic.net/4cIIB.png)
  
  Why does $\alpha=1$ mean batch MC Learning? I do not see this clearly when I compare with averaging returns formula.
  
  **Tags:** reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-25. Score: 1 | Accepted answer
  
  I did not read the link but I am giving a standard derivation as it can be found in Sutton-Barto for instance.
  
  The average return formula can be reformulated:
  \begin{align}
  V\_{n+1}(s) &= \frac{1}{n}\sum\_{i=1}^{n}R\_{i} \\
  &= \frac{1}{n}\left(R\_{n} +\sum\_{i=1}^{n-1}R\_{i} \right) \\
  &= \frac{1}{n}\left(R\_{n} +(n-1) \frac{1}{n-1}\sum\_{i=1}^{n-1}R\_{i} \right) \\
  &= \frac{1}{n}\left(R\_{n} + (n-1) V\_{n}(s) \right) \\
  &= \frac{1}{n}\left(R\_{n} + nV\_{n}(s) - V\_{n}(s) \right) \\
  &= V\_{n}(s) + \frac{1}{n} (R\_{n} - V\_{n}(s))
  \end{align}
  where $V\_{n}$ is the estimate of the value function after $n-1$ averages over the return of the state $s$. We can interpret each $i$ as a single visit to $s$ or as a batch of visits of $s$ in which case $R\_{i}$ would be the averaged return of this batch. Setting $\frac{1}{n} = \alpha$ directly results in the formula for incremental updates. Note that if we set $n = \alpha = 1$ we just set the value function of the state to the return of the first episode or the average of the first batch $V(s) = R\_{1}$.
  
  
  ---
  
  