Answers.Body: <p>Mystery solved thanks to Exercise 10.8 in the book. The reason is that we want the running mean to converge to the actual value of the average reward.</p>
  <p>With <span class="math-container">$\bar{R}_{t + 1} =  \bar{R}_t + \beta \delta$</span> with <span class="math-container">$\delta = {R}_{t + 1} - \bar{R}_t + \hat{v}(S_{t + 1}) - \hat{v}(S_t)$</span>, assuming value function estimation converged I would get</p>
  <p><span class="math-container">$\begin{align}
  \delta &amp;= R_{t + 1} - \bar{R}_t + v(S_{t + 1}) - v(S_{t}) \\
  &amp;= R_{t + 1} - \bar{R}_t + (R_{t + 2} - r(\pi^*) + R_{t + 3} - r(\pi^*) + ...) - (R_{t + 1} - r(\pi^*) + R_{t + 2} - r(\pi^*) + ...) \\
  &amp;= R_{t + 1} - \bar{R}_t - (R_{t + 1} - r(\pi^*)) \\
  &amp;= r(\pi^*) - \bar{R}_t
  \end{align}
  $</span></p>
  <p>And the estimation will converge to the average reward of the optimal policy, being updated as <span class="math-container">$\bar{R}_{t + 1} = (1 - \beta) \bar{R}_t + \beta r(\pi^*)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-03T12:07:21.953Z
Answers.LastActivityDate: 2022-06-03T13:16:37.000Z
Answers.LastEditDate: 2022-06-03T13:16:37.000Z
Answers.UserDisplayName: kmf
Body: <p>In Sutton and Barto's RL book, section 10.3 describes how to use average reward <span class="math-container">$r(\pi)$</span> to define the quality of a policy, re-defining action-value function <span class="math-container">$q_\pi(s,a)$</span> and value function <span class="math-container">$v_\pi(s)$</span> to measure the expected <em>differential return</em> <span class="math-container">$G_t^{diff} =\sum_{t=0}^\infty \left[R_{t} - r(\pi)\right]$</span> of policy <span class="math-container">$\pi$</span> in a state/action pair, or state, instead of its <em>discounted return</em> <span class="math-container">$G_t^{disc}\sum_{t=0}^\infty \gamma^t R_t$</span>.</p>
  <p>Of course, the average rate of reward for a given policy cannot be known in advance but only estimated, so the authors propose keeping a running estimate <span class="math-container">$\bar{R} \rightarrow r(\pi)$</span> that starts at 0 and is then updated as <span class="math-container">$\bar{R}_{t +1} = \bar{R}_t + \beta \delta$</span>, where <span class="math-container">$\beta$</span> is a learning rate and <span class="math-container">$\delta = R_{t + 1} - \bar{R}_t + \hat{q}(S_{t + 1}, A_{t + 1}) -  \hat{q}(S_{t}, A_{t})$</span> is the action-value TD error, in a SARSA context where <span class="math-container">$A_{t+1}$</span> is sampled; I understand we might as well use the value function TD error <span class="math-container">$R_{t + 1} - \bar{R}_t + \hat{v}(S_{t + 1}) -  \hat{v}(S_{t})$</span> if we are estimating that.</p>
  <p>My question is: why does the update rule for the average rate of reward estimate consider TD error, and why are we not updating it simply as <span class="math-container">$\bar{R}_{t + 1} = (1 - \beta) \bar{R}_t + \beta R_{t+1}$</span> ?</p>
  
Tags: <reinforcement-learning><deep-rl><rewards><function-approximation><sutton-barto>
Title: Why does the average-reward estimator for continuing tasks use the TD error?
UserDisplayName: kmf
ThreadText: # Why does the average-reward estimator for continuing tasks use the TD error?
  
  (Asked by: username_0 on 2022-05-28)
  
  In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q\_\pi(s,a)$ and value function $v\_\pi(s)$ to measure the expected *differential return* $G\_t^{diff} =\sum\_{t=0}^\infty \left[R\_{t} - r(\pi)\right]$ of policy $\pi$ in a state/action pair, or state, instead of its *discounted return* $G\_t^{disc}\sum\_{t=0}^\infty \gamma^t R\_t$.
  
  Of course, the average rate of reward for a given policy cannot be known in advance but only estimated, so the authors propose keeping a running estimate $\bar{R} \rightarrow r(\pi)$ that starts at 0 and is then updated as $\bar{R}\_{t +1} = \bar{R}\_t + \beta \delta$, where $\beta$ is a learning rate and $\delta = R\_{t + 1} - \bar{R}\_t + \hat{q}(S\_{t + 1}, A\_{t + 1}) - \hat{q}(S\_{t}, A\_{t})$ is the action-value TD error, in a SARSA context where $A\_{t+1}$ is sampled; I understand we might as well use the value function TD error $R\_{t + 1} - \bar{R}\_t + \hat{v}(S\_{t + 1}) - \hat{v}(S\_{t})$ if we are estimating that.
  
  My question is: why does the update rule for the average rate of reward estimate consider TD error, and why are we not updating it simply as $\bar{R}\_{t + 1} = (1 - \beta) \bar{R}\_t + \beta R\_{t+1}$ ?
  
  **Tags:** reinforcement-learning, deep-rl, rewards, function-approximation, sutton-barto
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-06-03. Score: 3 | Accepted answer
  
  Mystery solved thanks to Exercise 10.8 in the book. The reason is that we want the running mean to converge to the actual value of the average reward.
  
  With $\bar{R}\_{t + 1} = \bar{R}\_t + \beta \delta$ with $\delta = {R}\_{t + 1} - \bar{R}\_t + \hat{v}(S\_{t + 1}) - \hat{v}(S\_t)$, assuming value function estimation converged I would get
  
  $\begin{align}
  \delta &= R\_{t + 1} - \bar{R}\_t + v(S\_{t + 1}) - v(S\_{t}) \\
  &= R\_{t + 1} - \bar{R}\_t + (R\_{t + 2} - r(\pi^\*) + R\_{t + 3} - r(\pi^\*) + ...) - (R\_{t + 1} - r(\pi^\*) + R\_{t + 2} - r(\pi^\*) + ...) \\
  &= R\_{t + 1} - \bar{R}\_t - (R\_{t + 1} - r(\pi^\*)) \\
  &= r(\pi^\*) - \bar{R}\_t
  \end{align}
  $
  
  And the estimation will converge to the average reward of the optimal policy, being updated as $\bar{R}\_{t + 1} = (1 - \beta) \bar{R}\_t + \beta r(\pi^\*)$.
  
  
  ---
  
  