Answers.Body: <p>This is simply from definition of return in average reward setting (look at equation <span class="math-container">$10.9$</span>). The "standard" TD error is defined as
  <span class="math-container">\begin{equation}
  TD_{\text{error}} = R_{t+1} + V(S_{t+1}) - V(S_t)
  \end{equation}</span>
  In average reward setting, average reward <span class="math-container">$r(\pi)$</span> is subtracted from reward at <span class="math-container">$t$</span>, <span class="math-container">$R_t$</span>, so TD error in this case is
  <span class="math-container">\begin{equation}
  TD_{\text{error}} = R_{t+1} - \bar R_{t+1} + V(S_{t+1}) - V(S_t)
  \end{equation}</span>
  where <span class="math-container">$\bar R_{t+1}$</span> is estimate of <span class="math-container">$r(\pi)$</span>.  </p>
  
  <p>You can use <span class="math-container">$\delta_t$</span> in all 3 updates because neither of these updates depend on each other. For example if you update <span class="math-container">$\mathbf w$</span> you don't use that then to update <span class="math-container">$\mathbf \theta$</span>, or if you update <span class="math-container">$\bar R$</span> you don't use updated version to update <span class="math-container">$\mathbf w$</span> or <span class="math-container">$\mathbf \theta$</span> so you're not introducing additional bias. In each separate update you also don't have <span class="math-container">$\delta_t$</span> present multiple times so that you require multiple sampling per timestep to get the unbiased update.  </p>
  
  <p>Additionally this is semi-gradient algorithm, it uses bootstrapped estimate <span class="math-container">$V_{t+1}$</span> but it doesn't calculate full derivative with respect to it aswell, only with respect to <span class="math-container">$V_t$</span> so the algorithm is biased by default, but works well enough in practice for linear case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-31T15:25:53.880Z
Answers.LastActivityDate: 2020-01-01T10:36:27.310Z
Answers.LastEditDate: 2020-01-01T10:36:27.310Z
Answers.UserDisplayName: Brale
Body: <p>In chapter 10 of Sutton and Barto's book (2nd edition) is given the equation for TD(0) error with average reward (equation 10.10):</p>
  
  <p><span class="math-container">$$\delta_t = R_{t+1} - \bar{R} + \hat{v}(S_{t+1}, \mathbf{w}) - \hat{v}(S_{t}, \mathbf{w})$$</span></p>
  
  <p>What is the intuition behind this equation? And how exactly is it derived? </p>
  
  <p>Also, in chapter 13, section 6, is given the Actor-Critic algorithm, which uses the TD error. How can you use 1 error to update 3 distinct things - like the average reward, value function estimator (critic), and the policy function estimator (actor)? </p>
  
  <p><strong>Average Reward update rule</strong>: <span class="math-container">$\bar{R} \leftarrow \bar{R} + \alpha^{\bar{R}}\delta$</span></p>
  
  <p><strong>Critic weight update rule</strong>: <span class="math-container">$\mathbf{w} \leftarrow \mathbf{w} + \alpha^{\mathbf{w}}\delta\nabla \hat{v}(s,\mathbf{w})$</span></p>
  
  <p><strong>Actor weight update rule</strong>: <span class="math-container">$\mathbf{\theta} \leftarrow \mathbf{\theta} + \alpha^{\mathbf{\theta}}\delta\nabla ln \pi(A|S,\mathbf{\theta})$</span></p>
  
Tags: <reinforcement-learning><actor-critic-methods><temporal-difference-methods>
Title: What is the intuition behind the TD(0) equation with average reward, and how is it derived?
UserDisplayName: Maverick Meerkat
ThreadText: # What is the intuition behind the TD(0) equation with average reward, and how is it derived?
  
  (Asked by: username_0 on 2019-12-31)
  
  In chapter 10 of Sutton and Barto's book (2nd edition) is given the equation for TD(0) error with average reward (equation 10.10):
  
  $$\delta\_t = R\_{t+1} - \bar{R} + \hat{v}(S\_{t+1}, \mathbf{w}) - \hat{v}(S\_{t}, \mathbf{w})$$
  
  What is the intuition behind this equation? And how exactly is it derived?
  
  Also, in chapter 13, section 6, is given the Actor-Critic algorithm, which uses the TD error. How can you use 1 error to update 3 distinct things - like the average reward, value function estimator (critic), and the policy function estimator (actor)?
  
  **Average Reward update rule**: $\bar{R} \leftarrow \bar{R} + \alpha^{\bar{R}}\delta$
  
  **Critic weight update rule**: $\mathbf{w} \leftarrow \mathbf{w} + \alpha^{\mathbf{w}}\delta\nabla \hat{v}(s,\mathbf{w})$
  
  **Actor weight update rule**: $\mathbf{\theta} \leftarrow \mathbf{\theta} + \alpha^{\mathbf{\theta}}\delta\nabla ln \pi(A|S,\mathbf{\theta})$
  
  **Tags:** reinforcement-learning, actor-critic-methods, temporal-difference-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-31. Score: 4 | Accepted answer
  
  This is simply from definition of return in average reward setting (look at equation $10.9$). The "standard" TD error is defined as
  \begin{equation}
  TD\_{\text{error}} = R\_{t+1} + V(S\_{t+1}) - V(S\_t)
  \end{equation}
  In average reward setting, average reward $r(\pi)$ is subtracted from reward at $t$, $R\_t$, so TD error in this case is
  \begin{equation}
  TD\_{\text{error}} = R\_{t+1} - \bar R\_{t+1} + V(S\_{t+1}) - V(S\_t)
  \end{equation}
  where $\bar R\_{t+1}$ is estimate of $r(\pi)$.
  
  You can use $\delta\_t$ in all 3 updates because neither of these updates depend on each other. For example if you update $\mathbf w$ you don't use that then to update $\mathbf \theta$, or if you update $\bar R$ you don't use updated version to update $\mathbf w$ or $\mathbf \theta$ so you're not introducing additional bias. In each separate update you also don't have $\delta\_t$ present multiple times so that you require multiple sampling per timestep to get the unbiased update.
  
  Additionally this is semi-gradient algorithm, it uses bootstrapped estimate $V\_{t+1}$ but it doesn't calculate full derivative with respect to it aswell, only with respect to $V\_t$ so the algorithm is biased by default, but works well enough in practice for linear case.
  
  
  ---
  
  