Answers.Body: <p>We assume that our MDP is ergodic. Loosely speaking, this means that wherever the MDP starts (i.e. no matter which state we start in) or any actions the agent takes early on can only have a limited effect on the MDP and in the limit (as <span class="math-container">$t \rightarrow \infty$</span>) the expectation of being in a given state depends only on the policy <span class="math-container">$\pi$</span> and the transition dynamics of the MDP.</p>
  <p>This means that, eventually, <span class="math-container">$\mathbb{E}[R_t] = \mathbb{E}[R_{t+1}]$</span> for some large <span class="math-container">$t$</span>. Therefore, as we take the average of our expected values of the rewards received for an infinitely long period of time, this will have converged due to what I just mentioned of <span class="math-container">$\mathbb{E}[R_t] = \mathbb{E}[R_{t+1}]$</span>. To see why the two are equal, recall that the reward received is dependent on the current state and the action taken -- to better emphasise this I will briefly denote the reward at time step <span class="math-container">$t+1$</span> as <span class="math-container">$R(S_t, A_t)$</span>. If we are in the steady state distribution, that is, the state distribution is now fixed, and our actions are still taken according to our policy, then the expected value of <span class="math-container">$R(S_t, A_t)$</span> will be the same for all future <span class="math-container">$t$</span> since neither the policy nor the state distribution are changing (recall that the average rewards are a way of evaluating a policy in the average-reward setting so for sure this does not change).</p>
  <p>A way to think of this is that since we know that, eventually, <span class="math-container">$\mathbb{E}[R_t]$</span> will equal <span class="math-container">$\mathbb{E}[R_{t+1}]$</span>, and so if we keep have an infinite number of these, the average of them will of course converge to the same value. Imagine if I gave you the sequence 1, 2, 3, 4, 4, 4, 4, ........, 4 and asked you to take the average - if we had an infinite amount of 4's then the average would of course be 4.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-07T23:50:43.063Z
Answers.LastActivityDate: 2020-10-07T23:57:01.570Z
Answers.LastEditDate: 2020-10-07T23:57:01.570Z
Answers.UserDisplayName: David
Body: <p>In the average reward setting we have:</p>
  <p><span class="math-container">$$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$</span></p>
  <p><span class="math-container">$$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$</span></p>
  <p>How is the second equation derived from the first?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-07T19:17:44.423Z
Comments.Text: I think you are missing a factor of $\frac{1}{h}$ in the first equation. Otherwise the two equations are not describing the same thing
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-07T20:29:29.357Z
Comments.Text: at steady state distribution $E[R_t] = E[R_{t+1}] = E[R_{t+n}]$ If you substitute this into equation 1 you will end up with equation 2. And you are missing a factor $\frac{1}{h}$ in equation 1.
Comments.UserDisplayName: Swakshar Deb
Tags: <reinforcement-learning><rewards><function-approximation><sutton-barto>
Title: How do we derive the expression for average reward setting in continuing tasks?
UserDisplayName: ZERO NULLS
ThreadText: # How do we derive the expression for average reward setting in continuing tasks?
  
  (Asked by: username_0 on 2020-10-07)
  
  In the average reward setting we have:
  
  $$r(\pi)\doteq \lim\_{h\rightarrow\infty}\frac{1}{h}\sum\_{t=1}^{h}\mathbb{E}[R\_{t}|S\_0,A\_{0:t-1}\sim\pi]$$
  
  $$r(\pi)\doteq \lim\_{t\rightarrow\infty}\mathbb{E}[R\_{t}|S\_0,A\_{0:t-1}\sim\pi]$$
  
  How is the second equation derived from the first?
  
  **Tags:** reinforcement-learning, rewards, function-approximation, sutton-barto
  
  > **Comments:**
  > **username_1**: I think you are missing a factor of $\frac{1}{h}$ in the first equation. Otherwise the two equations are not describing the same thing  
  > **username_2**: at steady state distribution $E[R_t] = E[R_{t+1}] = E[R_{t+n}]$ If you substitute this into equation 1 you will end up with equation 2. And you are missing a factor $\frac{1}{h}$ in equation 1.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-10-07. Score: 3 | Accepted answer
  
  We assume that our MDP is ergodic. Loosely speaking, this means that wherever the MDP starts (i.e. no matter which state we start in) or any actions the agent takes early on can only have a limited effect on the MDP and in the limit (as $t \rightarrow \infty$) the expectation of being in a given state depends only on the policy $\pi$ and the transition dynamics of the MDP.
  
  This means that, eventually, $\mathbb{E}[R\_t] = \mathbb{E}[R\_{t+1}]$ for some large $t$. Therefore, as we take the average of our expected values of the rewards received for an infinitely long period of time, this will have converged due to what I just mentioned of $\mathbb{E}[R\_t] = \mathbb{E}[R\_{t+1}]$. To see why the two are equal, recall that the reward received is dependent on the current state and the action taken -- to better emphasise this I will briefly denote the reward at time step $t+1$ as $R(S\_t, A\_t)$. If we are in the steady state distribution, that is, the state distribution is now fixed, and our actions are still taken according to our policy, then the expected value of $R(S\_t, A\_t)$ will be the same for all future $t$ since neither the policy nor the state distribution are changing (recall that the average rewards are a way of evaluating a policy in the average-reward setting so for sure this does not change).
  
  A way to think of this is that since we know that, eventually, $\mathbb{E}[R\_t]$ will equal $\mathbb{E}[R\_{t+1}]$, and so if we keep have an infinite number of these, the average of them will of course converge to the same value. Imagine if I gave you the sequence 1, 2, 3, 4, 4, 4, 4, ........, 4 and asked you to take the average - if we had an infinite amount of 4's then the average would of course be 4.
  
  
  ---
  
  