Answers.Body: <blockquote>
  <p>Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?</p>
  </blockquote>
  <p>That is the definition of return.</p>
  <p>In fact when applying a discount factor this should formally be called <em>discounted</em> return, and not simply &quot;return&quot;. Usually the same symbol is used for both (<span class="math-container">$R$</span> in your case, <span class="math-container">$G$</span> in e.g. Sutton &amp; Barto).</p>
  <p>There are also other variations, such as <em>truncated</em> return (sum up to a given time horizon). They all share the feature that a return is a sum of reward values. You cannot really change that and keep the formal term &quot;return&quot;, that's how it has been defined.</p>
  <p>You can however define the value function to be something other than the expected return. Rather than looking for alternative definitions of return as your title suggests, you could be looking for alternative metrics to use as value functions.</p>
  <p>You do go on to ask about computing &quot;the value of a state&quot; without mentioning the word &quot;return&quot;, but it is not 100% clear whether you are aware that the way to resolve this is to not use return, but something else.</p>
  <blockquote>
  <p>Would it not make more sense to compute the value of a state as the following:
  <span class="math-container">$V^π(s)$</span> = <span class="math-container">$(r_{t+1} + γr_{t+2} + (γ^2)_{t+3}, + ...    \mid s)/k = {E}(∑γ^kr_{t+k+1}\mid s)/k $</span> where k is the number of elements in the sum, thus giving us the average reward for being in state s.</p>
  </blockquote>
  <p>Your example would nearly always result in zero for long-running or non-episodic problems, as you are summing a decreasing geometric series possibly up to very large <span class="math-container">$k$</span>, then dividing by the maximum <span class="math-container">$k$</span>. Notation-wise you are also using <span class="math-container">$k$</span> to be an iterator and the maximum value of the same iterator, that would need fixing.</p>
  <p>However, this is very close to a real value metric used in reinforcement learning, called the <a href="https://link.springer.com/article/10.1007/BF00114727" rel="nofollow noreferrer">average reward setting</a>.</p>
  <p>The expected average reward value function for a non-episodic problem is typically given by</p>
  <p><span class="math-container">$$V^\pi(s) = \mathbb{E}[\lim_{h \to \infty}\frac{1}{h}\sum_{k=0}^{h}r_{t+k+1}|s_t = s]$$</span></p>
  <p>Note there is no discount factor, it is not usually possible to combine a discount factor with the average reward setting.</p>
  <p>Sutton &amp; Barto point out in <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> chapter 10, section 10.4, that when using function approximation on continuing tasks, then a discount factor is not a useful part of the setting. Instead average reward is a more natural approach. It is also not so different, and quite easy to modify the Bellman equations and update rules. However, many DQN implementations still use discounted return to solve continuing tasks. That is because with high enough discount factor <span class="math-container">$\gamma$</span>, e.g. <span class="math-container">$0.99$</span> or <span class="math-container">$0.999$</span>, then the end result is likely to be the same optimal solution - the discount factor has moved from being part of the problem formulation to being a solution hyperparameter.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-17T12:50:09.103Z
Answers.LastActivityDate: 2020-07-17T16:46:03.407Z
Answers.LastEditDate: 2020-07-17T16:46:03.407Z
Answers.UserDisplayName: Neil Slater
Body: <p>Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?</p>
  <p>Would it not make more sense to compute <span class="math-container">$\mathbb{E}(R \mid s, a)$</span> (the expected return for taking action <span class="math-container">$a$</span> in the given state <span class="math-container">$s$</span>) as the average of all rewards recorded for being in state <span class="math-container">$s$</span> and taking action <span class="math-container">$a$</span>?</p>
  <p>In many examples, I've seen the value of a state computed as the expected return computed as the cumulative sum of rewards multiplied by a discount factor:</p>
  <p><span class="math-container">$V^π(s)$</span> = <span class="math-container">$\mathbb{E}(R \mid s)$</span> (the value of state s, if we follow policy π is equal to the expected return given state s)</p>
  <p>So, <span class="math-container">$V^π(s)$</span> = <span class="math-container">$\mathbb{E}(r_{t+1}+ γr_{t+2}+ (γ^2)_{t+3} + ...    \mid s) = {E}(∑γ^kr_{t+k+1}\mid s)$</span></p>
  <p>as <span class="math-container">$R=r_{t+1}+ γr_{t+2}+ {γ^2}r_{t+3}, + ...   $</span></p>
  <p>Would it not make more sense to compute the value of a state as the following:</p>
  <p><span class="math-container">$V^π(s)$</span> = <span class="math-container">$(r_{t+1} + γr_{t+2} + (γ^2)_{t+3}, + ...    \mid s)/k = {E}(∑γ^kr_{t+k+1}\mid s)/k $</span> where k is the number of elements in the sum, thus giving us the average reward for being in state s.</p>
  <p>Reference for cumulative sum example: <a href="https://joshgreaves.com/reinforcement-learning/understanding-rl-the-bellman-equations/" rel="nofollow noreferrer">https://joshgreaves.com/reinforcement-learning/understanding-rl-the-bellman-equations/</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-17T02:21:44.693Z
Comments.Text: Hi flying_costa. Interesting question! Would it be possible to write out your alternative definition for $\mathbb{E}(R\ |\ s, a)$ as an equation or explain it in more detail? I'm currently having a little trouble understanding your idea. Also, writing your question with capital letters and complete sentences may aid in clarity.
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-17T12:10:31.033Z
Comments.Text: I've added some more information to hopefully make the question more clear!
Comments.UserDisplayName: THAT_AI_GUY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-17T12:23:07.990Z
Comments.Text: Reading the reference more carefully, it is self-consistent at least in that both $R$ and $r_t$ (plus $s_t$, $a_t$) must be interpreted as distributions. My confusion stems from thinking it was following notation conventions from Sutton & Barto. It is not.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-17T12:31:02.257Z
Comments.Text: I'm not sure what you mean. The definitions in the site appear to be consistent. I'm asking why the return for being in state s is computed as the sum of returns given state s, not the mean of the returns for being in the state s.
Comments.UserDisplayName: THAT_AI_GUY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-17T12:32:36.343Z
Comments.Text: The return is not calculated as sum of returns, but of rewards. You get that right in the question, so I assume a typo in your comment? Either way I think I understand your question and am typing an answer  . . .
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-17T12:42:01.897Z
Comments.Text: Yeah,  sorry, my comment should read "I'm asking why the return for being in state s is computed as the sum of rewards given state s, not the mean of the rewards for being in the state s"
Comments.UserDisplayName: THAT_AI_GUY
Tags: <reinforcement-learning><q-learning><rewards><value-functions><return>
Title: Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
UserDisplayName: THAT_AI_GUY
ThreadText: # Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
  
  (Asked by: username_0 on 2020-07-17)
  
  Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
  
  Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking action $a$ in the given state $s$) as the average of all rewards recorded for being in state $s$ and taking action $a$?
  
  In many examples, I've seen the value of a state computed as the expected return computed as the cumulative sum of rewards multiplied by a discount factor:
  
  $V^π(s)$ = $\mathbb{E}(R \mid s)$ (the value of state s, if we follow policy π is equal to the expected return given state s)
  
  So, $V^π(s)$ = $\mathbb{E}(r\_{t+1}+ γr\_{t+2}+ (γ^2)\_{t+3} + ... \mid s) = {E}(∑γ^kr\_{t+k+1}\mid s)$
  
  as $R=r\_{t+1}+ γr\_{t+2}+ {γ^2}r\_{t+3}, + ... $
  
  Would it not make more sense to compute the value of a state as the following:
  
  $V^π(s)$ = $(r\_{t+1} + γr\_{t+2} + (γ^2)\_{t+3}, + ... \mid s)/k = {E}(∑γ^kr\_{t+k+1}\mid s)/k $ where k is the number of elements in the sum, thus giving us the average reward for being in state s.
  
  Reference for cumulative sum example: <https://joshgreaves.com/reinforcement-learning/understanding-rl-the-bellman-equations/>
  
  **Tags:** reinforcement-learning, q-learning, rewards, value-functions, return
  
  > **Comments:**
  > **username_1**: Hi flying_costa. Interesting question! Would it be possible to write out your alternative definition for $\mathbb{E}(R\ |\ s, a)$ as an equation or explain it in more detail? I'm currently having a little trouble understanding your idea. Also, writing your question with capital letters and complete sentences may aid in clarity.  
  > **username_0**: I've added some more information to hopefully make the question more clear!  
  > **username_2**: Reading the reference more carefully, it is self-consistent at least in that both $R$ and $r_t$ (plus $s_t$, $a_t$) must be interpreted as distributions. My confusion stems from thinking it was following notation conventions from Sutton & Barto. It is not.  
  > **username_0**: I'm not sure what you mean. The definitions in the site appear to be consistent. I'm asking why the return for being in state s is computed as the sum of returns given state s, not the mean of the returns for being in the state s.  
  > **username_2**: The return is not calculated as sum of returns, but of rewards. You get that right in the question, so I assume a typo in your comment? Either way I think I understand your question and am typing an answer  . . .  
  > **username_0**: Yeah,  sorry, my comment should read "I'm asking why the return for being in state s is computed as the sum of rewards given state s, not the mean of the rewards for being in the state s"  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-17. Score: 3
  
  > Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
  
  That is the definition of return.
  
  In fact when applying a discount factor this should formally be called *discounted* return, and not simply "return". Usually the same symbol is used for both ($R$ in your case, $G$ in e.g. Sutton & Barto).
  
  There are also other variations, such as *truncated* return (sum up to a given time horizon). They all share the feature that a return is a sum of reward values. You cannot really change that and keep the formal term "return", that's how it has been defined.
  
  You can however define the value function to be something other than the expected return. Rather than looking for alternative definitions of return as your title suggests, you could be looking for alternative metrics to use as value functions.
  
  You do go on to ask about computing "the value of a state" without mentioning the word "return", but it is not 100% clear whether you are aware that the way to resolve this is to not use return, but something else.
  
  > Would it not make more sense to compute the value of a state as the following:
  > $V^π(s)$ = $(r\_{t+1} + γr\_{t+2} + (γ^2)\_{t+3}, + ... \mid s)/k = {E}(∑γ^kr\_{t+k+1}\mid s)/k $ where k is the number of elements in the sum, thus giving us the average reward for being in state s.
  
  Your example would nearly always result in zero for long-running or non-episodic problems, as you are summing a decreasing geometric series possibly up to very large $k$, then dividing by the maximum $k$. Notation-wise you are also using $k$ to be an iterator and the maximum value of the same iterator, that would need fixing.
  
  However, this is very close to a real value metric used in reinforcement learning, called the [average reward setting](https://link.springer.com/article/10.1007/BF00114727).
  
  The expected average reward value function for a non-episodic problem is typically given by
  
  $$V^\pi(s) = \mathbb{E}[\lim\_{h \to \infty}\frac{1}{h}\sum\_{k=0}^{h}r\_{t+k+1}|s\_t = s]$$
  
  Note there is no discount factor, it is not usually possible to combine a discount factor with the average reward setting.
  
  Sutton & Barto point out in [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/RLbook2020.pdf) chapter 10, section 10.4, that when using function approximation on continuing tasks, then a discount factor is not a useful part of the setting. Instead average reward is a more natural approach. It is also not so different, and quite easy to modify the Bellman equations and update rules. However, many DQN implementations still use discounted return to solve continuing tasks. That is because with high enough discount factor $\gamma$, e.g. $0.99$ or $0.999$, then the end result is likely to be the same optimal solution - the discount factor has moved from being part of the problem formulation to being a solution hyperparameter.
  
  
  ---
  
  