Answers.Body: <p>RL agents - implemented correctly - do not take previous rewards into account when making decisions. For instance value functions only assess potential future reward. The state value or expected <em>return</em> (aka <em>utility</em>) <span class="math-container">$G$</span> from a starting state <span class="math-container">$s$</span> may be defined like this:</p>
  
  <p><span class="math-container">$$v(s) = \mathbb{E}_{\pi}[G_t|S_t=s] = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^kR_{t+k+1}|S_t=s] $$</span></p>
  
  <p>Where <span class="math-container">$R_t$</span> is the reward distribution at time <span class="math-container">$t$</span>, and <span class="math-container">$\mathbb{E}_{\pi}$</span> stands for 
  expected value given following the policy <span class="math-container">$\pi$</span> for action selection.</p>
  
  <p>There are a few variations of this, depending on setting and which value function you are interested in. However, all value functions used in RL look at <em>future</em> sums of reward from the decision point when the action is taken. Past rewards are not taken into account.</p>
  
  <p>An agent may still select to take an early high reward over a longer term reward, if:</p>
  
  <ul>
  <li><p>The choice between two rewards is exclusive</p></li>
  <li><p>The return is higher for the early reward. This may depend on the discounting factor, <span class="math-container">$\gamma$</span>, where low values make the agent prefer more immediate rewards. </p></li>
  </ul>
  
  <p>If your problem is that an agent selects a low early reward when it could ignore it in favour of something larger later, then you should check the discount factor you are using. If you want a RL agent to take a long term view, then the discount factor needs to be close to <span class="math-container">$1.0$</span>.</p>
  
  <p>The premise of your question however is that somehow a RL agent would become "lazy" or "complacent" because it already had enough reward. That is not an issue that occurs in RL due to the way that it is formulated. Not only are past rewards not accounted for when calculating return values from states, but there is also no formula in RL for an agent receiving "enough" total reward like a creature satisfying its hunger - the maximisation is applied always in all states.</p>
  
  <p>There is no need to somehow decay past rewards in any memory structure, and in fact no real way to do this, as there is no data structure that accumulates past rewards used by any RL agent. You may still collect this information for displaying results or analysing performance, but the agent doesn't ever use <span class="math-container">$r_{t}$</span> to figure out what <span class="math-container">$a_t$</span> should be.</p>
  
  <blockquote>
    <p>I found that for certain applications and certain hyperparameters, if reward is cumulative, the agent simply takes a good action at the beginning of the episode, and then is happy to do nothing for the rest of the episode (because it still has a reward of R</p>
  </blockquote>
  
  <p>You have probably formulated the reward function incorrectly for your problem in that case. A cumulative reward scheme (where an agent receives reward <span class="math-container">$a$</span> at <span class="math-container">$t=1$</span> then <span class="math-container">$a+b$</span> on <span class="math-container">$t=2$</span> then <span class="math-container">$a+b+c$</span> on <span class="math-container">$t=3$</span> etc) would be quite specialist and you have likely misunderstood how to represent the agent's goals. I suggest ask a separate question about your specific environment and your proposed reward scheme if you cannot resolve this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-11T08:02:18.500Z
Answers.LastActivityDate: 2019-08-11T08:17:19.080Z
Answers.LastEditDate: 2019-08-11T08:17:19.080Z
Answers.UserDisplayName: Neil Slater
Body: <p><strong>Should a reward be cumulative or diminish over time?</strong></p>
  
  <p>For example, say an agent performed a good action at time <span class="math-container">$t$</span> and received a positive reward <span class="math-container">$R$</span>. If reward is cumulative, <span class="math-container">$R$</span> is carried on through for the rest of the episode, and summed to any future rewards. However, if <span class="math-container">$R$</span> were to diminish over time (say with some scaling <span class="math-container">$\frac{R}{\sqrt{t}}$</span>), then wouldn't that encourage the agent to keep taking actions to increasing its reward?</p>
  
  <p>With cumulative rewards, the reward can both increase and decrease depending on the agents actions. But if the agent receives one good reward <span class="math-container">$R$</span> and then does nothing for a long time, it still has the original reward it received (encouraging it to do less?). However, if rewards diminish over time, in theory that would encourage the agent to keep taking actions to maximise rewards.</p>
  
  <p>I found that for certain applications and certain hyperparameters, if reward is cumulative, the agent simply takes a good action at the beginning of the episode, and then is happy to do nothing for the rest of the episode (because it still has a reward of <span class="math-container">$R$</span>).</p>
  
Tags: <reinforcement-learning><rewards>
Title: Should RL rewards diminish over time?
UserDisplayName: PyRsquared
ThreadText: # Should RL rewards diminish over time?
  
  (Asked by: username_0 on 2019-08-11)
  
  **Should a reward be cumulative or diminish over time?**
  
  For example, say an agent performed a good action at time $t$ and received a positive reward $R$. If reward is cumulative, $R$ is carried on through for the rest of the episode, and summed to any future rewards. However, if $R$ were to diminish over time (say with some scaling $\frac{R}{\sqrt{t}}$), then wouldn't that encourage the agent to keep taking actions to increasing its reward?
  
  With cumulative rewards, the reward can both increase and decrease depending on the agents actions. But if the agent receives one good reward $R$ and then does nothing for a long time, it still has the original reward it received (encouraging it to do less?). However, if rewards diminish over time, in theory that would encourage the agent to keep taking actions to maximise rewards.
  
  I found that for certain applications and certain hyperparameters, if reward is cumulative, the agent simply takes a good action at the beginning of the episode, and then is happy to do nothing for the rest of the episode (because it still has a reward of $R$).
  
  **Tags:** reinforcement-learning, rewards
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-11. Score: 5 | Accepted answer
  
  RL agents - implemented correctly - do not take previous rewards into account when making decisions. For instance value functions only assess potential future reward. The state value or expected *return* (aka *utility*) $G$ from a starting state $s$ may be defined like this:
  
  $$v(s) = \mathbb{E}\_{\pi}[G\_t|S\_t=s] = \mathbb{E}\_{\pi}[\sum\_{k=0}^{\infty} \gamma^kR\_{t+k+1}|S\_t=s] $$
  
  Where $R\_t$ is the reward distribution at time $t$, and $\mathbb{E}\_{\pi}$ stands for
  expected value given following the policy $\pi$ for action selection.
  
  There are a few variations of this, depending on setting and which value function you are interested in. However, all value functions used in RL look at *future* sums of reward from the decision point when the action is taken. Past rewards are not taken into account.
  
  An agent may still select to take an early high reward over a longer term reward, if:
  
  * The choice between two rewards is exclusive
  * The return is higher for the early reward. This may depend on the discounting factor, $\gamma$, where low values make the agent prefer more immediate rewards.
  
  If your problem is that an agent selects a low early reward when it could ignore it in favour of something larger later, then you should check the discount factor you are using. If you want a RL agent to take a long term view, then the discount factor needs to be close to $1.0$.
  
  The premise of your question however is that somehow a RL agent would become "lazy" or "complacent" because it already had enough reward. That is not an issue that occurs in RL due to the way that it is formulated. Not only are past rewards not accounted for when calculating return values from states, but there is also no formula in RL for an agent receiving "enough" total reward like a creature satisfying its hunger - the maximisation is applied always in all states.
  
  There is no need to somehow decay past rewards in any memory structure, and in fact no real way to do this, as there is no data structure that accumulates past rewards used by any RL agent. You may still collect this information for displaying results or analysing performance, but the agent doesn't ever use $r\_{t}$ to figure out what $a\_t$ should be.
  
  > I found that for certain applications and certain hyperparameters, if reward is cumulative, the agent simply takes a good action at the beginning of the episode, and then is happy to do nothing for the rest of the episode (because it still has a reward of R
  
  You have probably formulated the reward function incorrectly for your problem in that case. A cumulative reward scheme (where an agent receives reward $a$ at $t=1$ then $a+b$ on $t=2$ then $a+b+c$ on $t=3$ etc) would be quite specialist and you have likely misunderstood how to represent the agent's goals. I suggest ask a separate question about your specific environment and your proposed reward scheme if you cannot resolve this.
  
  
  ---
  
  