Answers.Body: <p>Discounted reward has its opposite undiscounted reward. The difference between <em>these</em> is that discounted has multiplier gamma != 1 and undiscounted is gamma = 1. Gamma tells the multiplier, how much previous values are valued in next iterations. [1]</p>
  
  <p>Additive refers to different thing, a note found in [2]:</p>
  
  <blockquote>
    <p>An additive reward function decomposes the reward into a number of contributions and can be represented as a (non-partiotioning) MPA.</p>
  </blockquote>
  
  <p>This short excerpt does not reveal lot, but was the only I could find that made some sense to me. What I <em>could</em> indeed find out was that although they seem similar concepts by name, they appear totally different by nature.</p>
  
  <p><strong>Sources:</strong></p>
  
  <p>[1] <a href="https://en.wikipedia.org/wiki/Reinforcement_learning" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Reinforcement_learning</a></p>
  
  <p>[2] The Logic of Adaptive Behavior: Knowledge Representation and Algorithms for Adaptive Sequential Decision Making Under Uncertainty In First-Order and Relational Domains.- M. Van</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-10T19:40:14.090Z
Answers.LastActivityDate: 2018-12-10T19:40:14.090Z
Answers.UserDisplayName: mico
Answers.Body: <p>I will disagree slightly with @mico. There is a usage of "additive rewards" that refers to decomposable reward functions (e.g. my reward in selling an item I do not want to own is composed of the reward of not having an unwanted item anymore, plus the monetary gain in selling the item). But, there is indeed a fundamental relation between additive and discounted rewards. Additive rewards are formulated simply as</p>
  
  <p><span class="math-container">$$
  R([s_0, s_1, s_2, ...]) = R(s_0) + R(s_1) + R(s_2) + \cdots
  $$</span></p>
  
  <p>whereas discounted rewards include a discount factor <span class="math-container">$\gamma \in [0,1]$</span> such that</p>
  
  <p><span class="math-container">$$
  R([s_0, s_1, s_2, ...]) = R(s_0) + \gamma R(s_1) + \gamma^2R(s_2) + \cdots
  $$</span></p>
  
  <p>Intuitively, the additive reward for a sequence of states is simply the sum of the rewards acquired at each state, while discounted rewards include a multiplicative discount factor that reduces the influence of rewards as time goes on. You will typically see additive rewards for finite-horizon problems, i.e. you have a discrete number of timesteps to optimize over, and discounted rewards are more relevant for infinite-horizon problems, i.e. you may need to optimize over an infinite number of timesteps (or at least a very large number). The discount factor governs the agent’s greediness in achieving immediate reward, where very small discount factors (closer to 0) encourage the agent to only seek rewards in proximal states, and very large discount factors (closer to 1) encourage the agent to think further into the future about what rewards it can expect to achieve in states it will visit later.</p>
  
  <p>The most direct reference I found for this distinction is in <a href="http://www-anw.cs.umass.edu/~barto/courses/CS383-Fall11/383-Fall11-Lec22.pdf" rel="nofollow noreferrer">these course slides</a>, which are somewhat authoritative since they from Andrew Barto, a co-author of <a href="https://mitpress.mit.edu/books/reinforcement-learning-second-edition" rel="nofollow noreferrer">the de facto text</a> on reinforcement learning. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-12T17:49:24.633Z
Answers.Text: I think your explanation of the discount factor (Gamma) is inverted, a large Gamma (closer to 1) allows the Total Reward to consider reward from further into the future (of similar magnitude to the current immediate reward), and a smaller Gamma will only consider rewards more immediate in time. This is due to the recurisve multiplication of Gamma with each time step, close to 1 will allow you to do that for more time steps before it decays to 0.
Answers.UserDisplayName: logan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-12T20:37:31.913Z
Answers.Text: @logan you're absolutely right, I said that backwards. I fixed it. Thanks.
Answers.UserDisplayName: adamconkey
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-26T17:16:09.033Z
Answers.LastActivityDate: 2020-04-12T20:36:23.027Z
Answers.LastEditDate: 2020-04-12T20:36:23.027Z
Answers.UserDisplayName: adamconkey
Body: <p>What is the difference between additive and discounted rewards?</p>
  
Tags: <reinforcement-learning><comparison><rewards><reward-clipping>
Title: What is the main difference between additive rewards and discounted rewards?
UserDisplayName: Marosh Fatima
ThreadText: # What is the main difference between additive rewards and discounted rewards?
  
  (Asked by: username_0 on 2018-12-09)
  
  What is the difference between additive and discounted rewards?
  
  **Tags:** reinforcement-learning, comparison, rewards, reward-clipping
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-12-10. Score: 0 | Accepted answer
  
  Discounted reward has its opposite undiscounted reward. The difference between *these* is that discounted has multiplier gamma != 1 and undiscounted is gamma = 1. Gamma tells the multiplier, how much previous values are valued in next iterations. [1]
  
  Additive refers to different thing, a note found in [2]:
  
  > An additive reward function decomposes the reward into a number of contributions and can be represented as a (non-partiotioning) MPA.
  
  This short excerpt does not reveal lot, but was the only I could find that made some sense to me. What I *could* indeed find out was that although they seem similar concepts by name, they appear totally different by nature.
  
  **Sources:**
  
  [1] <https://en.wikipedia.org/wiki/Reinforcement_learning>
  
  [2] The Logic of Adaptive Behavior: Knowledge Representation and Algorithms for Adaptive Sequential Decision Making Under Uncertainty In First-Order and Relational Domains.- M. Van
  
  
  ---
  
  ### Answer by username_2 on 2018-12-26. Score: 1
  
  I will disagree slightly with @mico. There is a usage of "additive rewards" that refers to decomposable reward functions (e.g. my reward in selling an item I do not want to own is composed of the reward of not having an unwanted item anymore, plus the monetary gain in selling the item). But, there is indeed a fundamental relation between additive and discounted rewards. Additive rewards are formulated simply as
  
  $$
  R([s\_0, s\_1, s\_2, ...]) = R(s\_0) + R(s\_1) + R(s\_2) + \cdots
  $$
  
  whereas discounted rewards include a discount factor $\gamma \in [0,1]$ such that
  
  $$
  R([s\_0, s\_1, s\_2, ...]) = R(s\_0) + \gamma R(s\_1) + \gamma^2R(s\_2) + \cdots
  $$
  
  Intuitively, the additive reward for a sequence of states is simply the sum of the rewards acquired at each state, while discounted rewards include a multiplicative discount factor that reduces the influence of rewards as time goes on. You will typically see additive rewards for finite-horizon problems, i.e. you have a discrete number of timesteps to optimize over, and discounted rewards are more relevant for infinite-horizon problems, i.e. you may need to optimize over an infinite number of timesteps (or at least a very large number). The discount factor governs the agent’s greediness in achieving immediate reward, where very small discount factors (closer to 0) encourage the agent to only seek rewards in proximal states, and very large discount factors (closer to 1) encourage the agent to think further into the future about what rewards it can expect to achieve in states it will visit later.
  
  The most direct reference I found for this distinction is in [these course slides](http://www-anw.cs.umass.edu/~barto/courses/CS383-Fall11/383-Fall11-Lec22.pdf), which are somewhat authoritative since they from Andrew Barto, a co-author of [the de facto text](https://mitpress.mit.edu/books/reinforcement-learning-second-edition) on reinforcement learning.
  
  > **Comments:**
  > **username_3**: I think your explanation of the discount factor (Gamma) is inverted, a large Gamma (closer to 1) allows the Total Reward to consider reward from further into the future (of similar magnitude to the current immediate reward), and a smaller Gamma will only consider rewards more immediate in time. This is due to the recurisve multiplication of Gamma with each time step, close to 1 will allow you to do that for more time steps before it decays to 0.  
  > **username_2**: @logan you're absolutely right, I said that backwards. I fixed it. Thanks.  
  
  ---
  
  