Answers.Body: <p>The main thing you will need to do is add the accumulated reward (<code>total_score_so_far</code>) to the state. In order to predict future reward with any accuracy, the agent is going to need to know this.</p>
  <p>You may still have some problems after doing this. The final return that the agent needs to predict is likely to have the following traits:</p>
  <ul>
  <li><p>A value that could range by orders of magnitude, making it hard to scale loss functions. You may be able to base the loss function for value predictions on mean relative error to help with this.</p>
  </li>
  <li><p>Large variance, making it difficult to learn expected returns, especially from the impact of early decisions. If your immediate rewards have any random element, this could be a major problem.</p>
  </li>
  </ul>
  <p>If value-based methods such as DQN are struggling in your case, and the optimal policy function is straightfoward, then you may want to try policy gradient methods, maybe even basic REINFORCE, to avoid needing to predict returns whilst dealing with such a high variance. You will still need to take care with scaling, as policy gradient methods scale update steps based on the return.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-31T17:00:12.960Z
Answers.Text: Thank you very much for your response. I forgot to mention, I have actually already added the accumulated reward to my observation space. While I'm not sure, I doubt the optimal policy is straightforward (it's complicated enough that I myself was unable to learn the optimal policy by manually playing with the data). I agree with every point you make, particularly the two bullet points. I had an idea to make my episodes shorter (less timesteps per episode), so that 1) there isn't enough time for rewards to get huge/tiny and 2) the "impact of early decisions" is reduced. What do you think?
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-31T17:03:41.527Z
Answers.Text: @VladimirBelik That's a good idea IMO. If you can simplify the environment and find an approach that works there, you may be able to scale it back up to the full environment by degrees based on what you discover. I cannot say what is likely to work though, as I have not really studied an environment with reward behaviour that you describe in the question
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-31T16:51:25.467Z
Answers.LastActivityDate: 2022-05-31T16:51:25.467Z
Answers.UserDisplayName: Neil Slater
Body: <p>In the context of my problem, the &quot;true&quot; reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to say, the real reward function is partially dependent on previously accumulated reward.</p>
  <p>Is there any way to implement this kind of dynamic successfully?</p>
  <p>I have tried to, but for some reason, the agent completely stops learning when I do this. I can implement a linear/additive reward function and the agent does learn good behaviors, but I feel that it's important for the agent to &quot;understand&quot; the true reward dynamic.</p>
  <p>Essentially, here is the reward function I have:</p>
  <pre><code>reward = points_gained_this_step
  </code></pre>
  <p>But here what I need:</p>
  <pre><code>reward = points_gained_this_step*(total_score_so_far)
  total_score_so_far = total_score_so_far + reward
  </code></pre>
  <p>Has anyone ever worked with something like this? Any ideas/insight for how to implement such a reward? I might be wrong, but it seems to me like part of the problem is exploding/vanishing gradients?</p>
  <p>EDIT: I have already added &quot;total_score_so_far&quot; to my observation space.</p>
  
Tags: <reinforcement-learning><deep-rl><reward-functions><reward-design>
Title: In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?
UserDisplayName: Vladimir Belik
ThreadText: # In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?
  
  (Asked by: username_0 on 2022-05-31)
  
  In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to say, the real reward function is partially dependent on previously accumulated reward.
  Is there any way to implement this kind of dynamic successfully?
  I have tried to, but for some reason, the agent completely stops learning when I do this. I can implement a linear/additive reward function and the agent does learn good behaviors, but I feel that it's important for the agent to "understand" the true reward dynamic.
  Essentially, here is the reward function I have:
  reward = points_gained_this_step
  
  But here what I need:
  reward = points_gained_this_step*(total_score_so_far)
  total_score_so_far = total_score_so_far + reward
  
  Has anyone ever worked with something like this? Any ideas/insight for how to implement such a reward? I might be wrong, but it seems to me like part of the problem is exploding/vanishing gradients?
  EDIT: I have already added "total_score_so_far" to my observation space.
  
  **Tags:** reinforcement-learning, deep-rl, reward-functions, reward-design
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-31. Score: 3 | Accepted answer
  
  The main thing you will need to do is add the accumulated reward (`total_score_so_far`) to the state. In order to predict future reward with any accuracy, the agent is going to need to know this.
  
  You may still have some problems after doing this. The final return that the agent needs to predict is likely to have the following traits:
  
  * A value that could range by orders of magnitude, making it hard to scale loss functions. You may be able to base the loss function for value predictions on mean relative error to help with this.
  * Large variance, making it difficult to learn expected returns, especially from the impact of early decisions. If your immediate rewards have any random element, this could be a major problem.
  
  If value-based methods such as DQN are struggling in your case, and the optimal policy function is straightfoward, then you may want to try policy gradient methods, maybe even basic REINFORCE, to avoid needing to predict returns whilst dealing with such a high variance. You will still need to take care with scaling, as policy gradient methods scale update steps based on the return.
  
  > **Comments:**
  > **username_0**: Thank you very much for your response. I forgot to mention, I have actually already added the accumulated reward to my observation space. While I'm not sure, I doubt the optimal policy is straightforward (it's complicated enough that I myself was unable to learn the optimal policy by manually playing with the data). I agree with every point you make, particularly the two bullet points. I had an idea to make my episodes shorter (less timesteps per episode), so that 1) there isn't enough time for rewards to get huge/tiny and 2) the "impact of early decisions" is reduced. What do you think?  
  > **username_1**: @VladimirBelik That's a good idea IMO. If you can simplify the environment and find an approach that works there, you may be able to scale it back up to the full environment by degrees based on what you discover. I cannot say what is likely to work though, as I have not really studied an environment with reward behaviour that you describe in the question  
  
  ---
  
  