Answers.Body: <p>Reinforcement learning already has the objective of maximising the sum of future expected reward.</p>
  <p>By making each reward the sum of all previous rewards, you will make the the difference between good and bad next choices low, relative to the overall reward guaranteed on each step.</p>
  <p>The best reward for the agent should be as direct measure of what you want it to achieve as possible. Sometimes you need to compromise because you cannot easily measure what you really want. Sometimes you may need to assist the bot, giving it a more frequent reward signal (e.g. change to predicted cash-out value of all the current holdings) - but you should note this is also a compromise, because an agent that maximises this &quot;reward shaping&quot; reward scheme may not actually maximise what you really want.</p>
  <p>You could view your attempt to use cumulative values as a failed experiment in reward shaping. I don't know if daily revenue would be a good for your bot, because a trading bot might have several different end goals for its creator.</p>
  <p>First thing you should do is figure out what your success criteria is for the bot, and find the measure for it that best captures that goal as directly as possible. E.g. if the goal is for it to create a reliable income after 5 years of trading, then the reward might be the monthly income that it able to generate in the long term. Perhaps simulating or measuring this is too hard, in which case you could fall back to a proxy, such as cash value of any sell orders with a penalty if it is not maintaining a certain amount of total of holdings.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-22T08:16:57.307Z
Answers.LastActivityDate: 2021-11-22T08:16:57.307Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over many years of data, the reward tends to become large. I have realized that after some training the agent becomes lazy, it just keeps the same position.</p>
  <p>Is it a bad practice to use cumulative rewards as rewards? Would the daily revenue be a better reward for the agent?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-21T18:11:04.913Z
Comments.Text: Can you please describe more in detail your environment or problem? "prediction on the position on an asset in a stock trading-like environment" is not very clear to me. It's also not clear what you mean by "cumulative rewards as rewards". Maybe you mean "cumulative cash/credit/money as reward"?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><rewards><ddpg><reward-functions>
Title: Is it a bad practice to use cumulative rewards in reinforcement learning
UserDisplayName: Leibniz
ThreadText: # Is it a bad practice to use cumulative rewards in reinforcement learning
  
  (Asked by: username_0 on 2021-11-21)
  
  I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over many years of data, the reward tends to become large. I have realized that after some training the agent becomes lazy, it just keeps the same position.
  
  Is it a bad practice to use cumulative rewards as rewards? Would the daily revenue be a better reward for the agent?
  
  **Tags:** reinforcement-learning, rewards, ddpg, reward-functions
  > **Comments:**
  > **username_1**: Can you please describe more in detail your environment or problem? "prediction on the position on an asset in a stock trading-like environment" is not very clear to me. It's also not clear what you mean by "cumulative rewards as rewards". Maybe you mean "cumulative cash/credit/money as reward"?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-22. Score: 2
  
  Reinforcement learning already has the objective of maximising the sum of future expected reward.
  
  By making each reward the sum of all previous rewards, you will make the the difference between good and bad next choices low, relative to the overall reward guaranteed on each step.
  
  The best reward for the agent should be as direct measure of what you want it to achieve as possible. Sometimes you need to compromise because you cannot easily measure what you really want. Sometimes you may need to assist the bot, giving it a more frequent reward signal (e.g. change to predicted cash-out value of all the current holdings) - but you should note this is also a compromise, because an agent that maximises this "reward shaping" reward scheme may not actually maximise what you really want.
  
  You could view your attempt to use cumulative values as a failed experiment in reward shaping. I don't know if daily revenue would be a good for your bot, because a trading bot might have several different end goals for its creator.
  
  First thing you should do is figure out what your success criteria is for the bot, and find the measure for it that best captures that goal as directly as possible. E.g. if the goal is for it to create a reliable income after 5 years of trading, then the reward might be the monthly income that it able to generate in the long term. Perhaps simulating or measuring this is too hard, in which case you could fall back to a proxy, such as cash value of any sell orders with a penalty if it is not maintaining a certain amount of total of holdings.
  
  
  ---
  
  