Answers.Body: <p>Your function could be called the <em>truncated return</em> - i.e. the sum of rewards up to some time step in the future.</p>
  <p>It would be unusual to perform reward shaping by taking the orginal reward from an environment and replacing it with the truncated return. It definitely breaks some things, so is not a general solution. The broken things include:</p>
  <ul>
  <li>Off-policy value estimations. The truncated returns would include results from future decisions that the behavior policy made, but the target policy would not make. By &quot;squashing&quot; the return into the new observed reward, you would prevent use of any approach, such as importance sampling, to correct for this mismatch.</li>
  <li>An environment with important rewards before step <span class="math-container">$n$</span>. These early rewards will be under-represented in the new full returns (i.e. returns calculated on your transformed environment), and this could change which early actions were optimal, changing the environment so that the new agent that is optimal in your reward-shaped environment would make different decisions to the optimal agent in the original environment.</li>
  </ul>
  <p>In comments you would use the calculated truncated return from the original environment as the reward signal observed by the PPO algorithm (effectively in a transformed or reward-shaped version of the environment). This is not necessary, because PPO <em>already</em> calculates and uses estimated returns from the observed rewards. This is true for <em>all</em> RL algorithms - they all take the observed return, or calculate an approximate expected return, or sometimes a value derived from the return, such as the advantage function. PPO uses a variation of the advantage function.</p>
  <p>Effectively, other that the above two problem cases above, what you would be doing for most on-policy algorithms is multiplying each reward by a factor <span class="math-container">$\sum_{k=0}^n \gamma^k$</span>, because the same immediate reward will be associated with every state, action pair that came before it, going back n steps. So, this should still work, but I don't think it will transform a problem to make it easier to solve for the agent. It is not much different, if you avoid the problem scenarios, from changing the learning rate.</p>
  <p>Some RL agents do use the truncated return internally. It <a href="https://lcalem.github.io/blog/2019/02/25/sutton-chap12#123-n-step-truncated-lambda-return-methods" rel="nofollow noreferrer">can be used for eligibility traces</a> for example.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-18T16:03:11.597Z
Answers.Text: Thanks for the explanation. I see your point explaining that this smoothing is not necessary for any RL algorithms, I just thought it would help.
  Thanks for the remark concerning off-policy value estimation, I didn't think of that.
Answers.UserDisplayName: Georg Schneeberger
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-11T11:23:46.407Z
Answers.LastActivityDate: 2023-12-11T11:30:06.020Z
Answers.LastEditDate: 2023-12-11T11:30:06.020Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have a problem with continuous time, observation and action space. I am discretizing the time to be able to apply the usual Reinforcement Learning algorithms (I chose PPO). The problem consists of a simulated car that has to drive through a series of goals without collisions.</p>
  <p>I read chapter 7 of the Reinforcement Learning Book by Sutton and Barto: <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">http://incompleteideas.net/book/the-book-2nd.html</a></p>
  <p>In chapter 7 they describe n-step bootstrapping which uses the rewards of the next n-steps and a value estimate of the nth step to update the value of the current step t. This describes extensions to reinforcement learning algorithms, I would like to use an approach inspired by the n-step bootstrapping.</p>
  <p>I would like to use a related approach to smooth my reward function. Unfortunately I am not sure how that approach is called and if it has already been talked about in research.</p>
  <p>I am searching a name or some research regarding the following reward function:</p>
  <p><a href="https://i.sstatic.net/G8doC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/G8doC.png" alt="SmoothedReward with discount factor gamma" /></a></p>
  <p>Reward(st, at) is simply the reward encountered at timestep t. I would then hand the SmoothedReward to the RL algorithm (PPO).</p>
  <p>Can you please direct me to research about something like my smoothed reward function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-10T09:58:25.880Z
Comments.Text: n-step bootstrapping is a way to approximate the expected return, and has the theoretical reasoning of being an estimate (combination of sampling and stats sumary) of the true value. What do you intend your "smoothed reward" being used to substitute (are you simply using your reward sum instead of the reward), and what is your thinking for justifying its use in PPO?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-11T08:54:52.850Z
Comments.Text: I would hand this smoothed reward to the PPO training algorithm instead of the simple Reward(s_t,a_t). 
  
  (Yes simple this discounted reward sum instead of the standard reward)
  
  
  
  I think this could be useful in my problem since rewards that are caused by the current iteration might only be awarded a bit later.
Comments.UserDisplayName: Georg Schneeberger
Tags: <reinforcement-learning><reference-request><reward-functions><reward-design><reward-shaping>
Title: What is the name of the reward function that utilizes the rewards of the next n steps?
UserDisplayName: Georg Schneeberger
ThreadText: # What is the name of the reward function that utilizes the rewards of the next n steps?
  
  (Asked by: username_0 on 2023-12-10)
  
  I have a problem with continuous time, observation and action space. I am discretizing the time to be able to apply the usual Reinforcement Learning algorithms (I chose PPO). The problem consists of a simulated car that has to drive through a series of goals without collisions.
  
  I read chapter 7 of the Reinforcement Learning Book by Sutton and Barto: <http://incompleteideas.net/book/the-book-2nd.html>
  
  In chapter 7 they describe n-step bootstrapping which uses the rewards of the next n-steps and a value estimate of the nth step to update the value of the current step t. This describes extensions to reinforcement learning algorithms, I would like to use an approach inspired by the n-step bootstrapping.
  
  I would like to use a related approach to smooth my reward function. Unfortunately I am not sure how that approach is called and if it has already been talked about in research.
  
  I am searching a name or some research regarding the following reward function:
  
  [![SmoothedReward with discount factor gamma](https://i.sstatic.net/G8doC.png)](https://i.sstatic.net/G8doC.png)
  
  Reward(st, at) is simply the reward encountered at timestep t. I would then hand the SmoothedReward to the RL algorithm (PPO).
  
  Can you please direct me to research about something like my smoothed reward function?
  
  **Tags:** reinforcement-learning, reference-request, reward-functions, reward-design, reward-shaping
  
  > **Comments:**
  > **username_1**: n-step bootstrapping is a way to approximate the expected return, and has the theoretical reasoning of being an estimate (combination of sampling and stats sumary) of the true value. What do you intend your "smoothed reward" being used to substitute (are you simply using your reward sum instead of the reward), and what is your thinking for justifying its use in PPO?  
  > **username_0**: I would hand this smoothed reward to the PPO training algorithm instead of the simple Reward(s_t,a_t).   (Yes simple this discounted reward sum instead of the standard reward)    I think this could be useful in my problem since rewards that are caused by the current iteration might only be awarded a bit later.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-11. Score: 1 | Accepted answer
  
  Your function could be called the *truncated return* - i.e. the sum of rewards up to some time step in the future.
  
  It would be unusual to perform reward shaping by taking the orginal reward from an environment and replacing it with the truncated return. It definitely breaks some things, so is not a general solution. The broken things include:
  
  * Off-policy value estimations. The truncated returns would include results from future decisions that the behavior policy made, but the target policy would not make. By "squashing" the return into the new observed reward, you would prevent use of any approach, such as importance sampling, to correct for this mismatch.
  * An environment with important rewards before step $n$. These early rewards will be under-represented in the new full returns (i.e. returns calculated on your transformed environment), and this could change which early actions were optimal, changing the environment so that the new agent that is optimal in your reward-shaped environment would make different decisions to the optimal agent in the original environment.
  
  In comments you would use the calculated truncated return from the original environment as the reward signal observed by the PPO algorithm (effectively in a transformed or reward-shaped version of the environment). This is not necessary, because PPO *already* calculates and uses estimated returns from the observed rewards. This is true for *all* RL algorithms - they all take the observed return, or calculate an approximate expected return, or sometimes a value derived from the return, such as the advantage function. PPO uses a variation of the advantage function.
  
  Effectively, other that the above two problem cases above, what you would be doing for most on-policy algorithms is multiplying each reward by a factor $\sum\_{k=0}^n \gamma^k$, because the same immediate reward will be associated with every state, action pair that came before it, going back n steps. So, this should still work, but I don't think it will transform a problem to make it easier to solve for the agent. It is not much different, if you avoid the problem scenarios, from changing the learning rate.
  
  Some RL agents do use the truncated return internally. It [can be used for eligibility traces](https://lcalem.github.io/blog/2019/02/25/sutton-chap12#123-n-step-truncated-lambda-return-methods) for example.
  
  > **Comments:**
  > **username_0**: Thanks for the explanation. I see your point explaining that this smoothing is not necessary for any RL algorithms, I just thought it would help. Thanks for the remark concerning off-policy value estimation, I didn't think of that.  
  
  ---
  
  