Answers.Body: <p>I think what you have here (with an important caveat, which I will get to later) is a common misunderstanding about how rewards should be structured for a reinforcement learning (RL) problem. It is <em>perfectly OK</em> to return a reward of <span class="math-container">$0$</span> for most steps, and only return measured rewards when the event they are related to happens.</p>
  <p>The task of figuring out that the critical action that caused a reward was some number of time steps ago is called <a href="https://ai.stackexchange.com/questions/12908/what-is-the-credit-assignment-problem">the credit assignment problem</a>. Sometimes this can make solving an RL problem hard. However, it is a core issue in RL, and <em>all</em> RL algorithms are built to solve this issue. This is usually achieved through algorithms that <em>backup</em> through time iteratively, assigning expected future values to earlier states (in value-based algorithms), or by associating <em>all</em> actions taken with rewards that occured in their future and allowing this to average out over multiple different experiences (in policy gradient methods).</p>
  <p>The important caveat is this: When the agent takes the earlier action, this choice must have had some observable influence on the <em>state</em>, and that change should still persist in some form and to some degree at the time the associated reward is observed, such that it is possible for the agent to make the association. In some cases - e.g. a game of checkers - then the state change appears naturally in how the environment is expressed. In other cases you may need to take extra steps. In the worst-case scenario, you would need to store a history of actions taken and add that to the state vector.</p>
  <p>Most simple evironments provided <em>with</em> Gym should provide you with a fully usable state vector where all influences on next state and reward are included. So you may need to do nothing other than apply a standard RL control method. Custom, or more advanced environments may require that you add something extra to the state yourself.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-05T22:51:46.407Z
Answers.Text: Exactly, you say it very well. I upvoted your answer
Answers.UserDisplayName: Zollikofen4
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-22T21:42:38.297Z
Answers.LastActivityDate: 2023-11-22T21:42:38.297Z
Answers.UserDisplayName: Neil Slater
Body: <p>My question is about whether is it possible to implement delayed reward logic within Gym environment.</p>
  <p>More specifically, I work on ride-pooling RL algorithm, when the action (choice of the parameters of ride-pooling heuristic) is taken once in several simulation iterations. However, trips that are influenced by this action may finish later than next Gym step starts. I am wondering if this is possible to send reward back to one of the previous Gym steps when it will eventually be calculated.</p>
  <p>For example, we made first action at simulation iteration 10, second action at 20, third action at 30, and only at 35 the reward for the first action is ready.</p>
  <p>In basic implementation of Gym step() one needs to return rewards in the end of the step.</p>
  
Tags: <reinforcement-learning><implementation><gym>
Title: OpenAI Gym implementation of the delayed rewards
UserDisplayName: Klavdiia Bochenina
ThreadText: # OpenAI Gym implementation of the delayed rewards
  
  (Asked by: username_0 on 2023-11-22)
  
  My question is about whether is it possible to implement delayed reward logic within Gym environment.
  
  More specifically, I work on ride-pooling RL algorithm, when the action (choice of the parameters of ride-pooling heuristic) is taken once in several simulation iterations. However, trips that are influenced by this action may finish later than next Gym step starts. I am wondering if this is possible to send reward back to one of the previous Gym steps when it will eventually be calculated.
  
  For example, we made first action at simulation iteration 10, second action at 20, third action at 30, and only at 35 the reward for the first action is ready.
  
  In basic implementation of Gym step() one needs to return rewards in the end of the step.
  
  **Tags:** reinforcement-learning, implementation, gym
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-22. Score: 2
  
  I think what you have here (with an important caveat, which I will get to later) is a common misunderstanding about how rewards should be structured for a reinforcement learning (RL) problem. It is *perfectly OK* to return a reward of $0$ for most steps, and only return measured rewards when the event they are related to happens.
  
  The task of figuring out that the critical action that caused a reward was some number of time steps ago is called [the credit assignment problem](https://ai.stackexchange.com/questions/12908/what-is-the-credit-assignment-problem). Sometimes this can make solving an RL problem hard. However, it is a core issue in RL, and *all* RL algorithms are built to solve this issue. This is usually achieved through algorithms that *backup* through time iteratively, assigning expected future values to earlier states (in value-based algorithms), or by associating *all* actions taken with rewards that occured in their future and allowing this to average out over multiple different experiences (in policy gradient methods).
  
  The important caveat is this: When the agent takes the earlier action, this choice must have had some observable influence on the *state*, and that change should still persist in some form and to some degree at the time the associated reward is observed, such that it is possible for the agent to make the association. In some cases - e.g. a game of checkers - then the state change appears naturally in how the environment is expressed. In other cases you may need to take extra steps. In the worst-case scenario, you would need to store a history of actions taken and add that to the state vector.
  
  Most simple evironments provided *with* Gym should provide you with a fully usable state vector where all influences on next state and reward are included. So you may need to do nothing other than apply a standard RL control method. Custom, or more advanced environments may require that you add something extra to the state yourself.
  
  > **Comments:**
  > **username_2**: Exactly, you say it very well. I upvoted your answer  
  
  ---
  
  