Answers.Body: <p>Given that you are not in an episodic setting (based on the comments), I would suggest the following: accumulate rewards for a period of time <span class="math-container">$T$</span>, that is, at time <span class="math-container">$t+T$</span> do as if you had received from the environment a reward <span class="math-container">$R = \sum_{k=0}^{T-1} R_{t+k}$</span> .</p>
  <p>Then, treat <span class="math-container">$T$</span> as a hyperparameter and see which value gives the best agent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-22T19:25:57.367Z
Answers.LastActivityDate: 2022-09-22T19:25:57.367Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Body: <p>According to the question in <a href="https://ai.stackexchange.com/questions/25178/how-to-deal-with-the-time-delay-in-reinforcement-learning?newreg=2cd2066b89d84102baf346b9a13634d3">How to deal with the time delay in reinforcement learning?</a>, we can tell the delay in the reinforcement learning can be observation delay, action delay and reward delay.</p>
  <p>I have a special case of the delay but I am not sure what kind of delay is it, and how to deal with it.</p>
  <p>For example, at a state St0, my agent takes action A1, but we need to wait for a while to gain the reward R1. Meanwhile, my agent keeps taking action A2 and A3. The trick part is A2 and A3 both influence the environment and may affect the R1.</p>
  <p>So the timeline is agent plays action A1, A2, and A3, all of them being effective in the environment immediately but we need to wait for a while to see the reward R1, R2 and R3.</p>
  <p>Shall we model this question as an observation delay or reward delay?</p>
  <p>When my agent receives the R1 but not R2 and R3, can I update my Q-table by eligibility trace or any other kind of method?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-12T20:15:30.227Z
Comments.Text: Are you in an episodic setting? If so, you could pretend that you get the sum of reward as the terminal reward, i.e. $R_T = \sum_{k=0}^T R_k$. Then you can use any RL algorithm you want (rather than trying to assign partial credit to actions during the episode).
Comments.UserDisplayName: Raphael Lopez Kaufman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-13T15:38:26.720Z
Comments.Text: that is the tricky part, my  setting is for the continues job, so it may not provide a clear bound as the episodic end.
Comments.UserDisplayName: CharlesC
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-14T19:38:03.173Z
Comments.Text: Instead of trying to find the "correct" mathematical approach, I wonder if you can get good results simply by applying a sufficient delay, i.e. you sum all rewards you got over a certain period of time T and make as if you received that summed reward at timestep $t+T$. Then you can play around with the value of $T$.
Comments.UserDisplayName: Raphael Lopez Kaufman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-14T22:36:17.093Z
Comments.Text: it might work. I was considering a similar approach which whenever I got the reward, I use the reward to update all the (State,action) in the waiting buffer. This means each pair might be updated multiple times. For example, when I got R1, I updaate for A1,A2, A3 with same number; when I got R2, I update A2 and A3. But it does not make any sense in the theory.
Comments.UserDisplayName: CharlesC
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-16T17:13:35.033Z
Comments.Text: Should I add my suggestion as an answer?
Comments.UserDisplayName: Raphael Lopez Kaufman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-16T18:21:14.940Z
Comments.Text: sure and thanks for the answer. we had a good discussion
Comments.UserDisplayName: CharlesC
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-17T21:16:08.377Z
Comments.Text: You ask what kind of delay you are facing, and claim you have a "special case", but don't explain anything about the nature of the environment that causes this delayed reckoning. A lot of time you don't need to change any reinforcement learning maths, but adjust your state representation. I cannot tell if that is the case here, but might be able to help if you explain more about the environment and how you have modelled it for RL.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-17T21:16:50.667Z
Comments.Text: In the meantime, I would be interested to know if this helps:  https://ai.stackexchange.com/questions/30502/delayed-state-observation-or-caching-action-in-openai-gym-can-it-still-learn
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><q-learning><eligibility-traces><delayed-rewards>
Title: How to deal with delay in reinforcement learning, an unclear case
UserDisplayName: CharlesC
ThreadText: # How to deal with delay in reinforcement learning, an unclear case
  
  (Asked by: username_0 on 2022-09-10)
  
  According to the question in [How to deal with the time delay in reinforcement learning?](https://ai.stackexchange.com/questions/25178/how-to-deal-with-the-time-delay-in-reinforcement-learning?newreg=2cd2066b89d84102baf346b9a13634d3), we can tell the delay in the reinforcement learning can be observation delay, action delay and reward delay.
  
  I have a special case of the delay but I am not sure what kind of delay is it, and how to deal with it.
  
  For example, at a state St0, my agent takes action A1, but we need to wait for a while to gain the reward R1. Meanwhile, my agent keeps taking action A2 and A3. The trick part is A2 and A3 both influence the environment and may affect the R1.
  
  So the timeline is agent plays action A1, A2, and A3, all of them being effective in the environment immediately but we need to wait for a while to see the reward R1, R2 and R3.
  
  Shall we model this question as an observation delay or reward delay?
  
  When my agent receives the R1 but not R2 and R3, can I update my Q-table by eligibility trace or any other kind of method?
  
  **Tags:** reinforcement-learning, q-learning, eligibility-traces, delayed-rewards
  > **Comments:**
  > **username_1**: Are you in an episodic setting? If so, you could pretend that you get the sum of reward as the terminal reward, i.e. $R_T = \sum_{k=0}^T R_k$. Then you can use any RL algorithm you want (rather than trying to assign partial credit to actions during the episode).  
  > **username_0**: that is the tricky part, my  setting is for the continues job, so it may not provide a clear bound as the episodic end.  
  > **username_1**: Instead of trying to find the "correct" mathematical approach, I wonder if you can get good results simply by applying a sufficient delay, i.e. you sum all rewards you got over a certain period of time T and make as if you received that summed reward at timestep $t+T$. Then you can play around with the value of $T$.  
  > **username_0**: it might work. I was considering a similar approach which whenever I got the reward, I use the reward to update all the (State,action) in the waiting buffer. This means each pair might be updated multiple times. For example, when I got R1, I updaate for A1,A2, A3 with same number; when I got R2, I update A2 and A3. But it does not make any sense in the theory.  
  > **username_1**: Should I add my suggestion as an answer?  
  > **username_0**: sure and thanks for the answer. we had a good discussion  
  > **username_2**: You ask what kind of delay you are facing, and claim you have a "special case", but don't explain anything about the nature of the environment that causes this delayed reckoning. A lot of time you don't need to change any reinforcement learning maths, but adjust your state representation. I cannot tell if that is the case here, but might be able to help if you explain more about the environment and how you have modelled it for RL.  
  > **username_2**: In the meantime, I would be interested to know if this helps:  https://ai.stackexchange.com/questions/30502/delayed-state-observation-or-caching-action-in-openai-gym-can-it-still-learn  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-09-22. Score: 0
  
  Given that you are not in an episodic setting (based on the comments), I would suggest the following: accumulate rewards for a period of time $T$, that is, at time $t+T$ do as if you had received from the environment a reward $R = \sum\_{k=0}^{T-1} R\_{t+k}$ .
  
  Then, treat $T$ as a hyperparameter and see which value gives the best agent.
  
  
  ---
  
  