Answers.Body: <p>The update rules are not any different.</p>
  <p>However, if you make many other decisions in the meantime, the timestamps that you are able to run estimate updates for will lag behind the current timestamp.</p>
  <p>You will need a buffer of pending rewards, noting the state and action taken. You can clear the buffer, turning it into training data and running updates, once you have the matching rewards.</p>
  <p>A lot of tutorial material will use the term <span class="math-container">$Q_t(s,a)$</span> to represent the current estimate of expected reward in update rules and to drive exploitation versus exploration. In a practical system with delays you will have to use the best available estimates instead. You don't need the subscript <span class="math-container">$t$</span> on the estimation function except for plotting learning curves - if you do so you will need to decide whether you use the decision time to label the estimates, or the update time when the estimates are revised (it doesn't really matter, the graph will look the same, and mean the same thing, just with an offset).</p>
  <p>Typically you will also plot total reward or regret, or some combination. This plot will be the same as before. The impact of the reward delay will be slower initial learning and slower responsiveness to non-stationary environments, but there is no way around that.</p>
  <p>Once you ignore (or change) the subscript <span class="math-container">$t$</span> on <span class="math-container">$Q_t$</span>, then all the equations work the same as before.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-02T19:34:50.113Z
Answers.Text: Thank you. regarding "You will need a buffer of pending rewards, ........ once you matching rewards.", How to build a buffer of pending rewards. When predicting, the buffer has a associated reward and will save it in trajectory (this is a dummy reward, since actual reward will be obtained after 3 days). Now after 3 days I wanted to update the reward in the saved trajectory (to observed reward). Is there a way to update the items in the saved trajectories ?  
  can you please clarify this too if possible -> https://stackoverflow.com/questions/72090295/how-to-write-a-custom-policy-in-tf-agents
Answers.UserDisplayName: tjt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-02T19:41:33.477Z
Answers.Text: @tjt Store the partially complete records in a file or datastore until you receive the rewards and can send them to the update routine or script. How to do so will depend on the language, libraries and available infrastructure for your project. It's not an AI issue as such, but a code/design one.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-03T01:29:55.217Z
Answers.Text: Sure, that's the path I had in mind initially. but wondered if I Can do it using customer agents, environments rather than using replay buffer. but looks like using the buffer is the only way
Answers.UserDisplayName: tjt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-02T19:19:35.973Z
Answers.LastActivityDate: 2022-05-02T19:26:38.400Z
Answers.LastEditDate: 2022-05-02T19:26:38.400Z
Answers.UserDisplayName: Neil Slater
Body: <p>All the examples I see in the tf_Agents for contextual bandits, involves a reward function we generated the reward instantly after an observation has been generated.</p>
  <p>But, in my real world usecase (say sending emails and waiting for the click rate), rewards will be observed after 3 days of generating the observation. How to include this scenario of delayed rewards in training the agent ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-03T08:50:32.170Z
Comments.Text: Note that questions about how to implement something in some specific library are off-topic here. So, I recommend to remove tf_agents from the title to remove the emphasis on the library, although you could ask, as an additional question, if someone knows how to do it in that particular library, but that shouldn't be your main question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-03T16:05:40.940Z
Comments.Text: @nbro sure done
Comments.UserDisplayName: tjt
Tags: <reinforcement-learning><tensorflow><experience-replay><contextual-bandits>
Title: How to handled delayed rewards in contextual bandits
UserDisplayName: tjt
ThreadText: # How to handled delayed rewards in contextual bandits
  
  (Asked by: username_0 on 2022-05-02)
  
  All the examples I see in the tf\_Agents for contextual bandits, involves a reward function we generated the reward instantly after an observation has been generated.
  
  But, in my real world usecase (say sending emails and waiting for the click rate), rewards will be observed after 3 days of generating the observation. How to include this scenario of delayed rewards in training the agent ?
  
  **Tags:** reinforcement-learning, tensorflow, experience-replay, contextual-bandits
  
  > **Comments:**
  > **username_1**: Note that questions about how to implement something in some specific library are off-topic here. So, I recommend to remove tf_agents from the title to remove the emphasis on the library, although you could ask, as an additional question, if someone knows how to do it in that particular library, but that shouldn't be your main question.  
  > **username_0**: @username_1 sure done  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-05-02. Score: 2
  
  The update rules are not any different.
  
  However, if you make many other decisions in the meantime, the timestamps that you are able to run estimate updates for will lag behind the current timestamp.
  
  You will need a buffer of pending rewards, noting the state and action taken. You can clear the buffer, turning it into training data and running updates, once you have the matching rewards.
  
  A lot of tutorial material will use the term $Q\_t(s,a)$ to represent the current estimate of expected reward in update rules and to drive exploitation versus exploration. In a practical system with delays you will have to use the best available estimates instead. You don't need the subscript $t$ on the estimation function except for plotting learning curves - if you do so you will need to decide whether you use the decision time to label the estimates, or the update time when the estimates are revised (it doesn't really matter, the graph will look the same, and mean the same thing, just with an offset).
  
  Typically you will also plot total reward or regret, or some combination. This plot will be the same as before. The impact of the reward delay will be slower initial learning and slower responsiveness to non-stationary environments, but there is no way around that.
  
  Once you ignore (or change) the subscript $t$ on $Q\_t$, then all the equations work the same as before.
  
  > **Comments:**
  > **username_0**: Thank you. regarding "You will need a buffer of pending rewards, ........ once you matching rewards.", How to build a buffer of pending rewards. When predicting, the buffer has a associated reward and will save it in trajectory (this is a dummy reward, since actual reward will be obtained after 3 days). Now after 3 days I wanted to update the reward in the saved trajectory (to observed reward). Is there a way to update the items in the saved trajectories ?   can you please clarify this too if possible -> https://stackoverflow.com/questions/72090295/how-to-write-a-custom-policy-in-tf-agents  
  > **username_2**: @username_0 Store the partially complete records in a file or datastore until you receive the rewards and can send them to the update routine or script. How to do so will depend on the language, libraries and available infrastructure for your project. It's not an AI issue as such, but a code/design one.  
  > **username_0**: Sure, that's the path I had in mind initially. but wondered if I Can do it using customer agents, environments rather than using replay buffer. but looks like using the buffer is the only way  
  
  ---
  
  