Answers.Body: <p>Say you have 2 deterministic slots, one that always returns 1 and the second that always returns 2 as rewards</p>
  <p>Your behavioural policy selects the first slot machines 80% of the times, thus the second one the remaining 20% of the times.</p>
  <p>Now, say you do 100 pulls, if you start with a uniform policy, you will do <span class="math-container">$1 \nabla \ln p(a_1)$</span> 80 times, and <span class="math-container">$2 \nabla \ln p(a_2)$</span> only 20 times, in other words, you get <span class="math-container">$80 \cdot (1 \nabla \ln p(a_1))$</span>, and <span class="math-container">$20(2 \nabla \ln p(a_2))$</span> as reinforcement... I'm pretty sure that your policy will think that option 1 is the best</p>
  <p>And that thinking that <span class="math-container">$a_1$</span> is the best, is only due to the bias that you are not correcting from the sampling</p>
  <p>PSA: there exists the off-policy counterpart for REINFORCE that corrects for this bias (is always, and just, an importance sampling ratio correction as always)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-26T12:22:54.993Z
Answers.Text: Thank you very much Alberto for your answer. However, I think this is handled by substracting a baseline from the return (e.g., the average reward). That is the reason I mention "relative rewards" in my post. Thus, by substracting the average reward, the action with 1 reward will actually have negative relative return.
Answers.UserDisplayName: gnikol
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-26T15:15:20.457Z
Answers.Text: @gnikol that’s just a case, take 3 bandits, with deterministic reward 10,5,-50 and a behavioural with policy 5%, 80%, 15%, the baseline is 1 (if i’ve not messed up the math) and the most reinforced one is the second action by a large margin, and thus you will end up with such action as the best even in the off policy one
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-27T22:18:12.690Z
Answers.Text: that makes sense. Thank you very much for your help. Except importance sampling are there any other methods to correct for this bias?
Answers.UserDisplayName: gnikol
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-28T09:55:11.073Z
Answers.Text: @gnikol yeah i also hate it, but never found alternative methods
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-26T11:19:49.277Z
Answers.LastActivityDate: 2024-05-26T11:19:49.277Z
Answers.UserDisplayName: Alberto
Body: <p>I would like to bring up a point regarding the application of on-policy algorithms, such as REINFORCE, to contextual bandit problems using data collected from other policies.</p>
  <p>Here are my thoughts:</p>
  <p>In a contextual bandit setting, the goal is to choose the action that maximizes the immediate reward given the current context. Unlike full reinforcement learning problems, there is no concern for future rewards or transitions between states.</p>
  <p>My thought is that the focus on immediate rewards aligns well with the assumption that the policy used to collect the data may not play a role as in more complex RL settings involving state transitions and future rewards and thus, data collected from any policy can be used to update the policy being optimized.</p>
  <p>However, on-policy algorithms like REINFORCE typically require data collected from the same policy that is being optimized. This ensures that the updates are directly related to the policy’s performance.</p>
  <p>On the other hand, in REINFORCE, the policy is updated by making the &quot;good&quot; actions, the ones with the higher relative returns, more probable. In full RL context the returns are the sum of the cumulative rewards produced by following the policy that is being optimized. But, in contextual bandit context we only have immediate rewards and thus it is not necessary to use data collected by the policy being optimized since we do not have future rewards.</p>
  <p>However, the primary concern I have with using off-policy data in on-policy algorithms is that the data may not accurately reflect the performance of the current policy being optimized and introduce potential biases and inaccuracies.</p>
  
Tags: <policy-gradients><off-policy-methods><reinforce><on-policy-methods><contextual-bandits>
Title: Does the off-policy/on-policy regime plays role in Contextual Bandits?
UserDisplayName: gnikol
ThreadText: # Does the off-policy/on-policy regime plays role in Contextual Bandits?
  
  (Asked by: username_0 on 2024-05-25)
  
  I would like to bring up a point regarding the application of on-policy algorithms, such as REINFORCE, to contextual bandit problems using data collected from other policies.
  
  Here are my thoughts:
  
  In a contextual bandit setting, the goal is to choose the action that maximizes the immediate reward given the current context. Unlike full reinforcement learning problems, there is no concern for future rewards or transitions between states.
  
  My thought is that the focus on immediate rewards aligns well with the assumption that the policy used to collect the data may not play a role as in more complex RL settings involving state transitions and future rewards and thus, data collected from any policy can be used to update the policy being optimized.
  
  However, on-policy algorithms like REINFORCE typically require data collected from the same policy that is being optimized. This ensures that the updates are directly related to the policy’s performance.
  
  On the other hand, in REINFORCE, the policy is updated by making the "good" actions, the ones with the higher relative returns, more probable. In full RL context the returns are the sum of the cumulative rewards produced by following the policy that is being optimized. But, in contextual bandit context we only have immediate rewards and thus it is not necessary to use data collected by the policy being optimized since we do not have future rewards.
  
  However, the primary concern I have with using off-policy data in on-policy algorithms is that the data may not accurately reflect the performance of the current policy being optimized and introduce potential biases and inaccuracies.
  
  **Tags:** policy-gradients, off-policy-methods, reinforce, on-policy-methods, contextual-bandits
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-26. Score: 1 | Accepted answer
  
  Say you have 2 deterministic slots, one that always returns 1 and the second that always returns 2 as rewards
  
  Your behavioural policy selects the first slot machines 80% of the times, thus the second one the remaining 20% of the times.
  
  Now, say you do 100 pulls, if you start with a uniform policy, you will do $1 \nabla \ln p(a\_1)$ 80 times, and $2 \nabla \ln p(a\_2)$ only 20 times, in other words, you get $80 \cdot (1 \nabla \ln p(a\_1))$, and $20(2 \nabla \ln p(a\_2))$ as reinforcement... I'm pretty sure that your policy will think that option 1 is the best
  
  And that thinking that $a\_1$ is the best, is only due to the bias that you are not correcting from the sampling
  
  PSA: there exists the off-policy counterpart for REINFORCE that corrects for this bias (is always, and just, an importance sampling ratio correction as always)
  
  > **Comments:**
  > **username_0**: Thank you very much Alberto for your answer. However, I think this is handled by substracting a baseline from the return (e.g., the average reward). That is the reason I mention "relative rewards" in my post. Thus, by substracting the average reward, the action with 1 reward will actually have negative relative return.  
  > **username_1**: @gnikol that’s just a case, take 3 bandits, with deterministic reward 10,5,-50 and a behavioural with policy 5%, 80%, 15%, the baseline is 1 (if i’ve not messed up the math) and the most reinforced one is the second action by a large margin, and thus you will end up with such action as the best even in the off policy one  
  > **username_0**: that makes sense. Thank you very much for your help. Except importance sampling are there any other methods to correct for this bias?  
  > **username_1**: @gnikol yeah i also hate it, but never found alternative methods  
  
  ---
  
  