Answers.Body: <p>The pragmatic and important K-arm dueling bandit problem is not easy and resistant to perfectly fit into the traditional RL framework because the feedback is pairwise preference rather than absolutely quantifiable scalar reward to form a meaningful upper bounded regret. <a href="https://www.cs.cornell.edu/people/tj/publications/yue_etal_09a.pdf" rel="nofollow noreferrer">Interleaved Filter Algorithms</a> are tailored to solve the dueling bandit problem which is a kind of <em>partial information</em> online learning problem.</p>
  <blockquote>
  <p>One important issue is formulating an appropriate notion of regret... a natural question to ask is whether users, at each iteration, would have prefered another bandit over the ones chosen by our algorithm. This leads directly to our regret formulation, which measures regret based on the (initially unknown) probability that the best bandit would win a comparison with the chosen bandits at each iteration... our results provide theoretical support for Langford’s observation about a qualitative difference between algorithms for supervised learning and those for learning from partial observations: in the supervised setting, “holistic information is often better,” whereas in the setting of partial observations it is often better to select a few points and observe them many times while giving
  scant attention to other points.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-10T07:40:13.390Z
Answers.LastActivityDate: 2024-12-10T07:40:13.390Z
Answers.UserDisplayName: cinch
Body: <p>The typical history format of RL is State, Action, Reward (single scalar value), state and action.  In dueling bandit the agent chooses two arms and is only told which has the higher reward. This does not fit the Reward spot directly.  So how should this be modeled?</p>
  <p>The core of most bandit problems is the agent seeks to always choose the optimal arm.  One might model it similar to a binary bandit, if the reward from the first selected arm is higher, reward of 1, zero otherwise.</p>
  <p>Is there an accepted way to adapt this problem to the RL history framing or any other ideas how it might be?</p>
  
Tags: <multi-armed-bandits>
Title: Modeling dueling bandits in a normal RL history format
UserDisplayName: foreverska
ThreadText: # Modeling dueling bandits in a normal RL history format
  
  (Asked by: username_0 on 2024-12-09)
  
  The typical history format of RL is State, Action, Reward (single scalar value), state and action. In dueling bandit the agent chooses two arms and is only told which has the higher reward. This does not fit the Reward spot directly. So how should this be modeled?
  
  The core of most bandit problems is the agent seeks to always choose the optimal arm. One might model it similar to a binary bandit, if the reward from the first selected arm is higher, reward of 1, zero otherwise.
  
  Is there an accepted way to adapt this problem to the RL history framing or any other ideas how it might be?
  
  **Tags:** multi-armed-bandits
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-10. Score: 2 | Accepted answer
  
  The pragmatic and important K-arm dueling bandit problem is not easy and resistant to perfectly fit into the traditional RL framework because the feedback is pairwise preference rather than absolutely quantifiable scalar reward to form a meaningful upper bounded regret. [Interleaved Filter Algorithms](https://www.cs.cornell.edu/people/tj/publications/yue_etal_09a.pdf) are tailored to solve the dueling bandit problem which is a kind of *partial information* online learning problem.
  
  > One important issue is formulating an appropriate notion of regret... a natural question to ask is whether users, at each iteration, would have prefered another bandit over the ones chosen by our algorithm. This leads directly to our regret formulation, which measures regret based on the (initially unknown) probability that the best bandit would win a comparison with the chosen bandits at each iteration... our results provide theoretical support for Langford’s observation about a qualitative difference between algorithms for supervised learning and those for learning from partial observations: in the supervised setting, “holistic information is often better,” whereas in the setting of partial observations it is often better to select a few points and observe them many times while giving
  > scant attention to other points.
  
  
  ---
  
  