Body: <p>Lets take an ad recommendation problem for 1 slot. Feedback is click/no click. I can solve this by contextual bandits. But I can also introduce exploration in supervised learning, I learn my model from collected data every k hours.</p>
  <p>What can contextual bandits give me in this example which supervised learning + exploration cannot do?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-29T12:37:49.060Z
Comments.Text: Hello. Can you please clarify what you mean by "I can also introduce exploration in supervised learning"? What's your idea more specifically?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-30T02:52:21.627Z
Comments.Text: I will learn a supervised model from the logging policy/model 's data and I will use it in a stochastic way which will promote exploration. If I do this iteratively(i.e. learn model with IPW supervised loss and collect data with exploration), i hope i can achieve what a batch learning with bandit feedback setup will also achieve. I will use inverse propensity weights in bandits and supervised learning loss as well. 
  
  then my question is that for this specific problem is, will bandit give me anything i cannot achieve with a IPW supervised loss.
Comments.UserDisplayName: dksahuji
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-30T02:59:03.080Z
Comments.Text: the whole point is working in a practical setup and trying to learn from partial observations i.e. in recommendation you dont have data for what you dont recommend. 
  
  learned stuff from this ( http://www.cs.cornell.edu/courses/cs7792/2018fa/ ) . batch learning with bandit feedback seems exciting to me. But how do i convince myself that for simple problem as explained in the question exploration plus inverse propensity weighted supervised loss is equal to a batch learned bandit. IPW supervised loss will obviously require less iterations over data.
Comments.UserDisplayName: dksahuji
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-30T03:05:42.900Z
Comments.Text: In the example in question description, i can have a softmax over candidates for that slot. exploration will let me try different candidates and hence collecting data by exploration.
Comments.UserDisplayName: dksahuji
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-20T12:21:44.857Z
Comments.Text: I'm wondering the same question. Bandit is not "an algorithm", it is a modeling setup. Your bandit will follow a policy π. 
  You say your policy π is based on a supervised model (disregard exploration for now). I believe that your supervision signal (click) would directly translate to a reward that you get at time t. In non-bandit agents, RL would optimize for long term future rewards. Bandit is a one-episode game, so I think supervised learning is equivalent to RL.  The benefits you get by posing problem as bandit is you can reason about and control your exploration
Comments.UserDisplayName: islijepcevic
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-07T09:45:12.983Z
Comments.Text: No. If I am using IPW weighted supervised loss then actually cross-entropy is my negative reward and click is consumed in the loss.
Comments.UserDisplayName: dksahuji
Tags: <supervised-learning><contextual-bandits><exploration-strategies>
Title: (explore-exploit + supervised learning ) vs contextual bandits
UserDisplayName: dksahuji
ThreadText: # (explore-exploit + supervised learning ) vs contextual bandits
  
  (Asked by: username_0 on 2021-06-29)
  
  Lets take an ad recommendation problem for 1 slot. Feedback is click/no click. I can solve this by contextual bandits. But I can also introduce exploration in supervised learning, I learn my model from collected data every k hours.
  
  What can contextual bandits give me in this example which supervised learning + exploration cannot do?
  
  **Tags:** supervised-learning, contextual-bandits, exploration-strategies
  
  > **Comments:**
  > **username_1**: Hello. Can you please clarify what you mean by "I can also introduce exploration in supervised learning"? What's your idea more specifically?  
  > **username_0**: I will learn a supervised model from the logging policy/model 's data and I will use it in a stochastic way which will promote exploration. If I do this iteratively(i.e. learn model with IPW supervised loss and collect data with exploration), i hope i can achieve what a batch learning with bandit feedback setup will also achieve. I will use inverse propensity weights in bandits and supervised learning loss as well.   then my question is that for this specific problem is, will bandit give me anything i cannot achieve with a IPW supervised loss.  
  > **username_0**: the whole point is working in a practical setup and trying to learn from partial observations i.e. in recommendation you dont have data for what you dont recommend.   learned stuff from this ( http://www.cs.cornell.edu/courses/cs7792/2018fa/ ) . batch learning with bandit feedback seems exciting to me. But how do i convince myself that for simple problem as explained in the question exploration plus inverse propensity weighted supervised loss is equal to a batch learned bandit. IPW supervised loss will obviously require less iterations over data.  
  > **username_0**: In the example in question description, i can have a softmax over candidates for that slot. exploration will let me try different candidates and hence collecting data by exploration.  
  > **username_2**: I'm wondering the same question. Bandit is not "an algorithm", it is a modeling setup. Your bandit will follow a policy π.  You say your policy π is based on a supervised model (disregard exploration for now). I believe that your supervision signal (click) would directly translate to a reward that you get at time t. In non-bandit agents, RL would optimize for long term future rewards. Bandit is a one-episode game, so I think supervised learning is equivalent to RL.  The benefits you get by posing problem as bandit is you can reason about and control your exploration  
  > **username_0**: No. If I am using IPW weighted supervised loss then actually cross-entropy is my negative reward and click is consumed in the loss.  
  
  ---
  
  ## 0 answers
  
  