Multi-armed Bandits with Compensation
We propose and study the known-compensation multi-armed bandit (KCMAB) problem, where a system controller offers a set of arms to many short-term players for T steps. In each step, one short-term player arrives at the system. Upon arrival, the player aims to select an arm with the current best average reward and receives a stochastic reward associated with the arm. In order to incentivize players to explore other arms, the controller provide proper payment compensations to players. The objective of the controller is to maximize the total reward collected by players while minimizing the total compensation. We first provide a compensation lower bound ⇥( P i i log T KLi ), where i and KL i are the expected reward gap and the Kullback-Leibler (KL) divergence between distributions of arm i and the best arm, respectively. We then analyze three algorithms for solving the KCMAB problem, and obtain their regrets and compensations. We show that the algorithms all achieve O(log T ) regret and O(log T ) compensation that match the theoretical lower bounds. Finally, we present experimental results to demonstrate the performance of the algorithms.
This paper is about multi-armed bandits. The twist is that the learner does not control directly the actions. Instead, in each round the learner chooses an amount of 'compensation' associated to each arm and the user chooses the arm for which the sum of the compensation and empirical mean is largest. The objective is to design algorithms for which the cumulative regret (with the normal definition) and cumulative compensation both have logarithmic growth.  The claimed results are as follows: (a) A lower bound showing that algorithms with subpolynomial regret necessarily must necessarily pay logarithmic cumulative compensation with the usual constants on *some* bandit problems.  (b) Upper bounds on the cumulative compensation for UCB, epsilon-greedy and a variant of Thompson sampling with appropriately chosen compensation levels.  To illustrate, the most obvious idea of offering the difference between the UCB and empirical mean as compensation leads to a sqrt(n) growth rate of the compensation, so something a little clever is needed. The authors point out the compensation is not required if the empirical mean of the arm chosen by UCB is the largest and then prove this happens with sufficiently few exceptions.  By the way, one might wonder if the DMED-style algorithm (Honda and Takamura, 2010) might not be adapted even more straightforwardly.  I have two reservations about this paper. First, the proofs are much more complicated than necessary. This is especially true of the lower bound, the proof of which is three pages. I include a suggestion for how to considerably shorten this proof below. Note that this sketch does not depend on the first arm having a mean of at least 0.9 and applies to all bandit problems, like Lai & Robbins result.  My second reservation is that the paper is not very cleanly written. There are many typos and the sketches are hard to follow. The proofs in the appendix even harder. And yet the claimed results do not seem especially surprising. I think a paper like this could be accepted at NIPS, but the analysis should be crisp and insightful.   Regarding the originality and significance. The setting is new to my knowledge. The techniques are not especially startling. The significance is hard to judge. There is some motivation given, but this is obviously a first step. In practical situations things would almost always be contextualized and so-on. Still, I think this aspect is Ok.   A few other comments:  1. I was wondering if perhaps you can prove a more generic result. Given /any/ algorithm can I design a black-box modification for which the compensation bound has the same order of the regret for that algorithm?  2. Perhaps you can get the constants right in the lower bound and upper bound. See below for the former. For the latter you could use the techniques from the KL-UCB algorithm.  3. All of the upper bounds involve simple modification of existing algorithms. Essentially what is required is choosing the compensation so that the behavior is the same as what you would expect in the standard setting. The lower bounds show you cannot do much better than this, but one still might wonder if the algorithms ought to change a little in the new setting. Mathematically it seems a pity the setting is not especially rich.  Overall the paper is below the bar for NIPS.   ********************************************** * LOWER BOUND SKETCH **********************************************  By Lai & Robbins result it holds that  P(N_i(T) >= (1-e) / KL log(n)) >= 1 - delta(T)  with delta(T) tending to 0.  Let epsilon > 0. I claim there exists a constant m that depends on the bandit, but not T such that the following four events hold jointly with probability at least 1/2 for all sufficiently large T: (a) N_i(T) >= (1 - e) / KL log(T) for all i (b) N_1(2m) >= m (c) Whenever N_1(t) >= m, then the empirical mean of arm 1 is at least mu_1 - epsilon (d) Whenever N_i(t) >= m, then the empirical mean of arm i is a most mu_i + epsilon for all i  Then since m does not depend on T we may take the limit as T tends to infinity when on the four events above it the algorithm must pay at least Delta_i - 2 epsilon for at least (1 - e) / KL log(T) - m rounds, which yields the desired form of the lower bound.  The claim is proven easily via a union bound. (a) using Lai & Robbins result, (b) by Markov's inequality and the assumption on subpolynomial regret. (c) and (d) by Hoeffding's bound and a union bound (Chernoff's is not required).    *********************** * MINORS ***********************  * The inequality in the sketch of Theorem 1 is the wrong way. * \Theta(f(n)) means that f(n) = O(f(n)) and f(n) = Omega(f(n)), so the claim of Theorem 1 is not accurate. In general I prefer not to see this notation in theorem statements.