A KL-LUCB Bandit Algorithm for Large-Scale Crowdsourcing 1 Multi-Armed Bandits for Large-Scale Crowdsourcing
This paper focuses on best-arm identification in multi-armed bandits with bounded rewards. We develop an algorithm that is a fusion of lil-UCB and KL-LUCB, offering the best qualities of the two algorithms in one method. This is achieved by proving a novel anytime confidence bound for the mean of bounded distributions, which is the analogue of the LIL-type bounds recently developed for sub-Gaussian distributions. We corroborate our theoretical results with numerical experiments based on the New Yorker Cartoon Caption Contest.
The paper proposes a novel PAC algorithm for the multi-armed bandit problem that combines ideas from KL-LUCB and lil-UCB. For the proposed algorithm (called lil-KLUCB) the authors show a sample complexity upper bound that improves on the corresponding result of both its predecessors. The core of this result is a novel concentration bound which uses the Chernoff-information - similiarly as in case of KL-LUCB, but this new approach results in a much tighter bound. Finally, the paper presents some experimental results demonstrating that the theoretical improvement has practical implications as well.
The problem I have found with the paper is that the proof of Theorem 1 contains errors that make it completely confusing, and thereby the correctness of the results remains unclear. What is exactly A_k and t_j? Their current definition do not makes sense: why should t in the definition of A_k run over the whole set of natural numbers? And why should t_k be bigger than, but still proportional to 2^k? And which part of the proof corresponds to the first term on the RHS of the equation below line 135, and which part to the rest of the terms?
Additionally, the proof sketch of Theorem 1 is is rather messy too (unfortunately, I was not able to check the corresponding part in the supplementary material):
- In line 180, what is U(f_t(\epsilon))? It cannot be deduced from the definition of U(t,\epsilon).
- Still in line 180, the equation in the second half of the line does not seem right either: the first term depends on gamma, whereas the second term depends on \epsilon. What exactly is going on there?
- In line 181 you are referring to some result from Theorem 1 - however, you are proving Theorem 1. (Presumably this reference is to the theorem in the supplementary.)
Nevertheless, the results are nice and plausible, and imprive significantly on previous bounds (although they are still somewhat worse than existing lower bounds); in fact, the novel concentration bound could be of interest on its own. Therefore, I would recommend acceptance given the issues discussed above could be taken care of in a convincing fashion.
Further remarks and questions:
- There does not seem to be a definition for D(z,x).
- How do you calculate L_i(t,\epsilon) and U_i(t,\epsilon)?
- In line 126: it seems that 1-\delta should be 1-2\delta.
- Line 142: the left bracket is missing in the term "D\mu,\mu)".
- In the line before line 142: "exp(\lambda S_t)" should be "exp(\lambda S_{t_j})".
- In the line below 135: S_t is not yet defined.
- In lines 194 and 195 you are referring to Fig. 5, whereas you only have Fig. 1.
- How close do you think the bounds are to the actual optimum of the problem?