Bounded Regret for Finite-Armed Structured Bandits
We study a new type of K-armed bandit problem where the expected return of one arm may depend on the returns of other arms. We present a new algorithm for this general class of problems and show that under certain circumstances it is possible to achieve finite expected cumulative regret. We also give problemdependent lower bounds on the cumulative regret showing that at least in special cases the new algorithm is nearly optimal.
Paper Summary:This paper treats a general multi-armed bandit problem in which the mean reward of each arm depends on a common unknown parameter. The authors consider a simple modification of the UCB1 algorithm. They show, unsurprisingly, that the algorithm satisfies a regret bound like that of UCB1. The main improvement of this paper is to show when the optimal arm can be identified perfectly by samples of the optimal arm, algorithm’s regret is bounded by a constant independent of the time horizon.Overall: The bounds provided are pretty strong, and the proofs in the body of the paper are clear and well organized. The explicit finite time analysis here is novel. My main issues with the paper are (1) it's already known in the literature exactly when finite regret is possible and (2) I worry that the algorithm that is considered is not appropriate for more practical structured bandit problems. If the paper is accepted, I think that these two points need to be acknowledged clearly. Quality and Clarity: This paper is well executed. The bounds provided are pretty strong, and the proofs in the body of the paper are clear and well organized. A few minor typos are listed at the bottom of this review. Originality[1] showed that finite regret is possible under the same condition as in this work, but otherwise regret must grow logarithmically with the time horizon. That paper is focused on providing an asymptotically exact characterization of regret and makes the technical assumption that the parameter space is finite. It’s mostly dismissed by the authors for this reason. But Graves and Lai later removed the condition that the parameter space is finite. There are also four existing papers on permutation bandits, a special case under which finite regret is possible.  In addition, the algorithm considered seems to have also been studied before, for example in [Mailard and Mannor; Latent Bandits; 2014]. The main original contributions are (1) to show that a simple modification of UCB is enough to attain finite regret, and (2) to provide an explicit finite time analysis of how large this finite regret may be. -Todd L Graves and Tze Leung Lai. Asymptotically eﬃcient adaptive choice of control laws in controlled markov chains. SIAM journal on control and optimization, 35(3):715–743, 1997Significance:I worry that the improvements are mostly relevant for some very simple structured bandit problems, and the algorithm considered is not appropriate for more practical structured bandit problems.The proposed algorithm is a very simple modification of UCB1. It maintains independent confidence intervals for each arm, and constructs a confidence set containing the parameters consistent with these intervals. This is nice, because it’s interesting that a simple modification of UCB is enough to attain finite regret. On the other hand, the proposed algorithm is in some ways a major step backward relative other UCB-style algorithms for problems with dependent arms. Consider, for example, a linear bandit problem. Even if no single arm were sampled many times, the confidence sets of (e.g. [Dani et. al; Stochastic linear optimization under bandit feedback;  2008]) could shrink around the true parameter because there is a known relationship among the arms. This is also true of algorithms in [10,18]. The confidence sets in this paper don’t fully exploit the known structure of the problem and therefore don’t share this property. Minor comments:-Lines 35-38: “This model has already been considered when the dependencies are linear by [17], but not as far as we are aware for the general non-linear case.” You actually cite two papers [1, 18] that consider the general non-linear case, so this is a strange comment. There is also a paper by Graves and Lai, and a Thompson sampling paper by Gopalan, Mannor and Mansour in ICML this year. - The motivating example in lines 45-49 is a strange choice given the preceding pitch, since arms are modeled independently in this example. - You sometimes slip up and write the true parameter as \theta instead of \theta^*. See for example Theorem 2 and its proof, or the proof of Lemma 5. - Similarly, you sometimes slip up in your notation and use \mu _i to denote the true mean reward of arm i when really this should be denoted by \mu_i(\theta^*). See e.g. lines 92-93.-In remark 4 you refer to a regime where n is large, but the preceding bound has no dependence on n. - This is a picky comment… in lines 240-241, you don’t assume concentration inequalities. You assume rewards are sub-Gaussian, which implies concentration inequalities that you can then reference in this step. - When referencing Lemma 5, you often use the expression given after equality ( e), so this should be in the statement of the result and not just the proof. - Line 272, equality (b) is an inequality. - In the proof of Lemma 7, given the current problem formulation it seems like theta-tilde and \theta are completely arbitrary. I believe you mean to specify that theta-tilde is in the confidence set, and to write theta^* instead of theta. -In line 324, the “if” after (b) is a typo.****************************************Post-rebuttal comments:****************************************This is a well executed paper that makes several technical contributions. I think it should be accepted.Let me apologize. The Lai + Graves paper doesn't seem to explicitly state that regret is finite, only that its growth rate is sub-logarithmic. So regret could grow as log(log(t)) for example. Still, the authors should clarify that the condition for finite regret -- "choosing the optimal arm is sufficient to prove that it is indeed optimal" -- plays a very important role in past work. Were this a fundamentally new insight, I would have given the paper a very enthusiastic review.