Combinatorial Multi-Armed Bandit with General Reward Functions
In this paper, we study the stochastic combinatorial multi-armed bandit (CMAB) framework that allows a general nonlinear reward function, whose expected value may not depend only on the means of the input random variables but possibly on the entire distributions of these variables. Our framework enables a much larger class of reward functions such as the max() function and nonlinear utility functions. Existing techniques relying on accurate estimations of the means of random variables, such as the upper confidence bound (UCB) technique, do not work directly on these functions. We propose a new algorithm called stochastically dominant confidence bound (SDCB), which estimates the distributions of underlying random variables and their stochastically dominant confidence bounds. We prove that SDCB can achieve O(log T ) distribution-dependent regret andÕ( √ T ) distribution-independent regret, where T is the time horizon. We apply our results to the K-MAX problem and expected utility maximization problems. In particular, for K-MAX, we provide the first polynomial-time approximation scheme (PTAS) for its offline problem, and give the firstÕ( √ T ) bound on the (1− )-approximation regret of its online problem, for any > 0.
Paper 903 generalizes the well-studied problem of stochastic combinatorial multi-arm bandit (CMAB) to the setting where the reward function is a Lipschitz - though not necessarily linear - function of the random realization of the arms. At each round, the algorithm selects a subset S - called a 'super arm - of m base arms, and receives a reward R() which is a Lipschitz and monotone function of the rewards of all individual arms. Like linear CMAB, the subset S is relies in some restricted class of subsets (say, [m]-choose-K), and the rewards of each arm assumed to be independent, and identically distributed over rounds.  The authors introduce an algorithm which obtains O(log T)-instance dependent and O(sqrt(T))-worst-case simple regret with respect to alpha * the reward of the best of simple arm S^*, which makes exactly one call to an offline alpha-approximation oracle per round. The paper establishes upper bounds even when each individual arm has an arbitrary, compactly supported distribution, which qualitatively match those for linear CMAB. Moreover, 903 introduces a PTAS for the offline oracle when the reward function is the max of all the arms selected - which they denote the K-MAX problem.
 All in all, the paper is well written, the proofs are clear, and the results are compelling. Though the algorithm is a natural extension of the linear CMAB algorithm in [Kveton et. al. 15], its key innovation is maintaining pointwise estimate of the arm's CDF, and pointwise lower confidence interval of the arm's CDF, which corresponds to the distribution of a random variable which stochastically dominates the given arm. This tweak makes the algorithm and its analysis easy to grasp, while still preserving connections to the lienar CMAB problem.   In what follows, we adopt the notation of the present work: T is the horizon, m the number of arms, K the largest cardinality of a super-arm, M the max-reward of a set, alpha the approximation factor, C the lipschitz constant of the reward function. When the distributions are finitely supported, they present an upper bound whose asymptotic alpha-regret in the limit of the horizon T-> \infinity does not depend on s, the size of the largest support of one arms distribution. Moreover, by appropriately discretizing the algorithm for finitely supported distributions, they extend their result to an upper bound for generalized CMAB with arbitrary (possibly non-discrete) distributions, which does not require foreknowledge of the time horizon.  Under the relatively mild assumption C / M = O(1), this upper bound matches the worst-case regret lower bounds for linear CMAB of O(sqrt(mKt)) up to a factor of sqrt(logT). This  reviewer encourages the authors to more fully elaborate how their guarantees compare with known lower bounds in their dependence in the parameter K, m, and T, so future readers can appreciate the strength of their result: i.e, for  general Lipschitz functions introduce no further worst-case statistical difficulty in the CMAB game than do linear functions.   Paper 903 gives detailed attention to the K-MAX loss function, where the goal is to select the subset of arms which maximizes Exp[max_{i in S} ] over all subsets S of size K. The work presents a PTAS for a 1 - epsilon approximation oracle for every epsilon, yielding an online algorithm for obtaining vanishing 1-epsilon regret which runs in polynomial time. The construction and analysis of the PTAS are very clever and original even though (as is true of many PTAS's). While the run-time of the algorithm ( O(m^O((1/eps^2)))) is still prohibitive for any large m and moderate epsilon (for example, in light of the Gollovin and Streeter work, it only makes sense to take epsilon < 1/e, which yields a run time of a high degree polynomial in m), the authors do include experiments which verify that the PTAS is successful in yielding vanishing regret. It would have been interesting to see a comparison in run-time between the PTAS and brute-force search for small values of K, m, and small discretizations.   Overall this is fantastic contribution: clearly written, original, and ostensibly correct. The only real weakness of the work is that the analysis is rather pessimistic, as the O(log-T) instance upper bounds only depend on the worst-case gaps between subsets, given by Delta_{min}. This gap dependence can be very, very loose. For example, consider the K-MAX reward function, decisision [m]-choose-K, and suppose all the arms are Bernoulli. In this special case, choosing the best subset of arms amounts to choosing those with the top-means. To make this concrete, suppose that there are K arms with mean mu + epsilon, and the remaining m-K arms have meaning mu. Then S^* is the top K arms,the gap between any arm i in S^* and j not in S^* is epsilon, but the gap in reward between S^* and any S \ne S^* is as small as Delta_{min} = (mu + epsilon)^K - (mu + epsilon)^{K-1}epsilon = epsilon(mu + epsilon)^{K-1}. Thus, the present works upper bound obtains a guarantee of the form O( m K log T/epsilon * 1/(mu + epsilon)^{K-1}). When mu + epsilon are bounded away from 1, 1/(mu + epsilon)^{K-1} = exp(-Omega(K)).  However, there are algorithms for top-K identification see [Kalyanakrishnan, Shivaram, et al. "PAC subset selection in stochastic multi-armed bandits." Proceedings of the 29th International Conference on Machine Learning (ICML-12). 2012] finds that the best subset of arms can be identified with probability 1-delta using at most O(log(K/delta) m/epsilon^2) samples. If we consider each pull of a super arm as one "sample", then [Best-of-K Bandits, Max Simchowitz, Kevin Jamieson, Benjamin Recht, COLT, 2016]. or [Top Arm Identification in Multi-Armed Bandits with Batch Arm Pulls, Kwang-Sung Jun, Kevin Jamieson, Robert Nowak, Xiaojin Zhu, AISTATS, 2016. PDF] can identify the best arm using O(log(K/delta) m/ (K*epsilon^2)). These can be turned into a low-regret algorithm with regret of roughly O( m Delta_{max}/K epsilon^2 * log KT), where Delta_{max} = (mu+epsilon)^K - mu^K, which is no larger than K * epsilon; and hence, the run time of adapting top-K identification is no worse than O(mlog(KT)/epsilon). Hence, in the regime where mu + epsilon is bounded away from 1, we see that the CMAB algorithm in this worst has guarantees which are exponential-worse than using top-K.   Obviously, there are sacrifices to be made for generality (indeed, the Bernoulli case is trivial and the innovation of this paper is that it works for non-Bernoulli distribution!). Moreover, I understand that even the linear CMAB upper bounds dont't have such a granular dependence on problem parameters. However, it would be nice to include further comments on the dependence on problem parameters, and potential looseness therein, in future revision. As an example of a highly-granular dependence on problem parameter, I recommend that you cite [Best-of-K Bandits, Max Simchowitz, Kevin Jamieson, Benjamin Recht, COLT, 2016]. The second half of that work provides upper bounds for the Bernoulli K-MAX problem under various partial feedback models (the first half is concerned with hardness results when the arms are not independent). While Paper 903's upper bound of O(mKlog T/Delta_min) can be exponentially (in K) loose for the semi-bandit feedback regime (as described above),  the upper bounds and lower bounds in Simchowitz '16 for Bernoulli K-MAX under bandit feedback (only observing the reward of a super-arm, rather than the rewards of its constituents) do in fact incur a dependence on so called "information" occlusion terms, which are related to the gaps in rewards between super-arms.   One other possibly area for improvement I seemed to find was that the algorithm for discrete distributions incurs an up-front (i.e., independent of T) regret of O(alpha * m * s), which seems quite hefty for short horizons. By adjusting the lower confidence bound in the line 12 of algorithm to look like O(sqrt(log (t * s)/T_i), it seems like one may be able to reduce this up-front regret to O(alpha * m) (i.e., no dependence on s), perhaps at the expense of a slight dependence on s in the asymptotic regret. I understand that the point was to have no dependence on s in the asymptote, but perhaps tweaking the confidence interval might allow for some nice interpolation. This may also yield improved analysis of the discretization for non-finitely supported distributions.    ------- After Rebuttal period ------- Have reviewed authors comments and maintain my initial review.