Discovering, Learning and Exploiting Relevance
In this paper we consider the problem of learning online what is the information to consider when making sequential decisions. We formalize this as a contextual multi-armed bandit problem where a high dimensional (D-dimensional) context vector arrives to a learner which needs to select an action to maximize its expected reward at each time step. Each dimension of the context vector is called a type. We assume that there exists an unknown relation between actions and types, called the relevance relation, such that the reward of an action only depends on the contexts of the relevant types. When the relation is a function, i.e., the reward of an action only depends on the context of a single type, and the expected reward of an action is Lipschitz continuous in the context of its relevant type, we propose an algorithm that achievesÕ(T γ ) regret with a high probability, where γ = 2/(1 + √ 2). Our algorithm achieves this by learning the unknown relevance relation, whereas prior contextual bandit algorithms that do not exploit the existence of a relevance relation will haveÕ(T (D+1)/(D+2) ) regret. Our algorithm alternates between exploring and exploiting, it does not require reward observations in exploitations, and it guarantees with a high probability that actions with suboptimality greater than are never selected in exploitations. Our proposed method can be applied to a variety of learning applications including medical diagnosis, recommender systems, popularity prediction from social networks, network security etc., where at each instance of time vast amounts of different types of information are available to the decision maker, but the effect of an action depends only on a single type.
The paper deals with the problem of contextual bandits in the following setting: The context is composed of the product of D continuous sets, which are throughout considered (w.l.o.g) to be the interval [0,1]. The gain of an arm is assumed to be a random variable with mean equal to some Lipschitz function that depends on exactly one of the D intervals. The contexts are assumed to be arbitrary. In each round the player chooses an arm and gains its defined gain. The player may choose to query the value of that gain at a cost. Eventually, the objective is to minimize the regret defined as the sum of gains of the optimal strategy minus the sum of gains earned by the player plus the sum of exploration costs.The novelty w.r.t the setting is the sparsity of the function, meaning the fact that it depends on only one interval rather than all of them. This additional assumption is very natural and has been considered in other scenarios as well. Having it allows a significant increase in the performance of the algorithm. Specifically, it allows the authors to obtain a regret of T^c with constant c that is independent of the context dimension D (without the assumption there is a lower bound of c \geq (D+1)/(D+2)). In order to deal with the continuous regime, the authors use a (somewhat) standard technique of maintaining partitions of the interval with adaptively shrinking lengths. To deal with the sparsity of the gain function, the algorithm maintains, for each arm, estimates of the function for pairs of intervals originating from different intervals and determines the relevant interval (i.e., the interval that determines the value of the arm) by finding the one whose expected value is the same for all pairs containing it. Overall, the techniques are mostly technical but are definitely non-trivial.The authors provide experiments in a setting where algorithm is used to combine several different classifiers. These experiments do not do justice with the main contribution of the paper since the dimension of the context is chosen to be 3. I would expect the experiments to compare the current algorithm with those that do not assume sparsity, and an advantage would only occur for large values of D. In particular, for D=3, previous results provide a regret of T^{0.8} while this paper guarantees a regret of T^{0.83}.Comments:•	line 253: D should be |D|•	Line 361: The paper should state the exact dependence on D_{rel}•	The following paper could be relevant: Agarwal, Alekh, et al. "Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits.". In particular, it would be interesting to see whether the mentioned setting can be solved using their techniques as well as the resulting regret.•	Appendix, line 103: log(tAD/\delta) should be log(t^4AD/\delta)•	Appendix, line 161, missing L