Stochastic Multi-Armed-Bandit Problem with Non-stationary Rewards
In a multi-armed bandit (MAB) problem a gambler needs to choose at each round of play one of K arms, each characterized by an unknown reward distribution. Reward realizations are only observed when an arm is selected, and the gambler's objective is to maximize his cumulative expected earnings over some given horizon of play T . To do this, the gambler needs to acquire information about arms (exploration) while simultaneously optimizing immediate rewards (exploitation); the price paid due to this trade off is often referred to as the regret, and the main question is how small can this price be as a function of the horizon length T . This problem has been studied extensively when the reward distributions do not change over time; an assumption that supports a sharp characterization of the regret, yet is often violated in practical settings. In this paper, we focus on a MAB formulation which allows for a broad range of temporal uncertainties in the rewards, while still maintaining mathematical tractability. We fully characterize the (regret) complexity of this class of MAB problems by establishing a direct link between the extent of allowable reward "variation" and the minimal achievable regret, and by establishing a connection between the adversarial and the stochastic MAB frameworks.
Secondary Review: The authors have replied to my concerns in a satisfactory manner. In particular, the problem with proof of Thm 1 can indeed be mitigated by the choice of epsilon suggested by the authors, although the second last inequality in their proposed derivation seems strange (the additional V_t^{2/3} would appear unnecessary) and, after noting that the "first part still holds" they can move directly to the case when \epsilon = 1/4 \sqrt{K/\tilde{\Delta}_T}. About line 424, their meaning is clear, however I see no reason to exclude the T^{1/3} term, as it provides information about the initial performance, and leaving it out all together would be misleading.Thanks to the authors for correcting their derivations, and I am happy to change my review scores accordingly.Initial Review:Summary of paper: The authors consider the multi-armed bandit (MAB) problem in a setting where the rewards are generated by non-stationary distributions. In particular they introduce a quantity ($V_T$ in the paper) which upper bounds the amount that the means of the reward distributions are allowed to vary by. They consider classes of MAB problems defined through admissible ranges for $V_T$, and they measure the worst-case regret of algorithms against a (non-stationary) oracle over these classes. They give a result lower bounding this worst-case regret of any strategy for these problems in terms of $V_T$, the time horizon, and the number of arms, and they give a matching upper bound on an epoch-variant of the Exp3 strategy. Thus they show a tight dependency of this class of problems on the quantity $V_T$, and motivate its interpretation as a quantitative measure of the cost of introducing non-stationarity into the standard MAB problem.Quality and Clarity: The paper is well written in parts, but not in others. In particular I found it hard to follow the details of the calculations in the proofs, and the many new notations. First two major issues:- It seems to me that there is a mistake in the proof of Theorem 1. Specifically on line 119 in the appendix I do not understand how the second inequality follows from the assumption that $T\ge KV_T$. Indeed, it seems to me that the numerator on the LHS of this inequality is greater than or equal to $T^{4/3}/8$ iff $(KV_T)/T \ge (1+\sqrt{\log(4/3)}/2)^3/2$, which is not implied by the assumption mentioned. I would appreciate it if the authors could clarify how the argument proceeds at this point.- I have difficulty in reconstruction the bound (a) on line 424 in the proof of Theorem 2. In particular, using the authors assumption that $V_T\ge K^{-1}$ I can bound the second term on the RHS of the inequality above (a) by $K(\log K)^{2/3}T^{1/3}$. Crucially, here there is a linear dependence on K. I would appreciate it if the authors could clarify how this becomes a dependence on $(KV_T)^{1/3}$. This last derivation is of great importance, as if it does not hold, then the bound is not tight with the lower bound provided in Theorem 1.I list also a few further suggestions that would have helped me to read the paper more fluently:- Some notation and notation and nomenclature differed from my experience of related literature, whereas it could be easily brought in line with it. In particular I found the "Admissible policies, performance and regret" paragraph did not introduce policies in a standard way, as they are introduced in many bandit and MDP papers (I think it is more normal to treat \pi as a distribution over the arm set that is measurable with respect to the history, and to denote by a_t, or I_t the chosen arm at time t).- The two paragraphs after Theorem 2 could be emphasised more since they are important and interesting consequences of the theorem and its proof.- I do not understand why the proof of Theorem 2 is in an Appendix within the main submission. It would be more natural for it to be a section before the discussion, or that it come immediately after Theorem 2.- The connection with reference [27] is often alluded to, but not in the end explained in any detail. It seems all of these allusions could therefore be in one place, for example the discussion section.Originality and significance: The contribution is original, and of significant interest to the NIPS community. It adds new understanding to the interface between stationary and non-stationary MAB problems. This should be of practical interest, although the authors do not directly suggest any real-world applications where the quantity $V_T$ can be interpreted.