A Drifting-Games Analysis for Online Learning and Applications to Boosting
We provide a general mechanism to design online learning algorithms based on a minimax analysis within a drifting-games framework. Different online learning settings (Hedge, multi-armed bandit problems and online convex optimization) are studied by converting into various kinds of drifting games. The original minimax analysis for drifting games is then used and generalized by applying a series of relaxations, starting from choosing a convex surrogate of the 0-1 loss function. With different choices of surrogates, we not only recover existing algorithms, but also propose new algorithms that are totally parameter-free and enjoy other useful properties. Moreover, our drifting-games framework naturally allows us to study high probability bounds without resorting to any concentration results, and also a generalized notion of regret that measures how good the algorithm is compared to all but the top small fraction of candidates. Finally, we translate our new Hedge algorithm into a new adaptive boosting algorithm that is computationally faster as shown in experiments, since it ignores a large number of examples on each round.
This paper proposes a new scheme for constructing and analyzing algorithms for online learning based on the equivalence of a broad class of online optimization games and the class of so-called drifting games. This newly found equivalence not only allows the authors to provide an alternative derivation for the well-known exponential-weights algorithm for prediction with expert advice and online convex optimization, but also leads to a new parameter-free learning algorithm called AbnormalHedge (a variant of NormalHedge originally proposed by Chaudhuri, Freund and Hsu, 2009). The authors also use their technique to prove strong guarantees on the so-called \epsilon-regret of the resulting algorithms, defined as the regret against the best \epsilon-quantile of the pool of experts/decision space. Finally, the authors apply AbnormalHedge in a boosting setting and highlight several useful properties of the resulting boosting algorithm.Originality & Significance--------------------------The results of the paper clearly constitute a substantial step forward in understanding potential-based algorithms for online learning. While this is not the first work to draw connections between drifting games and online learning, the general recipe of constructing near-optimal online learning algorithms by relaxing the minimax optimal strategy is novel. These results tie in nicely with the recent work of Rakhlin, Shamir and Sridharan (2012), who proposed a recipe for constructing online learning algorithms that work by relaxing the minimax strategy for the online learning game itself. The power of the proposed approach is clearly demonstrated by the simple and straightforward analysis provided for AbnormalHedge, which is especially elegant when compared to the complicated original analysis of NormalHedge. To the best of my knowledge, this is also the first time that the continuous exponential-weights algorithm is derived via a (relaxed) minimax analysis. The extension of the results to the multi-armed bandit setting -- while providing no spectacular new results -- can also serve as basis for interesting future work.Clarity & Quality-----------------The writing is superb, there is really not much to fix except a handful of typos (see below). The technical quality is also very good for the most part, the derivations are clean and seem correct. The only exception is Section 3.4, where the authors claim that their approach can yield high-probability regret guarantees without relying on concentration-of-measure results. Although I can see how this result follows from a modification of Theorem 2, it would be probably better to provide this proof at least in the appendix for completeness.Detailed comments-----------------212: \Phi -> \Phi_t in the expression of the optimal value of w. Same typo on line 229.212: "0-1 loss function" -> "the 0-1 loss function"234-235: This discussion is a bit difficult to follow. How does one "solve \Phi_0(0)<\epsilon" and how does such a procedure "give R > \underline{R}"?289: The logical expression in the indicator is a bit difficult to understand: how is a constant compared against an asymptotic expression? This should be rephrased.309: This result probably follows from an optimized setting of "a", which should be given in the text.381: The EWOO algorithm of Hazan, Agarwal and Kale (2007) precedes the work of Narayanan and Rakhlin on continuous EXP, and this is also probably a better reference for the trick used for proving Thm 6 than [7].443: The correct date for [5] is 2014.Reference---------E. Hazan, A. Agarwal and S. Kale: Logarithmic regret algorithms for online convex optimization. Machine Learning Journal 69(2-3), 2007, pp. 169 - 192.